Learning from mistakes
It sounds like D&D 5 is going to at least try to avoid some of the pitfalls of 3rd Edition/D20 OGL, which is a good thing.
The first issue, which the link addresses, is that WOTC held onto the design principles and didn’t share them with the public (at least until the end of the lifecycle…). This contributed greatly to the amount of crappy D20 material that came out – part of the reason you’d buy a book of monsters, feats, or classes is that you’d want to put it into your existing game – but if they’re all mechanically unbalanced, it doesn’t make a good fit. By making the DMG a “hood pulled back” look at the mechanics, you help people not just hack the rules for themselves, but develop rules that others can use as well – they get a common language on what’s going on.
The second issue was the fact that open source development works because a lot of people can see how something works and at least SOME of those folks will have an idea of how to improve it. D20 fell down because most of the material was hidden – you’d have to pay for a full product to see the OGL material AND very often due to the licensing, people would keep the best bits for themselves and only put the weakest, blandest parts up for OGL.
Indie rpg scene, the OSR scene
Meanwhile, the indie rpg scene and later the OSR scene effectively DID use open sourcing as open sourcing. A lot of development for games was done publicly, a lot of playtest drafts were free to download. Games like The Shadow of Yesterday used the Creative Commons License to simply open the whole game up for reuse, and many other designers put it out there, “If you want to do a variant/supplement for my game, just email me and ask.” So we saw a lot of fast development there.
The OSR scene first used the OGL to reproduce a lot of clones of their favorite versions of D&D, but quickly started spinning off into their own games and development from that. Again, often a lot of these were free or accessible for folks to build on ideas and cross develop. Even things like E6 variant D&D developed under the same kind of logic.
The WOTC Challenge
Probably the easiest path for them would be to create a version of the Creative Commons License – let fans make stuff, but not charge money for it. It fulfills the original goal of the D20 OGL – get people making material that feeds back to D&D, but it also avoids a lot of the pitfalls about paywalls. There’s probably also something to be said about competition, especially in the post-Pathfinder era, though I don’t think that’s actually a real issue in the long run.
I know there’s the usual cry of “but if no one can charge money, no one professional will ever do this and quality will suffer!” but we can look at a lot of the OSR stuff being works of love rather than profit, and being successful despite lowered production quality. Likewise, one can look at the amount of fanfiction which apparently has been decent enough to become transformed into professional bestsellers with the names scraped off, so there’s plenty of examples of “good enough to people to love it” from fan work.
The less easy path, if they wanted to take closer to the D20 OGL route, where people could create products and charge money, would be to have a central database or wiki of all the open source material so anyone could see it and work with it. That said, you still have to figure out how to make sure it’s the best stuff coming in and not the dregs of any product, as well as dealing with the issue of maintenance. (Any platform the public can have input on, must be constantly handheld or it fills with violent racist, misogynist, homophobes and penis pictures as a basic law of the internet…)
The 4E route of “pay to license” is not going to work and we can see how much of 4E kinda sat on the edge with fan creations. To be sure, there was hacks and stuff posted, but we can probably say that a lot of 2nd tier publishers who were putting out solid material for D20 decided to stick with what was working for them rather than deal with the hot mess of confusion and restriction the 4E license provided.
In many ways, I wonder how much of Pathfinder came out of a love of d20 as opposed to Paizo seeing 4E’s license as being unworkable. It’s sort of like how Windows puts out a crap version every other time and no one upgrades because it’s easier to stick with what you’ve got than to deal with the hassle. (mind you, I’m talking about the license, not the game itself. I think 4E had a lot of great design choices in it, and, would have been one of the better systems for 3rd party support/hacking, because of the modular nature of powers…)
The Indie Lesson of Fan base
One of the better things that came out of the 2000′s Indie rpg scene was the understanding of how to better engage with your game fan base. People were playing games that were long out of print, with nothing new coming out and nothing in the news, but having a passionate community still playing and pushing it. Ron Edwards pointed out that play communities mattered more to rpgs than regular publishing cycles. We can see that is true with ever re-release Kickstarter of an old RPG which does exceedingly well because the fan base is still present.
So how do you best engage with that fan base?
Roleplaying games are a creative activity, so it’s no surprise that gamers want to share their ideas as well. This doesn’t mean anything has to become officially part of your game, but it does mean you need to allow spaces for it and to some level, encourage it. To a great deal, we can see this is what Enworld and some of the blog spaces on the WOTC site have become. Getting that to flourish in more spaces becomes more ‘free advertising’ for your core game.
The other one which is still relatively underutilized is cross promoting other games that you also think are good. Tabletop rpgs are too small of a hobby to act as if you gain some competitive edge by pretending no other games exist. Not only does cross promoting help grow/maintain the hobby by making sure people can find a game that does what they want (even if it’s not your game), it also sets up a reciprocal space of gaming promotion. Not everyone is going to promote your game in return, but enough will and it makes a big difference. “One True Way” logic has hurt roleplaying significantly as a hobby, and remains one of the big roadblocks to D&D groups forming and continuing with play. If D&D gobbled up every other tabletop RPG company’s profits from actual games? It’d be nothing to them. What you want is more gamers overall, and it makes more sense to promote tabletop roleplaying as a whole, rather than fighting for crumbs.