This post was prompted by an old Adventuring Party episode, which seems fitting (although less old than I thought it was when I first wrote this).
The premise was, basically, "are gamers too nostalgic? where is all the new stuff?" and based on a listener question. The group discussed this at length and with enthusiasm, and eventually settled mostly on "no" and "everywhere, on Kickstarter and stuff". While an interesting discussion, I felt like it didn't quite get to grips with things. I spent a lot of it on the tangent of thinking about why people tend to substantially gravitate towards the same handful of older games - D&D, World of Darkness, Traveller and Star Wars. Why isn't there a large market share for new shiny games?
(and yes, one of the answers is "it's questionable how much those should be considered old")
Most of my discussion will focus on D&D because I've played that.
Inertia
I think a large part of this can be attributed to general inertia. Humans are, for the most part, inclined to stick with what they're already doing, and some of the social issues around gaming specifically interact with that.
Conservatism is the basic point. Once you've got a game that does X, why learn another one? This tendency is intensified by the group-based nature of gaming. To play, you need to get a group of 3-6 people to agree on a game; it's not something you can just do yourself. For many people this means persuading their existing group (or one of just a couple of groups) to try the new system. In some groups, people are all willing to experiment with new games - it depends a lot on what the group is for, I suppose. There are game groups which are (explicitly or tacitly) about relaxing and socialising, and an old familiar system fits the bill well.
There are other factors which play in here. A group of laid-back players with an evangelical GM may be perfectly happy to have their choice of game driven by the GM's enthusiasms. Some groups are pretty stable, others are highly changeable depending on what game is currently being played; the latter are perhaps easier to crack into with a new game.
The social contract, or rather a social contract, reinforces this conservatism. When you do persuade them, if some people are reluctant in the first place because they don't see the point of changing from [game that does something similar], this tends to weigh against them changing their mind. If they've given it something approaching a fair chance, and still don't really buy into it, the social contract pushes towards going back to Old Game. That's down to the dynamics of things like how we approach favours and agreements - there's generally an unspoken sense that if Jane agrees to try MegaSpaceWhizz and doesn't care for it, you'll go back to Traveller.
Worse, if you've tried something and it wasn't a great success, there's a fairly strong pressure to never try it again.
This is particularly significant when you think that Old Game is probably played with the ease of long practice; people are comfortable with the rules and expectations and the kind of characters they can be. A first session of New Game, or even first few sessions, will be full of explanations and rules checking and mistakes, and people finding out they can't do something they want to.
It's also common for there to be a long and painful chargen process, not least because you normally have 1 copy of the rules and nobody knows them so you have to share. It's a very straightforward system, but I can make a Call of Cthulhu character without consulting the rules at all (okay, yes, I don't remember the exact list of skills or their base percentages, but I can remember many and I can certainly allocate all my points to stack above base). New Game will almost always seem painful, clumsy and slow in comparison.
Of course, New Game may also have cool and exciting aspects, like new character concepts and new powers and finding out you can do something very cool, but these tend to occur after you've already bought into it. How much this applies depends a lot on how much people already know about the game, the type of game, and how the GM presents it. A tie-in game or one that draws very strongly on specific tropes known to the players may strongly spark their imagination. On the downside, if players are not familiar with those tropes, it's a much harder sell.
To make things personal for a minute, I am really quite bad at pop culture. Luckily I'm pretty easygoing about this stuff, because I usually don't really know the tropes for games people want to run. My mates are also very good at quickly giving me the basic tools I need to pretend I know what I'm doing.* I am, however, quite familiar with a lot of D&D content because before I even got near it, I'd read plenty of generic fantasy, I played some D&D games and absorbed a lot of the setting and tropes. If you want me to play a new game inspired by Popular Zeitgeist Thing, I can almost guarantee I know nothing about it.
* Although I do have a tendency to interpret all settings as basically Call of Cthulhu
I don't want to paint this as a negative thing. People stick with old games they enjoy because they already enjoy them. Most gamers I know have a stack of ideas saved up for next time they get to run something - they're already excited with possibilities for doing more of the games they're already running. There isn't necessarily any need for something new.
First gamer advantage
The old games that people come back to get that for a reason. They normally do something well. There are other old games which are obscure or forgotten.
This is actually much more complicated than the standard "first mover advantage". It's not just about economics, though I'm sure that's an issue.
Old games had the huge advantage of an untapped well of sources to draw on. This is no longer true for any but the most specific of games. This may be a factor unique to games, as RPGs are a relatively derivative genre.
This meant D&D was able to draw on a huge range of fantasy, including swords and sorcery, weird, technofantasy, gothic, fairy tales and mythology, when establishing its setting and tone. This allowed it the great benefit of being flexible and adaptable; even if a GM chooses to run in an existing setting, individual elements can be highlighted, muted, subverted or ignored altogether, flavouring the in-game reality accordingly. Something as simple as tweaking the amount of background politics and treachery can shift a setting from "bickering city-states full of backstabbing nobles heedless of the evils around them" to "loose alliance of city-states keeping the torch of human civilisation lit in the face of darkness". It does tend towards a high-magic interpretation of the swords and sorcery end, but it hits enough notes from many of the others that it feels right. It covers Generic Fantastical well enough that it's a perfectly viable game to use for it. Anyone else coming into the scene now has to do something more specific, and thus more limited.
Well, they don't have to. They can try to create another Generic Fantastical game. It's an astonishingly difficult challenge, though, because there is already a game for that and it's very, very well-established, well-known, well-supported and widely-played. Trying to persuade people to swap out their well-worn-in game for one that seems mostly quite similar is a massive undertaking, because people have a lot of traction with their existing one and need a very strong reason to change. A vast mechanical improvement might do it, and this is what game publishers try to offer with new editions. The problem for a rival publisher is that in most cases you won't be able to offer anything too similar - copyrighted monsters, magic, setting details and so on. So you're asking people to change to either a game that's a bit like what they already do perfectly well, or a (hopefully) polished ripoff of the game they already play. And for the latter, even if it's better, people do tend to instinctively dislike ripoffs.
The cases I think you can sort of count as successes here are things like GURPS versions of particular genres, and this generally seems to work because people are fans of the system and want to run their favourite genres in it. But GURPS is, in its own way, one of the Elder Games. You can't make the new GURPS, although FATE is a similar beast.
The older, vaguely generic games are generally broad enough that they support a wide range of playstyles, preferences, tones and subgenres. This means you can use the same system, and much of the same content, to do different things. Part of the reason people don't seek out a new game to do XYZ based on newish sources is that they don't need to: they can often adapt an Elder Game that's in the right area, adding new content or throwing some out to produce the game they want. You can put content from The Witcher straight into a D&D game. You can grab ideas from modern sci-fi and insert them into Traveller.
D&D (and Vampire, and Traveller) also got to grab a lot of ideas from around the cultural sphere and be the first game that incorporates them. They've had a lot of time both to raid literature and film, and to evolve their own ideas. This just leaves slightly less breathing space for newer games. There's already a beholder, a mind flayer, a mimic, a gibbering mouther... a clan of occult vampires, a sect of highly religious vampires, vampires who want to tear down the secretive vampire social order. A lot of the cool ideas have already been used. You need new ones.
And, to be honest - older games got away with more. They were there first, and they got away with throwing together rag-tag bundles of whatever different people thought was cool with minimal explanation. Supplements and expansions threw in even more stuff that might count as part of D&D or Vampire or Star Wars. Over time, edges have been smoothed, weirdness has been explained away in various ways, and this stuff has taken on the mantle of making sense because That's Just How It Is. Your exciting new game doesn't get that leeway. It's competing with other games that seem to have coherent settings. If people decide it's a random set of what you thought was cool and makes no real sense, it's unlikely to succeed.
Limitations
One of the challenges facing newer games is that because old games are often so broad, they have to define themselves substantially by exclusion. It's very hard to create a broader game, or a substantially different mix of game elements that can't already be homebrewed by a GM, if everything conceivable is already part of the setting.
To take a D&D example, if you want to create a fantasy game that's postapocalyptic (maybe based on Shannara or something), you can just play D&D: there's the refuse of previous civilisations everywhere, there's a fair amount of technological elements. If you want to create a Gothic fantasy game, that's Ravenloft, but you can also do a Gothic-skinned pulp fantasy game by running D&D with lots of ghosts, skeletons, curses, hags, witches, shadows and vampires. If you want to create a folklore-based fantasy game, you can run D&D with a lot of fey and constructs and less in the way of dungeons.
Probably none of those are quite what you meant, but for a lot of players they'll scratch the flavour itch. Basically, the difficulty is you can't say "it's like Dungeons and Dragons, but with fey/post-apocalypticism/vampires" because those things are already in Dungeons and Dragons. And similarly, Vampire already incorporates almost any conceivable type of vampire and can be tweaked to run a wide variety of vampire subgenres.
What your new game needs to do is be more restrictive: it only has fey and fairy tales, or it only has Gothic elements; it specifically excludes orcs and elves and flesh golems and half the D&D PC archetypes and so on. This allows it to have a strong and distinctive flavour, but it's a harder sell, I think. Not least because quite a few players will think "sure, or I could run that by eliding a bunch of stuff from D&D".
Often what distinguishes fantasy, say, is not just the setting details, but the tone and the flavour of the world. But this tends to melt away in the face of a fairly generic setting, because D&D doesn't rely on any particular tone or flavour. Traveller can be a hard-boiled mercantile game, or a swashbuckling pirate game, or a heroic saving-the-Imperium game. Vampire can probably be a game of personal horror if you're determined, but it's more likely to be either Vampires Investigate or Vampire Buffy, let's be honest.
Common knowledge
What's more, this breadth is very welcoming to new players. A new player won't grasp the vast details of any D&D setting, but they don't have to - having some knowledge of some fantasy is a very good start. Almost everyone knows that elves are graceful and noble, orcs are monsters that you fight, and mysterious people in robes send you on quests. Of course, there's a lot of room for confusion if you've seen Lord of the Rings and you're supposed to be playing Fafrd and the Grey Mouser, but you can work that out in play based on what everyone else does - and importantly, deciding which of those stories you're playing is more down to the GM than to the game itself, which has room for both.
In other words, even if you aren't quite right in your assumptions about the game, it's likely that you can make some in the first place. This is not necessarily the case in newer games, which try to contrast by having novel content which prospective players aren't familiar with. If a player needs to do homework before they can meaningfully join a game, the barrier to entry is high. I know some games, like White Wolf, got a lot of buy-in while expecting players to embrace a lot of their very convoluted ideas, but it's rooted in popular conceptions, and it's notable that the weirder and more experimental games have been much less successful.
I own several newer games with their own unique settings and worlds. I haven't played any of them. I haven't had the energy to read through it all. The sole exception here are the Warhammer 40,000 line, and they are Elder Games by dint of their huge existing fanbase and canon.
Mechanics
The Elder Games tend to be quite robust: they have been played a great deal, and thus through weight of numbers and through market share they tend to have rigorous playtesting. They tend to be good at what they're doing. Their systems don't suit everyone, but they tend to work well at what they're trying to do. They have rules for everything they need. They have a broad array of tools for GMs, from antagonists to hazards to setting details. There are lots of people playing them who talk about them at length on the internet, as well as critics, which makes it relatively easy to check up on possible weirdness (how does this interact with that?) or gauge which elements will work well in your campaign. For example, if you want to know which D&D classes or GURPS advantages might end up causing problems for your specific game concept, it's likely someone somewhere has talked about it already.
Evolution
The Elder Games have been around for a long time, and there are certainly very old-established aspects to them. D&D isn't going to stop having classes or levels, for example, because there's a point at which it wouldn't feel like the same game any more. However, Elder Games have been able to reinvent themselves to suit changes in the market. Editions of D&D are quite different from each other. You can do quite a few different things without ever leaving D&D.
If you want a quick, brutal and gritty dungeon-bashing temple-raiding experience, you can play the earlier editions. If you want a fairly tactical experience focused on combat choices with powerful PCs, you can play 4th edition. If you want a much looser adventuring experience with powerful and often bizarre PCs, with a lot of player control, you can play 3rd edition. If you want something fairly smooth and simple but with survivability that supports long-term play, you can play 5th edition.
More generally, this means they have responded to changing tastes and preferences from the player base, and that means they haven't been left behind as you might expect from an older product. Sure, younger players may not be enthused for AD&D, but that's okay because they can play the new edition instead. Because there's an awful lot of continuity between editions (in terms of expectations, setting, monsters, magic and so on) it's relatively easy for players to transition between them both permanently and temporarily. Older players can shift to a new edition to join a group of younger players; younger players who started on 5th edition can get a rough handle on a 3rd edition campaign. For the most part it's still much easier to switching to a completely different game.
To a reasonable extent, some of those Elder Games are also New Shiny Games. They have the easy comfort of familiarity, together with the exciting promise of nicer rulesets, new cool stuff, and maybe even That Thing You Always Wanted.
New Game Blues
So having thrown out all that in explanation of the strengths of older games, what about the challenges for newer games?
Licensing
For the most part, as I said, RPGs are a quite derivative medium. They are an opportunity to participate in a story that's like that story you enjoyed. That means they usually draw on other sources - perhaps heavily mixed up and chopped about, but still.
The Adventuring Party mentioned games based on newer stories. I think one of the chief difficulties here is probably licensing. If you really want to publish a game that's very much like Space Captain Smith or Winds of Khalakovo, and that players will know is based on them, you probably need to license the property so as to reuse names (and preferably stick a big "The Space Captain Smith Roleplaying Game!" on the front). This was not true of many older games, since they were able to draw on stories that were both out of copyright (especially as it was much less absurb in those days) and widely known. The works of many pulp authors that inspired D&D were no longer in copyright, and many vampire stories had the same benefit. Of course, the games were substantially influenced by much newer works (we all know White Wolf's ideas of vampires draws heavily on Anne Rice, though not always directly) but the game could lean on the out-of-copyright stuff explicitly for familiarity value.
Licensing is expensive, especially for the kind of popular new hits that are likely to have a large potential audience. A good designer may be able to take an obscure subgenre and see the potential for a brilliant RPG, but Harry Potter is frankly going to get more copies sold. Not only is this a problem in an industry with poor margins, and a particular problem for a small underfunded company trying to create the next hit, but there's the permanent risk of losing the license. With that, you lose all your hard work and the future sales potential. After all, if you can create a game people are playing decades later, there's the potential for a substantial trickle-in income, further supplements (hopefully with a lower baseline cost now that the initial work is done), growing the playerbase and so on. All that helps to fund more work, improve the game and create a virtuous cycle.
Statistics
I think sheer numbers are a non-trivial part of the problem for new games. A high proportion of potential players are already playing Old Game that is at least superficially similar. As a GM, you have an idea for a campaign. It's probably easier to pitch your campaign as one run using Old Game which some people play, than New Game which nobody plays yet.
If I want to run something that's roughly a fantasy game, I'll probably run it in D&D. Investigative and low-combat? Call of Cthulhu. Turned up to 11 and fairly gritty? Warhammer 40,000. Modernish and not particularly gritty? World of Darkness. Not because those are the absolute best games to run those things in, but because they work, people are familiar with them and it requires minimal extra effort beyond convincing people the campaign would be fun at all.
Longevity
I think there's genuinely also an extent to which Elder Games are good at providing long-term entertainment, through a mixture of factors including the same flexibility and breadth I mentioned earlier.
Will your game provide long-term entertainment, or exhaust its possibilities quickly? Mission-style games or "thing of the week" can feel more like the latter. Long-term games tend to have some inbuilt progression, often a mechanical one of growing power that ties into escalating challenges. D&D isn't just fun because you can get XP and become more powerful, but because that progression allows you to move onto newer and greater challenges. Moreover, long-term play with the same groups of characters allows for greater investment and depth as characters develop over the course of play through choices, new bits of backstory and characterisation.
To take another example here, my current Planescape character has been fleshed out a lot over two years of play. Beginning as the simple concept of "Gap Yah student, but an elf" he's developed specific relationships with the rest of the party, long-term goals which clash with those of some party members, a homeland that has grown from a simple name to a relatively detailed society, actually grown up a bit, and seen his throwaway National Service background (as a ranger, he has favoured enemies) manifest in radical changes of persona when faced with abberations. In a game with much shorter arcs, this just wouldn't have happened.
In the Pathfinder campaign I was recently playing, there's a huge difference between punching an orc at low levels, and at high levels watching the frontline fighter get battered to a near-pulp by an abyssal monstrosity before striding over and using a single high-level spell to restore him to full hit points. Retreating from arrows, versus clearing a whole chamber with a storm of fire. Summoning a triceratops to trample enemies into the ground. Transforming into a gigantic elemental to breach the castle walls.
I'm honestly not very sure what the newer games are like that might be trying to do a similar thing. The new games I've tried are mostly indie things that expect one-off play. FATE supports gradual changes of character through tweaks to their Aspects, which I could see working. However, it also felt to me (in my very, very limited experience) as though the game would get samey over time.
As I said, people I know tend to have ideas for the campaigns they'd like to run next. But this is much easier in a broad game where you can do a lot of different things. If a game is designed for quite a narrow pool of experiences, it may well serve them very well, but it's also less likely that you'll want to keep playing it. And if you don't, you're no longer part of the current audience, which makes it that much less likely that this game is a Big New Thing Everyone is Playing.
Market Share
Market share for all media has splintered over the past decades. There is more of everything available, it's easier to find out about, and easier to find other people with similar interests.
When I was at school, people talked about The Programme that was on TV last night. This was easy. There were a handful of channels, very few programmes people our age might watch (between parents, mealtimes, homework etc.), and live broadcast only. If you had any interest in that sort of thing, you watched the show that was roughly the sort of thing you wanted. So everyone watched Thundercats, Teenage Mutant Hero Turtles and, to a lesser extent, Buffy. To a lesser extent because by that point, media was already shifting so there was much more choice of what to watch. You didn't have to watch Buffy because you could watch... some other programme instead, that did something similar but in a way you enjoyed more.
There are more books published every year. While finding a book that's exactly what you want isn't necessarily easy, it's more likely now that that book exists at all. People are spread more thinly, even people with quite similar tastes. They read different books, and so Jane A isn't particularly interested in Cyber-Regency Black Ops: the RPG because she's been reading a grittier set of books about Regency spies with more violence and less kissing. This doesn't mean she wouldn't like the game, it's just less likely she'll immediately find it appealing or even hear about it.
I suspect the similar interest thing also helps to create self-reinforcing bubbles. If you can readily seek out other people who want to play Old Games I Like, because you live in a large city or similar, you may just not really encounter newer games.
So there we go, usual unfiltered ramblings about why it's hard for new games to take over the world, hope it was interesting.
