I think the fundamental issue is that, after Gygax, nobody paid attention to the process of DMing, or the way procedures drive play. And Gygax never really described how he handled out of dungeon play. So when non-dungeon adventures came into vogue, designers just puked their ideas up on paper. So you get a wall of text trying to either create a railroad or describe every possible outcome, instead of short entries with procedures that help you to improvise.
Why do they keep writing them this way? Because very few people are writing better modules for this playstyle; and those that do are relatively obscure and aren't really competition. The WotC stuff continues to sell, Hasbro is risk averse and seemingly always happy to fire people, so who is going to take the risk of experimenting with other approaches?
I think customers started buying modules to collect and read them more than play them as well. A pretty product with a cool story became more desirable. Modules became aspirational. I remember my friends, who were players rather than referees in the group buying adventures. I remember when I had a couple of dry years where I couldn't scrape together a group, buying adventures anyway. Playability wasn't necessarily the first criterion. I know I wasn't alone in that buying habit; the corporation followed the money.
@bryce0lynch always says this too, but I find that type of module completely unreadable.
my guess based on what I've read and also observed about WotC is that module writers basically follow Steve Brust's Cool Stuff Theory of Literature
That would be fine if they wrote like Brust.
For me I just ask up-front what the game is "about" from the 8 Kinds of Fun perspective (
https://theangrygm.com/gaming-for-fun-part-1-eight-kinds-of-fun/, best article AngryGM ever wrote), and then decline based on that instead of getting into a semantic argument about what "D&D" means.
No, the best articles the AngryGM ever wrote were his early skill system and adjudication articles. Unfortunately, the last time I tried to look at them it appeared the webpages had been hacked.
It's some kind of horrible Stockholm Syndrome. I've been running this open-ended sandbox for a few months and it's been a fight to get the players to make their own choices. Their conditioning is deep-set. They're constantly looking for narrative hooks and much happier once they're on the railroad (of their own choosing).
I mean, then they go to great lengths to sabotage the railroad. But they needed that framework there. They needed a box to think outside of.
Yeah, there need to be perceived constraints or they get completely lost. Basically, you need hooks, and the hooks need to either have obvious paths of investigation associated with them (say, because the hook is a trope that invites a particular type of action), or the hooks need to have the path of investigation built in (for example, a treasure map).
I note using the AD&D treasure tables, if magic items were indicated you had about a 10% chance of getting a map. That's an early hook. I don't like to be that obvious, but I often include hooks in treasure - say, a deed to property, or treasure with a history that will get noticed when you sell it (B1 did a lot of that). Finding one part of the Rod of Seven Parts is a hell of a hook.
I also think sages are badly underused, and information-providing NPCs are generally used poorly. Modern modules have NPCs dispense information, instead of suggesting how PCs can find information. Like, what is going to drive play more, telling the players some relevant history, or telling them that the answers lie in the diary of the Dread Pirate Roberts, which you heard was buried in his tomb in the Saltmarsh Cemetery, at least until the Yellow Skull Gang robbed it last year? A common fault of modern modules is they don't encourage players to play actively.
Also (and this just occurred to me), it would probably help if you had useful elements introduce themselves. I get frustrated that players don't take the initiative and seek out purveyors of information, but it occurs to me that information brokers want to find clients, and low level information brokers can't be too choosy; they might seek out adventurers and hint that they can be useful in finding stuff out. The concierge at the fancy hotel might make it
really clear that they can hook you up with all kinds of bizarre and illegal stuff. The local thief taker might advertise his ability to find people and things.
Hell, maybe you just tell the players that they know people with information. The cleric knows that temples and monasteries tend to have collections of historical and theological works. The wizard knows where the sages like to hang out. The rogue knows how to ask around to find fences and the like. The fighter knows who the court gossips are.
