Palindromedary
*eyeroll*
I think the shift was one part players and designers wanting something different (because by the mid 80s D&D was already ten years old and for a lot of people there were only so many open-ended dungeons one could venture into), and one part not yet understanding that the difference they wanted / latched onto (comparatively rigid storyline play) was far more limiting in some ways (because it was new, after all, so its limitations hadn't really had the chance to be felt).
However, the big element for me is one last part: that the core intended purpose and gameplay loop of old D&D was so poorly explained in the rulebooks that there was lots of room for people to miss it. So after you get the mass hirings of new TSR staff in 81 and 82, just as Gygax begins stepping back from development duties, followed by the mass purges of 1984, you've got pretty much an entirely new design staff, including tons of people who largely were playing since the 70s but weren't part of the original Lake Geneva crew and so never got to learn firsthand how the game was intended to be played. They had developed their own styles of play, and a lot of those didn't mesh with the original playstyle.
One of the things that really sticks out to me as being obviously true but somewhat bewildering is how TSR never seems to have developed a House Style that was enforced at the corporate level. Perhaps because Gygax was big into the idea that each DM should handle things how they thought best, each designer really had their own ideas of how an adventure should be put together and that was readily evident on the page. TSR developed a common look and feel for their modules over time, but there seems to have never been an internal TSR document that said "this is the core purpose of D&D and, following from that, adventures should do these broad things and avoid these broad things": a sort of best-practices document. As such, as soon as he moves on (and really, even before) you get all sorts of random styles and approaches in modules, and the original style in particular is allowed to languish, because people have different ideas of how to play and because it's old and been done and there seems to be new and exciting methods of play popping up. That these new paths were often creative and gameplay dead ends wouldn't be apparent for a while (and to be fair would remain the preferred way for some regardless: there's a reason they've lasted this long no matter how much we tend to disparage them).
However, the big element for me is one last part: that the core intended purpose and gameplay loop of old D&D was so poorly explained in the rulebooks that there was lots of room for people to miss it. So after you get the mass hirings of new TSR staff in 81 and 82, just as Gygax begins stepping back from development duties, followed by the mass purges of 1984, you've got pretty much an entirely new design staff, including tons of people who largely were playing since the 70s but weren't part of the original Lake Geneva crew and so never got to learn firsthand how the game was intended to be played. They had developed their own styles of play, and a lot of those didn't mesh with the original playstyle.
One of the things that really sticks out to me as being obviously true but somewhat bewildering is how TSR never seems to have developed a House Style that was enforced at the corporate level. Perhaps because Gygax was big into the idea that each DM should handle things how they thought best, each designer really had their own ideas of how an adventure should be put together and that was readily evident on the page. TSR developed a common look and feel for their modules over time, but there seems to have never been an internal TSR document that said "this is the core purpose of D&D and, following from that, adventures should do these broad things and avoid these broad things": a sort of best-practices document. As such, as soon as he moves on (and really, even before) you get all sorts of random styles and approaches in modules, and the original style in particular is allowed to languish, because people have different ideas of how to play and because it's old and been done and there seems to be new and exciting methods of play popping up. That these new paths were often creative and gameplay dead ends wouldn't be apparent for a while (and to be fair would remain the preferred way for some regardless: there's a reason they've lasted this long no matter how much we tend to disparage them).