Dr. Peter Kreeft (Catholic apologist) speaks about his love of Tolkien in this interview starting at 14:10. I post it here because we've gone back and forth about Bombadil several times, with the majority disliking his inclusion on the LotR.
Kreeft starts speaking about Bombadil at 19:00. Interesting is the interviewers take (siding with the anti-Bombadil crowd) versus Kreeft's---which beautifully articulates my feelings on the matter.
I have no issue with Bombadil as a concept, or his existence in Middle-Earth; the Bombadil poems are fine. My issue is with the pointlessness of his appearance in LotR. In fact, I see his as am impediment to good storytelling, because not only does nothing happen - the main characters accomplish nothing, and learn nothing - but he actively removes their agency.
The hobbits get caught by Old Man Willow, are rescued by the Hand of God, and learn and accomplish nothing. The hobbilts get caught by the barrow-wights, are rescued by the Hand of God, and learn and accomplish nothing. IIRC, they don't even find Merry's magic sword on their own, he hands the swords to them, with them having done nothing to earn them. The whole thing has about as much function as "Merry finds a magic sword in a field."
I understand how Bombadil first got there, because as I understand it he was telling these stories to his kids as a serial, and a random episode works fine in that context. So he added a crossover with a old character AND his rogues' gallery, who (as with many poor crossovers and sequels) are up to the exact same tricks they were in the original poem, and defeated by Bombadil in the same way (he may have even had the same dialog). I think the work would have been better if he had the discipline to excise an element that added nothing to the overall work, or publish it as a separate short story, or shove it into an appendix like Arwen's story.
"...if they come from the concious mind..." is a major part of what's wrong with much Adventure-writing/fiction in my opinion. The creatively is too predictable and meditative, and as a result feels too tidy and artificial.
I agree with this assessment. WotC adventures are formulaic, antiseptic, and have a far too 21st century sensibility for what amounts to, at the latest, an early modern or 19th century political, social and technological environment. They exist in a clockwork universe in which every element has a logical explanation
which must be explained to the DM, even if the logic is a bit dodgy, and eschews any mystery. The OSR's focus on the mythic underworld is a reaction to this, if nothing else.