Tom Bombadil is definitely one of the weirdest characters in all of The Lord of the Rings, he just comes out of nowhere, stumbling upon Frodo and Sam while singing. His actions are just incredibly unusual compared to the other characters in LotR. My question is for what reason does Tolkien add Bombadil into his story? How is he important to the development of the story?
Bombadil is indeed an anomaly, and does not appear to fit very well into Tolkien's overall narrative. He comes from nowhere (although he has been there all along, unobserved), and disappears equally abruptly from the story. Unless the author has given us some hints about the character's relevance, we are reduced to speculation, so let us just consider the mechanics of the story.
Tolkien has to get the four hobbits from Crickhollow to Bree, where they will meet Aragorn, unharmed, and without Gandalf to protect them; and yet they need to have some adventures on the way, otherwise the story becomes uninteresting. Gandalf's absence is Tolkien's biggest problem, and so Bombadil is basically a (slightly unreliable) stand-in for Gandalf.
The hobbits are in no position to face the nine riders on their own. Neither can they just walk through the Old Forest and have a pleasant hike to Bree. So, we have Old Man Willow, which Bombadil rescues the hobbits from, and then the barrow wights, where he rescues them again, and in between these two escapades, a pleasant interlude in Bombadil's house, with Goldberry and plenty of food and rhyming verse.
I have a suspicion that Tolkien had some attachment to Bombadil. Maybe Bombadil represents the author, since he is older than any other living thing in Middle Earth, and is unaffected by the power of the One Ring, and presumably, he is also unaffected by the destruction of the ring at the story's climax. Only an author has this much power to do as he pleases in his own stories. Tolkien is, perhaps, pulling a Hitchcock on us, since the director liked to make cameo appearances in his own films.
In the Council of Elrond, Gandalf is right when he says that they cannot give the Ring to Bombadil, since he would probably just lose it. With the Ring in his own pocket, Tolkien loses the plot.
Tolkien said that Bombadil represented a sort of passive pacifism, which was important to represent in the story but couldn't play much of a role in the actual plot.
From Tolkien's Letters, letter #144:
Tom Bombadil is not an important person – to the narrative. I suppose he has some importance as a 'comment'. I mean, I do not really write like that: he is just an invention (who first appeared in the Oxford Magazine about 1933), and he represents something that I feel important, though I would not be prepared to analyze the feeling precisely. I would not, however, have left him in, if he did not have some kind of function. I might put it this way. The story is cast in terms of a good side, and a bad side, beauty against ruthless ugliness, tyranny against kingship, moderated freedom with consent against compulsion that has long lost any object save mere power, and so on; but both sides in some degree, conservative or destructive, want a measure of control. but if you have, as it were taken 'a vow of poverty', renounced control, and take your delight in things for themselves without reference to yourself, watching, observing, and to some extent knowing, then the question of the rights and wrongs of power and control might become utterly meaningless to you, and the means of power quite valueless. It is a natural pacifist view, which always arises in the mind when there is a war. But the view of Rivendell seems to be that it is an excellent thing to have represented, but that there are in fact things with which it cannot cope; and upon which its existence nonetheless depends. Ultimately only the victory of the West will allow Bombadil to continue, or even to survive. Nothing would be left for him in the world of Sauron.
In Letter #20, Tolkien also reveals the out-of-universe inspiration for Bombadil:
Do you think Tom Bombadil, the spirit of the (vanishing) Oxford and Berkshire countryside, could be made into the hero of a story? Or is he, as I suspect, fully enshrined in the enclosed verses?
The other answers have some good analysis of Bombadil and his role and purpose, but I thought it would also be useful to have what the author himself said about his intentions in introducing the character.
There is also some existing discussion of this and other related issues on the Science Fiction & Fantasy Stack Exchange, which I checked before posting this answer:
This is opinion: I have no way to back it up.
I believe Tolkien introduced Tom Bombadil as a symbol of mankind before the fall, a sort of creature of pure, unadulterated nature. (Others have speculated this as well.) Think of him a bit like Adam, of the Adam and Eve in Genesis, before it all went south. He lives in harmony with nature and has some control over it, which makes sense because nature was a thing created for man's use and there was no enmity between man and nature before the fall. He's peaceable and happy because he's not tainted by evil or regret - he knows what evil is but he's not affected by it. He's unlike other characters because, well, no other character is like him - unfallen and unaffected by brokenness. Of course the ring has no effect on him, and neither do wights. Tolkien, a Christian, was (I think) making a point about the limits of the power of evil, and introducing a point of hope and optimism (here is something evil will never win against) in a story that then gets bleaker before it gets better.
It's important not to push the analogy too far: T.B. might be a symbol of the purity of the original creation, or Adam himself, but not a Jesus or God one. Tolkien was clear about that; this link might be of interest: http://www.thetolkienwiki.org/wiki.cgi?FAQ/Who__is__Tom__Bombadil
In LotR there are other echoes of the idea that someone who has never done X (X some bad thing) cannot be affected by X. For T.B, X is evil itself.
Tolkien has to handwave a bit during the Council of Elrond because of Bombadil - people propose just giving the ring to Bombadil and calling it a day, but Gandalf says no. I always thought the explanation was weak; why would Bombadil be so carefree and detached as to forget he had it? But I think I see the opposing problem - if people could just pass every awkward, evil bit of magic to Bombadil, he'd end up as the caretaker of a growing mountain of other people's dark efforts and failings. It wouldn't be a peaceful existence - there would constantly be evil characters going to Bombadil's land to grab the bad stuff. Bombadil, the ultimate symbol of freedom from the effects of evil, would get trapped in the role of taking care of everyone else's evil schemes and follies. It's no way to live, so Bombadil simply doesn't. He withdraws instead, only joining the world when he sees value in it to others, such as bumbling into the hobbits when they need help.
Bombadil is also part of the world-building. He's part of the world, and his whole purpose in it is something that nobody short of Eru fully understands.
Tolkien mentions things like this in passing. Gandalf, after his returning from his trip Outside, mentions some creatures at the very nadir of his fall in Moria:
Far, far below the deepest delving of the Dwarves, the world is gnawed by nameless things. Even Sauron knows them not. They are older than he. Now I have walked there, but I will bring no report to darken the light of day.
The maps that Tolkien made include a lot of places the the hobbits never see, and some that are not even named in the narrative.
It's all done to give the reader a picture of a world that is not only larger than the Shire, but is also far older.
Do you think it's a coincidence that the very oldest resident of Middle-Earth (aside from Maiar), and by far its biggest enigma, dwells within a day's walk of where the story begins and ends?
I'm reading Fellowship (again) right now, and I had a similar thought when I reached the Forest/Barrow section, and I thought for a long while why it might be.
I found my answer ('my' rather than 'the' because it's just an opinion) when the Council are discussing what to do about the ring. Bombadil is suggested as a solution - in fact the suggestion veers towards the idea of 'weaponising' Bombadil, then towards him keeping the Ring within his protection/seclusion, but both are dealt with swiftly as impossible. I (chose to?) interpret this as Tolkien's in-line explanation of why greater entities (Bombadil, or the Valar, or a greater collection of Maiar) in general couldn't or wouldn't just resolve the issue themselves. This is probably dealt with extensively elsewhere in the Histories, or Silmarillion, though I don't remember where, but if you take the Lord of the Rings series as a set by themselves without the rest of the legendarium, it seems to me to give the satisfactory 'answer' for why the West doesn't just come back and go to open war with Sauron.
The story of the Ring is, after all, a vehicle for the closing of the Third Age and the beginning of the dominion of Humankind, so it makes sense (to me) that a more powerful being choosing a pacifist stance is included. The Elves are already departing for the West, and the remaining Elves in Middle-Earth are already but a small number of the previous population.
Bombadil's existence (nor that of Goldberry) is never really 'dealt with' at least not in my recollection (this read-through is after maybe 10 years since I last read the series) so one can only assume they sort of just fade into the background, not being corporeal any more. I have wondered before whether the 'feeling' that Tolkien mentions in the letter #144 (quote above) is due his environment at the time (wartime and after) where simple enjoyment of things is basically non-existent because everyone is so preoccupied.
One reason that Bombadil does not really "fit" in the narrative may be that The Lord of the Rings was originally a sequel to The Hobbit. At that time, The Hobbit was only indirectly connected to the world of the Silmarillion and was "only" a children's story with ogres speaking in cockney accents and other silly things.
So the adventures until the hobbits reach Bree still very much have a feel of that type of children's story and Tom Bombadil actually fits very well into that type of story.
As Tolkien further developed The Lord of the Rings and rolled both it and The Hobbit into the mythology of the Silmarillion, the story became much more serious and "adult", but some of the earlier elements were left in place, even though they don't have quite the same feel as the items developed later.
Quality literature is rarely about the plot, but instead the plot serves to deliver a message (and perhaps to engage the reader long enough to get the message). Other elements besides the plot also serve to deliver the message (e.g. places, names, auxiliary characters, etc.)
The Lord of the Rings is full of messages - about Truth, war and peace, virtue, character, bravery, honesty, commitment, greed, etc. A key message of the work is the archetypal battle of good versus evil. Evil is hell-bent on destroying all that is good: how will it be defeated? How will the innocent be protected?
As @Rand al'Thor explains in his answer, Bombadil is an important part of the message: good triumphs over evil, not through an escape to innocent pacifism, but by honest individuals standing forth with courage give and risk all they have and are. These message is repeated as the Hobbits leave the Shire, as the Ents rally against Isengard, as Rivendale supports Rohan at Helm's Deep, as Rohan responds to Gondor's call for help, etc.
There is something beautiful and refreshing about a complete release of the cares of the world (Bombadil), but the absence of Bombadil from the rest of the story IS a deliberate part of the message: evil is not defeated by innocence - we must proactively fight against evil in order to establish peace.