You are correct to point out that Joyce's use of the stream of consciousness technique developed over his career. While there are only inklings of it in his early work, there is quite a bit of it in Ulysses.
Ulysses, however, is not the apogee of the stream of consciousness technique in Joyce. That spot is reserved for Finnegans Wake, a work in which Joyce carried the technique (and all of its attendant experimental devices) to the limit.
As to why Joyce used the technique, a probably unsatisfying answer is that it was simply something that greatly interested him. Over the course of his career, as his ideas about writing and his personal aesthetics developed, he used the technique more and more because it was in tune with those ideas and that aesthetics.
In particular, Joyce came to believe that following the automatic impulses of his own mind in his writing was exceedingly valuable. So he experimented with the technique to a greater degree. Further, he was adventurous and unafraid of potential failure. All of these sentiments are expressed in the following quote:
Emotion has dictated the course and detail of my book, and in
emotional writing one arrives at the unpredictable which can be of
more value, since its sources are deeper, than the products of the
intellectual method. In the intellectual method you plan everything
beforehand. When you arrive at the description, say, of a house you
try and remember that house exactly, which after all is journalism.
But the emotionally creative writer refashions that house and creates
a significant image in the only significant world, the world of our
emotions. The more we are tied to fact and try to give a correct
impression, the further we are from what is significant. In writing
one must create an endlessly changing surface, dictated by the mood
and current impulse in contrast to the fixed mood of the classical
style. This is ‘Work in Progress’. The important thing is not what we
write, but how we write, and in my opinion the modern writer must be
an adventurer above all, willing to take every risk, and be prepared
to founder in his effort if need be. In other words we must write
dangerously: everything is inclined to flux and change nowadays and
modern literature, to be valid, must express that flux. . . . A book,
in my opinion, should not be planned out beforehand, but as one writes
it will form itself, subject, as I say, to the constant emotional
promptings of one’s personality.
Conversations with James Joyce (1974), by Arthur Power (also quoted here)
While this quote is from Finnegans-Wake-era Joyce, and directly references that work, we can expect that Joyce held similar sentiments when writing Ulysses. This quote reveals why, in general, Joyce was fond of the stream of consciousness technique.