24601 has developed into being an iconic part of both the Les Miserables book and musical. Was that number special to him, or was it simply a random number he chose (I doubt it)?
Yes: it corresponds to the date of Hugo's conception.
This is part of a pattern of similarities between the character of Jean Valjean and the author himself: both are of similar age, have similar habits and similarly austere lifestyles, and even share the same dreams. This is according to David Bellos's The Novel of the Century: The Extraordinary Adventure of Les Misérables, which concludes Part Three with:
But Hugo put in a secret sign to remind himself and maybe those who knew him well that there were deep links of identity between the dutiful and persistent ex-convict and himself. Hugo was born on 26 February 1802, but because he was a slightly premature baby, he always believed he had been conceived on 24 June 1801. Valjean's prison number on his first incarceration at Toulon is 24601.
-- p. 162 (emphasis mine)
As a bonus, Valjean's second prisoner number was apparently also chosen for its special significance to Hugo: it refers to the month of his daughter Léopoldine's death. From footnote 46 in Kathryn Grossman's Figuring Transcendence in Les Miserables: Hugo's Romantic Sublime:
Gaudon demonstrates that this image, which represents genius in one of the poet's earliest carnets (8 July 1820), is a sign of his "deep identification" with Valjean (Temps 547). As further evidence that the hero is an alter ego, Gohin cites the two numbers assigned to him in prison. The first, 24601, refers to the supposed date of Hugo's conception (24 June 1801), while the second, 9430, alludes to Léopoldine's death in September 1843 ("Une histoire" 47; see also 52).
-- p. 165 (emphasis mine)