1. Professor Moriarty
I'd like to go into Moriarty in more detail, because Worth was not the only inspiration for Moriarty.
Doyle did not simply draw on Worth for inspiration. Masters of Crime states that Doyle himself only acknowledged being inspired by one criminal: Jonathan Wild, a 17th and 18th century master of the criminal underworld. In The Valley of Fear, he is even referenced by Holmes:
Everything comes in circles—even Professor Moriarty. Jonathan Wild was the hidden force of the London criminals, to whom he sold his brains and his organization on a fifteen per cent commission.
The evidence pointing to Worth as another inspiration is largely circumstantial but quite apparent. He was once referred to as "the Napoleon of crime" by a Scotland yard detective (see Masters of Crime), a phrase Doyle my have appropriated for Moriarty. Like Moriarty - and like Wild - he was famous for creating a large criminal network in London to which he could seldom be traced.
Adler, who appears in A Scandal in Bohemia may have been inspired by several real-life singers and courtesans. The New Annotated Sherlock Holmes lists quite a few, including Lola Montez (mistress of Ludwig I of Bavaria, which hearkens to the story), Lillie Langtry (whose career is similar to Adler's), Pauline Lucca, Laura Bell, and others. The Doctor and the Detective also mentions one Ludmilla Hubel, whose love affair with Archduke Johann Salvator parallels Adler's relationship with the King of Bohemia. Viktor Adler, an Austrian politician, may be the source of the name "Adler".
3. Charles Augustus Milverton
Doyle's famous blackmailer, who appeared in The Adventure of Charles Augustus Milverton, bears a striking similarity to the 19th century Charles Augustus Howell in both name and occupation (blackmailer and art dealer). London After Midnight states the Doyle himself admitted as such, although I'm still trying to confirm this.
4. John Clay
It's also worth noting that Masters of Crime, which drew parallels between Moriarty and Wild and Worth, same book states that John Clay, the hidden antagonist of The Red-Headed League, can be similarly traced. He shares characteristics of Worth and a criminal rival of Worth's named Max Shinburn - even though Doyle did not, at the time of the story's publication, know about either of the two. The author describes Clay as a "dummy run" for Moriarty, although Doyle could not have had Worth in mind at the time.
It seems possible that Doyle based Clay on Wild, though. The obvious similarity is the ability to hide behind accomplices, distance oneself from the crime, and avoid capture, which Clay has, according to Holmes, used quite often in the past. Both, too, seem to have spun vast webs - perhaps not quite at the level of Moriarty - covering a wide range of illegal activities, from petty theft to murder.