First, we should note that Elizabeth is very specific here: she is only talking about her uncle and aunt, not of any other relatives. While it is perhaps natural that her thoughts should first go to those she was in company with, it is not so natural that they should not continue to her closer family. If it was a matter of shutting out all her family, then surely her father and Jane would be missed as much or more than the Gardiners.
So, why the Gardiners? They certainly seem like the most respectably behaved of all her relations we meet. The answer lies in the class system of the time, and what was considered a genteel occupation. In another passage, Elizabeth describes hers and Darcy's position in life as equal:
He is a gentleman; I am a gentleman's daughter; so far we are equal.
This is at least formally correct. While the staggering difference in income would likely have precluded such a match in practice, there were no formal difference in rank, partly because the incomes were still derived from the same sources: land, and interest.
One formal requirement of being genteel was how money was generated. Land and interest were the most unobjectionable sources of income, and what the country gentry that is the focus of most of Austen's novels lived on. There were also a few occupations that were also considered acceptable, suitable for younger sons without any hope of a grand inheritance: the army, the navy, the church, and law. These are the occupations of all the romantic interests we meet in Austen that has to earn their living. Anything else, and you were not strictly genteel. Medicine was on the rise in respectability, but not there yet.
Back to the Gardiners: We are told that Mr Gardiner is a merchant. This is not acceptable society to someone who is strict about this kind of thing, and Mr. Darcy even says that such relatives
[...] must very materially lessen [the Bennet sisters'] chance of marrying men of any consideration in the world.
The way out of this situation for someone who has made a large sum in trade is to buy a suitable country estate; this was the aspiration of the late Mr. Bingley, and the novel is kicked into action by his son attempting to do this very thing: it might take a generation or two, but eventually the stigma of having been in trade would disappear. By the standards set forth already by Mr Darcy and Lady Catherine de Bourgh, the Gardiners are of far too low a class to be acceptable company.
Thus, given that Mr Darcy has indeed show great pride before, Elizabeth really has no hope that he would entertain such relatives, when he had already shown how deeply he had objected to most of her immediate family, who at least formally belonged to the same class as him.
I should probably also say much of Austen's writing actaully is an examination of this class system: in Pride and Prejudice, the character that is perhaps the best behaved is Mr Bingley, who does not care very much for such matters and instead is generally kind and attentive. A large part of the redemption of Mr Darcy in the eyes of both Elizabeth and the reader is when he adopts such an attitude to the Gardiners and treats them as being of similar social standing.
I've used the essay "Class" by Juliet McMaster in The Cambridge companion to Jane Austen; apart form pointing out useful passages in the novel it also discusses the class system as it was during Austen's time and how she depicted it in her novels.