The following paragraph is from Charles Dickens, Little Dorrit, Chapter 27.
It was now summer-time; a grey, hot, dusty evening. They rode to the top of Oxford Street, and there alighting, dived in among the great streets of melancholy stateliness, and the little streets that try to be as stately and succeed in being more melancholy, of which there is a labyrinth near Park Lane. Wildernesses of corner houses, with barbarous old porticoes and appurtenances; horrors that came into existence under some wrong-headed person in some wrong-headed time, still demanding the blind admiration of all ensuing generations and determined to do so until they tumbled down; frowned upon the twilight. Parasite little tenements, with the cramp in their whole frame, from the dwarf hall-door on the giant model of His Grace’s in the Square to the squeezed window of the boudoir commanding the dunghills in the Mews, made the evening doleful. Rickety dwellings of undoubted fashion, but of a capacity to hold nothing comfortably except a dismal smell, looked like the last result of the great mansions’ breeding in-and-in; and, where their little supplementary bows and balconies were supported on thin iron columns, seemed to be scrofulously resting upon crutches. Here and there a Hatchment, with the whole science of Heraldry in it, loomed down upon the street, like an Archbishop discoursing on Vanity. The shops, few in number, made no show; for popular opinion was as nothing to them. The pastrycook knew who was on his books, and in that knowledge could be calm, with a few glass cylinders of dowager peppermint-drops in his window, and half-a-dozen ancient specimens of currant-jelly. A few oranges formed the greengrocer’s whole concession to the vulgar mind. A single basket made of moss, once containing plovers’ eggs, held all that the poulterer had to say to the rabble. Everybody in those streets seemed (which is always the case at that hour and season) to be gone out to dinner, and nobody seemed to be giving the dinners they had gone to.
What does "nobody seemed to be giving the dinners they had gone to" mean?