This is a vast subject; entire books have been written on the subject of coincidence in fiction. So I’ll attempt a very brief survey.
Were coincidence plots popular?
Charles Dickens was the most popular novelist in English in the mid-19th century, and his novels are full of coincidences. In his introduction to the 2002 Penguin edition of Oliver Twist (1837–9), Philip Horne listed a few of the more conspicuous ones:
The pocket picked by Charley and the Dodger when Oliver first goes out from Fagin’s house happens to be that of Mr. Brownlow, the oldest friend of Oliver’s father, and once in love with Oliver’s aunt (now dead), who happens to have on his wall a portrait of Oliver’s mother, which so resembles Oliver Mr. Brownlow is awestruck. When Sikes takes Oliver after his recapture to commit his second crime, at Chertsey, hours away from London, it turns out to be the house where Oliver’s other aunt, Rose Maylie, lives. Oliver’s father’s will, destroyed by the father’s wife, happens to have stipulated that Oliver inherits only if “in his minority he should never have stained his name with any public act of dishonour, meanness, cowardice or wrong”—such a stain is just what falling into Fagin’s hands puts Oliver at risk of; Oliver’s legitimate but wicked brother Edward Leeford (‘Monks’) happens to be the only one to know of this will, and to see and recognize Oliver, whom he has never seen (this time by his uncanny resemblance to their common father), on the one occasion he is away from Fagin’s (and rescued by Mr. Brownlow). Somehow Monks connects with, and somehow finds, Fagin, whom he employs to recapture and criminalize him.
Other novels that make use of coincidence were and remain immensely popular, for example Jane Eyre by Charlotte Brontë (1847; in which Eyre, wandering the moors, is rescued and taken in by the Rivers family, who turn out to be her cousins, whom she has never previously met) or Les Misérables by Victor Hugo (1862; in which Valjean, fleeing the police in the back alleys of Paris, meets the man whose life he saved many years previously in a distant town in another part of France).
There was a little contemporary criticism of Dickens’ coincidences. In 1842, Edgar Allan Poe wrote a long review of Barnaby Rudge, in which, among more pressing concerns, he touches briefly on coincidences thus:
There are many coincidences wrought into the narrative those, for example, which relate to the nineteenth of March; the dream of Barnaby, respecting his father, at the very period when his father is actually in the house; and the dream of Haredale previous to his final meeting with Chester. These things are meant to insinuate a fatality which, very properly, is not expressed in plain terms—but it is questionable whether the story derives more, in ideality, from their introduction, than it might have gained of verisimilitude from their omission.
For Poe this is clearly a question of balance: coincidences detract from the quality of verisimilitude, but add to the quality of fatality, and he values both qualities.
(Aside: Poe’s review refutes the idea, raised in comments, that “readers back then were much less nitpicky”. The review is a monument of nitpicks! But Poe’s nitpicks are directed at the effects Poe values, which are dramatic unity, which he thinks is spoiled by the inclusion of the Gordon Riots, and suspense, which he thinks is spoiled by the sloppy way the mystery plot is handled. Poe recognizes that coincidences are unrealistic, but realism is far from his most important concern.)
What did readers make of coincidences?
One way to think about this kind of question is to turn it around:
Reading 21st century literature, it is striking how often the plot turns on outrageous elements of fantasy. For example, in the ‘Harry Potter’ novels the characters cast spells, drink potions, and fly about on broomsticks. Did contemporary readers just accept this? Were they put off? Was the magic necessary to reassure them that it wasn’t real in some way?
Obviously contemporary readers understand that magic is a literary convention in the fantasy genre. We know that it’s not realistic, but since we don’t require fantasy novels to be realistic, that’s not a problem. Instead, we appreciate the dramatic or comic situations that that author can create using magic.
Coincidence does something similar in the 19th century novel: it’s an economical and effective means for the author to create dramatic or comic situations, and it “carries with it an element of surprise or astonishment that derives from the lack of apparent causal connection” (Thomas Vargish, The Providential Aesthetic in Victorian Fiction). If Oedipus kills a stranger, that’s homicide, but if by coincidence the stranger is his father, then that’s tragedy.
Vargish argues that early 19th century English culture was steeped in the Christian idea of ‘divine providence’, that is, a belief that the world is ordered by “a Supreme Being who foresees and controls human events for some divine purpose”. In this view of the world, coincidences are the traces of the divine plan, that rewards the virtuous and punishes the wicked.
I doubt it was true of all authors—surely like any trope it drifted from its origins—but Dickens seems to have thought about coincidence plots in this way. In an 1859 letter to Wilkie Collins he wrote, apparently in response to the latter’s suggestion about a change to A Tale of Two Cities to build up an event in a more convincing manner:
I do not positively say that the point you put might not have been done in your manner; but I have a very strong conviction that it would have been overdone in that manner—too elaborately trapped, baited, and prepared—in the main anticipated, and its interested wasted. […] I think that the business of art is to lay all that ground carefully, not with the care that conceals itself—to shew, by a backward light, what everything has been working to—but only to suggest, until the fulfillment comes. These are the ways of Providence, of which ways all art is but a little imitation.
What happened to coincidence?
It never went away, it just changed its appearance. Watch any television crime series and you’ll see that it’s commonplace for the murderer to be revealed as someone who coincidentally appeared earlier in the episode; or for the detective to see or hear something unrelated to the mystery that coincidentally suggests the solution; or for an innocent suspect to appear guilty because of a coincidental similarity with the criminal or the crime. Coincidence remains an economic way of creating drama.
But certainly literary novels no longer include the same kinds of egregious coincidence as Dickens, Brontë, and Hugo. That’s a consequence of the success of literary realism in the mid-to-late 19th century. Originating in France with the works of Stendhal and Balzac, realism was a reaction against the prevailing mood of romanticism. The progress of science, and especially the publication of On the Origin of Species in 1859, made it hard to hold on to the idea of a providential universe, and the rise of the social novel required a realistic treatment.
By the publication of Middlemarch (1871–2), George Eliot found it necessary to apologise for making use of a coincidence (by which the rogue Raffles happens upon a letter which he can use to blackmail Bulstrode):
As the stone which has been kicked by generations of clowns* may come by curious little links of effect under the eyes of a scholar, through whose labors it may at last fix the date of invasions and unlock religions, so a bit of ink and paper which has long been an innocent wrapping or stop-gap may at last be laid open under the one pair of eyes which have knowledge enough to turn it into the opening of a catastrophe. To Uriel watching the progress of planetary history from the sun, the one result would be just as much of a coincidence as the other.
* OED: “clown, n. 1.a. A countryman, rustic, or peasant.”
One has to go somewhat later to find full-throated denunications of the use of coincidence. George Gissing, an important realist novelist and critic, was scathing on the subject:
I have left it to this place to speak of the sin, most gross, most palpable, which Dickens everywhere commits in his abuse of “coincidence”. Bleak House is the supreme example of his recklessness. It seems never to have occurred to him, thus far in his career, that novels and fairy tales should obey different laws in the matter of incident. When Oliver Twist casually makes acquaintance with an old gentleman in the streets of London, this old gentleman of course turns out to be his relative, who desired of all things to discover the boy. When Steerforth returns to England from his travels with Emily, his ship is of course wrecked on the sands at Yarmouth, and his dead body washed up at the feet of David Copperfield, who happened to have made a little journey to see his Yarmouth friends on that very day. In Bleak House scarcely a page but presents some coincidence as glaring as these. Therein lies the worthlessness of the plot, which is held together only by the use of coincidence in its most flagrant forms. Grant that anything may happen just where or when the interest of the story demands it, and a neat drama may pretty easily be constructed. The very boldness of the thing prevents readers from considering it; indeed most readers take the author’s own view, and imagine every artificiality to be permitted in the world of fiction.
(Charles Dickens: A Critical Study, 1898.) For Gissing, realism is the most important thing, or maybe the only thing.