Central to your question is: what is the definition of "literary fiction".
The general definition is: fiction that has literary merit. What that means is that it will hold up under re-reading, and that it has many of the characteristics that make it art rather than merely entertainment. What are these characteristics? The list undoubtedly varies between literary critics, but some generally-agreed on characteristics are
- well-drawn characters,
- an elegant or interesting writing style,
- an exploration of larger themes, such as the human condition, or the nature of love and/or friendship,
- interesting and vivid imagery,
- the use of symbolism, metaphor, or allegory.
There is no question that Latin American magical realism is considered literary fiction — for instance, consider Gabriel García Márquez's One Hundred Years of Solutide, Isabel Allende's The House of the Spirits, and Laura Esquivel's Like Water for Chocolate. These all have fantastic elements, these are all great books, and they're all considered literary fiction.
So why do some critics refuse to admit that English-language novels with fantastic elements can have literary merit?
Literary critics, like any community of human beings, are subject to fads. For a long time during the mid-to-late 20th century, the fad among English-speaking literary critics was for realism. This meant that English-language novels which had fantastic elements were generally considered not to be literary fiction (there were some exceptions, like Salman Rushdie's Midnight's Children, although I believe Midnight's Children was excused for being a “foreign” novel). And authors who were trying to write literary fiction felt that they had to stick with the critics' guidelines, so they could get good reviews from them. So even if literary fiction authors wanted to incorporate fantastic elements in their fiction, they didn't feel like they could unless they were already famous enough to ignore the critics.
This meant that only authors who weren't trying to write literary fiction used fantastic elements. And while some of these genre or popular authors wrote novels with real literary merit, it was not until fairly recently that they were accepted into the canon of literary fiction.
Some genre writers, like Ursula Le Guin, complained vociferously about this.
For example, see this blog entry, which contains most (if not all) of Le Guin's essay “On Despising Genres”.
And in my opinion (probably not widely shared), Stephen King's book Bag of Bones is a raspberry blown in the general direction of these critics. For the first half of the novel, it reads like literary fiction, with the narrator suffering and recovering from writer's block brought on by the death of his wife. In the second half of the novel, King deconstructs this and brings in a supernatural cause for many of the events in the first half, thus placing the second half firmly into the genre of horror. So in this book, Stephen King is indirectly complaining about not being accepted as writing fiction of literary quality.
I want to add that looking at the Booker Prize short list lately, this fad seems to be drawing to a close, although I'm sure that some literary critics will interpret the inclusion of non-realistic novels on the short list as a decline in the standards of the Booker Prize.
