It seems to me that the confusion over what is termed "LitFic" in part stems from the question as to whether the novels that have become canonical -- Crime and Punishment, Wuthering Heights, The Grapes of Wrath, etc. -- actually still qualify as LitFic. I tend to agree with a previous commenter that LitFic is more of a recent market designation and that these examples of "classic" literature don't really fit in with the designation. LitFic could perhaps also be designated as Contemporary Literature, that is, literary output that is, in its own way, vying for inclusion in the canon, which is ever-expanding. That may be an elitist view of LitFic as a genre, but so be it.
I also agree that it is an outgrowth of Modernism, and that in itself points to a certain elitism as well.
Let me be clear: I'm not saying that LitFic targets a specific audience, or that it attracts a specific audience. But its stylistic focus and emphasis on characterization, often at the expense of plot, present challenges that Genre Fiction (often, but not always) does not.
LitFic writers are, I think, by nature a little more experimental with technique and tend to draw attention to the medium of language itself, thus the connection with Modernism.
I'm sure I've said something here that's going to make someone angry. Fire away.