The Library of Babel cannot be taken as plausible, realistic worldbuilding. "How can real-world languages exist" is not going to have more of an answer than "How does an immense number of bound books exist," or "How can they know what bread or pyramids are, when they can't possibly have either." There is excellent reason for Borges to use real languages, but they're not in-universe reasons.
Borges uses languages here in order to demonstrate yet another facet of infinity: the idea that the already-infinite selection of books and combination can furthermore be interpreted according to a near-infinite number of languages or cipher keys.
This is a fundamental theme of the piece: levels upon levels of infinity; and the blurring between the meaninglessness of random chance, the immense significance that's possible when everything exists somewhere. As the books are infinite, and therefore meaningless and also hiding books of impossible truth and value; the ways to interpret an impenetrable book are also infinite, and therefore each book is both meaningless and also possibly hiding impossible truth and value.
A lot of the brilliance of this piece is in the crafting of clear, intuitive imagery, for impossible, paradoxical concepts. In this case, foreign languages are exactly the imagery Borges needed. It's an amazingly simple, elegant, graspable way to convey the idea that gibberish to one person might hold meaning for another.
It is my opinion that any modification of this concept might have made the story more plausible, but would have made its imagery weaker, less immediate.
- Consider how using imaginary languages -- which would certainly be more plausible -- would be a stumbling block. It would take the reader time and effort to understand what is being conveyed. It would make sense, but it wouldn't be immediately intuitive, which is key here.
- Non-language models - e.g., maybe all the books are written in cryptographic cyphers, which can be decoded - are absolutely possible, and Borges touches on these (e.g. with the repeating MCVs, and how they struggle to come up with some form of language under which the text could conceivably make sense), but would take much more explanation and would be less vivid by orders of magnitude.
- In contrast, consider how effective Borges's description is: "A Samoyedic Lithuanian dialect of Guarani, with classical Arabian inflections." In the real world, this would be an utterly bizarre, implausible combination. That's the point. By taking familiar elements and combining them at random, Borges arrives, again and again, at his intriguing portrait of infinity. And for that to work here, you need to be able to immediately grasp the image he's created, and how very strange it is.
So, yes, those languages should not technically exist within the Library. But the Library is a dazzling thought experiment; not a "real" or rigorously-speculated world. In the service of the thought experiment, real languages serve Borges far better than imaginary ones could.