Alfred Nobel's will states that
[Priset] [...] för litteratur [utdelas] af Akademien i Stockholm[...]
or, in English
The award for Literature shall be awarded by the Academy in Stockholm
All the awards are granted by academies based in Stockholm, except for the peace prize, which is awarded by a five-member committee chosen by the Parliament of Norway.
The Academy has then elected to organize the work in the following manner:
- A large group of people, including Academy members, professors, and former winners, are able to suggest people for consideration
- A four or five-member sub-committee, elected for three years, does the heavy lifting of registering, investigating and recommending winners. They can ask for expert opinions of outsiders, and ask for translations if not enough such are available. They reduce all the suggestions to a short-list of about twenty.
- Then the entire Academy discuss the list, and it is reduced further to five.
- Over the summer, the members read such authors that they need to get further acquainted with.
- Early autumn, they vote on the winner.
As for the nationality of the Academy, I have not been able to find the statutes, but this LTE makes it clear that only people of Swedish nationality are eligible (and thus not, e.g. Swedish-speaking Finns). They could, I guess, involve more non-members in the preparatory work, but I think it would be hard to wiggle away from the wording of the will, even if it is clearly the work of Nobel's own hand, not a lawyer. In the end, it is the Swedish Academy, and no other body, that is responsible for awarding the prize.
Also, remember that there was actually an uproar about the fact that the awards should be granted "without regards to nationality" when the contents of the will first became known. Even so, while the prizes have always been of a very substantial sum, it was in no way clear from the start that they would take on the significance they have today.