First of all, let me just point out two things which it may be useful to bear in mind when considering how seriously to take this answer: whether I'm overanalysing or underanalysing here.
Firstly, some of the choices of animals are significant. It's not completely random; the author spent some considerable time thinking about which animals to use to represent certain groups, even to the extent of having specific reasons for rejecting some suggestions. From this blog post:
At the very beginning of the second volume (page 9) Art is thinking about what species he should choose to depict his wife’s nation, the French. Francoise votes for a ‘bunny rabbit’ but Spiegelman dismisses the idea claiming that it is a choice too sweet and gentle for the French. This scene reassures me in my view that in Maus the animals chosen to represent the different nationalities are selected for a reason, and they are not mere results of an arbitrary pick.
On the other hand, some of the choices don't really matter at all. For national or ethnic groups which played only a minor role in the story, the choice of animals used to represent them was much more arbitrary, because it didn't really matter at that point. As the author said:
As the book was coming to a close, I really couldn’t have cared less about my metaphor, but I was stuck with it. People would ask me, “Oh, how would you draw us Italians?” and I as always stumped. I just had to deal with each of these issues as they came up, and it led to the whole sequence in Maus II of talking to Françoise about how to represent her. In a way I started reaching for the absurd to make sure one didn’t take the ruling metaphor at, um, “face” value.
With that out of the way, let's start considering the different groups and their animal representations one by one. I'll go through them in approximate order of their importance to the story, so the choices of animals will become more arbitrary and less significant as we go on.
Jews as mice
Spiegelman has actually spoken about this in an interview, and told us what inspired him to create this depiction. It seems that as well as the cat-and-mouse allegory (about which more later), there has been a long tradition of portraying Jews as rats or mice in anti-Semitic works.
The most shockingly relevant anti-Semitic work I found was The Eternal Jew, a 1940 German “documentary” that portrayed Jews in a ghetto swarming in tight quarters, bearded caftaned creatures, and then a cut to Jews as mice—or rather rats—swarming in a sewer, with a title card that said “Jews are the rats” or the “vermin of mankind.” This made it clear to me that this dehumanization was at the very heart of the killing project.
In fact, Zyklon B, the gas used in Auschwitz and elsewhere as the killing agent, was a pesticide manufactured to kill vermin—like fleas and roaches.
As I began to do more detailed and more finely grained research for the longer Maus project, I found how regularly Jews were represented literally as rats. Caricatures by Fips (the pen name of Philippe Rupprecht) filled the pages of Der Stürmer; grubby, swarthy, Jewish apelike creatures in one drawing, ratlike creatures in the next. Posters of killing the vermin and making them flee were part of the overarching metaphor. It’s amazing how often the image still comes up in anti-Semitic cartoons in Arab countries today.
Germans as cats
In the same interview, Spiegelman spoke about how the perceived grace and nobility of cats ties in to the supposed Aryan supremacy which was part of the Nazi German ideology:
There was one rendering of a cat in full Nazi drag that looked sort of like Marlon Brando in The Young Lions. It was the most noble and savage version of the Nazis, tying into the stereotypes that presented Nazis as somehow sexy. It reminded me of the whole Night Porter genre of pornography that involved SS uniforms and scared me away from drawing Maus with really large-scale cats.
Also, of course, as you've already noted, "Jews as mice, Germans as cats" can be seen as a symbolic representation of the Holocaust as a "cat and mouse" affair with the Germans hunting down the Jews. Interestingly, Spiegelman originally had this idea for an entirely different case of racism in a different continent:
Actually, it all started with me trying to draw black folks. [...] I wanted to do something in that melodramatic pulp illustration mode, complete with venetian blind shadows, but with animal faces in which the denouement would have the protagonist getting crushed to death by a giant mousetrap that snaps shut on his body. I made some sketches but I was floundering. A filmmaker I had become close friends with, Ken Jacobs, was teaching an introduction to cinema class. On this particular day, Ken showed a bunch of old racist animated cartoons from the silent and early sound era. The blacks were cheerfully represented as subhuman, monkeylike creatures with giant minstrel lips—stereotypes stealing chickens, stealing watermelons, playing dice, all singin’ & dancin’, just the daily stock in trade of our racist cartoon heritage.
…it all led me to my Eureka moment: the notion that I could do a strip about the black experience in America, using an animated cartoon style. I could draw Ku Klux Kats and an underground railroad and some story about racism in America.
[...] After my self-excoriating doubts settled in, I realized that this cat-mouse metaphor of oppression could actually apply to my more immediate experience. This development took me by surprise—my own childhood was not a subject for me. But I did realize that if I shifted from Ku Klux Kats and anthropomorphized “darkies” to the terrain I was more viscerally affected by, the Nazis chasing Jews as they had in my childhood nightmares, I was on to something. It became my three-page contribution to Funny Aminals.
French as frogs
Calling the French frogs is actually a long-running thing, at least in Britain. I've always assumed this is due to the French practice of eating frogs (many derogatory nationalist terms are based on cultural delicacies: the French are frogs, the Germans are krauts, the British are roast-beefs, and so on.) According to the Racial Slur Database (TIL this is a thing):
The French are said to laugh like frogs. When they laugh, their adam's apples bulge out of their necks like frogs. Also perhaps from the French delicacy of frog-legs. Another possible derivation is the Fleur-de-Lys displayed on the French king's banner in the Middle Ages, which, to the English enemy, looked like squatting frogs. UK origins.
Poles as pigs
This is a deep and sticky quagmire to get into. Many critics have condemned Spiegelman for his portrayal of Poles as pigs, accusing him of racism against the Polish people. Entire academic papers have been written on this issue. In an effort to keep it short, I'll quote just from this 82-page article entitled "Poles as Pigs in MAUS":
Portraying Poles as pigs is offensive. In fact, it has been acknowledged as such by literary critics. In the biographical introduction to the excerpt from MAUS that appears in The Norton Anthology of American Literature, 7th edition (New York: Norton, 2007), Volume E, p. 3091, editors Jerome Klinkowitz and Patricia B. Wallace describe Spiegelman’s representation of Poles as pigs as “a calculated insult” leveled against Poles. A similar point was made by Harvey Pekar, a celebrated underground comic book write, who describes himself as a Jew with a background similar to Art Spiegelman’s: “When he [Spiegelman] shows them [Poles] doing something admirable and still portrays them as pigs, he’s sending a mixed message.”
Characteristically, Spiegelman has dismissed Poles’ concern about their depiction as pigs as “a squeal,” the sound pigs make.
The pig imagery or “metaphor,” on the other hand, is rarely, if ever, explained – whether in MAUS itself, or in available reading materials. One handout provided to students states that the animals have a “symbolic quality,” without any further explanation of the role of the pigs.
“You pig,” is universally considered to be an insult. In many cultures, pigs are viewed as disgusting, filthy, and greedy animals. They are often considered to be vulgar and stupid. The implication, therefore, is that there is something unsavoury about the pig people. This is one obvious negative connotation that would not be lost on the students, especially since that image is reinforced by the negative stereotypes used to portray Poles, who even manage to remain fat while imprisoned in Auschwitz.
For Jews and Muslims, pigs are “unclean” animals. Jewish culture in particular views pigs, and pork, as nonkosher, or unclean. [...] It is telling that Spiegelman chose this supremely unkosher animal to depict the Poles, rather than the Germans.
Unfortunately, the image of Poles as being “unclean” has a long and shameful tradition. In Poland, when a Jew wanted to insult a Pole, he called him a “Polish pig.” [...] Polish inmates of Nazi camps were often called “Polish swine” by German officials and kapos (prisoner functionaries). Poles are also referred to as “pigs” in Jewish memorial books.
MAUS employs the same imagery of the Poles as found in Nazi propaganda, where Poles were often referred to as “pigs.” Art Spiegelman was, of
course, aware of these problematic associations when he chose to portray Poles as pigs.
Apparently Spiegelman has even admitted that bias against Poles played a role in his choice of the animal used to represent them. Quoting from the same article:
In MetaMaus: A Look Inside a Modern Classic, Maus (New York:
Pantheon, 2011), belatedly, Spiegelman divulges his actual reasons for portraying Poles as pigs: it is to bash Poles. With reference to his father’s attitude towards Poles, he quips, “So my metaphor [mice to be killed outright, and pigs to be exploited and eaten] was somehow able to hold that particular vantage point while still somehow acknowledging my father’s dubious opinion of Poles as a group.” (P. 122.) Despite the fact that Poland had for centuries given sanctuary to Jews persecuted elsewhere, Spiegelman adds: “‘And considering the bad relations between Poles and Jews for the last hundred years in Poland, it seemed right to use a non-Kosher animal.’” (P. 125.)
Americans as dogs
This could again be interpreted as tying into the symbolism of the "cat and mouse" game of Germans and Jews. In the end, the American dogs come and chase away the cats to rescue the mice. Although there's not much love lost between dogs and mice in real life, the enmity between dogs and cats is real enough, and could represent the war between Americans and Germans. Again, an interview with Spiegelman seems to confirm this:
Turning again to my simpleminded ur-text of American popular culture: cats chase mice, and dogs, by God, chase cats – it’s a direct food chain. [...]
But dogs were easy; it’s almost the Family Feud answer to what animals come to mind and how do you perceive them. The dogs were heroic vanquisher of cats, so there was that. besides, as soon as you’re a cartoonist drawing a dog, you’ve got lots of different kinds of dogs to draw. You’ve got Collies and Dachshunds and Cocker Spaniels and Chihuahuas and their species or sub-species are much more clearly delineated than cats, even though cat fanciers will say otherwise. Here, the fact that there were so many possible dogs got me to actually verbalize to myself: “Oh, I get it. Americans are a mongrel race, a bunch of mutts.” Bill Mauldin’s panel cartoons of Willie and Joe – the “dogfaces” of World War II as GIs were called – came to mind as soon as I started trying to figure out what it might mean to draw a dog in an army uniform.
Gypsies as moths
Well, for one thing, they're gypsy moths, so even the name fits. Also, moths fit with an imagery of itinerancy - flitting from place to place, but never stopping for long - which makes them a reasonable choice to symbolise gypsies.
Also, Spiegelman has a story about where the inspiration came from for this:
I vividly remember drawing the sequence where my mother went to see a fortune-teller – I was in a small cabin, deep in the woods of Connecticut that summer. I prefer to work at night when I can, and these giant moths kept flinging themselves against the glass, trying to get in. Most of them looked like casting calls for Mothra. They were insane and enormous. I got really fascinated by what their faces looked like. And it was at precisely the moment I was trying to figure out how to draw the gypsy, so it was preordained that I’d use gypsy moths.
Brits as fish
By this point, we're getting to nationalities which are less important to the story, and their choices of animal are becoming more arbitrary. But here's what Spiegelman said; take it with a pinch of salt if you like:
When Vladek looks for Anja after the war, he goes to a large displaced persons center at Belsen. The British are in charge of that camp. I guess I could have avoided the whole issue since they just appear in the mise-en-scène for a panel or two, but I decided to give the Brits a walk-on part – or, as it finally resolved itself, a swim-on part. I thought about fish and chips, an island culture, fish out of water. All those things just seemed to lead me toward drawing fish without bicycles but with jeeps.
Swedish as reindeer
There are plenty of reindeer in Sweden. I'm not sure if there's any deeper meaning than that. The author doesn't have much to say about it either:
Sweden was quite welcoming to refugees after the war. I thought of the Swedes as somehow far outside the loop of my Eastern European narrative and finding an animal so totally out of the scale with mice, cats, and mutts – those large galumphing and gentle reindeer – struck me as amusing.
All bold emphasis is mine in the quotes above.