Size doesn't matter ...
You may find Are the Redwall creatures their natural real-world size? an interesting read. The long and the short of it is: don't worry too much about the animals' size in the Redwall series. The author didn't pay too much attention to consistency there, and if you try to analyse it too much, you'll lose your immersion in the story. We don't generally notice much difference in size between mice, hedgehogs, squirrels, moles, rats, stoats, weasels, ferrets, etc. - they interact with each other much as humans would. Badgers, some birds, and possibly hares are noticeably bigger, but the relative sizes aren't always consistent. Just go with the flow and don't worry about it.
Brian Jacques has said the following in response to questions from fans:
Do the animals in your stories walk on all fours or, do they walk erect?
The creatures in my stories are as big or small as your imagination wants them to be. My imagination does not see them on all fours.
I have just started your books. I have read Redwall and am now working on Mattimeo. At the beginning of Redwall there is the "riderless horsecart" and now in Mattimeo there is mention of a ship that sank. Are there humans somewhere in the realm of Redwall?
No, there are no humans, my first book Redwall did mention the horse and cart but no humans are ever in the stories and I don't intend that they should ever be. The ships are generally crewed by vermin.
His answer to the first question supports what I said above (based on my memory of reading all the Redwall books) about lack of consistency: the size of the animals isn't usually all that relevant to the story, so you can imagine them however you want. (And when it is relevant, it's not always consistent, so thinking too hard about it will break your suspension of disbelief.)
Relevant to the second part of the quote above, there's something else to take into account: what TV Tropes has dubbed Early Installment Weirdness. At the time of Redwall, the earliest published book in the series, a lot of what would later become established tropes of the Redwall universe were still in flux. Horses never appear in any of the later books; vehicles such as carts do, but the animals (regardless of species) always seem to fit into them in a comparable way to how humans do in reality, rather than an entire army of vermin fitting into a single cart. So it may be that this mention of a stag beetle's horns fitting nicely on a rat's helmet is something that would never have happened in later books. Generally, you should worry even less about inconsistencies in Redwall than in later books.
... but stag beetles are actually pretty big.
However, all that aside, it's not too weird for a stag beetle's horns to be of noticeable size on the helmet of a rat. Stag beetles of the type found in Britain can be nearly 8cm long, while brown rats are around 20-25cm long, not counting their tails. Assuming that Cluny is an unusually large rat, and that the beetle he used was also unusually large (he'd want an impressive battle helm), it would be a little under one-third his size. Imagine a half-metre-long stag beetle compared to an average human man; that doesn't sound too unrealistic for using its horns as a helmet ornament.
So your premise that "Beetles are generally... quite smaller than a rat" is incorrect. However, I'll stress again that all of this is way overkill for thinking about animal sizes in the Redwall series. Cf., for example, Urgan Nagru the Foxwolf, who was a fox wearing a wolf's skull on his head:
Foxes are considerably smaller than wolves in reality, yet nobody batted an eye at this.
One last piece of evidence for beetle size in the Redwall series is the water beetle Grubwhacker in the book Mossflower (one of very few other mentions of beetles throughout the series), who seems to be roughly the size of a large dog when compared to the shrew Log-a-Log:
An old black water beetle sat by the fire. [...] He sat stroking the beetle's back.
"I call this fellow Grubwhacker. He lives nearby, comes in and out of here for his food, just like a pet." [...] Taking a loaf and a piece of cooked fish, he placed them upon Grubwhacker's back, where they could be carried without falling off. Log-a-Log patted his pet affectionately.
"Go on Grubwhacker," he told him. "Back to your missus and the little uns."
The beetle trundled off obediently.
Again, it doesn't seem unrealistic for a creature this size to have its mandibles on a rat's helmet. Bear in mind, too, that your quote doesn't even say the horns are a major part of the helmet - just that they're "adorning it". They could conceivably be much smaller.
