Compare this with a situation where someone visits you, you offer them food, and they say: "No, thanks, I have lots of food at home!"
This would be extremely rude in many cultures. It may imply any of the following:
Guests eat because they don't have food at home
The host may not have enough food, so the guest may be putting considerable strain on them