I don't think the poem is value-laden about these things, but I do think that the poem is about (which is to say, interested in) issues around:
temptation (aestheticizing an immoral/sinful act is a good way to make oneself feel less bad about committing it)
how we ask for forgiveness (grammatically, the speaker does not ask for it at all)
remorse (which, does the speaker show it?)
excuses (there is no punctuation between "Forgive me" and "they were delicious" as if the speaker was rushing to get the justification out)
and similar concerns.
A good meta-question for the poem might be: Is how the speaker goes about the act of eating the plums and making their apology a morally good, emotionally effective way, or a way that assuages their own feelings without actually resolving the root problem? The poem to me has always seemed a warning against passive-aggressiveness, and an argument for authenticity in social engagement. Think about how you would feel if your roommate left this note!
The prosaic quality in this poem that some have noted ("remove the line breaks and it's just a couple little sentences", by the way, is descriptive of a vast swath of English poetic history) lends to imagining it as an actual, material object. That imagination gives the document an insistently embedded, social dimension that can't be ignored: the poem, the note on the fridge, is an act of communication that is inescapably politicized.