The meaning of "lovers' food" is definitely not poison.
Stanley Wells' edition of the play (New Penguin Shakespeare) glosses the phrase as "the sight of each other". This makes sense in the context of this speech, since Hermia says,
Keep word, Lysander: we must starve our sight
From lovers' food till morrow deep midnight.
The gloss also makes sense when looking at As You Like It, Act 3, Scene 4, where Rosalind says,
The sight of lovers feedeth those in love.
However, the food metaphor is also present indirectly in an earlier line in Hermia's speech:
And in the wood, where often you and I
Upon faint primrose-beds were wont to lie,
Emptying our bosoms of their counsel sweet,
There my Lysander and myself shall meet
In Shakespeare's works, "sweet" can be interpreted as "tasting sweet" or as "sounding sweet". See for example the following lines in A Midsummer Night's Dream, Act 1, Scene 1, where these two meaning appear to meld due to use of the word "tongue" (both the organ and "speech"):
My ear should catch your voice, my eye your eye,
My tongue should catch your tongue's sweet melody.
The idea that "lovers' food" can be something auditory is also present in one of the best-known opening lines from Shakespeare's plays, namely from Twelfth Night (emphasis mine):
If music be the food of love, play on;
Give me excess of it, that, surfeiting,
The appetite may sicken, and so die.
Of course, when lovers are in each other's presence, they can enjoy both the visual and the auditory "food", i.e. the visual and/or auditory enjoyment of each other's presence.