Looking through Day by Day, I found two poems that might be talking about impotence, although neither of them mentions it explicitly.
In Ear of Corn, about an old wine baron, the following lines at least show the inability of the subject to make young women be attracted to him.
someone's young wife is on his right—
She cannot cure his hallucination
he can bribe or stare
any woman he wants into orgasm...
He fills her ear
with his old sexual gramophone.
he makes nothing happen.
And The Downlook is also suggestive of impotence. It even contains the word (although possibly in another context, unless "two-way traffic" is a metaphor for sexual intercourse, which I suspect it is). Here are what I think are the most suggestive lines from this poem.
nothing lovelier than waking to find
another breathing body in my bed ...
glowshadow halfcovered with dayclothes like my own,
caught in my arms.
Last summer nothing dared impede
the flow of the body's thousand rivulets of welcome ...
winding effortlessly, yet with ambiguous invention—
safety in nearness.
Now the downlook, the downlook—small fuss,
each miss must be a mile
if one risk the narrow two-lane highway.
It's impotence and impertinence to ask directions,
while staring right and left in two-way traffic.
Ah, loved perhaps before I knew you,
others have been lost like this,
yet found foothold
by winning the dolphin from the humming water.
Robert Lowell suffered episodes of depression for much of his life. However, depression can cause impotence, and I believe that this is indeed a subtext here.
The Downlook seems to be primarily about depression (which is also one of the definitions of the word downlook). And this is indeed how Lowell used the word: Google finds:
In a letter to Blair Clark, Lowell speaks of the “downlook” as an inward feeling of depression (the aftermath of manic attack)