The speaker in the poem is a single woman who entertains married men. Comparing her position as "the other woman" to the men's domestic life, the speaker imagines that they probably praise her to their wives.
This praise could be the men's justification for their affair, claiming that they were carried away by the speaker's charms and also denigrating their wives' housekeeping in comparison to hers:
They said my house was licking clean,
no word I spoke was ever mean,
I had an air of mystery,
But... They went home.
Alternatively, they might not be admitting to the affair but just praising the speaker as a double bluff to deflect suspicion. In either case, while the men are happy to have a fling with her, they ultimately return to their wives:
they liked my smile, my wit, my hips,
they'd spend one night, or two or three.
But ...
The poem is a commentary on the asymmetrical power relations between the sexes. On the one hand, as "the other woman," the speaker has no claim on the men: "They went home." On the other hand, the wives they go home to either have no idea that their husbands are breaking their marriage vows, or have to deal with the knowledge that their husbands find them inadequate and the fallout of the affair.
The shift in the addressees of the men's words also show that Angelou does not pit mistress against wife, but rather sees them both as situated within these asymmetric gender relations. In the first verse, the men are praising the mistress to their wives. The second verse is ambiguous; the men could be either praising the speaker to their wives, or complimenting her directly. In the third verse, the addressee is the mistress as they praise her hips while spending a couple nights with her. This skillfully handled shift, where both the wives and the mistress are spoken to by the men, places the women in solidarity with each other even as their relationship is not direct, but mediated through the speaking men.
Whether as wife or as mistress, then, the women Angelou depicts in this poem find their social and self-worth dependent on their relationship to unreliable men. The poem thus critiques social norms that evaluate women largely, if not solely, in terms of whether they have a man, while simultaneously allowing men to treat the women as disposable.