This is called allusion, a reference to something outside the text. It is common, not just in poetry, but in literature of all types. It can be an extremely effective technique for an audience that understands the references.
However, as you have "alluded" to, the problem with allusion is that its effect depends on your familiarity with the the source. Work that refers to transitory pop-culture phenomena, or to personal inside knowledge, or to a not-widely-known sub-culture, tends to not travel as well, or age as gracefully.
The best bet is to make sure your work stands up without the allusions, and then they serve as a kind of "bonus" for the people who do understand them. In the case of the Williams poem, plenty of people have enjoyed it or been struck by it without necessarily knowing what it means (or perhaps, because they don't know what it means).