I think Milton expresses the same emotional pattern of thought in Tetrachordon (1645) where, commenting on Matthew 5:32, he says (my emphasis):
[His Wife. ] This word is not to be idle here, a meere word without a sense, much less a fallacious word signifying contrary to what it pretends; but faithfully signifies a wife, that is, a comfortable helpe and society, as God instituted; does not signify deceitfully under this name, an intolerable adversary, not a helplesse, unaffectionate and sullen masse whose very company represents the visible and exactest figure of lonelines it selfe. Such an associate he who puts away, divorces not a wife, but disjoyns a nullity which God never joyn'd, if she be neither willing, nor to her proper and requisite duties sufficient, as the words of God institute her. And this also is Bucers explication of this place.
The imagery recalls common patterns of Biblical condemnation of idols, such as in Psalm 115:5-8, which typically say that foreign gods are simply lumps of wood/stone/gold/etc. which are not capable of feeling or acting. Milton's extended language, here and elsewhere, presents the idea that a marriage can be a total sham, having the form of a marital relationship but without the real partnership that ought to exist. Further, earth and phlegm are base substances: this is no golden idol, but something even more crudely fashioned.
Rand al'Thor suggests in comments that there is a connection to the "four humours" of classical medicine. This is certainly possible as something in Milton's mind, as he was familiar with the model (and was later to use it in such passages as Samson Agonistes, "anguish of the mind and humours black"). The phlegmatic nature was commonly said to be typical of women, and its negative manifestation is coldness, dullness, lack of affection, and lethargy. Earth is associated with black bile, or melancholy, which in the negative sense would mean inwardness, withdrawal, sadness, and so on. Milton invokes this in Paradise Lost 11, with "moaping Melancholie" (485) and Michael's warning that "in thy blood will reigne / A melancholly damp of cold and dry / To weigh thy spirits down" (543-545); he also mentions "melancholy despair" in the passage quoted in the question above. All of this is consistent with Milton's broader point here, even though he is not making a "medical" argument through explicit invocation of the theory of humours.
There may also be an ironic echo of John 9:6, in which Christ heals a man's blindness by anointing his eyes with a mixture of earth and spit. At the time of writing, Milton had not yet lost his sight completely, but he may have been feeling the start of the process. Perhaps the "image of earth and fleam" represents his resentment at his difficulties with vision, with an unsympathetic wife and a relationship which was far from the miraculous power of Jesus to elevate common matter into a healing substance. Even is this particular nuance is an over-reading, Milton's imagery is still remarkably bitter, as he is not simply noting that his relationship lacks the qualities it ought to have, but specifically blames his partner for her temperament.