"Dirt is matter out of place" is a phrase apparently originating from Lord Palmerston, but nowadays associated more with Mary Douglas who cited it to Lord Chesterfield.
The origin of this phrase is a fascinating topic! Perhaps much more so than you realised when you asked the question. It's commonly attributed nowadays to Mary Douglas and her 1966 anthropology book Purity and Danger, but since Douglas was born decades after the publication of In the Midst of Alarms, clearly we must dig deeper. Douglas wrote in her book:
If we can abstract pathogenicity and hygiene from our notion of dirt, we are left with the old definition of dirt as matter out of place. This is a very suggestive approach.
Since she refers to it as an "old definition", even without the evidence of In the Midst of Alarms we'd know that this is not a concept original to her. In a volume of field notes compiled during her second trip to study the Lele people in the Belgian Congo in 1953, she attributes a very similar idea to Lord Chesterfield:
Contrast: idea of dirt, with “good clean mud” etc. Chesterfield “Dirt is any matter displaced” e.g. hair, crowning glory etc. and hair in the soup.
And she later attributed her own exact quote, "dirt is matter out of place", to Chesterfield, in her 1968 essay "Pollution" for the International Encyclopedia of the Social Sciences, later anthologised in the collection of her essays entitled Implicit Meanings (1975)
In English-speaking cultures, the key word is the ancient, primitive, and still current 'dirt'. Lord Chesterfield defined dirt as matter out of place. This implies only two conditions, a set of ordered relations and a contravention of that order.
This source would certainly pre-date In the Midst of Alarms, but in fact Chesterfield too is not the real origin of the idea! A detailed investigation into the origins of "dirt is matter out of place" was done by Richard Fardon, Mary Douglas’s literary executor, in his article "Citations out of Place" published in Anthropology Today 29(1) (2013). He concludes that Chesterfield did not write "dirt is matter out of place":
Did Chesterfield ever write that dirt was matter out of place? Short of reading his every word, I cannot be sure he did not, but it seems unlikely. The Letters to his son on the art of becoming a man of the world and a gentleman opine that personal cleanliness was a common decency, but he does not do so with the generalizing force, or in the precise terms, that Douglas suggests. While he could not be certain the phrase occurred nowhere in Chesterfield, as a scholar of the eighteenth-century writer, Colin Franklin could not recall it (personal communication).
The earliest written source that Fardon was able to find for this idea was Lord Palmerston:
Lord Palmerston (1784-1865), recently Foreign Secretary, soon to be Home Secretary and later Prime Minister, had spoken about urban dirt, ‘During an address to the Royal Agricultural Society in 1852, he said “I have heard a definition of dirt. I have heard it said that dirt is nothing but a thing in a wrong place. Now, the dirt of our towns precisely corresponds with that definition” (Anon 1852, 8).’
The above is not the exact phrasing "dirt is matter out of place", but it expresses the same notion, even though Palmerston himself doesn't claim to be the original inventor of the phrase. He did, however, continue using and adapting the phrase in the following years:
Palmerston must have been pleased with both the immediate reaction to his witticism and to its literary uptake, since he used it more than once, and doing so brings his remark half-way closer to the version we know from Douglas when ‘a thing in a wrong place’ becomes ‘matter ... in the wrong place’. The ditty “Our filth and our felons” published in Punch on 3 January 1857, cited it in this form,
‘Lord Palmerston once, with that off-hand felicity,
Which belongs to his lordship in stating a case,
To a new definition of “dirt” gave publicity,
As “nothing but matter left in the wrong place.”...’ (1857: 9)
A year later, Punch for 30 January 1858 reported, ‘Lord Palmerston was the other day repeating the saying which gained him so much approbation in the sewerage debate, namely, that “Dirt was only matter in the wrong place.” The Lord Privy Seal said that it was the rudest speech he had ever heard’ (1858: 47).
Fardon's further investigation finds at least a possibility for the origin of dirt being matter "out of place", rather than Palmerston's "in the wrong place":
William James’s The varieties of religious experience (the 1901-2 Gifford Lectures delivered at the University of Edinburgh) featured on Mary Douglas’s course outline for ‘Religion, morals and symbolism’ in the early 1970s when I attended her lectures, though I can only guess how much earlier she had read it. James adapts Palmerston’s aphorism (albeit unattributed) in a passage in which he is discussing the ‘gospel of healthy-mindedness’ in its optimistic rejection of the essential character of evil.
[...] Here we have the interesting notion fairly and squarely presented to us, of there being elements of the universe which may make no rational whole in conjunction with the other elements, and which, from the point of view of any system which those other elements make up, can only be considered so much irrelevance and accident – so much ‘dirt,’ as it were, and matter out of place. (James 1902: 107)
James’s use of the redolent adjective ‘excrementitious’ to qualify the noun ‘stuff’ leaves no doubt that he was attuned to Palmerston’s original context. Here, finally, we find the canonical Douglas phrasing of the aphorism that ‘dirt is [as it were] matter out of place’. Indeed, this ‘splendid passage’ is quoted in the concluding chapter of Purity and danger (1966 [1984]: 165). So perhaps Douglas absorbed her maxim from James but this fact escaped her when writing the article on ‘Pollution’ (1968) around the same time? A 1952 edition of James’s book is referenced in Purity and danger which might mean a copy of this work was in her possession before she wrote the entry in her 1953 field-notes mentioned at the outset.
So the history of this phrase is a long and tortuous one: starting perhaps from an unknown pre-Palmerston source (or Palmerston just used "I have heard it said" to make his own phrase sound wise and well-accepted), popularised and adapted by Palmerston, further adapted by James, further popularised by Douglas but misattributed by her to Chesterfield.
What about this phrase's usage in In the Midst of Alarms?
Some of the passages quoted above make clear the meaning of "Dirt is matter out of place": anything can be considered dirty if it's in an inappropriate place, and so "dirt" can be defined as a thing that's "out of place", not where it's supposed to be. Yates is applying this aphorism, in a somewhat tongue-in-cheek way, to himself: he is completely "out of place" riding a horse in the countryside, and worth as much as dirt in that situation.