The character Fiennes immediately explains the remark:
But he stopped in the torrent of his talk in a momentary bewilderment before he saw the priest’s very simple meanmg.
“You mean that people make too much of them?” he said.
The word “dog” spelt backwards is “god”, so what Father Brown means is that he doesn’t like the way that some people treat dogs as gods: that is, they assign their dogs’ behaviour a significance it does not deserve.
This is the theme of the story, as indicated by the title. There are two instances in which Fiennes interprets the behaviour of the dog ‘Nox’ as having a mystic or oracular significance. First, the dog fails to retrieve the stick thrown by Harry Druce into the sea, and sets up “a howl of wail of woe” which Fiennes interprets as the dog knowing that his owner has just been murdered. Second, on returning to the house, the dog rushes up to the lawyer Traill “barking at him madly, murderously” which Fiennes interprets as the dog denouncing Traill as the killer.
But Father Brown explains that once you stop treating the dog as if it is a god, then its behaviour provides a clue to how the murder was committed:
“The dog had everything to do with it,” said Father Brown, “as you’d have found out if you’d only treated the dog as a dog and not as God Almighty judging the souls of men.”
He goes to to explain the dog’s behaviour as follows:
First, the dog failed to retrieve the stick from the sea because it was a sword-stick, and being made largely of steel it was heavier than water, so that it sank to the bottom. The dog howled because he couldn’t find it. Second, the dog barked at Traill because the lawyer was nervous of dogs and “dogs hate nervous people”.
This is a common theme in Chesterton’s stories, in which modern, secular people are actually credulous and superstitious, while only Catholics like Father Brown are properly skeptical and rational.