From The Kite Runner:
Hassan milled about the periphery of my life after that. I made sure our paths crossed as little as possible, planned my day that way. Because when he was around, the oxygen seeped out of the room. My chest tightened and I couldn’t draw enough air; I’d stand there, gasping in my own little airless bubble of atmosphere. But even when he wasn’t around, he was. He was there in the hand-washed and ironed clothes on the cane-seat chair, in the warm slippers left outside my door, in the wood already burning in the stove when I came down for breakfast. Everywhere I turned, I saw signs of his loyalty, his goddamn unwavering loyalty.
What's the meaning of "Hassan milled about the periphery of my life" in this passage?