A couple of comments have led me to believe I probably should have made my answer clearly.
Yes, the poem is written in the first person. That is to say the entire poem is a first-person narrative, told from the perspective of the poet, Shirley Toulson. This is made abundantly clear in the first stanza:
A sweet face,
My mother’s, that was before I was born.
And the sea, which appears to have changed less,
Washed their terribly transient feet.
"I" refers to the poet herself. This is first-person narration. And focus of the poem is on the mother and the daughter (Toulson)'s connection with the mother.
The sea holiday
Was her past, mine is her laughter.
"Mine" refers to my (the poet's) past corresponding to "her past". Now let's turn to your troubling lines:
Both wry / With the laboured ease of loss.
"Both" refers to both the mother's past and the mother's laughter. Both are wry and both with labored ease of loss. Why wry? Note that wry has a strong denotation of distortion and has the sense of "twisted", "bent", "abnormal", "deliberate", which makes it a perfect reverberant point echoing two words that follow: "labored" and "loss". The mood is nostalgic and melancholy. Both her past and her laughter are twisted because it is the loss of our childhood, out past, and our beloved ones that we have to grapple with. Hence the "labored ease". We need to and will eventually come to terms with our loss but it takes work for us to reach inner peace with those losses.
At the end of the poem lies the verse you quote. That verse reveals to us the mother has been dead for many years "as many years / As that girl lived". The mood is sorrowful and mournful. The poet is missing her mother. She mourns her passing by starting the poem on a happy, celebratory note in the first stanza, slowly coming to a pensive and melancholy second stanza, and ending on a lugubrious note.
And of this circumstance
There is nothing to say at all.
Its silence silences.
This circumstance, as Old Brixtonian has pointed out, refers to the fact that her mother died at a young age. The approximate age the mother died at is "hidden" in something of a riddle in the poem. Let's decipher it:
And she the big girl — some twelve years or so.
Then in the second stanza:
Some twenty — thirty — years later
She’d laugh at the snapshot.
The mother would have been anywhere between 32 and 42. The third stanza tells us:
Now she’s been dead nearly as many years
As that girl lived.
So her mother has been dead for about 12 years. Since we don't know how much time has elapsed since the poet and the mother shared a moment looking at the old photo, it is unclear how old exactly the mother was when she passed. But the idea that the mother died young, was cut down in her prime, is clear. That is what "this circumstance" is. The pronoun its also points here. This situation.
There is nothing more to be said, declares the poet who sits in silent mourning of her mother. The silence is sorrowful and full of happy memories and lingering thoughts. The silence itself silences [more words].