There's plenty of ways how a pleasant thought can bring sad thoughts to mind but in this context, Wordsworth is grieving the condition of humanity.
As the proceeding lines below state,
To her fair works did Nature link
The human soul that through me ran;
And much it grieved my heart to think
What man has made of man.
And again, at the last stanza, Wordsworth writes,
If this belief from heaven be sent,
If such be Nature’s holy plan,
Have I not reason to lament
What man has made of man?
What exactly is this "Nature's holy plan?" The poem seems to suggest pleasure, joy, and rest.
Wordsworth remarks that he believes "that every flower // enjoys the air it breathes". He notices the birds around him "hopped and played", and "the least motion they made // it seemed a thrill of pleasure". He writes,
The budding twigs spread out their fan,
To catch the breezy air;
And I must think, do all I can,
That there was pleasure there.
Throughout the poem, we see the repetition of words relating to beauty ("fair" and "sweet") as well as the word "pleasure" or "pleasant". This is contrasted by the realization of "what man has made of man". The poem seems to lament the fact that nature around him is so beautiful and peaceful, but the world around him - humanity as a whole is destroying the natural beauty of the world and ultimately destroying each other.
The pleasantry of the environment around him made him realize that what he had around him was simply not realistic to the real-world he lived in, and that's why it brought him sad thoughts.