The key to this poem is the title, ‘The Penitent’, that is, someone who ‘repents’ their sins:
repent, v. To review one’s actions and feel contrition or regret for something one has done or omitted to do; (esp. in religious contexts) to acknowledge the sinfulness of one’s past action or conduct by showing sincere remorse and undertaking to reform in the future
Oxford English Dictionary
In Christianity, sincere repentance is a precondition for forgiveness of sins and admission or reconciliation to the Church, and forgiveness of sins is a precondition for being saved (admitted to heaven). It should be remembered that Anne Brontë’s father Patrick (1777–1861) was a priest in the Church of England, so that the Anglican religion must have been omnipresent in Anne’s childhood at the parsonage in Haworth. In this Christian context we can interpret the poem as follows.
The penitent regrets their former wrongdoing (‘thou shouldst sorrow so’), and the speaker feels the same way (‘I mourn with thee’). But in a neat paradox, the speaker also ‘rejoices’ because the penitent sincerely repents, and so they can be forgiven and saved. The ‘angel choirs’ are a reference to Luke 15:10:
Likewise, I say unto you, there is joy in the presence of the angels of God over one sinner that repenteth.
In line 4, ‘to bless the sinner’s woe’ is another paradox. This is metonymy: it is not the woe itself that is blessed, but the consequences of that woe, namely contrition, repentance, and salvation.
The second stanza describes the obstacles facing the penitent, namely the mockery of the non-religious (or insincerely religious) friends and family who would rather the penitent continue sinning. The ‘Redeemer’ is Christ, whose words from Luke 6:21–22 are referenced here:
Blessed are ye that weep now: for ye shall laugh. Blessed are ye, when men shall hate you, and when they shall separate you from their company, and shall reproach you
In the third stanza, ‘thy course’ means the course of repentance and salvation. ‘Earthly cords’ are the ties that bind the penitent to ‘Earthly’ (secular) concerns, and these are ‘riven’ (cut) by religious conversion. ‘Wondrous change’ is a common phrase in evangelical contexts meaning ‘conversion or awakening to Christianity’. In the poem this is lamented by ‘Man’, meaning the penitents’ acquaintances from the second stanza. The last line is a reference to Luke 15:7:
I say to you that likewise there will be more joy in heaven over one sinner who repents than over ninety-nine just persons who need no repentance.