The relevant senses are:
work, v. II.9. transitive. To cause, bring about, produce as a result; to accomplish, achieve, attain.
annoy, n. 1. A feeling of discomfort, displeasure, or weariness; annoyance, vexation. Also more strongly: affliction, suffering, tribulation.
Oxford English Dictionary.
I think the stronger senses of “annoy” work better in context, since Giaffir has observed “rebellion there begun” in Selim’s eye, and since the Ottoman Empire was notorious for violent transitions of power. So “work me more annoy” means “cause me more tribulation”.