It may refer to the Common Law Tort of ‘criminal conversation’, where ‘conversation’ means ‘sexual intercourse’. Criminal proceedings could be brought by a spouse, usually the husband, against a third party seeking compensation for breach of fidelity.
Wikipedia states:
Suits for criminal conversation reached their height in late 18th and early 19th-century England, where large sums, often between £10,000 and £20,000 (worth upwards of £10–20 million in today's terms), could be demanded by the plaintiff, for debauching his wife.
The article further cites:
Sir Richard Worsley lost his case against George Bisset, after it had been found that he had colluded in his own dishonour by showing his friend his wife, Seymour Dorothy Fleming, naked in a bath-house (technically he won the case, as the fact of adultery was not contested, but the jury awarded only one shilling damages).
Which appears to confirm that juries would ‘cast up’ (calculate) the compensation due to the husband in respect of the value of his wife.
The cases brought were private prosecutions, and the sum is technically compensation for damages experienced by the spouse who brought the case, rather than a fine for breaking the law.
These cases have no direct relevance to prostitution, referring solely to a situation where a spouse is ‘debauched’ by a third party. The compensation is to the partner of the debauched individual, rather than the individual themself.