I haven't (yet) read the novel in question, but I can explain the passage's simple meaning as a fluent English speaker. Let's take this apart, one piece at a time.
As one of the commenters mentioned, "by Jove!" is an exclamation, similar to "By God!". In fact, according to Wiktionary, the term comes from another name for the god Jupiter.
he'd sailed pretty near the wind once or twice! But he'd always got
away with it!
You're right; "sailing near the wind" means to do something that is "only just legal" or dangerous. (The Free Dictionary, citing the Cambridge Idioms Dictionary)
There wasn't much he drew the line at really.
You're close with your interpretation, here, but I think it needs a slight change to be perfect. I don't think this only means that he has always gotten away with "sailing close to the wind" without consequence. To draw a line means to make a limit for yourself -- to tell yourself that I'll drive 5 miles over the speed limit, but not 30. In this statement, the speaker is saying that they don't really have any self-imposed limits.
No,there wasn't much he'd draw the line at. He fancied that he was
going to enjoy himself at Soldier Island.
The word "fancy," as a verb (in the fourth entry here), means to imagine or suppose. These two statements together imply that the speaker expects to enjoy himself a lot on Soldier Island, because he doesn't have any self-imposed limits. I haven't read the book, but this statement is designed to make one feel uncomfortable about this character, because this character will have "fun" in a place away from regular society, probably even at the expense of other people. I would be wary about this one, if I were you.