If you enter the google books search "part second" you will see this construction was a perfectly normal practice in the 1700 and 1800s. Here are some of the hits I got: to The Ploughboy: A Poem, Part Second (W. Cook, 1855), to Sadlier's Catholic first reader: part second, Part 2 (James A. Sadlier, 1883), and to Zoonomia ; Or, the Laws of Organic Life: Part Second. By Erasmus Darwin, M.D. of 1797. There are many others.
I think "part the second" is more common, as in Both Sides of the Gutter: Part the second by P. Byrne.
Both it and "part second" mean, of course, "the second part". You are possibly familiar with Carroll's The Hunting of the Snark, whose cantos are "Fit the first", "Fit the second", which is more of the same sort of thing.
For yet more instances of this construction, try google books "chapter fifth": I got hits on Walter Scott, Jules Verne, and so on.
To answer your question: your book says "Part first" and so on because that's what the author wrote. This style seems odd or ungrammatical to you only because you are imperfectly familiar with 19th century English.
As a speculative aside: I think usages like "Part the first" and so on might once have been more common than "Part one", with the ordinal form of numbers being more common in such contexts than the cardinal form, and that the use of Arabic or Roman numbers in spelling might be connected with this shift. To this day when we read aloud the written words "King Richard III" we actually say "King Richard the third", and sometimes when we see "December 25" we say "December twenty-fifth" or "December the twenty-fifth". Although written English is "alphabetic", the phonetic basis or theory of the alphabet (which in English is applied in notoriously quirky ways, but still applied) does not govern the pronunciation of Arabic or Roman numerals. According to this speculative theory, the usage "Part first" that OP reports would once have been a correct reading of "Part 1" and its use here reflects both the conservative habit of using ordinals in this context and an archaizing tendency to avoid the use of Arabic numbers in the form "Part 1".