"Garn" is Cockney slang, in this context a reference to Pygmalion / My Fair Lady.
For the direct meaning of the word, you can check for example Wiktionary:
From go on.
Interjection
garn
(Cockney slang) A response that expresses disbelief or mockery.
The example quote provided in that Wiktionary page is from George Bernard Shaw's play Pygmalion, which was adapted into a famous musical and later an award-winning film both titled My Fair Lady. The story is about a linguistic professor who teaches a Cockney flower girl to speak "proper" English in order to pass her off as a duchess - the passage you quote from Rosemary's Baby is a direct and clear reference to this. Quoting from the play Pygmalion itself (emphasis mine):
THE FLOWER GIRL [tickled by the performance, and laughing in spite of herself] Garn!
THE NOTE TAKER. You see this creature with her kerbstone English: the English that will keep her in the gutter to the end of her days. Well, sir, in three months I could pass that girl off as a duchess at an ambassador's garden party. I could even get her a place as lady's maid or shop assistant, which requires better English. That's the sort of thing I do for commercial millionaires. And on the profits of it I do genuine scientific work in phonetics, and a little as a poet on Miltonic lines.
You can also listen to the line "Garn" from the film adaptation, and read more about its meaning from our friends at the English Language & Usage Stack Exchange.
In the passage you quote from Rosemary's Baby, it seems that Hutch is trying to raise Rosemary's intellectual and social level - responding to "improper" question by sending her to philosophy classes. He likens himself to Henry Higgins, the linguist who trains flower seller Eliza Doolittle to speak like a duchess. Rosemary has wit enough to reply in kind, saying "Garn!" in Eliza Doolittle style.