I'm pretty sure I once (a long time ago) read a Joyce quote where he said that only twelve (or was it thirteen?) readers would ever fully understand Finnegans Wake. But I couldn't convince Google to find a source for this. Do I remember correctly or did my memory make this up?
I found a quote that may be what you're remembering, or slightly misremembering. It's not from Joyce himself, but I think it's close enough to be worth posting as an answer. It concerns Work in Progress, the title Joyce used to refer to Finnegans Wake before its complete publication, and the idea that only twelve or thirteen people might be able to fully understand the book.
Anyone who is prepared to make the reading of James Joyce’s new, incomplete book a life’s career, and who has the lexicographical skill to construct a James Joyce encyclopædic dictionary might conceivably derive emotional benefit from Work in Progress, and might procure the same benefit for at most a dozen other bizarre human beings. Apart from such thirteen human beings, Work in Progress will not be read, because it cannot be read by any individual normally constituted.
Ulysses has had many respectable imitators. Work in Progress will never be respectably imitated. I think it ought to rank as the oddest novel written. It will probably be unique.
If James Joyce is content with a possible thirteen readers, that is his affair and his alone.
-- Arnold Bennett, ‘Books and Persons: The Oddest Novel Ever Written’, Evening Standard (8 August 1929), 7. Reproduced in Robert H. Deming (ed.), James Joyce: the Critical Heritage, Volume 2: 1928-41.