TL;DR: Joyce criticized dramatic flaws in Hamlet, but never condemned the play as a “failure”.
Richard Ellman’s biography of Joyce makes it clear that Joyce thought Henrik Ibsen a better dramatist than Shakespeare, and in 1908 he criticized the dramatic aspects of Hamlet in a conversation with his brother Stanislaus. However this has to be balanced against the fact that Joyce gave a long series of lectures on Hamlet in 1912–13, and included many allusions to the play in Ulysses and Finnegans Wake.
Joyce’s comparison with Ibsen also needs to be understood in context. It is a commonplace that Shakespeare was a better poet than dramatist: his plays are admired not because the plots are watertight, or the morals uplifting, but because his “skill in placing one syllable beside another has given acute pleasure to generation after generation of English-speaking people” (George Orwell). So when Joyce said that Ibsen “towers head and shoulders above him [Shakespeare] when it comes to drama” he did not mean that Shakespeare’s plays lacked other kinds of merit.
The key participant in the game of whispers was William H. Quillian, who summarized Ellmann like this:
On the whole, however, I accept the evidence put forward by Richard Ellmann […] that Joyce considered Shakespeare a “second-rate” playwright (after Ibsen) and Hamlet a “rude and barbarous” play (à la Voltaire).
Quillian’s use of quotation marks suggests that these phrases come from Ellmann or maybe even from Joyce as quoted by Ellmann. But in fact neither phrase appears in Ellmann, and both are misleading summarizations. Joyce considered Ibsen a better dramatist than Shakespeare, but that makes Shakespeare ‘second only to’, not ‘second-rate’. And although Ellmann presents evidence that Joyce once read out Voltaire’s criticism of Hamlet at a lecture, this does not mean that Joyce agreed with Voltaire.
Wikipedia cites McDougal.
McDougal cites Quillian:
In “Hamlet” and the New Poetic: James Joyce and T. S. Eliot (Ann Arbor, Mich., 1983), William H. Quillian analyzes the literary response to Hamlet in the decade from 1911 to 1922. Quillian demonstrates the enormous and positive hold that Hamlet exerted on the literary imagination of the nineteenth century before examining the “fundamental shift in orientation of critical inquiry” represented by the “condemnation of the play as an artistic failure by Eliot and Joyce.”
Stuart Y. McDougal (1989). ‘T. S. Eliot’. In Jackson R. Bryer (ed.), Sixteen American Authors, Duke University Press, p. 202.
Quillian cites Ellmann:
By Eliot’s “condemnation” I mean the famous condemnation of the 1919 essay (the subject of my 3rd chapter). The situation with Joyce is somewhat more complex. In my second chapter I try to show that when Joyce lectured on Hamlet in Trieste in 1912/1913, he adopted a unique method of “unravelling the mystery” of the play. On the whole, however, I accept the evidence put forward by Richard Ellmann, James Joyce (New York, 1959; rev. 1982), p. 398; that Joyce considered Shakespeare a “second-rate” playwright (after Ibsen) and Hamlet a “rude and barbarous” play (à la Voltaire).
William H. Quillian (1983). Hamlet and the New Poetic: James Joyce and T.S. Eliot, p. 147. University of Michigan Research Press.
Ellmann describes three occasions on which Joyce said that Ibsen was a better dramatist than Shakespeare. First, based on Joyce’s brother Stanislaus’s Trieste diary for 1908:
To Stanislaus he [James] continued to disparage Shakespeare at Ibsen’s expense. As he had objected to Macbeth at University College for its improbabilities, now he took Hamlet to task. He had just seen Salvini play the part in Trieste when, on February 6, 1908, he complained of the play’s gross dramatic blunders. ‘He said,’ wrote Stanislaus, ‘that Ophelia’s madness took all the force out of Hamlet’s simulation, and that her love for her father, whom the audience have seen to be a paltry old imbecile, is a caricature of Hamlet’s passion; and the evil in the King’s character that accounts for Hamlet’s hatred must be supposed for it is not dramatically explained.’ Ibsen was much better, although perhaps lacking in variety; Joyce remarked on March 2, 1908, ‘Ibsen has persisted in writing what was essentially the same drama over and over again. I suspect that Ibsen met the four of five characters whom he uses throughout his plays before he was twenty-five.’
Richard Ellmann (1982). James Joyce, p. 266. Oxford University Press.
Second, based on an “interview with [Felix] Béran’s widow, Frau Lisa Béran, by Dr. Alfred Dutli, 1956”:
At the end of 1915, he [Joyce] obtained some work from a bearded messianic professor from Vienna, Siegmund Feilbogen. […] But they began to talk of Ibsen, and Joyce proved Ibsen’s superiority to Shakespeare so eloquently that he won Feilbogen over.
Ellmann, p. 398.
Third, based on “Ole Vinding, ‘James Joyce in Copenhagen,’ in Willard Potts, ed., Portraits of the Artist in Exile (Seattle, 1979), 139–52”:
‘Do you then place Ibsen higher than Shakespeare?’ asked Vinding. ‘He towers head and shoulders above him when it comes to drama [said Joyce]. No one approaches him there. It’s very difficult to believe that Ibsen will grow stale; he will renew himself for every generation.’
Ellmann, p. 694
The only relevant occurrence of ‘Voltaire’ in Ellmann is in a description of Joyce’s lecture series:
The following notices of Joyce’s lectures on Hamlet (translated here into English) appeared in the Piccolo della Sera, Trieste. […]
11 February 1913 (p. 2):
Dr James Joyce’s Lectures. Yesterday evening Dr. James Joyce concluded his series of lectures in English on Hamlet. […]
As Joyce indicated yesterday, he had purposely refrained from critical or philsophical disquisitions about the play he was reading and interpreting. His first task was to explain the words. His original and slightly bizarre talent changed the nature of his commentary, which might otherwise have been dry, into attractive ‘causeries’. The words, the manners, and the dress of the Elizabethans stirred the lecturer to literary and historical recollections which proved of keen interest to an audience which had been his for so many hours.
Yesterday evening, accepting the duty of closing such a work with a critical synthesis, he read (in English translation) the attack of Voltaire on Hamlet, and then, suddenly, the eulogy of the same work by Georg Brandes. We think that many in an audience that was capable of following these lectures will find themselves encouraged, as Joyce intended, to read other works of the great Englishman in the original.
Ellmann pp. 775–6.
Of course, the mere reading of Voltaire’s attack does not mean that Joyce agreed with it, any more than his reading of Brandes’ eulogy means he agreed with that. This is the passage from Voltaire:
I certainly cannot justify everything in the Tragedy of Hamlet; it is a rude and barbarous piece, which would not be approved even by the most wretched people of France or Italy. Hamlet goes mad in the second act, and his mistress goes mad in the third; the prince kills his mistress’s father in the belief that he is killing a rat, and the heroine throws herself in a river. A grave is dug on stage; the gravediggers make jokes while holding the skulls of the dead in their hands; prince Hamlet replies to their abominable vulgarities with lunacies no less repugnant; meanwhile, one of the actors conquers Poland: Hamlet, his mother, and his step-father drink together on stage; they sing at table; they quarrel; they kill each other; we would believe this outrage were the fruit of the imagination of a drunken savage. But amid these vulgar absurdities, which are performed to this day to the absurd and barbarous English, we find in Hamlet sublime features, worthy of the greatest geniuses. It seems that nature pleased itself to mix in the head of Shakespeare, the greatest strength and grandeur imaginable, with whatever witless vulgarity can make basest and most detestable.
Voltaire (1748). ‘Dissertation sur la Tragédie’. In La Tragédie de Sémiramis. My translation.