TL;DR: No convincing explanation has yet been put forward.
Perhaps nowhere else in Browning scholarship has more critical ink been expended on fewer words than the three of “Hy, Zy, Hine”.
James F. Loucks (1974). ‘“Hy, Zy, Hine” and Peter of Abano’. Victorian Poetry 12:2, p. 165.
Criteria
A good explanation would meet the following criteria.
In this stanza the speaker describes a plan to pretend to sell his soul to Satan in return for the latter destroying Brother Lawrence’s rose-acacia. (A rose-acacia is a delicate ornamental shrub imported from the American southwest, so Lawrence is understandably proud of succeeding in growing one.) After “Hy, Zy, Hine…” the speaker seems to be interrupted by the evening prayer service. It would be a point in favour of an explanation if it fit into this context.
If the explanation depends on an obscure source, it would be a point in its favour if Browning could be shown to have had some knowledge of the source prior to 1842.
Browning’s immersion in medieval sources means that his references are often obscure, but this does not make it likely that he would deliberately obfuscate his sources, for example by misspelling words. It would be a point in favour of an explanation if it didn’t rely on Browning doing this.
The words “Hy, Zy, Hine…” are printed in italics in all editions of the poem. In the rest of the poem, phrases in italics are consistently used for the words of Brother Lawrence which the speaker mocks. In the second stanza the speaker mocks Lawrence’s poor Latin (‘salve’ is an imperative, so ‘tibi’ is otiose) and dull conversation:
At the meal we sit together:
Salve tibi! I must hear
Wise talk of the kind of weather,
Sort of season, time of year:
Not a plenteous cork-crop: scarcely
Dare we hope oak-galls, I doubt:
What’s the Latin name for “parsley”?
What’s the Greek name for Swine’s Snout?
At the beginning of the fourth stanza, “Saint, forsooth” mocks Lawrence’s mild oath after he snaps a lily. And “Plena gratiâ / Ave, Virgo!” at the end of the last stanza mocks Lawrence’s poor knowledge of the vesper service (in which the word ‘virgo’ does not appear: the hymn begins “Ave Maria gratia plena”). It would be a point in favour of an explanation if it were consistent with this typographical convention.
Vesper bells
Nineteenth-century commentators, if they tried to interpret the words at all, took them for the sound of the vesper bells. James Stirling had a detailed theory about the meaning of each syllable:
The Hy-Zy-Hine represents the sound of the bell—Hy, as it opens its mouth left, Zy, as it pours itself out right, and Hine, as it drops its own extinguisher on its own sound in the midst.
James Hutchison Stirling (1868). ‘The Poetical Works of Robert Browning’. The North British Review 98, p. 402. Published anonymously, but authorship confirmed by Amelia Hutchison Stirling (1912), James Hutchison Stirling: His Life and Work, London: Unwin, p. 188.
These details seem to me unconvincing and even self-contradictory: ‘extinguisher’ must mean ‘damper’ which implies that the bell was fixed in place and rung with a hammer; but “opens its mouth left” implies that it was swung and rung with a clapper.
Later commentators followed Stirling in outline if not in detail, though Peterson’s “apparently” suggests some skepticism:
Hy, Zy, Hine: represent the sound of the vesper bell.
Hiram Corson (1888). An Introduction to the Study of Robert Browning’s Poetry, p. 313. Boston: D. C. Heath.
This is apparently meant to represent the sound of the Vesper’s bell.
Hans Christian Peterson (1903), Inductive Studies in Browning, p. 14. Chicago: Ainsworth.
Twentieth-century scholars tended to favour other possibilities and criticized the ‘vesper bells’ interpretation, first, because Browning’s choice of syllables is unnecessarily obscure if bell-ringing is intended:
As an echo of bell-notes the syllables seem curiously inept. English ding-dong, French din-dan, Spanish tin-tan might have served. Our speaker’s mood is hardly such that he can be supposed to shift instantly into fanciful imitative mouthings; and had Browning meant to throw in original bell-syllables of his own, could he not have done better than hy, zy, hine? When he employed onomatopoeia it was not usually subtle or obscure.
Fred A. Dudley (1957). ‘Hy, Zy, Hine’. Research Studies of the State College of Washington XXV, p. 65.
I cannot believe monastery bells to be very different from other bells, which do not put forth such strange sounds.
Loucks (1974), p. 165.
Second, if the sound of the bells is to fit into the soliloquy we have to imagine the speaker (or Brother Lawrence) singing along with them, but then:
[They] have not satisfactorily explained why the speaker appears to be interrupting himself (“’St, there’s Vespers”) when the actual bells begin to ring.
Loucks (1974), p. 165.
Richard Wear tried to rescue this interpretation by suggesting that it is Brother Lawrence who is singing along with the bells:
It is precisely his simple, loving nature that would cause Lawrence to burst into joyous, if somewhat foolish, song at the sound of the Vesper bells. Because the speaker is so riveted in his rage on Lawrence’s every act, he mocks the absurd syllables, Hy, Zy, Hine, only to realize with a start what they indicate: “’St, there’s Vespers!”
Richard Wear (1974). ‘Further Thoughts on Browning’s Spanish Cloister’. Victorian Poetry 12:1, p. 69.
Satanic conjuration
Charlotte Porter and Helen Clarke were the first to suggest that the words might be the interrupted beginning of a conjuration. They did so, without any indication that they were introducing a novel interpretation, in the middle of a list of self-study questions on the poem:
Does the conjuration he begins (line 70) suggest Arabic words? How do they contrast with the interruption of the call to vesper service, and the Latin “Hail Mary” following?
Charlotte Porter and Helen A. Clarke (1900). Browning Study Programmes, p. 527. New York: Thomas Y. Crowell.
Fred Dudley drew attention to Porter and Clarke, and suggested that the words might have been inspired by De occulta philosophia by Cornelius Agrippa, which Browning had quoted from in his note to ‘Paracelsus’ (1835):
A glance almost anywhere in [De occulta philosophia] would have turned up elaborate discussions of sorcery, charms, astrology, necromancy, numerology, or the powers of names. There are complicated instructions for “deriving” the secret names of angels and devils from the names of God. The procedure requires tracing the letters of known names through tables marked up one side and down the other with the the characters of the Hebrew alphabet, across the top and backward across the bottom with the signs of the zodiac or of the planets. One direction of tracing yields names of good spirits; the reverse, those of evil spirits. Browning’s Zy might just conceivably echo “the name of the evill spirit Schii … which signifies a spirit that is a work of engines.”
Dudley (1957), p. 67.
but Dudley recognized that this was a poor effort and immediately retreated to the idea that the phrase is a nonsensical invention:
But surely the outright invention of three bizarre syllables was not beyond the powers of Robert Browning.
Dudley (1957), p. 68.
Gordon Pitts suggested that Browning might have come across the Beauvais manuscript of the ‘Feast of the Ass’ in the British Museum during his research for Sordello (1840).
The refrain of the mass throughout is “Hez hez sire asnes hez,” Old French for “Heigh, heigh, Sir Ass, heigh”. Elsewhere in the mass too occurs the phrase Hinham, Hinham, Hinham (he-haw). These refrains could easily become slurred in the mind of speaker (or of Browning) into “Hy, zy, hine”
Gordon Pitts (1966). ‘Browning’s “Soliloquy of the Spanish Cloister”: “Hy, Zy, Hine”’. Notes and Queries 13:9, pp. 339–340.
James Loucks suggested that the words are altered forms of names appearing in the Elementa Magica attributed to Pietro d’Abano, which Browning might have encountered during his research on Paracelsus, since it is printed as an appendix to De occulta philosophia.
The following words serve to open the [“Coniuratio diei Martis”]:
Coniuro & confirmo super vos, angeli fortes & sancti, per nomen Ya, Ya, Ya, He, He, He, Va, Hy, Hy, Ha, Ha, Va, Va, Va, Aie, Air, Air, El, Ay, Elibra, Eloim, Eloim [etc.]
What follows is the concluding sentence of Abano’s “Exorcismus Spirituum Aereorum”
Venite ergo in nomine Adonay, Zebaoth, Adonay, Amioram: venite, venite, quid tardatis? festinate, imperat vobus Adonay Saday, Rex Regum, El, Aty, Titeip, Azia, Hyn, En, Minosel, Achadan, Vay, Vaa, Ey, Haa, Eye, Exe, a El, El, a, Hy, Hau, Hau, Hau, Va, Va, Va, Va.
It will be noted that the words “Hy” and “Hyn” appear in these passages; I am guessing that Browning borrowed the word just before “Hyn” in the second passage (“Azia”), altering it to “Zy” to echo “Hy.” As for “Hyn,” presumably Browning altered its spelling to make its rhyme with “swine” more exact.
Loucks (1974), pp. 168–9.
Susan Hardy Aiken suggested that Browning, rather than finding the ‘Feast of the Ass’ in the Beauvais manuscript in the British Museum, might have found it in Ancient Mysteries Described (1823) by William Hone.
Hone bases his account of the Feast of the Ass on the Beauvais manuscript and a thirteenth-century missal composed by the archbishop of Sens, both printed in Latin and Old French by Du Cange. Noting that the service begins immediately “before vespers,” he describes “the clergy [going] in procession to the door of the cathedral,” where they chant, “in a minor key, or rather squeaking voices," mock vesper anthems in adoration of the Ass. He then translates the first and last stanzas of the Mass, rendering its refrain as “Huzza, Seignor Ass, Huzza!” and
other parts of the service were terminated by the burden of Hin-Han, Hin-Han, in imitation of the braying of an ass […]
Hone’s Huzza (which unlike the French hez requires pronunciation of the z) becomes Hy, Zy; and the Hin-Han ending the mock Vespers becomes “Hine…”—its second syllable interrupted, as Browning’s ellipsis suggests, by true Vespers.
Susan Hardy Aiken (1979). ‘“Hy, Zy, Hine” and Browning’s Medieval Sources for “Soliloquy of the Spanish Cloister”’. Victorian Poetry 17:4, pp. 377–383
These explanations are in my opinion all quite implausible: they depend on Browning picking indiviual words from an obscure text and altering them so as to make the allusion unrecognizable, without having any understandable motivation for doing so.
Other explanations
Alexandra Orr managed to give the impression of glossing the phrase, but without committing to any particular meaning:
He is turning over some pithy expedients, when the vesper bell cuts short his meditations.
Alexandra Orr (1885). Handbook to the Works of Robert Browning, p. 252. London: Bell & Sons.
Joseph Baker’s suggestion has at least the merit of simplicity:
These meaningless words represent a growl of hatred
Joseph E. Baker, ed. (1947). Pippa Passes and Shorter Poems, p. 71. New York: The Odyssey Press.
Adam Roberts noted the similarity between ‘Hy’ and Greek ‘hys’ meaning ‘swine’, and wondered whether
the words are there to evoke a speaker with poor Greek, vaguely remembering a Greek-language magic charm for transforming an enemy into the shape of a pig?
Adam Roberts (2014). ‘Hy, Zy, Hine’.
The prize for ridiculous contrivance goes to James Anderson:
Blended, perhaps, with the tones of the vespers bells of the next line, but ascribable neither to them nor to Brother Lawrence nor to the soliloquist himself, these words are best read in literal Miltonic terms as a truly satanic utterance, the intrusion of the Devil’s alien voice, summoned into the cloister garden by the soliloquist’s human evil. Satan himself, father of many a curse, must logically be speaking the words “Hy, Zy, Hine…” […]
Strange imprecation though they may be, these satanic words nevertheless cannot be dismissed as a mere nonsense-formula. In fact they belong to English, albeit English of a dialectal variety. […] Thus Browning’s italicized words “Hy, Zy, Hine…,” however mysterious or incantatory they may seem, also make perfectly good lexical sense as a satanic imperative: “Hurry up, sink down, you (slavish) lout!”
James E. Anderson (1997). ‘Robert Browning’s “Soliloquy of the Spanish Cloister”: Themes, Voices, and the Words, “Hy, Zy, Hine”’. Victorian Poetry 35:3, pp. 322–3.
No-one has yet built a theory on the coincidence that ‘hy zy’ means ‘he is’ in West Frisian and ‘hine’ meant ‘theirs’ in Old Frisian.