T. J. B. Spencer's edition of Hamlet (New Penguin Shakespeare, 1980, page 236) provides the following endnote for "glimpses of the moon":
the earth illuminated by the uncertain light of the moon
G. R. Hibbard's edition (The Oxford Shakespeare, 1987, page 182) offers the following gloss:
flickering gleams of moonlight
A Shakespeare Glossary by C. T. Onions (revised by Robert D. Eagleson, Oxford University Press, 1986) defines "glimpse" as
Transient brightness, flash
I'm rather skeptical about the idea that the Elizabethans saw the moon flicker or flash.
The more plausible explanation for this characterisation of the moon's light as unreliable is that it intensifies the changes during the lunar cycle.
Remember Juliet's words in Act II, scene 2 in Romeo and Juliet:
O swear not by the moon, th'inconstant moon,
That monthly changes in her circled orb
The ghost of Hamlet's father only appears at night, when the moon is shining, and disappears again at or before dawn.
See for example Act I, scene 1, where Horatio says,
I have heard,
The cock, that is the trumpet of the morn,
Doth with his lofty and shrill-sounding throat
Awake the god of day, and at this warning,
Whether in sea or fire, in earth or air,
Th'extravagant and erring spirit hies
To his confine.
The description of the moonlight as uncertain fits with the uncertainty about the identity of the ghost:
is it really the ghost of Hamlet's father or malicious spirit attempting to lead Hamlet to his doom (as Horatio suggests later in Act I, scene 4).