"Once" here means "ever", so Hamlet is asking, "Would the heart of man ever think it or imagine it / this?"
The phrase "the heart of man" may refer to a conception of the heart as "the seat of intelligence, motion, and sensation", which goes back to Aristotle [1]. (Aristotle also held that "the heart as being the hottest of all the bodily parts, is the counterpoise of the brain." [2])
Based on this theory of anatomy, Hamlet's question means, "Would man's intelligence ever think it / this?"
However, Galen's theory of physiology was also very influential. According to Galenist medicine, "[r]eason is located in the brain, emotion (particularly anger) in the heart, and desire in the liver" [3]. With this in mind, one may paraphrase Hamlet's question as asking how his emotions would deal with this. His anger about what the ghost has revealed is very clear, for example in lines such as "O villain, villain, smiling damnèd villain!" earlier in the same scene.
[1] Paula Findlen: A History of the Heart (History Department, Stanford University)
[2] Aristotle: On Sense and the Sensible, translated by John Isaac Beare.
[3] Amy M. Schmitter: Ancient, Medieval and Renaissance Theories of the Emotions. Stanford Encyclopedia of Philosophy, 2010.