To understand this, once needs to know a little Irish history. Essentially Ireland has been an island riven by violence almost continually for over a thousand years. First it was the Vikings, raiding, pillaging and settling. Then it was the Normans followed by the Tudors who, through the reformation, then lead a bloody, divisive series of religious wars which sucked in people from all over the United Kingdom. The repercussions of this are still being felt today and were very much in evidence at the time that Ulysses was written. This is the "nightmare" of Ireland's history.
How Stephen frames his statement is purposefully cryptic: it's possible that it's meant to be one of those things which sounds quite profound but is in fact relatively meaningless. He is a teacher, and clearly a clever and well-learned one at that. However, during the scene where he utters this phrase he's been cornered by Mr. Deasy, the headmaster who, although an authority figure, seems far less imaginative than Stephen. Deasy is trying to reverse roles and place Stephen in the role of the pupil in order to press his authority: Stephen resents this and says a number of cryptic things to Deasy apparently with the purpose of confusing him.
Stephen jerked his thumb towards the window, saying:
-- That is God.
Hooray! Ay! Whrrwhee!
-- What? Mr Deasy asked.
-- A shout in the street, Stephen answered, shrugging his shoulders.
The statement is also a direct rejection of two other interpretations of history that have been pressed upon Stephen. In the first chapter, Haines says:
Haines detached from his underlip some fibres of tobacco before he spoke.
-- I can quite understand that, he said calmly. An Irishman must think like that, I daresay. We feel in England that we have treated you rather unfairly. It seems history is to blame.
This frames History as something impersonal and relegated to the past (i.e. the English can "blame" it, squarely and without repercussions). Stephen's characterization is of history as something that is both ongoing ("history is") and personal ("I am trying").
Deasy, on the other hand, agrees that history informs the present but seems it as moving toward a single, unified outcome which has been decreed by divine will.
-- The ways of the Creator are not our ways, Mr Deasy said. All history moves towards one great goal, the manifestation of God.
This implies that, since God is merciful, the terrors of the present (and the past) will lead toward a positive outcome. It also, like Haine's statement, uses history as a way to avoid blame: in this case the blame can be placed on those who choose to reject the gospel. Stephen's characterisation of it as a "nightmare" refutes all aspects of Deasy's opinion, since a nightmare is awful, uncertain and something over which you have no choice.
The last history which Stephen invokes with this statement is his own. Compared to the other characters we have met so far in Ulysses, Stephen is not high in wealth or social status, yet he finds himself at University in the company of the wealthy, the well-connected and the powerful. He feels guilt about his behavior around his mother's death. His own history is - comparatively - a nightmare, which he is trying to escape.
