The caged bird’s “shadow” is “screaming”. This is in vain, as shadows cannot scream. The most the bird’s shadow can do is extend beyond the cage. So, let’s look at the description of the bird’s cage:
But a bird that stalks
down his narrow cage
can seldom see through
his bars of rage
The cage is “narrow”, and the bird “can seldom see through” it. The bird is trapped inside, but its shadow can slip out and scream. Scream helplessly, yes, but the very fact that the bird is shadow-screaming in the first place indicates that it knows its situation is unfair.
Now, there’s a qualifier to the scream - it is a “nightmare” scream. This is important for two reasons. First, it indicates the bird has dreams. Its screams may be prompted by nightmares, but the existence of dreams means it still clings to a hope of a better place. Second, this is more evidence that the bird knows how its situation is wrong, unnatural, unfair. It can acknowledge that it is living in a “nightmare” while simultaneously entertaining dreams of a change for the better.
The chorus of the poem explains what the bird’s dreams are for:
The caged bird sings
with a fearful trill
of things unknown
but longed for still
and his tune is heard
on the distant hill
for the caged bird
sings of freedom.
The caged bird is singing of “freedom”. This is what it is dreaming of, what it so desires while it whiles away its life trapped in a “nightmare”. Yet it does not know what freedom is. The concept of freedom, of flying out of the cage as the other bird does, is “unknown / but longed for still”. So the caged bird is calling out for an escape that it desperately wants but can’t articulate.
To directly answer your question - the line “his shadow shouts on a nightmare scream” means that the caged bird is calling out, using its shadow as a method of reaching beyond its cage. The scream still indicates that the bird nurses a dream of freedom. But without the ability to properly articulate its dreams, the bird’s shadow is left to fruitlessly, soundlessly reach out of its cage.
All quotes from the poem linked in the question.
Sources of inspiration:
https://www.shaalaa.com/question-bank-solutions/in-caged-bird-what-does-line-and-his-shadow-shouts-nightmare-scream-mean-writing_83557
https://www.enotes.com/homework-help/caged-bird-what-line-his-shadow-shouts-nightmare-662741
https://englicist.com/questions/explain-his-shadow-shouts-on-a-nightmare-scream