Presumably it implies that you can't get inside except by flying.
This stanza of the poem is describing the physical strength of the castle and its defences. There is no way to scale the walls, no way (so the poet claims) for an unwanted attacker to get in. The only creature that can get in is one that can fly: of course, being able to travel through the air would render strong walls useless without an equally strong roof. So birds can get in, but nobody else can.
A bird is frequently used in literature as a representative of the category of creatures that can fly. (Technically, insects and bats could also have got in to the castle, but poetry and pedantry don't mix well.)