I'm no expert on this particular poem. However, we know that Stevens is obsessed with (what he views as) the abstract, highly subjective distinctions between concepts such as reality/language, truth/imagination, idea/thing-itself, order/chaos, poem/mind, black/white [or'the first idea'/nothingness], and the scope of human potential versus the limited time which we are allotted to comprehend and articulate the world in order to truly and comprehensively understand anything. This is why we observe Stevens regress to terms like 'seem', ' or 'part', evidently reluctant to address reality as an objective fact, perhaps out of respect for Cartesian metaphysical Skepticism from which he draws the version of phenomenology which provides the foundation for everything he writes.
If you look at his many other poems, and understand how his poetry progresses from Harmonium to the less optimistic The Rock, this one becomes clearer. A Postcard From The Volcano, for example: 'what we said of it became / A part of what it is'. Similarly, some other examples sprinkled throughout his canon:
Of Modern Poetry: "poem of the mind in the act of finding / What will suffice"
Re-statement of Romance: "The night knows nothing of the chants of night. / It is what it is as I am what I am: / And in perceiving this I best perceive myself"
Not Ideas About the Thing But the Thing Itself: "a scrawny cry from outside / Seemed like a sound in his mind"
Man on the Dump and Of Mere Being are also good reference points for this.
On this basis, I am going to say that the line which you question refers to
Stevens talks about the book and reality throughout the poem, and it's all about solitude, solipsism, the sublime, the transcendental experience, and the process by which the reader becomes the creator, not the poet, through the act of reading itself (a concept I learn from Sartre's Being & Nothingness and What is Literature, which I'm sure originates elsewhere). I settle on this interpretation in particular because:
a) it recurs throughout American writing, in many of my favourite authors and poets, and
b) it's not only philosophically poignant, but also akin to Stevens's own preestablished philosophy.
Stevens digresses further on these philosophies in his meta-theoretical poetry. Comedian as the Letter C, Notes Toward a Supreme Fiction, and An Ordinary Evening in New Haven -- as well as his essays from The Necessary Angel -- all give a more direct version of his methodology and theoretical framework.
Similarly, you can actually track the poignant, highly Existentialist thoughts which he notes in his early poetry (Harmonium, Ideas of Order) and see how they break down, remain, or develop later. And this is no exception; it's very clear that Stevens is indebted to many philosophers focussing on temporality and being (Heidegger, Sartre, some Camus, Collingwood) as he usurps their lexicon and recreates the form, giving these European ideas an American Modernist platform
Sidenote: I believe also that Stevens is indebted to William Faulkner in this respect (the 'is was' concept from As I Lay Dying; the entirety self-reflexive historiographical form of Absalom, Absalom), but have not as of yet found any biographical evidence of this. Must be a symptom of the shared American experience? Very interesting regardless.
Hope this answers your question!