There are a number of reasons why Ishmael might be considered unreliable, although none of them is conclusive. In most cases, there is an alternative explanation for Ishmael's seeming duplicity.
It starts, of course, with the famous opening "Call me Ishmael" which can be read in the context of an assumed name. I won't pursue this further as you have an independent question on this.
The opening then goes on to cast Ishmael as a troubled individual, as indeed he might be given his story.
Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin warehouses, and bringing up the rear of every funeral I meet; and especially whenever my hypos get such an upper hand of me, that it requires a strong moral principle to prevent me from deliberately stepping into the street, and methodologically knocking people's hats off
These do not seem to be ruminations of an entirely sound mind. And if the mind is not sound then nor may be the story that it is relating.
It is striking, for example, that past the opening chapters of the book, we hear very little about Ishmael. He rarely describes his own part in the narrative or his actions aboard the ship. We learn little about him: there are only two mentions in the whole book about his family, for example, an uncle and a step-mother.
These seem peculiar omissions for a man describing a narrative and suggest he may have something to hide. Alternatively, however, it may simply be a literary device. Melville needed a survivor to tell the tale, but the tale is really Ahab's. So it makes sense that the focus of the story moves more toward the captain as it unfolds.
Through the book, Ishmael describes on the thoughts, feelings and motivations of many other characters. He often does this in moments when he is not present. There are many examples: a particularly extreme one is chapter 109, a short chapter which describes in detail a conversation between Ahab and Starbuck in the captains' cabin.
A relatively lowly crew member such as Ishmael simply would not have been present at such a meeting, let alone able to relate in such detail. However, again, Melville has constrained himself by his choice of narrative device. Often, to tell the story, it is necessary to relate incidents outside of Ishamel's direct experience.
In contrast, however, Ishmael is hazy about many of the details of his own life:
Some years ago – never mind how long precisely….
We learn that he is a sailor who has previously only served on merchant's vessels. This fails to explain how he seems to have an almost encyclopaedic knowledge of whaling. Far more, you would imagine, than a novice whaler could learn on their first and only voyage. Yet again, though, Melville wants to talk about whaling, in detail, so he has to furnish Ishmael with this knowledge in order to include the desired content.
Finally, there are occasions when Ishmael makes some very odd narrative decisions. Take this, for example, from chapter 54:
For my humor's sake, I shall preserve the style in which I once narrated it at Lima, to a lounging circle of my Spanish friends, one saint's eve, smoking upon the thick-gilt tiled piazza of the Golden Inn.
Why would Ishmael take the trouble to let us know that he's telling us a story in a given "style", rather than simply relating the story? This may be a stylistic flourish my Melville, but it may also be an indication that Ishmael isn't telling the whole truth when he relates his tale.
