According to A Shakespeare Glossary, the adjective "kind" had several meanings:
- natural, appropriate, proper (…)
- favourable, gracious (…)
- affectionate, loving, fond (…)
Based on this, the relationship between Hamlet and Claudius is described as a little more than just kinship (Claudius is not just his uncle but now also his stepfather) but less than affectionate. Hamlet is more than just a "cousin" now; "cousin" could mean "nephew" but could also refer to "any kinsman more distant than a brother" (Jenkins).
What is also interesting about this word choice (i.e. "kind") is that in Act II, scene 2, Hamlet describes Claudius as a "kindless villain", i.e. an unnatural villain, someone "lacking in feeling for one's own kind" (Harold Jenkins's gloss).
G. R. Hibbard points out that Hamlet here speaks his own version of the proverb "The nearer in kin, the less in kindness" (cited in M. P. Tilley's A Dictionary of the Proverbs in England in the Sixteenth and Seventeenth Centuries). This would imply that by becoming closer in their relationship (stepfather instead of just uncle), the dislike has increased. From this point of view, "less than" would refer to a reduction rather than total absence of any affection.
Note also that the line "A little more than kin, and less than kind" fits the usual metre of Shakespeare's verse, namely the iambic pentameter. Replacing "less than" with "not" would result in only nine syllables instead of the usual ten. If the intended meaning is "not", Hamlet is using understatement or, as Mary's answer pointed out, meiosis. None of the Hamlet editions I consulted seem to find this worth mentioning.
The question says, "I completely understand why Hamlet must use litotes if he's chiding Claudius (...)". There is no evidence that Hamlet's words are actually about Claudius. T. J. B. Spencer points out that "we see no action of the King towards Hamlet which is not, at least on the surface, affectionate" (p. 220). G. R. Hibbard's gloss implicitly assumes that Hamlet's comment is about himself. Bernard Lott's gloss makes this more explicit than the other ones: "closer to you than a mere relative (because you are now in a sense my father), but not very kindly disposed to you".
Bernard Lott's edition is the only one that points out that "kin" and "kind" is a play on words, since both words derive from the same Old English word. None of the editions mention that Hamlet might be making a bilingual pun.
References
- Onions, Charles Talbut; Eagleson, Robert D.: A Shakespeare Glossary. [1986] Enlarged and Revised Throughout by Robert D. Eagleson. Oxford: Oxford University Press, 1993 [1988].
- Hamlet. Edited by T. J. B. Spencer. Introduced by Anne Barton. The New Penguin Shakespeare. London: Penguin, 1980.
- Hamlet. Edited by Bernard Lott. New Swan Shakespeare: Advanced Series. Harlow, Essex: Longman, 1990 [1968].
- Hamlet. Edited by G. R. Hibbard. The Oxford Shakespeare. [1987] Oxford: Oxford University Press, 1989.
- Hamlet. Edited by Cyrus Hoy. Norton Critical Editions. [W. W. Norton & Company, 1963] Second Edition. New York; London: W. W. Norton & Company, 1992.
- Hamlet. Edited by Harold Jenkins. The Arden Shakespeare. [Methuen, 1982] London: Routledge, 1993.