It appears that the "contes" are written in regular verse and use an older diction, while the "nouvelles" are written in irregular verse, with rhyme scheme and meter varying in the course of the poem. I have not been able to read the Tiphaine Rolland book you mentioned in your comment, but a review in the Italian language by Federico Corradi of said book mentions in passing:
l’alternanza tra nouvelles in versi irregolari e contes in vieux langage risponde ad un’esigenza di varietà.
the alternation between nouvelles in irregular verse and contes in vieux langage responds to a need for variety.
Corradi, Federico. Review of L’Atelier du conteur. Les “Contes et nouvelles” de La Fontaine. Ascendances, influences, confluences by Tiphaine Rolland. Studi Francesci: Rivista quadrimesta fondata de Franco Simone Volume 60 No. 2, Issue 179, 2016. pp. 321–322. https://doi.org/10.4000/studifrancesi.4349. Quoted line is on p. 322. Translation mine and AYOR.
A glance at de la Fontaine's text does seem to confirm this. For example, the conte "La Clochette" ("The Little Bell") begins:
O Combien l'homme est inconstant, divers,
Foible, leger, tenant mal sa parole !
J'avois juré, même en assez beaux Vers,
De renoncer à tout conte frivole;
Et quand juré? c'est ce qui me confond,
Depuis deux jours j'ai fait cette promesse;
Puis fiez-vous à Rimeur, qui répond
D'un seul moment. Dieu ne fit la sagesse
Pour les cerveaux qui hantent les neuf sœurs:
Trop bien ont-ils quelque art qui vous peut plaire,
Quelque jargon asses plein de douceurs;
Mais d'être sûrs, ce n'est-là leur affaire.
Oh, how men are inconstant, variable, fickle, light, and poor keepers of their word! I had sworn, even in verses beautiful enough, to renounce all silly tales. And when did I swear this? That's what defeats me; I promised two days ago. Then put your faith in the rhymer, who is answerable only for a single moment. God did not make wisdom for the brains that the nine sisters haunt. They have far too much of some art that can please you, some jargon quite full of sweetness; but it is not their business to be reliable.
La Fontaine, Jean de. "La Clochette: Conte." Contes et Nouvelles en vers. 1665. Vol. 1. London, 1755. pp. 170–172. Lines quoted are on p. 170. Translation mine and AYOR.
The poem is fairly short compared to some of the nouvelles, and the rhyme scheme of abab is maintained throughout.
By contrast, "La Coupe Enchantée: Nouvelle tirée de l'Arioste" ("The Enchanted Cup: A nouvelle taken from Ariosto") is irregularly rhymed from the get-go:
Les maux les plus cruels ne font que des chansons
Près de ceux qu'aux Maris cause la jalousie.
Figurez-vous un fou , chez qui tous les soupçons
Sont bien venus , quoi qu'on lui die,
Il n'a pas un moment de repos en sa vie.
Si l'oreille lui tinte , Ô Dieux! tout est perdu,
Ses songes font toujours que l'on le fait cocu;
Pourvu qu'il songe , c'est l'affaire.
The most cruel evils are mere songs when compared to those things that cause husbands to be jealous. Imagine a madman who welcomes all suspicions, regardless of what they claim. He has not a moment of rest in his life. If his ear rings, O gods! all is lost. His dreams are always that he is being cuckolded. And he believes that whatever he dreams is the case.
la Fontaine, "La Coupe Enchantée." Contes pp. 96–112. Lines quoted are on p. 96. Translation mine and AYOR.
The verse-pattern of these lines is not maintained over the rest of the nouvelle. It seems ad hoc, and subsequent lines follow equally ad hoc patterns. They are creative and delightful to work out, but not regular in the way of "La Clochette." And this nouvelle is considerably longer than that conte.
Insofar as these two poems are representative, they confirm Corradi's remark that the nouvelles are in irregular verse and the contes more formal. They also bear out your hypothesis that length is a factor in deciding whether a given poem is nouvelle or conte. Perhaps the strictly regular verse forms used in the contes make them unsuitable for more extended narratives.
However, all of this is only a hypothesis based on a hint in Corradi. I think Rolland's work would be most useful in supporting or disproving it, but since it has been some months and no answer has been put forward, I thought this might be better than nothing. Hopefully someone who has access to Rolland's monograph and/or is an expert on La Fontaine and/or has a more competent grasp of French (and, for Corradi, Italian), will provide a better answer soon.