The corresponding passage from Chapter VI in the French text goes as follows:
Pour la première fois peut-être, j'ai pensé vraiment que j'allas me marier.
Masson voulait se baigner, mais sa femme et Raymond ne voulaient pas venir. Nour sommes descendus tous les trois et Marie s'est immédiatement jetée dans l'eau.
The phrase "ne voulaient pas venir" literally means "did not want to come [with us]", whereas the English translation quoted in the question reads "were disinclined to move". "Disinclined" sounds less direct and sounds more formal than Meursault's language in the rest of Part 1 (compared to the French text, at least).
However, after the swim, Meursault tells us nothing about any unusual interaction between Raymond Sintès and Masson's wife. And later, Meursault, Masson and Raymond go for a walk on the beach while Marie and Masson's wife wash the dishes. During that walk, Meursault tells us that Raymond and Masson have known each other for a long time and had even lived together for some time. This would be a good moment to suggest that their might have been some cheating or a ménage à trois, but we hear about nothing of the sort. It would also have been a good moment to suggest an affair during this walk, since this is also when the three men encounter the two "Arabs", one of which had been beaten up by Raymond because he had suspected him of cheating him with his mistress; if Raymond had been having an affair with Masson's wife, the dramatic irony would have been particularly strong.
The corresponding passage from Chapter III in the French text goes as follows:
Juste à ce moment est entré mon deuxième voisin de palier. Dans le quartier, on dit qu'il vit des femmes. Quand on lui demande son métier, pourtant, il est "magasinier".
(Instead of using a noun, the French text uses a verb construction: "vivres des femmes".) Meursault also says:
Je trouve que ce qu'il dit est intéressant. D'ailleurs, je n'ai aucune raison de ne pas lui parler.
My translation:
I think [literally: I find] that what he says is interesting. Besides, I have no reason not to speak with him.
Elsewhere in the novel, Meursault never expresses a negative opinion about Raymond. (The most critical comment we get is in Part II, where Meursault says that Céleste is "better" ("vaut mieux") than Raymond.) After Raymond has beaten up his mistress, Meursault is even willing to serve as his witness at the police station. During the trial in Part II of the novel, the public prosecution says that Raymond is a procurer and Meursault was his accomplice and friend; his main function in this part of the novel seems to allow the prosecution to present Meursault as a "moral monster". Masson is also called as a witness. However, even during his long time in prison, Meursault never speculates about an affair between Raymond and Masson's wife. Based on this, I have no reason to assume that such an affair existed.