To say that Lem applied game theory to the writings of the Marquis de Sade is a little bit of an overstatement. In the essay Markiz w grafie (“The Marquis in the Network”) he first argues how game theory can be used in general to understand the world, ranging from industrialisation, philosophy, theology, and religion. He then moves on to literary analysis of works of fantasy, and divides them into four categories: utopias, fairy tales, dystopias, and “anti-fairy tales” . He initially thought that the final, most unusual category, of “anti-fairy tales” would not have any examples, but on reflection found that de Sade’s works fitted perfectly into this category and added the final two pages to the essay to make this point, almost as an afterthought. In the essay he certainly does not analyse de Sade's works in detail - he actually discusses them more thoroughly in the interview quoted in Rand al'Thor's answer. Indeed Lem writes:
My argument would not have the value it proposes - to show a blank
space on the table of fantasy types, coinciding with de Sade's
writings - if that was my intention in writing… Although I know de
Sade's work, I did not think about it when I started to classify the
elements of fantasy under the aegis of game theory… and it was only at
the very end of the analysis that I was surprised by the analogy of
"anti-fairy-tale" and de Sade's writings. However, I put aside the
unfinished text and only now added the last two pages to close it. Of
course, I could no longer return to the original ignorance of said
similarity. Nevertheless, the thing seems worth publishing.
If we look in detail into the essay, it begins by introducing Lem’s concept of game theory:
While living, man constantly makes decisions both in thought and in
deed. These decisions are never supported by complete knowledge. The
person has to decide on the basis of incomplete information whether to
take risks. This is a typical game situation. Upon entering the world,
a person is thrown into a game whose rules are unknown to him. But
also at the lowest levels of development, life is tangled in a
conflict situation, and thus in a game where the victory is the
deferment of death. Therefore, all the phenomena of life, from the
simplest to human, can be investigated with decision theory,
especially in its section on conflict situations, namely in game
theory.
Lem goes on to to consider various kinds of works of fantasy by considering them as games played between the protagonist and the rest of the world. The first step is to decide whether the world is favourably or unfavourably disposed towards the protagonist (“The overriding question of the ontology seen through the glasses of game theory is the attitude of the inhuman partner”). This bifurcation is the first step in the analysis shown in the graph below:

On the positive branch, the world can either be favourably disposed to all the inhabitants, making a utopia, or to just a few inhabitants, which produces a fairy tale. On the other side of the divide, where the world is negatively biased against the inhabitants we have the same argument:
the world acts negatively against individuals, or for entire
communities. The second option is the world of dystopia. The first has
not filled by collective works: in folklore there is no such thing as
an "anti-fairy tale"
Lem finds utopias boring, because everyone is equally blessed and there is no dynamic:
Utopia is that everything can be unsurpassedly good at the same time,
that there are no values to be given up in favor of the values of
others. Being such an ideal of collective life, utopias must be
unchanging. Therefore, it is not as compelling as a fairy tale: an
ideal is not a wonderful adventure.
In a fairy tale, though, only the hero is the recipient of good fortune. In his language of game theory:
Whatever such a hero does, he will kill the dragon, marry a princess,
he will become king. This is because when you look at the whole thing,
there is no losing strategy in the fairy tale. This is understandable
as an attempt at the efficiency of this world… At the same time,
according to the principle of symmetry in the game structures, there
is no winning strategy for negative characters in a fairy tale.
…
In the light of game theory, a fairy tale is a zero-sum game, because
the hero's win equals the antagonists' defeat. What the bad guys lose,
the good win. It is probably not an arithmetic sum. It is hard to say
whether the relationship with the hideous dwarf would be as unpleasant
for a princess as it is delightful to marry a beautiful knight, but
since it is commonly believed that this is the case, the equation
holds true.
Having described the first two branches of his diagram, Lem briefly considers worlds where everybody is equally oppressed, dystopias, and then goes onto the final category in his classification, “anti-fairy tales”, and notes that there appear to be no examples of this genre:
one is struck by the absence of anti-fairy tales - in folklore there
has never been such a genre. Nevertheless, we can describe its
features precisely. The world of anti-fairy tales should be biased
against an individual, and the sum of good and bad is also constant,
as in a fairy tale, but with a reverse distribution: good will be
punished, and evil rewarded. There are no winning strategies for the
heroes, but all the strategies of the rogues are optimal. The payout
function is a reverse ethics function. The sum of the game should also
be zero, since good losing equals evil winning. The questions are,
what is the price of the game, what should be the strategies and what
distribution reveals the payout function.
...
The pleasure in fairy tales is only felt at the end, because it is not
pleasant to sit in a witch's cage, in the belly of a wolf, or to fight
a terrible monster. It is done either out of need or out of
self-denial. The noble and the weak succumbs to evil temporarily, and
the strong hero rushes to their aid for higher reasons… A fairy tale
postpones the reward for kindness; the anti-fairy tale temporarily
rewards evil in the course of inflicting it, because the suffering of
others is the protagonist's joy. This changes the course of the game.
Usually, fairy tales start with an evil attack that ruins the state of
the opening, and the game is about fixing it. Thus, the trajectories
of the game must be different: in a fairy tale there is little evil
first, then a lot, and finally there is none at all. In the anti-fairy
tale, however, there is a constant surge of evil.
Lem goes on to contrast the structures of fairy tales and these anti-fairy tales, and concludes that an anti-fairy tale is intrinsically contradictory, and should not exist because of its lack of internal logic:
While in a fairy tale the game is about happiness deferred to the end
game and there is a payoff, in an anti-fairy tale the happiness must
be the misfortune of others, and thus happiness ceases when that
disappears. Thus, an inalienable contradiction appears in the
structures of the game. The virgins prey on the dragon, but when they
eat all of them, they will starve to death.
A further drawback is how the world of the anti-fairy tale must have an “arbitrary unreality”. While fairy tales naturally contain fantastic elements, we can willingly suspend our disbelief in a way that Lem thinks we would not do for anti-fairy tales:
anti-fairy tales cannot retain even a trace of conventional naivety
and innocence, which soothes the often present manifestations of
cruelty in fairy tales (e.g. in the fairy tales of the Brothers
Grimm). Here is another paradox that we encounter: the atmosphere of
anti-fairy tales will not be fairy-tale-like. And even what is
emphatically fantastic will be perceived as a bloody dream or a
nightmare.
The second consequence is that the anti-fairy tale will teeter on the edge of self-parody, with the antihero furiously indulging in evil.
Having built up this argument, Lem then makes an almost parenthetic remark:
It is puzzling that our - after all logical - reconstruction of the
genre, which has never been created, corresponds to the work of the
Marquis de Sade.
but does not give any further discussion of de Sade's works.