I'm a mathematician, and to me the style of narration in this passage sounds very familiar. Because of the extreme abstraction from reality of most of (pure) mathematics, we who study it often dread being asked what we do by non-mathematicians. The difficulty of trying to explain our work to laypeople in any level of detail is a common joke among mathematicians.
This varies between different fields of maths, as some can be more directly related to relatively concrete objects, such as numbers or geometric figures, than others. Those who study number theory or geometry have an easier time explaining at least the basics of their field: it's easy to show people patterns among whole numbers, or to draw visual diagrams. Perhaps the hardest branch of maths to explain to the layman, due to its abstraction from anything remotely visualisable, is algebra. (Even category theory, for mathematicians the most abstract field due to its extreme generality, can be more easily explained, either in terms of points and arrows and commutative diagrams or just by saying "it's what everything else in maths is a special case of".)
Just as an example to illustrate the difficulty and abstraction of algebra, here's the definition of a
scheme, one of the most basic concepts in modern algebraic geometry:
An affine scheme is a locally ringed space isomorphic to the spectrum of a commutative ring. A scheme is a locally ringed space X admitting a covering by open sets U_i, such that the restriction of the structure sheaf O_X to each U_i is an affine scheme.
In order to understand this definition, you need to know: what a locally ringed space is and what it means for two such to be isomorphic; what a commutative ring is; what it means to take its spectrum; and what open sets mean in topology. In order to understand the definition of a locally ringed space, you need to know: what a topological space is (this also explains the meaning of "open sets"); what a sheaf is; what its stalks are; and what a local ring is. In order to understand what a spectrum and a local ring are, you need to know what an ideal of a ring is. Rings and topological spaces are among the most basic objects of algebra and topology respectively, although there are entire textbooks and undergraduate courses about these objects and the things associated with them such as ideals and open sets. Sheaves and their stalks are both very tricky concepts to understand even for high-level maths graduates, and realistically you need to have taken advanced courses in differential geometry, category theory, and analysis in order to fully grasp these definitions.
So just for this one definition, of one of the foundational objects used in algebraic geometry (which despite its name is mostly pure algebra), you need to have a deep understanding of abstract algebra, topology, differential geometry, category theory, and analysis. And an introduction to schemes is only part 2, the first part after "Preliminaries", of the standard treatise on algebraic geometry (warning: 800-page PDF link), which some students are required to know in full before starting a doctoral degree in algebraic geometry.
Are you scared yet?
Coming back to the point ...
... the passage you quoted sounds very much like how real mathematicians sound when trying to explain their work to laypeople. Unfortunately, because it's written for a lay audience, it's not enough for an expert to understand exactly what mathematical objects are being referred to here. The language of mathematics, above all, is precise, and most or all of that precision is lost when it's translated into an ordinary language such as English or Polish. If the imaginary narrator were speaking to a colleague instead of to a lay audience, he would be using strange letters and symbols instead of words that anyone can understand, and it would be easier for me to tell what actual mathematics is behind it. As it is, all I can do is make an educated guess at which mathematical objects are being 'approximated' by his rough description.
It was from this position that I proceeded to the translation of the problem into the language of mathematics. What I did I cannot present plainly, since our everyday language lacks the required concepts and words.
This is essentially the same point that I've just been making. He's not explaining any of this properly, because he can't do that without resorting to symbols and formulae.
I can only say, in general, that I studied the purely formal properties of the "letter"--treating it as an object mathematically interpreted
This is obvious. If you're going to analyse a real-world problem mathematically, you need to turn it into an ideal mathematical model, and consider only the formal properties of the objects involved rather than their actuality.
It may interest you to know that letters and words can be turned into ideal mathematical objects: in algebra, a letter is simply a symbol (with no particular meaning - it doesn't matter what symbol is used) and a word is a sequence of such symbols. The study of these 'formal' letters and words leads to interesting results in group theory.
--for features that are of central interest in topological algebra and the algebra of groups.
Topological algebra is a real thing: it's the study of mathematical objects which have both topological and algebraic properties that mesh nicely with each other, such as topological groups, topological rings, and topological algebras. (Yes, a topological algebra is an object studied in the field of topological algebra. Deal with it.)
The algebra of groups is also a real thing - it just means group theory, the study of groups, which is a subfield of algebra.
In doing this, I employed the transformation of transformational sets, which gives the so-called infragroups or Hogarth groups (named after me, since I was the one who discovered them).
As far as I know, there's no such thing as a transformational set or an infragroup. As you note, Hogarth groups are clearly fictional, being named after a fictional character.
If I obtained, as a result, an "open" structure, that would still prove nothing, because it could be that I had simply introduced an error into my work, going on some false assumption (such an assumption might be, e.g. the assertion of the number of code signs in a sing e"unit" of the message). But it happened otherwise. The "letter" closed beautifully for me, like an object separated from the rest of the world, or like a circular process (to be more precise, like the DESCRIPTION, the MODEL of such a thing).
Unfortunately, this is hard to interpret. There are several different properties in various branches of mathematics which could be described roughly as "openness" and "closedness". The analogy to a circular process makes me suspect that "closedness" here describes something more like a closed curve from geometry than a closed set from topology, but most likely it's not exactly either of these.
The concept that a result one way would be inconclusive due to the possibility of a false assumption, but a result the other way is conclusive - again, this has a ring of truth to it. When approximating a real situation by a mathematical model, you always have to make assumptions, and some of those assumptions have the potential to make your model useless. But if you're sensible about what assumptions you make, you'll make sure to do it so that if you get the result you're hoping for, you know it actually works; it's only if you don't get the desired result that you need to go back to the drawing board, and maybe try different assumptions.
I spent three days setting up a program for the computer, and the computer carried out the task on the fourth. The result said that "something, in some way, closes." The "something" was the letter--in the totality of the interrelations of its signs; but as for the "how" of that closing, I could only make certain guesses, because my proof was indirect.
This again is a concept I'm very familiar with: that of an existence proof, one which is not constructive. In maths, you can often prove the existence of an object satisfying certain properties, even if you can't actually find an explicit example. For instance, any result which uses the Axiom of Choice is likely to involve knowing something exists without being able to find it.
Clearly, the fact that the letter is "closed" (whatever that means - he hasn't given us a rigorous mathematical definition or anything approaching one) is somehow significant. It proves the existence of something, even without giving a way to construct it.
The proof showed that the "described object" was NOT "topologically open."
OK, interesting. It seems like my guess before was wrong and we're talking about openness and closedness in the topological sense rather than in the sense of curves. Evidently, our narrator has used a topological model for the letter, and he's just proved that some ideal mathematical object used in this model is not open.
Important note! In topology, closed is NOT the opposite of open. Some sets can be both closed and open; most sets in most topologies are neither closed nor open. It's probable, then, that when the narrator says "open" he means open in the rigorous topological sense, but when he says "closed" he means "not open", rather than actually topologically closed. Either that, or there's some property of the topology he's using, or of the specific set within his topology, which implies that it must be either open or closed. Or, of course, it's simply a mistake and Lem didn't know that open and closed aren't opposites in topology.
But to reveal the "means of closure" with the aid of current mathematical methods was impossible for me; such a task was several orders of difficulty greater than the one I had managed to surmount. The proof, then, was very general--one could even say vague.
This is just repeating what we already know - that his proof is non-constructive. He knows the set is "closed" (in whatever sense he's using that word), but he can't prove how it's closed, only that it's closed.
One the other hand, not every text would have displayed this property. The score of a symphony, for example, or a linear coding of a television image, or an ordinary linguistic text (a story, a philosophical treatise) does not close in that fashion.
This is interesting. It tells us that the mathematical property exhibited by the letter - the property described as "closedness" - is something that many texts don't have. But that much is obvious: if every possible text was "closed", there would be no need to prove that the letter is, and doing so wouldn't be a significant result. The curious thing is that texts with meaning, such as stories or symphonies, don't exhibit the same property as the letter. Thus, the mathematical analysis of the 'letter' may suggest that in its fundamental nature, it's not really akin to a message written by an intelligent entity.
But the description of a geometric solid closes, as does that of something as complex as a genotype or a living organism. The genotype, true, closes differently from the solid.
Putting this together with the preceding sentences, it looks like the fundamental mathematical nature of the "letter" is more like an object of natural symmetry, as found in geometry and biology, than an intelligently designed and meaningful message.
This kind of discovery can be very significant in science: the revelation that something once thought to be close in nature to X actually behaves more like Y. Such breakthroughs have caused the remodelling of particle physics, the reorganisation of biological taxonomy, and new connections between parts of maths once thought to be very different. The discovery of an identical pattern cropping up in two apparently unrelated branches of mathematics was what ultimately led to not only the proof of Fermat's last theorem, one of the most famous mathematical problems in the wider world, but also, as a step along the way, and more importantly within mathematics, the proof of the Taniyama-Shimura conjecture which links those two branches together.
But by going into such distinctions and details I fear that I will be confusing the reader rather than explaining to him what I did with the "letter."
This is Lem's way of breaking off the painful "maths for a lay audience" passage and going on to something different.
Conclusions
After performing a close reading of the passage quoted in your question, I've come to the following conclusions regarding the narrator's work on the "letter":
- he was working at the intersection of two fields of maths, topology and algebra;
- he used a non-constructive proof to establish a new property of the mathematical object used to model the message;
- this proof shows that, in some sense, the nature of the message is more like that of a naturally symmetric object than that of a meaningful communication;
- the vagueness of the description (necessary due to its audience) makes it impossible to confirm that this is real maths rather than gibberish, but it's certain that some of the mathematical objects used are fictional.
Postscript: on Lem and mathematics
Lem wasn't formally qualified in maths, but nonetheless he was quite knowledgeable about it.
In fact, he was knowledgeable enough to be able to write about fictional mathematical events which were so plausible that they actually came true later!
