06-Cave-Ch06

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 6 The Gnostic Machine Artificial Intelligence in Stanisław Lem’s Summa Technologiae Bogna Konior Nothing which implies contradiction falls under the omnipotence of God. St. Thomas Aquinas, Summa Theologica, 1273 Loyal engineers must evaluate what biology has accomplished, and not reduce their judgment to mocking its Creator, who aside from life gave us also death, and gifted us more sufferings than pleasures. Their judgment should illuminate him as he was. And he was, above all else, far from omnipotent.1 Lem, Summa Technologiae, 1964 6.1 Introduction: evolving technology Stanisław Lem’s first-contact novel Solaris (1961) is the Polish writer’s most celebrated and enduring legacy. It features an ocean planet that conjures up life with no apparent purpose, at best perversely reflecting human wounds and desires. Unintelligible, alien, and therefore not easily communicable, the ocean in Solaris could be an archetype of inhuman intelligence: indifferent, foreign, and creative in its distorted reflection of humanity. A quick Google search reveals that multiple artificial intelligence (AI) companies christened their products after Lem’s novel. Solaris could be a prototype for computational cognition design that is not human-centred: ‘that inscrutable alien might be wise or cruel or unconcerned’, not an all-seeing God but rather ‘a nebular and numinous totality . . . [a] big alien brain’ (Bratton, 2020, p. 97). Summa Technologiae (1964a), Lem’s abstruse, 600-page-long theoretical work is less known, though it contains the rationale for his whole oeuvre. Titled after Summa Theologica (written 1265–1273, published 1485) by St. Thomas Aquinas,

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 90 Europe Lem’s philosophical tour de force might make one reread Solaris, and its descriptions of inhuman intelligence, as a theological—at least, metaphysical— problem. Indeed, many of Lem’s novels trace the limits of knowledge in an indifferent cosmos. Overviewing the most cutting-edge scientific discourses of the times, Summa is a work of futurology composed of eight chapters, each of which reframes theological questions as technological problems. The first chapter presents Lem’s views on the purpose of futurology, and subsequent chapters discuss similarities between technological and biological evolution; SETI; the possibility of computational intelligence; whether humans could be technologically omnipotent; whether humans could create artificial worlds and virtual cosmoses enclosed in themselves; machine knowledge; and whether humans could, akin to gods, engineer new lifeforms. Though this chapter cannot investigate the matter in exhaustive detail, I explore how—given Lem’s skilful portrayal of fictional alien intelligences—the idea of AI appears in Summa. Lem is one of only a few writers from beyond the anglophone scene who made it into the popular global canon of science fiction. Recent monographs on his work in English (Gajewska, 2021), new widely reviewed English translations distributed by MIT Press and published together with essays by media scholars such as Katherine Hayles, and multiple events held globally to celebrate the hundredth anniversary of his birthday, prove that there is a rise of interest in his work. This happens alongside a general rise of interest in technological narratives beyond western Europe and the US. Questions have appeared, for example, about what ‘non-western AI’ might be, but when ‘non-western perspectives’ are brought up, they can sometimes be essentialized as ‘better’ or even inherently more moral for AI design—they are argued to be more ‘pluralistic’, ‘metaphysically inclusive’, ‘relational’, or ‘ethical’ than the ‘western scientific ones’ (Goffi, 2021; Hui and Amaro, 2020; Vickers and Allado-McDowell, 2020). Various local belief systems can appear as exoticized ‘others’ to be deployed in the ethical redesigning of ‘Western AI’ into a seemingly more relational, harmonious, and pacified technology (Williams and Shipley, 2021). Given its ambivalent position with regards to the ‘West’— whether Poland is ‘east’ or ‘west’ has been hotly debated for at least a couple centuries (Czapliński, 2016)—we find no proximate salvation discourse projected onto Eastern Europe. It is not perceived as foreign ‘enough’ to serve as a useful foil against which mainstream American technology discourse can evaluate its ‘ethical failings’. In fact, Summa sets out not to take an ethical perspective and instead takes a disinterested, detached look at technology’s role in human evolution. Rather than taking an ‘AI ethics’ or ‘area studies’ approach to Lem’s work, I prefer to see him as a formidable technology theorist whose work is in line with contemporary media theory and can enrich its canon. Though some attention is devoted to Central and Eastern European media and technology studies,

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 91 it is mostly within the areas of news media, communications, and political economy (Demeter, 2020; Downey and Mihelj, 2012). We are yet to see serious attempts at media theory—or philosophy—in the tradition of Haraway (1991), Hayles (1999) or McLuhan (1964) to come from this region; that is, an inquiry into how technological mediation revises certain fundamental axioms of humanism. Lem’s interest lies precisely in that, to the extent that he even discards human-centred ethics as a relevant perspective. Summa cannot be used to redeem perceived sins of ‘Western AI’, but it does contribute hypotheses on the very nature of technology and how it mediates human knowledge and evolution. Before diving deeper into Summa, I first sketch the context: Lem’s novels, his life under the Soviet regime, his Catholic intellectual scene in Kraków, and the debates around cybernetics in the USSR. In contrast to Promethean ideals of both American and Soviet science fiction of the time, Lem was interested in the Great Silence, in miscommunication and failure. He remained devoted to science and engineering as existential tasks, not as an idealist but as a metaphysician, and with an understanding that humans operate from a position of partial knowledge. This he believed true not only because humans are constrained by ideologies—such as Soviet communism and its restrictions on science and arts—but also ontologically, by the inherent incommunicability of the cosmos. Understanding the debates that were happening in Lem’s intellectual circle—his proximity to the Catholic Church in Poland on the one hand and the Polish Cybernetic Society on the other—helps frame his predictions in Summa about AI as a ‘gnostic machine’, whose function is both epistemological and evolutionary: the production of new knowledge at the limit of human comprehension as a factor in the civilizational trajectory of our species. 6.2 Lem’s intellectual circle: science, literature, theology and cybernetics Summa was not planned as a 600-page monograph. Though several of Lem’s essays previously published in Przegląd Kulturalny (Cultural Overview), liquidated in 1963, constitute the core of Summa, the finished book is a long treatise on the evolution of human science and technology, more resembling a dissertation than a collection of articles. ‘Let’s consider the future of civilization from the perspective of potential scientific developments,’ Lem proclaims (1964a, p. 125). While one might expect that the philosophical magnum opus of a science fiction writer would be devoted to writing or world-building, Summa is rather a work of popular science. Biophysicist (and Lem fan) Peter Butko (2006) considers it equal (or even superior) to formidable nonfiction books at the overlap of science and philosophy, such as Richard Dawkins’s The Selfish Gene (1976) or

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 92 Europe Douglas Hofstadter’s Gödel, Escher, Bach (1979), and notes that it was Summa that first comprehensively addressed some of the issues that made these subsequent books so influential and widely celebrated. Indeed, though he was a voracious reader of his fellow novelists, Lem held scientific papers in higher regard than science fiction as a genre—it is no surprise that he desired to be seen as a theoretician of science rather than only of science fiction. He lamented that science fiction rarely transcended its designated category of genre novels, which he considered ‘trash’, due to dynamics in the publishing industry, consumerist preferences of the average science fiction fan, and also a lack of ambition, talent, and education among its writers (1984b, ‘Science-Fiction: A Hopeless Case—With Exceptions’). His interest in technological trajectories and the future motivated him to study paradigm shifts in science. In consequence, his nonfiction essays, most of which have not yet been translated into English, were like guidebooks for Polish readers to scientific debates of the respective decades, and he was frequently commissioned by the Polish press to comment on the latest developments. After his retirement from fiction in the late 1980s, Lem penned hundreds of such essays, seemingly on every possible subject, from 9/11 to social psychology, from politics to literature and religion, and possible development routes for artificial life and AI (Bomba Megabitowa, 1999; Rasa Drapieżców, 2006; Tajemnica Chińskiego Pokoju, 2003). Peter Swirski, the most prolific English-language scholar of Lem’s work, describes Summa as ‘Lem’s philosophical and futurological opus magnum which contains his most wide-ranging reflections on the nature and regularities of technological evolution, including its pervasive effects on human culture and society’, and, just like Lem’s novel Niezwyciężony (The Invincible) (1964b), published in the same year as Summa, ‘highlight[s] the similarities between biological evolution and the evolution of culture—not least the contingency of their development’ (Swirski, 2015, p. 137). In his usual grumpy assessment of his own work, Lem wrote in a 1964 letter to his friend, Polish scriptwriter Aleksander Ścibor-Rylski: My Summa technologiae has just been published . . . It’s such a cybernetic, theoretical and pretentious [book] so if you have no desire to bite your way through this dreadful text, and only intend to keep this brick on your bookshelf as a memorabilium, [please let me know now] because I only have 3000 author copies to give away, and only 260 copies have been sent to bookstores in Kraków. (Orliński, 2017, p. 395) ‘This dreadful text!’—so Lem describes what has been called his crowning achievement, a work that has been celebrated for predicting multiple

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 93 developments in contemporary science and technology, from virtual reality, which he in Summa calls ‘phantomatics’ ( fantomatyka), to developments in artificial life and AI, which we find under the speculative labels ‘imitology’ (imitologia) and ‘intellectronics’ (intelektronika). Lem rather treated these then-imaginary technologies as thought experiments—only sixty years later can we assess how accurate he indeed had been. As Anton Pervushin explores in Chapter 7 in this volume, the question of bio-technological evolution was influential in Soviet science fiction in general, such as in the stories of Anatoly Dneprov, which concerned machines that could evolve. Yet, rather than simply imagining humanoid robots as an additional link in the evolutionary chain, Summa considers the broader role that technology as an activity plays in the evolution of human civilizations: how it produces or staves off existential risk for our species. Published in 1964, it is unmistakably influenced by the debates around cybernetics, evolution, and space travel—all hot topics at the time. In fact, young Lem first learned English when his mentor, psychologist Mieczysław Choynowski, assigned him to read Norbert Wiener’s Cybernetics (1948) in the original (Bereś and Lem, 2013, p. 47). Lem’s task was to review books sent to Choynowski by American and Canadian universities for the Polish academic publication Life of Science (Życie Nauki, 1946–1953). The young aspiring writer spent many nights in a small annex, ‘a tad bigger than a wardrobe, but much smaller than a room’, with the English-Polish dictionary in hand, making his way through Cybernetics or Claude E. Shannon’s (1948) A Mathematical Theory of Communication (Orliński, 2017, pp. 175–6). In 1952 Lem began writing Dialogi (Dialogs), a precursor to Summa. Dialogi (1957) are part-philosophy, part-fantasy, and contain fictional conversations between Filonous and Hylas, characters named after Three Dialogues between Hylas and Philonous (1713) by Bishop George Berkeley. This eccentric collection fictionalized many of the conversations Lem had had about cybernetics at Choynowski’s conservatory, commenting especially excitedly on how versatile a science it promised to be, with implications for warfare, governance, arts, and literary theory. This obscure collection of strange, fantastic conversations between imaginary sages drew little attention, not only because it was not as readable as Lem’s subsequent science fiction novels, but also because of political conditions at the time. The first readers of Summa were largely the intellectual and technocratic elites of the Communist bloc in the 1960s (Butko, 2006, p. 83). Lem, a lifelong member of the Polish Cybernetic Society, established in 1962, played a part in rehabilitating the framework. In the 1950s, under Stalin, cybernetics was considered a forbidden, bourgeois, imperialist pseudoscience. It was not until the reforms of Nikita Khrushchev that Soviet scientists embraced the potential of this new science, albeit seeking to integrate it completely within

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 94 Europe the teleological goals of communist governance (Gerovitch, 2008). In Lem’s early novel The Magellanic Cloud (Obłok Magellana) (1955), wary of using the term ‘cybernetics’, he instead labelled a similar fictional science ‘mechaneuristics’ (Bereś, 2013, p. 71). Even so, the manuscript was held by the censors for a year (Orliński, 2017, p. 260). Competing with evolutionary science, communism found its own idea of civilizational trajectories in a Hegelian reading of Marx, where all social behaviours must be directed towards the fulfilment of communism (Sayers, 2019). Within this tautology, all of science and culture should serve the purpose of social betterment and the only permissible definition of social betterment is communism. Scientific and cultural inquiry that was not aligned with this ‘noble’ goal had to be suppressed under the aegis of social responsibility. Soviet censors were not unjustified in their worries—already in Dialogi, Lem proposes that cybernetics disproves totalitarianism as a governance mechanism. In theories of cybernetics, feedback loops are a key feature of all functional systems, whether countries or weapons. Feedback informs the ‘steersman’ whether the system is working, so that appropriate corrections can be made. Yet in totalitarian systems, as the characters in Dialogi discuss, because of censorship and propaganda, the ‘managers’ do not receive correct feedback—they only receive positive reinforcements. As Filonous says to Hylas, such a machine will collapse, like a steam engine without a safety valve. Even so, though when it came out in 1964, no one quite knew what to make of Summa—neither science fiction nor sustained scientific debate was part of the Polish literary scene—with time it became ‘a textbook for a generation of Soviet scientists and engineers obstructed by the Iron Curtain from contacts with world science’ (Swirski, 2015, p. 34). When the idea for Summa began crystallizing in the early 1960s, Lem envisioned it as a continuation of Dialogi—another attempt at popularizing cybernetics. The root of the word cybernetics is the Latinized form of Greek kybernan—to steer or pilot a ship. In a nod to cybernetics, Summa looks for ways to ‘steer’ the evolutionary structures of human civilization, however imperfect they may be. Yet, in his 1962 letters, Lem already realized that the book would become more than that—no longer a treaty on cybernetics, but ‘a pilgrimage to the border of human comprehension’, a futurology that aims to describe the civilizational trajectory of humanity in a feedback loop with scientific and technological discoveries (Orliński, 2017, p. 448). It is no accident that the characters in Dialogi are named after a canonical work by a Christian philosopher—the question of technology was for Lem from the very beginning a question of metaphysical proportion. Cybernetics appeared to Lem as a theory of totality, as skilled at describing the world—and engineering it!—as theology promised (but failed) to be. Lem combined his work for Choynowski—who became a model for numerous ‘mad scientist’ figures in Lem’s later novels—with literary and

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 95 philosophical ambitions. His mentor in these endeavours was Jerzy Turowicz, the legendary journalist and editor of Tygodnik Powszechny, a Polish Roman Catholic weekly magazine of social and cultural issues, around which many intellectuals and artists gathered throughout the Soviet times and until today. As Wojciech Orliński writes in his biography of Lem: Why was Lem so different from other science-fiction writers internationally, and from Polish writers en masse? Because he was intellectually shaped by different readings. He read what was recommended by Turowicz and Choynowski, and so he understood international science better than the average Polish writer, and he understood the humanities better than the average science-fiction writer. (Orliński, 2017, 178) Born to a Jewish family and brought up as a Catholic, Lem later identified as an agnostic. Summa explicitly discounts all religious traditions as potential bases for futurology. And yet, Tygodnik Powszechny, published by the Kraków Curia, its associated social scene, as well as the significant role that the Polish Catholic Church played in anti-communist resistance and the cultivation of arts and philosophy, all mean that we cannot easily discount the influence of religious themes on Lem’s work. It was in Kraków, in its specific intellectual environment, that his work was supported and read by the best Polish intellectuals at the time, many of whom were Catholic and deeply preoccupied with the problems of human dignity and free inquiry under communism: Tygodnik Powszechny was an anomaly, the only such publication to the east of the Iron Curtain . . . while it did not take an anti-communist position—the censors would not allow that—it did take a non-communist one. This alone constituted a challenge to the very essence of a totalitarian system, which by definition sought to subordinate everything to one ideology. (Orliński, 2017, p. 179) Given this proximity to Catholic intellectuals, it is not surprising that Lem titled his defining work after St. Thomas Aquinas’s great theological treaty Summa Theologica. Akin to a work of theology, Summa Technologiae aspires to grasp totality and investigate the relationship between technological practice and human destiny. Where its inspiration—Summa Theologica—sought to describe the relationship between Creator (God) and His creation (humans), Summa Technologiae wants to reveal the evolutionary processes that dictate the development of human science and society and ask whether creation (humans) can steer its Creator (Nature, to use Lem’s term). If there appears a deity in Summa, it engineers the world in obscure ways and wields alien tools, akin to what Richard Dawkins famously called ‘the blind watchmaker’ in his 1986 book of

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 96 Europe that title about evolution. In another essay, Lem remarks that the genetic code is inhuman in construction, assembling itself without a conscious engineer; it is not simply ‘devilishly complicated’ but rather ‘burdened by unnecessary complexity’ (Lem, 2016, p. 361). This theme of indecipherable logic, common in Catholic theology, returns also in Lem’s novels. In His Master’s Voice (Głos Pana) (1968) scientists struggle to decode an extra-terrestrial message and are unable to determine its meaning, or even whether it is a message at all. In Fiasco (1987), narrated by an untrustworthy hero, human–alien communication also fails, as it does in Solaris. If there is no intelligible divinity, and reason cannot be found with the ‘blind watchmaker’, where could intelligent design be exercised? Such is the guiding question of Summa: are humans capable of being the drivers of their own evolution? Could they rise up to this challenge through scientific inquiry and engineering? 6.3 Political contingency: science fiction in Poland Lem’s immediate intellectual community provides context for Summa and so does his life as a Polish citizen under Soviet rule. It is also worth looking into the larger circumstances of Poland and how—alongside his intellectual influences—Lem’s political life shaped his interests as a novelist and intellectual. If not for his experience of war, and later the Soviet regime, Summa might have been a much more Promethean work. As it stands, Lem notes that human agency is faulty and complex—we cannot say ‘let there be light’ and expect there to exist a strict causality between our initial design and what we actually produce (1964a, p. 9). Omnipotent, divine agency is as incorrect a model for actual technological design as are utopian ideals of the Soviet regime. Lem hints at this throughout his work. Commenting on the relationship between science fiction and politics, Ivan Csicsery-Ronay writes, ‘the nations that have produced most of the science fiction in the past century and a half . . . are precisely those that have attempted in modern times to expand beyond their national borders’, including Britain, France, Germany, the USSR, Japan, and the United States (2006, p. 130). Lem’s works, like the novels of Karel Čapek in neighbouring Czechoslovakia, are notable exceptions. Poland’s national identity has little to do with the idea of techno-scientific expansion and, though once a country with some amount of geopolitical power, over the last two centuries, Poland had frequently found itself in the midst of imperial conflicts, struggling for sovereignty and undergoing modernization while under occupation by foreign powers. As the partitions of Poland (1772–1918) came to a halt with the end of the First World War, Poland regained its sovereignty for a mere twenty years before

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 97 the co-ordinated invasion by the Nazi and Soviet Armies in 1939 commenced another occupation that lasted until the fall of the Soviet Union in 1989. Under the Soviets and as is often true of regions under colonial or imperial occupation, seized territories were like free zones of technological and social experimentation, laboratories where modernization happened with dazzling speed, an ‘imperial future shock . . . at a speed that made resistance futile’ (Csicsery-Ronay, 2006, p. 133). The Poles nevertheless ‘practiced resistance simultaneously through the refusal of modernization (for example, in agriculture) and through mathematics and theoretical science’ (Csicsery-Ronay, 2006, p. 146). As mentioned in Section 6.2, Lem was interested in how scientific developments, such as cybernetics, could prove the dysfunctionality of the political system under which he had to live. This often plays out through allegory or allusion in his novels—naturally, because of censorship—but is rarely the main theme. One of the notable exceptions is Memoirs Found in a Bathtub (1961), a Kafkaesque pastiche on totalitarianism, which could only be published if, as suggested by the censors, it were set in the Pentagon rather than the USSR, to suggest that it was capitalism, rather than communism, that was totalitarian (Bereś, 2013, p. 72). We can suspect that Lem’s characters in Solaris carry American names because the writer would not want to suggest that Soviet astronauts could fail their missions. Later in life, when it was no longer dangerous or detrimental to his career, Lem revealed his difficulties in navigating intellectual and creative interests under the communist regime. He also described his distaste for the social realist stories he wrote as a young man—due to financial needs but also misguided youthful idealism—as well as his fear for his family and colleagues later when the regime frequently imprisoned misbehaving intellectuals and artists (Bereś, 2013). However, he remained on good relations with many Soviet scientists and writers—an endearing gesture of intellectual defiance. For example, Lem first discussed his novel Golem XIV (1978) in a dissident venue in Moscow called the Institute of High Temperatures, where the scientific elite of the USSR gathered. He chatted about it to, among others, Iosif Shklovsky, who worked with Carl Sagan on extra-terrestrial communication (Fiałkowski, 2015, p. 91). Though with time, the principles of Marxism–Leninism were modified to allow for the idea of scientific–technological revolution, especially to justify the USSR’s space programme, Lem’s characters rarely have any real power or fit with the Promethean agenda. The Lem hero is not a Promethean revolutionary taming the cosmos with technology, but rather a defiant metaphysician-scientist operating within a negative theology, perpetually aware of how incomplete knowledge is. Additionally, it wouldn’t be an overinterpretation to say that Lem’s recurring theme of technological systems evolving to full autonomy and bypassing humans entirely—as in his novel

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 98 Europe Golem XIV (1978)—originates from his position as a Polish citizen caught in the midst of the American–Soviet technological race, which happened over Poland rather than in Poland. Lem’s characters frequently participate in momentous events that remain beyond their understanding and control. Describing how dissociation from historical events is common to the Polish modern psyche in general, Andrzej Leder writes in his book A Dreamt-through Revolution: Exercises in Historical Logic (2014) that twentieth-century Polish history is ‘transpassive’. Poles experience modern history, from Nazism to communism, as carried out by the hands of strangers and occupants, in a dreamlike state, where historical changes are happening through them, but are not done by them. We find a similar logic in Lem’s narratives, where characters have to continually find their bearing in conditions not of their own making, where miscommunications and lack of agency abound. This did not fit so well with what the Communist Party wanted to promote in Soviet science fiction at the time: a futuristic ‘ethical imperialism based on a romantic, rhetorical, nineteenth-century vision of humanistic values in full control of science and technology’ (Csicsery-Ronay, 2006, p. 142). Where Soviet science fiction was rooted in a utopian narrative of socialist values in control of science and nature, in Lem’s oeuvre as a whole, save for his earliest work, attempts to perfect society produced unexpected disasters. Though Lem’s early novels, such as The Astronauts (1951) and The Magellan Cloud (1955), both of which he resented later in life, aligned with the Promethean imperative of the Soviets, ‘manifold heresies soon entered Lem’s evolutionary narratives through the back door’ (Jarzębski, 2006, p. 106). Contingency and miscommunication rule Lem’s technological cosmology, placing him in the company of twenty-first-century scholars of technology, rather than the ideologues of his time who wanted to use art and science for utilitarian ‘betterment’. As Paul Virilio (1999), for example, noticed decades later, each technology is an invention of a corresponding accident. In a recent book Ironic Technics (2008), Don Ihde comments that, while many have the tendency to describe technologies as utopian or dystopian, technological development is mostly dominated by the unexpected. Lem knew this not only in theory, but also through his life experience: There is no doubt about the harrowing effect the war years had on the shaping of Lem’s intellectual and emotional outlook on life. The mature writer’s almost obsessive return to the subjects of chance, luck, survival, aggression, the military mindset, and the inhumanity latent in humanity is clearly rooted in the young man’s experiences in the war zone [and later under communism—BK]. (Swirski, 2006, p. 6)

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 99 Indeed, later reflecting on his work in The New Yorker, Lem admits that having to assume fake identities and living under a constant threat of ideological impurity, as well as near-brushes with death, made the idea of full control over history (or technology) sound like ‘deluded hubris’ (1984a, p. 88). Living under communism also instilled in him a lifelong mistrust of ‘ideological artifices’, throwing light on his subsequent distaste for French and American postmodernism, especially for Jacques Derrida (Lem, 1996). In Summa, he describes the fraught evolution of ideologies in evolutionary parallels: biological monstrosities cannot survive, contrary to the monstrosities of linguistic nonsense or intercultural illusion. The punishment for faulty biological construction is disability or death. An analogous metaphysical sin [in the realm of culture] costs no penance, because the human mind is a far more liberal environment for vegetating (or penitential) ideas than the natural environment is for live systems. (Lem, 1964a, p. 446) Lem believed that careful mapping of contingency and limits of human ability can pose a challenge to the political ideologies under which he had to live, characterized by the double problem of utopian hubris and censorship of science and art alike. In addition to Poland’s fraught geopolitical situation under Soviet communism, in no small part he was also informed by his intellectual circle in Kraków, where the Church was the core of intellectual, artistic, and political resistance, and where the Polish Cybernetic Society in time provided his intellectual home. Let us now take a close look at how these interests appear in Summa when it comes to AI specifically. 6.4 AI in Summa: artificial knowledge Subsequently reason itself—which, in this period of Lem’s career, seems to be a kind of self-contained entity initially functioning as a subsidiary agent in the processes of adaptation and struggle for survival—evolves, then acquires some autonomy, and through human agency is transferred into a more convenient environment of machines . . . reason does not necessarily serve living creatures: it simply treats Homo sapiens as selfishly as it does Dawkins’s famous gene, using each human being as a temporary vehicle for its own transformation and not caring for that individual’s fate. (Jarzębski, 2006, p. 110) Lem’s predictions about AI in Summa are framed by similar concerns that guide his novels, where humans are unable to communicate with or understand

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 100 Europe that which is beyond them. Constrained by our perception, not ourselves being able to know, could we engineer a different mode of knowing? Is it possible for something constructed by humans to exceed the knowledge-capacity of its makers? Summa aims to answer these questions on evolutionary terms, considering how humans could ‘evolve’ a different form of knowledge, even despite our own limitations. In his later essays, such as one on the Chinese room experiment (2003), Lem continues to understand computation in terms of evolution. This lens prevails in many of his articles published long after Summa. In ‘Artificial NonIntelligence’, he discusses robotics engineering inspired by insect swarms, citing his own essay from 1984 where he suggested that we should develop ‘artificial instinct’ rather than intelligence (Lem, 2006, p. 377); that is, computer modelling based on the neural anatomy of insect swarms. He also expresses interest in ‘biomorphs’ or ‘live machines’ (Lem, 2006, p. 379) and would be probably thrilled to know that today roboticists model their designs after viruses (Monosson, 2013). In an essay from 1994, ‘Language and Code’, he brings up AI in the context of understanding language as a simultaneously biological and computational form, again discussing it alongside the genetic code (Lem, 2006, pp. 323–42). We could multiply such examples; in the end, Lem left behind hundreds of nonfiction essays. Suffice to say that we already find these preoccupations in Summa. In Summa, Lem describes human evolution as remarkable, but not astounding. What we humans have evolved is the ability to do good and harm to each other in ever-more technologically sophisticated ways (Lem, 1964a, p. 12). But we do not yet have the ability to ‘light up and dim the stars’ (Lem, 1964a, p. 13). Our interest in existential technologies that could alter the course of our species, rather than just better or worsen relations between humans, has been largely confined to the domain of fiction. Even so, Lem firmly believes that ‘the only solution to [any] technology is another technology’ and ‘promises rooted in socalled humanism’ have no power to alter the evolution of the Homo sapiens, even though they exert significant power over interhuman relations (1964a, p. 12). Accordingly, he was interested in the existential potential of AI—could it help us avoid scientific stagnation, for example?—rather than its social dimension. Though in Summa Lem does not use the term ‘artificial intelligence’, chapter 4—‘Intellectronics’ (‘Intelektronika’)—is often considered the most relevant section, for it is here that Lem cites Alan Turing’s essay ‘Computer Machinery and Intelligence’ (1950), W. Ross Ashby’s (1956) work on intelligence amplifiers, and Mortimer Taube’s Computers and Common Sense: The Myth of Thinking Machines (1961). Lem sets up this chapter by describing a hypothetical existential risk scenario, which he labels ‘the megabit bomb’, when science, or human comprehension in general, is not anymore able to keep up with the

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 101 volume of information it receives (Lem, 1964a, p. 135). Human civilization, in consequence, plateaus and then declines. He envisions three outcomes: first, the creation of ‘synthetic researchers’ that could augment the work of human scientists, second, the creation of a ‘cybernetic-socio-technical shell’ where civilization could close itself off from ‘Nature’ and control the volume of incoming information, and third, defeat, which culminates in creating super-specialized microworlds where each science can only apprehend itself: a patchwork of scientific filter bubbles (Lem, 1964a, pp. 138–41). As a futurology, Summa indeed feels quite contemporary. Lem anticipates John Searle’s Chinese room experiment when he proposes that we should complicate Turing’s imitation game by imagining two different language machines. The first one successfully imitates the human brain, can converse with humans about anything, and might be called intelligent. The second is akin to a sophisticated gramophone—it contains ‘let’s say a hundred trillion recorded replies to all possible questions, but it does not understand anything’ (Lem, 1964a, p. 211). (This also somewhat recalls the Shannon–McCarthy objection to Turing from 1956.) To identify such a gramophone, Lem says, tell the machine a joke, ask if it found it funny, and if so, ask it to rephrase it so that it retains its humour, and you shall unmask the imposter soon enough. A mere few pages later, he also in passing describes how AI should rely on what we today call cloud computing, describing a thinking machine built from distributed blocks, corresponding to, let’s say, cerebral ganglia. Now we divide the blocks and place them across the whole planet, one in Moscow, second in Paris, third in Melbourne, fourth in Yokohama. Separated from one another, the blocks are ‘psychically dead’, but connected (through, for example, telephone cables) they become one, integral ‘persona’, one ‘thinking homeostat’. (Lem, 1964a, p. 213) After Frank Rosenblatt, he goes on to discuss ‘perceptrons’—systems equipped with ‘a visual receptor’ analogous to the eyes, and ‘pseudoneuronal elements, connected randomly, able to recognise simple images, thanks to learning under the supervision of a simple algorithm’ (Lem, 1964a, p. 229). He also proposes that we should build ‘antagonistic machines’ that restrict and supervise each other, what we might recognize as an early description of generative adversarial networks (Lem, 1964a, p. 237). Summa is full of such delightful speculations, which, in retrospect, strike us as especially prescient. In contrast to these, Lem is overall unimpressed with what he calls ‘the cybernetic return’ of ‘the Medieval mythos of the homunculus, an artificially created thinking being’ (Lem, 1964a, p. 145). While he considers that it may be

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 102 Europe possible to create ‘an electric brain’ as a copy of the biological one, he dismisses this task as belonging to the realm of religious desire, not scientific priority: There will be no artificial humans, because it is unnecessary. There will be no ‘rebellion’ of thinking machines against humans. There is an older myth that informs these concepts—the satanic myth—but no Intelligence Amplifier will become the Electronic Antichrist. All of these myths come down to a common, anthropomorphic denominator, which supposedly can account for the behavior of thinking machines. (Lem, 1964a, p. 149) Clearly, Lem was not at all interested in anthropocentric or anthropomorphic models of AI, but rather in intelligence amplifiers, black boxes, and cybernetic models of governance, inasmuch as each of these forces us to delineate (and maybe exceed) the limit of human knowledge and comprehension, pushing us to confront otherness and contingency, a theme common in many of his novels. Precisely because he worries that human civilization at large is like the characters in Solaris, who turn any potential knowledgesurface or encounter with the alien into mirrors, he is interested in the possibility of machine knowledge that could help us deal with the surplus of information. Throughout his work, Lem perceived cognitive chaos and the inability to reason as obstacles to civilizational development. He was concerned that not only do we humans fall prey to ideological artifice, such as communism, but also that the domain of science itself generates a lot of noise. The desire to know and the volume of knowledge is accelerating but human mechanisms for discerning patterns are weak, preventing our species from understanding the trajectory of our own evolution, much less being able to influence it. Referring to Ashby’s (1956) influential book on cybernetics, Lem writes that principles of knowledge can emerge from within a system that operates chaotically and without internal logic. He believed that we should aim to find ways to sift through this noise and continue producing knowledge, otherwise, submerged in it, human civilization is ‘like a ship built without a blueprint’ and with ‘no steersman’ (Lem, 1964a, p. 418). 6.5 The gnostic machine One of the most powerful ideas in Summa with regards to what we may term ‘artificial knowledge’ is the brief mention of a gnostic machine. I believe it to

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 103 be especially important not only because it departs from common anthropomorphic imaginations of AI, but also because it connects so well to Lem’s work as a novelist and intellectual. It is key to what I would call his ‘secular theology’ of human technological evolution, where the Catholic tradition of negative theology, with unknowability of the divine as its centre, overlaps onto Lem’s life experience of contingency as a Pole caught in the USSR-Soviet cybernetic arms race. The gnostic machine—only mentioned a few times in Summa (1964a)—is described thus: A gnostic machine must take into account, for the purpose of creating a theory of complexity, a huge number of variables, such that the algorithms of contemporary science cannot produce . . . At the outset we would receive a theory, coded as, let’s say, a whole system of equations. Would humans be able to do anything with these equations? (pp. 234–5) [...] [With regards to thinking machines], how much of the matter is under human control is—let’s admit it—a matter of perspective. That man can swim does not mean that he can swim the ocean on his own without a ship, not to mention an analogous situation with jets and space rockets. A similar evolution is now taking place, somewhat in parallel, in the information universe. A human might direct a gnostic machine towards a problem that he—or his descendants—might be able to eventually solve on their own, but the machine might during its work open his eyes to problems he had not even suspected. In the last instance, who has the agency here? (p. 441) In a way, the gnostic machine is the opposite of the totalitarian machine of ideology—the former is governed by opaque, indecipherable, inhuman logic, where the former is homogenous and overly controlled, reduced to the artifice or utopianism of human ideology. In a subsequent chapter, ‘On the Creation of Worlds’, on the surface unrelated to AI, Lem elaborates on this further, focusing on how we could automate science and metaphysics by creating technologies that operate at the limit of human comprehension, rather than ‘imitating’ human-level cognitive functions, which are prone to failure. The goal is further removal of humans from the knowledge-production process, the production of the ‘autognostic or cybergnostic machine’ (Lem, 1964a, p. 409). If gnosis is a form of revealed knowledge based on direct participation in the divine, without mediators, Lem’s description of our technological future

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 104 Europe resonates here: ‘could we extract information from Nature without the mediation of brains, human or electric—to create something like a breeding farm facilitating the evolution of information?’ (Lem, 1964a, p. 139). For Lem, this project follows closely the logic of evolution—he calls this convergence ‘imitology’ (Lem, 1964a, p. 427). One potential method is imitating the workings of in vitro reproduction in order to generate new metaphysics: doing with information what we already are able to do with life. What interests Lem here is that humans, by using machines, are able to create life in an artificial setting, while not completely understanding how ‘life’ works. This is a black box—a system whose total inputs and outputs are not fully comprehensible to the designer, all while the design itself remains workable. Could there be an analogous process but aimed at creating an artificial epistemology? Akin to genetic engineering, could we ‘do something analogous in the domain of information science? Breed and cross-breed information, initiate its fertilisation so that in the end we receive a grown organism—a scientific theory?’ (Lem, 1964a, p. 398). In evolutionary terms, for Lem, ‘information should originate in information as one organism does in another. It should fertilise, cross-breed and be subject to mutations’ (Lem, 1964a, p. 399). A total synthesis between biological and technological evolutions. Such breeding of information potentially creates epistemic configurations beyond humans, just as in vitro fertilization lets humans ‘create’ other mammalian life. While ‘there is no thought that a human hive mind would think that an individual brain could not think on its own’ (Lem, 1964a, p. 443), ‘imitology’ opens up speculative possibilities of transcending these limitations. This proposition is also a reversal of theological and evolutionary principles: a scenario in which a less-intelligent being—humans—might be able to create a more intelligent one. It is as if monkeys were somehow able to engineer divinity. The evolutionary significance of machines is revealed, partly, in this process. If in theology, ‘gnosis’—knowledge—denotes insight into the infinite and the deliverance of humanity from earthly constraints, a parallel phenomenon in Lem’s narrative of techno-biological evolution must be black-boxed, to some extent automatically performed or mediated through a machine. ‘Automatic gnosis’ or ‘artificial epistemology’ rather than ‘artificial intelligence’ is one of the latent promises of computation. One of Lem’s later works, Golem XIV (1978), considers an example of how this could pan out. In the novel, a superintelligent computer evolves beyond its initial military tasks—its traditional cybernetic framework—and learns to perceive human designs as lacking in internal consistency. It subsequently lectures humans on existential trajectories, informing them that they are mere vessels for otherthan-human intelligence, just like their bodies are evolutionary vessels for life. The evolution of intelligence needs ‘reason and consciousness but [is] not controllable by the latter’ (Jarzębski, 2006, p. 110)—intelligence might evolve well

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 105 beyond consciousness and organic life. No novel better demonstrates Lem’s intellectual trajectory from Dialogi to Summa than one that features a computer evolving beyond its cybernetic, military, and ultimately human applications in order to play a part in the techno-biological evolution of intelligence and confront humans with the limits of knowledge. 6.6 Conclusion Joanna Zylinska, media theorist and translator of Lem’s Summa into English, writes in her book AI Art: Machine Vision and Warped Dreams that the question of AI is ultimately about ‘rescaling our human worldview’ (2020, p. 152). Though what we commonly understand as AI today—machine learning, natural language processing, expert systems, deep learning, computer vision, and robotics—are only indirectly discussed in Summa, AI as an idea draws on multiple ‘sociotechnical imaginaries’ that end up providing paradigms for understanding the relationship between our imaginations of technology, the social order, and the designs we actually carry out (Jasanoff, 2015, p. 6). As discussions about ‘non-western AI’ continue, Lem’s work provides an intriguing counterpoint to the increasingly common practice of using ‘eastern’ or ‘non-western’ theories of media and technology as salvation narratives for the perceived sins of ‘Western AI’. Framed by his life and work as a Polish writer, his unique Catholic-cybernetic intellectual scene in Kraków, and his interest in how technology alters axioms of human cognition and evolution, the concept of a ‘gnostic machine’ can be a starting point for fruitful discussions about the possibility of ‘AI’ as an existential technology, not confined to its possible ethical implications for humans. References Aquinas, St. T. (1485) Summa technologiae. Basel: Michael Wenssler. Ashby, W. R. (1956) An introduction to cybernetics. London: Chapman & Hall Ltd. Bereś, S. and Lem, S. (2013) Tako rzecze Lem: Ze Stanisławem Lemem rozmawia Stanisław Bereś. Kraków: Cyfrant. Berkeley, G. (1713) Three dialogues between Hylas and Philonous. London: Henry Clements. Bratton, B. (2020) ‘Synthetic gardens: Another model for AI and design’, in Vickers, B. and Allado-McDowell, K. (eds.) Atlas of anomalous AI. London: Ignota Books, pp. 91–112. Butko, P. (2006) ‘Summa technologiae: Looking back and ahead’, in Swirski, P. (ed.) The art and science of Stanislaw Lem. Montreal: McGill-Queen’s University Press, pp. 81–104.

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 106 Europe Csicsery-Ronay, I. (2006) ‘Lem, Central Europe, and the genre of technological empire’, in Swirski, P. (ed.) The art and science of Stanislaw Lem. Montreal: McGillQueen’s University Press, pp. 130–53. Czapliński, P. (2016) Poruszona mapa. Kraków: Wydawnictwo Literackie. Dawkins, R. (1986) The blind watchmaker: Why the evidence of evolution reveals a universe without design. New York: WW Norton & Company. Dawkins, R. (1976/2016) The selfish gene. 40th ann. edn. Oxford: Oxford University Press. Demeter, M. (2020) ‘Representation of Eastern Europe in media studies’, Journalism & Mass Communication Quarterly, 97(3), pp. 586–9. Downey, J. and Mihelj, S. (eds.) (2012) Central and Eastern European media in comparative perspective: Politics, economy and culture. London: Routledge. Fiałkowski, T. (2015) Świat na krawędzi: Ze Stanisławem Lemem rozmawia Tomasz Fiałkowski. Kraków: Cyfrant. Gajewska, A. (2021) Holocaust and the stars: The past in the prose of Staniław Lem, tr. K. Gucio. London: Routledge India. Gerovitch, S. (2008) ‘InterNyet: Why the Soviet Union did not build a nationwide computer network’, History and Technology, 24(4), pp. 335–50. Goffi, E. (2021) ‘Escaping the Western cosm-ethical hegemony: The importance of cultural diversity in the ethical assessment of artificial intelligence’, AI Ethics Journal, 2(1), pp. 1–11. Haraway, D. (1991) Simians, cyborgs and women: The reinvention of nature. London: Free Association Books. Hayles, K. (1999) How we became posthuman: Virtual bodies in cybernetics, literature, and informatics. Chicago: University of Chicago Press. Hofstadter, D. R. (1979) Gödel, Escher, Bach: An eternal golden braid (Vol. 13). New York: Basic Books. Hui, Y. and Amaro, R. (2020) ‘Designing for intelligence’, in Vickers, B. and AlladoMcDowell, K. (eds.) Atlas of anomalous AI. London: Ignota Books, pp. 53–75. Ihde, D. (2008) Ironic technics. Copenhagen: Automatic Press/VIP. Jarzębski, J. (2006) ‘Models of evolution in the writings of Stanislaw Lem’, in Swirski, P. (ed.) The art and science of Stanislaw Lem. Montreal: McGill-Queen’s University Press, pp. 104–17. Jasanoff, S. (2015) ‘Future imperfect: Science, technology, and the imaginations of modernity’, in Jasanoff, S. and S. H. Kim (eds.) Dreamscapes of modernity: Sociotechnical imaginaries and the fabrication of power. Chicago: University of Chicago Press, pp. 1–33. Leder, A. (2014) Prześniona Rewolucja: Ćwiczenia z Logiki Historycznej. Warsaw: Wydawnictwo Krytyki Politycznej. Lem, S. (1951) Astronauci. Warsaw: Czytelnik. Lem, S. (1955) Obłok Magellana. Warsaw: Państwowe Wydawnictwo Iskry.

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA Artificial Intelligence in Stanisław Lem’s Summa Technologiae 107 Lem, S. (1957) Dialogi o zmartwychwstaniu atomowym, teorii niemożności, filozoficznych korzyściach ludożerstwa, smutku w probówce, psychoanalizie cybernetycznej, elektrycznej metempsychozie, sprzężeniach zwrotnych ewolucji, eschatologii cybernetycznej, osobowości sieci elektrycznych, przewrotności elektromózgów, życiu wiecznym w skrzyni, konstruowaniu geniuszów, epilepsji kapitalizmu, maszynach do rządzenia, projektowaniu systemów społecznych. Kraków: Wydawnictwo Literackie. Lem, S. (1961) Pamiętnik Znaleziony w Wannie. Kraków: Wydawnictwo Literackie. Lem, S. (1961) Solaris. Warsaw: Wydawnictwo Ministerstwa Obrony Narodowej. Lem, S. (1964a) Summa Technologiae. Kraków: Wydawnictwo Literackie. Lem, S. (1964b) Niezwyciężony. Warsaw: Wydawnictwo Ministerstwa Obrony Narodowej. Lem, S. (1968) Głos Pana. Warsaw: Czytelnik. Lem, S. (1978) Golem XIV. Kraków: Wydawnictwo Literackie. Lem, S. 1984a. ‘Chance and order’, tr. F. Rottensteiner, The New Yorker, 30 January, pp. 88–9. Lem, S. (1984b) ‘Science-fiction: A hopeless case—with exceptions’, in Rottensteiner, F. (ed.) Microworlds: Writings on science-fiction and fantasy. San Diego: Harcourt, Brace and Company, pp. 45–105. Lem, S. (1987) Fiasko. Kraków: Wydawnictwo Literackie. Lem, S. (1996) Sex Wars. Warsaw: Niezależna Oficyna Wydawnicza Nowa. Lem, S. (1999) Bomba Megabitowa. Kraków: Wydawnictwo Literackie. Lem, S. (2003) Tajemnica Chińskiego Pokoju. Kraków: Towarzystwo Autorów i Wydawców Prac Naukowych Universitas. Lem, S. (2006) Rasa Drapieżców. Teksty ostatnie. Kraków: Wydawnictwo Literackie. Lem, S. (2016) Planeta Lema. Felietony ponadczasowe. Kraków: Wydawnictwo Literackie. McLuhan, M. (1964) Understanding media: The extensions of man. Toronto: McGraw-Hill. Monosson, E. (2013) ‘Can life evolve from wires and plastic?’ Aeon [online]. Available at: https://aeon.co/essays/can-life-evolve-from-wires-and-plastic (Accessed 4 September 2021). Orliński, W. (2017) Lem: Życie nie z tej ziemi. Warsaw: Wydawnictwo Czarne & Wydawnictwo Agora. Sayers, S. (2019) ‘Marx and teleology’, Science & Society, 83(1), pp. 37–63. Shannon, C. E. (1948) ‘A mathematical theory of communication’, The Bell System Technical Journal, 27(3), pp. 379–423. Swirski, P. (2006) The art and science of Stanislaw Lem. Montreal: McGill-Queen’s University Press. Swirski, P. (2015) Stanislaw Lem: Philosopher of the future. Liverpool: Liverpool University Press. Taube, M. (1961) Computers and common sense: The myth of thinking machines. New York: Columbia University Press.

OUP UNCORRECTED PROOF – SECONDPROOFS, Mon. Mar 8 2023, INTEGRA 108 Europe Turing, A. (1950) ‘Computer machinery and intelligence’, Mind, LIX(236), pp. 433–60. Vickers, B. and Allado-McDowell, K. (eds.) (2020) Atlas of anomalous AI. London: Ignota Books. Virilio, P. (1999) Politics of the very worst: An interview with Philippe Petit. Lotringer, S. (ed.), tr. M. Cavaliere. Los Angeles: Semiotext(e). Wiener, N. (1948/2019) Cybernetics or control and communication in the animal and the machine. Reissue of the 1st edn. Cambridge, MA: MIT Press. Williams, D. H. and Shipley, G. P. (2021) ‘Enhancing artificial intelligence with indigenous wisdom’, Open Journal of Philosophy, 11(1), pp. 43–58. Zylinska, J. (2020) AI art: Machine visions and warped dreams. London: Open Humanities Press. Endnote 1. All translations from the Polish by the author.