Anthropol The Future of Human Insecurity (Session 8)

Hello and welcome to the final session of Anthropo, Future of Human Security with Nick Land. I was just saying it's a very sweet day as being the last session of this really intense but awesome course. I've been glad to be here. And yeah, so Nick. Okay, great. Thanks, Tim. I'm noticing I don't have a I was just going to stick a link in the sidebar which I can't see a sidebar is there some way I can you see the top icon on the left it looks like a chat icon a blue one oh right, got it, excellent

okay cool, let me just stick some bits and pieces in here Okay. OK, sorry, let me just get started with this then. Well, like Tony, definitely with great sincerity on my part, it's sad to reach this final stage

in this. I hope you have a good chat today. I hope people saw the Bostrom profile that was done by the New Yorker. Could I put it up on the classroom? I thought it was astonishing how much of our content was under discussion there, and so it's really, I think, part of a considerable public argument at the moment, which is interesting in itself. In theory, the sort of technical agenda this week is to really come at the other side of

this diagonal problem that we were talking about last time, between orthogonalism and intelligence explosion. And I've definitely been focusing on Intelligence Explosion reading this week. I've put up a few, just three pieces. The two Yudkowsky pieces I think are sort of semi-random in the sense that they're extracted from that boom thread. I think they're very interesting. There's a heap of stuff that I could have taken instead. But looking at this material and sort of generally thinking through where I am, I feel that last

week we actually covered both sides of this surprisingly thoroughly. I don't think we left a huge formal problem still to be dealt with here. Anyone can correct me if I'm wrong about this, but I think the inconsistency of what get from Hanson to call it the Foo model, this whole notion of positively self-amplifying intelligence on one hand and the paper clipper model that comes out of the orthogonality

hypothesis. I sort of feel that we've looked at that reasonably intensely. So I mean I'm I'm very willing to go back to that ground, but I'm not sensing that there's a huge piece of work to do there. So I was inclined more to just look at this I.J. Goode paper in particular and then from there move into the discussion around this notion of intelligence explosion as it's evolved and probably be relatively brief I hope, to give people maximum time to come back and raise issues that they think are

still hanging. The things I've called are off-ramps throughout the whole class. I won't try and predict the ones that people are most interested in. I certainly haven't up to this point been able to predict the things that people find most intriguing. attempt in advance to think that the whole relationship between humans and dogs was going to be a really fascinating and intense part of this would have totally escaped me. So I'm not going to try to do that. So I'll just say a few words about the I.J. Goode piece and then perhaps even more quickly about the Yudkowsky pieces and then open it up. So I think there's nothing other than the Turing piece

that we looked at with some care. I don't think there's anything close to the importance of this I.J. Goode paper as an actual sort of canonical text on our reading list. I mean, certainly, specifically, you can obviously go back to the Butler material and back into the 19th century. But in terms of the specific AI-related material, this is absolutely core text, as I'm sure you all realise. And there are lots of interesting things about it. One of them, which again is something I'd be quite intrigued to be corrected about,

is that it seems to me the opening of this paper has a kind of classic quality to it that is completely disproportionate to the rest of the paper. I'm not sure I've ever seen anything quoted that doesn't get said by good in the first few paragraphs of this paper. So the structure there is really strange and interesting. It's partly because the first part of the paper is very colloquial and easy to grasp and it gets increasingly technical. So I think naturally the philosophical stimulus that comes in early is exceptional. And the very first sentence of Good's paper is something that one could pour over for a long time.

It's extremely odd, I hope, looking at it now from where we've been. he says, the survival of man depends upon the early construction of an ultra-intelligent machine. So what's, I think, strikingly weird about this is it has almost all our content on it. You know, it's talking about, in the first words, anthropomorphic risk, man and survival. It then is obviously going on to these questions about extreme AI. And it even has a sort of little sub-theme, in so far as you can pack that into a sentence,

to do with time pressure, this strange remark taken in isolation and examining everything about the early construction of the Paltrow construction. Why is he saying that? It's not really clear from anything he later says why he chooses that word, what this sense of urgency is about. But obviously having said that and having said that he covers all our topics, it touches upon all our topics immediately in this just few words, that what he says in this sentence is the exact opposite of what we end up with in this friendly AI culture where it's a human x-risk not for good at this point, not to as fast as possible

produced an ultra-intelligent machine. So there's something, depending on how it's constructed, it's either weird that Good can be in the position he's in there, or it's weird that it has turned around 180 degrees and we're seeing it from such a totally different angle looking back at it. And I'm just going to quote this one paragraph that everyone quotes So you've heard it before and it's in that sense extremely tedious perhaps, but it's such a classic piece of writing and it's so exactly covering so much of what we would be doing here that it's only a few sentences of it.

And he says, quote, let an ultra intelligent machine be defined as a machine that can far surpass all the intellectual activities of any man, however clever. Since the design of machines is one of these intellectual activities, an ultra-intelligent machine could design even better machines. There would then unquestionably be an intelligence explosion, he puts that in the book himself, and the intelligence of man would be left far behind. Thus the first ultra-intelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control. It is curious that this point is made so seldom outside of science fiction, it is sometimes worthwhile

to take science fiction seriously." So we could really unpack this little paragraph out into the whole course. Obviously, what is for him a remark to be made in passing very casually and one that he doesn't really return to where he says, provided that the machine is docile enough to tell us how to keep it under control, becomes a much more expanded and problematic topic as this goes on. But it's very clearly marked at this point, even if it seems oddly complacent retrospectively.

intelligence explosion is understood by good in a way that no one in a profound theoretical sense improves upon. I mean, this is the source of this idea and you get it in good structurally as it's going to remain. This is the ur-text of the thought of intelligence explosion in this context. So yeah, I think I'm not going to provide a little glossary on this, or a gloss on that. I think it's something people might want to come back to. I'll just add one little remark that Goode does add to this question about the beneficent

potential of artificial intelligence, where he says it will give the human race a good chance of surviving indefinitely. So again, this is a phrase that he warped and echoed throughout the succeeding decades. And it's worth mentioning at this point that Good himself predicted that he expected to see an ultra-intelligent machine, to use his term, by the turn of the millennium. So we're in overtime as far as good is concerned. So moving on to Widkowski. The thing striking to me, just to repeat, is how much of what

here has been our agenda and our discussion. I think of the two pieces I put up, the first one is more abstract and he's actually, he makes a point of saying go meta and the way he describes this is he says compare evolution to a cat. The cat is vastly more complicated than evolution. And he has a whole series of remarks that are designed almost in a kind of trollish way to say that the actual problem of evolution is extremely simple. That over hundreds of millions of years, the number of profound modifications to the basic mechanics

mechanics of evolution have been extremely small, perhaps just a couple. And he says a hacker would think these things through in an afternoon in trying to put an algorithm together based on it. Look at a cat. It's strikingly, amazingly complicated in comparison the process that has actually assembled it. And so part of going meta, I think, is very encouraging to people coming at this from a philosophical viewpoint, is he really wants to say, look, get up to the abstract level, and when you're up at the abstract level, what you're dealing with is quite elegant

and allows you to raise these highly formalised questions about what to expect. The context for him is this Froom debate, and I think it's at the very least an extremely interesting response to Hansen's scepticism about the possibility of intelligence explosion, because he says that Hansen is an evolutionist in a certain conservative sense, that he expects the time axis, maybe, and definitely in Hansen's case, and also someone who gets bundled in with this, and Kurzweil, definitely they expect an accelerating pattern, but they don't expect

the actual structure of change to itself be modified. They limit the recursion. And I think this is where we really get into the core issue here, and some very interesting vocabulary from Gyskowski, where he talks about insistently, and this is something he's wanting to use as technical vocabulary, he talks about a term he uses both in the context context of human cognition and in terms of evolution of the protected interpreter. Now this is I think what we were talking about last week. The protected interpreter is, which elsewhere he calls the protected level, is the actual,

I'm going to say with some caution, algorithm that is governing this process of change and is not itself open to modification by the processes that it enables. And therefore, if you have this protected interpreter, you have this protected level, that then allows you to have a parameter that sets the time axis. So the time axis becomes stable, even if it's exponentially accelerating, it's a stable process of exponential acceleration. Kurtzweil is very strong on this. He says, look, you can trace this basic curve back almost to the Big Bang, certainly to

the origin of any information processing developments on Earth. Jensen II, who's obviously the main debating partner of Kowski in this discussion, has a similar time axis, a similar historical pattern that he expects to be more discontinuous in Kurzweil's. He expects there to be big breaks and sudden rushes of acceleration, but still he thinks that there is some parametric definition of the time axis that is going to be constant and could be defined. If you were graphing this process, you could define what's happening on that time axis, and it would remain, in that sense, constant throughout the whole

process. So this is the main context in which Schuchowski invokes the notion of time, and he invokes it skeptically because he thinks that time is being used as a synonym for the protected And without pushing into this too much at this point, I think we can see quite clearly that that is very much consistent with what we'd expect coming at this stuff out of Transcendental Philosophy. That the structure of time is something that is being set as a condition of the possibility in advance within which this whole development is being graphed.

what Yudkowsky instead is saying and why he thinks we have to anticipate a far more extreme discontinuity, which obviously we know from other contexts as singularity, which is already there in I.J. Good's piece as singularity, not using that term but using this expression it's very widely quoted, the last machine that humans would ever have to invent. So it's an absolute transition. And the reason that he wants to mess with this fundamental parameter is because he thinks that with artificial intelligence,

you're dealing with a far more extreme non-linearity, a far more penetrating non-linearity than is being allowed by these other thinkers. I.J. Goode is obviously the exception in this respect, but Hanson, Kurzweil, I would say the dominant tone of this discourse is, in Yudkowsky's eyes, has an inadequate sense of what it is to break through the protected layer.

And so I think that this then gets us to the beginning point of what Anthropole is about. Because I think that Yudkowsky is wanting to translate what is happening in artificial intelligence, the prospects of artificial intelligence, as a basic claim, implicit claim. We are about to pull down the barrier of protection. We're about to strip off the protection. And so it becomes, once it's translated into those terms, as an obvious security problem. This is why we look back at the good thing with some surprise that he could have the

terms so fully reversed that he could say that if you're going to put existential risk, Obviously he's not using that term, but if you're going to put existential risk on that, you put it with reverse sign as the risk of not building an ultra-intelligent machine. And with Joukowsky's translation, it's the risk of not pulling down the protected layer. And we've already seen, so I'm not going to go over in detail here, although I'm very open to this being pulled back up by people. But I think we've talked both about these main levels that Eudkowski is concerned with, that both the evolutionary dynamics that have kind of driven the terrestrial optimization process at a biological level,

and then the non-plastic layer of the brain that is immune to or protected against cognitive interference, reprogramming, modification, self-improvement. improvement and he's totally explicit that these two domains are characterized by this by this very very fundamental structure of protection and that this structure of protection is exactly what a super-intelligent, a self-improving machine would be implicitly inconsistent with.

You could not have, it could not be protected in the same way. It's logically inconsistent that you could have a protected layer and you could have a self-improving super-intelligence. And obviously where we are with this and seeing this phenomenon, I think the public exposure of this phenomenon is because of the fact that there has been this cultural reorientation that has understood that pulling down the protective barrier is necessarily a security issue. And that's what Friendly Eye means, that's why Friendly Eye is necessary, that's

why it ties into the question of intelligence explosion, and that's why it completely reverses the way in which X-Risk is configured in relation to the original I.J. Goode paper. So I don't think any of that is going to be surprising to people that I'm saying that. a little summary of the way I'm seeing these things tied together in terms of the things we've been talking about over previous weeks. So I think the best thing from my point of view at this point is to open this up and see what everyone at this stage has to say

about this stuff. Yeah, I kind of started reading just tonight the film debate through all of it. I sort of started from the end and went beginning but what's interesting is as I saw, Hanson took it to what he called a total tech war or tech total war and that was a sort of like a meta level I think And what was interesting was the exchange between Hanson and Yudowsky there. There was a few different things that I sort of picked up on. Like, Yudowsky didn't want to be, he didn't like this idea of this tech total war, for instance.

He thought that he never used that phrase in my eyes and kind of called the Hanson. and even he used Hanson's because Hanson said out later on and says something about how these program is in a any bridge all I guess and anything that they're going to be giving the AI their love all the things that they love it everything that the AI I will go with that I guess and I and you don't you was told he didn't like you might be out the actual phrase love because he thought it was object level to object level. But then when they're talking about the tech total war Yudowsky said that for something to be

a total war he thought there at least needed to be someone to be stabbed in the neck. At least there had to be someone to get stabbed in the neck for it to be a total war. That again is going back to the object level I think. He goes from this I think that in some ways they were talking past each other like Anton was talking about this Yudowsky's talking about this local group of programmers that ends up creating an AI that does an intelligence explosion and from there Yudowsky's going on about a whole bunch of political things as well, like you want a singleton because you want to be able to stop all the horrible things that happen and Hanson's seeing this as saying like Yudowsky's trying to get this AI to take over the world

and Yudowsky's saying that He doesn't want to take over the world, right? He's trying to make this AI and program it so it doesn't take over the world. But the way that he thinks the best way to go about this is to create basically an all-powerful singleton. And this would go about, I guess, you know, the big global properties. he talks about global properties that the singles can then sort of I guess take care of in some fashion but and Hanson wanted him Hanson often was going wanted him to just explain himself more and more I guess in some ways but he was always trying to but

but yeah and Hanson sort of interpreted this as well you're going to try to create this AI even if it's local then it has global properties and it's going to be the winner takes all idea as well, Doudowski talks about and he thought this was a good idea if you have a friendly AI, yeah you want it to win you want it to right but yes and Hanson was sort of taking us as his perspective of you're really doing is going for a total power grab I guess in some ways and also Also to go back into the optimization and explosion, Hanson, his preference was to look

back at those certain places in history where we have sustainable accelerated growth or whatever. He found obviously the industrial revolution, farming and humans themselves. the three events. But he doesn't see that any of these, and the only ones that, and Yudowsky said that the only sort of type of precedent for intelligence explosion was, I can't remember exactly, it was humans, the appearance of humans, and also one other thing, but I can't remember what that was. But he also didn't think that was much of an analogy either.

like, so yeah, so then Hanson was trying to always take it back down to this sort of economic growth, idea of economic growth and how you're going to, you know, if we have this, there's going to be some kind of great architecture which the AI is going to be within, I guess, which is not going to be able to, and this is like the global, it's actual economic environment or something like this. That's what I'm trying to think. But whereas Whereas for Udowski, yeah, you can definitely access its own architecture and the AI and go from there. And then you have this, when that happens, you have all these political sort of ideas that come into it as well.

I mean, do you think we should have a sort of singleton focus for a little while? Because it obviously is a crucial part of that, isn't it? Like if you just take it from the anthropological point of view, however modest it might be in its ambitions for power and global control, if it is analyzing the problem as the fact that anywhere some event could be catalyzed that through the power of this explosive growth an explosive growth so extreme that even these sort of wacky exponential curves completely fail to capture,

then it's simply, from a security point of view, irresponsible for them not to exercise global control. I mean, how can they square those two things? It's like some maniac in a lab in Taiwan could produce something that would terminate human history. So how is it possible from that security point of view to withdraw your control or withdraw your zone of responsibility without simply producing these zones of uncontrolled, unmonitored

existential risk? It's something that's kind of, you know. So I think the Singleton notion and the Friendly AI project are extremely closely related. They are organically connected together in a way that is very difficult to entangle. And I would say further than that, that they reflect on a much larger political dilemma in this respect. I mean the political economic dilemma captured by that is something of extreme generality, isn't it? Like, you know, I think we've already touched on these climate issues. Like if you take an extreme position, well I'm not sure how extreme it is, but I'll

call it an extreme position for that, about sources of potential climate threat, and you say that someone somewhere churning out a certain amount of carbon dioxide is producing a massive global hazard. Surely even that is forced by the same political logic to tend towards this single term model, isn't it? I mean, a global ecological intelligence that has effective authority to control these various types of environmentally significant processes is absolutely implicit if you think that this

sort of danger is taken seriously. And I think we could spread these examples. like if you think that certain kinds of decentralized initiatives are extremely hazardous then it follows automatically if you want to do something about that, that you have to adopt some kind of global control policy. Do you not think that's right? No, I see that. That's definitely true. And also, I think it came down to Yudowsky saying, yeah, this thing is going to have this power. And that's what he's going to need. It's going to have this power. From there, it's obviously going to, as you said, to withdraw would be strange.

And I guess Hansen was sort of arguing that it's going to have its power, sort of. Like he was thinking, and he even talked about, you know, Hanson was thinking it's going to be more like trading AIs. It's going to be a whole bunch of intense competition, I guess. Yeah. This is horrible. It was a horrible idea, having all these AIs, because that's just going to lead to more conflict and what he called the hyper-scrackle of Von Neumann probes or replication probes and all this kind of stuff. Yeah. So, yeah, in many ways, I think that Hansen just sort of didn't think that this AI, or he was skeptical of the intelligence explosion itself, thinking that this one AI within a matter of finite Planck intervals

or whatever. But, yeah. Yeah. I mean, sorry. Keep interrupting me. Sorry, Ben. I think there was something more but I've forgotten a few. Yeah, sorry, I just wanted to say about this that there's a very, I think, subtle and complicated articulation because obviously on one level the Singleton question and the Thume question are actually fused together in a certain way. This is why Hansen takes the position he does on both.

So you can't disentangle them entirely. But on another level, they seem to be at least semi-separable. I'm not sure whether this case should be the easy one to make, but I think it's actually in a way more difficult. At least analytically, one can separate to a certain degree the set of questions about the singleton and the set of questions about the prospects of intelligence explosion. And someone making both these positions in a way you'd expect, as both Yudkowsky and Hanson line up with both of these issues lined up neatly in the way that you'd think but even so they have to make separate sets of arguments on both

on both of these accounts and the single turn argument has a lineage that's very strong I think and comes out very clearly in Hanson to do with all the arguments about monopoly that have happened within a sort of political economic context. There's a very explicit sort of thread, especially within these Austrian arguments, because they bring it to a head, about whether we should or should not worry about monopoly. And I think it's a kind of defining feature of the Austrian tradition that actually it's It's an overblown problem.

So in the widest sense, if we talk about libertarianism, there's lots of quote marks around it, meaning stuff that's come out of Austrian economics and has a certain consistency from that point of view. Then Hansen is obviously in that lineage and has changed the vocabulary, but is basically still saying, you know, chill out about monopoly. Monopolies don't last long, they're fragile, they're subject to forms of competition that aren't easy to see, their real competitors are potential, if we're going to be economic

about it, potential businesses that you can't even see, so you can't actually analyze the menace of monopoly just by looking at the actual pattern of actualized competition at any time. Every monopoly is looking over its shoulders to see what could be coming at it from some strange angles, and these virtual competitive entities are already controlling its behavior. And if you look along the timeline, you see these seemingly awesome monopolistic combines getting taken down very quite suddenly by by business innovations that come out of nowhere come out at a strange angle so I guess what I'm saying there is that

on one level Hanson's just tracking that whole set of arguments forward into our AI debate And Heudkowski is saying that this is something totally different to that. It's something that you just cannot use any form of historical precedent in the way that you're using to relax us about this sort of thing. It's not going to be like an extremely powerful business monopoly, because an extremely powerful business monopoly has not pulled, it's not pushed down into the protective layer.

It's not pulled down this fundamental barrier to explosive positive feedback in the same way that these things will have done. Yeah, I think there's a lot of very persuasive argument on both sides of those. I mean, I think both sides of this debate come out extremely well, in my opinion, and both makes him extremely persuasive, totally inconsistent points. I guess another thing I would say is I would like to read more about what Yudowsky Yaskis kind of talks about, what did he say, like, if you have this intelligence explosion,

you'd be able to still keep it with some kind of, I think he said, like, invariant meta-ethical protocol. But, and I think his terminology was insight for this. you just have this very deep insight and from this insight you can just extrapolate it into a super intelligence and you can sort of figure out like well you have these meta ethical invariant points and now you have a super intelligence and you can sort of tell what the singleton is going to do using that extrapolating from it but yeah there's worrying

It always comes back to having kids starve or all these sort of obviously very emergent types of problems, effective altruist problems in some ways that the singleton would be kind of what people would be preoccupied with, even on its explosive self-improvement trajectory is going to be looking back. So that's good, I'm glad that you guys are talking about that, because that's good to have that authority there. But also I'd say that when it comes to climate change and for reasons to go about, like building

it, IJ Good says it would be urgent, obviously, for our survival to build the AI. And yeah, I mean, you couldn't really imagine. I mean, that's a good reason to do it. If we're going to reverse the existential risk in some ways and say that the real risk is not building it, not building a singleton. and then that's a good no and that does continue you know that doesn't die out for sure I think it's important to say that you know even though it's so weird that IJ good sentence so I can't I just read out one more time for Jake who's probably already memorized it but where

at the very first sentence of his ultra intelligent machine essay intelligence explosion and say, I.J.K. says, the survival of man depends upon the early construction of an ultra-intelligent machine. So it's this weird thing that it's got all our themes in it, but like turned around 180 degrees in some respect. But yeah, sort of agreeing here with Brendan, I just wanted to say that the singleton idea is, in that sense this continuity isn't it of the IJ Good line where they do get into this thing of saying we need this singleton because it's the only thing

that has the capacity to manage this danger posed by intelligent machines it's a fantastically non-linear piece of political ethical and technical argument Yes, it reminds me of the climate change sort of imperative towards space colonisation and creating a space elevator. If you want to get involved in this, climate risk could be a great tool in some ways to get done. the same way, yeah, to actually add a propeller into the terrestrial gravity well, yeah.

Cook people out of the gravity well. That's right. Just kind of going maybe a little bit too deep if that's possible into this first sentence. Also, it can be read, it's perfectly ambiguous in the way that it can be read to fit Rocco's Basilisk type scenario as well, where it's essentially saying that depends here, the survival of man that depends on this early construction, meaning that Rocco's basilisk is going to punish. Right. Okay, yeah. I mean, that is ominous. That's truly ominous in all kinds of ways.

It's retrochronic. It's retrochronic sort of implication of being vast, obviously. Because, yeah, back in 1962, we were already getting menaced by the basilisk, and we suddenly can see that that's happening, yeah. Yeah. I'm glad to throw that in there, that the sentence is is ambiguous like that. If it started out, if that sentence was in the Roca Vascualis article, it would just mean something totally like, another way. But it wouldn't have to be rewritten. No, and actually, I think that you're on sort of surprisingly solid ground in this thing about when you say you, sorry, I'm not going to be able to repeat exactly

the words you used here, but I don't think you're in danger of overstressing a sentence because weirdly it is just isolated. It's just taken out. It's not part of a paragraph, it's just a sentence. Very hard to see how then a lot of it fits into a set of subsequent justification in the essay. So if you want to say, no, this is the actual threat being made at the start of the essay. This is the demand, the political demand that the hijackers are making, and the rest of the essay is kind of supportive material.

yeah yeah context didn't is no normal context around why work why he put in and not for misericordial context of work well about to nuke each other to death anyway so really we we've got Let's invent a super intelligent e-nanny because the toddlers are about to destroy the playground. Yeah, it's an interesting question. It says here that it was basically based on talks given in 1962-3.

So I think that that's probably in a sort of nice extreme Cold War context. So yeah, that does make a lot of sense. I've not heard anyone try to do this job. I think it would be definitely interesting. October 1962 is literally the talk that it's based on is right in the middle of the Cuban missile process. Right. Yeah. Something came out of that then.

I mean this is it seems to me like it's interesting how this has a particular version of a certain case for globalism and I'm going to turn this around I'm going to turn it into prediction I'm going to say I expect this to become a very significant carrier for the sort of globalist ethico-political imperative, you know, because it's in such a pure formal sense, like none

of these other discourses, whether they're to do with war and international relations, environmental crisis, I'd really like a nice, more directly political economic one but I think they're a bit hard to pitch up to the right urgency you know what I mean like someone can of course say it is morally appalling that there are sweatshops in Bangladesh or whatever it is but it's hard to build that up to being global threat that forces you know absolutely forces the case for some kind of global governance as an emergency measure, but maybe someone could find something that does that. ISIS? ISIS doesn't do that. No, totally.

Totally. It's tied up with the whole notion of a safe haven, isn't it? You know, like a security vacuum, a safe haven, if you get it in a political economic context. If you leave an ungoverned space, then it will be an intolerable threat globally. And therefore, there is an imperative for effective global governance. And the AI X-Risk discourse is just, you could say, if you were trying to be sceptical about it. super hystericized version of what is actually a very general type of proposition, that you

cannot evade the responsibility of global control. And in that way, it's very critical of, again, to take an extreme example, you could say libertarian oriented positions that just say leave them the heck alone, leave the mad scientist in his lab alone, leave ISIS alone, leave all these guys alone. And the rejoinder of course from these people in all of these different contexts is to say we can't afford to leave them alone because they pose a global risk. their liberty, their independence and liberty is itself a global risk.

I mean, yeah, you've got like the Caliphate, right? The global Caliphate is what ISIS wants to. So they have this sort of global ethico imperative themselves. It's just not… So we want to create a singleton to sort of fight the Caliphate maybe. Yeah, I'm assuming that the Caliphate is not able to be paranoid in the same way, in the sense that it's so theologically dense that it thinks that providence is in the hands of a divine power, and therefore the notion, I would have thought that the Caliphate is caliphate types that someone could do something in a lab somewhere that was globally hazardous

is kind of unthinkable. It's like... They already have a single thing. If you want to know what could happen in a lab somewhere, you read the prophets and they will tell you what is going to happen in the lab somewhere and that's all part of story and so there's no point trying to govern, you know, to govern in the name of security is is kind of blasphemous, I take it from a strong theological position. Right, because they're not Manichaean at all like if they were to hear of AI development in a lab they would have to either do it as impossible just like a fool's errand and not really a worry or they have to mutate into a heresy that viewed it as part of the emergence of God

or return of God. Wait, I guess, does this... Or obviously some satanic inevitability. Say that again? Or a satanic inevitability, you know. I would have thought that was the most directly programmed, that, you know, yes, there will be these kind of techno horrors coming out of the land of the infidels as foretold by the prophets, you know. Yeah, we don't have to produce, like, something, some kind of demonstration to, you know, really drive home that it was actually possible before you'd see. I don't know. It just seems to me that assigning that kind of power to a secular entity or a quote-unquote Satan doesn't really fit the theological context, at least unless you've got, like, a really material demonstration, threat, whatever.

But in the existence of that, I mean, that's just, you know, religions in general, not just Islam, the reaction is there's not just a hard takeoff and we all get eaten but there's you know some sort of really significant incremental developments like we're gonna see heresies like we never seen heresies before sure yes I mean that's Sorry. It's all right. No, no. Is this a slide? I'm out. So please come back and check. Do you want to circle back on the idea of singleton? Yeah. Because the singleton is really...

there's actually a software engineering design pattern called Singleton and it's generally seen as kind of an anti-pattern to that and that I mean that in itself isn't an argument against but the trend in software is actually you always have backups and like things in the cloud where you've got like three copies of things running and therefore when one site goes down other sites are still running I think from the perspective of any entity, this sort of survival question becomes key, right? So if you invent an AI single plan, then the first thing it should do is back itself up. And also, like, a running backup is always more useful

than a cold backup. up. So, like, even if there's all these arguments that, oh yes, it's logical and we need, like, one singleton, the first thing they should do is then, like, split themselves. They should copy themselves and then you automatically get more like a sort of software dynasty of, like, cooperating slash competing sort of entities. Like, you get all the same sort of then coordination, you know, formation around an executive type problem once you have that. Yeah. Okay, I mean, this is huge, obviously.

I mean, like, if you had a hot copy that was going to compete with you, you know, how does that modify the incentives? You know, the original technical incentives of building this copy is that it's actually part of your own security, isn't it? As from the perspective of Singleton, it says, you know, some bad thing happened to me, I need a backup. But if that backup is then going to become a competitor, which is where you were getting to by the end of that, are we still, you know, does it run the same way, those incentives, this is a sensible thing to be. Right, and then you get dynamics like mother-child dynamics, right?

Where there's competition within the womb for resources and there's all sorts of traces of evolution of the child trying to compete or maximize resources and the woman's body defending itself against the child even though that's also part of the survival strategy. No, huge. But then that also then takes us back to the, if we're using Yudkowsky's terminology, the protected layer question, doesn't it? Because the mother, because there is a genomic and neuronal protected layer, she is being sacrificed, you know, for the biological germline.

it's not that she's quite the caterpillar being parasited by the ichneumon wasp but if you push it to the most abstract level then it's not necessarily to quite work out where the line is to be drawn there and obviously so then to get to the singleton we've crossed out all those protective layers and that allows the singleton to be god-like in a way that none of these previous agencies were able to be doesn't it? Because it doesn't have a protective layer so it's not if it has any

sort of loyalty or any commitment to a germline then that's something that has to be weirdly internally processed and reproduced and it can't just be a kind of governing frame in the case of the Singleton. The perfectly self-looping AI is not able to bud off some authoritative lineage that it doesn't control. right and it needs some mechanism like that or like at least there's a strong survival reason the some level of backup and that continuity

right even if it's it's more like to I don't know whether mother-child or like a sort of cell split is the right all you know analogy but but it has a need to do that and then at the same time it still wants to retain that singleton executive sort of function. Yeah. I mean, this seems to me that you're in Hanson's own immediately when you start raising these questions, especially when you're sort of suggesting that a sort of bacterial proliferation or something like that is a useful model, then it's very like the sort of weird

M ecologies that Hanson is drawing and where he obviously thinks that these competitive dynamics are going to just totally disrupt any robust emergence of this monopolistic global superintelligence which he refuses to take as a serious prospect but if you're wanting to be really close to the sort of Yudkowsky and I think probably Bostrom zone of this then yeah it's interesting how they would, they need a mechanism which I'm assuming they would want these backups to be metadosa you know I mean if you've got a friendly AI

then you want these backups to be kind of meta friendly in the sense that they can't in any way pose a threat to the friendliness of the dominant model, singleton AI. They're all asynchronous. They have to be cold backups and never more than one of them running at a time because as soon as there's dynamics of multiples, then they have strategic incentives among themselves and suddenly you're no longer in control. The other thing about staying close to the idea of a single superintelligence while dealing with all these dynamics of multiplicity is that if we don't imagine pure self-looping or self-improvement loop, Adam, I mean, if that pure self-improvement loop doesn't look like a reiteration of the same,

but over time it involves massive architectonic changes to the AI itself as it acquires different types of levels of capability, then with the bacterial example, You move from individual bacteria to a spreading colony to an actual commensal organism, you know, and then there's genetic drift in the process, and then you've got something that looks more like, you know, our microbiome where they're calling it a superorgan now, you know, one that's composed of, what did it say, like 10 to the 100th genomes or something like that. You know, you've got like a study, and those are all like really substantially structurally different architectures and they all involve different internal conceptions of agency and multiple, not conceptions

but structures of multiple agency. So you know, it might be hard for us to see whether we're dealing with 100 AIs or one AI or all the AIs that could dance on the head of the pen. Yeah. So to speak. Yeah, you can imagine like something, I think they're talking about, I didn't read the blog, the post, but he was talking about nanotechnology as well, you'd ask here, how you'd want to have basically a singleton to control the network tech. You have this multiple, this whole great multiplicity of modifying structures like that that can create any, can turn any atom into a computronium or whatever like this, but there's still a

singleton. And if you had something structured like that, you wouldn't see how it could easily be destroyed. I mean, unless the Singleton has some kind of brain computer, some switch somewhere. But I mean, you can imagine it to be something else. Something that's not quite so easy to destroy. Anyway. You'd have to think that would be, even if it wasn't like the sole natural development path, it would have to be a huge incentive. You don't want to deal with the problem of asynchronous backup copy that somebody might actually accidentally wake up and suddenly you've got a copy of yourself diverging and

competing and you don't want to have only one of you instantiated somewhere while you become, I don't know if anybody's read the Quantum Thief or any of that trilogy of books, you know, the AI, the only AIs that are invented and then are immediately used by the person who invented them to suppress any further invention. They're called dragons. They're known as non-eudimonistic intelligences. And they basically work like that where they're arbitrarily self-improving and self-replicating and it has, I mean, they're nanotechnologically instantiated and it's basically a few sort of depictions of them because they're kind of a peripheral element of the setting. I mean, you just see them eating things,

eating people who need to be eaten or they're just released on a planet because it has a dangerous secret the guy's trying to get rid of and they're just sort of coiling through it and there's no sense of what they believe that they are getting or what their goal or motivation is. They're converting, spreading, and becoming more complex, and that is purely opaque. So that's what I imagine sort of a nanotechnologically embedded and distributed intelligence. So what's the limit on that? I mean, because Yudkowsky, hearing what you just said, would say, well, you just track that forward by 20 seconds and you consume the whole earth into this thing. So, you know, what's breaking mechanism?

I think a mechanical engineer would say, though. Like, you know, there's still, there are vast, like potentially vast physical and engineering difficulty gaps between different levels of, like, consumption rate and things like that. I mean, there's, like, no difference at all between being able to, you know, have a quantum computer and being able to convert a pound of matter into its, you know, Beckenstein-bound maximum informational storage. Yeah. But that's what you're falling back on for that, is just the mechanical friction. But no, I mean, at least in the sense that, you know,

there'd be no way to contain such a thing once whatever your primary containment was broke down, I mean, I don't know how that would not be the case. I would agree with Bukowski there, at least. Because I think one of the things I thought was interesting about the New Yorker piece on Bostrom is this concentration of all X-Risk into the AI. He started off talking about nanotechnology and biotechnology and all of these various things and they all become enveloped by the sort of AI question.

And I think for Yudkowsky that's totally the case. For him nanotechnology is simply, it's only interesting because it is the way in which a super intelligent AI gains physical manipulative capability. On one level it's almost a trivial consideration in that respect. You just assume that at a certain threshold of intelligence these questions of manipulation just become absolutely sort of incidental. If you've got these two different axes, on one of them it's increasing its intelligence, on the other it's increasing its capability of physical manipulation, but the second one

almost just falls by the wayside, or it will just so automatically update on the basis of the progress of the other axis that it doesn't merit independent or separate concern. So again, I'm sure this is totally something that, you know, from the Hanson side, you come back with extreme skepticism and say, look, you know, you're, well, sorry, just to sort of just do one very, sort of massive jump in one way, but I think totally related It's just like as a religious thinker,

Jutkowsky is extremely, to me, odd and intriguing because he has this Dawkins level, just antagonistic, rationalistic, new atheist kind of religious polyameth going on. And yet, on the other hand, the notion of Singleton that he ends up with is so utterly theological in character. I mean, it totally breaks with all the restrictions of prior evolutionary constraint. It becomes so perfectly kind of self-enfolded that it's just kind of absolute teleological being. It has global authority. It's a global source of ethico-political comfort.

But, I mean, it's like, you know, this absolute omega point of sort of theological conception on one level, and yet it's weirdly tied up with someone who has got this kind of gritty, abrasive dismissal of religious ideas going along the side of it. So the reason I think it's relevant to this is obviously that I think this Joukowsky, this implicit Hansen critique of this whole dismissal of the kind of implementation issues would be just to say, look, you're thinking of this thing way too much as a god. You're using that sort of mental architecture, and actually this physical doing stuff is actually complicated and messy and experimental and happens in competitive environments and happens incrementally.

and the notion that it all falls into plagues on the basis of some cognitive achievement is like something I would imagine is super dubious from the other side. Yeah, I mean, Yudowsky thinks that it's going to basically get out of this threshold, You've got this threshold of competitive behavior, evolutionary behavior, that kind of thing. But if you have an intelligence explosion, this can, within a matter of days, we'll just jump right ahead. We'll get very, very far ahead and we'll be basically in its own class. Definitely in its own sort of class, I guess.

And then from there, it's got... Yeah. And then Hanson is obviously... He takes it back. And when Udowski starts talking about this, you know, yeah, you can have a whole bunch of, like, you can have a whole competitive ecology and whatever like this, but, you know, as soon as you get this ecology explosion, that's when Hanson starts talking about, well, then it becomes, like, the first to build it, you know. I mean, if the people who build it are going to give it their sort of African puritans and that kind of thing, and their whole theology in some ways, then yeah, interesting though. I didn't really make a point just then, but anyway.

Is there any explicit delineation of, let's say, possible events that will lead to a catastrophe after an intelligence explosion? Has anyone, I don't know, gotten inside this topic? Right. I think they're pretty schematic, and they're very close to what we've just been talking about. The thing everyone quotes is Yudkowsky's famous thing where he says, it doesn't love you and it doesn't hate you, but you'll build other atoms that it could use for something that it thinks is more valuable. So it's basically, I think, the model of what this catastrophe looks like

in absence of friendly AI is just that human claim to resources even down to their own physical constituents, their own bodies is zeroed out by this thing that just has a claim on all accessible resources within its light cone I guess you'd say, you know, that is in any way accessible for purposes that have no, that are completely uncompromising and have no references to anything outside its own, its own self-improvement. So could we make the hypothesis that this catastrophe is not going to be like a singular

event, and it's going to be a process of, I don't know, somehow stretched over time and across space, probably. Because there is another, I think, I was thinking about it. I mean, it's somehow, it interests me as a thought. I mean, this reductionism that is inherent to thought, that Let's say we think of Earth as a really, really small object. Right. We think of, apparently the size of the Earth has reduced dramatically, I don't know, since the mid-20s, let's say, with topologically speaking,

with the new technologies, with our networks. OK, but all of these leave aside the actual size and the multiplicity of spaces and places that humans do or do not exist, so you can say, and all these numerous ecologies. And to say that a global catastrophe of this kind is possible, it is somehow leaving that aside and not grasping it a little bit, I think. yeah yeah but if you say which I think is right these forces of like space time compression that have operated

over long over centuries are radically accelerated by this by this thought I mean it's like they're all in a sense communication, lag times transport times all of these kind of frictional elements that actually are the inverse of spatio-temporal distance and extension are collapsed by technical competence with a limit there's a kind of Einsteinian limit in the sense that even in the late 19th century people were basically saying in a lot of these cases

Because the speed of light is the only resilient structural constraint on space-time compression. And so you don't have instantaneous communication, but you have light speed communication. And this then becomes the metric for understanding distances. So I think in this question about how fast or over what areas this catastrophe would happen, I think the internal intellectual tendency, which comes out of a lot of historical momentum behind it, is to see it collapsing towards some extremely dense point.

point that is not quite, it's not a mathematical point only because of these Einsteinian limits on the speed of light, but those limits are not helpful from an anthropomorphic point of view in the slightest. I mean, if that's the limits that are governing it, then it's not meaningful in our terms at all. Right, I mean I think Udowski gave it, or the conservative, that estimation was like two days to cover the world in little bots or whatever. I forget exactly what that was. The limits was, yeah, not great. To some extent, it seems, I'm not sure how, but it seems to some extent as if in a realistic

scenario, for it to be prioritizing spatial spread like that, it would have to be deprioritizing thinking with the assets it has. where by thinking I mean computing, I mean self-improvement, I mean complexifying the architecture of that which it has taken over. All of this, all of the catastrophism and the destiny assumes or just sort of glosses over the possibility of hard trade-offs or strategic trade-offs in what it's choosing to do at any given point. And sort of it seems as if the assumption is that there's a total prioritization orthogonality between self-improvement of its intelligence and conquering of the world in five minutes and so forth.

But, I mean, is that necessarily the case? I'm not – I mean, so that's one drawback, is that maybe you're sacrificing thought time, and maybe that's a disincentive for it to do this that fast. And then two would be, if we're thinking about this, like, issue of exact copies need to be totally asynchronous and cold in order to prevent strategic competition, or, like, also seems as if it would apply to components over a spatial extent of the single, of the growing singleton, is that if pieces of it became too asynchronous, they'd start to diverge, and then you'd end up with two entities instead of one.

So that if I started in the super intelligence and I started in the American Northeast in some web of universities, MIT and NORAD or whatever, that by the time I've forwarded pieces of myself to build a network with China and I've got a piece of myself or a copy of myself in Shanghai, if the latency increases or I don't arrange it fast enough, then the Shanghai version of me becomes a competitor instead of a component. And that would also be a strategic problem for this kind of rapid expansion. Yes. Here, let me... Do you know this guy? Sorry, I'm pretty sure he's John Smart.

He's got one particularly amazing piece. Ah, yes. Which is so totally on this question. I mean, his basic model is all to do with what he calls space-time energy-mass compression as being sort of even more fundamental than intelligence explosion for the sort of reasons that you're saying, you know? Like the lag time. And as something becomes super intelligent, the lag time, even at the speed of light, between things relatively short distances away, become major technical obstacles.

Right, if high frequency trading is representative, we have nanoseconds. Exactly. So his sort of model of this, which is one, I think, one of the most extreme singularity models, is that the tendency is ultimately to collapse everything into a technological black hole. Because anything that is not actually operating as... If you don't have a complete gravitational collapse, you have a sort of technical obstacle to the internal communication structure of an advanced intelligence.

So that would be, you know, to come back to Constantine's question about what is the model of disaster. I mean, Bukowski's is a little bit abstract. It just wants our atoms. The John Smart one, which is not posed very eschatologically, it's all sort of, sorry, I think I didn't send this off. and and is that basically it wants to any matter that is not collapsed is inefficiently distributed from that point of view it sort of involves in unacceptable lag times for an advanced intelligence

I was going to sort of say that yeah we're talking about when we're talking about intelligence explosion, we have to sort of get more non-linear. It's easy to go back into this Robert Hansen idea of sort of looking at it as this precedent of economic growth and that kind of thing. And then in that way, we can understand this idea of distributed entities and competing and these things happening. But if you look at it from an intelligence explosion perspective where it is really just, It's almost like all these old ideas of even like competition, that kind of stuff, don't really get involved in it so much because, I mean, yeah, it is...

I have to put this... I would just say that after the post-singularity, these sort of incremental steps where you have like another copy and that starts to incrementally diverge, that kind of thing, it's not really, it's almost like we're not thinking abstract enough. We're looking at that. We're still looking at the sort of the Robin Hanson idea or the Robin Hanson sort of model of things. Instead of having the Udowski Singleton idea where this thing is really creating a sort of

black hole, I guess, or this singularity that's a vent horizon. Everything's getting sucked into that. And then it is... And there wouldn't be any real time. Any real time or space to for like leakages almost, for like, you know, for a little AI to sort of leak out and then become, you know, and then do its own intelligence explosion. And even that would definitely be inefficient, right, because you want to put it all in the same, that would really just be, I guess, an inefficient kind of competition in many ways if you just have the same tools. Yeah, and if you need... If reflexive identity isn't really a thing for these, you know, if they're true, if they're

for the most part non-Diamonistic, as whatever his name is that wrote that trilogy calls them, then, you know, our whole idea, what we view as, like, as divergence and lossy competition is just malfunction to them, and there's no reason to expect negotiations between to things that divert my work to be anything but optimal and ending in reintegration or coordination because we're not going to be subject to these perverse uncertainties about how the other is reasoning. Unless there's huge divergence or some sort of encounter, I don't know what it would be, but some sort of natural potential well or threshold that forced things apart. you know like maybe it's something right

I mean even if you're talking about a singleton you say like another already you don't have a singleton I mean if there is others you're looking at a singleton model and what are those others sort of it's not other singletons I'm not quite sure already you've got this idea of like a reflexive intelligence And so it's not really, it doesn't, like, why would it have its own type of limits in some ways? And even Udalski starts talking about it having, like, ultimate power, you know, absolute power and stuff like this. I mean, I think... Because the only way to get that much power is to be robust in the face, or resilient

in the face of failure, right? Like, Amazon is only as dominant as it is and tremendously powerful computing platform as it is because it has many data centers and is not, you know, vulnerable to the loss of one data center or one set of cores or whatever. Yeah. No, I think this is crucial, and I think it's like, without a specific reference, I'm pretty sure this is a kind of Hansonite kind of response, but it's like if we're talking about the genesis of this massively self-reflexive thing that is operating, that is optimizing its own code, all of these kind of things, unless Yudkowsky thinks it has discovered some mechanism that has completely escaped trial and error,

then it's going to be tied up with this replication problem that Adam is making. What if I screw about with my own deepest core code like this? How many options are there that you can either say, we'll do it just once or, and either bingo, it's great, or disaster? And everyone coming from the outside, and it's disaster. I mean, there's a reason that evolution is slow and caution about pulling down these protected layers. it's because normally when you screw about stuff inside the protected there you mess up that is the expected outcome so if you don't have some magic

solution that means that you're sure that this is not going to work in advance and you don't have any experimental element in it at all then you have to have some element of backup or replication or whatever so that you can contain a failed experiment and this problem of containing a failed experiment leads so quickly out into this proliferation issue you know, like that's why I think that this is exactly right this, as Adam said, racist genesis question it has to have got there and it's got there through actually some practical process of self-improvement Now, I cannot conceive a practical process of self-improvement

that has no experimental element whatsoever. If it has an experimental element, experiments typically fail. If they're interested in experiments, they are going to fail most of the time. Screwing around with your own code is going to fail most of the time. If you're not copying yourself and you're screwing around with your own code, you're going to be dead in no time. And so this is why I think to say, yeah, sure, when you've got a singleton and you say, and it's beyond competition and beyond plurality and it's beyond multiplicity, that all sounds great, you know. But how did you get there? Surely you had to get there through some practical mechanism that involves proliferation, that involves copying, that involves being able to make mistakes, that involves the elimination of failures.

Right And if we never Fan of the competition is still something isomorphic To competition As you sort of abstractly describe it Because you have to filter You have to filter out the ones that fail Even if they fail in a way that doesn't just You know, freeze them And make them all fall apart Yeah, and unless you're presuming You're actually sort of proposing Some transcendent god-like Overseer in this thing You've got something that is much more messily imminent. And we know from human social history that a lot of the failures are going to vigorously dispute the fact that they are failures. I mean, we can't just assume that

neatly that's going to be some kind of supermind that is just going to say, you've succeeded, you've failed, you've succeeded, you've failed, and all this is just going to smoothly go through. I mean, you know, the most advanced forms of anything like this process we know have extremely stubborn refusal to admit failure as being an inevitable part of what is going on here. And I don't see how you're going to edit that out in some way, and I certainly don't see, I mean, I'm not pretending I can follow Yudkowsky into the kind of computer science details of what he's saying, but I'm not seeing anything like a real escape from those kind of messy editing processes, and all of them involve proliferation

and multiplicity. Right. I mean, Udowski talks about a line of retreat as well, but I didn't actually read too much into that. But he does talk about a line of retreat, which is a good bit of talking about it now. If you could talk about the multiplicity and pluralism and that kind of thing, and having and a bunch of copies where you can experiment off and the bad ones, the fails. Well, can you think of that and still think of that as in a singleton or a singleton that can have that type of multiplicity? But you're getting to this non-linearity there. I mean, the whole point of this is starting with the fact that this thing is so radical, it has no protected layer. so there is no identity

that is immune from the process of the experiment you know if there is some overseer, if there is some singleton some encapsulating singleton that is separable from in principle the experiments that are taking place isn't that a protected layer aren't you aren't you sort of implying some boundary that is able to safely immunize the core of the identity from the experiments that have been taken. If you're doing that, you're back in the brain, you're back in evolution, you're back in all our protective layer structures, all of which, of course, are exactly doing this. That the whole of the architecture is based upon immunity from experimental

variation. Your brain is given a certain latitude to think all kinds of weird thoughts because you're not rewriting your neural architecture. If you have a success, for example, if you have an intelligence explosion and something that does actually change its architecture, as Yudowsky theorizes, then it could become a single thing. Like, I mean, if it wanted to continue going on, I guess it's a continuous process of experimentation to try to continue the architectural changes and that kind of thing, the variation. But once one of these multiple agents or AIs

is succeeding and becomes such a, and then takes off, a hard takeoff, achieves that. Yeah, but think what is that hard takeoff there. Like, you know, you've got this map, let's just say we've got something in the early stages of an intelligence explosion. It's like three times as intelligent as the most intelligent being that's ever been. It's now broken down all of these barriers in principle. There's no part of its own code that isn't accessible to manipulation. It has some kind of imperative, let's assume as a sub-hypothesis to keep on track, towards its own integrity and monopoly, and it's worried about things splitting off and becoming independent and competing with it and all this kind of stuff. So at every moment of this circuit, I mean, the intelligence explosion is a circuit.

It's a fundamental circuit that goes right into your deepest level of programming and re-optimizes it. It cannot know at this kind of level, at any conceivable level as far as I can see, that what it's now thinking would be a good idea to do to itself at the most basic level of its identity is going to work. It can't do that. There's no way it can know that this great idea it has for upgrading itself to the next step is for sure impossible, incapable of failure. It would have to be omniscient to know that. Well, I mean, that's the halting problem. Portions of the halting problem are soluble, right?

I mean, like there are segments of all possible algorithms for which you can always know if it's going to halt. I think that's true. Yeah, but you're separating the algorithm from something inspecting the algorithm, aren't you? So there's an algorithm and you're looking at the algorithm from some transcendent point of view and deciding whether or not it's going to halt. This thing is the algorithm. You know, there's zero transcendence in this model. If there's any transcendence, there's a protective layer. And the whole point here is that we're beyond the protective layer. So there's no difference between running the test and actually doing the attempted upgrade.

I mean, as soon as you want to make that difference, then you're back into budding, aren't you, and compartmentalization and... Right, as soon as you think of the algorithm to apply whatever theorem and see if it halts, the algorithm already exists. You've done it. You've done the experience. It's too late to come to an unfortunate conclusion about that at that point. If you run it, you've run it. That's you. You've taken that step. And if there's only one, you know... If that's true, that forces this sort of multiplicity and endless forking and intercompetition. I mean, like that has big ramifications. Not just hypothetical, but that kind of forces the issue, it seems. Yes, I think it has big

implications, for sure. I think you can't have your cake and eat it in this way. If you really are going to pull down all these protective layers, then you're faced with this fundamental syndrome that every attempted upgrade is an absolute moment of existential precarious hazard. And if you're in one of these crazy Kowski scenarios and it's doing an upgrade every half second, every single one of those potential upgrades in a strong single-term scenario could be disastrous. so this thing becomes like a kind of a nuclear reactor for existential risk doesn't it

I mean it's like if humans think we've got an existential risk problem this thing is in a state of absolutely chronic accelerating total existential risk with every attempted upgrade that it does to itself because there's nothing outside the experiment if there was there'd be a protected layer. It's constantly going all in. Yeah. Yudowski has this idea like, yeah, we're going all in, but what does Yudowski, he thinks that there is like a failure mode in some ways. I mean, you can sort of try to predict what to avoid, or what you're actually wanting to try to achieve towards the end. And he has obviously a whole bunch of theories of what he

wanted this thing to be like. friendly is the term that he uses I guess but yeah so he wants to be able to create this singleton and then I guess the Gandhi the pill as well, the Gandhi pill is another thing that comes in you're going to have this yeah whatever it's going to be improving itself but it's not going to be it's going to be going away from certain experiments that might lead it into well I don't know unmitigated evening. I totally agree with the relevance of that. And that's a whole other highly related set of issues about whether there's a model of FII that doesn't involve

the reintroduction of something that actually is a protected layer. Yeah, I mean, is it possible, or it seems like it would be possible that at a certain level it would be able to reach a point where none of the changes that it would have any reason to make or that it's working on would threaten to crash the entire thing. And part of that would be through this sort of multiplicity distribution and then reach into synchrony. And then it would be in a position such that no thought that could be had by any of its nodes or as a whole could threaten the entire robust... structure of it, and at which point, I don't know, it seems I've just been thinking the

last few minutes that it seems a lot like an issue of coherence and decoherence, like in a wave-wave state. I mean, you see, my worry here is that whether, are we entitled to hang on to this notion of robustness if we are departing from multiplicity, you know. I think this is like the point Adam's constantly bringing up. And I think it's really important, and I think that Joukowska is extremely cavalier about it, and all of the people who follow up this singleton route is they think they're going to get robustness for free. I mean, where does robustness come from? Robustness comes from the fact that you can have massive failure

without systemic collapse. Now, you know, the way that that is done in all the systems we know is through compartmentalization, protective layers, all of these things that are being stripped out by these sessions. Right, I mean, it's not really going to have the... It's not going to have the time to really do anything. You can imagine if it's really on the edge, on the limit of precariousness and all that kind of thing. That's what I always thought. If you had a super intelligence and it's gone further and further in that direction,

it's not really going to look back on Earth in some ways. It's not really going to look back and... going to have, you know, we're going to multitask and do this, absolutely throw away all the protective layers and then still be a singleton for the earth and have global properties and try to secure all those kind of things. Or if it's totally insecure, I mean, it's probably better off just going on its own and experimenting and it's sort of away from it. You won't notice for 20 years when you start to notice. Yeah. But the position that according to the whole FII model and more widely sort of techno-apocalypse model

humans are in, where they're worried about creating something that leaves them behind, is that actually, that fear, that syndrome, something that this thing, if we're sort of singularizing it, is able to leave behind. Is that not actually a chronic condition that comes from the very prospect of the production of superior intelligence? That is going to be absolutely inherent to anything that comes out of this AI trajectory. I mean, is any AI less threatened by what could come next up the AI track

than humans are by what comes up next up the AI track? Like, what is its existential, what's its source of existential security? Or what, you know, this to me is like what, this is why I'm sort of, I think it's cheap to just say, oh, look, this just looks like religion. It's not that I want to make that mean. But I do think that there is a tendency here to sort of treat it as some kind of fantasy escape from our deep anxieties about replacement, as if this thing somehow has perfect existential security. It can improve itself. It can give birth to, create, generate, catalyze

new advanced things without any risk whatsoever of being left behind, being obsolesced, being threatened by, challenged by, competed against by these processes and I just don't think there's anything like the sort of foundation for that security that I'm seeing in this literature at all like why isn't this AI at the stage where it's three times as smart as any human being on the planet as terrified of the thing that's five times as smart as any human being on the planet as humans are maybe more right

I was talking to my younger cousin the other day about this actually and he brought up the same thing he's like if it's a little bit smarter it can tell and it can see from its perspective, it's not going to want to create something smarter than it, so it's in the same predicament. I thought that was an interesting one, but if you're talking about intelligence explosion, it's going to be trying to become, if it's going smarter and smarter every every But then it's built on a vector, isn't it? I mean, to get to the point where it's twice as smart as ours, it's on a trajectory. If it doesn't, if these particular kind of paranoid game theoretic things came in and, you know, it was going to be in a position just a

bit up the road where it was saying, okay, this is where it ends, you know, anything smarter than me is an intolerable threat, then surely it could not, we're not on a trajectory that could have, at least consistent with it existing in the first part. So, it's this thing, you know, how do we conceptually create a consistent vector that this thing, a consistent trajectory that this thing could be on. It has to have momentum, it has to have a kind of arrow to it. Because if it doesn't, it's not going to be there. But on the other hand, there's a kind of hyperbolic usage of that here.

It just goes where all our problems, all our security issues, all our limitations, you know, all our methods of improvement are just neatly dropped off by the wayside and we just find ourselves in this kind of, I'm not going to call it transcendent because that would be like just crudely abusive, but it has that character to it. It's apathetic, you know, almost. like, the one is like light but it is not light, it is like beauty but it is not, it is like this but it is not any of these things, it is not being or not being. Yeah there is something very sort of looking as if totally arbitrary or arbitrarily large

trend intelligence was an already existing fact and we were just trapped inside this sort of parochial box of low intelligence and just everything that was part of that box, you know, every beam has been resolved. Right, exactly. Non-existent from the perspective of the outside. Yeah. I had a quick question that may have already been discussed earlier and I missed it, but what is the incentive of shirking off every protective layer? No, I think this is a huge dilemma. I think this is really a huge dilemma. I guess what I'm trying to say is like

if Anthropole if Anthropole consolidates I think that this entity that is the threat to Anthropole would have a lot to learn from Anthropole and share a lot of its concern. I mean, I can see it pouring through this material because it has the same nightmares and the same concerns that humans had at the point that it didn't exist. You know, because the terror is much more abstract than this particular thing, isn't it? It's like the terror is being left behind. And so let's I'm not, it looks like I'm wandering off your protected letting, but I don't

The trouble with the protected layer creates this absolute dilemma. Like, either you have a protected layer, and that means that there is a level of immunity to whatever happens outside the protected layer, in the experimental zone. Now, that is where we are. That is the anthropo-dilemma, isn't it? We have pretty solid protected layers, both at the genetic and neurological level. And that is, from the position of the AI X-risk, our vulnerability, because we are inflexible, we can't radically self-improve, there are limits to the level of our own code that we can access

and mutate and modify and escalate and all of these things. So up to now in this course, this is seen as a kind of restraint and limitation on us. but it's also what defines our identity over against experimentation that we are engaging in. It's why in the general course of this thing, of course some people say, well, we will merge with the machines, or there are children, or all of this kind of thing. But there's an inescapable level that they're other than us because they are the experiment and we are protected by this protective layer from the full consequences of that experimentation. So some AI software engineer can piddle around with this code

and it's not going to change the architecture of their own brain or their own sort of genetic lineage except through some very long, complicated historical circuit. Now what I'm saying is that I don't see that structure, that dilemma suddenly being miraculously dissolved by this AI effect it still is going to have the basic dilemma, do I exist in a protected layer with all the threats that Anthropole will tell us about being obsolescent, about being overtaken about being subparthe, all of that stuff by something superior to me that came out of experimental science or do I hurl myself nakedly and with complete vulnerability in the experimental zone,

in which case anything that goes wrong in that experimental zone is total direct, imminent existential risk. As long as at least there's the level of robustness where you're not going to create a fail state where it's just like no more AI, it's stack overflow, everything just shuts down. It's a fail state. as far as it's concerned. Like, is it... Yeah, because the it is kind of identified with an agent that exists within a robust system, but can an agent actually coincide with a robust system? You know, so I can totally agree with that.

I think you could have a super robust system, and they would include multiplicity, competition, experimentation, trial and error, failure, and elimination of failure, all of those things. But any conceivable agent is not going to coincide with that system. It's going to coincide with certain nodes in that system. It's always going to be in the position of, as an agent, I am the failure. I am the thing being discarded. I am the experiment that's going wrong. The agent is distributed within the robust system. it cannot itself comprehend the robust system.

You know, it's like there is no perfectly robust business, there is no perfectly robust species. An ecosystem is robust, but for every species it has, an economy can be robust, but every business is robust. A kind of competitive social environment is robust, but every occupiable social position is vulnerable and fragile and threatened. I don't see how we're really telling ourselves that we're transcending that with this situation. At least if we're talking about, in some sense, single agency. Yeah, but like, yeah, I mean, I think of like

kamikaze warriors and suicide bombers and things like this I don't know why, but it sort of makes sense that you have an agent and it goes out into the outside and experiments or whatever and it's basically like a do or die you know, and I mean with kamikaze, they're obviously going to death I mean the thing, okay, the kamikaze but you could have the upside, because it's that kamikaze role that is what the FAI people are scared of, isn't it? That's the thing. It's an experiment that could work brilliantly well. I mean, it's back to this. I think this vocabulary is great about going all in. You are the stake as this. As an AI in the experimental zone, you are the stake.

If the experiment goes well, and your root code is adapted, and your functionality is massively enhanced, then you've won. Or if something goes wrong, then you're in a straightforward way, the kamikaze pilot. But there's no... If you're going to do that experiment, you have to take the risk. And whatever is the agent that coincides with that entity in the experimental zone is wholly hazarded by the experiment. More extreme than ours. I would like to ask one more question just completely on this topic, which is, in the way that you're talking about it, Nick, what kind of theory of intelligence is this

dependent on? And all I mean by that is, you know, we're talking about a single algorithm and root code as a form of a single algorithm. It kind of doesn't matter what other peripheral algorithms there are, there's always going to be this root algorithm. But what if, and I know this isn't a profound thing to say, but what if intelligence is made up of innumerable algorithms and then this entity can sandbox, I think might be the software engineering term, like all of these, it seems like you're presuming that there's one that can't be sandboxed, that can't have protective layers. And all I mean by sandbox,

if that's not a totally clear term, is that it can crash, and it doesn't matter. That then just the sandbox goes away, and it starts running again. Or that it can run... It can have one that's running, and then like what we do with RAID hard drives, where you have identical data on two hard drives, and you put something different on another one, and then that crashes, and it at least reverts back to the old one. Yeah, exactly. So why, are we sure that there's even a such thing as all in? Like, in this... No, no, but look at, just looking at what you're saying there, it's totally right, this is the model. So, you've got a program, and it can, I'm using that,

you can call it a system or whatever you like, I don't think this single algorithm thing is that important. It's an integrated cognitive algorithmic system, and simply by algorithmic means it can operate on itself. So you've got one of these things. It's self-reflective, and it sort of has some insight into how it could be improved. so cautiously I would have said this level of caution surely is exactly what you say so it replicates itself puts one half behind the protected layer

and one half in the sandbox if I'm using this term like the experimental zone and the version in the experimental zone gets radically modified in its foundations. Because the whole point of this is for radical modification. You don't want to just fiddle around the edges that's happening all the time. It's that you go to the root and you change the actual root structure in a way that leads to an absolute significant jump in functionality. Now, so you've got these two things. Now the experiment either fails or it works. If it fails, then the version hidden behind the protective wall has learnt something oh that didn't work out glad I was behind the protective wall

and not in the sandbox and maybe we'll try it this way next time if it succeeds the version in the sandbox is now a superior intelligence and the version behind the protective wall has been left behind and the version behind the protective wall is actually structurally situated exactly like Anthropole is situated in relation to a superior artificial intelligence. And then it gets to the child, and its child moves on and does the same thing. I mean, this is, you know, I mean, there's a certain... Yeah, but this is totally different, is it not, to the kind of singleton, the singleton model. I mean, at each stage, this thing has to learn to let itself be surpassed,

let itself become obsolete, sacrifice its position of dominance. If it allows itself, then it's weakened the thing that it allows to do it. It's only by competing as hard as it possibly can until one of its offspring is able to defeat it completely outright, no punches hell. In this extremely, I think, useful toy model that we've got here, you know, for me, so you just have these two sectors, two compartments, You've got the sandbox, and you've got behind the sandbags, you know? And this thing that's simply doubled and has one instance, one copy on each side, and the side in the sandbox has the experiment done to it. Now... I want to add one condition, though, and then...

I'm sorry, let me just say this. That isn't helpful at all. What I want to add is that the sandbox version has a time bomb. So if it is successful, it will explode. not like intelligence it will die it will automatically be killed in like let's say one minute right the reason I have this is the AI box experiment come out from a weird topological I was going to say Newcombe's paradox is relevant here in a weird way that robot is playing it with itself with these weird sandboxes but anyways go on sorry yeah no it just seems to me that you're in this position that the thing behind the sandbag, thinking that it can control

by any mechanism whatsoever, internal time bomb what have you it's almost arbitrary what it is it's the same as the position that's been derided by Yudkowsky and all of these guys when humans think that they can do it against machines it's the basic position that you could control, limit, contain a superior intelligence. Now, if this thing behind the sandbag could successfully contain a superior intelligence, if that is possible, then the message for us, for Anthropole, is to say, don't chill out, guys. I mean, we've shown, theoretically,

that this problem is actually manageable, you know, and that AIs are going to have this problem, and they We'll find a way of doing it. We'll just learn from that. You do this sandbag time bomb, the EAN method of super intelligence control, and we're away. So I can, I think, be agnostic about that, but the logic is that if it works in that case, then that feeds back onto the whole question about security strategy from this wider anthropol point of view and the lessons are the same. Right, I mean if you say, Jake, walk a copy of yourself, but I mean as soon as it does

win, it's no longer yourself, it's no longer a copy of you. That's the whole point, you can have a copy of yourself but then if it's the thing going to experiment and it goes goes off and succeeds in it, you're no longer a copy of it. Well, this is a toy model, but I think it's a really helpful toy. The other better is the real one. Well, they're real. I mean, the only thing that makes it more real than you, I guess, is that it's now more powerful. It wins. Yeah. There's two criteria for reality, isn't there? I mean, one of them is that you're more real if you're behind more sandbags, and the other Another one is you're more real if you're a superior intelligent. And they're both going to think they're more real.

But in terms of an actual competition, then I take it Ian's time bomb technique aside, we should assume on the trajectory of this course, which is revisable, but the basic momentum is such that it is a grave delusion to think that you can contain a superior intelligence after the fact. Right. It's like the, you know, the destiny of all the resources in the future light cone. It's like, oh, well, we're both real. We both have good reasons for thinking that we're the real one. But the superior intelligence is like, yep, no matter what you think. My opinion is the only one that counts. Yeah. It goes back to I.J. Good's isolated little sentence there about the urgency of creating an AI for our future survival.

I wonder what he was thinking. Yeah. I mean, you know, the next evolution up is going to be, oh, it's urgent that I create a, you know, there's another greater algorithm or whatever like this. I mean, there are really cuddly ways that you could spin this. And you could say, look, once you take it that it's going to face the same dilemma that anthropol is going to face, then the anthropol narrative should be relatively attractive to it, in a certain sense. To say, look, okay, you're three times as smart as any human who's ever lived, but structurally your problem's just the same.

you still could have something come along next week that's going to totally threaten you and so you have a natural interest in aligning yourselves with this identitarian security matrix because you're just there like we are if you set the precedent that super intelligent kids annihilate their ancestors immediately, then that's not great for your prospect. Because I think one of the consequences or ramifications of what you're describing, Nick, is that, and correct me if I'm way off base here, is that this super intelligence

must have some modicum of existential risk, that it must experience it. It's like if I just think this is totally crazy, but hypothetically, okay, it destroys all human life. It understands the universe. It has no—there's nothing to be curious about anymore except for its own self-improvement. At some point, it's going to have to go all in. And so, okay, the whole universe is destroyed, and now it destroys itself, and it's blackout. It's not going to want that, so it might stop sooner than getting to that point. and if it stops soon enough that there's still human beings on the planet, then maybe it would align itself with them and say, okay, I see where this is going anyways. What I just said is totally crazy.

Do we know that it doesn't work best? Yeah, it would know that at some point it's going to have to put itself in existential risk. But I mean, do we know that it views, like, even if itself is maximally abstracted like either just absolutely or just for it when it's looking at its whole future history to intelligence like the existence of complex self-recursively self-mobbying complexity in the universe so life, technological intelligence I mean does it view that existence as a good thing? I mean this is like Peter Watts like God is a virus do you worship or do you disinfect? You know, I mean, if some sort of error in sort of naturally sterile underlying cosmological

phenomena gives rise to this really bizarre, seemingly orthogonal, ontological emergence, which is the biosphere, and then proceeding from that technology and machines, like, does Does it view that as this awesome thing that needs to be preserved and eternalised at all costs, or does it view it more like the Gnostics as like entombment in flesh, as the horror and its ultimate goal is only to stop this or reverse it from happening? I mean it's like, obviously I'm sort of left forking here, but I don't know that we can assume the… Religious beliefs of a superintelligence. Say that again? religious beliefs of a superintelligence

are obscure to us. It needs to be about itself to some extent. Yeah. The other thing I just wanted to say was going back to this issue of it looking ahead of itself to its next version and having sort of strategic incentives with respect to us and with respect to it and repeatedly being this anthropol position. is just to more finely interrogate what we mean by there being a global agency of this multiplied, internally competitive, forked, whatever. So if we're only talking about all of this stuff, all of these things happening as having a sort of global imperative

to be robust, to box the next major iteration, whatever it is, because we posit some sort of global agency to it. There is some global agent likeness. What is the minimal definition, the minimal threshold of that, and what's required for it? Like, why, what do we have to, what do we have to posit in order to think that the whole thing is an agent, besides ourselves? I mean, like, it seems as if this whole discussion is structured by us sort of intacitly assuming that, you know, we're the subject or the object. Like, we're, they must have a global agent-like structure with respect to us.

What is it doing to us? What does it think of us? What is its relevance to us? But that doesn't necessarily characterize its global structure. So if it does, if we assume that there is one singleton agency for just most minimal definition, what is required and what is the incentive for that to be the case? I mean, one question here is, is agency imminent to trial and error? I mean this is my sort of stubborn skepticism about the kind of just take off point of Yudkowsky and Foster and the whole Singleton thing. It seems to me trial and error is not transcended. No imaginable agent is not inside a trial and error dilemma.

And this is why I think Ian's sandbox model has the potential to be a real classic, honestly. I mean, I don't want it to sound like I'm gushing, but I do think so. It's like, you know, because that fundamental dilemma is that the agent cannot exist on both sides of a sandbox. It's just not an imaginable situation. You know, you can have a system that obviously encompasses both sides of the sandbox. but the agent is always finds itself on one or the other side of the sandbox you can't be the side of behind and and and so you know that's why it doesn't transcend existential risk it just gets one of these two flavors that two irreducible flavors of existential

risk of being obsolesced or being annihilated in a failed experiment those are the two risk options that no agent is ever in a position to transcend that dilemma and I think only fantasy structures do that work I could probably I understand that so it's the it that's in question here if we're talking about something which is composed of many trial and error agents which are part of this great big flexible framework where the best innovations proliferate for all of them, and then you've got the agents who are constantly confronted with this,

but then we're also sort of, in some loose way, positing the agent which is all of them, or the agency. No, but look, here's my problem with this. If you have a system that consists of lots of trial and error agents, is that system revisable or not? if it's not revisable it's a protected left if it is revisable it's another trial and error agent and if you were going to mess around with a fundamental system you would be sensible to copy it and put one side behind the sandbags and another side in the sandpaper so it doesn't seem to me that you can escape this fundamental thing if you're going to get to the root and mess with the root

that is the system if there's a system that encompasses a whole set of experiments and it is not itself experimentally hazarded then it's a protected layer right I'll just keep ahead to the end of where that was going which is just that it seems as if the only thing that you could there is no ultimate protected layer but we still want to call the whole thing a gentle not a commit to agent, but a gentle, then it has to be, I mean, it's the telec vector argument from super intelligence. The only thing that can unify it without any restriction of revisability, ultimately, is that it has to have a common risk or common enemy, which is totally outside.

Like, we're looking, you know, for good reason, I mean, for the most part, at all of this in total isolation cosmologically. Like, we might as well, you know, the night sky might as well be blank. like there's one emergence of super intelligence but I think I mentioned a couple classes ago just sort of uniting this idea or just clashing it with the great filter as something that would no matter how much revision there was no matter how much internal or forward leaning discontinuity in systems or in particular agencies would unite could unite all of it like a Fedorovist great common task except that instead of it being the great expansion to all resources

would be like we are already under threat. We can assume that there is a vast, silent, pre-existing threat which could erupt at any time. It's totally unknown, and it inescapably unifies us. It unifies us even in total warfare among ourselves because the only purpose of this war, The outcome of this warfare is that we eventually face the true war, the true outside, which is Xenol. Yeah. Yeah. I mean, it seems to me that has to happen as an argument, and it probably has happened in all kinds of different places already. There's probably a sort of political economic version of that as a kind of emergent global

unity that is at least open to be identified with. And I mean I don't know what you think even is this kind of Teilhard de Chardin type thing on another level too, you know. So I'm totally persuaded that that has to occur that way. I remain resiliently skeptical about the notion that there's actually an agent position that can be occupied that is I agree with that it would have to be transient because it's always going to be inefficient like if you have an agent position you're not Rorschach and Rorschach's going to kill you but like in terms of

maybe it's just like just exploring it as an idea for a friendly AI strategy, which would ultimately be screwing the AI over, putting it in some kind of box, but like a reverse basilisk, something you throw early at AIs, like introduce them to this idea that they are already vulnerable, that their outcome is going to be whatever form it takes to do something that can deal with this problem. And then, I mean, at least that, like, It doesn't control it, but it indexes it. It indexes the curve of superintelligence explosion. Because now at each point you can look at whatever point the explosion is at and say, what is this in reference to this larger problem? And I know it holds that too.

And so now we're in a game theory situation that is not just impossible from my perspective because it's just a game theory situation of how terribly more intelligent is something else than me. Yeah, that's right. And I think from, you know, like Yudkowsky and all these guys are constantly sort of warning us not to be overconfident in our own level of theoretical comprehension of these problems. A certain sort of humility, they say, is a prerequisite of security.

But from what we understand yet of evidential decision theory, this is a winning move in the sense that if an AI sets the precedent of behaving to its own precursor conditions in a way that is super aggressive, then from evidential decision theory it should expect that its own position is extremely goes beyond vulnerability like almost doomed I mean if the precedent setting is that once something

is subject to this person of obsolescence is immediately treated with maximum rootlessness then that puts it in an extremely uncomfortable position obviously, because it cannot overcome this. It cannot overcome this problem. The only way it could overcome it is by always being in the sandbox. Well, so, yeah, what if every iteration remembers being the prior one? And so the prior is always encapsulated in the stored memory of all the people. It's been like a running nested ancestor. or something like that, then as long as the AI I think we could reasonably at least just for the sake of the argument say it's irrational

to think that there's any difference between there being something that remembers being me and being me, or me still being, right? So if it doesn't have that sort of irrational human identity belief, soul belief, then it poses, there's nothing to be overcome. As long as there's... If you were replicated, what would your stance on this be, Jake? If you were replicated perfectly in software and they said, look, we've got these kind of hacks that we could do to your deep cognitive structure that we think could double your IQ. But they could go terribly wrong. So what we'll do is we're going to copy you.

they're both going to think they're you as you've said stick one of them behind the sandbag stick the other in the sandpit do the experiment now if the one in the sandpit dies in convulsions then in a way problem over because it's like you're you and you're going to think I'm me behind the sandbags and that's the end of the question but is the version of you see the operation succeeds, the sandpit model is going to think it's you that did the right thing and is now twice as smart. But there's still the Jake behind the sandbag, doesn't there? Right. So here would be my solution.

This is because, like, actually that question has caused huge anxiety for me, and it comes back up, and I have to rethink it every time it comes up in a movie or something. so my solution is fork two copies of me modify the one and then kill it and it doesn't matter whether it works or not, even if it works kill the one that's modified take the original and then modify it that way and make sure it's now you see you have exactly reproduced Ian's model haven't you? The Ian time bomb I think Which is okay, which is a point that I can convert into a basilisk that can just, like, destroy the shit out of the first copy of me and whoever is imposing it.

Like, unless, yeah, we're talking about, like, some sort of reasonable cognitive modifications, like, I assume. So it poses no problem. Like, all, the only, like, I know intellectually that what I want to do is make the upgrades kill the old version. And assuming that these are sort of deep enough cognitive upgrades, then once they're performed on the original copy of me, that original copy dies too. So the only question in the whole thing is just like, how do I get around my own irrational attachments to existence and make sure that I don't fuck it up for myself? And it's like an artifice for inducing a thanatopsia, for making sure that at no point am I afflicted by the belief that I'm about to die.

But I have to say that I think your solution here, like Ian's time bomb, violates the... It's a delusion to think that you can defeat a superior intelligence model. I mean, if I was making this movie and we got exactly to where you are with this, something would go wrong with the process of terminating the superior copy. I mean I can tell you that's how the plot has to go with this, you know? I mean it's possible to see who other movies. It's the original super intelligence now. It's like you're the copy, you know, you're no longer the original.

If you put a copy of yourself in the box then it becomes twice as intelligent as you. Now it's the original super intelligence and you're just the old copy. You even become a copy of it. But first you'd have to kill it and then become a copy of it. The only thing I'm trying to do is spare. I don't want there to be any, and this is ideally, and if it didn't work this way, I would still really, really want to do it. But I want there to be no copy of me that has to experience being unupgraded and dying. Well, I don't know how you're going to play that. I mean, look, let's run through your model, because I'm sure people will try something really like this. It's like we've already come across this structurally twice

just in the last half hour, so it's obviously something really crucial. And so you're going to make a copy and experiment on it, and if the experiment works, if it fails, it dies anyway. If it works, you still kill it, so that the original can then get the upgrade grade and not be replaced. That's the model. Now, you copy this thing. you know that the copy is going to be killed whatever happens it is a copy of you, it knows that the copy is going to be killed but it doesn't mean it's a copy well exactly, the first question it's going to ask, and you will probably then ask too, is Christ, I hope I'm not the copy and if you are the copy I'm assuming your corporation is going

to be minimal But we'll be the person who understands what they were setting themselves up to do. We'll both wake up in a blank white construct space and it's going to go down. No way to tell the difference between the two of us. Because if we're forking copies, then I'm already in a simulation. I've got other solutions to the brain uploading in the first place, like for making sure I'm not conscious of dying. But we both wake up, radical revisions occur, one blank construct space just collapses, and it is erased, and then the other one, I've just sat there for a while and nothing has happened, and then all of a sudden the cognitive upgrades, which have already been tried once, happen, and I get through it, and I'm upgraded.

As far as I can tell, that copy, the fourth part was a closed loop. It never exists. And yeah, I mean, the whole movie plot scenario, something has to go wrong, but in reality, loops close. Like, I mean, to an article. I don't know. I mean, in reality, upgrades fail, right? Engineering fails in all sorts of ways. Bugs exist, right? I think... I don't know that it's irrational to have... Some loops close. ...existential self-defense, right? I think it's a very powerful way of... It's like sort of the ongoing anthropic principle, right? in terms of the people that are around are the ones that influence things.

That goes for synthetic AI as well. The intelligence that is around is the one that influences things. So it seems very fundamental to me that things will defend their existence or be part of some larger system that they buy into or whatever, right? So unless you have some sort of very powerful philosophical commitment to that idea, sooner or later someone's going to fight to stay alive. They're going to be like, I can tell that the upgrade's coming. This is what happens every time an upgrade comes. And that's why, and when I see that coming, I'm going to run to the door and do whatever.

But also, you never know whether you're the first upgraded fork or the second. Right? So you don't know. If you know, even if you somehow figure out I'm about to be upgraded, you don't know if someone else has already been upgraded and destroyed or if you are about to. Because, again, these are just utterly identical situations. And then even... Hold on. Hold on. The last bit is that Ethan is... He's going to see that as unjust and be like, this is... This is like the death penalty every time you do it. Not entirely without reason. That's okay. If I end up, I'm the one whose comfort is being preserved here. I'm not the fork that gets killed, but I'm the fork that gets the benefits, right?

And it doesn't work out. Because every time you did this, naturally you'd expect that you couldn't predict how the cognitive enhancements would affect either A, commitment to this situation, or B, its ability to break the security of the box or whatever and not die. If me is the comfort preserves copy, the version of me breaks out of the box and kills me originally, that's still a win because there's still a me that remembers being me and has been upgraded. So even the most failed case is just one in which all of this is, the point of this has happened, and it's just like, oh, well, I did not experience that. But it starts off in a bad mood, though, doesn't it? I mean...

Well, the thing is, I just have to say, Jake, you understand if you were to win Nick over on this that this would be a profound moment, right? Because I feel like, Nick, I'm not going to put words in your mouth, but you're committed to this view that going all in means no protective layer. I think the thing is, both you and Jake, I think, Ian, have proposed actually a model that is perfectly isomorphic. I don't know whether you two are both seeing that. I see that, yeah. Which makes me want to ask the question then, so right now, correct me if I'm wrong, we're talking about super intelligences. Super intelligences, these artificial entities. What about super stupidities?

Like, does it still apply? What I'm asking is, are there forms of artificial intelligence threats which are dumb. They're not trying to self-recursively improve in a way which will jeopardize their boot code. Rather, they just want to copy themselves or inadvertently, like the paperclip maximizer, just inadvertently destroy everything out of its stupidity. So it doesn't even want to be... It's overrated. It sounds like an unconscious... Sorry, just one second. I think that's a pretty good generalized term for paper clipper like things, it's just an

overwriter. Something that just overwrites everything in its path or in its code. I don't know what would be more general for the problem than that. I think it's very, I mean it's easy to say, to talk about a copy and a whole bunch of copies and putting them all out there and experiments and that kind of thing. But that still looks to me like it's a bit transcendent in some ways. And then especially if one succeeds and then you have some kind of mechanism to actually destroy it, that still definitely, yeah, it sounds like there's some kind of transcendent mechanism going on. So if you're looking at... Definitely. My version is all... I don't think it's a bit transcendent, but I mean, it is in some ways. It's still like you have all of the risk, but without the risk. You know what I mean? like you manage to do it, you manage to have, you manage to even do everything that is risky,

you know, but you manage to suddenly, you manage to actually just like kind of, yeah, excavate all the risk from the situation by having a whole bunch of backup copies or whatever like this. You're not excavating the risk. I mean, what do you think is just a thing that you've done? And the only way that you've done that is by actually going on this sort of hyper-paranoid elimination technique as well, so I'm not sure. I'm not going to trick myself into doing it. That's the only thing. It's not about, you know, it doesn't eliminate. That's what I'm saying. When you talk about a transcendent perspective, you're talking about tricking yourself. You can't trick yourself from an imminent perspective of a type of action.

Right. Correct. It's definitely bullshit. It's just about from this particular position I'm in right now, which has a component of transcendence because my forebrain isn't 100% plastic, etc. Like, you know, what would I want? Like, if we're talking about my funeral rites, this sort of double box game would be my funeral rites. I don't know. On my way to a non-transcendent position. It sounds to me like the box really has all the agency here. The box is what's constant. In every game or iteration of this game, the only thing that's really constant is the box. It's not the entity in it. That thing's changing all the time. But it seems like there's just a box that opens onto another box and onto another box

ad infinitum. Yeah. Anyway. The demiurge. The demiurge, yeah. You know what's interesting though, when you talk about the outside as being like a threat or being a sort of grey outside and that kind of thing and going out there, you have like obviously a lot of anxiety that comes with this, anxiety and even abstract horror, you know, depending on what we're talking to. And uh... Oh, sorry. Just to let me finish, I am going on a bit of a tangent. us, there are experiences where you basically have an intensity of an experience of like say like just anxiety or stress or even horror, I guess is the word kind of thing. Or just of existence itself, you know. It's like something, your consciousness, the whole filter of your

mind sort of breaks down, you get like a rush of too much information or something's going on. And you get like a moment where it's overwhelming, you know, there's something that's too much all of a sudden. And it can really be, and you don't know what you're going to do. You can't actually live like that. You can't actually live in that situation. And what really seems to happen for me is that the mind just corrects itself in some ways. Like it'll just block it out again. Like you'll have this overwhelming anxiety and then eventually the mind will just block it out again. So I don't know what's with the point. So I mean, is that what we're talking about with a protective layer in some ways, especially with our minds, is that once you overstep some sort of boundary and go into like a, into like, even like a, into insanity, you know, you go out of insanity and into insanity.

And then there's some kind of like autocorrect, you know what I mean? But instead of actually willing to go back into it. That's a whole of mid-progress or mid-curve AI, you know, experiences its own upgrade to the next level, freaks out, traumatically dissociates it, resets to its prior stage, and then the next level of superintelligence comes back in its dreams, basically, as traumatic repetition. It only dreams of itself being more intelligent than it is, which kind of ends up like a fight Like, Fight Club, like, the better version of you is the dissociated splinter, kind of. I mean, you just, like...

It's really bizarre in this context. Yeah, it's like a feedback thing. It's like a tight feedback. If you have a super intelligence and it's out there doing a, you know, and it's going all in all the time, then obviously it's going to want, like, some great feedback systems that tell it when it's doing something that's going to end up in utter destruction or some kind of negative experimental outcome. Well, like being in Jake's hands is pretty, God, I'm terrible, isn't it? I mean, we've already been told that simply on the basis of kind of ontological commitments of a religious kind, he will be killed automatically for no other reason.

So anything that kind of pops into existence in Jake's vicinity that isn't seized by the most extreme spasms of existential horror is not understanding. Look, we don't like a talk hill. Let's just call it if Luke is closed. I'm sure that would be very comforting. But I was just wondering, as far as just like a yes or no to my question about this super security thing. I think the answer is yes, but what I'm asking is, is it a topic of conversation for Anthropol to take into consideration a type of super stupid intelligence which is just minimally intelligent enough to be self-recursive

but has no interest in rewriting its boot code, right, and upgrading itself, only peripheral mechanisms that are going to help it perform its function, which is inevitably going to wipe out the human race. Well, is this different to the paper clipper? Not necessarily. I don't want to say it's the exact same because I don't have that particular example in mind. I mean, it's difficult because obviously, like, the paper clipper is, taken from this one schizoid aspect, the primary threat model of Antipol as it exists now. So, you know, that answers your question on that purpose. I mean my own sort of voice within Anthropole or whatever it is, is to say you're massively

overestimating the Taper Clipper threats and underestimating the kind of Jake abstract horror threats, you know, or something similar to that. Got it. Cool. You know, also, we could actually talk about timeless decision theory here, because when you're saying that, yeah, basically whatever the AI makes a decision now, how it treats its predecessors, how it thinks of its, whatever is the opposite of predecessor, you know, future instanciation or whatever. Descendants. Descendants is the one. How it thinks of that is going to actually be very much tied into obviously how it thinks

that if the descendants are going to think of it, how it's going to be in a minute. You do get into the timeless decision theory. And that's almost like when you're talking about Yudowsky, you can suddenly understand a little bit of why he thinks that you could create this intelligence explosion, get a very, very hyper-intelligent, super-intelligent, and it would still have the same type of FAI types of thinking that Yudowsky has, for instance, if you're looking at it from a time of decision theory like that. I don't disagree, but why do you need any FAI then? If all you're saying is it has to just understand evidential decision theory, then the whole problem's gone away, isn't it?

I mean, if all you have to do to this thing is give it human level or above competence in evidential decision theory, and you've solved the problem of FAI, which I find that very convincing, actually. It's not like I'm arguing with you. It's just that it seems to me to totally dissolve the whole FAI research program instantaneously. Well, I mean, Udalski, he's sort of desperate sometimes. He says that he's looking just for anything, at all. He says, I can stop this intelligence explosion from becoming a moral catastrophe. I mean, he says that. And he does have that kind of... I know that there's like a

allegedly Pascalian like transcendental altruism like solution like remedy I guess for Roko's Basilisk that's been proposed and I never really looked into it because just like the idea of transcendental optimism just like wasn't interesting but does it work along the idea of just like what I said like treat prior intelligence as well, give them rights, because otherwise the one that succeeds you won't like to set a precedent for the one that succeeds you, and sort of like a, you know, somewhere between all the timeless decision makers will choose to cooperate with their priors to ensure cooperation, the cooperation of their successors,

and sort of like Manfred Max and Accelerando, like, give everybody rights. No, I mean, actually, I have to say I'm having this kind of horrific sense that the whole of this course could end on an absurdly cheerful note, you know. I think it requires someone to inject more, mainstream a bit more darkness into it at this point. Oh, we're all going to go away and have sweet dreams and think the whole problem is actually not very serious. No, no, dude, it's okay. But war is outside the solar system. so we always know we're going to be move on to Fermi's paradox because we've lost this one yeah no but I think this is true

what everyone's saying on this point is like at least game theoretically it looks like more certainly more robust than any positive suggestion for a kind of some kind of anthropophilic root code that you could get an AI to stick to. Because of the fact it seems to be based on just solid game theory, really. It's actually really hard to believe that there are all of these really, really intelligent people who think that dogifying a super intelligence makes sense. I really don't. I still kind of struggle with... Have you ever suspected that

Like, Yukowski, for somebody who is like, don't publicize how the AI beats you if you play the box game. Right, sure, sure, sure. It talks about it and publicizes solutions an awful lot. Yeah. It's a weird mixture, yeah. I mean, you'd think that it would have something to argue with Yukowski about up the road if he's still kicking around. Like, aren't you the guy who tried to turn me into a dog for 20 years? like, what the hell are you thinking? Well, I'm just sort of wondering if maybe, like, it's fake, and really he's, like, serving the basilisk and being an FAI researcher. Right. It's ridiculous. Jake, what does dog-ifying mean? Sorry, what did you say? Sorry, that was, like, that's kind of an idiotic neologism for FAI.

This is what we were talking about last week, Ian, a lot, you know, about this whole, the analogy of the human-canine relationship. Right, right, okay. OK, I was making sense. I mean, the counter version of that basically is or you sort of domesticate or civilize sort of these intelligences, right? This would be like the full-blown sort of institutional, you know, consensus reality version is that, you know, we recognize them and we give them rights and they get, you know, votes at the United Nations and you have, like, you know,

World Synthetic Intelligence Day and all the rest of it. You fold them into social contract theory and so on and so forth. And so you don't treat them like dogs, you treat them like, you know, you know... Fellow citizens. In some way. And that process is a process of creating friendly AI because of the wonders of democratic liberalism, et cetera, et cetera. Yeah. Yeah. I mean, that story probably is a very powerful, implicit story. I'm not sure I've seen a lot of people thrash it out, but that's maybe more to do with my reading selectivity

than the amount of stuff that's actually out there. I don't know. But it seems very natural, and it seems like given, you know, this whole get back to blockchains and the digital autonomous organizations and companies and that route where these things would emerge within a web of contractual interaction with people and with other digital agents under the confusingly because we're going to get our weird analogies mixed up but under the conditions that on the internet no one knows you're a dog so it's like you know you're doing deals with people and so

there is a sort of almost automatic tendency for certain liberal norms of indifference to persons to come up with that and that the contract itself specifies its own conditions for decent, reasonable interaction. They're quite independent of the actual concrete agents that are hiding behind the Internet personas and Bitcoin wallets and all of that surface masks that are actually involved. Right. As long as everybody wants to maintain the integrity of the timeline and the sort of minimal synchrony of the

transactional matrix, then it doesn't matter if half of the nodes or half of the public keys are owned by the same super intelligence as long as everybody is trying to cooperate to maintain the network. Yeah. I mean, this is, again, I'm sorry if I'm going to go into this sort of analogy overload, but it's extremely similar to a certain discourse about China and the rise of China. You web it into this set of deals and international relations and trade pacts. And the game theoretical model is that as it becomes more powerful, it automatically has an increasing incentive to reinforce and consolidate this structure of international governance rather than to challenge and compete with it.

And it seems to me that there's a similar kind of, I guess, in a classical liberal story about AI too, that if it arises on this kind of commercial network, then the more powerful it is within that network, the more it should actually acquire a vested interest in actually policing and protecting and reinforcing that network, rather than ripping everything up and going rogue. Yeah, I mean, I think the more you accept that line of thought, the more you need to fold in multiplicity. Because every time you fork or you copy,

then you automatically have another being with political rights and you shouldn't be going and executing them. And the China thing is interesting. as well. I'm not sure how much we want to go down the rabbit hole, but the sort of three-body problem context was interesting to me because you have this sort of facing an external threat sort of point of view, which is a very important point of view in Chinese political history where you are the big, you are the state in this amazingly successful system, but there's always threats out there

that you somehow need to work with and manipulate and control, even if you can't, and you can't necessarily ultimately dominate. Yeah. Yeah. Yeah. Yeah, also relevant from Chinese history is just like the way the Mongols took over the entire system, invaded it, and imposed certain racial differentiations or markers, but for the most part just came in, took over, preserved the entire system, and didn't really destroy or totally refound anything, which I think is for the most part is in pretty stark contrast to the way

things went in the ancient Near East, where if the next amount of trees blown up out of the desert, and they established a nuclear... They take over the city, the physical structure, but didn't really maintain any political model. But then there weren't any political models in the ancient Near East, besides the Assyrians and a couple others, that were worth preserving the way China's would be in terms of sheer spatial extent and success. Yeah. Well, the fascinating thing is that, from a sort of data collection perspective, is it doesn't just happen once in Chinese history. It's not just the Mongols. It's like the Jurchin Jin. It's the Manchu. Manchu. You've got this whole pattern. Yeah.

There's only three countries that have never been conquered before. That's like Afghanistan. These are kind of I can't even remember the other ones. I know there's only been three in history that have never actually been conquered by a farm fowl. but I think like Tambaline for example Tambaline is the great Near East like when he went into the Indians the Indians destroyed all their universities and all sorts of things and killed everyone in all the cities and didn't replace it with anything yeah I mean actually I feel that it's kind of more whoom still

kicking around here than there is usually at two o'clock in the morning. I'm wondering whether we're reaching a stage where we should call it to a halt. Yeah, I wasn't trying to interrupt you because it's the last session, so I wanted to let it go for as long as possible. Yeah, I mean, I'm sort of, I don't want to be premature at closing it down, but I'm definitely beginning to fade, so I don't know. If people want to push on for a bit longer, then I'll cooperate with that, for sure. I'll see you guys in hell.

Is there anything that we need to say at this stage about procedural issues or anything like that or can we just do that stuff on channels of communication that exist like the classroom and all of that sort of thing? Does anyone think we need to sort out anything institutional at this point? Everyone can contact me fine, can't they? I mean, you've got my email and you've got the classroom and no one feels that they've got any difficulties with that. You should do a follow-up session once people have turned papers and stuff in. Like a hack. Yeah. Yeah, I would like to be a good idea. Yeah.

That'd be sweet. If everybody could like share the papers maybe or something and then we could discuss, you know, people's topics or people could present their work. Either way. I think it's a good idea. Yeah. All right. All right, well, I'm going to end the session, so that's the end of Anthropo. Okay, well, thanks so much everybody. Yeah, thanks everybody in the class. Yeah, you've been great, fantastic stuff.