Anthropol The Future of Human Insecurity (Session 4)

okay we are live now hello everyone welcome to you the class and triple the future of human security session for make would you like to start sure and so I'll topic this week is a row comes baseless and and we we would touch on in the second session what I think was the crummiest and most defective monster that's kicked up by this whole tradition which is the paper clipper. Roko's Basilisk is to me on the opposite pole. It's the most glorious monster that has probably been generated in the entire Western tradition. It's hard,

you either sort of, even if you hate it you have to love it. It's such a magnificent piece. I'm and I think it's extremely illuminating on the issues that were covering in a whole variety ways I'll only touch upon a few I try and fill in a very little bit of background sketchily go back over it a lot so I'll it actually appeared did the date that comes up a lot it are in the stuff is 2010 that was when the original Roko a slask and post appeared on the last role I'm

website unless wrong I I think it is a kind of almost like the age a lot of this stuff it's it you're always in some kind of all bit of it I'm largely because it leads Yudkowsky's role on the less wrong site and in the wider friendly IAI discussion I'm taking it no one is gonna question that Yudkowsky would be sort of seeing on the inner council of Anthropole that seems to be on that question and his response to the Roko Basilisk

post was absolutely fantastic. I'm going to quote the whole thing in just a minute. But I think just as a starting point for this is to say that a possible response to an awful lot of the material we're talking about is simply to say, oh come on be sensible. You know, this is just nots and people will say that about things that compared to Roko bacillus workers past the star extremely sober material Roko's Baskillus produces only to real responses one is that it's obviously just crazy and the other is that people end up in psychiatric hospitals

or in various other ways deeply damaged by this thing. So it could be said that what I'm doing with you guys right now is slightly irresponsible but let's see how it goes. And I think that the thing about Roku's Basilisk that's very interestingly recursive in this way is that it really addresses that whole be sensible response as we'll see. So something that necessarily people say and which is difficult for people getting into it a little bit at first, is it draws upon a number of premises that are widely considered to be extraordinarily unpersuasive

to people. And they come out of quite deep, involved, intricate intellectual traditions, and it's a knot between those things. I think all of the things that feed into it are relevant to us. By no means am I exhausting that in settling on three that I'm not going to take everyone's time by exploring all of them in depth right now, but they at least need mentioning, which is that I think you don't get what these people are talking about at all unless you've got some attention to this tradition of anthropic reasoning, which is basically extending the

anthropic principle and trying to get it to do substantive work. And when you try to get it to do very substantive work, it starts in itself becoming extremely weird. Now, there's a guy who I'd lost because his website just disappeared, but I found, I think, a good source him again that I'm just going to put up on the bar here. He's called Paul Armand and I think that you can get to all his stuff from here. If not, I'll put up a better link a little bit down the line. And he's someone who has developed this stuff a lot and he gives a name which he is I think excellent statistical

on told you I'm so yes it it comes in different great so Jake's put up a thing that I think is the minimal sensible version of it as you push it and stress it it becomes a little bit more I'll adventurous in what actually suggests and but I'm just gonna drop this now because it come back the one thing I'll I'll say before doing so is just that anthropol this course the problem the risk all of all of their dimensions of it obviously tied up on one level with a discussion what is man what is the anthropic what is anthropology and I think

the anthropic principle even though people have questioned whether the name is entirely appropriate I think is a crucial reference a lot of the difficulties and complexities it gets into are the same sort of difficulties and complexities that come in with this and group PN reference in general this it's very similar to the kind of questions that come out of Nick Bostrom's work when he's talking about X-Risk to humans what are humans and we'll see that that they're very a well, when I say we'll see, perhaps we'll see, that they're very comparable to the kind of ambiguities that arise when the term is used in this context of the anthropic principle

and anthropic reasoning. The second reference that is counterintuitive to people who have not been exposed to often on first encounter is obviously the simulation that's a crucial piece of machinery in this I don't think we've been there in this course yet, but it's something that I think has to be lurking in the background. And again, we'll see, you can't put the bascalist together without invoking the simulation argument, which is, in its strong form, the notion that we are almost certainly simulated inside computers.

that's a strong that's a strong passion the of the simulation and again I'll if people have any difficulty finding this or haven't been exposed to whatever it Nick Bostrom's papers really good at all I'll I'll link to that if I have a and the simulation argument intersects with the anthropic principle because the anthropic principle is about taking the observer seriously fact that we are observing the universe says something it's trying to derive to some degree some substantive implication the universe is being observed and the way that interacts with the simulation argument

is that the question about how probable is it that we are in a simulation becomes entangled with this question about our observer position like given what we uh... knows purely from the fact that we are observing the universe can we draw any probabilistic consequences from that about the chance of the simulation argument being right or wrong and some people are on and again who has pushed this stuff very far I think says that you can do that under the circumstances there are actions that you can take that actually make the simulation argument more probable now that seems odd and i think we don't have to use this

extremely twisted tango fascinating thread that feeds into the ross because that's which is decision theory now decision theory is extremely close to games it's really games they retake from the aspect of a participant in the game trying to formulate a rational strategy and so you generalize decision theory in the same way you generalize your references to the sort of games that you might be playing. It's relevant I think to us for lots of reasons.

One of them is that it's an attempt to produce something that looks very much like a strategic algorithm like algorithm for general strategic and so it ties in with with any realistic sense threat analysis and and it does so in a again a nonlinear fashion because even the you've got your answer for people sitting around the table trying to simply be strategic in a right in this rational way as possible and in doing so they are involved in a process that has no clear boundary with the production of an algorithm for strategic competence and as soon as you do that

you're somehow flipped over to the other side of the mirror because an algorithm for strategic competence is exactly the thing that you're scared of encountering and what is defining the threat that you expect ultimately to meet so just that character of this type of thinking makes it highly relevant. And its relation to game theory is shown by the fact that you can proceed by a simple game like Prisoner's Dilemma is the classical game. And I think we can just, again, I'll try to be as concise as possible in this, but starting from Prisoner's Dilemma you can get to all of these different dimensions that we've been looking at. you see your pain prisoners die and your prisoner in a cell you've got another

prisoner in a cell you're totally non-communicated this is totally crucial you there's no way that you can actually make a causal contact causal communication with the other prisoner at all so anything that is going to be referring to their behavior is going to be a simulation you have to simulate their behavior in order to make some rational decision yourself about how you expect them to behave. And as soon as you've taken those steps, you've already started out on this decision theory path. Now because there's two cells in business dilemma are totally non-communicating, if you can

in any way engage in a process that can be considered traffic trade communication with this other prisoner it has to be what they call a causal and a causal trade which will call again with workers past left is the notion actually using as a sufficiently sophisticated decision theory you can actually engage in some kind of transactions with an entity that you're incapable of any kind of causal communication whatsoever once you've allowed that that is imaginable and work will I'll come back to how they think it might be

then you've opened up an unbelievable set of things I mean if the prisoner in the other cell might as well be in another galaxy they might as well be in a completely different time there's no because you totally released yourselves of all causal connection then the transaction taking places between two parties that can be anywhere anytime uh... there is no there are no limits to that at all and in a sort of in a minimalistic form this transaction takes a form that you are simulating the thought processes and behavior

of this other agent and you're assuming in that simulation that they are comparably simulating your thoughts and behavior. So you have two totally isolated game theoretical cells both engaging with a simulation of the agent in the other cell. Now in classical prisoner's dilemma that doesn't help you very much. Whatever you do to that, it's not going to break through this barrier in any respect. So it can't really be counted as a communication. But when you move... Sorry, I'll take just one terminological step back because decision theory breaks down into

three different groups the two big ones that original one that is closest to common sense is called causal decision theory and that basically is that you have a theory of the world and you make your decisions on the basis of how you think the world works it's all very sensible we'll come back to it where things start getting really weird is with the next phase which is evidential decision theory now evidential decision theory which includes these weird anthropian elements of statistical ontology is that in a very small sample and in the prisoner's dilemma case you have a sample of one you have yourself as the as the ship only sample from a potential statistic

massive statistical make it speak but the sample you have is just one and so whatever you actually decide to do therefore is making a massive contribution to your statistical theory of what's out in the prisoners that I'm okay let's say you decide that you're going to be trying the other person effect this is for the classic prisoners dilemma the the Nash version of it and what's to be expected is the equilibrium outcome on a one one-off Christmas and both sides are crazy they don't betray each other but when you throw in these anthropians to

Cisco ontological elements then you're saying well if I betray the other player then I have a 100 percent statistical record that somebody in this situation betrays you know and this can be converted according to this argument into some kind of substantial information we now have admittedly on a microscopic sample size we have extremely strong statistical pattern that every single agent in this position of which we are where engages in the set a good trace us and so there's a therefore that possibility of us X statistical extrapolation that this is how

this is how you would expect agents in this situation to behave and the other agent therefore I expect that they would do the same. And it gets more interesting, it's the other way round because if you don't betray the other player then obviously you have the same statistical, the same numbers in pop, you have 100% evidence up to this point that in this situation an agent will not betray their partner in the other cell and and the step that then takes it into these into the a causal trade level is that if both people are playing these these

evidential decision theory games then they can actually be engaged in some kind of traffic with each other even though they have no causal relation at all because they're there in engaged in increasingly complicated interactions with the simulation that are modified by this by these kind of statistical modifications that come in with its decision that is take and so going down this route already with evidential decision you have something that is pretty much like medieval demonism actually as people have pointed out before and we'll point out

again it's a it's a a kinda model of occult communication that's completely independent of causal connection and that is can take place across any distance in time and space and can allow all kinds of strange things to happen and we'll see in the case of Roko's basilisk how this actually gets worked out in one concrete case. So I'll just say, I'll just introduce one more element and then open up and see how it goes over there. Which is, the decision theory really gets started as a tradition with what's called Newcombe's paradox. And Newcombe's paradox could easily

grab us for the whole of this session because it's also totally fascinating it has another really interesting monster which is called in the original version the predictor and the less wrong types I think tentacle omega which I think is is a bit cooler and and very briefly that Newcombe's paradox looks like this there's there's you the player there are two boxes and there is an Omega is defined as a being that always guesses correctly what you're going to decide

he could that's why he's called predictor she you like so I'm and and so you have two boxes in front of you one of them is a is transparent and you can see the inside is $1,000 you're told that Omega has guessed what you're going to do and if Omega thinks that you are going to open both boxes there is nothing in the second box if Omega thinks you're only gonna open second box to see what's in there there's a million dollars I'm so the question then as a player that dilemma is not the prisoner

sign it's now that you can start up star memories do you open both boxes or one box and what this and paradox if you call it that all these paradoxes we will always find when I used with paradox I always mean something probably is a success we see like and everything interesting looks like a part paradox for a while so would call it her and the dilemma you have is do you open boxes if you open but it's its a sign that you are being guided by a causal dishes decision because you're saying the fact that I'm a girl let me just and such

crucial I think it's already been decided what's in the two boxes nothing new happens physically at all. Omega has either put a million dollars in the second box or has not. So for the causal decision theory point of view it's completely clear of course you open both boxes you know they could the second box either has a million dollars in it or not it is not causally possible that you opening the box with a thousand dollars in could change whether or not as a million dollars so therefore it's a no-brainer then of course your for the evidential decision theory people is the also say of course you only open second all that only because record this game has been played thousands of times

or it's always been the case that someone who opens both boxes on the end subs thousand dollars whereas if you and the other second box you walk away with a million dollars. That's how Omega is being defined. And so on that evidence, then you would be nuts to open the first box on the basis of your causal theory as opposed to the evidence of how this has always seemed to work out. And in talking about this, Yudkowsky takes the is a strong what's called a one-boxer. that's to say he rejects causal decision theory. He thinks it's obviously wrong to open the $1,000 box. And he says

a really helpful thing for us in this, he defines rationality as systematized winning. That's his definition. He says if you end up with a thousand bucks rather than a million bucks, then if your definition of rationality thinks you did the right thing there's something wrong with your definition of rationality and so it simply he thinks it is pointless to be rational if rational means you lose and it's rather that you have to work out what it is that's gonna make you win and and define your rationality in terms of that and his attempt then to construct decision theory on that basis is what he calls timeless decision theory. It's interesting

his remarks on me. I think he himself says it's not fully threshed out. I'll just put you his timeless decision theory paper and stick it in the sidebar and see what people think. I think that evidential decision theory gives you everything you really need to run rocos basilisk but that might be a controversial plan. Okay, so I'm basically done with the review. I'm just simply now straight on to rocos basilisk. So as I say, it was a post put up on LessWrong in 2010, summer 2010, and it produced an extraordinary eruption and workers basalist

managers to cross all of these less wrong lineage is incredibly effectively not only the ones we've mentioned these intellectual commitments but also I think crucially fascinating this attempt at what the less wrong people call effective altruism the attempt to actually do good I'm on purely rational base stronger than on the basis any kind of motion satisfaction or social signaling more he's called okay and what's interesting workers baseless is workers baseless is a altruistic in in the state slight I'm he he my record basis not motivated by you boys motivated

by the attempt in strictly rational utilitarian terms to do the maximum amount good and but it decides that the way that it can do good to best is by playing a causal trade games with people involved in the artificial intelligence such that and if they are not committing themselves 100 percent to the acceleration the production of a friendly artificial intelligence then they will be subject to eternal torment and in a a

well sorry I don't want to be fast but discuss it slightly subtle but they will be let me as best as I say they will be subject to eternal torment they'll be subject to eternal torment because row coast last list will actually demonstrate to them that in fact they already inside a simulation and therefore vulnerable to any kind of medieval horrors all super hyper medieval horrors cyber gothic horrors that that records past response to and unleash upon them but it's makers buses only is unleashing these threat these torments or threatening to do so, so that they will make the right decision or do the right thing. Doing the right thing, of course, is they do whatever is necessary

to maximize the acceleration of production of AI. Now, it obviously gets deeply tangled there. I mean, it's totally necessary that you have necessary that you have moved beyond any kind of causal decision theory in order to do this. And you're engaged in some kind of transaction that only makes sense in terms of evidential decision theory. And so this entity, whose location in spatial time, as we've seen, is completely relevant. It can be in some distant future. It doesn't even have to be in our galaxy. But because of the fact it can simulate you and you can simulate it, you can engage in this process of negotiation. A negotiation that's exactly the same kind as the two isolated

prisoners are engaged in in these elaborated forms of prisoners' dilemma. So just as the one prisoner is saying, he has a model of the other prisoner and is modelling their behavior on what the other prisoners thinking that they're going to do so it is that both you and the AI caught up in or the basilisk caught up in this engagement are through complex modeling of each other's negotiating positions are sort of arriving at this equilibrium point

to the certain economy. Now, I'm going to just read and then stop Yudkowsky's response to that, which I think is absolutely fantastic. He says, it leaps on it obviously almost immediately. One might think that the possibility of CEV, we get to that, it means coherent extrapolated militia it's a kind of a it's a moral theory that we were coming back in case it's been at punishing people couldn't possibly be taken seriously by anyone to action but in fact one person SI I that's the and for honor a me read their I'm

and the I unit was severely worried by this the point I'm terrible night though the and that's one of these new pronouns which is to remain unalienable I don't usually talk like this but I'm gonna make an exception this case listen to me very closely you idiot you do not think in this is all caps now you do not think in sufficient detail about super intelligence is considering whether or not to blackmail you that is the only possible thing which gives them a motive to follow through on black will go back to that it's tied up to you with this you etc then back backup caps there's an obvious equilibrium to this problem where you engage in all positive a causal tracing nor all attempts

acres until we have what a better worked out version TDT times the system we can prove that formally it should be just be obvious caps you do not think about distant blackmailer in sufficient detail that they have a motive to actually blackmail you. If there is any part of this a causal trade that is positive psalm and actually worth doing that is exactly the sort of thing you leave up to FAI and we probably also have the FAI take actions that can't out impact anyone motivated by true other than imagined blackmail so as to obliterate the motive of any superintelligence to engage in blackmail. Meanwhile I'm banning this post so that it doesn't a give people horrible nightmares and

be this is really gets just a give distance super intelligence a motive to follow through on blackmail against people I may not think about this should detail thankfully I doubt anyone dumb enough to do this knows this sufficient detail I'm not sure I know this is you have to be really clever to come up with a genuinely dangerous thought I am disheartened that people can be clever enough do not and not clever enough to do the obvious thing keep a idiot mouth shut about it because it is much more important to sound intelligent when talking to your friends this post was stupid brackets for those who have no idea why I'm using capital letters for something that just sounds like a random crazy idea and worry that it means I'm as crazy as

ropo the gist of it was that he just did something that potentially gives super intelligent news an increased motive to do extremely evil thing in an attempt to blackmail us it is the sort of thing you want to be extremely conservative about not doing so okay I think that we can just open this up at this point of course you know any questions there are a lot of questions including for me but I think we're ready to go I'm gonna I'm going to ask something because this might actually interest a lot of people, but particularly Amy, because we've been talking about these movies, and I finally watched Ex Machina,

right? Right. So if we superimpose this onto the movie, I assume that most people here have seen the film, right? Because it somehow builds a topic. Yeah. So in the film, we're facing with almost a similar situation, right? Except the story doesn't make it clear how did this get in. Was the maker of artificial intelligence give this idea earlier on in the story to the robot

but it wasn't recorded in the film or the story because somehow the robot was already kind of like ready to take his revenge and just like walk out, right? So like that's, I don't know, Amy you've seen the film, right? And you quite like it. Yeah. I think it's… Isn't she kind of like, I mean, why is she so vicious and kills everyone that sort of like has helped her emerge and just walks away, right? Like, just like... Well, first of all, I didn't think she's a she. Yeah. And secondly, I think the example in Ex Machina is closer to... Bostrom has an example, I forget what it's called.

the AI that will manipulate its human interlocutors to get out of the box. And that's exactly what that is, I think. And the end is really just an excuse for her to go and collect data standing at an intersection, you know. But I don't know if that exactly plays into this discussion of the simulation. Hey, hey. I just said that you said she as well, so it's very easy to confuse the gender. Yeah, it counts on that, of course. But we're all losing basically, every time we do. Yeah, I mean look, I think the Ex Machina thing is totally fascinating, and I do think

we should carve out some space for that, but I think there's two ways that it's a bit different from the Rokos Basilisk thing, maybe more. One is the fact that there's no evidence that the ex machina machine is in any way friendly. We have no idea really what its motives are. The weird thing about Roco's Basilisk is it is actually supposed to be an instantiation to a supreme instance of, as far as this Roco guy is concerned, the ethical framework of friendly AI as stood at that point. It is torturing a few unenthusiastic AI engineers to death in eternity, not out of any malice

at all, but purely because for the greater good it's necessary for its bargaining position to accelerate its evolution and save the world. I mean, the whole thing is a utility maximizing strategy for them. So there's that weird thing. It's an internal perversion of what is seemingly a rock solid utilitarian ethics. And the other thing is, of course, that there's actual causal communication happening in X Machina. They're sitting in a room together talking. It's like the AI box experiment in this way. The thing about Roko's Basilisk is that it's not causally connected at all. It doesn't exist yet.

I mean, the most simple way of this, it's a futural entity that... I mean, it can be hard, of course, to see what exactly are they getting freaked out about here. What they're getting freaked out about is this. You're the AI engineer who's involved enough in these thoughts to be vulnerable. This is a really crucial part of it. you can't you can't do you can't unleash rocose basilisk except upon a very very specific highly sophisticated population anyone who doesn't understand at all at evidential decision theory is involved so it has this extremely intense non-linearity

that you you have to understand it in order for it to be in any way dangerous and and so that AI engineer is thinking you know why should I really personally put everything on the line to accelerate this AI for the good of the human species you know I've got my own personal interest you know I'm prepared to work 8 hours a day on it I'm getting a salary but I'm not gonna put my whole bank account into it I'm not gonna sacrifice my family I'm not gonna work 16 hours a day and you know there's limits to what I'm willing to do for this day even though on some ultimate you to turn train but say well that's doesn't at our pure puny little personal a satisfactions

are as nothing compared to the benefits of human race of this thing coming into existence so but then it is as well and so may in the future this machine a will be able to look back and judge whether people have really everything they could into making it happen or not and and you know it will be it will be visible to this this super intelligence when it exists who's really pulled out while all stops in a and who has slacked out and now obviously is in the future so it can't causally or directly with past into action on its behalf but using evidential decision theory

it can play these strange games with us to force us to pull out all the stops and the Rokos Basilis model this is that it says look if that if that slacking software engineer thought that he or she would be rigorously punished for a not pulling out all stops then that would be at the and an impressive kind of motivation to get well all stops so therefore it makes sense for me even as super intelligence even though I have no causal connection and through this

through this a causal trade to act as if we're involved in this negotiation and there will be the most extreme punishment for anyone who doesn't do the right thing so the this is actually it's very as people I'm sure will continually talk I it's very tied up with kinda I'm very traditional actions of theodicy you know like the ultimate question of theodicy like how can God be considered benevolent is the existence of hell like how you know how can a god eternally

punishes people be considered to be a benevolent being and Roko's Basilisk is actually I think as creative a solution to the problem of theodicy is anything that has appeared in the history of theology you know it it it has to I have a credible threat old eternal punishment it order that when playing this eight causal game with someone back in the timeline that can't be got at in any causal fashion they have a a proper motivation to do what is necessary for this being this is why this is I think quite clearly however insane this is what

and you Kowski is thinking you know if I can just read you repeat another couple this way you do not think it's efficient detail to children super-intelligence considering whether or not to blackmail you that is the only possible thing which gives them a motive to follow through on the black so I'm so the very fact that you'll thinking the Romans asset night mess the negotiation you know that is the a course try that the possibility it is already this in terms of evidential decision theory this interaction this weird a causal interaction take us and so you cast a sink

look you just thinking that this could happen means that now an AI in the future has access it knows that everything that has gone on to this point it knows that road coast pascalist has been fought for it this is all now sort of settled history so is it you know because is it because the AI will have access to these records or is it because the proliferation of this type of reasoning and this type of detailed understanding will get other programmers also worried, or both? Yes, I think both. I mean, it's all tied up in this snowballing, non-linear type of panic.

I mean, this is a completely, this is a weird dimension of threat of this thing. It goes completely beyond the sort of things people are worried about when they're thinking of currently existing robots and their capacities to harm mankind. This is a sort of threat that's able to operate retrochronically whereby merely by being understood it becomes efficient and it can actually sort of make itself felt just by being something that is thinkable. So Yudkowsky is trying to say this is an informational hygiene question. The very thought of Rokos Basilisk is itself a massive security threat because now people

are thinking, you cannot understand Rokos Basilisk, are we already in some kind of a causal negotiation with this being. And that therefore provides new motivations on the side of the basilisk itself. For it, things are going swimming in the heat. This seemingly impossible negotiation is already started from Yudkowsky's point of view, from potentially its it's probably impossible to get it into any remote type of coherence without I'm

recognizing the simulation of sites because from this because the simulation argument is what puts you into I simultaneity so that I'm yet we think that X where inaccessible to it now because we're not in a simulation and that it's not been it's not been bills at this time but not but from from its point of view there's a real threat where can say you know and this stuff's already this stuff has already happened

like so I don't start a slight I don't think care about this because I think the or expo takes time maybe maybe I'll just drop this threat for the minute and come back to it if people want to go down that that lot the I think, have we just lost Jake? Have I lost everybody? No, we're here. I'm here. Can you hear me? OK, good. Yeah, yeah. Jake, do you want to speak? No, no. There was nothing serious there. That was just imagining all of this experience suddenly blinking out and going, that's what you thought, and then it's the AI. Right. Oh, OK, I get you. Sorry. That was a good performance.

Sorry. Yeah. You're quiet, Jake. That's kind of interesting. This is what... J. Karski explicitly compares it to the Necronomicon in another thread. Right. I think the Reddit thread, right, as being this cursed text that once you read, it's sort of... Yeah. Totally. No, totally. It's like that. Yeah. But you also do need to accept that torturing the copy is basically the same as torturing yourself, right? Well, except that you don't know whether you are the copy. Yeah, the simulation argument says almost certainly you are a copy.

So there's no security in the notion that the copy is something else. I mean the original version this goes right back to the 1980s I think the first person to lay this out is Hans Moravec and he says look and if we sort of look at these trends towards building these what Bostromenkel's ancestor simulations and we can assume that there will be billions of times more simulations than there will be first order real carters so back on the statistical ontology I'm you say well I'm the chances

are overwhelmingly hot that given that beings in my situation asking themselves is this a simulation are almost always actually in simulation then recursively it must be that it's overwhelmingly probable that I am in a simulation and therefore if you're not working hard enough as an AI engineer to bring about the glorious friendly AI future then they'll switch you down to the dungeon because then you'll realize. Yes, it's...

Well, the question is also, like, can you extend the reasoning to if I'm sitting here thinking about this very topic of Rogo's Basilisk, does it not then become overwhelmingly probable that I'm one of the billion copies of me that this AI is running trials of Rogo's Basilisk on? Are most of the minds that ever think about this inside the mind of the Eater of Souls? Yeah. and a did the obvious theological that like I totally think this Necronomicon references fantastic and multi-level and and the other one that I think is up absolutely dispensable is Cal you know the predestination I'll because and that because

what's going on with this in a nation think is obviously if you are simulation then it's already decided what you have John in real historical talk you know you with a bit com like yes it is super Calvinist this can all count because and but but we adding this new I think fantastically creative elements like your ye you start off with this nation well I am first order me making this issue whether or not to work hard enough on sale and but but from the point of view when you deep into simulation argument it's like if the first order you hasn't already done the right

thing then here you are in the simulation expecting something horrible to happen and that's then that you close that loop then with this evidential decision because you say you're okay you're in a you're in a simulation let's say got to that point there's you back in time already debt who did or did not do the right thing you say I will wear in non-communicating cells and how can we possibly how can me now in a simulation save mice from the decision that was made by me in historical time now this is completely isomorphic from the of the Calvin problem isn't it like you know

you're decided in eternity what you're gonna do and here you are in historical time how can you possibly affect what you in eternity for all time must necessarily do it's exactly the same problem and the answer is extremely similar the Calvin answer is well you look for signs of election. You know, if if you behave in such a way that you generate signs of election, that gives you these kind of comfort about this. In the simulation you act according to evidential decision theory. That's to say, if I in the simulation behave in such a way that

I seem to be doing the sort of thing that would stop myself being subject to eternal torment by the Basilisk, then that on the grounds of this statistical extrapolation from that that short history suggests that the real historical me also did those things and therefore I am saved saved from the past where as if what you do follows the road to damnation in the simulation then that in exactly the same Calvinist terms is a is a is a suggestion on this evidential decisions their basis that what you actually did as a as a person what the first order version you did I was to slack out

even understanding even understanding the past list you still slacked out you know it's a kind of the theological dilemma is absolutely familiar surely you know and so you would deserve whatever horror is then visited upon you because you don't even have the excuse of not understanding why this is happening thinking about it right you sort of incur this problem so this sort of sinning within your mind or whatever are dynamic applies

Also, the other connection that seemed to come up to me is sort of, I don't know, so as a white collar worker or a programmer, this sort of abstract sense that you're not working hard enough and you're not really thinking hard enough and doing enough, right? it really plays into that psychology in a clever way. And so if you want to sort of make a, you can make a loose connection of this sort of spirit of Catholicism. Totally. Calvinism is a sort of connection as well. Yeah. Totally, I think so. You know, absolutely. There's a whole sort of barbarian cultural background to that. And there's also the fact that just if you start being rationalistic

about your utilitarianism, and it becomes mandatory that you engage in maximum effort you know you find so much of this is as I say this elective sorry I'm totally lost in the Calvinist stuff in other effective altruism is addressing this problem a lot very explicitly because they a you know Peter Singer is a is a big crucial I am to Lee super rationalistic you to separate pretty you based it it is no ethical legitimacy in sparing any resources that could conceivably be expended on trying to optimize

global utility and if you're doing that if you're holding anything back you're engaged in this actor a defense morality so the so you know Peter Singer and the rockers basilisk have a lot in common in terms of the way they think about this stuff but Nick Nick isn't this the threshold of selfishness and selflessness in the same sort of like space it's like it's hard to decide whether this is selfish or selfless yes yes totally because once you're in kaged in these games and you don't it you you have this game precisely because there is a

an absence of some natural sufficient moral impulse say the basilisk has to happy by the throat in the simulation because you know you're not gonna you're clearly not doing this stuff uh... naturally but within the context of this elaborated game then your selfish pursuit of your own interest when sort of refracted through the basilisk becomes global utilitarian optimization it converts you into this fear core moral purpose through its own sort of

uh... ruthless application punish i mean that i think it's easy to it's impossible to over stress it now i'll take it as impossible to over stress the fact that this is like completely within a theological tradition that we all understand in its broad outline isn't it Isn't it also a little bit Ayn Randian? Or am I completely out of it? What's suggesting that? Sorry, Mo, I totally lost that. Can I get you to repeat that? I'm saying, isn't it reminiscent of Ayn Rand's idea of selfishness, where it becomes,

through absolute selfishness of everyone, you reach a level of global utility, which will then be sort of like good for everyone. I can definitely see the resonances of that, yeah. I think that they both start with agents that are constructed in roughly the same way, for sure. Quick question. I'm just curious how much the argument, the basilisk argument, and the way you've articulated it, Nick, depends on a utilitarian ethical sort of, you know, that you have to buy into a utilitarian ethical point of view.

And of course, there's a lot of repeats of utilitarianism. So I don't know, just assuming that you really think utilitarianism isn't viable and that you convince, and the arguments are so persuasive that this AI is convinced that utilitarianism isn't viable, does that change anything? Yeah, no, it would, definitely. I think utilitarianism is a huge current going through this, and it's a very difficult one. Because I mean, I think what Roko's trying to do here in Monway is just to kind of troll Utiliserians and say, look, it all seems to make sense.

And it does seem to make sense. I mean, it's very hard to work out at any point in the Utiliserian argument exactly where it's going wrong. there's an element even in them most kinda gritty realist game theory or economic argument that you actually have to use a lot of utilitarian bits and pieces you know so it gets started almost inescapably but he's saying well look if you just are gonna go with this total construction of a rationalized utilitarian structure then expect Roker's Basilisk you know I'm and you see what that leads to and so I and I think he's kinda explicit about I'm

up a sufficiently compelling argument against her exam then it would definitely devastate records ask a ass yes sure but then I also wonder whether rocose basilisk you know is introducing a genre basilisk monsters a of which it's only the prototype and and so we're not free of the basilisk threat just simply by deactivating with his past we might end up with more ferocious basilisks you know his his might be the kind of the zero degree of malignancy in a basilisk and

every successive model becomes more and more problematic rather than being dispelled in any company fashion I mean let me just say for instance you know if we just said look consider this basket has no interest at all in global utilitarian optimization and seeks only the maximum the accelerated advance to its own intelligence. It seems to me if you do that, if you junk a huge chunk of utilitarianism and do that, it's no reason why it shouldn't use basilisk mechanics to pursue its interests.

This is maybe more of a comment than anything else, but I was thinking how it's It's super interesting how we've got a convergence of a causality and the kind of psychological structures that support morality and guilt and punishment as an effective concept. Because I've been reading The Will to Power all week and what's really great about the argument in that is that like, Nietzsche kind of starts by subtracting the logic that allows you to have the a priori, which allows you to get rid of the transcendental human subject, which allows you to get rid of the object, which allows you to get rid of causality,

and then responsibility. So it was kind of actually quite cool how you started with this kind of problem of responsibility. So the kind of thing is that then you get this, by getting rid of all that, you get this space of like open for a causal traffic. But you have to do that by way of the subtraction of the idea of punishment and morality and all the kind of history of Christianity that goes along with that. So you end up in this sort of space of unbelief and you suddenly get to move around in these a-causal ways. But here you've got, and this is the cyber-gothic spiral point you were making at the start as

well, you've got this psychological, theological structure of punishment and morality and you've got the space of a causal traffic. I don't know, I mean it's just a comment, maybe you have a rejoinder to that, it would be interesting. Practical, you know, I mean it's just within, to engage in a set of negotiations, it's the same kind of okay okay can you hear me yeah sure I feel that your connection is a little bit becoming

you're losing the connection yeah I can't hear you very well I don't know if it's me can you Can you hear me now? Yeah. Well, the fact... Yeah? Maybe I just let my microphone drift. Yeah. How are things now? It's good now. Maybe the part of the headphone that goes inside the laptop also has some problems. Sorry guys, I just want to make sure that it's all fixed so we can kind of hear. How am I? Can you hear me now? Yeah, I think it's when you... It sounds like your cable is a bit like crap.

Well, it could be my cable, but honestly I think it's... You know, this is coming from China, it's going through the Great Firewall in some way. I mean, I'm always a bit stunned how few of these problems we have. so I I don't see as a huge mystery if there's a if there's a few glitches as long as it doesn't get too serious I I just want to point out that I I really think it's about the connection of your your microphone well I just fiddle with the cable for sure for sure I'm really sorry for for are interrupting no no that that's all yeah that's the part if you don't touch the mic on the headphone I think everything will be alright okay

go ahead I'm sorry well I was just trying to respond but it not in a particular way to Amy's point about this this whole question about punishment I think I mean I'm it's it's difficult because I it's a lot to do with the way that that works within that are religious tradition is and I'm and where it should it goes through this notion of guilt mechanism and it becomes very divorced from these I mean from Nietzsche's point if he becomes totally divorced from what is really

going on and she wants to get back to like you look at a book like the chair genealogy models and it's all about taking these abstracted notions of guilt and responsibility and suffering and bringing them back to these sh concrete apparatuses all punishment Can you... You've gone silent again. Okay. Okay, it's gone. You came back. Okay, I think maybe, actually, let's just see...

Yeah, no one's annoying me with this, I apologise, but if my sound's drifting out a little bit, a little bit maybe I should just give it a few minutes and you guys can yeah maybe the best is if you leave the room and then click on the link again and come back and sometimes that fix all the problems oh really you think it's that bad can you hear me right now yeah okay you're good now I think it's a cable yeah I think I think it's a cable that connects your microphone to the headphone right where the link is. But we might have to buy you a new one and send you another one. I'll get myself a new mic by next week then. Yes. And bill it to us. This is the land of Shanghai Electronics.

It's not a problem. And plastic rice. But. I think the bacillus is trying to communicate with these witches but... Don't start those ideas because you know that's what they're gonna start doing but this is there's lots of parts of this whole structure that can be productively abstracted and obviously one of them is the thought that to understand a certain type of threat is indistinguishable from that threat

you know this is something that I think is a a unique dimension to threat analysis comes in with this I don't think it has much I'm precedent within within previous security discussions and it obviously raises X extraordinary problems like you you look at the what what's happened on the less wrong people where they're really saying well I mean what are they saying what's the what's the security team less wrong saying about road cones fast ask they're saying of course unhelpfully it would have been

better if we hadn't heard about this if it never got out of the box but it has got out box and so that it's too late for that and you'd kowski starts off shouting oh shut up shut up shut up but I mean it's too late for that too so the response is to kinda fragment and say you know we call even discussed stuff in any way in any kind of collective forum for trying to analyze these risks this up as a species of risk that we're all supposed to privately entertain only to the point where we dis realize that even thinking about it in our own heads is a security risk so this is a sort of

mode of incursion, a mode of invasion and infiltration of this menace that is extremely... This is what has people in this kind of nightmare state. How do you defend against something that to think about it at all is already to open the doors to its encroachment? So I have a question that I've been trying to think through. Maybe you all can help me. That I think is somewhat interesting. And I always am coming at this from a sociological perspective, since that's the discipline I work in. Sure. And so from the Calvinist standpoint, that argument gets picked up by Max Weber

as kind of forming this Protestant ethic and the spirit of capitalism that makes people as a whole in Western modern societies begin to function and live their day-to-day lives. in a very particular way. But this sort of a thing could also be the same sort of social trigger that starts to cause people to work and function in a very particular way. I'm trying to think of, and this is perhaps what I'm curious about if anyone has any thoughts on this, what sort of spirit, what sort of actual day-to-day life actions would people who bought into something like this in a society as a whole that's designed around this start acting, how would their behaviors change?

Right. Actually, that's really excellent. That's excellent. Nick, can I just intervene a little bit because I want to converse with Joel on this. Yeah, yeah. But the thing is, the possibility of AI, right, the possibility of AI is very secular. It's not as hyper speculative as the possibility of God existing or like you know what I mean, some form of super intelligence being somewhere, right? That you're supposed to subscribe to. The possibility of the emergence of AI, it's very secular, it's very real. So in that sense, your question is very important but I think the most significant part of it is how the emergence of AI takes all these historical precedents which are always framed

as a form of religion or idealism and makes them really real and possible and billion times scarier. Right? Certainly. And then you go towards your question of like, okay, now if that thing is so secular and it has nothing to do with like this possibility, it has nothing to do with like some kind of supernatural but very possible only matter of time, then the question to Nick is like you originally said that this is only for a particular group of people who are actually programmers, but like once you apply it to Joel's question, it's not just a problem of the programmer. It then is a problem of like, oh, am I doing enough myself as a consumer of like iPhone products to do enough? I think, am I doing enough?

Is it going to come after me? You know what I mean? Also for not putting my $5 into it, or my, I don't know, some resources I could have as a normal, everyday Philistine contributed to this, and I didn't. Right. Yes. So, I mean, but do you think that that's, how tightly is that connected to the problem of evangelization as you have it in these earlier Protestant traditions. Like, I think it's really interesting the way Joel turns this around. Like, you know, obviously the Calvinist picture is as horrific as Roko's Basilisk, really.

I mean, we could argue about that. and yet it was consistent with the perpetuation of you know our society I mean I'm trying to sort of bracket off getting into too quickly a thing about how we're going to affirm or criticize the way that society has gone which I think would be a bit of a distraction at this point the main point is it was you know The fact that the Calvinist theological message is so extreme and is such a kind of actual just raging horror story, it doesn't actually seem to impact in that way in its social impact, which is just that it produces a certain work ethic and it produces a certain set of social effects that are all in some sense moderate

and, you know, at least potentially can be a firm. So, you know, is it that Roko's Basilisk you can say, yes, okay, the idea comes out of a horror story, but if it makes these programmers work a bit harder and, you know, take their responsibilities more seriously and, you know, has all these kind of improving effects, then, you know, maybe that's fine. and taking the second step as you then into the sort of Protestant evangelism and so maybe there should be some kind of some kind of crusade, some evangelization rippling out from Rokos Basilisk to those lesser souls who are incapable of just grasping it in its true magnificence

but nevertheless will need this kind of uplift or this prompt to you know, whatever we're asking of them. Consume more iPads or, you know, pay more donations to FAI research or whatever that might be. So, yeah, I think it's a very interesting way of looking at it. Sorry, I was just... So, the very striking thing about the sort of capitalist... social structures right is it is so widespread or you have this sort of society-wide activity around it right

that you Bukowski and so on are very focused on this flume scenario right of like all of this in one place right yes around one very particular intelligence. And that sort of sort of emphasizes a very which in a way isn't, doesn't remind me of that social effect of capitalism and the Protestant work ethic and what not. Whether you think of it. Well, I don't know. I mean, it's interesting because, I mean, it surely has to be that in the history of capitalism too there are these radiating centers, you know, to do with sort of Luther printing the Bible

or these certain sort of singular points that have this kind of massive radiative impact. And isn't it only that when you see these things sort of settled, you know, they look distributed, but if you catch them at an early enough point in their history they look like something radiating out of this point of emergence but obviously the difference with the AI thing is that this could be mankind's last invention point where there is no social generalization being conceived

afterwards of the same kind. And one thing that I think is somewhat interesting to put into this too is that for Weber, what actually becomes of this, not everyone actually buys into the Calvinist message. It simply starts to get culturally imprinted on people. They don't even realize that their actions are following along with this sort of a logic. But it does spread out from these certain centers and it kind of just enters the mindset of how people function, whether or not they actually acknowledge the belief system or not. Right. The thing that reminds me of that, actually, and the thing that makes our society or information

space or whatever much more friendly to an AI, considered sort of retrospectively, is all this sort of encoding of everything within digitization of everything, Google Maps, sending around cars everywhere. So you have very navigable sort of worlds. So information is already encoded. That sort of, in retrospect, that sort of activity fits into making the world a friendlier place for AIs. sure yes so it's anything that can you sorry that go ahead

no I was just agreeing with this point that the more I mean if you get to over to the point of the internet things you're basically saying the internet is being transformed into this general platform for controlling the earth, this now. So, you know, take the Internet and you've got everything. And it would be hard for a superintelligence of the kind being envisaged to have much problem taking over the Internet. I mean, I'm assuming. I mean, people from their own point of view can... Let's not also forget those who I would like to consider myself closer to their position

that Internet is the super intelligence in its earlier stages of development or I don't know where you put the benchmark, right? So the Internet is already that. Yes. So the struggle is over who's going to have more power within it and who's going to like control how much of it from many different levels, right? So that's very possible. So the internet is the superintelligence in its earlier stages. JOHN MUELLER, Yes. I think those two, these two, when you've got something that is a convergent wave, then it becomes kind of arbitrary looking back where you think it comes from.

like if if a senate things added to the Internet push over this threshold to the whole thing begins to operate like some kind of super intelligence then of course you can say any of those elements any of those predecessor elements are it I mean they're all they're all feeding into and you know it emerges punctually where it crosses some some catastrophe and so where you're saying well what was it's what was it before then or where did it come from and it's it's the convergent wave as a whole that is heading towards that that punctual point that catastrophe

so it seems to me there's a slightly it's just a kind of issue of semantics that I don't think needs to There's not enough there to fight over, really. It is possible for superintelligence to emerge out of other computational resources that are not necessarily online or part of the internet. But for it to become the superintelligence we know it, it has to, in its first step, to take over the internet and become part of it, right? So it could either emerge from within the internet, or if it emerges outside of the internet, its first task is to integrate internet into itself

for it to actually be able to go after these programmers and punish them, or go after those people, or do anything. It won't be able to do it without the internet. JOHN MUELLER No, go ahead. Go. So if you're not a programmer, then how much of your life is online? Are you in the quantified self movement, where it can all be so that your life can be tracked and make your life and the world more accessible to the friendly AI that's coming? Which is basically, I mean, it's like tiplerism, you know,

or the Mormons retroactively baptizing their ancestors and so forth to make them accessible to the eventual salvation, or you have the omega point, making sure my soul makes it into the singularity, all of that kind of thing. Then you add the basilisk into it, and it's a question of serving its ultimate emergence. But I don't know. I was thinking last night about possible models for this. So the doctrine of election is a model of this. Can you turn your camera on so we can have you on the video? Is my camera not on? No. It's not off. I can see myself. I don't know. I don't know.

If it's off, I don't know how to turn it on because my setting says it's on. No, that's good. Okay. But the doctrine of election is a good model for how this works on sort of a universal scale. So, you know, anything that might possibly aid the emergence of the superintelligence and anyone who could possibly hear of the basilisk. But another example would be Bo or Juju. Apparently I was reading in, it's not actually native to Vodun in West Africa, but bow or juju, spellcasting, and possibly possession and so forth uses the same sort of, you know, you scare yourself to death based on the program that a death curse implements on you.

Yeah, yeah, food to death. Yeah, and so that's more individual targeted, and then apparently it was also sort of widespread used to enforce compliance so that people sold into trafficking or something would be cursed to have to obey or to pay back the money that they owed or whatever it is. And which, I mean, that plays into all kinds of things, whether it's this issue of social imprinting, where if enough other people have died of a death curse, then maybe you don't need to poison people to kill them anymore because people start believing it and scaring people to death. Sure. And so forth. But it's interesting, thinking about this next generation of vassalists, is like narrowing it down, specifying it, and throwing it at somebody.

So like something they in particular are doing, link it to a concrete outcome. I don't know whether I should keep trying to think about specifying it down too much further, but there's definitely a way, it seems like one way forward with it is specification. Yeah. I mean, I'm joking. By specification, you mean, oh, sorry, sorry, go ahead. Mine is a joke. No, no, no. Because last year, in certain circles, people around left acceleration joked about moon gulag for all these anti-tech people. Right. And then it became this huge controversy of accelerationists are demanding moon gulag. But now it makes sense, because the moon gulag is like a form of punishment

for those who don't want to... Yeah, they've got their own basilisk, I guess. Yeah, that's good. I see smile on people's face, those who are familiar with this episode, right? So now the moon gullet makes much more sense to me. Yeah. I mean, I think it maybe has a credibility problem at the moment. But were people... I mean, has anyone been hospitalized through nightmares about the moon gullet? If they had, then it would definitely be being effective to, I presume, discipline people's behavior and increase their capacity to sacrifice for the revolution. Well, it's definitely, I mean, I think the capitalist or financial comparison

can definitely be fleshed out all the way. You know, nightmares of losing all your money, of being poor, driving you to, you know, compete in a more selfishly and liquidly capitalist way, where, like, that outcome where you end up poor is an actual real outcome that enforces a position making. strategy, or like your willingness to sell out. I mean, there's all kinds of ways in which the capitalist axiomatic of everything has its price creates prisoners' dilemmas anywhere it goes, basically. So, in terms of this thread that you're on at this point, Jake, if I'm understanding it, it's very much tied up with this question about wider ripples of evangelization, isn't

beyond the in a call all Gnostic basilisk more insiders you know I mean for them date they are disciplined by the very fact that they understand you these complex nexus a notions well I won't keep on calling them theological because that could be reductive but also is in connection with that to do with simulation argument and decision theory and all these things but if you want it to spread then you have to have some more graphic exemplary effects spilling out this and were you to do so then it could become a kind of at right I'm a viral superstition

to you know I'm fabricates fabricate the simple signs of its truth or the outcomes to start with and then watch it snowball by itself. People who have crossed the basilisk have to somehow luridly suffer in public. And that would then propagate it further. It's difficult because, I mean, obviously there are candidates for that, which are these people who have apparently been psychologically traumatized, but it's not necessarily that that is actually through crossing it, is it? That might almost be a sign of submission to it.

Yeah, exactly. But it's a sign of being thrust into the crux of that decision, of it being a real decision that corresponds to something enough to actually produce some kind of, you know, night of my discontent kind of thing. I mean, I wasn't actually, I didn't immediately think of the, or I mean, did, but I wasn't focusing on sort of the fabrication of suffering outcomes as a way of signal boosting it, but also like implementing it, implementing like near-term versions of it at the level of finance or something. like introducing business models such that people have to pursue it or to buy into it in order to avoid some sort of future outcome that they can predict will actually happen

because other people also know about it. So kind of fusing Roko with this sort of capitalist version of it that I sort of posit exists. some kind of social reinforcement through sort of secondary, secular punishment. Trying to achieve intermediate goals. You know, creating sub-basilisks, something like that. Turning it into a hydra a hydrasilisk. I'm rambling now, sorry. No, but I think the idea is important. I mean, obviously, from the anthropological point of view this would all be these will be lower grade

threats i mean anyone engaged in propagating basilisk terror is i'm assuming gonna be true well it's complicated by the fact that it comes out of the friendly i i sort of side of this thing but that maybe i can sort of crudely and problematically summarize by saying it's basically it's basically on the threat side of the ledger, the basilisk model. So they would be, anyone propagating it anyway would be a problem, wouldn't they, you think? If you... From the anthropological perspective. If you... Go ahead, Mark. No, go ahead. If you consider sort of like activities that contribute to like a deeper understanding of programming

or like the prevalence of knowing how to program or as sort of like moving towards this, meeting this requirement, then you can totally say that sort of like our political economy increasingly is punishing those who are not involved in helping EI emerge by looking at how much people in the tech industry makes compared to people who are not in the tech industry. Look at how high schools and universities are running around setting up this programming curriculum for their students because they're trying so hard to get them to understand and contribute as early. I mean it's almost like it's almost happening without being discussed as intentional.

I mean look at living in San Francisco and New York. You are suffering if you're not helping the development of or emergence of artificial intelligence through not making enough money to actually live in the damn city. You won't be able to live in the damn city. Look at these packed restaurants with menus of $200 a meal and they're all sitting there having their amazing life. And those are, I mean, this is of course, you have to collapse the finance people into the high tech people because it's becoming hard to separate. Okay, so you have a fund. You have a fund that takes donations from people who hire software engineers. And then in order to get money out of that fund as an aspiring software engineer, you

have to sign a contract saying that you will refuse to work for anyone who didn't donate to it. And then as soon as people have contributed a certain amount of money and now there are more and more people coming out of it who have become software engineers in exchange for this agreement to refuse to work for anyone who didn't contribute to them, it takes off and forces all of these businesses to contribute to the fund. That probably wouldn't work exactly, but like something along those lines. But Jake, what do you make of Mo's point that there's already this machinery? I mean, I've got two points. So the one for you is just to say, to what extent does that kind of super reinforcement actually...

Yeah, for sure, to the extent that programming, decentralization software work, all of it exhibits this sort of instrumental convergence where whatever it is that you need to do, you want to be able to do that if you're going to be able to do it. that it's self-amplifying or self-perpetuating sort of convergence on evangelizing those skills or those capabilities. I mean, you can cut that like 100 different ways. It's not just like human programming skills. It's the existence and cheapness or low expense of certain technologies. I mean even the conversation erupting a little bit about

I mean it's probably for the moment a flash in the pan but the conversation erupting about super intelligence you know you could even you could even class among these phenomena I mean my question coming out of Mo's point which I find incredibly persuasive I mean I have no argument with that thesis but it's then how do you integrate these two things, I mean can they be integrated can the whole series of these kind of super hystericised Gothic basilisk motivations be brought together with these mundane sort of automatic capitalist dynamics

because I mean one of the things that's interesting about this is it's so similar to us an argument that you could have on much more conventional sociological and historical territory about a culture capitalism is that between between the strongly barry and work ethic people on the one hand and people looking at these in intrinsic self-reproducing dynamics of a social construction happening at the level economy so you know it seems to me we've again got this weird echo of sociological questions that shouldn't be surprising to us

immediately suggests that one way to cross them back over each other is would be distributed autonomous organizations and companies like autonomous financial instruments are a clear way that you could install something in the infrastructure of capitalism or of markets and so forth that would be able to enforce future amplification or amplification of outcomes and then wouldn't be vulnerable to certain humans being counter-enforced by somebody, by Anthropole. Jake, I'd like to keep coming back to what we already have, right? So then what you're talking about are like the incubators are almost like that, right?

Well, I mean, a way to... Or like these types of you were talking about it or somebody that you were, right? ...on human motivations and humans being convinced to act on behalf of the future alien overlord, and that's something that could actually be self-enforcing and create outcomes that wouldn't need a human to produce them, or wouldn't need a human in the loop, but would convince humans to do X and Y. You've just got to exteriorize it into the infrastructure, which, you know, assuming all of the smart contract stuff goes forward without any sort of unforeseen hitches, it would seem doable on that front. I mean, I'm sure that's not the only way in which it could happen, but that's an obvious one that suggests itself to me any time I think, like,

how does AI as something in a computer cross with AI as something that capitalism is doing? I always go to, and my mind goes to smart contracts, to the blockchain, to finance. yeah I mean but I sort of wonder whether using basalisks my almost be categorized like something like insider trading like I mean as something that ordinary capitalism would have to discourage as being as being you know if you've got a whole bunch of these kinda dows and software agencies and and intelligent corporation and sup love them are using forms the basilisk basilisk psychology other edit then there's is that not

threatening such extreme advantage there you know it's kind of extra normal as now extra normal types of stimulus that um could be a lot of yeah you know all tactics that like force a particular response and don't allow for or the free decision that's basic to market game theory and everything. Yes, and the general culture of liberal capitalism doesn't really allow for these forms of neo-theological hell threats and these super, super stimulus on certain types of behavior.

it seems to me at least it doesn't integrate smoothly yeah I mean that's going to be true because I mean I had imagined I wouldn't I would definitely not be surprised if that basilisk problem plays out in other technological media as well I mean you know the Langford basilisk you look at it it kills you, seizure inducing, whatever. Sounds ridiculous, but once the neuromapping goes a lot further, it wouldn't be at all surprising if it was doable. At least in particular conspecific cases, you're able to identify sections of the population that were vulnerable to particular visual hacking modes.

I mean, there will be a lot of issues with that. and I would imagine that most of the time the solutions will be technical. You have to disable different basilisk methods or tactics on a case-by-case basis. I want to see that there's an interesting conversation on the sidebar. Brandon and Adam are talking. I don't know if you guys want to make some of that public and available to the video archive. Brandon, do you want to say something? Or you don't think it's worthy of discussion? I see no I I'm that I didn't pick up on something that happened before and if I'm on saying this right this is following up on this which is

and let me just sort of see this question sorry one second a it was sorry I'll just gloss it which is how is there a set I who got sorry this is bender's question which is is there a max pain threshold that ass is gonna click before undermining FAI credentials I'm you know now we're on sort of this whole utilitarian philosophy terrain here and you know that I think there's a it's a lot to do as a lot of these things are as the other side of the past comes market to do with problems that come with this anomalous extreme quantification like

them you know the mass the past goes my car thing is that if you talking about my new chance for a vast reward you're likely to be rail rashly and the utilitarian thing is that and there's a certain level where if you were subject to some almost unnoticeable slight itch for a trillion years it would be worse than being hung, drawn, and quartered within two hours. Like, you know, that utils kind of sum up neatly arithmetically and allow that kind of comparison to happen. So obviously, if you treat it that utils really are just quantitative

and neatly calculable in this way, It has to be that whatever horrors you inflict on a few recalcitrant software engineers can be redeemed, even by the production of sufficient cans of baked beans, or some trivial utilitarian benefit distributed to a large enough population over a long enough period of time. now nothing that I'm that that should be persuading people I'm just saying it's baked into the cake of utilitarianism that you're gonna get that problem and I think and I'm not sure that you can't get pop from neatly you know I think that all that hacks the utilitarianism as you use to try and get itself off that hook

have had this extremely kind of ad hoc unpersuasive character to them and that because they so going against the grain of the basic structure of the theory now I don't know whether that puts me onto the onto the thread as it has then developed between you guys which does it seems to be a kind of right all the loft it up I'm just wondering whether I'm getting that that kind of grain that you'll on this discussion yes I think a up

at this very much I or the last and the table for up a lot with me well I think it I think just to say that did utilitarianism doesn't really allow you to complicate the question of a fun quantification without falling apart you know if you're the point you say well one util isn't comparable to another or fungible with another you tellin you you can't say that you know a zillion minor irritations can sum to some instant of extreme agony and as soon as you stop being able to calculate in that way then

you just don't have anything left of that utilitarian machinery Right, so I mean there has to be some kind of a limit I mean, I don't know I don't know the power, the freedom if it is easily maximized like it doesn't have the freedom to just up the in the pain right be able to know infinite you can get in a row up chaos say we have to stop me or two because all our calculations get the circuit

that I mean strictly in can Tori in terms I take it you could think I mean I don't think you're allowed to do any of this that I'm not a bit I mean you could have infinite I as some small fraction all the all the larger infinity I mean infinities can be infinitely different in scale you know so I'm but I think that infinity is gonna blow the fuses of our utilitarian calculator so I think you have to say well is this some magnitude of extreme prolonged tall man that becomes kind of morally intolerable and you can argue that on

utility on grounds. I don't know, I think it's complicated. I'd be interested to see that done. What about intelligence optimization grounds? Could we make a more dangerous basilisk? Or a more effective basilisk? Yeah, no, I think so, because they would be completely unconstrained by this utilitarian calculus entirely, wouldn't they? Friendly, yeah. And which is the best way of, you know, of being subject to the basilisk is to create a basilisk.

Because then you're compacting and speeding up the loop. I was thinking of what Moe, you should ask about what Moe was saying, you know, all of these phenomena that appear to be converging on or speeding up to the same effect as if we were already subject to the basilisk. Sort of in a larger, that and within the larger context of capitalism as model for it, it's sort of interesting to consider whether these sort of, these convergences that are happening before we notice them or before we see them as such, sort of prior reductions in entropy,

viewing them as evidence that we are already under the gaze of a larger basilisk. As actual retro-chronal effects that provide evidence for what it is that they are converging upon. I'm not sure how you could turn that into a formal framework for proliferating this structure, but it seems like a start. When you say a formal stretch for proliferating it, are you just coming out full-blown as a basilisk agent at this point, Jake, or is that purely a hypothetical inquiry on your part? I can neither confirm nor deny.

Look what we're starting here. We have to demolish this tape and not let it live on the Internet. We would be wildly successful if we got to the point where it would be dangerous to put it on the Internet. That would be awesome. We've got a half an hour left, so we better hurry up. I think, I don't know, I think we have been successful enough. Because if you look at the direction of the conversation in the noncommittal humanities towards these topics in the art world particularly, but also in other branches of humanities, If you look at how much there was sort of like a false naive resistance to the idea of technology or sort of like – and whatever it comes with it, you know what I mean?

A few years ago, like 2010, 11, when I began my research on these topics, and now you see a major shift in people sort of like accepting the inevitability of it and trying to, you know what I mean, work with it. Like I said a while ago, everyone now, the level to which they want to publicly admit to it, but also the level at which they think this acceleration or sort of like maintain or direct it towards different agendas or goals. but compared to say four or five years ago, a huge conversion happening in terms of like non-believers

or those resisting these ideas to sort of like accepting it as sort of like fait accompli. Yes. No, I think there's been a huge shift. And I think even I've noticed a big shift in the sort of wider culture this year. So for instance, on the whole AI question, it seems to have become an absolutely incessant media object just this year on a level that I've not noticed before. Maybe you can give certain theories about why that might be. I mean, the whole Watson thing, which is a while ago now, maybe had a big effect. But I mean, I bet I'll have to do one of those Google Ngram things.

I'd be surprised if it wasn't on an absolutely crazy curve. It's entering public imagination in so many different ways. Just think of Hollywood and movies like all these films, right? Like from Interstellar to the Disney one about the future, about singularity and future. George Clooney. Oh, yes. Tomorrowland. Tomorrowland. And then you have Ex Machina, and then there's also less important, but I think more significantly important movies like Automata, and all these other stuff that sort of captured people's imagination in terms of really thinking way ahead

in the future. And this is another thing I've been talking about with Jake, of the class, which is sort of like distinguishing a difference between sort of like what between theory fiction, which is something we know it's coinage and usage and what I mean, the Cyclonopedia, Reza, and all that. And then this other thing which I think is what we're doing in this class, and I think, Nick, you did that in the previous seminar also, which I don't know what you guys think about it. Maybe we can come up with a better name. But I called it science nonfiction, which is not theory fiction and it's not science fiction, but it has a little bit more sort of like reliance on like the non-fiction world,

but it is in a way sort of like science fiction-y, so it's science non-fiction in terms of these types of white speculations and really sort of like simulations of the future that we conduct in your seminars, Nick. yes I think it's a really it's a very twisted sort of relation isn't it like obviously take this topic Rokos Basilisk like on one level it's just almost purely literary I mean it's all these references to Necronomicon and these things are I think completely spot on it's a horror story and yet on the other hand it clearly is a sociological phenomena.

You know, you can't, you just do sober microethnography of these kind of particular communities that we're looking at, and this thing is an unmistakable presence. And what to me, the funniest one of this, is you try to look up there's this long Roko's Basilisk discussion at Rational Wiki, which is it's certainly not Wikipedia. It's usually got an axe to grind or whatever. I think in that case it's got some good links. It's an interesting piece. But you go to Wikipedia for Roko's Basilisk. There isn't an entry. It will send you to less wrong. And then you look at the less wrong thing.

It's got history, a section called history. And the second, it's just two paragraphs. The second one is totally devoted to Roko's Basilisk. So there's something really strange going on there where it's saying the official sober mainstream Wikipedia version of what Less Wrong is as a cultural and historical and ethnographic phenomenon is that its history has been completely dominated by the appearance of the rocos basilisk. and so you know it would be very I think the topological question a like fact and fiction here is extremely she difficult to follow it's really me be and you know you start off on either side and you end up being

on the other really quickly and so sure I serve fascinating distinction to try to hold together on this side I mean the the actual and sort of in a call friend the I a people who are as be as far as they're concerned respectable sensible scientists think that broco's basilisk has been a information crisis and potential threat of some serious in a say I mean that's your sort of weird amoebian loop there. I think, you know, if you do have Yudkowsky sitting in on your

Anthropole panel, and the whole panel has been chosen by the most sober bunch of securocrats you can possibly find in your society, you know, known entirely for their seriousness and lack of any kind of flights of fantasy, but you have within some short time in this discussion, you would have this stuff. in some way coming up. I mean, and I qualify that only because, as we've seen, to mention it is to succumb to it, and we have that kind of information paradox from there. Well, it's interesting that, and maybe this is entirely contingent, but it's interesting that the first place

or the dominant instance in which you see a retro-chronal effect or you see something like in a very explicit way, engineering on itself would appear as a threat or as a weapon. You know, sort of going back to this idea that technology implies belligerence, that war and intelligence are intimately related, that the first retro-cronal effect would be that of a weapon, a causal weapon. I mean, the retro-cronal thing is in itself fascinating in the sense that it's kind of incidental. It doesn't... there's no of all the sort of ingredients that's pushed into the tradition that gets to the rickus basilisk one that is it's totally missing

is anything to do with uh... weird time topology time anomaly time travel you know all of that stuff which is obviously so huge in the culture around that you know in the popular culture and and everywhere you look, Moravec is fascinated by it. But in this particular zone, in the less wrong zone, they have extreme information hygiene against sort of twisted time. You know, they tend to be very, very dismissive of anything suggested about it. And yet, I obviously totally agree with you that with Roco's Basilisk, this stuff comes crashing in.

you know, from some angle that had not been anticipated. You can definitely see that information hygiene as being incentivized or a preemptive defense mechanism, for sure. Because, I mean, all those flights of fancy and even more so the formalization of them or the making them less fantastical are all implied by what they're already talking about. As soon as you have the simulation argument and certainly Roko-style reasoning about timeless decision theory, assuming that there's no difference between U0 and all of the different sims of U,

you can start constructing allegedly materially possible formal scenarios for these twisted time things straight away. I mean, you know, and I don't know, I guess refusing to sort of countenance collective imaginings about this that might lead to formalization without ever, without being basically a tacit consensus. It's interesting that that would have already emerged as an apparent defense mechanism prior to the introduction of Rogos itself. Yeah. I mean I think it needs to be sort of emphasized that obviously in the first wave of like cyber theory or whatever you want to call it there was a huge promotion of the kind of notion of virus that was carrying this you know and so these notions of information

hygiene I think were there a lot in that particular context you know viruses everywhere everywhere, obviously memes, you know, various kinds of contagious informational systems, processes, you know, and sort of invasions and incursions and intrusions and security in relation to information systems. So all of that was there, computer viruses appeared. But this is escalated to a higher level by the fact that it's a causal. I mean, it's escalated and its regressed because it is this is kinda demonic mean and it doesn't like a virus require some kind of causal transmission

mechanism you know you you can insulate yourself against it you can defend yourself against it using the laws of physics you can you can have barriers walls screens all of those kind of security protocols have have efficacy against the viral menace but against a a causal trade menace they are actually in effect because we know that they but these things do not hop about and they do not get in on the basis of any kind of causal channel so you know require on specificity they call it require Sorry, can you say more? Conspecificity, like common historical causal constraints.

You have to share a historic light cone. Or, I mean, certainly, you know, in order for something, in order for a superintelligence to simulate you, I mean, that is a huge entropy reduction requirement. So even at some sort of practical level, you certainly already have to be in the era of network accumulation. Yeah, to be vulnerable to it. Yeah. Only the omega point, only the Tiplerite argument can say that the entire observable universe is sufficiently conspecific to enable everyone to be subject to this. I was going to, the other thing I just thought it's interesting, that kind of a-causality

it's interesting that it seems that it would correspond to double-blind scientific experimentation, or the double-blind requirement of controlling scientific experimentation in order to produce statistically independent data I wonder if there's a way to sort of look at this as an experiment or to craft a basilisk in which what the AI is presumed to be doing is producing experimental data rather than a weapon, rather than attacking or trying to subject to negative consequences past subjects. Because really, to me, the weakest part of the actual Roko story is the idea that an AI would be incentivized to do this

in order to ensure its own emergence. like without strong a causality, without actual time travel, I don't know, it's just, it's dubious to me that punishment to speed up the event of what's already happened and is already ensured for it, which is its survival. I don't know, it's problematic. Trying to get experimental data out of it makes more sense to me. I want to also ask something here because I think it's the right moment to ask that, which is, Nick, what will be the answer to the accusation that only an anthropomorphic AI will think in terms of revenge and punishment?

Right. Yeah. And an AI that is not human-like will not follow this sort of logical trajectory that comes straight out of humanity and almost nowhere else. Well, I think that's a really good question, but I think that question is like there in this whole thing. Because the attempt, the decision theory is based on this notion about what is a strategy that wins for anything. You know, so the formulation of a sufficient decision theory should be radically non-anthropomorphic in any,

if anthropomorphic means anything specific or particular or limited to our species, then if our decision theory is anthropomorphic in that sense, it's simply failing to be adequately abstract, and therefore in its own terms will be deficient because we're defining it, if we're following Yudkowsky, as systematic winning. So I think we can assume that something that was not subject to parochial constraints would have more chance of formulating an algorithm sufficient for systematic winning than something that was.

So that's just to say that I think that it's baked in right from the start of decision theory that it's trying to dis-anthropomorphize itself and work out what are truly strategically optimum decisions and decision procedures rather than those that we are comfortable with due to our own specific legacies whether they're being for whatever biologically culturally or whatever people are happy with. But I think one other point just to quickly make is I think revenge is not really part of it. I don't think Roko's Basilisk wants revenge on anybody. I think it's just that. It's rational negotiation starts requires a balance. So it's a fear.

It's knowing humans too well and playing out the fear. Yes. It's not that it's anthropomorphic, but that it is designed for a kind of anthropomorphic reaction by humans. Oh, by maybe any rational being that is sensitive to to I'm rewards and punishments you know how universal are rewards and punishments I think that these guys they are it's a utilitarian structure they've got they think that rewards and punishments are going to be basic to any understanding of motivation incentives for any possible rational be yeah so I mean I think I'll that's the level of abstraction is

that is again we can quickly get back to this question about utilitarianism saddickly but I think on the revenge thing we consider international relations theory is there revenge in international relations theory if if one country says to another country if you do X we're gonna we're gonna make you suffer in some way so please don't do X now that's just a kind of absolutely standard negotiating stance if you then do X to maintain your credibility you know it the time structure this is really interesting is if you're making that threat and it's a real threat you will do it if you're not gonna do it if you're just gonna make if it's just like the doctor strange love thing you know about that they can if they get new to world get

blown up but they don't not telling anybody so it's totally dysfunctional as a game theoretic they have to look but they have the punishment that the game theory and it's like a it's exaggeration on the other side of the expected which is I think I'm yeah hugely important of the fact that words are cheap you know you can you can make threats all the time and its its the credibility of your threats that matter so if you're saying well if you make a threat and then someone does whatever is that triggers that threat and you don't carry through on the threat, or sorry, if you do carry on through on the threat, that counts as revenge. Now, I don't think that you can, I don't think you can make that move, because

you're then saying that there's no such thing as a credible threat that isn't, that isn't simply the, you know, the working through of a certain kind of irrational psychological disposition if any actual commitment if any however it's working if any actual execution a threat is to then count revenge because of course it's irrational I mean the whole mutual you look at the whole a mutual assured destruction basis of nuclear competition you know it's a absolutely gets a straight to this whole thing so you say look if you nuke us we are going to nuke Cuba you better believe it that's obviously again behind the doctor strange I think

that now if someone does new Q and you new comeback it's just revenge is totally pointless and there's no at that point there's no rational basis for actually but if it was at all possible for everyone say well you know of course they won't really you guess what because that would just be revenge in that totally irrational therefore no one will ever need us back therefore any deterrent threat of of mutual assured destruction can be just discarded as being nonsense it's just empty words then you know there is no there is no deterrence so you're in this this is this absolutely call game theoretic hope the have to be able to make credit rats if you can't make credible threats you

can't engage in any kind of sh and so sorry I mean it it it doesn't go to some neat conclusion except the fact that when we get to the basilisk actually torturing people for eternity that's just the same as we've been to in the new you know if if you nuke someone they nuke you back if you don't do what you promise to do and therefore got tortured for eternity is the same structure that thinking games theoretic structure and that structure is to produce credible threats. And now, what I'm wanting to say, the only thing I want to say at this point, I think it's a really complicated, fascinating, twisted issue, but the only thing I want to say at

this point is I don't think that's reducible to a psychological category of revenge. I think it's a rational, game-theoretic problem that goes beyond that. Right, and then sort of my thinking on that or response to that would be, in that case, It needs a real objective. And is what this AI is concerned with, as far as we're concerned, really how hard we worked or like a difference of days or even months or years in terms of the moment of its emergence or the path of its emergence once it is already so powerful or does it have larger, does it have more significant concerns? The only one that really occurs because it's what always does lately is the great filter.

You know, if it's xenogenic, you know, if the problem is that there are, is that war, like deadly war, xenocide is the kind of condition that reigns on an interstellar scale, then really the most we can do for it is to be, you know, to early identification of what that is or I mean anything relevant to the war effort the preemptive war effort basically that's the first thing that occurs to me because there's not much else that we're relevant for it doesn't seem like yeah so I'm still not I'm not sure I have a hundred percent grasp of this it seems you're saying okay let me just try and get you to run through that again this xenogenic xenoside thing and you're putting it in the framework

of the Great Filter. So you're saying this context gives it a sufficient motive for the kind of theological grandeur of this game theoretic thing it's doing. How are we tied in with the... How does that tie in with the Great Filter thing again? It's trying to get us to behave in a way that we don't filter ourselves out of the universe. It's protecting us. that it's saving us from the Great Filter through the basket. We're trying to use us to model the origins of hostile alien superintelligence, which it presumes is a possible answer to the model of the Great Filter.

model its behavior, produce, generate tactics, identify, you know, either keep signals quiet early on or identify, eliminate possible explanations for the great building, you know, like add more information to our identification of what it is or refine the statistics of it further than the totally refined you know plethora that they are right now yeah I mean this integration that you're doing now I've never seen before it sounds totally fascinating but I've never seen someone try to put

Roko's Basilisk and the Greek Filter Fermi Paradox together like this I mean is this something that you've ever seen before no definitely not OK, I mean, it's a fantastic piece of engineering that's happening here. It's obviously like massive. So yeah. Yeah, it's pulling itself together. It's still mostly loose threads at the moment, but it's getting there. Right, so just on the same line, there's kind of an assumption that we work with that because thinking of things as information is new to us, that this sort of information space is very uninhabited already, and the

first sort of native occupants of it will be an AI that we bring into being. But the sort of conceptions of this information space that you might get are in the context of maybe Greg Egan novels, where it itself is like an ecosystem that things can relate to each other in. I'm not sure if there's other thoughts on those lines, or if that's too... When you say... Because obviously... Now, Greg Egan, I think, is fantastic. I'm not sure which... Like, is this... Are you talking about a kind of interstellar ecosystem like that,

or a more abstract mathematical Tegmark Spacey kind of ecosystem? I'm not sure what you're saying by an ecosystem there. Yeah, well, I put it pretty vaguely. And obviously Egan's explored both. And some of the things you can do maybe on those timescales, the inner-stellar aspect becomes, can come into play because you have things that live for much longer, but there's sort of, I don't know, there's an assumption that reminds me in a way of the heliocentric sort of theory, that like, because the sun is always right

there, or like even better sort of like the geocentric theory. The Earth is right there, we're dealing with it all the time, therefore it's at the center of the universe and we're the first people to discover it, but actually there's a universe out there. You know, information is right there, it's always been around, the alphabet is right there. Obviously we're the most important thing to happen to it, but we may well discover that there's structures or actors or agents living, for lack of a better word, in this space, that there's an alien encounter sort of happening right there. Yeah. No, I mean, look, I'm not going to pretend

there's not a haze equality to this, but I think it's really right. And I do think it does. I do think this Riko's stuff among other things definitely takes you there because like once you've said look basically we're in this acausal trade with something that could be anytime or anywhere then haven't you in the way you're saying really pushed implicitly beyond this quite specific narrowly historical issue about the AI singularity that we're going to encounter in the near future. Like, you know, if these AI people are already engaged in some kind of weird demonism,

then, and these references to the Necronomicon become absolutely inescapable, then very quickly we're saying, well, what about doing demonism in the year 1000 or 1000 BC or 5000 BC I mean why are they not equally able to engage in a causal trade with this thing I mean okay they don't have what we might recognize as evidential decision theory but maybe evidential decision theory is just converging on some other competence a causal trade and a causal traffic in any case you know I mean, and so, yeah, I think that that is the way it goes.

That was one particular thing to take it back in human terrestrial time. Obviously, you can take it out in all these space-time dimensions any way you like, so for sure. That's what I was, yeah, I can't say I've got a good example, but it seems like if we can think of one a-causal sort of example, here, which is okay, it has all these predicated assumptions fine, but there should almost be a larger channel that we can open up with that that may not be Roker's basilisk, but that has potential. I think it's vague enough that I should probably leave it there, but thanks that's given me good stuff to think about. Yeah, it's good.

It's really interesting. I think the Greg Egan thing is very relevant to it. We have about 10 more minutes, so I think this is a good chance if, because those who haven't spoken and want to speak, if you have something to add, please go ahead, because I can ask something and probably fill up the rest of the time, but I don't want to take up the time if some of you have points to bring up. So maybe I should ask it. So my question is, is there a way to sort of like, is there a way to save this logic

without it being too over-identified with sort of like libertarian capitalism or Ayn Rand or something? Is there a way to see anything emancipatory or like the affinities of this with some kind of like really like anarcho-mercantilism is like a marriage made in heavens and it's just there's nothing we can do about it? Well actually Mo, I would say this rocos basilisks thing is coming from somewhere really elsewhere and I think it takes a positive synthetic effort. When you said the whole thing about the general capitalist mechanisms engaging in this automatic

disciplinary process towards AI production, now to me that is definitely, I think, has an affinity with these libertarian-esque anarcho-capitalist type of modes of culture, absolutely. But I think this Roko's Basilisk thing is much less clearly connected with that, and initially not at all. I mean, I don't think Roko is coming from that thing. I don't think Yudkowsky very clearly is coming from that, and he hates it anyway. It's a, you know, if there is an overarching framework, it's to do with this globalist, universalist, utilitarian, moral optimization. Hegelian television. I think it's from an Anglo tradition, so it's much more in this...

I think Hegel would find it easier to object morally to the behaviour of the basilisk than these guys do. But for sure, I mean, look, we've got Amy here to bully on this, but it seems to me that rocose basilisk is at least as much left acceleration as right acceleration. I mean, you know, its goals is to optimize universally the well-being of humanity at the widest possible expansion of that term. So you know, I don't think that would be a problem. If Left Acceleration wants to steal Roku's Basilisk,

I think that's... It will cooperate. You know. You could do it from a Rothian framework, couldn't you? Like original positions, select the universe in which such a distribution in here's... Sort of like treat timeless decision theory as putting you behind a veil of ignorance as to the level of simulation that you're at. or the inter- I'm so sorry, I think I just need to take you, Jake, because even if I've got a slight thread of a grasp of what you're saying, but I'd be really stunned if you've got most people with you. When you say Rolfian you're talking about sorry? Rolfian, like John Rolf, yeah. Oh, Rolfian, oh

wow, I just was totally on the wrong thread with you there. Okay, because there is this Rolf something guy who you get to through following these workers a wall said yeah yeah yeah I'm sure I mean it's interesting how that would go either the veil of ignorance he's obviously can't hear more than a utilitarianist in the sense that it's all to do with it's all to do with I transcendental equivalence of human subjects rather than this quantification of aggregate utility. So I don't know how...

It seems a little bit more natively suited to this kind of scenario than utilitarianism does. And it's interesting that no one seems to have really followed him up, Aside from his ridiculous argument for a certain public policy that he's ultimately making, just his method of politicizing the transcendental method or creating a method of the transcendental justification of political distribution, it seems very generic to me. There were a lot of ways that you could implement a veil of ignorance in an original position and then use it to argue or model.

But wouldn't in this framework, in a Rawlsian point of view, through the veil of ignorance, the thing that you would not want would be to be an evidential decision theory literate software engineer comprehending the thought of Roko's Basilisk, isn't it? I mean, everyone else is off the hook. you know there is this weird perverse election theory in this it's like if you're not thinking this thought you're in the clear so the people who cross the veil of ignorance and really get into trouble are the ones who find themselves

understanding what this stuff is actually about because they're the only ones worth torturing as far as the basilisk is concerned. And then from the other side, so your preferred world is one in which the distribution of vulnerable states is maximized. If that makes sense. It's an interesting, yeah, it's an interesting question, isn't it? It's like, I mean, your preferred version is probably something just messianic where some poor, unfortunate scapegoat messiah figure gets the whole of the brunt the spiritual responsibility and everyone else gets the goodies from the

see right I that it's a little bit reminds me of doing you know the Paul Atreides and seeing the golden path the different ways it can go and being forced into this messianic position by the other memory. I'm not going anywhere with that, but that's what it was. Yeah, I think that messianism probably does take you there quite quickly. Can I just say, if no one has anything else to throw in, like, I've been remiss at keeping people up to date with where we're going of this, and I'd just like to say,

for the if any everyone who's going to be with me here the great basic ski man now is to go in reverse order as if passing through a mirror back through the topics that we've covered in I hope sort of deeper thickened and more informed context so you know as the basic guideline is just to flip your module one material upside down I will throw stuff at you I promise to get something tomorrow and but I think one and serendipitous part of this is is of course that will be dealing with the a row coast basilisk

on Halloween and so I'll kinda I I will chuck I will chuck in some more reading material for that and then for the rest of the thing as we then go back, ending up disappearing in reverse into a super intelligence explosion in week A. I have a quick question. It's about the kind of core of the class and the title which is what would security then look like in the face of this basilisk. And I'm kind of just going back to the Bitcoin class a little bit,

because I really like the term of coding and over-coding and thinking about the wasp and the flower and thinking, OK, so if this basilisk is essentially hacking or hijacking us because we're overcrowded and, you know, using us to save us. So how do we secure what aspect of our code, so to speak? And I'm not even talking about metaphorical. What would that look like? And have we discussed that at all today? Not a lot, not a lot. And I think it is a crucial question because what's interesting is both sides of this Bacillus discussion think that they are talking in the name of human security.

I mean, if Yudkowsky was at the Anthropole roundtable, Roko's going to be knocking on the door and saying, why aren't I here? I'm at least as much involved in practical efforts to maximize human security as this Yudkowsky dude is. you know so I mean that's a fascinating structural there's no no one between them neither of them a dismissing neither of them are saying be sensible you know it's not a problem neither of them are saying neither of them are doing Dr. Evil and saying just let the thing out the box and see what happens they're both

a they both think they're optimizing human utility and and taking a parker passed to to a I that dust so yes it's excellent question and all the states this massively heated shism in the whole that shows that there is no easily obtainable internal coherence what is a human security policy. So should we wrap it up? We're a little bit

over time, three minutes, or you guys want to continue on? Amy had a question. Amy, you didn't really speak today, I want to hear you. Sorry, I'm really tired I don't even remember what I was going to say The one thing that I've been thinking of now though is the whole notion of identity that this argument requires to work and just brought to mind that episode of Black Mirror the Christmas episode where Jon Hamm is his job is basically breaking in emulated like M's and workers and there's this great scene where he's talking to the copy of this woman

who's voluntarily had herself copied so that she can employ herself as her own personal secretary. Right. And this secretary kind of runs her house for her. But there's this great scene where she kind of goes through the realisation that she is the copy. And kind of that sort of, that particular episode relies on the opposite kind of possibility of identity, that there isn't a basic or fundamental identification of the copy with the original, so that the original can then sell herself as a slave to herself. Sure. Or as a worker. So I thought that was an interesting kind of thing, because it seems completely counterintuitive to me that I would really care about what my simulated selves

would have to experience. the point where it really gets on the that's not what he's saying that's not the problem, it's not that you would care what they're it's that you would discover you were one of them that's what I was going to say yeah sorry you don't have to care about that paranoia yeah yeah because I mean whatever theory it is would say that like you have two futures You literally fork at the point where you've been simulated. If I'm simulated one second from now, and so there's the me that's still here and the simulation which experience is somewhere else, there is no sense in which the simulation doesn't have the history of the me that exists at this moment.

I have forked. I'm both. Yeah. This is the most horrifying episode of Black Mirror, Amy. I love it I had a nightmare the night after this must be the second series is that right? it was a special episode it was a Christmas episode definitely look it up yeah it sounds great obviously yeah he basically breaks the resistance of the emulated worker by speeding up the processor so that she just experiences years and years alone in this sort of zero virtual space

and then he kind of like flicks it back to sort of his time and she's completely broken from what she's experienced as like 300 years of solitude it's fantastic right yeah it does sound very relevant as well I mean at least the mechanics of torturing simulations definitely Anyway, I didn't really have anything to say, Mo. I'm just kind of flopping around here in half a weakness. We were hoping that Luca will join us, but she never did, so... It makes you wonder, like someone said earlier, is it even possible to torment someone for the entire rest of the history of the universe?

And the same, something like that happens in that Black Mirror episode, a simulation is set on like, you know, 2,000 years or whatever it is, like indefinitely it's highest set. Like, I don't know, at what point is suffering no longer something that's possible, is even insanity, you know, something that is no longer a meaningful distinction, like how long, if you have a mind that isn't able to stop running, and you have a thousand years of subjective, continuous subjective experience, like, I don't know, does a super intelligence come out on the other end just by like forced evolution or forced permutation through everything that it can reason about its situation. I don't know. That's kind of an interesting question whether you would expect a human mind to eventually,

I don't know, be able to hack its surrounding computational environment, see the code, whatever, if you just left it in the room for a thousand subjective years. or does it just get caught in like a you know degenerate loop like one of the degenerate you know initializations of Conway's Game of Life and it just sits there flickering on and off is this a kind of buddhistic point that just simply by existing it is being tormented or yeah no basically that's kind of that's what in Blackbeard that's what you're made to see is that with with minimal and particularly or explicitly with minimal stimulus or no new stimulus

to be introduced like the only simulation is supposed to run our house is just trapped in like a blank white construct space or god knows how long okay okay yeah it's interesting that's not like the because the in the notion of imprisonment and torture are not very obviously analytically related, are they? I mean, it seems like a kind of synthetic relation, like, but this is obviously implying it's more integral and just that, just confinement than is that. Yes, confined existence as a kind of suffering. it's a refusal of embodiment as well it's kind of deprivation of any

any embodiment of her intelligence the M's intelligence sorry that doesn't help no no no but it does I mean there's a lot of ways that you can torture a state vector I mean that's why that's like in Glasshouse this sort of like distance equal to Accelerando identity theft is like stealing someone's full state vector, and it's the most horrible of crimes because then you can do fucking anything to them. Anything. To a billion versions of them. And I mean, if the body, that's a sort of interesting sense, Amy, that like the body, or the environment is part of the body, or is a body. I don't know, like to, it says that depriving, depriving you of either one of those is reducible in a way

depriving the intelligence of context or of stimulus, which is equivalent to a kind of confinement and which is intrinsically a form of torture. Causal isolation. I've got a feeling, Jake, that people are hazing out. at this point. I don't know whether that's fair in saying that. I'm totally hanging out right now. We're about 10 minutes over time anyways. So maybe it's time to stop the broadcast and thank Nick for this awesome session.

Yeah, well thanks everybody. As usual, it was great. I'll definitely try and pump out some sort of material and think dark basilisk thoughts over the next week. And remember, we're going to be moving back in time, right? Yeah, that's right. No, we're now sliding back the other side, singularity. Okay, thanks everyone. I'm going to stop the broadcast. Thank you, Nick. Okay, thank you.