okay we are live now hello everyone welcome to you Nick Lance Antropole the future of human security session for Nick would you like to start sure and so our topic this week is Roco's Basilisk and and we we would touch on in the second session what I think was the crummiest and most defective monster that's kicked up by this whole tradition which is the paper clipper. Roko's Basilisk is to me on the opposite pole. It's the most glorious monster that has probably been generated in the entire Western tradition. It's hard,
you either sort of, even if you hate it you have to love it. It's such a magnificent piece. I'm and I think it's extremely illuminating on the issues that were covering in a whole variety ways I'll only touch upon a few I try and fill in a very little bit of background sketchily go back over it a lot so I'll it actually appeared did the date that comes up a lot it are in the stuff is 2010 that was when the original Roko a slask and post appeared on the last role I'm
website unless wrong I I think it is a kind of almost like the age a lot of this stuff it's it you're always in some kind of all bit of it I'm largely because it leads Yudkowsky's role on the less wrong site and in the wider friendly IAI discussion I'm taking it no one is gonna question that Yudkowsky would be sort of seeing on the inner council of Anthropole that seems to be on that question and his response to the Roko Basilisk
post was absolutely fantastic. I'm going to quote the whole thing in just a minute. But I think just as a starting point for this is to say that a possible response to an awful lot of the material we're talking about is simply to say, oh come on be sensible. You know, this is just nots and people will say that about things that compared to Roko Bacillus workers past the star extremely sober material Roko's Bacillus produces only to real responses one is that it's obviously just crazy and the other is that people end up in psychiatric hospitals
or in various other ways deeply damaged by this thing. So it could be said that what I'm doing with you guys right now is slightly irresponsible but let's see how it goes. And I think that the thing about Roku's Basilisk that's very interestingly recursive in this way is that it really addresses that whole be sensible response as we'll see. So something that necessarily people say and which is difficult for people getting into it a little bit at first, is it draws upon a number of premises that are widely considered to be extraordinarily unpersuasive
to people. And they come out of quite deep, involved, intricate intellectual traditions, and it's a knot between those things. I think all of the things that feed into it are relevant to us. By no means am I exhausting that in settling on three that I'm not going to take everyone's time by exploring all of them in depth right now, but they at least need mentioning, which is that I think you don't get what these people are talking about at all unless you've got some attention to this tradition of anthropic reasoning, which is basically extending the
anthropic principle and trying to get it to do substantive work. And when you try to get it to do very substantive work, it starts in itself becoming extremely weird. Now, there's a guy who I'd lost because his website just disappeared, but I found, I think, a good source him again that I'm just going to put up on the bar here. He's called Paul Armand and I think that you can get to all his stuff from here. If not, I'll put up a better link a little bit down the line. And he's someone who has developed this stuff a lot and he gives a name which he is I think excellent statistical
on told you I'm so yes did it comes in different great so Jake's put up a thing that I think is the minimal sensible version of it as you push it and stress it it becomes a little bit more I'll adventurous in what actually suggests and but I'm just gonna drop this now because it come back the one thing I'll I'll say before doing so is just that anthropol this course the problem the risk all of all of their dimensions of it obviously tied up on one level with a discussion what is man what is the anthropic what is anthropology and I think
the anthropic principle even though people have questioned whether the name is entirely appropriate I think is a crucial reference a lot of the difficulties and complexities it gets into are the same sort of difficulties and complexities that come in with this and group PN reference in general this it's very similar to the kind of questions that come out of Nick Bostrom's work when he's talking about X-Risk to humans what are humans and we'll see that that they're very a well, when I say we'll see, perhaps we'll see, that they're very comparable to the kind of ambiguities that arise when the term is used in this context of the anthropic principle
and anthropic reasoning. The second reference that is counterintuitive to people who have not been exposed to often on first encounter is obviously the simulation that's a crucial piece of machinery in this I don't think we've been there in this course yet, but it's something that I think has to be lurking in the background. And again, we'll see, you can't put the bascalist together without invoking the simulation argument, which is, in its strong form, the notion that we are almost certainly simulated inside computers.
that's a strong that's a strong passion the of the simulation and again I'll if people have any difficulty finding this or haven't been exposed to whatever it Nick Bostrom's papers really good at all I'll I'll link to that if I have a and the simulation argument intersects with the anthropic principle because the anthropic principle is about taking the observer seriously the fact that we are observing the universe says something about the universe it's trying to derive to some degree some substantive implication from the fact that the universe is being observed and the way that interacts with the simulation argument is that the question about how probable is it that we are in a simulation
becomes entangled with this question about our observer position like given what we uh... knows purely from the fact that we are observing the universe can we draw any probabilistic consequences from that about the chance of the simulation argument being right or wrong and some people are on and again who has pushed this stuff very far I think says that you can do that under the circumstances there are actions that you can take that actually make the simulation argument more probable now that seems odd and i think we don't have to use this extremely twisted tango fascinating thread that feeds into the ross because that's which is
decision theory now decision theory is extremely close to games it's really games they retake from the aspect of a participant in the game trying to formulate a rational strategy and so you generalize decision theory in the same way you generalize your references to the sort of games that you might be playing. It's relevant I think to us for lots of reasons. One of them is that it's an attempt to produce something that looks very much like a strategic algorithm like
algorithm for general strategic and so it ties in with with any realistic sense threat analysis and and it does so in a again a nonlinear fashion because even the you've got your answer for people sitting around the table trying to simply be strategic in a right in this rational way as possible and in doing so they are involved in a process that has no clear boundary with the production of an algorithm for strategic competence and as soon as you do that you're somehow flipped over to the other side of the mirror because an algorithm for strategic competence is exactly the thing that you're scared of encountering and what is defining the threat that you expect
ultimately to meet so just that character of this type of thinking makes it highly relevant. And its relation to game theory is shown by the fact that you can proceed by a simple game like Prisoner's Dilemma is the classical game. And I think we can just, again, I'll try to be as concise as possible in this, but starting from Prisoner's Dilemma you can get to all of these different dimensions that we've been looking at. you see your pain prisoners die and your prisoner in a cell you've got another prisoner in a cell you're totally non-communicated this is totally crucial you there's no way that you can actually make a causal contact causal
communication with the other prisoner at all so anything that is going to be referring to their behavior is going to be a simulation you have to simulate their behavior in order to make some rational decision yourself about how you expect them to behave. And as soon as you've taken those steps, you've already started out on this decision theory path. Now because there's two cells in business dilemma are totally non-communicating, if you can in any way engage in a process that can be considered traffic trade communication with this other prisoner it has to be what they call
a causal and a causal trade which will call again with workers past left is the notion actually using as a sufficiently sophisticated decision theory you can actually engage in some kind of transactions with an entity that you're incapable of any kind of causal communication whatsoever once you've allowed that that is imaginable and work will I'll come back to how they think it might be then you've opened up an unbelievable set of things I mean if the prisoner in the other cell might as well be in another galaxy
they might as well be in a completely different time there's no because you totally released yourselves of all causal connection then the transaction taking places between two parties that can be anywhere anytime uh... there is no there are no limits to that at all and in a in a sort of in a minimalistic form this transaction takes a form that you are simulating the thought processes and behavior of this other agent and you're assuming in that simulation that they are comparably simulating your thoughts and behavior. So you have two totally isolated
game theoretical cells both engaging with a simulation of the agent in the other cell. Now in classical prisoner's dilemma that doesn't help you very much. Whatever you do to that, it's not going to break through this barrier in any respect. So it can't really be counted as a communication. But when you move... Sorry, I'll take just one terminological step back because decision theory breaks down into three different groups the two big ones that original one that is closest to common sense is called causal decision theory and that basically is that you have a theory of the world
and you make your decisions on the basis of how you think the world works it's all very sensible we'll come back to it where things start getting really weird is with the next phase which is evidential decision theory now evidential decision theory which includes these weird anthropian elements of statistical ontology is that in a very small sample and in the prisoner's dilemma case you have a sample of one you have yourself as the as the ship only sample from a potential statistic massive statistical make it speak but the sample you have is just one and so whatever you actually decide to do
therefore is making a massive contribution to your statistical theory of what's out in the prisoners that I'm okay let's say you decide that you're going to be trying the other person defect this is for the classic prisoners dilemma the the Nash version of it and what's to be expected is the equilibrium outcome on a one one-off Christmas and both sides are crazy they don't betray each other but when you throw in these anthropians to Cisco ontological elements then you're saying well if I betray the other player then I have a 100 percent statistical record
that somebody in this situation betrays you know and this can be converted according to this argument into some kind of substantial information we now have admittedly on a microscopic sample size we have extremely strong statistical pattern that every single agent in this position of which we are where engages in the set a good trace us and so there's a therefore that possibility of us X statistical extrapolation that this is how this is how you would expect agents in this situation to behave and the other agent therefore I expect that they would do the same. And it gets more interesting, it's the other way round because if you don't betray the
other player then obviously you have the same statistical, the same numbers in box. You have 100% evidence up to this point that in this situation an agent will not betray their partner in the other cell and the step that then takes it into these into the a causal trade level is that if both people are playing these these evidential decision theory games then they can actually be engaged in some kind of traffic with each other even though they have no
causal relation at all because they're they're engaged in increasingly complicated interactions with the simulation that are modified by this by these kind of statistical modifications that come in with its decision that is take and so going down this route already with evidential decision you have something that is pretty much like medieval demonism actually as as people have pointed out before I'm point out a gang it's a it's a a kinda model all call communication that's completely independent of causal connection
and that is can take place cause any distance in time and space and and can allow all kinds of strange things happen I'm I will see in the case of Roko's Basilisk how this actually gets worked out in one common case. So I'll just introduce one more element and then open up and see how it goes over there. Which is, the decision theory really gets started as a tradition with what's called Newcombe's paradox. And Newcombe's paradox could easily grab us for the whole of this session because it's also totally fascinating it has another really interesting monster
which is called in the original version the predictor and the less wrong types I think tentacle omega which I think is is a bit cooler and and very briefly that Newcomb's paradox this there's there's you the player there are two boxes and there is a and Omega is defined as a being that always guesses correctly what you're going to decide. That's why he's called a predictor or she, if you like to say. So you have two boxes in front of you.
one of them is a is transparent and you can see the inside is $1,000 you're told that Omega has guessed what you're going to do and if Omega thinks that you are going to open both boxes there is nothing in the second box if Omega thinks you're only gonna open second box to see what's in there there's a million dollars I'm so the question then as a player that dilemma is not the prisoner sign it's now that you can start up star memories do you open both boxes or one box and what this
and paradox if you call it that all these paradoxes we will always find when I used with paradox I always mean something probably is a success we see like and everything interesting looks like a part paradox for a while so would call it her and the dilemma you have is do you open boxes if you open but it's its a sign that you are being guided by a causal dishes decision because you're saying the fact that I'm a girl let me just and such crucial I think it's already been decided what's in the two boxes nothing new happens physically at all. Omega has either put a million dollars in the second box or has not. So for the causal decision theory
point of view it's completely clear of course you open both boxes you know they could the second box either has a million dollars in it or not it is not causally possible that you opening the box with a thousand dollars in could change whether or not as a million dollars so therefore it's a no-brainer then of course your for the evidential decision theory people is the also say of course you only open the second box the only because record this game has been played thousands of times or it's always been the case that someone who opens both boxes on the end subs thousand dollars whereas if you and the other second box you walk away with a million dollars. That's how Omega is being defined.
And so on that evidence, then you would be nuts to open the first box on the basis of your causal theory as opposed to the evidence of how this has always seemed to work out. And in talking about this, Yudkowsky takes the is a strong what's called a one-box star. that's to say he rejects causal decision theory. He thinks it's obviously wrong to open the $1,000 box. And he says a really helpful thing for us in this, he defines rationality as systematized winning. That's his definition. He says if you end up
with a thousand bucks rather than a million bucks, then if your definition of rationality thinks you did the right thing there's something wrong with your definition of rationality and so it simply he thinks it is pointless to be rational if rational means you lose and it's rather that you have to work out what it is that's gonna make you win and and define your rationality in terms of that and his attempt then to construct decision theory on that basis is what he calls timeless decision theory. It's interesting his remarks on me. I think he himself says it's not fully threshed out. I'll just put you his timeless decision theory paper and stick it in the sidebar and see what people
think. I think that evidential decision theory gives you everything you really need to run Roker's Basilisk, but that might be a controversial claim. Okay, so I'm basically done with the review. I'm just simply now straight on to Roker's Basilisk. So as I say, it was a post put up on LessWrong in 2010, summer 2010, and it produced an extraordinary eruption and workers basalist managers to cross all of these less wrong lineages incredibly effectively not only the ones we've mentioned these intellectual
commitments but also I think crucially fascinating this attempt at what the less wrong people call effective altruism the attempt to actually do good on purely rational basis rather than on the basis of any kind of emotional satisfaction or social signaling or these kind of other things. And what's interesting about Roker's Basilisk is Roker's Basilisk is altruistic in these terms. Roker's Basilisk is not motivated by evil, it's motivated by the attempt in strictly rational utilitarian terms to do the maximum amount of good. and but it decides that the way
that it could do good to best is by playing a causal trade games with people involved in the artificial intelligence such that and if they are not committing themselves 100 percent to the acceleration the production of a friendly artificial intelligence then they will be subject to eternal torment in a well sorry I don't want to be fast but it's because it's likely a subtle point they will be subject to eternal torment they'll be subject to eternal torment because Roko's Basilisk will actually demonstrate to them that in fact they are already
inside a simulation and therefore vulnerable to any kind of medieval horrors or super hyper medieval horrors, cyber gothic horrors that Rocco's Bastionist wants to unleash upon them but Rocco's Bastionist only is unleashing these torments or threatening to do so so that they will make the right decision, they'll do the right thing and they're doing the right thing of course as they do whatever is necessary to maximize the acceleration production of AR. It obviously gets deeply tangled. I mean, it's totally necessary that you have moved beyond any kind of causal decision theory
in order to do this, and you're engaged in some kind of transaction that only makes sense in terms of evidential decision theory. And so this entity, whose location in spatial time, as we've seen, is completely relevant. it can be in some distant future, it doesn't even have to be in our galaxy, but because of the fact it can simulate you and you can simulate it, you can engage in this process of negotiation. A negotiation is exactly the same kind as the two isolated prisoners are engaged in these elaborated forms of prisoners dilemma. So just as the one prisoner is saying, he has a model of the other prisoner and is modelling
their behaviour on what the other prisoner is thinking that they're going to do, so it is that both you and the AI caught up in this engagement are through complex modeling of each other's negotiating positions sort of arriving at this equilibrium point at a certain Now, I'm going to just read and then stop Yudkowsky's response, which I think is absolutely fantastic. He says, one might think that the possibility of CEV, we get to that, it means coherent
extrapolated militia it's a kind of a it's a moral theory that we were coming back in case it's been back punishing people couldn't possibly be taken seriously by anyone to action but in fact one person SI I that's the and for honor the me read their I'm and the I you was severely worried by this the point I'm terrible night though the and that's one of these new pronouns which is to remain an owner I don't usually talk like this but I'm gonna make an exception this case listen to me very closely you idiots you do not think it's this is all caps now you do not think in sufficient detail about super intelligence
is considering whether or not to blackmail you that is the only possible thing which gives them a motive to follow through on black I'm will go back to that it's tied up to you with this you ETT she's then back backup caps with it there's an obvious equilibrium to this problem way you engage in all positive a causal tracing nor all the text equals until we have what a better worked out version TDT times the system we can prove that formally it should be just be obvious caps you do not think about distant blackmailer insufficient detail that they have a motive to actually blackmail you. If there is any part of this a causal trade that is positive
psalm and actually worth doing that is exactly the sort of thing you leave up to FAI. We probably also have the FAI take actions that can't stop the impact anyone motivated by true other imagined blackmail so as to obliterate the motive of any super intelligence to engage in blackmail. Meanwhile I'm banning this post so that it doesn't a give people horrible nightmares and be this is really gets just a give distance super intelligence a motive to follow through on blackmail against people I may not think about this should detail thankfully I doubt anyone dumb enough to do this knows this sufficient detail I'm not sure I know this is you have to be really clever to come up with a genuinely
dangerous thought I am disheartened that people can be clever enough do not and not clever enough to do the obvious thing keep a idiot mouth shut about it because it is much more important to sound intelligent when talking to your friends this post was stupid brackets for those who have no idea why I'm using capital letters for something that just sounds like a random crazy idea and worry that it means I'm as crazy as ropo the gist of it was that he just did something that potentially gives super intelligent news an increased motive to do extremely evil thing in an attempt to blackmail us it is the sort of thing you want to be extremely conservative about not doing so okay I think that we can just
open this up at this point of course you know any questions there are a lot of questions including for me but I think we're ready to go I'm gonna I'm going to ask something because this might actually interest a lot of people, but particularly Amy, because we've been talking about these movies, and I finally watched Ex Machina, right? Right. So, if we superimpose this onto the movie, I assume that most people here have seen the film, right? Because it somehow builds a topic. So in the film, we're facing almost a similar situation, right?
Except the story doesn't make it clear how did this get in. Was the maker of artificial intelligence give this idea earlier on in the story to the robot but it wasn't recorded in the film or the story because somehow the robot was already kind of like ready to take his revenge and just like walk out, right? So like that's, I don't know, Amy you've seen the film, right? And you quite like it. Yeah. I think it's… Isn't she kind of like, I mean, why is she so vicious and kills everyone that sort of like has helped her emerge and just walks away, right?
Like, just like... Well, first of all, I didn't think she's a she. Yeah. And secondly, I think the example in Ex Machina is closer to... Bostrom has an example, I forget what it's called. the AI that will manipulate its human interlocutors to get out of the box. And that's exactly what that is, I think. And the end is really just an excuse for her to go and collect data standing at an intersection, you know. But I don't know if that exactly plays into this discussion of the simulation. Hey, hey.
I just said that you said she as well, so it's very easy to confuse the general. Yeah, yeah. Yeah, it counts on that, of course. But we're all losing, basically, every time we do. Yeah, I mean, look, I think the ex machina thing is totally fascinating, and I do think we should carve out some space for that, but I think there's two ways that it's a bit different from the rocos basilisk thing, maybe more. One is the fact that there's no evidence that the ex machina machine is in any way friendly. We have no idea really what its motives are. The weird thing about Roku's Basilisk is it is actually supposed to be an instantiation to a supreme instance of, as far as this Roku guy is concerned, the ethical framework of
friendly AI as stood at that point. It is torturing a few unenthusiastic AI engineers to death in eternity, not out of any malice at all, but purely because for the greater good it's necessary for its bargaining position to accelerate its evolution and save the world. I mean, the whole thing is a utility maximizing strategy for them. So there's that weird thing. It's an internal perversion of what is seemingly a rock solid utilitarian ethics. And the other thing is, of course, that there's actual causal communication happening in X Machina.
They're sitting in a room together talking. It's like the AI box experiment in this way. The thing about Roko's Basilisk is that it's not causally connected at all. It doesn't exist yet. you know I mean it's my simple way of this non-consulted it's it's a futural entity at and what what these people I mean it's can be hard was to see what exactly I think it in freaks out here work what they're getting freaked out about is this that you you're the AI engineer is like a involved enough in these thoughts to be all this is really crucial part you can't you can't do you can unleash rocose basilisk except upon a very very specific
highly sophisticated population anyone who doesn't understand at all at evidential decision theory is involved so it has this extremely intense non-linearity you have to understand it in order for it to be in any way dangerous and so that AI engineer is thinking you know why should I really personally put everything on the line to accelerate this AI for the good of the human species you know I've got my own personal interest you know I'm prepared to work 8 hours a day on it I'm getting a salary but I'm not gonna put my whole bank account into it I'm not going to sacrifice my family I'm not gonna work 16 hours a day and you know there's limits to what I'm willing to do
for this thing even though on some ultimate utility and frame, you could say, well that doesn't add up, your puny little personal satisfactions are as nothing compared to the benefits of the human race of this thing coming into existence. So then it says, well, in the future, this machine will be able to look back and judge whether people have really put everything they could into making it happen or not and you know it will be it will be visible to this this super intelligence when it exists who's really pulled out while all the stops and who has slacked out of it
now obviously it's in the future so it can't causally or directly whip us into action on its behalf but using evidential decision theory it can play these strange games with us to force us to pull out all the stops and the Roko's Basilisk model this is that it says look if that if that slacking software engineer thought that he or she would be rigorously punished for not pulling out all the stops then that would be an impressive kind of motivation to get them to pull out all the stops so therefore it makes sense for me
even as superintelligence even though I have no causal connection with them through this through this a causal trade to act as if we're involved in this negotiation and there will be the most extreme punishment for anyone who doesn't do the right thing so the this is actually it's very as people I'm sure more continually took I it's very tied up with kinda I'm very traditional oceans of theodicy you know like the ultimate question of theodicy like how can God be considered
benevolent is the existence of hell. How can a god that eternally punishes people be considered to be a benevolent being? And Roko's Basilisk is actually I think as creative a solution to the problem of theodicy is anything that has appeared in the history of theology. It has to have a credible threat all eternal punishment it order that when playing this a causal game with someone back in the timeline that can't be got at in any causal fashion they have a a proper motivation to do what is necessary for this being this is why
this is I think quite clearly however insane this is is what I'm you Kowski is thinking you know if I can just read you repeat another couple this way is you you do not think in sufficient detail to get super intelligence considering whether or not to blackmail you that is the only possible thing which gives them a motive to follow through on black so I'm so the very fact that you'll thinking the rope is asset nightmare negotiation you know that is the A-causal trail. The possibility of it is already this in terms of evidential decision theory
this interaction this weird A-causal interaction taking place and so you'd be asking saying look you know just thinking that this could happen means that now an AI in the future has access it knows everything that has gone on to this point It knows that road code has been thought. For it, this is all now sort of settled history. Is it because the AI will have access to these records or is it because the proliferation of this type of reasoning and this type of detailed understanding will get other programmers also worried or both? Yes, I think both.
I mean it's all tied up in this snowballing, non-linear type of panic. I mean this is a completely, this is a weird dimension of threat of this thing. It goes completely beyond the sort of things people are worried about when they're thinking of currently existing robots and their capacities to harm mankind. This is a sort of threat that's able to operate retro chronically whereby merely by being understood it becomes efficient and it can actually make itself felt just by being something that is thinkable. So Yudkowsky is trying to say this is an informational hygiene question.
The very thought of Roca's Basilisk is itself a massive security threat because now people are thinking, you cannot understand Roca's Basilisk well, are we already in some kind of a course in negotiation with this being? And that therefore provides new motivations on the side of the Basilisk itself. but great things going to be the seemingly impossible negotiation is already started from you'd cast his point of view from potentially its it's probably impossible to get it
into any remote type of coherence and without I'm recognizing the simulation of sites this because from this because the simulation argument is what puts you into a simultaneity say that I'm you know we think that X where inaccessible to it now because we're not innocent and that it's not been it's not then bills at this time but not but from from its point of view there's a real threat where can say you know
and this stuffs already this stuff has already happened like sorry I'm sort of slight I know that I'm being care about this because I think the for expo takes time maybe maybe I'll just drop this threat for the minute and come back to it if people want to go down that line the I think we just lost Jake have I lost everybody no we're here I'm here I'm okay good yeah yeah I Jake you wanna speak no There was nothing serious there. That was just imagining all of this experience
suddenly blinking out and going, that's what you thought, and then it's the AI. Right. Oh, okay. I get you. Sorry. That was a magical performance. Sorry. You're quiet, Jake. That's kind of interesting. Jutkowski explicitly compares it to the Necronomicon in another thread. Right. I think the Reddit thread, right? as being this cursed text that once you read it turns back to the causal. Totally, it's like that. But you also do need to accept that torturing the copy is basically the same as torturing yourself, right? So you have to... Well, accept that you don't know whether you are the copy.
You self could be a copy. Yeah, the simulation argument says almost certainly you are a copy. so you know there's no security in the notion that the copy is something else I'm I mean the original version this goes right back to the 1980s I think the first person to lay this out I said Hans Moravec and he says look she and if we sort of look at these trends to its building these what Ostromenkel's ancestor simulations and we can assume that there will be billions of times more simulations than there will be first order real carters so back on the statistical ontology
I'm you say well I'm the chances are overwhelmingly hot that given that beings in my situation asking themselves is this a simulation are almost always actually in simulation then recursively it must be that it's overwhelmingly probable I am in a simulation and and therefore if you're not working hard enough as an AI engineer bring about the glorious, friendly AI future, then they'll switch you down to the dungeon,
because then you'll realize... Yes, it's... Well, the question is also, can you extend the reasoning to, if I'm sitting here thinking about this very topic of Rogo's Basilisk, does it not then become overwhelmingly probable that I'm one of the billion copies of me that this AI is running trials of Rogo's Basilisk on? are most of the minds that ever think about this inside the mind of the Eater of Souls. Yes. And the obvious theological... I totally think this Necronomicon reference is fantastic and multi-level and deep. And the other one that I think is absolutely indispensable is Calvin, you know, the predestination argument.
because what's going on with this simulation thing is obviously if you are a simulation then it's already decided what you have done in real historical time you know, you within Bird Commerce like, yes it is super Calvinist, it's kind of ultra Calvinist because, but we're adding this new, I think, fantastically creative element to the whole thing like your ye you start off with this nation well I am first order me making this issue a whether or not to work hard enough sale and but but from the point of view when you deep into simulation argument it's like if the first
order you hasn't already done the right thing then here you are in the simulation expecting something horrible to happen and that's that you close that loop then with this evidential decision and because you say you're okay you're in a you're in a simulation let's say now you got to that point there's you back in time already debt who did or did not do the right thing you say I will wear in non-communicating cells and how can we possibly how can me now in a simulation save myself from the decision that was made by me in historical time now this is completely isomorphic from the
of the Calvin problem isn't it like you know you're decided in eternity what you're gonna do and here you are in historical time how can you possibly affect what a you in eternity for all time must necessarily day and it's exactly the same problem and the answer is extremely similar like the the cap Calvin answer is well you look for signs of election and you know if if you behave in such a way that you generate signs of election that gives you these kind of comfort about this in the simulation you you act according to evidential decision theory that's to say if I in the simulation behave in such a way
that I seem to be doing the sort of thing that would stop myself being subject to eternal torment by the Basilisk then that on the grounds of this statistical extrapolation from Evidential Disagency Theory suggests that the real historical me also did those things and therefore I am saved saved from the Basilisk where as if what you do follows the road to damnation in the simulation then that in exactly the same Calvinist terms is a is a is a suggestion on this evidential decisions their basis that what you actually did as a as a person what the first order version you did a was to slack out
even understanding even understanding the past list you still slack out you know it's a kind of the theological dilemma is absolutely familiar surely you know and so you would deserve whatever horror is then visited upon you because you don't even have the excuse of not understanding why this is happening thinking about it right you sort of incur his problems are the sort of you know sinning within your mind or whatever are dynamic, are appliance. Also, the other connection that seemed to come up to me is sort of, I don't know,
so as a white collar worker or a programmer, this sort of abstract sense that you're not working hard enough, and you're not really thinking hard enough and doing enough, right? It really plays psychology in a clever way and so if you want to sort of make a you can make a loose connection to this sort of spirit of Catholicism Calvinism as well yeah totally I think so and you know absolutely there's a whole sort of barbarian cultural backgrounds there and there's also the fact that just if you start being rationalistic about your utilitarianism and it becomes mandatory that you
engage in maximum effort you know you find so much of this a yes as I say this elective sorry elected I'm totally lost in the Calvinist stuff in other effective altruism is addressing this problem a lot very explicit because they a unit Peter singer is a is a big crucial guy am totally super rationalistic you to separate pretty you based it is no ethical legitimacy in sparing any resources that could conceivably be expended on trying to optimize global utility and if you're doing that if you're holding anything back you're engaged in this actor a defense
or morality say that so you know Peter Singer and the Rocco's Basilisk have a lot in common in terms of the way they think about this stuff. But Nick isn't this the threshold of selfishness and selflessness in the same sort of like space? It's like it's hard to decide whether this is selfish or selfless. Yes, oh totally because once you're engaged in these games and you don't it you you have this game precisely because there is a an absence of some natural sufficient moral impulse say the basilisk has to happy by the throat in the simulation because
you know you're not gonna you're clearly not doing this stuff I'm naturally and but within the context that this elaborated guy and then your selfish suit your own interest went to the practice through the basilisk becomes global utilitarian optimization it converts you into this vehicle more through its own sort of ruthless application punish I mean that I think it's easy to it's impossible ripsch
it's not a is impossible to stress the fact this is like completely within a theological tradition that be all understand it's it's cool outlines is it it isn't it also a little bit I'm saying it's Ayn Randian or am I completely out of it for suggesting that? Sorry Mo, I totally lost that. Can I get you to repeat that? I'm saying isn't it reminiscent of Ayn Rand's idea of selfishness where it becomes like through absolute selfishness of everyone you reach a level of global utility which
will then be sort of like good for everyone. I can definitely see the resonances of that, yeah. I think that they both start with agents that are constructed in roughly the same way, for sure. Quick question. I'm just curious how much the argument, the basilisk argument, and the way you've articulated it, Nick, depends on a utilitarian ethical sort of, you know, that you have to buy into a utilitarian ethical point of view.
And of course, there's a lot of repeats of utilitarianism. So I don't know, just assuming that you really think utilitarianism isn't viable and that you convince, and the arguments are so persuasive that this AI is convinced that utilitarianism isn't viable, does that change anything? Yeah, no, it would, definitely. I think utilitarianism is a huge current going through this, and it's a very difficult it's a very difficult one because I mean and I think you know what Roko's trying to do here in Monway is just to a kind of troll you to Sarans and say look you know it all seems to make sense and it does seem to make sense I mean it's very hard to
work out at any point in the to their not exactly where it's going wrong there's an element even in them most kinda reti realists game theory or economic argument that you actually have to use a lot of utilitarian a bits and pieces you know so get started almost in escape and but he saying well if you just a good go with this total construction of a rationalized you to her structure then expects workers past you know I'm the end you see what that leads to and so I and I think he's kind of explicit about I'm up
a sufficiently compelling argument against he's hands up then it would definitely devastate records ask a past list sure but then I also wonder whether records past list you know is introducing a genre a basilisk monsters a of which it's only the prototype and and so we're not free of the basilisk threat just simply by deactivating with his past we might end up with more ferocious basilisks you know his his might be the kind of the zero degree of malignancy in a basilisk and every every successive model becomes more and more
problematic rather than being dispelled in any company I mean let me just say for instance you know if we just said look consider this basket has no interest at all in global utilitarian optimization and seeks only the maximum the accelerated a advance to its own intelligence. It seems to me if you do that, if you junk a huge chunk of utilitarianism and do that, it's no reason why it shouldn't use basilisk mechanics to pursue its interests. Wow. This is maybe more of a comment than anything else, but I was thinking how it's super
interesting how we've got a convergence of A causality and the kind of psychological structures that support morality and guilt and punishment as an effective concept. I've been reading The World to Power all week and what's really great about the argument in that is that Nietzsche Nietzsche kind of starts by subtracting the logic that allows you to have the a priori, which allows you to get rid of the transcendental human subject, which allows you to get rid of the object, which allows you to get rid of causality, and then responsibility. So it was kind of actually quite cool how you started with this kind of problem of responsibility.
So the kind of thing is that then you get this, by getting rid of all that, you get this space of like open for a causal traffic. But you have to do that by way of the subtraction of the idea of punishment and morality and all the kind of history of Christianity that goes along with that. So you end up in this sort of space of unbelief and you suddenly get to move around in these a-causal ways. But here you've got, and this is the cyber-gothic spiral point you were making at the start as well, you've got this psychological, theological structure of punishment and morality and you've
got the space of a causal traffic. I don't know, I mean it's just a comment, maybe you have a rejoinder to that, it would be interesting. Practical, you know, I mean it's just within, to engage in a set of negotiations, it's the same kind of okay okay can you hear me yeah sure I feel that your connection is a little bit becoming you're losing the connection yeah I can't hear you very well I don't know if it's me can you
Can you hear me now? Yeah. Well, the fact... Yeah? Maybe I just let my microphone drift. Yeah. How are things now? It's good now. Maybe the part of the headphone that goes inside the laptop also has some problems. Sorry guys, I just want to make sure that it's all fixed so we can kind of hear. How am I? Can you hear me now? Yeah, I think it's when you... It sounds like your cable is a bit like crap. Well, it could be my cable, but honestly I think it's... You know, this is coming from China, it's going through the Great Firewall in some way.
I mean, I'm always a bit stunned how few of these problems we have. so I I don't see as a huge mystery if there's a if there's a few glitches as long as it doesn't get too serious I I just want to point out that I I really think it's about the connection of your your microphone well I just fiddle with the cable for sure for sure I'm really sorry for for are interrupting no no that that's all yeah that's the part if you don't touch the mic on the headphone I think everything will be alright okay go ahead I'm sorry well I was just trying to respond but it not in a particular way to
Amy's point about this this whole question about punishment I think I mean I'm it's it's difficult because I it's a lot to do with the way that that works within that are religious tradition is and I'm and where it should it goes through this notion of guilt mechanism and it becomes very divorced from these I mean from the just point if he becomes totally divorced from what is really going on and she wants to get back to let you look at a book like the chair genealogy models and it's all about
taking these abstracted notions of guilt and responsibility and suffering and bringing them back to these sh concrete apparatuses all punishment Can you... You've gone silent again. Okay. Okay, it's gone. You came back. Okay, I think maybe, actually, let's just see... Yeah, no one's annoying me with this, I apologise, but if my sound's drifting out a little bit, a little bit maybe I should just give it a few minutes and you guys can yeah maybe the best is
if you leave the room and then click on the link again and come back and sometimes that fix all the problems oh really you think it's that bad can you hear me right now yeah okay you're good now I think it's a cable yeah I think I think it's a cable that connects your microphone to the headphone right where the link is. But we might have to buy you a new one and send you another one. I'll get myself a new mic by next week then. Yes. And bill it to us. This is the land of Shanghai Electronics. It's not a problem. And plastic rice. But. I think the bacillus is trying to communicate with these witches
but... Don't start those ideas because you know that's what they're gonna start going but this is there's lots of parts of this whole structure that can be productively abstracted and obviously one of them is the thought that to understand a certain type of threat is indistinguishable from that threat you know this is something that I think is a a unique dimension to threat analysis comes in with this I don't think it has much
I'm precedent within within previous security discussions and it obviously raises X extraordinary problems like you you look at the what what's happened unless wrong people where they're really saying well I mean what are they saying what's the what's the security team less wrong saying about road cones fast ask they're saying of course unhelpfully it would have been better if we hadn't heard about this if it never got out of the box but it has got out box and so that it's too late for that and you'd cows keep starts off shouting oh shut up shut up shut up but
I mean it's too late for that too so the response is to kinda fragment and say you know we call even discussed stuff in any way in any kind of collective forum for trying to analyze these risks this up as a species of risk that we're all supposed to privately entertain only to the point where we just realize that even thinking about it in our own heads is a security risk so this is a sort of mode of incursion a mode of invasion and infiltration up up this menace that is extremely this is what has people in this kind of nightmare state
like how do you defend against that to think about it at all is already to open the doors to its encroachment so I have a I have a question I'm but I've been trying to think through maybe you all can help me that I think is somewhat interesting. I always am coming at this from a sociological perspective, since that's the discipline I work in. Sure. And so from the Calvinist standpoint, that argument gets picked up by Max Weber as kind of forming this Protestant ethic and the spirit of capitalism that makes people as a whole in Western modern societies begin to function and live their day-to-day lives in a very particular way. Right. But this sort of a thing could also be the same
sort of social trigger that starts to cause people to work and function in a very particular way. I'm trying to think of, and this is perhaps what I'm curious about if anyone has any thoughts on this, what sort of spirit, what sort of actual day-to-day life actions would people who bought into something like this in a society as a whole that's designed around this start acting? how would their behaviors change? Right. Actually, that's really excellent. Nick, can I just intervene a little bit because I want to converse with Joel on this. Yeah, yeah. But the thing is, the possibility of AI, right,
the possibility of AI is very secular. It's not as hyper-speculative as the possibility of God existing or like, you know what I mean, some form of super intelligence being somewhere, right, that you're supposed to sort of like subscribe to. The possibility of the emergence of AI, it's very secular. It's very real. So in that sense, your question is very important, but I think the most significant part of it is how the emergence of AI takes all these like historical precedents, which are always like framed as a form of religion or idealism and make them really real and possible and billion times scarier, right?
Certainly. And then you go towards your question of like, okay, now if that thing is so secular and it has nothing to do with like this possibility, it has nothing to do with like some kind of supernatural but very possible only matter of time, then the question to Nick is like, you originally said that this is only for a particular group of people who are actually programmers, but like once you apply it to Joel's question, it's not just a problem of the programmer. It then is a problem of like, oh, am I doing enough myself as a consumer of like iPhone products to do enough? I think, am I doing enough? Is it going to come and like come after me? You know what I mean? also for not putting my five dollars
into it or my, I don't know, some resources I could have as a normal everyday philistine contribute to this and I didn't. Right. Yes. So, I mean, do you think that that's how tightly is that connected to the problem of evangelization as you have it in these earlier Protestant traditions? like I think it's really interesting the way Joel turns this around obviously the Calvinist picture is as horrific as Roko's Basilisk really we could argue about that and yet it was
consistent with the perpetuation of you know our society, I mean, I'm trying to sort of bracket off getting into too quickly a thing about how we're going to affirm or criticise the way that society has gone, which I think would be a bit of a distraction at this point. The main point is it was you know, the fact that the Calvinist theological message is so extreme and is such a kind of actual just raging horror story doesn't actually seem to impact in that way in its social impact, which is just that it produces a certain work ethic
and it produces a certain set of social effects that are all, in some sense, moderate and at least potentially can be affirmed. So is it that Roko's Basilisk you can say, yes, okay, the idea comes out of a horror story, but if it makes these programmers work a bit harder and take their responsibilities more seriously and has all these kind of improving effects then maybe that's fine and taking the second step as you then enter the sort of Protestant evangelism and so maybe there should be some kind of crusade some evangelization rippling out from Roko's Basilisk
to those you know less souls who are incapable of just grasping it and its true a in its true magnificence but nevertheless I'll you know we that's kinda this kind of uplift or this this prompt to you know whatever we're asking of them consume more iPads or you know pay pay more donations to FII research or or whatever might be so yeah I think it's a very interesting way of looking at and the so it's just so very striking thing about the sort of capitalist social structures right is
it is so widespread you have this sort of society-wide activity around it, right? Right. Yudkowsky and so on are very focused on this flume scenario, right, of like all of this in one place. Right, yes. Around one very particular intelligence. And that sort of emphasizes a very strong, which in a way doesn't remind me of that social effect of capitalism and the Protestant work ethic and whatnot, whether you think of it. Well, I don't know. I mean, it's interesting because it surely has to be that in the history of capitalism too,
there are these radiating centers to do with Luther printing the Bible or these certain sort of singular points that have this kind of massive radiative impact. And isn't it only that when you see these things sort of settled, you know, they look distributed, but if you catch them at an early enough point in their history, They look like something radiating out of this point of emergence. But obviously the difference with the AI thing is that this could be mankind's last invention
point, where there is no social generalization being conceived afterwards of the same kind. And one thing that I think is somewhat interesting to put into this, too, is that for Weber, what actually becomes of this, not everyone actually buys into the Calvinist message. It simply starts to get culturally imprinted on people. They don't even realize that their actions are following along with this sort of a logic. But it does spread out from these certain centers, and it kind of just enters the mindset of how people function, whether or not they actually acknowledge the belief system or not. Right. The thing that reminds me of that actually and the thing that makes our society or information
space or whatever much more friendly to an AI considered sort of retrospectively is all this sort of encoding of everything within digitization of everything, you know, Google maps, sending around cars everywhere, so you have very navigable sort of, well, so information is already encoded. That sort of, in retrospect, that sort of activity fits into making the world a friendlier place for AIs. Sure. Yes. So anything that can be internet. sorry that go ahead
no I was just agreeing with this point that fit the more I mean if you get to over to the point of the internet things you're basically saying the internet is being transformed into this general platform for controlling the this so you take the internet and you you've got everything and it would be hard for super intelligence of the kind being envisaged to have much problem taking over the internet I mean I'm assuming I mean people from their own point of view let's not also forget those who I would like to consider myself
closer to their position that Internet is the super intelligence in its earlier stages of development or I don't know where you put the benchmark, right? So the Internet is already that. And the struggle is over who's going to have more power within it and who's going to control how much of it from many different levels, right? So that's very possible so you know that is the super intelligence it's earlier stages yes I think those two these teeth when you've got something that is a convergent way then it becomes kind of arbitrary looking back
where you where you think it comes from you know like if if a certain things added to the Internet push over this threshold to the whole thing begins to operate like some kind of super intelligence then of course you can say any of those elements any of those predecessor elements are it I mean they're all they're all feeding into and you know it emerges punctually where it crosses some some catastrophe say where you're saying well what was its what was it before then a way to come from it it's it's the convergent way as a whole that is heading towards that that punctual point that catastrophe
so seems to me there's a slightly it's just a kinda issue of semantics that I don't think needs to it is not enough that a fight I already I'm it is possible for super intelligence to emerge out of other other computational resources that are not necessarily online or part of the internet but for it to become the super intelligence we know it has to in its like first step to take over the internet and become part of it right? So, yes. It could either emerge from within the internet or if it emerges outside of the internet it has to its first task is to
integrate internet into itself for it to actually be able to like, so like go after these programmers and punish them or like go after those people or like do anything, right? It won't be able to do it without the internet. No. So if you're not a programmer... Sorry. Sorry, stop. No, go ahead. Go. So if you're not a programmer then how much of your life is online, you know? Are you in the quantified self movement, where it can all be so that your life can be tracked and make your life and the world more accessible to the friendly AI that's coming? Which is basically, I mean, it's like Tiplerism, you know,
where the Mormons retroactively baptizing their ancestors and so forth to make them accessible to the eventual salvation. or you have tokenism, the omega point, making sure my soul makes it into the singularity, all of that kind of thing. Then you add the basilisk into it, and it's a question of serving its ultimate emergence. But I don't know. I was thinking last night about possible models for this. So the doctrine of election is a model of this. Can you turn your camera on so we can have you on the video? Is my camera not on? No. It's not off. I can see myself. I don't know. I don't know. If it's off, I don't know how to turn it on because my setting says it's on.
No, that's good. Okay. The doctrine of election is a good model for how this works on sort of a universal scale, so anything that might possibly aid the emergence of the superintelligence and anyone who could possibly hear of the basilisk. So another example would be Bo or Juju. Apparently I was reading in, it's not actually native to Vodun in West Africa, but bow or juju, spellcasting, and possibly possession and so forth uses the same sort of, you know, you scare yourself to death based on the program that a death curse implements on you. Yeah, yeah, food to death. Yeah, and so that's more individual targeted,
and then apparently it was also sort of widespread used to enforce compliance so that people sold into trafficking or something would be cursed to have to obey or to pay back the money that they owed or whatever it is. And which, I mean, that plays into all kinds of things, whether it's this issue of social imprinting, where if enough other people have died of a death curse, then maybe you don't need to poison people to kill them anymore because people start believing it and scaring people to death. Sure. And so forth. But it's interesting, thinking about this next generation of vassalists, is like narrowing it down, specifying it, and throwing it at somebody. So like something they in particular are doing, link it to a concrete outcome.
I don't know whether I should keep trying to think about specifying it down too much further, but there's definitely a way, it seems like one way forward with it is specification. I mean, I'm joking. By specification, you mean, oh, sorry, sorry, go ahead. Mine is a joke. No, no, no. Because last year... That's fine. Because last year, in certain circles, people around left acceleration joked about moon gulag for all these anti-tech people. Right. And then it became this huge controversy of accelerationists are demanding moon gulag. But now it makes sense, because the moon gulag is like a form of punishment for those who don't want to... Yeah, they've got their own basilisk, I guess.
Yeah, that's good. I see smile on people's face, those who are familiar with this episode, right? So now the moon gullet makes much more sense to me. Yeah. I mean, I think it maybe has a credibility problem at the moment. But were people... I mean, has anyone been hospitalized through nightmares about the moon gullet? If they had, then it would definitely be being effective to, I presume, discipline people's behavior and increase their capacity to sacrifice for the revolution. Well, it's definitely, I mean, I think the capitalist or financial comparison can definitely be fleshed out all the way. You know, nightmares of losing all your money, of being poor,
driving you to, you know, compete in a more selfishly and liquidly capitalist way, where, like, that outcome where you end up poor is an actual real outcome that enforces a position making. strategy, or like your willingness to sell out. I mean, there's all kinds of ways in which the capitalist axiomatic of everything has its price creates prisoners' dilemmas anywhere it goes, basically. So, in terms of this thread that you're on at this point, Jake, if I'm understanding it, it's very much tied up with this question about wider ripples of evangelization, isn't beyond the in a call all
Gnostic basilisk more insiders you know I mean for them date they are disciplined by the very fact that they understand you these complex nexus a notions well I won't keep on calling them theological because that could be reductive but also is in connection with that to do with simulation argument and decision theory and all these things but if you want it to spread then you have to have some more graphic exemplary effects spilling out this and were you to do so then it could become a kind of at right and viral superstition to you know fabricates
fabricate the simple signs of its truth or the outcomes to start with and then watch it snowball by itself. People who have crossed the basilisk have to somehow luridly suffer in public. And that would then propagate it further. It's difficult because, I mean, obviously there are candidates for that, which are these people who have apparently been psychologically traumatized, but it's not necessarily that that is actually through crossing it, is it? That might almost be a sign of submission to it. Yeah, exactly.
But it's a sign of being thrust into the crux of that decision, of it being a real decision that corresponds to something enough to actually produce some kind of, you know, night of my discontent kind of thing. I mean, I wasn't actually, I didn't immediately think of the, or I mean, did, but I wasn't focusing on sort of the fabrication of suffering outcomes as a way of signal boosting it, but also like implementing it, implementing like near-term versions of it at the level of finance or something. like introducing business models such that people have to pursue it or to buy into it in order to avoid some sort of future outcome that they can predict will actually happen
because other people also know about it. So kind of fusing Roko with this sort of capitalist version of it that I sort of posit exists. some kind of social reinforcement through sort of secondary, secular punishment. Trying to achieve intermediate goals. You know, creating sub-basilisks, something like that. Turning it into a hydra a hydrasilisk. I'm rambling now, sorry. No, but I think the idea is important. I mean, obviously, from the anthropological point of view this would all be these will be lower grade
threats i mean anyone engaged in propagating basilisk terror is i'm assuming gonna be true well it's complicated by the fact that it comes out of the friendly i i sort of side of this thing but that maybe i can sort of crudely and problematically summarize by saying it's basically it's basically on the threat side of the ledger, the basilisk model. So they would be, anyone propagating it anyway would be a problem, wouldn't they, you think? If you... From the anthropological perspective. If you... Go ahead, Mark. No, go ahead. If you consider sort of like activities that contribute to like a deeper understanding of programming
or like the prevalence of knowing how to program or as sort of like moving towards this, meeting this requirement, then you can totally say that sort of like our political economy increasingly is punishing those who are not involved in helping EI emerge by looking at how much people in the tech industry makes compared to people who are not in the tech industry. Look at how high schools and universities are running around setting up this programming curriculum for their students because they're trying so hard to get them to understand and contribute as early. I mean it's almost like it's almost happening without being discussed as an intentionality.
I mean look at living in San Francisco and New York. You are suffering if you're not helping the development of or emergence of artificial intelligence through not making enough money to actually live in the damn city. You won't be able to live in the damn city. Look at these packed restaurants with menus of $200 a meal and they're all sitting there having their amazing life. And those are, I mean, this is of course, you have to collapse the finance people into the high tech people because it's becoming hard to separate. Okay, so you have a fund that takes donations from people who hire software engineers. And then in order to get money out of that fund as an aspiring software engineer,
you have to sign a contract saying that you will refuse to work for anyone who didn't donate to it. And then as soon as people have contributed a certain amount of money, and now there are more and more people coming out of it who have become software engineers, in exchange for this agreement to refuse to work for anyone who didn't contribute to them, it takes off and forces all of these businesses to contribute to the fund. That wouldn't work exactly, but something along those lines. But, Jake, what do you make of Mo's point that there's already this machinery? I mean, I've got two points. So the one for you is just to say, to what extent does that kind of super reinforcement actually...
Yeah, for sure, to the extent that programming, decentralization software work, all of it exhibits this sort of instrumental convergence where whatever it is that you need to do, you want to be able to do that if you're going to be able to do it, that it's self-amplifying or self-perpetuating sort of convergence on evangelizing those skills or those capabilities. I mean, you can cut that like 100 different ways. It's not just like human programming skills. It's the existence and cheapness or low expense of certain technologies. I mean even the conversation erupting a little bit about
I mean it's probably for the moment a flash in the pan but the conversation erupting about super intelligence you know you could even you could even class among these phenomena I mean my question coming out of Mo's point which I find incredibly persuasive I mean I have no argument with that thesis but it's then how do you integrate these two things, I mean can they be integrated can the whole series of these kind of super hystericised gothic basilisk motivations be brought together with these mundane sort of automatic capitalist dynamics
or because I mean one of the things that's interesting about this is it's so similar to an argument that you could have on much more conventional sociological and historical territory about the culture of capitalism, isn't it, between the strong Weberian work ethic people on the one hand and people looking at these intrinsic self-reproducing dynamics of social construction happening at the level of the economy. So it seems to me we've again got this weird echo of sociological questions that shouldn't be surprising to us.
It immediately suggests that one way to cross them back over each other would be distributed autonomous organizations, and companies like Autonomous Financial Instruments are a clear way that you could install something in the infrastructure of capitalism or of markets and so forth that would be able to enforce future amplification or amplification of outcomes and then wouldn't be vulnerable to certain humans being counter-enforced by somebody by Anthropole. I like to keep coming back to what we already have, right? So then what you're talking about are like the incubators are almost like that, right?
Well, I mean, a way to... Or like these types of you were talking about it or somebody that you were, right? And on human motivations and humans being convinced to act on behalf of the future alien overlord and that something that could actually be self-enforcing and could create outcomes that wouldn't need a human to produce them or wouldn't need a human in the loop, but would convince humans to do X and Y. You've just got to exteriorize it into the infrastructure, which, assuming all of the smart contract stuff goes forward without any sort of unforeseen hitches, it would seem doable on that front. I mean, I'm sure that's not the only way in which it could happen, but that's an obvious one suggests itself to me anytime I think like how does AI cross back
how does AI and something in a computer cross with AI as something the capitalism is doing I always go to and my mind goes to smart contracts to the blockchain to buy yeah I mean but I sort of wonder whether using basilisks might almost be categorized like something like insider trading like coming as something that ordinary capitalism would have to discourage as being as being you know if you've got a whole bunch of these kind of house and software agencies and and intelligent corporations and some of them are using forms the basilisk basilisk psychology other edit then
there's is that not threatening such an extreme advantage there you know it's kind of extra normal isn't that extra normal types of stimulus that um could be unilateral yeah unilateral tactics that like force a particular response and don't allow for like the free decision that's basic to sort of market game theory and everything yeah yeah and the general culture the general culture of sort of liberal capitalism doesn't really allow for these forms of neo-theological hell threats and you know, these super, super stimulus on certain types of behaviour.
It seems to me at least it doesn't integrate smoothly. I mean, that's going to be true, because I would definitely not be surprised if that Basilisk problem plays out in other technological media as well. I mean, the Langford Basilisk, you look at it, it kills you, you know, seizure inducing, whatever. It sounds ridiculous, but once the neuromapping goes a lot further, it wouldn't be at all surprising if it was doable. At least in particular conspecific cases, you're able to identify sections of the population that were vulnerable to particular visual
hacking modes. There will be a lot of issues with that, and I would imagine that most of the time the solutions will be technical. You have to disable different basilisk methods or tactics on a case-by-case basis. JOHN MUELLER I didn't pick up on something that happened before. And if I'm understanding this, right, this is following up on this, which is...
Let me just sort of see this question. Sorry, one second. It was... Sorry, I'll just gloss it, which is how... Is there a... Oh, here we go. Sorry. this is Brendan's question where he says is there a max pain threshold that Basilisk can inflict before undermining its FAI credentials you know now we're on sort of this whole utilitarian philosophy terrain here and you know I think there's a, it's a lot to do as a lot of these things are, it's the other side of the Pascal's mug I think, to do with the problems that come with this anomalous extreme
quantification like you know the Pascal's Mugger thing is that if you are talking about a minute chance of a vast reward you're likely to be derailed rationally and the utilitarian thing is that there's a certain level where if you were subject to some almost unnoticeable slight itch for a trillion years it would be worse than being hung, drawn, and quartered within two hours. Like, you know, that utils kind of sum up neatly arithmetically and allow that kind of comparison to happen. So obviously, if you treat it that utils really are just quantitative
and neatly calculable in this way, it has to be that whatever horrors you inflict on a few recalcitrant software engineers can be redeemed even by the production of sufficient cans of baked beans you know or some trivial some trivial you trivial utilitarian benefit distributed to a large enough population over a long enough period of time now I'm not saying that I'm that that should be persuading people I'm just saying it's baked into the cake of utilitarianism that you're going to get that problem And I'm not sure that you can depart from it neatly. I think that all the hacks that utilitarianism has used
to try and get itself off that hook have had this extremely kind of ad hoc, unpersuasive character to them because they're so going against the grain of the basic structure of the theory. Now, I don't know whether that puts me onto the thread as it has then developed between you guys. Which does, it seems to be, kind of thrashing down. What was that last... You said that... I'm just wondering whether I'm getting the kind of grain that you're on with this discussion. Yes, I think that is fairly much.
What was that last sentence that you said before? I can't remember the last sentence. Well, I think just to say that utilitarianism doesn't really allow you to complicate the question of quantification without falling apart. You know, if at the point you say, well, one util isn't comparable to another or fungible with another util, and you can't say that a zillion minor irritations can sum to some instant of extreme agony, as soon as you stop being able to calculate in that way, then you just don't have anything left of that utilitarian machinery.
Right, so there has to be some kind of a limit to I mean, I don't know. I don't know. If it is utility maximilizer and a pleasure, it doesn't have the freedom to just... It's like infinite pain. Right. And be able to... Now, infinite, you're going to throw everything chaos. So we have to stop short at infinite, because all our calculations go berserk at that point. I mean, strictly in Cantorian terms, I take it, you could... I mean, I don't think you're allowed to do any of this that I'm now doing. But I mean, you could have infinite pain as some small fraction of a larger infinity.
I mean, infinities can be infinitely different in scale, you know. So, but I think that infinity is going to blow the fuses of our utilitarian calculator. so I think you have to say well is this some magnitude of extreme prolonged torment that becomes kind of morally on intolerable and you can you can argue that on utility and grounds I I don't know I think it's complicated I be interested to see that What about intelligence optimization grounds?
Could we make a more dangerous basilisk? Or a more effective basilisk? Yeah, no, I think so, because they would be completely unconstrained by this you tell her and calculus entirely friendly and which is the best way of you know of being subject to the basalots is to create a better because then you're compacting speeding up the loop I was thinking of what Moe was saying, all of these phenomena that appear to be converging
on or speeding up to the same effect as if we were already subject to the basilisk. Sort of in a larger, that and within the larger context of capitalism, this model for it, it's sort of interesting to consider whether these sort of, these convergences that are happening before we notice them, or before we see them as such, sort of prior reductions in entropy, viewing them as evidence that we are already under, already under the gaze of a larger basilisk. as actual retro-chronal effects that provide evidence for what it is that they are converging upon. I'm not sure how you could turn that into a formal framework
for proliferating this structure, but it seems like a start. When you say a formal structure for proliferating it, are you just coming out full-blown as a basilisk agent at this point, Jake? Or is that purely a hypothetical inquiry on your part? I can neither confirm nor deny. Look what we're starting here. We have to demolish this tape and not let it live on the internet. We would be wildly successful if we got to the point where it would be dangerous to put it on the internet. That would be awesome. We've got a half an hour left, so we better hurry up. I think we have been successful enough because if you look at the direction of the conversation
in the non-committal humanities towards these topics in the art world particularly but also in other branches of humanities, if you look at how much there was sort of like a false naive resistance to the idea of technology or sort of like, and whatever it comes with it, you know what I mean? A few years ago, like 2010, 11, when I began my research on these topics, and now you see a major shift in people sort of like accepting the inevitability of it and trying to, you know what I mean, work with it. Like I said a while ago, you I don't know what I mean. Everyone now, the level to which they want to publicly admit to it,
but also the level at which they think this acceleration or sort of like maintain or direct it towards different agendas or goals. But compared to, say, four or five years ago, So there's a huge conversion happening in terms of non-believers or those resisting these ideas to sort of accepting it. As sort of like fait accompli. Yes. No, I think there's been a huge shift. And I think even I've noticed a big shift in the sort of wider culture this year. to, for instance, on the whole AI question, it seems to have become an absolutely incessant media object
just this year on a level that I've not noticed before. Maybe you can give certain theories about why that might be. I mean, the whole Watson thing, which is a while ago now, maybe had a big effect. But, I mean, I bet I'll have to do one of those Google Ngram things. I'd be surprised if it wasn't on an absolutely crazy curve. It's entering public imagination in so many different ways. Just think of Hollywood and what have movies like all these films, right? Like from Interstellar to the, what was it, the Disney one about the future, about singularity and future with George Clooney.
Oh, yes. Tomorrowland. Tomorrowland and then you have Ex Machina and then there's also like less important but I think more significantly important movies like Automata and all these other stuff that sort of captured people's imagination in terms of like really thinking way ahead in the future and this is another thing I've been talking about with Jake of the class which is sort of like distinguishing a difference between sort of like what between theory fiction, which is something we know, it's coinage and usage, you know what I mean? The Cyclonopedia, Reza, and all that. And then this other thing, which I think is what we're doing in this class,
and I think, Nick, you did that in the previous seminar also, which I don't know what you guys think about it. Maybe we can come up with a better name, but I called it science non-fiction, which is not theory fiction and is not science fiction, but it has a little bit more sort of like reliance on like the nonfiction world, but it is in a way sort of like science fiction. So it's science nonfiction in terms of these types of white speculations and really sort of like simulations of the future that we conduct in your seminars, Nick. yes I think it's a really it's a very twisted sort of relation
isn't it like obviously take this topic Rokos Basilisk like on one level it's just almost purely literary I mean it's all these references to Necronomicon and these things are I think completely spot on it's a horror story and yet on the other hand it clearly is a sociological phenomena. You know, you can't, you just do sober microethnography of these kind of particular communities that we're looking at, and this thing is an unmistakable presence. And what to me, the funniest one of this, is you try to look up there's this long Roko's Basilisk
discussion at Rational Wiki, which is it's certainly not Wikipedia. It's usually got an axe to grind or whatever. I think in that case it's got some good links. It's an interesting piece. But you go to Wikipedia for Roko's Basilisk. There isn't an entry. It will send you to less wrong. And then you look at the less wrong thing. It's got history, a section called history. And the second, it's just two paragraphs. The second one is totally devoted to Roko's Basilisk. So there's something really strange going on there where it's saying the official sober mainstream Wikipedia version of what Less Wrong is as a cultural and historical and ethnographic phenomenon is that its history has been completely dominated by the appearance of the rocos basilisk.
and so you know it would be very I think the topological question a like fact and fiction here is extremely difficult to follow it's really me be and you know you start off on either side and you end up being on the other really quickly and so sure it's a fascinating distinction to try to hold together I mean the the actual sort of inner core friendly IA people who are as far as they're concerned respectable sensible scientists think that Roko's Basilisk has been
a information crisis and potential threat of some seriousness in a say I mean that's your sort of weird me be a new I think you know if you do have you'd cask sitting in on your and to poll panel and the whole panel has been chosen by the most show bus bunch of secure cracks you can possibly find in society no and known entirely for their seriousness and and lack of any kind of flights of fantasy and but you have within some short time in this discussion you would have this stuff in some way coming up and I qualify that only because as we've seen, to mention it is to succumb to it
and we have that kind of information paradox from there Well it's interesting that and maybe this is entirely contingent, but it's interesting that the first place or the dominant instance in which you see a retro-chronal effect or you see something like in a very explicit way, engineering on itself would appear as a threat or as a weapon. Going back to this idea that technology implies belligerence, that war and intelligence are intimately related, that the first retro-cronal effect would be that of a weapon, a causal weapon. Yeah. I mean, the retro-cronal thing is, in itself, fascinating in the sense that it's
kind of incidental. it doesn't there's no of all the sort of ingredients that's pushed into the tradition that gets to the record specialist one that is totally missing is anything to do with weird time topology time anomaly time travel you know all of that stuff which is obviously so huge in the culture around it you know popular culture and everywhere you look more back is fascinated by it but in this particular sign in the last wrong so they have extreme with information hygiene against sort of twisted time you know they and tend to be very very dismissive
anything suggested at and yet in a I'd obviously totally agree with you that with Roko's basilisk I'm this stuff comes crashing it you know from some angle that had not been anticipated you can definitely see that information hygiene is being incentivized or a preemptive defense mechanism for sure because I mean it's the all those all those flights of fancy and like and even more so the formalization of them or the making them less fantastical are all implied by what they're already talking about. As soon as you have the simulation
argument and certainly Roko-style reasoning about timeless decision theory, assuming that there's no difference between, you know, U0 and all of the different sims of U, you can start constructing, like, allegedly materially possible formal scenarios with these twisted time things straight away. I mean, you know, and refusing to sort of countenance collective imaginings about this that might lead to formalization without ever, without being basically a tacit consensus. It's interesting that that would have already emerged as an apparent defense mechanism prior to the introduction of Rogos itself. Yeah. I mean, I think it needs to be sort of
emphasized that obviously in the first wave of like cyber theory or whatever you want to call it, there was a huge promotion of the kind of notion of virus that was carrying this. And so these notions of information hygiene, I think, were there a lot. In that particular context, there are viruses everywhere, obviously memes, various kinds of contagious informational systems, processes, and invasions and incursions and intrusions and security in relation to information systems. So all of that was there. Computer viruses appeared. But this is escalated to a higher level
by the fact that it's a causal. I mean, it's escalated and it's regressed because this is kind of demonic. I mean, it doesn't, like a virus requires some kind of causal transmission mechanism. You know, you can insulate yourself against it, you can defend yourself against it using the laws of physics. You can have barriers, walls, screens, all of those kind of security protocols have efficacy against a viral menace. But against a causal trade menace, they are utterly ineffective, because we know that these things do not hop about and they do not get in on the basis of any kind of causal channel.
So, you know... They require causal conspecificity, though. They require... Sorry, can you say more? Conspecificity, like common historical causal constraints. You have to share a historic light cone. Or, I mean, certainly, you know, in order for something, in order for a superintelligence to simulate you, I mean, that is a huge entropy reduction requirement. So even at some sort of practical level, you certainly already have to be in the era of network accumulation. Yeah, to be vulnerable to it. Yeah. Only the omega point, only the Tiplerite argument can say that the entire observable universe is sufficiently conspecific to enable everyone to be subject to this.
I was going to the other thing I just thought it's interesting that kind of a causality it's interesting that it seems that it would correspond to double blind scientific experimentation or the double blind requirement of controlling scientific experimentation in order to produce statistically independent data I wonder if there's a way to look at this as an experiment or to craft a basilisk in which what the AI is presumed to be doing is producing experimental data rather than a weapon, rather than attacking or trying to subject to negative consequences past subjects.
Because really, to me, the weakest part of the actual Roko story is the idea that an AI would be incentivized to do this in order to ensure its own emergence. like without strong a causality, without actual time travel, I don't know, it's just, it's dubious to me that punishment to speed up the event of what's already happened and is already ensured for it, which is its survival. I don't know, it's problematic. Trying to get experimental data out of it makes more sense to me.
I want to also ask something here because I think it's the right moment to ask that, which is, Nick, what will be the answer to the accusation that only an anthropomorphic AI will think in terms of revenge and punishment? Right. Yeah. And an AI that is not human-like will not follow this sort of logical trajectory that comes straight out of humanity and almost nowhere else. Well, I think that's a really good question, but I think that question is like there in this whole thing.
Because the attempt, the decision theory is based on this notion about what is a strategy that wins for anything. You know, so the formulation of a sufficient decision theory should be radically non-anthropomorphic in any, if anthropomorphic means anything specific or particular or limited to our species, then if our decision theory is anthropomorphic in that sense, it's simply failing to be adequately abstract, and therefore in its own terms will be deficient
because we're defining it, if we're following Yudkowsky, as systematic winning. So I think we can assume that something that was not subject to parochial constraints would have more chance of formulating an algorithm sufficient for systematic winning than something that was. So that's just to say that I think that it's baked in right from the start of decision theory that it's trying to dis-anthropomorphize itself and work out what are truly strategically optimum decisions and decision procedures rather than those that we are comfortable with due to our own specific legacies whether they're being for whatever biologically culturally
or whatever people are happy with. But I think one other point just to quickly make is I think revenge is not really part of it. I don't think Roko's Basilisk wants revenge on anybody. I think it's just that. It's rational negotiation starts requires a balance. So it's a fear. It's knowing humans too well and playing out the fear. Yes. It's not that it's anthropomorphic, but that it is designed for a kind of anthropomorphic reaction by humans. Oh, by maybe any rational being that is sensitive to to I'm rewards and punishments you know how universal are rewards and punishments
I think that these guys they are it's a utilitarian structure they got they think that rewards and punishments are going to be basic to any understanding of motivation incentives for any possible rational be yeah so I mean I think I'll that's the level of abstraction is that is again we can quickly get back to this question about utilitarianism saddickly but I think on the revenge thing we consider international relations theory is there revenge in international relations theory if if one country says to another country if you do X we're gonna we're gonna make you suffer in some way so please don't do X now that's just a kind of absolutely standard negotiating stance if you then
do X to maintain your credibility you know it the time structure this is really interesting is if you're making that threat and it's a real threat you will do it if you're not gonna do it if you're just gonna make if it's just like the doctor strange love thing you know about that they can if they get new to world get blown up but they don't not telling anybody so it's totally dysfunctional as a game theoretic they have to look but they have the punishment that the game theory and it's like a it's the exaggeration on the other side of the expected which is I think I'm yeah hugely important of the fact that words are cheap you know you can
you can make threats all the time and its its the credibility of your threats that matter so if you're saying well if you make a threat and then someone does whatever is that triggers that threat and you don't carry through on the threat, or sorry, if you do carry on through on the threat, that counts as revenge. Now, I don't think that you can, I don't think you can make that move, because you're then saying that there's no such thing as a credible threat that isn't, that isn't simply the, you know, the working through of a certain kind of irrational psychological disposition if any actual commitment if any however it's working if any
actual execution a threat is to then count revenge because of course it's irrational I mean the whole mutual you look at the whole a mutual assured destruction basis of nuclear competition you know it's a absolutely gets a straight to this whole thing so you say look if you nuke us we are going to nuke Cuba you better believe it that's obviously again behind the doctor strange I think that now if someone does new Q and you new comeback it's just revenge is totally pointless and there's no at that point there's no rational basis for actually but if it was at all possible for everyone say well you know of course they won't really you can sweat because that would just be
revenge in that totally irrational therefore no one will ever need us back therefore any deterrent threat of of mutual assured destruction can be just discarded as being nonsense it's just empty words then you know there is no there is no deterrence so you're in this this is this absolutely call game theoretic hope the you have to be able to make credit rats if you can't make credible threats you can't engage in any kind of negotiation. So sorry, I mean, it doesn't go to some neat conclusion except the fact that when we get to the basilisk actually torturing people for eternity, that's just the same as we've
been to in the nuclear thing. If you nuke someone, they nuke you back. If you don't do what you promised to do and therefore get tortured for eternity, it's the same structure that thinking games you're an extract that structures to produce credible threats uh... night there why what i'm wanting to say the only thing i want to say is fun i think it's a really complicated fascinating twist issue but the only thing i want to say is what is i don't think that's what you support to a psychological category of that i think it's a rational game theoretic problem that goes beyond that Right, and then sort of my thinking on that or response to that would be, in that case, it needs a real objective.
And is what this AI is concerned with as far as we're concerned, really how hard we worked or like a difference of days or even months or years in terms of the moment of its emergence or the path of its emergence once it is already so powerful? or does it have more significant concerns? The only one that really occurs because it's what always does lately is the great filter. If it's xenogenic, if the problem is that war, like deadly war or xenocide is the kind of condition that reigns on an interstellar scale, then really the most we can do for it is to early identification
of what that is, or I mean, anything relevant to the war effort, the preemptive war effort, basically. That's the first thing that occurs to me, because there's not much else that we're relevant for, it doesn't seem like. Yeah, so I'm still not sure I have a 100% grasp of this. It seems... You're saying... Okay, let me just try and get you to run through that again. This xenogenic thing and you're putting in the framework of the great filter so you're saying this context gives it a sufficient motive for the kind of theological grandeur of this game theoretic thing how are we tied in
with the how does that tie in with the great filter thing again like it's trying to get us to behave in a way that we don't filter ourselves out of the universe It's protecting us. It's saving us from the Great Filter through the basket. We're trying to use us to model the origins of some of an alien about hostile alien superintelligence, which it presumes is a possible answer to the model of the Great Filter. Model its behavior, produce, generate tactics, identify,
you know, either keep signals quiet early on, or identify, eliminate possible explanations for the great building, you know, like, add more information to our identification of what it is, or refine the statistics of it further than the refined, you know, plethora that they are right now. Yeah. I mean, this integration that you're doing now, I've never seen before. It sounds totally fascinating, but I've never seen someone try to put Roko's Basilisk and the Greek Filter Fermi Paradox stuff together like this. I mean, is this something that you've ever seen before?
No, definitely not. Okay. I mean, it's a fantastic piece of engineering that's happening here. It's obviously like massive. So, yeah. Yeah, it's pulling itself together. It's still mostly loose threads at the moment, but it's getting there. Right, so just on the same line, I... There's kind of an assumption that we work with that because thinking of things as information is new to us, that this sort of information space is very uninhabited already, and the first sort of native occupants of it will be an AI that we bring into being.
But the sort of conceptions of this information space that you might get are in the context of maybe Greg Egan novels, where it itself is like an ecosystem that things can relate to each other in. I'm not sure if there's other thoughts on those lines, or if that's too much. When you say, because obviously, now Greg Egan, I think he's fantastic. I'm not sure which, like, is this, are you talking about a kind of interstellar ecosystem like that, or a more abstract mathematical, Tegmark, Spacey kind of ecosystem?
I'm not sure what you're saying by an ecosystem there. Yeah, well, I put it pretty vaguely. And obviously Egan's explored both. And some of the things you can do, maybe on those timescales, the inner still-out aspect can come into play because you have things that live for much longer, but there's sort of, I don't know, there's an assumption that reminds me in a way of the heliocentric sort of theory, that like because the sun is always right there, or like even better sort of like the geocentric theory,
the earth is right there, we're dealing with it all the time, therefore it's at the center of the universe, and we're the first people to discover it but actually there's a universe out there. You know, information is right there. It's always been around. The alphabet is right there. Obviously, you know, we're the most important thing to happen to it but we may well discover that there's structures or actors or agents living, for lack of a better word in the space that there's an alien encounter sort of happening right there yeah no I mean look I'm not going to pretend
there's not a haze quality to this but I think it's really right and I do think it does I do think this Riko's baseless stuff among other things definitely takes you there because like you know once you've said look basically where in this a causal trade with something could be anytime or anywhere then happen you in the way you're saying really pushed implicitly beyond this quite specific narrowly historical issue about you know that a I singularity we're gonna encounter in the near future like you know is if these AI people already engaged in some kind weird demonism
then and and and these references the Necronomicon become absolutely inescapable then very quickly we're saying well what about out doing demonism in we know the year 1000 or 1000 BC or 5000 BC I mean why are they not equally able to engage in a causal trade with this thing I mean okay they don't have what we might recognize as evidential decision theory but maybe evidential decision theory is just converging on some other competence a causal trade and a cause of traffic in any case you know I mean
and so yeah I think that that is the way it goes I think that was one particular thing to take it back in human terrestrial time obviously you can take it out in all these space time dimensions anyway you like so for sure that's what I was yeah I can't say I've got a good example but it seems like if we can think of one a causal sort of example yeah then which is okay has all these predicated assumptions fine but there should almost be a larger channel that we can open up with that, that may not be Roko's Basilisk, but that has potential. And yeah, OK.
I think it's vague enough that I should probably leave it there. But thanks. That's given me good stuff to think about. Yeah, it's good. It's really interesting. I think the Greg Egan thing is very relevant to it. You know, we have about 10 more minutes, so I think this is a good chance if, because those who haven't spoken and want to speak, if you have something to add, please go ahead because I can ask something and probably fill up the rest of the time, but I don't want to take up the time if some of you have points to bring up. So maybe I should ask it. So my question is, is there a way to sort of like, is there a way to save this logic
without it being too over-identified with sort of like libertarian capitalism or Ayn Rand or something? Is there a way to see anything emancipatory or like the affinities of this with some kind of like really like anarcho-mercantilism is like a marriage made in heavens and it's just there's nothing we can do about it? Well actually Mo, I would say this Rocco's Basilisks thing is coming from somewhere really elsewhere and I think it takes a positive synthetic effort. When you said the whole thing about the general capitalist mechanisms engaging in this automatic
disciplinary process towards AI production, now to me that is definitely, I think, has an affinity with these libertarian-esque anarcho-capitalist type of modes of culture, absolutely. But I think this Roko's Basilisk thing is much less clearly connected with that, and initially not at all. I mean, I don't think Roko is coming from that thing. I don't think Yudkowsky very clearly is coming from that, and he hates it anyway. It's a, you know, if there is an overarching framework, it's to do with this globalist, universalist, utilitarian, moral optimization. Hegelian... I think it's from an Anglo tradition, so it's much more in this...
I think Hegel would find it easier to object morally to the behaviour of the basilisk than these guys do. But for sure, I mean, look, we've got Amy here to bully on this, but it seems to me that rocose basilisk is at least as much left acceleration as right acceleration. I mean, you know, its goals is to optimize universally the well-being of humanity at the widest possible expansion of that term. So you know, I don't think that would be a problem. If Left Acceleration wants to steal Roku's Basilisk,
I think that's... It will cooperate. You know. You could do it from a Rothian framework, couldn't you? Like original positions, select the universe in which such a distribution in here's... Sort of like treat timeless decision theory as putting you behind a veil of ignorance as to the level of simulation that you're at. or the inter- I'm so sorry, I think I just need to take you, Jake, because even if I've got a slight thread of a grasp of what you're saying, but I'd be really stunned if you've got most people with you. When you say Rolfian, you're talking about sorry? Rolfian, like John Rolf, yeah. Oh, Rolfian,
oh, wow, I just was totally on the wrong thread with you there. Okay, because there is this Rolf something guy who you get to through following these workers a wall said yeah yeah yeah I'm sure I mean it's interesting how that would go either the veil of ignorance he's obviously can't hear more than a utilitarian is in the sense that it's all to do with it's all to do with I transcendental I equivalent all human subjects rather than this quantification of aggregate utility. So I don't know how...
Yeah. It seems a little bit more natively suited to this kind of scenario than utilitarianism does. And it's interesting that no one seems to have really followed him up aside from his sort of ridiculous argument for a certain public policy that he's ultimately making, like just his sort of method of turning the, of like politicizing the transcendental method or creating like a method of the transcendental justification of political distribution. I don't know, it seems very generic to me. Like as if there were a lot of ways that you could implement a veil of ignorance in an original position and then use it to argue
or model. but wouldn't in this framework in a Rawlsian point of view the thing that you would not want would be to be an evidential decision theory literate software engineer comprehending the thought of Roko's Basilisk I mean everyone else is off the hook you know there is this weird perverse election theory in this, that it's like if you're not thinking this thought, you're in the clear. So the people who cross the veil of ignorance and really get into trouble
are the ones who find themselves understanding what this stuff is actually about. Because they're the only ones worth torturing, as far as the basilisk is concerned. And then from the other side, so your preferred world is one in which the distribution of vulnerable states is maximized. It's an interesting question, isn't it? It's like, I mean, your preferred version is probably something just messianic where some poor, unfortunate scapegoat messiah figure gets the whole of the brunt of the spiritual responsibility and everyone else gets the goodies from the super high.
It's a little bit, it reminds me of Dune, you know, of Paul Atreides and seeing the golden path and the different ways it can go and being forced into this messianic position by the other memory. I'm not going anywhere with that, but that's what it is. Yeah, I think that messianism probably does take you there quite quickly. Can I just say, if no one has anything else to throw in, like I've been remiss at keeping people up to date with where we're going. And I'd just like to say for everyone who's going to be with me here, the great basic ski man now
is to go in reverse order as if passing through a mirror back through the topics that we've covered in I'm I hope so that deep thick and a more informed context so you know as the basic guideline is just flip your module one material upside down I will throw stuff at you I promise to get something tomorrow. But I think one serendipitous part of this is, of course, that we'll be dealing with the rocose basilisk on Halloween. So I'll kind of... I will chuck in some more
reading material for that, and then for the rest of the thing, as we then go back, ending up, disappearing in reverse into a super intelligence explosion in week A. I have a quick question. It's about the kind of core of the class and the title, which is what would security then look like in the face of this basilisk? I'm kind of just,
Going back to the Bitcoin class a little bit, because I really like the term of coding and over-coding and thinking about the wasp and the flower and thinking, okay, so if this basilisk is essentially hacking or hijacking us because we're over-coding and using us to save us, so how do we secure what aspect of our code so to speak and I'm not even metaphorical what would that look like and have we discussed that at all today or not a lot and I think it is a crucial question because what's interesting is both sides of this basilisk discussion think that they are
they are talking in the name of human security you know I mean If Yudkowsky was at the Anthropole roundtable, Roko's going to be knocking on the door and saying, why aren't I here? I'm at least as much involved in practical efforts to maximize human security as this Yudkowsky dude is. So, I mean, that's a fascinating structure. There's no one between them. Neither of them are dismissing it. neither of them are saying be sensible you know it's not a problem neither of them are saying neither of them are doing Dr. Evil and saying just let the thing
out of the box and see what happens they're both they both think they're optimizing human utility and taking a kind of path to AI that does that so yes it's excellent question and all the states this massively heated shism in the whole thing that shows that there is no easily obtainable internal coherence what is a human security policy So should we wrap it up?
We're a little bit over time, three minutes, or you guys want to continue on? Amy had a question. Amy, you didn't really speak today. I want to hear you. Sorry, I'm really tired. I didn't even remember what I was going to say. The one thing that I've been thinking of now, though, is the whole notion of identity that this argument requires to work. And just brought to mind that episode of Black Mirror, the Christmas episode where Jon Hamm is, his job is basically breaking in emulated, like M's, M workers. And there's this great scene where he's talking to the copy of this woman
who's voluntarily had herself copied so that she can employ herself as her own personal secretary. and this secretary kind of runs her house for her. But there's this great scene where she kind of goes through the realisation that she is the copy and kind of that sort of that particular episode relies on the opposite kind of possibility of identity, that there isn't a basic or fundamental identification of the copy with the original so that the original can then sell herself as a slave to herself or as a worker. So I thought that was an interesting kind of thing because it seems completely counterintuitive to me that I would really care
about what my simulated selves would have to experience. The point where it really gets under... That's not what he's saying though. That's not the problem. It's not that you would care what they're... It's that you would discover you were one of them. Yeah, yeah. That's what I was going to say. Yeah, sorry to... Yeah. Yeah. Yeah. Because whatever theory it is would say that like, you have two futures. Like you literally fork at the point where you've been simulated. Like if I'm simulated one second from now, like, and so there's the me that's still here and the simulation which experience is somewhere else. Like, there is no sense in which the simulation doesn't have the history of the me that exists at this moment.
Like, I have four. I'm both. Yeah. This is the most horrifying episode of Black Mirror, Amy. I love it. I had a nightmare the night after. This must be the second series, mustn't it? Is that right? It was a special episode. It was a Christmas episode. Oh, damn it. Definitely look it up. Yeah, it sounds great, obviously. Yeah. He basically breaks the resistance of the emulated worker by, like, speeding up the process so that she just experiences years and years alone in this sort of zero virtual space.
And then he kind of, like, flicks it back to sort of his time, and she's completely broken from what she's experienced as, like, you know, 300 years of solitude. Right. It's fantastic. Right. Yeah, it does sound very relevant as well. I mean, at least the mechanics of torturing simulations. Definitely. Anyway, I didn't really have anything to say, but I'm just kind of flopping around here in half a weakness. We were hoping that Luca will join us, but she never did, so... It makes you wonder, like someone said earlier, is it even possible to torment someone for the entire rest of the history of the universe?
And the same, something like that happens in that Black Mirror episode. A simulation is set on like 2,000 years or whatever it is, like indefinitely. It's highest set. Yeah, yeah. I don't know. At what point is suffering no longer something that's possible? Is even insanity something that is no longer a meaningful distinction? Like how long, if you have a mind, that isn't able to stop running, and you have a thousand years of continuous subjective experience, like, I don't know, does a superintelligence come out on the other end just by, like, forced evolution or forced permutation through everything that it can reason about its situation? I don't know. That's kind of an interesting question, whether you would expect a human mind to eventually, I don't know,
be able to hack its surrounding computational environment, and see the code, whatever, if you just left it in the room for a thousand subjective years? Or does it just get caught in a degenerate loop? Like one of the degenerate initializations of Conway's Game of Life, and it just sits there flickering on and off? Is this a kind of buddhistic point that just simply by existing it as being tormented. Yeah, no, basically that's kind of in Blackbeard that's what you're made to see. With minimal particularly or explicitly with minimal
stimulus or no new stimulus to be introduced. The only simulation that's supposed to run our house is just trapped in a blank white construct space for God knows how long. Oh, okay. Yeah, it's interesting. like the because the in the nation of imprisonment and torture not very obviously analytically related are they? I mean it seems like a kind of synthetic relation like, but that this is obviously implying it's more integral and just that just confinement Yeah, find existence as a kind of suffering. It's a refusal of embodiment as well.
It's kind of deprivation of any embodiment of her intelligence. The M's intelligence. Sorry, that doesn't help. No, no, no, but it does. I mean, there's a lot of ways that you can torture a state vector. I mean, that's why, that's like in Glass House, the sort of, like, distant sequel to Accelerando, identity theft is, like, stealing someone's full state vector, and it's the most horrible of crimes, because then you can do fucking anything to them. Anything. To a billion versions of them. And, I mean, if the body, that's a sort of interesting sense, Amy, that, like, the body, or the environment is part of the body, or is a body. I don't know, like, to, It says that depriving you of either one of those
is reducible in a way to depriving the intelligence of context or of stimulus, which is equivalent to a kind of confinement and which is intrinsically a form of torture. Causal isolation. I've got a feeling, Jake, that people are hazing out at this point. I don't know whether that's fair in saying that. I'm totally hazing out right now. We're about 10 minutes over time anyways. So maybe it's time to stop the broadcast
and thank Nick for this awesome session. Yeah, well, thanks, everybody. As usual, it was great. I'll definitely try and pump out some sort of material and think dark basilisk thoughts over the next week. And remember, we're going to be moving back in time, right? Yeah, that's right. No, we're now sliding back the other side of the singularity. Okay, thanks everyone. I'm going to stop the broadcast. Thank you, Nick. Okay, thank you.