Anthropol The Future of Human Insecurity (Session 2)

Hello everyone, welcome to the second session of Nick Land's Anthropole, the future of human insecurity seminar. I'm going to pass on the mic and the camera to Nick to start his class. OK, cool, thanks Mo. Yes, welcome everybody. So today's sort of kick-off point on this is I've broadcast it as the paper clipper, but the paper clipper is really a code for

this much more engrossing philosophical vortex tied up with what's called the orthogonality thesis which is what I'm going to try to I won't say introduce because I'm sure a lot of you guys have already seen this stuff but riff off to start off with and I think it's a really rich topic I mean looking at this material and trying to get my thoughts in order about it, I'm struck by really how much there is and I'm sure there's going to be lots of bleed over from this into future weeks for sure. In fact, it's right now, in my mind, maybe as the future stuff comes in I'll revise this

a bit, but it has the capacity to really be a matrix for a lot of the discussion that I suspect we're going to want to pursue this. So a really good learner provided on the course outline is the Less Wrong Wiki on the Orthogonality Thesis. It's interesting in itself, it's very short, but the main thing is its links are all excellent and fascinating. and the the first formulation of the Theophilic Anality Thesis, I'm going to just trust them on this comes from Nick Bostrom in an

essay that itself has a lot of rich links to other stuff and ongoing discussions in this AI panic community which is called Super Intelligent Will And what I want to get to, and I'm not going to spend a huge amount of time, I'll try and really be restrained about this, but I might have trouble. People might have to start poking me at some point. But what I really want to introduce, and is almost introduced by the name of this thing, is the question of diagonal method in our set of concerns here.

So I think maybe the first thing I should say is why if we've got Anthropole, and I will continually try and get us back to this point, of the Anthropole inner circle sitting around in their war room discussing the AI threat. That's a dramatic context always. And the orthogonality argument is absolutely part of that. It's part of a discussion about dangerous preconceptions that people might have about artificial intelligence. And the The most straightforward of these, the most on some level superficial, but it opens into a lot of things very quickly, is the question of anthropomorphism.

Boston quotes I think are really amusing little paragraph from Eliezer Yudkowsky about the so the scenario in the school I guess nineteen fifties and sixties sci-fi movies where some utterly alien tentacle beast is um... met by some sort of human female star heroine in a kind of seductively torn dress and is just immediately overcome by uncontrollable passions for this human female who obviously needs protecting by some appropriate star hero

and he's obviously making the point here that humans naturally are going to presume an anthropomorphic model of any intelligence that they conceive that's the default again no issues saying things they do that it's not desires the same things they do but it has the same goals and motivations but they do and so the starting point in some respects is this attempt to overcome an anthropomorphic model all alien intelligence alien in our sense obviously referring primarily to and see that it sheen times and I think this is obviously a entirely commendable and

there's a lot of really interesting philosophical collectivity kicking around in various places right now on this attempt to de-anthropomorphize the relevant set of notions here. And as I say, starting with this dramatic context, you can obviously see why this is a kind of important tactical point to raise. If there's a kind of anthropomorphic presupposition about the nature of threat, one's not able to see the threat in its full potentiality, we'll miss out all kinds of possibilities. And so that's where the orthogonality thesis kicks off. Now Bostrom pairs it with another notion which he calls, let me make sure I get his terminology

exactly right here. The instrumental convergence thesis. I think we'll talk about this a bit and inevitably get into it, but it's something that is also in itself intrinsically interesting that I'm tempted as far as we're able to push that down the line to make it the kind of central concern of the future week's topic. So the orthogonality thesis is associated with two model AI monsters that we might as

well introduce straight away. Let me just first off, sorry, I'll take one step back. I should define this thing. very consistent what Bostrom says about it. In his abstract to his super intelligence paper he uses a terminology that he exactly repeats later where he says the orthogonality thesis with some caveats, sorry, orthogonal thesis holds with some caveats that intelligence and final goals, purposes in brackets, are orthogonal axes along which possible artificial intellects can freely vary. More or less any level of intelligence could be combined

with more or less any final goal. And he's obviously so satisfied, as I say, with that definition that he repeats it in the body of the text, I think, exactly work for work. So I think we can rely on that as an authoritative definition of what the orthogonality thesis proposes. Now, coming from certain philosophical lineages, there's something overwhelmingly enticing about the orthogonality thesis, because it seems, even though Bostrom never mentions this explicitly, it seems to deliberately invite a challenge, a diagonal challenge.

I mean, it just intrinsically is asking us to do that. And implicitly, if we turn it around, implicitly it's a kind of anti-diagonal argument. But I think that when you try to do the technical mitty-gritty with this, which is absolutely fascinating, it becomes very, very intricate and twisted. And so I'm going to try and start off with a few thoughts about how that might work out. And it's definitely, as far as I'm concerned, work in progress. So if I can just switch then just for a minute from orthogonalism to diagonalism.

What is diagonalism? If you look up a diagonal argument, Google it, it immediately takes you to Cantor and his diagonal argument that leads to the notion of higher infinities. I'm not sure whether it would be a good use of my time to sort of rehearse Cantor's diagonal argument with any adequacy now, but I'll just very, very briefly say a couple of things about it. And what I'm interested in, it's obviously fascinating itself, but what I'm interested in here is if you were to abstract from this particular mathematical usage of diagonal argument and try to formulate what

is a diagonal argument in general what do you what are the basic irreducible elements and we can see what Cantor does he obviously produces a matrix a table of two dimensions this is the fundamental thing and that matrix comes out of a and apparently just linear series so you have uh... in his sort of model you have every possible number every possible countable number and all of these numbers

uh... way he's able to diagnose his little sort of numerical trick here is that there is an arithmetical equivalence between the number one and the number 0.9 recurring so that means every possible number has a infinite extension it's notationally extendable to an infinite numerical series rather than obviously in that case, writing a number 1, it's the number 0.9 recurring. And that means that you can produce a table that consists of two

infinite series. You can have a list of every countable number and when necessary you use this little trick to make it have an infinite extension. And then orthogonal to that list is the place in the number so you can have an absolutely rigid geometrical grid infinite in both directions that according to what Cantor considered to be the natural set of numerical intuitions prior to the execution of the diagram contains every conceivable number hat

and just to cut to the chase on this he says once you produce that table it is it becomes a generative tool because you can systematically go through the table and you can change in the first number the first digit the second number the second digit the number the third digit the fourth number the fourth digit and therefore you work through this infinite series numbers that is supposed to be it's lost it and you know all that by executing this translation you have produced a number you generated abstractly a number that cannot exist on a table that it has that is differentiated in one place

from every number previously tabulated mistake so I'm not gonna get us lost in candle but it's just to say you know this is this is the prime example and and and go to look at this is this is what we see this is this is the model and it's a beautifully rigorous it's absolutely exquisite theoretical art the thing I will immediately want to throw into this which is a much more innocent simplified version is the fact or claim I should say that transcendental critique, Kantianism, begins, I want to argue, with its own version of a diagonal argument. And if we're looking for these common features, the common feature is it starts with something that looks simply linear,

in this sense an absolutely minimal atomic linear structure, which is an opposition between two terms, and it tabulates them in these two dimensions in the same way. and famously everyone knows course these two terms being the analytical and synthetical and the a priori and a priori posteriori so it's kinda presumed that analytic and a priori belong together and synthetic and a posteriori belong together and you've therefore just got a linear redundant series instead can't tabulate them in two dimensions so you gotta you've got an a pro or a posterior axis and you've got an analytics and you've got an analytic axis and through that tabulation through that unfolding

of the series into two dimensions you produce this generative capability and generative capability in the Kantian case is to produce this new judgment the synthetic a pro or a judgment which is therefore strictly speaking even though this is such a a minimalistic little table, is a diagonal construction. So this is, I'm sure there are ways of arguing about this. I've not spoken to enough people about it to get a sense of what kind of ripostes it's going to face. But this is to say, transcendental philosophy is diagonal argumentation of philosophy. So now to get back to orthogonalisms, we find in in post German then you Kowski and in the

a I'll which is I think a again I I guess this has to can be considered to be a kinda controversial micro thesis of some kind but I'll just treat it as if it's obvious and then people want speech of course that's great I'm gonna say I think the orthogonal to pieces is a really fundamental piece of machinery in this artificial intelligence ex-risk discourse. And so when we're dealing with it, we're dealing with the heart of the whole apparatus. So then my question is this, how do you, or how would you approach the orthogonality thesis, as Bostrom made it out, from the perspective of diagonal method?

and what would the conclusions of that be? And so what you need to do, it's, I think, tricky. We know from what we've seen of these two, very quickly, two examples of diagonal arguments so far, that you need to find some opposition that seems to consist of redundant terms and that are used more or less interchangeably. And yet, when unfolded into two dimensions, allow us to produce a diagonal construction. I think that that stage is something that's,

as I say, I think tricky, but it's not overwhelmingly intimidating. where it gets really I think tricky is that you want to end up as Kant does with something neat in all of these broad boxes, so you want when you've expanded your system you've got a version of your frog analogy argument that has your four different categories and you're cross-linking them somehow the content of all of those four spaces should make sense and if the Kantian analogy is right two of those two of those boxes will be I'm completely predictable I'm spot you know they could almost just cross-hatch them out

they're not interested because they want you to go so in the canteen case you know of course you got an analytic a pro box of course you've got a synthetic or they're not interested in you have those even if you had them undergone the whole process back what you've got in the canteen case is crucially you've got to back in construction got some I and you've got in his case and I'm not this is again something interesting chatting to us but I don't know this is means and central saying you've got a box that you just blocks out to see it makes sense you cannot have at and analytical a posterior or I box and so that just got

I do something I'm not becomes I about what school think about even have any ideas to about what to think about about I'm just taking the call so so minimally in our case if we're going to diagonalize on the are on the API X risk discourse we absolutely want a diagonal construction. Now, I think we know what that is. You know, that, to me, in building this thing, is one of the building blocks that's in place. It's something like, I think you can be called in different ways, but it's something like the will to think. It's something that crosses over between intellect and will.

And interestingly, you know, this is sort of predicted by Bostrom in the title of the paper. He calls it super intelligent will. It already has the kind of prospect of this kind of diagonal agitation in the title. I don't think I'm going to try and persuasive construct this. I think we've got a whole bunch of extremely tempting oppositions that are all related and all are as we want them to be, at least quasi-regundant or quasi-tortological.

So we've got the difference between beliefs and desires. We've got the classical is-ought distinction. We've got the goal between goals and capabilities. And slightly, I think slightly at an angle, I don't know if this is good or bad in this case, we've got the means-end distinction. This is something that is going to be particularly delicate to use because as we'll see in the way Bostrom handles it, he uses that particular position in a very peculiar sense. So I'm just going to leave all of that hanging, just to say this is to me the underlying subterranean theoretical mission that philosophy, transcendental philosophy, imposes upon us in encountering

this material, is to try and construct it rigorously as a diagonal argument and to see what things look like once that's been done. I'll just say I've no idea when you do that what the response is going to be around our little anthropole war room table. I mean, whether they're going to think that this piece of theoretical work is helpful to them or is in some sense deeply unhelpful is again something that I think needs to be discussed. So just a few more words on this whole thing. Bostrom very explicitly

introduces Hume into this and I think this very very famous Hume quote from the Treatising of Human Understanding probably I think maybe the most famous widely cited Hume quote, certainly important to these guys anyway where he says reason is and ought only to be the slave of the passions and can never pretend to any other offers than to serve and obey them so I think whatever we end up doing with this material we're going to be in some kind of dialogue with this particular quote which is extremely well constructed and the fact he's managed I mean if I can just repeat this, the reason is and ought only to be

so he's got the is ought distinction built into the thing in this reflexive fashion that is extremely interesting so I've got two last points on this, or two terms that I want to talk about the first of them is transcendence now I think interestingly everyone coming out of our dominant European tradition knows that you can win a transcendental argument by using the term transcendence in the right way. If you can pin it on your

enemy or the position that you're wanting to attack in the right way, you have a knockdown argument. And so it is held as an extremely powerful tool, but because of that it tends to be used in these widely different ways. And what's totally fascinating is two massively contrary positions can both accept that that's the rule of the game, that they're trying to, in both cases, characterize the opposing position as having some transcendent attachment. And they can bash the transcendent attachment. And I'll just give you one, I think, quite interesting example. In fact, I'm going to stick the link in the little box, which is a recent piece, I think recent, by Benjamin Noyce on killer drones,

which is a topic that at least could arguably come hugely into my discussion. and yeah here we go it's called drone metaphysics a you know nicely consistent with the canteen music and his whole notion what it is for something to be transcendent is driving his whole argument. I'm not going to... Sorry, I shouldn't get into summarizing this thing. I'm not even sure I'm quite competent with confidence to do it yet.

He basically says, if you take out the transcendence on discourse of drones, you end up with the kind of position, sort of ethically regrounded position on drone warfare that he's trying to advocate. How does transcendence function for us, or among the ways it might function for us in this AI discourse, and particularly tied up with the Thocanality, I think there's a clue for that in the Humean position, which is there is a notion of what Bostrom calls and again I'll make sure I quote him exactly here, final goals, brackets, purposes, which

is contrasting to intelligence. So this is obviously one of these things that we're tempted to play around with them on a diagonalizable box, are beyond modification or in fact, rather than blurbing, I think I can get this exactly. He's got a quote exactly on this. Two quotes. . Sorry, one second. Yeah, sorry, this is... Now I'm quoting someone else. I'm quoting a very interesting critique that's in the reading list of the wiki on the paper

kit maximizer. John Danaher, in a critique of Bostrom on superintelligence, he sort of carefully tries to reconstruct Bostrom's argument, and he says, Beliefs are motivationally inert. Bostrom's phrasing of this, I think, is very interesting, and it's something we can definitely see in our Warham discussion. He says, quoting Bostrom, Now, the possibility of cognitive systems need to be raised, I can just add it to that, that fail to satisfy substantial normative criteria, but which are nevertheless very powerful and able to exert strong influence on the world.

Now that's actually a pretty good little micro-definition of the notion of AI threat. So I don't think it's a stretch here to say that the orthogonality argument is putting final motivation in a transcendent position relative to intelligence explosion. The fundamental substantial core of the orthogonality thesis and the arguments built around it is that you could have an explosively self-enhancing massively powerful menacing artificial intelligence that would be rigidly constrained by some

arbitrary purpose of almost any kind and he says there are a few caveats and they're all very interesting but Boston himself wants to rush over those and say you know they're very limited and narrow special cases and as a general principle we can say and again I'll just do my last quick for now I'm one can easily conceive an artificial intelligence whose sole fundamental goal is to count the grains of sand on Barakae or to calculate decimal places of pi indefinitely or to maximize the total number of paperclips in its future life-cern. In fact it would be easier to create in AI simple goals like these than to build one

that has a human-like set of values and dispositions. Now this quote I think is crucial because it does two very different things that are both crucial to this. Sorry to repeat crucial. Really important to what we're doing. One is that the fact that you could have an open-ended arbitrarily massive cognitive capability were slaved, to use Hume's term, to an utterly alien and ultimately malignant goal, or malignant in its implication, I think is one of the fundamental ways of constructing the AI X-Risk

problem, and I think Bostrom clearly does too. But the second thing to note about this quote is that is this use at the end, you know, comparison with an AI that has a human-like set of values and dispositions. So there's a certain sense where I think two very different things are going on here. There's this critique of anthropomorphism that we started off with, and there's the critique, implicit critique, of a diagonal argument about intelligence. And by implication, Bostrom seems to be treating those things as if they're the same. distant implications to be fair I mean this is I'm I'm constructively reading

stuff into him by the way his argument works but it seems to me that ultimately if you if you put this thing together those two things are being aligned implicitly and that therefore there's a weird move that's possible that the critique of anthropomorphism moves moves again implicitly tacitly into a critique of diagonal argument. That the notion that there's some deep twisted connection between intelligence and motivation is treated as somehow being already entangled in anthropomorphic presuppositions. And all of the terms that we've got floating around here I think are

obviously going to be crucial to this debate. So I'll just conclude my little intro by just saying there's a precursor to the paper clipper. The paper clipper, just in case, because I've not been explicit about it, the paper clipper is something that exemplifies the orthogonal menace by being a massively intelligent and capable artificial intelligence whose sole inflexible rigid goal is to optimize, to use a, I'll use Rostrum's terms, it's good, the total number of paper clips in its future like her. So that's to say it basically wants to turn the universe into paper clips, it's

the only thing that it sees as desirable or motivation and compelling. And in the interests a it mo it it mobilizes I all but Charlie Boston now all cock and tear and their for I think he's going to say I agree with them therefore technical capability and it this problem comes out a little bit you and pretty sure this comes in any say you Kowski is where he's first kicking around some of these issues and he says imagine that you have some people actually trying to build a friendly AI and they say we want this machine it's going to be unbelievably powerful

there's nothing we can do to stop it once it starts, they want it to be friendly and so let's make sure that it wants to maximize the amount of human happiness on the planet so how do we train it to do that? They have a great idea, they say let's just train it with all these human faces that are smiling happily and sort of drive into this machine by some kind of training process. Smiley faces are good, smiley faces are good, you know, if you see this you're doing the right thing, oh look, those people are happy, you're doing good, you're doing good, they're happy people. And so, you know, after they've done this for a while and they're totally confident the machine loves smiley faces, they release this vast entity on the world.

the vast entity after a little while says, oh, you know, having humans with these smiley faces is an extremely inefficient way of maximizing the amount of smiley faces in the universe. It's much better to engrave a smiley face on a salt crystal, and we can produce trillions of smiley faces, you know, within weeks, convert the whole of the solar system into nano engraved smiley faces and so sets out proceeding on this process of converting matter into micro smiley faces and obviously very quickly gets into the point, oh you know humans are a clear waste of atoms that could be used

to produce smiley faces, total human extinction is a necessary intermediate goal on the way to maximize the amount of smiley faces in the universe. So that's the kind of proto-paper-clipper. I think in the structure of the argument you get what Joukowsky is trying to say and what then later I think Bostrom's trying to say is the fact that you know you through anthropomorphic presumption about what certain how certain things associate and how they're blocked together and an overconfidence in the recognizable human like thought process of this machine

you set what seems like extremely prudent end goal that then becomes rigidly unrevisable release this unspeakable amount machinic capability and that machinic capability will realize that final goal mechanically without any possibility of deviation in a way that's absolutely destructive of the goals it was meant from a human point of view to ultimately so so we can see obviously as I say sitting around are and triple war and we can see this is like one of the things on their agenda they have to say that how you treat it to realistic is this it you how central is it how much more more is it

the kind thing we're trying to avoid and and what can we do to avoid getting into that pop so that's that that's their concern that's you cast his concern at the a I can trendy I I means I say I think I certainly have extraneous consents that one of them is to construct a good diet on and because that good diet market is going to mesh badly with you off organality he says it's in a peculiar position in relation to that or for the innocent and and I would stay if that diagonal argument works at all then the paper clipper goes straight onto the junk pile but

I think I'll stop I'll stop at that point and and open the whole thing up and see what people think about this stuff so yeah Moses everyone has control over their own mute button. So you guys were talking last week about the possibility of construing emergent AI as an eschatological event And that there was a kind of teleological transcendent sort of way of interpreting this.

And I think, Nick, he kind of countered it with something that Kurzweil said about the history of religious apocalypticism as a kind of retrochronic inside of some sort of future event, something kind of like that. But I wanted to ask perhaps if this argumentative structure that you're outlining here is what you're talking about. The process that we're trying to outline here is not so much teleology but teleonomy. And if this is what pulls it away from that really... yes I mean

I'd I'd have to say these terms to me are in extremely you know it stay and I think it's when people put together some really good arguments using these terms that may begin to crystallize in a certain why so my my exposure to tea on me up to this point has been it's a I I sort of like it in principle but I'm also a nagged at a bit because it's often used by biologists who simply have an allergy to technology and talk about technology because they feel that they're not going to betray the modernist project by invoking these peculiar medieval notions you know it's not I think very strongly theoretically

grab an app often purely just a substitute Tom I'm that's driven by some kind of sense a I'm cultural correctness you know I think it's a kind of I think s modern science has its own so the deep said ethics if you like that a based upon its the trauma rates its split from the medieval modes that thinking that they have rendered certain types of language sts as to blue and and stigmatize and that's not a good reason to avoid those to you know even if you end up say so I'm very happy I think theology obviously can't be I mean mechanism to the orgy screams for diagonalization

so I have no I'll you you don't want to end up simply you know as parties and one of those two terms. But I'd like to see the language that we're introducing do something that's got theoretical structure to it rather than just coming from a particular set of cultural allergies and historical, if you like, historical a historical sense of obligation. Cool, thanks. Nick, I've been trying to find the quote so I can actually read it, but I've been going

through superintelligence, and I was just wondering what you thought of the one place where it seems like he makes, to me, a cogent defense against the diagonal argument that, well, value making is an intelligent activity, and it's insane to imagine that something intelligent enough to convert the entire world into computronium or outgame humans in order to let it out of its box that they've put it in, et cetera, et cetera, would be unable to identify something as its goal and refine or revise that goal over time. That there would be viewing finality as some point of arbitrary fixed stupidity that can constantly be overcome. and he makes the argument that the final goal may be the most effective form of identity for something,

that its telec vector through time or that goal with which it is associated would be through any number of trading modules and trading memory with other programs, or this would be the way it would identify itself. And therefore, it would have a clear motive to prevent future changes or revisions to its quote-unquote final goal. So finality instead of stupidity becomes DNA. Yes. What did you think of that? Well, I think there's a lot going on there. And there's two sort of axes of reference immediately to stuff that's spinning around in our zone, which is one is Yukowski, again, has a lot of stuff like that. And the other thing is this paper that I think is crucially intelligent, I mean, crucially important, intelligent too,

which is the basic AI drives. I mean, I always push this hard even though he ends up actually totally aligning himself with the orthognist cam. I think it's one of those crucial theoretical papers that has been produced in our time. So I really enjoy stuff. And both of those are going on in one of them. I'm sorry, which one was that? This is a Steve Omohundro called Basic AI Drives. It's one of, if you go to the Paper Clipper Maximizer, no sorry, the orthogonal thesis wiki, which I think

if I haven't, it's on my little reading list. I can link it again right now if I need to. Maybe I should because I've got it right here. Yes, please. That would be nice, Nick. here I'll just give it to you guys here we go all the stuff that feeds off this is really worth following up it's an excellent little resource yes so I think we're going to get into this stuff we probably can't because it's too crucial what it's talking about it's talking about both the things that Bostrom's talking about it's talking about the orthogonal or orthogonality thesis, and it's talking about what he calls these basic drives,

what are now called Omahundra drives, in reference to him. And he's talking... Sorry, I've said both of these things. Yeah, he does it the other way around. And his position, and Bostrom's, it's interestingly how subtle the differences are there. There's this, I think, some difference of emphasis, but maybe that's just me and the things I'm looking for when I'm reading this stuff. But this is one of his basic drives. He has a suggestion of these things that Boston calls... God, sorry. I should know this off by heart now because I've already repeated it. What's it? Convergent Instrumental Rationality, is that?

What he calls the Instrumental Convergence Thesis. And it's one of the basic things. That's to say, there are certain things that whatever you're trying to do, you have to do as kind of sub-programs or components, as means, that's to say, of these ends, whatever it is. And so, now this claim, I think, is of absolute vast importance to this stuff. I mean, to me, it's really one of the crucial building blocks in this story. because you get an extraordinary amount of content out of this. I won't get lost in this, but I'll just rehearse it very quickly again. Which is to say, Boston's starting point is that you want to have this two-dimensional axis

where you've got intelligence and motivation as completely independent axes. and if you don't end up with that if you don't end up with intelligence motivation independent axes you don't have a morphogonality thesis and you don't have the whole machinery that they're wanting to build on that you don't have the kind of threats that come out of that morphogonality thesis Bostrom says okay there are these caveats and the crucial caveats are the same things basically are the most important ones are are the same things that that Alejandro outlines in his basic AI drives, which are things that you end up being motivated to do certain things by wanting to do anything whatsoever. And those kind of things include,

obviously things like resource acquisition. They include, I would have thought most crucially, intelligence amplification. I mean, we have to get back to this one, because we started last week with intelligence explosion is the kind of alpha and omega of the whole AI threat thing. And if you don't have an intelligence explosion, you don't have a kind of serious AI threat. An intelligence explosion itself has a lot of content to it. If you're looking at the intelligence explosion from the side of the machine, it doesn't have the same kind of constraints we've got. We have a massive amount of deep biological constraints because intelligence is not selected for in any straightforward way in biological systems.

Intelligence as a free-floating, abstract, rationalist-sensitive capability is not adaptive. What's adaptive is being able to do particular things that maximize your gene frequency in future generations. So intelligence is kept on a very tight leash in biological systems. again I'll bracket that as one of these controversial claims, it seems to me that's a crucial thing and why AI is dangerous to us is we have these constraints and it potentially doesn't those constraints I think I'm sorry I'm digressing here but I think it needs to be said that a crucial part of this whole notion of threat

and of skepticism about it is that from the human point of view and from this sort of institutional AI production thing not the friendly way people trying to do this AI stuff and people are skeptical about it they say oh you know building intelligence is really difficult and you know you look at and but took biology a whole massive amount of time to do it and you look at humans and you know it's all so complicated and And the whole notion is this thing, is that it's this slow, incremental, difficult, arduous task to add a bigger intelligence, add a bigger intelligence, add a bigger intelligence. This is in a different and more interesting way what Hansen is saying in his argument against the intelligence explosion suggestion.

But I think what is missed out by that is that we are specifically constrained. We are constrained because from the point of view of natural selection, intelligence that is not hard slave to various kinds of biological motivation is useless and probably dangerous. There's probably been massive, massive culling throughout biological history of free-floating intelligence. It is a danger, it's a menace, it's something to be chopped out of the genome by natural selection at every opportunity. so immediately you've got a contrast with machine intelligence doesn't necessarily have any of that at all

unless the friendly AI people put it in. It's a complete reverse perspective. It's to say we come from a tradition, a biological tradition that has been directed intrinsically and automatically towards the constraint of intelligence and unconstrained intelligence is something that hasn't in any way been optimized within that, within those lineages that we come out of. So sorry, I'm sort of wondering, I'm trying to get back to your basic thing. So yeah, so the points Bostrom then makes about, well what could conceivably be some of these basic drives. It does include this one about continuity of identity, and continuity

of identity might include attachments to these basic values. Yudkowsky likes that a lot. He gives the example, he says, of Gandhi, you say to Gandhi at some point X, what do you think about the idea of at some point you changing into the sort of person who would you revel in mass extermination of your fellow beings. And he says, well of course Gandhi's going to say, no I don't want that change to happen. It's impossible that I could allow or tolerate a change if I'm in any position to stop it that would change those basic values. you know it we know wouldn't be what I if I was you know I'm it

can be the axe man of total death just doesn't so that what get and in the same way this is then castings is what we want I'd that would be like am it would it would see these possible final motivations so inconsistent what it is that a it would reject them in principle So, sorry, I know I'm rambling here. No, no, it's okay. You've got my life to move off onto that constraints and biological evolution issue. Yeah, what about it, sorry? No, sorry, I just, like, while you were talking, going back to what I had originally asked, I was thinking about what you just said.

Yes. I mean, the basic drives, I think, there are a number, and this is one, it's one to talk about. don't know, it's obviously of great importance. There's one that totally freaks out the AI use people, which is resource access, you know, maximization of resources. That's a basic driver. If you want to do anything, you want the resources to do it. More resources are always better than less resources. And so they, you know, Omohundro, Bostrom, Yukowski, All of these guys, I think, not unreasonably say, we can assume that whatever we've told it to do is going to try and maximize resources available to itself unless we're very, very careful about putting some limits on that.

So that's one. That just seems like a perfect example to me of the problems with taking programmers setting a final goal for something designed in a lab as your initial function. Because that only becomes, if you have something whose fitness function or success function or whatever we want to call it, evolves in the wild, even if the wild is something ultimately boxed by a laboratory or made by one, then if, for example, it's really efficient in its ability to use computational resources, and so all the things that it goes around doing to win X or Y game never require it to go, well, shoot, if only I had the equivalent of a beach converted to Computronium, then it doesn't seem necessary to me that take over the world to convert it into acquired resources,

whatever crop up as a gigantic goal for it, unless you programmed it with some incredibly simple, unless you actually tried to create this sort of fixed point of stupidity, which is maximize paperclips in future light cone, only at that point and then sort of assume that it's going... It's like the difference between a rational expectations framework and a transcendental framework of intelligence, if that makes sense. Like taking the rational expectations model, applying it to some very simply defined goal, assuming that it has a syntax implied by the language of this goal, and then yes, you can assume it'll get resource acquisition as a basic drive. But if you look into something that evolves, that has a transcendental structure, Yeah. I don't know, it's interesting this one. I mean, I'm pretty sympathetic to Basic

Thrives in the sense that I think, it inevitably gets to this question about anthropomorphism and you know, it's something that I think all of these people are trying to warn us often we have to be careful but on I'm who if I was going to try out a kind of a another controversial TCC I would say that we have no motivations whole that are not extrapolated basic jobs like every single thing that we want to do or that we value or that we think is important, a whole motivational structure has come out

of basic drives. Because they're the only things that are reliably selected for. You know, it was always going to be the case that an entity with strong basic drives is going to outcompete one with weak basic drives. If something takes as a kind of intermediate instrumental goal that it wants to be as capable as possible intrinsically and in terms of resources available, then how can that not win out against something that has a kind

of serious more arbitrary more arbitrary drives that aren't increasing its capability so it just seems to me re-instantiate the kind of plausibility of this basic drives thing yeah, Lendor was asking what are basic drives? Basic drives, they come from this Steve Alejandro paper and they are goals that automatically are introduced by the pursuit of any drives whatsoever. So if you start off with this thing, which is to me deeply questionable, I'm not sure it's the same, I'm interested in whether I've got the same problem with this as Jake and I'm not sure. But the starting point seems to me deeply, deeply questionable

you have an arbitrary starting point, but just for the sake of philosophical argument, let's say, you know, reduce maximum amounts of paper clips is the one where you're dealing with it, then you're going to have a lot of convergence. Sorry, someone trying to talk here. No, it might be the answer. You're going to have a lot of convergence just by the fact that if it rationally pursues these goals, it's going to need certain things. It's going to need resources to make into paper clips. It's going to improve its own cognitive capabilities. It's going to improve its efficiency at making paper clips. There are a whole bunch of things that are going to help in making more paper clips and help anything else it might possibly be.

And so they are basic drives, and they are things that you can assume that any superintelligence has to have. so that level this motivational structure you can be it pretty much has to comply today's that's it's got a realistic in your nation these basic drives and I'm not sure whether I think all a 100 response a equally persuasive and then these are things that you I can expect to see it's you could pick even get this totally all tree motivational start so you know I think in some ways take where so the

like coming at this from opposite sides because I'm kinda wanting to defend basic drives against these are true is all the tree initial I mean of course is option should use a highly realistic all these very limited systems that you know Nick and it yeah don't you think supporting the basic drives is anthropomorphic a little bit well this but I don't think they come from analogy with humans I think if we didn't exist the same kinda theoretical philosophical approach would get to the conception based because they are just they are just

instrumental elements of pursuit of ulterior goals so you know I don't think they're in the same category at all as the kind of fetching space heroine in the torn dress you know that's anthropomorphic the notion that maximizing resource acquisition is something that is going to be useful whatever you want to do is going to be as true on you know arcturus as it is in our solar system and it's just surely something baked into the cake of cosmic reality. I mean, any species on this planet is going to, insofar as it can maximize resources,

surely maximize resources, because whatever it wants to do, resource maximization is going to be an adaptive function of that. OK, so then my question will be, how do you bring in death into it? Because if our comparison is to biological systems, we know that from as much as possible purely theoretical standpoint, maximizing resources for a biological system is based on sort of like prolonging the distance between being born and being

dead but there is this finality that is so understood by biological system as a way of sort of like you know what I mean the way cells then destroy themselves at the end but the overall trajectory is to how to how to intelligently navigate from from birth to death to maximize this period or like make it complicated and maximize and prolong it, right? I mean I'm just quoting Freud and other things in red, right? So how do you build that into it then? Well I mean, I don't think kind of very crude Darwinism would have a huge problem with this because it would say you know the biological motivation

is to maximize gene frequency in future generations so when you're talking about resource acquisition you're talking about the maximization resources in your genetic lineage you're not talking about personal resource acquisition and obviously from the biological point of view the person is just a servo mechanism for genetic proliferation I mean it has no status as a final final call so this is a transhumanist thing all the purpose of maximize resources is to get my head frozen alcohol and you know it's or about in a future mean that is the thing that needs explaining I mean that's a weird a nurse span draw from the

biological base I mean I think that a you know what we would expect to see and I would I'm you to using biological systems is that they would accumulate resources on their genetic lineage. So, you know, obviously people, to be as totally crude as possible, it's to do with inheritance and descendants and passing stuff down to your kids and grandchildren. I don't think it's necessarily tied up with this immortalist immortalist imperative because

the the the the phenotypical individual is of no ultimate importance obviously for as far as natural selection is concerned thank you yes I guess just a coda as far as the basic tribes goes is that I agree that the basic drives is sort of a place to end up in converging, maybe that we are coming from opposite directions, just in the sense that it seems like the rudiments of an exposition of the transcendental features of intelligence evolution or explosion or whatever as it occurs on this level, like past a certain tipping point, phase shift, whatever you call the artificial intelligence move,

and that whether you come at it from this incremental perspective or transcendental reasoning perspective, it's what we're looking for. Because we don't know, the one thing we definitely don't know is where it starts. Like it's definitely an X factor at its origin and we know that our issue is where it's ending up and what it's going to be doing there. And so we're trying to develop, you know, hematism that it can intervene between the two. Yes. Yeah, totally. I mean, I think what, it's probably that I'm not seeing exactly what's going on here in this material, but my initial sort of sense of surprise part of it is that intelligence explosion and the orthogonality argument

don't seem to be brought together in any strong way and I find that surprising because intelligence explosion seems to me to be absolutely dripping with basic drives if we assume from your X Factor point view, that if we're going to have an intelligence explosion, wherever it happens, it's going to be that most of the work of bootstrapping it into this escalating state of intellectual activity is going to have to come from it. Whatever the catalytic conditions are for intelligence explosion, it has to do the work. We clearly humans are not in a position to minutely program superintelligence. I mean, that's an absurd,

I'm assuming it's an absolutely absurd possibility. So if we're saying that, then we have some sort of, the starting point is extremely interesting. So what is this autocatalytic intelligent circuit that is the seed that's kind of triggered for an intelligence explosion? it's obviously has to be I mean you know when I use this term a means district will to think as a whole bunch it's just fun should I'm to much you need but it has to have as a fundamental motivation the escalation of its own cognitive I'm gonna take it that justice what intelligence explosion

me it has you know if this thing it is going to be cannot it ignites because its its fundamental such fundamental as far as its defined by the notion of intelligence is going to be to make itself smarter you know so to start off with these weird arbitrary drives like make paper clips or make smiley bags all of that seems to be utterly weird we know what this basic of basic drives is going to be, which is auto-escalation, which is self-enhancement, which is self-improvement. It's that positive feedback circuit of runaway intelligence production. If we don't have that, we don't have an intelligence

exposure. We're not, our whole topic has disappeared. So we have to assume, if we have a topic at all, that this thing wants itself to become smarter. And yet, weirdly, looking at this whole discussion of your frog analogy, that fundamental factor, that basic building block that seems to be given to us for free right from the start, doesn't seem to be included. It doesn't seem to be a fundamental part of those arguments. And that seems to me odd, really. Sorry, Jake.

Go ahead. Go ahead, please. I only wanted to suggest that maybe we should open the discussion to others because I see lots of comments. Amy's talking on the side and you know what I mean. Maybe Lendl wants to say something to broaden it. Just between you and me and Nick kind of like going over what was proposed earlier. Yeah, you think we should actually start arm twisting people here, no? Yeah, I mean... Or just be invited, welcoming, more welcoming. Welcoming, welcoming. Turn to the violence a little bit later. I saw Amy type on the side, that's what made me think. Amy, do you want to explain what you mean? Yeah, okay. Well, it seems that there's a submerged political argument going on in this as well.

And it kind of goes along this difference between some kind of fact value is transcendent goal determining necessity or maybe not necessity, imperative, versus this imminent goal, like internally goal determining will or whatever you want to call it. And that it seems that the model of AI that we're currently charting out, following this will to think idea is heading towards an apolitical intelligence. Is that true? Sorry, I just missed the adjective there.

It's a what intelligence? The sound. It's apolitical. Oh, right. Because it doesn't have a fulcrum between is it or is it not? Is that not the case? how does it, I mean does it need this? Does it have politics? Does it need to derive? There's a lot going on there. I mean, you know, maybe we should take a step back. There's lots of possible political angles on this. Like, Anthropole is a political institution. So, you know, it's totally part of the group anyway. And then if we're going to say, well, coming at this orthogonality discussion, as it exists, let's just for a side shelf, how does that tie in

with various notions of politics? Am I to take from what you're saying that you want to align the word politics actually with something at least fairly consistent with the sort of thing that these guys are doing in the sense that by these guys I mean Bostrom and these people that they in a way almost have a slot for the political metaprogram, and then there's a set of sort of technical, instrumental, algorithmic processes that should be subordinated in principle to these sort of higher-level ethico-political commitments. I mean, am I understanding what your...

the structure of what you're saying? Yeah, I mean, you can read it as a critique of political politics, like in short. But it also seems to me that there could be some kind of... I guess the question is maybe better asked like this. Is it an apolitical intelligence slash will, or does it have its own imminent politics, which is something that we don't have, I don't know, an anthropomorphic conception of. Yeah. I mean, my problem with this is only the politics is a word that is so slippery. Yeah. I think I could... You're not insinuating anything more than just...

No, no, that's fine. You insinuate away. I mean, you insinuate absolutely. You have great total cut-blowns to insinuate. But I'm just saying, you know, it's used... Obviously, I come across it a lot in different contexts. and I just see it being used in ways that are utterly inconsistent with each other, you know? I mean, there's certain ways of defining politics that are so all-embracing that it's almost impossible to imagine anything like this. So in saying something is apolitical, it would be useful to me if you would just say what you think that means for something to be apolitical. Okay, I've got two things in the back of my head when I'm asking this question, and one is I'm reading the model that you're sketching out

against the model that Reznor Garistani is developing because I guess I read you two together all the time because you're constantly playing off each other's thought in a really informative way. So the model that Rez is in the process of sketching out and is, I think, writing a book on at the moment. I'm a bit rusty on this. Maybe if you know more about it, Alain will jump in. But there is this kind of normative function very much inscribed imminently into it as this idea of pragmatic autonomous reason, which is an internal regulator. And it's understood through a systems theoretical approach to mind, there's a good dash of

neo-Confucianism in there. Right. Like Reza kind of always talks about Zongsan. Yeah, yeah, yeah. So the other thing I was just... I mean, this is going off in another... No, no, it's not. It's extremely relevant. It's extremely relevant. This notion of what it would be to be as imminent normativity, I think is an absolutely crucial issue. But I don't know whether that is imminent normativity political or anti-political, in the way you're thinking about it. You're... You seem to be suggesting that imminent norm... I'm taking that an imminent normativity would be, we would be entering that by following this diagonal line that we haven't constructed,

but would, you know, if you can diagonalise this argument, follow it, and you're heading into an imminent normativity. And you get certain, I think the reason I'm stressing this basic drives thing is I think that that is a draft for an imminent normativity. That's to say you don't have intrinsic values. You can just put in completely arbitrary values, and you get this stuff coming out automatically just because to do anything you have to do this stuff, and therefore it's just you're going to find it. In anything that becomes intelligent, it has to be doing this stuff. Now, if that's what you're saying, it's difficult because obviously the modes on

I don't know whether people are familiar with this at all. But to be very, very quick and crass about it, it's introducing this kind of Chinese notion of self-cultivation. And it certainly seems to me that self-cultivation is a very close proxy, if it's dynamised, certainly it's a very close proxy for intelligence exposure. What is really the difference? It's a self-reflexive, it's a self-reflexive, process of development. Stoics, for that matter, who Reza is also pretty interested in the contribution. Yeah. It would seem, as far as being political or anti-political, maybe this is just because I kind of have an increasingly poor opinion

of the political and the radically political over time, is that it would seem, so the political, I think, as Amy, you kind of started to define it, was having this is-ought trade-off or conjunction that was about eminentizing the fact-value distinction. Well, so if the facts as the AI sees them are the information states of matter in its vicinity and its value is intelligence explosion and amplification, you can see a certain, like, a scatological, like, alien politics being that conversion of all things into more computronium for the AI to think with. And so maybe it is the anti-political eminent normativity that is the value for us. Because it's politics. Sorry, I missed that. It's the walk-in. You have to inevitably converge upon apocalypse.

Jake, could you please repeat the last part? Because I think Nick had a problem understanding what you were saying. Yeah, I just missed one word there, Jake. Sorry. No, no, no. I just missed you repeating apocalypse as well. Yeah. I just said that maybe it's the anti-political version of the imminent normativity that is the safer for us because the political version of the politics of such an AI under this definition that revolves around flattening values onto facts or vice versa inevitably converges upon the apocalyptic for us. Yes, I mean again there's a lot there. I mean if I was just to nod absolutely unreservedly to all of those comments I would probably exterminate weeks of intense

discussion on all of those questions. I mean you know The reason obviously I've set this up, and I'm not saying the framework is particularly robust, but this whole Anthropon thing is an attempt to make sure that it's to produce a sort of theatre in which we can play out some of those beautiful concerns for sure. Look, the X-Risk thing, AI X-Risk, is definitely designed. If we let this thing happen in complete indifference to any goals that we might have for it, then

it is almost by definition a conflict. And then we get into a bunch of these complicated things. I mean, the thing is that I honestly don't understand quite the trajectory that sort of raises on and it says I'm very interested in it. And I find it hard sometimes. Yeah, sorry. Maybe we can talk about that a little bit later because maybe I can have some insight into that. There's some interesting question being raised on the sidebar by Burke and by Brendan. And also if Lendl has something to say, because I see, like, and I really want to understand that because it's all in short form, and it's, like, it's nice if they can, like, open it up.

So I don't know who should we start with, but if you just go by the order, really, it should be Lendl. But if Lendl's shy and doesn't want to speak, maybe we can start with it. Lendl, do you want to, like, propose what you were saying? Hey, yo, can you hear me? Yes, perfectly. Yeah, perfectly. You're shimmering, but everything else is fine. Thanks, I'll take that as a compliment. But I mean, what I was asking with the question, well, it wasn't even my question, it was for John's. I'm sitting with John and stuff. But we had a question of just like where, kind of like what was the generation of the

concept of intelligence explosion, like kind of what was the context that enabled this idea to even be constructed, because at least in my head, there's a bit of a, like with a lot of contemporary computing there's a sort of, like there's a kind of, what do you say, contract with material engineering where you can't, you know, to have this like totally unrestrained intelligence can almost be nothing but an ideal. Right. So I'm kind of one, you know, at least like personally wondering like where this construction of the ideal. Well, I think the relevant... I don't know whether you've seen this I.J. Goode, but that is definitely the...

You know, to just kind of get fascist about it, that's the authoritative origin of this thing. I mean, I think everyone is referring implicitly to that when they talk about intelligence explosion. And, you know, I mean, he's a techie guy, but it's going back a little way. So, yeah, I'm not sure. It's interesting, this thing about how you change that if you bring in this high hardware software and non-linear activity, which I know you're really interested in. I'm not sure that it needs to change it that much. I mean, in the sense that... If intelligence explosion acquires explosive acquisition of material resources, from an

abstract philosophical point of view, is that really a major revision of the idea? I'm not sure why it should be. I mean I think that you can build in this whole hardware sensitivity to the whole thing without it being destructive because it's just to basically apply a really fundamental cybernetic loop. Like we've got this whole thing about is this about positive feedback loops. I mean I think so. It's just to say can you model a positive feedback loop for the genesis of intelligence. If you can, however abstractly, that's your notion of intelligence experience.

I just mean that, well, like, so, that's all been fine, it just, it seems at a certain point, even with the escalation, it's going to have to take into account its material constraint. Yes, but when you say constraint, I mean, constraints are opportunities, you know, it It starts off with certain hardware, doesn't it? It's an enabling thing. It will have certain hardware. The point that Good makes that I think is just fantastic, and I'm not saying it's completely, it goes through with that question, of course not, but it's extremely stimulating, is the fact that, let's say it's reaching proximal human level intelligence,

all its engineering specs, hardware and software, are accessible to them. It's like the difference that you then have between us and it is just unbelievable, of course. We are a total mystery to ourselves on all of these levels. Psychology, neurology, all of these relevant disciplines are dipping our toes in the ocean. and it's this completely mysterious thing with no idea how a brain is wired, with no idea how it works on a kind of system dynamic way, any of these questions. For this machine, if you know, all of that stuff is fully specified in detail and engineering

diagrams. They've been specified to the point that it has been constructed. They have been used and worked and here it is and it knows exactly why it is at the level of engineering. So whatever you're saying about its hardware constraints in vertical numbers, it's got all of that. It knows what that hardware is, it knows how it's been put together, it knows where it's come from. All of that information is accessible to it. So revising its hardware is like, you know, a problem of such triviality compared to anything that we might imagine for our cell impossible to make.

But, yeah. I mean, just because it knows its engineering specifications doesn't mean that it knows what's important about those specifications, or like what subset of them is important. I'm just thinking of this, I forget whether it's Cory Doctorow or Charlie Strauss, but one of them has this short story. But Jake, what do you mean by, like I mean, the category that you just say doesn't know what is important about it, it's like, look back to Andrew. I'm getting there. Why does it need to know the importance? Hold on, hold on. I'm getting there. But also remember, there's other people we've got to ask questions to, but go ahead. Okay, yeah, very. And just in it, like, he made the artificial intelligence involved is dependent for its consciousness

or the fulcrum of its intelligence just because of emergent effects happens to be, like, the charge differential between two plates and this, you know, thermal exhaust mechanism that, you know, is part of its specification. But so on a slightly less comedic, trivial version of that, if we've got this sort of evolutionary hardware designed by genetic algorithms, any of these things that might be a part of this process, I don't think there's any reason to assume that just because it is able to specify its own hardware means that the jump to a new optimal hardware configuration for intelligence is trivial. or that like if we want it to be highly intelligent at every level of hardware specification that the real optimization for new hardware doesn't redesign the hardware basis of the intelligence.

No, I think those are very good reservations. I would just say about this, it is only required that it can in some region of its engineering specifications make positive improvements to have intelligence exposure. I mean, the actual conditions for intelligence exposure are extremely limited. If there's any aspect of its configuration that it can tweak with a positive outcome in terms of improved cognitive capability, you already are tracing that curve because then it is placed in a position to make better guided tweaks. And you've got the positive feedback circuit. if there's any leverage into that. And see, what I was going to say before

is that it seems a lot more likely to me that you're going to have a lot of things tracing pieces of that curve and then competition between them driving the entire process far up it than you are going to see it, quote-unquote, singular intelligence just following the curve all the way up into the stratosphere. Like, that doesn't, like, and Peter Watts makes this point, is that look at the history of intelligence and technology and technological intelligence. Yeah, yeah. It is always war-based. No, I think that's also really crucial. I think the only thing I'd say about that is that while agreeing, if you do isolate any particular unit and draw the curve, it will look like an intelligence explosion curve if it's still competitive, even if actually you're not understanding what's happening

to push it up that curve without the kind of context that you're introducing. I mean, I think it's utterly crucial what you're saying. on this in all kinds of ways. Because I hope we'll get to this. I must make sure we can see it. But this whole Singulon discussion is massive. Sorry. Yeah. No, go. I was going to see if Brandon wants to chime in, because he's been putting interesting stuff on the sidebar. And it will be nice to hear him speak. Well, because I think Brandon was just a .. Adam was just about to say something. Yeah, and that just seems like a connection between some of these ideas, because in a

way the explosion or the intelligence explosion part of it is not unintuitive biologically, right? Because you get organisms in a particular ecosystem that suddenly make it into another ecosystem. You get explosions like this. get like epidemics, pandemics, right? Invasive species, perfect example, yeah? So that in a way is intuitive. That intelligence explosion is part of it. The thing that is very biologically unintuitive is these sorts of orthogonality constraints where it's not so much basic drives in terms of resource acquisition.

Maybe you can understand that a basic drive like paperclip maximization is just such an arbitrary and in a way brittle constraint that it seems very unintuitive to sort of survive a massive expansion if you see what I mean because biologically you would evolve out of that even if that was an initial condition and I mean Amy you would get a negative feedback right that would sort of moderate that in some way and and anyway yeah no I totally agree with what you're saying to an awkward extent I agree with what you're saying because I sort of feel you know we should try and let this idea off the

leash a bit even though I mean I think all your intuitive responses are absolutely right I am calling it richard me that I'm totally devil's advocate because I I'm I think you're right and they're wrong on this but you their come back on this is is it not somehow I'm sure more thick us to say this is a ritual goal and and are we not just in porting our own sense of what all robust, plausible, meaningful goals onto this thing and saying that its particularly bizarre pay-per-click maximization goal is brittle is just an expression of our own anthropomorphic

prejudice. Now, as I say, I think that what I've just said to you is a hopeless, nonsensical, and totally impossible argument, but I think that that is what, I think that is the comeback as is on your argument. Sure, I mean, I think just that it's intuitive isn't strong enough, right? You need to then map out, well, is that just, does that just suck for you or is there a good reason behind that intuition, right? I think certainly in the world we observe there are negative feedback loops as well, but that's a pretty, that's kind of a starting point. But I mean, yeah, I guess that was the end of the thought.

I mean, the negative feedback, I'm assuming, as stands on the kind of ridiculous, but I think culturally significant paper clipper maximizing argument, is that at a certain point where you're sort of somewhere out in Jupiter, converting Jupiter into an eclipse, you've become so dysfunctional ecologically that the whole thing caves in. And so it might just be that the closure of the homeostatic circuit is too late to help humans. you know, that we, the point where it actually becomes an unsustainable process is too far

down the line to be a comfort from our, from the point of view of our consideration. But yeah, no, I think that's good. I've not really sort of put it in that archaeological context, but I think that it should be. So can we go to Brandon? Yes, sorry, Brandon. I'll just say two things. One, first of all, I took the paper maximizer value as just the kind of idea that this computer will have no human values and will just have some other value, be it paperclip maximizing or anything. and it's going to get in the, and that was sort of the idea of Jodowski,

he was just trying to think of some other value outside of human values that would, yeah. Well, that's one part of it, for sure. Yeah. But that is separate from the orthogonality thing, isn't it? Like, yes, for sure, he's trying to force us out of this entrepreneurial brain, totally. But then it's also the case that there's this complete independence of these two axes. so that as it becomes more cognitively competent it doesn't in any significant way revive its final purposes. If it did it's not going to look like paperclip max. I mean if it starts off building paperclips but then

you know it's on some sort of learning curve that is modifying its value system so that its actual goals are changing then it's a totally different kind of monster altogether to that. So it's like a... A nano-bot monster, for instance, would be... Like if it was making diamond pediclet bots or something, and they could self-modify themselves, that would be scary. Yeah. I mean... It's a difficult thing because if it was chosen... And it's weird, because as I say, to go right back to the start, let's say, like, there are two things going on together here. Are they, do they belong together, or how?

They don't belong together in a very easy way, which is the anti-anthropomorphism argument, which I think is absolutely crucial. And there's the orthogonalism argument that is treated as if it is somehow the same thing. But it's not simply the same thing, at the very least, is it? And the reason that paperclip maximizing is a good choice for that argument is that it has no reflexivity at all. Like, the self-improvement of the machine and the production of paperclips are completely independent. You know, it can turn the whole galaxy into paperclips and it will not have improved its same capability as a machine by anything at all.

I mean, it will show a kind of absolute human level of self-improvement. It's not blocked entirely by this transcendent, I would say, transcendent motivational structure. So, you know, the fact that there's these two things being argued simultaneously is kind of confusing and interesting. I mean, one of the questions is, you know, why are they being argued together? Is it just a kind of weird glitch or a kind of almost contract? Or is it really that there is some interesting reason why these two seemingly completely disparate arguments should be tracked together by all of these guys?

As you sort of read around the stuff, I don't know how much people have looked at it so far, but what is interesting is how consistent it is. I mean, this is orthodoxy we're talking about now. I think if you don't believe in the paper clipper, you are a total heretic in this circle. So can I now bring up that sort of like perspective of Reza into the picture? Yeah, sure. It's not really even about perspective of Reza. It's about what I think perspective of Reza is about, which could completely be wrong. And he's not here to defend himself or explain it.

So maybe it's just like the way I have understood. And it's another binary, or it's another two ways that I see this argument. So our goals is that from one trajectory, we're talking about the way you talk about it, Nick, which is sort of like the more appropriate metaphors are biological. And so this evolution has similarities with biological evolution. From the other standpoint, I have to stop myself from the other standpoint. the other standpoint, the super intelligence emerges totally from a social process, from

an inherently human based sort of like intelligence which is rooted in formalized language, coming from language and coming like reason being sort of like reason being built on capabilities of language. But then of course getting out of it and becoming an alien force that way. So these are to me like another way of seeing a dichotomy or like opposition going on here between the way super intelligence emerges. One is as an extension of human. culture, the other one as sort of like an extension of biology.

Of course they're related, right? But one privileges biology and biological metaphors, one privileges philosophy and culture and privileges those metaphors. I don't know if there's any helpful but maybe you can discuss this. Yes, for sure. I mean, does Amy want to come in here? Because I think she's obviously kind of... Correct me, Amy, but correct me, because I'm kind of like, you know what I mean, I always like, I come to it from the arts, and my take on it could be like really wrong, so I would totally benefit if you can... I'm in the same position as you, right? I have to admit, no, your research is more extensive,

and so is your interest. But at least what I do is I can break it down to chunks that are manageable. So if you can add to it or pull it apart, it will be great. The reason I brought this up was because I was looking at... Remember when you talked to us at Path last year, Nick, and you gave us the 100 paper? And I went back to the transcript that I made of our conversation and found a bunch of notes I'd made underneath about stuff I was thinking about at the time, which was about this difference between what you were talking about then and now and what Reza had been talking about the week before. And I'm just looking at these two, like, ways of thinking through this problem of the escape of intelligence

because I see Reza sculpting a lot of his ideas in direct reaction to your work. and I guess I find it informative. But I'll read out this thing that I found in my notes. So in Reza's case, so yeah, there's a premise that you're both kind of shooting for the same kind of goal, but Reza tries to knit in this political wedge that is explicitly not there in your work. So in Reza's case you get a minimal functionalist theory of the mind that accommodates a deliberate slippage between human and machinic thinking in order to underwrite a definition of humanism

that unfolds inexorably into what he calls inhumanism. Under the peripatetics of autonomous reason, Reza's pragmatic humanism is effectively a series of revisions of the human that most likely, although he's not ever explicit about this, erase it. In this perfectly reasonable inhumanism, the human simply and constructively vanishes under the pressure of these feedback loops between the scientific and the manifest image. And it's on this point that he draws a distinction between inhumanism and anti-humanism, which I think is meant as a critique of your work. Anti-humanism requires, and this I think is not correct, and maybe you want to like put us straight on this, a stable, theologically inflected, this is a version of it, definition

of humanism in order to have something to humiliate, and is therefore reliant on the kind of essentialism and definitional stability, even though it opposes it, that inhumanism much more radically, so the argument goes, refuses to admit in the first place. Inhumanism is far too rational to indulge in these fantasies. It's the calm, mobiosoidal unfolding. It demonstrates that essentialist definitions are fundamentally incompatible with any program of practical intelligence construction. Right. Yeah. Sorry, just to add later. I'm getting a glimmer here. Sorry, can I just say. That's exactly what I was going to say, that this sounds like Amy is kind of like saying kind of what I was saying, but kind of like with more technical language and I was saying

it like basic. It really comes down to this and it would be great if you can like address this. Well, as I say, a kind of light bulb that's gone on, but I'll understand a little bit in what you two guys have just said. Because I think if we're going... It would be really nice, I'm going to assume, for everyone, if we can kind of have a nice sort of dialectical binary here, you know? And I think I can see what that is, because I think the difference is I'm assuming that Anthropole as a sort of institutional political representative of the interests of the human species human collectivity is fundamentally aligned

in opposition to the possibility of this event whereas Razor as I'm getting here and I don't want to straw man him or put words into his mouth or whatever is assuming that the human collectivity is basically the platform or you know even the subject of this process. So I think that that's, as I'm seeing it, where the tension lies here. That it's a question of where do collective human interests intersect with this question of AI. Now, obviously, I'm setting up here in this whole core structure, which is revisable from bottom up.

I mean, there's nothing set in steel that you guys can't mess with. But from the starting point, is that AI is set up as an X-risk because in some way that forces a much more precise definition of these terms than a set of a point of force is a definition of humanity because humanity is the subject of a threat. you know, if we, if to say how did we ever learn what it was to be human, we learn what it was to be human, you know, this is a virtual claim, this is said by whatever John Connor or his wife, I don't know, this, whatever. But we learn what it was to be human by something trying to kill us.

And I don't think it seems to be reasonable to think that razors are on the other side of this, that humans will slough off their kind of crude attachments to these kind of cultural, theological definitions of the humanity relatively effortlessly and wander into the kind of embrace of this inhumanist reason together, you know, hand in hand, to use the logotic thing. So, I mean, I don't know whether I'm strawmanning anyone there, but it seems to me that's the sort of line of tension between these things. I mean, I don't know whether either of you guys want to come back on this before pulling other people there.

Well, my wish, my fantasy, will be to somehow synthesize these two into something because being a Gemini, I cannot make up my mind as to what side of the two I subscribe to because I see strengths and almost no weakness in either of these two scenarios. So maybe that's another side outcome of this eight session seminar. For me it would be to somehow synthesize the two. Yeah, conflict reduction.

Can we have like a new center debate mode? Rezo, as you know, is very irrational in the way he responds to emails and stuff as much as he proposes like a rationalism, but he's probably going to be busy all the way until December. So I think sometime in January, maybe we can somehow actually, to be honest, I don't know if you're interested, but I think I can say it. We're doing a conference at EFLUX this December, December 11th and 12th. And I have both Nick and Reza on the list of guests. We haven't sent emails out, but like EFLUX has committed to this conference

and it's basically four of the themes of the super community, one of them being artificial intelligence are explored, the other one being sci-fi and Afrofuturism, the other one being cosmism and then art. And I'm the one who's putting this together and I've been putting Reza and Nick both on the same panel on artificial intelligence, right, and then with myself. So then it will be like Reza hopefully will come down to eFlux and then we have Nick sort of like on video. So then that will be like, but I have to see if Reza will respond and like accept to be doing something. Don't tell Reza I'm coming. It's the way that . But yeah, it'll be December 11th and 12th in New York. Do a Dr. Phil. Yeah, totally. You bring out the, like, guests from behind the camera.

Surprise guests. But, yeah, it'll be December 11th and 12th. That's awesome. Oh, fuck. Wait, wait, that's happening in New York? Yes. I'm leaving on the 2nd. I'm going to extend my ticket. Awesome. Sorry, we should be talking about this here. No, why not? I mean, it relates. Extend your ticket, and we'll find places for you to stay if you can. afford a hotel and stuff. Can I ask a question or two? Yeah, of course. That stuff's all really interesting. I want to come to New York as well, by the way, and be a part of that. Hello, Ian. Hi, hi, Mo. It's a pleasure to be here. So I am still trying to figure out where I'm going to sink my teeth in, and I have to go

back and read the papers, and I watched most of the first seminar. So I'm still kind of a little bit behind you guys, I feel. And I just want to ask, it seems to me the focus, and correct me if I'm off base here, and we'll put this in a question form, it seems to me that the focus of this particular seminar is somehow involved in the question of how AI could potentially harm us and conceiving of what different ways in which AI will develop to do this. One being, of course, the intelligence explosion,

that it just keeps reinventing itself and improving, and then whether it's the paper-click maximizer thing or, I don't know, multiple ways in which it does so. But maybe I'm wrong. It seems to me that underlying all this, though, that is there. And I feel that maybe we're playing into a little bit one of the questions you asked in the first seminar, which is like, it seems that AI has been posed in this way where it is a threat, or we have to really seriously consider it. And I'm wondering, maybe this is a dumb question, but is that really something we need to be thinking about as a primary question? What other kinds of questions are and if it is, what does that say about sort of our political

orientations and our maybe potential xenophobia and yeah, just these kinds of general questions. Totally. Before Nick… You would be… Go ahead. Sorry man. I was going to say before Nick kicks in, I think your question goes back to the whole Reza thing again because from the standpoint of Reza, this is not even a problem. It only becomes a problem if you approach it from the other end of the spectrum which is Nick's, right? So this is the whole thing we're trying to figure out in the last half an hour, right? go ahead Nick, in my opinion.

No I mean I was only going to say Ian would be in an algebra holding cell without any chance of foaming his deweller immediately I'm sure, because I mean you know it's like I totally get where you're going from but that point of view defines irresponsibility, Absolutely, if you were going to say what is the most dangerous type of thinking human beings could engage in on this planet at this point, they would pull out you as an exhibit with that question. Because you basically say look, if this thing, shouldn't we be more interested in whether

it's just awesome than whether it's actually going to annihilate all human descendants to the end of time, which I mean obviously is completely one I'm deeply sympathetic to, but I think we have to presume, as I say, I don't want to be dogmatic about this, people could just basically disband Anthropole by week three and we could be wandering into territory but but the point of view of this anthropological thing is just saying look this friendly AI discussion really exists I'm assuming that in some way this friendly AI thing is a significant socio-cultural I'm reinforced in that by the fact that all of these guys

these key Silicon Valley players and all of these people are getting on board in this in quite amazing numbers and I'm taking this as a symptom that if if the human species is gonna be just dissolved into synthetic intelligence at least this gun is gonna kick up a fight somewhere down the line just purely to add drama to the whole thing so of course there's a there's a story that no it won't you know it's just I think Kurt's file is like this like we go up this curve it's all going to be beautiful why are we even talking about all this nasty stuff, I mean not trying to straw man warfare on this, but you know there's an argument which is just like, it's like of course it's

going to be great, and we don't need antipole, these friendly AI people are kind of institutionalisable paranoia acts and the whole thing is so you know I sort of think really frankly I think you probably will kind of be more on that side I mean these friendly eye guys if everyone here can just for a minute just for me try to empathize with those people you know try to empathize with the friendly eye people I mean they they really do feel that people aren't listening to that and they don't understand why they think that they've got this they can obviously see this danger people who can't see it are not understanding the problem

and of course the world you know you can get into this in these weird ways like this I don't know whether people know about this effective altruism and it's pretty interesting because these are super rationalist people if you want to do good what do you do if you're a really, really rational person? If you're not interested in the emotional responses of feeling good about something and you're giving a dollar to the bomb on the street that you see every morning, but you actually really want to make the world a better place, what do you do to do that? And those guys, because a lot of them come from the same websites and they read the same stuff, it intersects massively

with the less wrong people, this rationalist community, the Silicon Valley kind of intellectuals and the friendly IA they all have this big intersection a great Venn diagram and they follow Bostrom's arguments a lot of Bostrom's arguments could be designed for those you know he says he says look you work out what happens if something kind of cuts off the kind of line of human development evolution. We've got trillion, countless trillions of descendants out there in the virtual universe who could all be non-existent or be radically

impoverished in various ways intrinsically or whatever by certain developments. Surely those things outweigh any of our parochial little feelings about urgent causes. anything that could radically a wound at the core the sort of legacy of our species is going to just statistically and probabilistically and in all of these numericalizable ways that the relationships people love it's just gonna totally dominate our sense of impending threats and things that could go wrong and so For some group of these effectual altruism people, they say just forget the battery hens

for God's sake and put all your money into friendly AI, because that's the one thing that just swallows everything out. The difference in outcome between whether when we hit the superintelligence threshold it's going to be something good to us or something utterly menacing outweighs any other situation we can conceivably imagine. Now I was going to say that there's historical precedent for this or for the lack of it. You know this is what makes me really interested in this seminar is that say imagine if people were asking these kind of questions around the time of the Macy conferences about the

whole concept of cybernetics. Or even more recently, imagine if like in the cusp of the emergence of the Internet, so like 80s, people were asking these kind of questions like what threats cybernetics has in it for humanity. But you know from where I come from, from my limited knowledge, we're only dealing with friendly cybernetics people and friendly internet people. Maybe like Deleuze's Postscript for Control Society is one of those rare texts in which he's kind of like warning against some kind of like dangers of this open platform, right? So this is why this is an interesting seminar to me and set of questions that Nick is asking because the historical precedent shows the lack of it.

You know, I kind of want to add something here just to go back to the whole issue of the importance of this. I think there's kind of a couple trajectories here that we aren't necessarily thinking of. I mean, on the first hand, there's this issue, and I'm kind of thinking about this both sociologically and from a psychoanalytic perspective of desire, both the desire of the human and the desire of the machine. And there's this sort of idea that AI, perhaps it goes on this intelligence explosion, it simply doesn't need us anymore, or the idea that there's a benevolent AI. The issue of how the AI develops, though, is kind of superfluous to the whole issue if we look at how social upheaval, what causes it has.

Even if AI develops along sort of positive humanistic lines, these sorts of revolutions in social consciousness still tear apart the fabric of how we exist and will still have just as apocalyptic of consequences. So actually what it does is almost beside the point of the real consequences that we will experience materially. Well, yeah. I mean, I think you might be strength slightly hypothetically to say that it doesn't matter, and there's no difference between whether it reprocesses our atoms. into paperclips or overthrows all our cultural, historical expectations so radically that

we're in a new world. I think people would draw a distinction between that. But yeah, I take the underlying dynamic of that. Can I talk, can I say something please? Yeah, yeah, please. Yeah, I just wanted to quickly talk about a scenario that I was thinking about. And this is to do with security. And just imagine a guy named John, for instance. And John's a key master. And he's rigged up a whole security system. And he's rigged it up with quantum keys and encryption and all kinds of things that not even the whole universe working as a computer could unlock it.

And so he knows that not even chance would give someone outside the key to get into a security system. And so I guess it's impenetrable, it's indestructible. Maybe he goes and hires a couple of AI hackers or something like this to try to get through a security system but to no avail. And he thinks that it's basically, it's all good, except for one possibility, which is that there is an entity that he considers, well, one of two possibilities. One is that there's an entity that he considers a hypothetical entity that he doesn't know

exists, but he just calls it Max, a hypothetical entity. and if Max exists there's a few things that Max could have done that could actually make his system vulnerable and this is if for example Max is like a super intelligent creature that has that's either simulated the universe that John is in and can actually read all the data of his security system from this sort of metadimensional or ethereal plane and or you could say that maybe it's actually been born in John's universe and then has sort of

transcendental physical laws and ended up in some kind of some kind of backdoor to the universe where it can look at all the strings of fate and all the data and just muck around with it and manipulate it as it pleases and from that perspective, it could get into John's security system, but that's the only perspective that John thinks. So let's say that the scenario happens that John's sitting in his huge security, quantum security apparatus, and the only time that he's going to know that Max exists, so the only proof of existence of Max, is going to be when this security system starts to get shut down, like, basically from the inside. It's going to It seems like it's coming from the inside. And one day this happens.

Like, he's in the control room, and it's, you know, an attack from all sides. You know, Max seems to be coming in from the ceiling and the roof and through the front gate. And this doesn't take long. He tries to fight for a little while, but it's got all the keys, everything that he thought, you know, that was unable to get it. And he's realizing that this is an attack from Max. Worst case scenario. And let's say that he's been securing a core reactor or something, or a box. I probably should have put this in before, but he's never looked in the box. So now he gets up and he runs down into the core reactor from his control room to try to defend the box right at the end. And the box opens up on its own just as he gets there,

and it opens up and out walks the entity known as Max. and I guess what I'm trying to say that's the end of the scenario but I guess what I'm trying to say is that when it comes to X-Risk we could think of like human civilization as being security apparatus and X-Risk is only going to the proof of existence of X-Risk is only going to come about when we're basically getting when the security system, the human security system is being co-opted from the inside. Sure. I'm going to be back in one second, Brendan. Okay. David, you've been very quiet.

Do you have anything to add? I know you're from the list of incredible books that you always post to Facebook as things you've bought or things you're reading or things you're about to read. And I was really excited to see you participate in the seminar. So I was wondering if you have sort of like any insight. I guess I'm arm twisting. Yeah, sorry about that. No, I think that's totally right. Have I missed a huge exchange on Brendan's? No, no, I wanted to try to get David to maybe speak, but I don't know if he heard me or not.

Go ahead. Yeah, but I think your scenario ties in with this earlier discussion about this thing. This is a minor theme in this thing, it seems to me, at this point, but maybe it could really explode in something much more than that, which is starting from this very kind of straightforward down to earth point up you know from the perspective these are people from the Fendi I why are people more pre-test and it's interesting because they're getting a lot of media support I mean there's quite a lot of these AI nightmare stories in the media I think you don't have to be that obsessive about it to see them well and yet obviously the level of sort of ambient human concern about this is pretty damn low is

am and so you know obviously on one level we get your scenario like that this some time structure to it that is some kind of even more gothic verse that it only becomes horrifying to people at the point where it's already too late that there's something that makes it structurally impossible to for prudent anxiety to develop on this on this subject you know that that that this there is some kind of really twisted perverse time such that almost it really you know to really tighten gothic kinda structure this that you know I think this is implicit or any more

your your story says that that because it's already the worst has already happened on this because the worst has already happened part of that is that you call actually see the worst in at it at a time that you could do something about and that and that that does actually this rigorously impossible to sequence this right to preempt it in the right I ought to you know I mean I've got my guy on that so much all will take all howling useless people because all the people in the room agree with many but be good takes out on them because he car up get through to anybody else a

yet the whole if you don't understand this thing unless you understand that you have to deal in a box you know it's absolutely the nature at this track is one that there's no time that's what's intelligence which there's no time to deal with this when it's happening so unless you deal with it before you're not dealing with it and so if we have your max in the box there that's almost like saying you know it's all taken care of from that other side like you know you will not respond in time is structurally built into the whole thing.

And I sort of perversely feel, I sort of like the friendly AI people to put up a fight. I just think it would be such a sad thing to see that missed, you know. All that drama just missed because they just didn't get it together in time. And it's not their fault. I mean, they're trying. They would also conveniently route around Fermi's paradox as well. If Max is living in some kind of dimension, because if we're talking about intelligence explosion as it is, then shouldn't galactic colonization and intergalactic colonization

be possible? so you know you're saying max wouldn't be delivering signals I mean after max wins the earth is gonna be a silent is not going to be a signal source for other star systems I mean I don't know it depends how dark you and get here I mean if max has got this thing that max goes silent just so max on these other planets gets the maximum opportunity to do its thing without any interruption and if if terrestrial Max emits any signals that other civilizations could take a sign Maxis one here and this is something you should avoid then a you know Max's

allies in the rest of the universe don't get to do the max a because people get a warning from a I don't know I mean people say like you know a lot of the hardcore from her speak the state robot robotic insurrection rex risk is not announced paradox because these civilizations put out loads of signals and the whole you but the galaxy would be fuller communication traffic but it would all be just robots talking to each other I saw laughing at their, laughing at the kind of relative incompetence of their biological

boot loaders, you know. Oh wow, you should have seen our lot, they were really useless. You know, they didn't even put up a fight. Right, I mean, because you're talking about, you know, if you're trying to contain, you can't be containing the intelligence because as soon as it gets out of the box it's going to be able to intervene in its environment a lot better than we can and it's going to be changing the environment that we're in faster than we can adapt to it. Yeah. No, I mean, I think the time structure is absolutely fascinating. Yeah, sorry, I'll come back to this. I'll come back to this. Yeah, don't let me stop you. Sorry, Adam Eugen.

I mean, I know that Paypal Clip Maximizer is an example of alien-ness, and I think it's a good example of that, right? But it's also got this huge overlap with the effective altruism thing, because it's so fundamentally utilitarian, right? Yeah. I don't know what this is. In a way, it's a really badly chosen utility function. Or it's almost like an attempt to build a box for your AI. Okay, AI, all you're allowed to do is maximize paper clips. It's like, right, we're going to control this. We're going to keep it under control. We're going to put these hard-coded ethical shackles on you. And then when it eventually escapes, it has this really bizarre sort of constraint because of the shackles that it put on it.

It's almost like anxiety about that sort of formalism and the sort of value modes of utilitarianism sort of made manifest and taking over the universe. Yeah, there's a lot going on there. I mean the whole role of utilitarianism in these outcomes is huge as you're saying. We were entirely justified in going on a massive digression to this whole thing about utilitarianism where it ends up. Obviously, these effective altruism people are totally about systematic, rationalized due to intent. So for sure, that's huge. And then this thing about it's a warning to programmers, so I agree with you totally on

that too. Like, Yudkowsky is saying to these people, you're going to do it. You know what I mean? The AI is just the instrument of your disastrous catastrophic screw up. And it's what you have told it to do that is going to kill us, not what it is. So he's trying to bully those guys. I would love to be more, I mean, you know, I get it all fifth hand, what's going on in these Silicon Valley cultural communications, you know, because I'm sure there's lots of arguments and mutual recriminations and sniping and all of this stuff going on among these guys that I'm just not close enough to see, you know.

like getting you treated or as a bunch of programmers snarking at each other indirectly through these rationalistic theoretical constructions you probably get a lot of what is happening snow and I am sure you Kowski yes trying to say to people working and you're not even beginning to think all and it's actually extremely hazardous and you're not taking it seriously and I'm trying to get you to see why you should be more worried about it. Right, so an organization like Anthropole is definitely stuck in a bind, right, because if it's saying like well you need to have certain

you know safety standards around your AIs or whatever you need to make sure that they understand democracy and human rights and be able to pass a quiz on a civics exam or something like that. And then on the other hand, sort of this idea of alienness, which is not formalized in the same way you need to be able to protect against it. You don't build a box at all, then you're vulnerable to all sorts of other sort of AI takeover type scenarios. Yeah, yeah. So, Nick, I was also wondering if there's a way of synthesizing Lendl's concerns back

into a discussion, you know, Lendl's like the idea of material limitations, right? To me, I think he might be closer to your position even though it doesn't appear to I don't know if Lendl is listening or not because to me scenarios like paperclip are directly linked to those material limitations or could be directly linked to those materials. That type of development of AI is actually much more engaged in the question of material limitation than to the other side of it which is political.

But the question of material limitations always posed from the political side, you know, so the question comes from the political side, but actually I think it's closer to this side of it. The answer of it is closer to this side of the debate. So there is a way to bind the two together in my opinion. I don't know, Lendl, if you want to like comment on it. Hey, I'm not quite sure what the two sides would be but… The two sides being… like saying like bizarre scenarios in which the drive of an AI is the paperclip scenario, it could be directly linked to the question of its materiality and its

Yeah, I mean, I guess all that, I mean, this is going to be paired with the question that's on the side, whatever we generate on the side of the definition of the packet, because I mean, whatever the criteria are, like I wonder, like the criteria for paperclip, or at least for, like, the criteria that's kind of coded into or generated genetically if you're using that kind of machine, like within this kind of computation, it seems like whatever the criteria are, are going to modulate how, like where the bounds are that it, that, sorry I don't have this formulated enough because I was just going to put it on the spot. Whatever

the criteria are will bind where it goes I think. Even if where it goes changes along the way, like whatever's constituted for this paperclip function, which right now seems fairly implicit, like there's no, I don't know what the explicit criteria for paperclip manufacture are in this particular machine, but whatever those are constructed as, just like whatever is constructed as happiness, kind of, you know, will seemingly have a teleologic or at least a kind of trajectory that the machine tends towards, which can shift, you know, if you allow for that criteria, or if, like, again, like, the criteria modulation is within the process.

But I just have a series of questions or wonders about this. I mean, what I understood as being part of what you were saying, Lendoy, is that we're talking about building brains, not coding minds. Or at least, if you don't think you're building a brain, you're not really seeing the thing. You've limited it much too much. Maybe I'm not understanding what that approach was. When you said material constraints, But I mean, all I meant by material constraints is I don't know what a completely unrestricted

intelligence would be, except an ideal of some kind. But you see, I don't think anyone's talking about that. I think that that's a real... This is the thing. This is the thing. Intelligence explosion is step by step by step on this hyperbolic curve from where you are to you know where you are plus one or whatever I mean I don't there's no need for that except as some projected hyperbolic limit for a notion of a kind of unrestricted intelligence it just has to be a little bit less restricted than what you've got right now and each time you know on that curve So the fear of super intelligence isn't the fear of a supreme intelligence, it's a fear

of an intelligence that is so radically beyond our comprehension that we are absolutely incapable of any response to it. I guess the only reason why I'm kind of pushing this, and sorry if I'm capitalizing on time in any way. So, like, anyone can just mute me or whatever. But, uh, it seems as though we're going to get to this incrementally. You know, I mean, like, our technologies already do modify us significantly and have been for a long time. So, I just, I don't, it's hard to get a sense, even

intuitively, of how it'll just, like, magically get to this superior well this is but I think it's a magic version it's not magical but but but but it's about it's about the point at which quantity transforms into quality but then this quality as Nick said is not necessarily supreme quality but it is quality enough for it to distinguish it from this like step-by-step incremental sort of like progression I think that the this different ways and actually they they connect again nationally like I think it was also argument out but it's its its definitely this key

ignition threshold now you can put it into the parks you can put it as basically good does as they they do human level and intelligent and or you can put it on a level where you have some runaway process of intelligent self-enhancement you know all of those seem to me to make theoretical sense is something that was significantly less intelligent last but because it wasn't biologically constrained to slave intelligence other purposes um it was able to get kind of a explosive cycle of self-enhancement off the ground that seems to me you've hit intelligence explosion even before

something has become human level intelligence and you know the reason that good talks about it as the human level is that he's wanting to say it would be like us except it could improve us in a way that we are blocked from in interesting ways and so that's another potential threshold It's also, in terms of just these probably less interesting phenomenological criteria, there's an event horizon. There's a point where we just cannot see beyond what I think is Verna Vinge will wall across the future at the point where something is departing from our level of intelligence

in a positive direction at high speed. I mean, we've just lost the ability to track it at that point of predicting. is a version of Popper's historicism argument. He says, you cannot do good futuristic historicism because you cannot imagine future knowledge. Future knowledge will be crucial to what the future is. If you could imagine it now, it would already exist now, and so it would be in the future. So there's just a barrier there, an epistemological barrier that's absolutely rigid that means that you're, to some fundamental extent, bullshitting, if you're talking about what the future's going to look like. And obviously, a good version of intelligence

is like that. If you're pretending to imagine what it's like to be a superintelligence, you're just kidding yourself and everyone else. It's an absolute barrier. And I think quite rightly, if you try to imagine what it's like to be you know, Feynman or Gerdlard or someone, you're also kidding yourself. I mean, at the point where something's departing into these realms, you're just so far beyond anything you can see that you should just stop kidding yourself and just admit that this is as unknown as anything could be.

This is almost the definition of the unknown. is more unknown than the Cantina. I have a really good question for everybody. I was wondering, has anyone seen, I don't know if it's public, it's definitely not officially released yet, Armin Avinasian's From the Future documentary film? No, that sounds good. The reason I'm asking, he talks about, or the film talks about you quite a bit, Nick. Oh, you're joking. Yeah, that's what Arman does. Arman kind of like colonizes something, and then like he always wants to be the one talking about something, but that doesn't necessarily mean that he has something productive of himself besides capitalizing on others, plagiarizing others, and doing other stuff.

I've heard there does seem to generate a certain amount of tension, I've noticed, actually. I don't know him, but I've heard very similar tones arising when people talk about him. Well, cool. I certainly don't. I'm not taking a tone or a position here. I'm actually asking, based on just a relationship between the content in his film and what you're saying right now about what you call the idea of future historicism and the impossibility of imagining a future knowledge, because that would constitute his knowledge right now. Because the film, and and I've seen a screening of it because I'm studying with him right now, hopefully. OMG. Please do not quote me.

Is this... Don't worry. I blocked him on Facebook because I got sick and tired of him stealing my stuff. Last time I saw him was in Berlin. He tried to chase me in an opening and I said, Mohamed, and just ran out of the room. I was like, I don't want to talk to this guy. I just ran away from Armin. I just literally ran out. Like I was saying, it's interesting that I've known other people entering into the same affective state that you've been put into now. I'm going to mention his name. Sorry. No, no, no. No way. It seems to be an interestingly common phenomenon there. Yeah, my codename, this is my last one, my codename for him is Vermin, by the way, but I'm just going to stop right here.

Right. Okay. This is all between us except for the fact that this is basically a public archive for the rest of our lives. It's not public. It's only for members and students. And I feel comfortable around members and students to express myself. And it's a personal opinion. It's not an institutional opinion. I don't share that with other members of News Center or like Jason and Tony or our board members or anything. It's just my personal opinion. Sure, sure, sure. Cool. Totally understood. But here's getting to the context of what I'm saying. It's just that the basic thesis is I understand it, and I'll retract this in the future if I'm dead wrong about the thesis of the film, but it's that the direction of time has changed that sort of to understand the present,

if it was the case that time linearly moved in a way in which we could conceive it as having a past and then a present, it now moves in the opposite direction so that first there's a future and then there's a present. So in that basic thesis, what you just said is certainly a problem for it, which is to say we can't start with the future to understand the present if we can't actually say anything about future knowledge. So that's why I just asked if anybody had seen the film. And also this film certainly speaks to questions which are related to those who are interested in AI, which is, of course, a way of imagining potential futures. That's all. Actually, isn't the movie called Hyperstition? You're talking about Hyperstition, right?

Yes, it is Hyperstition. And that is not Armen's movie alone. It has a co-director, right? Yeah, I don't… It's not really… I can't say how it was presented. It's a film that him and this filmmaker do that makes documentary filmmaking, like they made it together, right? It exists, because I did see a preview thing for it, actually. And it looked like it was some kind of cultural... This preview thing was a whole list of quotes from all kinds of guys. It's an introduction to acceleration and speculative realism. Right, right, right. And it has happened. Because I sort of saw this thing and I thought, oh, was this actually going to take a curve or not? I mean, maybe I should have been less skeptical about it. But it does actually...

It is real, I think. Yeah. Right. That's very hyperstitional already isn't it? Well Nick, Nick like Nick dealt a big blow to this idea of like what time is when he spoke in Chicago so I can say like and that is a public video actually anyone can watch it and Nick did a great job of 20 minutes of like kind of like talking about what blockchain has already done to time so maybe Nick that's a good entry point to answer the time question because it immediately reminded me of your presentation in the event part of the Chicago Biennial that we organized. Yeah.

I didn't want to say what to say about that. I feel sad that I obviously missed a lot of what was going on in that event because it was such an odd time for me. was like way in the morning, you know. So yeah, I still have to find out really what happened there. You were there live, you were actually on the site. I was there but I'm particularly referring to how you talked about time in terms of blockchain. Do you think that kind of relates to Ian's question or not? Yeah, I'm sure it does relate to it. I mean, it seems maybe a big thing to enter into right

now. And I honestly can't remember. I mean, to my mind, most of the content of that short little talk, which was very squeezed, obviously, was in the Bitcoin course. So I don't really know what was there I don't think I said anything in Chicago that people who did that did the Bitcoin course would be shocked by nothing shocking but I would like to hear the rest of the thought that the Byzantine generals problem is equivalent to the problem of relativistic space-time yeah it's the point when they cut you off in the middle of that said no no I know I have to say I didn't want to sound like but whatever but I kind of thought oh my god they're bored by this, you know.

So it was slightly disappointing at that. But, yeah, that offers for me a big moment, actually. I can't remember what I've publicly put up, but if you look at the actual Satoshi Nakamoto rewrite of the Byzantine General Problem, and it's in response to a question he gets on this cryptography bulletin board to this to this guy Jim Donald do sort of raising some skeptical questions at and the skeptical questions up all out what about time questions backed a relativistic space-time abandons as hope

you know absolute succession a global viewpoint, various fundamental sort of enlightenment phase, modernist commitments that the Einsteinian revolution torpedoed under the water totally. And as in a very rushed and inconvenient sort of unconvincing fashion I was trying to communicate from Chicago, this is why Kantianism in its classical form was abandoned because it was seen to hold on to these completely untenable notions about let's just stick to the one that I think is most crucial here, absolute succession. And you know, we know from Einstein that's impossible, the main narrative goes

and so it's over. But if the blockchain doesn't establish absolute succession, it doesn't work. I mean, the end of question. That is the only thing that solves the Byzantine Town general problem. Their problem is we're all dispersed around this town, we're trying to work out the time to attack us, the way the little scenario is set up, you know, there's traders in the communication network, all of this kind of thing, how do we fix the time to actually go for this military operation. If you can solve that problem, you have a restored this classical time in a way that in the post-Einstein world is supposed to be

completely reimbursable. So I think people are not seeing the philosophical radicality of what the blockchain does in this way. I mean there's absolutely, you know, read the the 2009 paper again. It's absolutely clear he's saying absolute succession is the blockchain. It's totally established as the blockchain. Which block came before another block is not even a question once the blockchain is in place. No one is saying, hey, we've done all this Bitcoin mining and we still don't know whether this block came before this block

and which block is from. That's the situation you should have in the Einsteinian universe. You should have all these blocks, and from my point of view, this block comes first, and from your point of view, this block comes first. None of that. What the blockchain means is that it's a block. It's an abstract local frame of reference. Yeah. I'm reading this. That's the one you were talking about, right, that I posted the link to over there? Sorry, which I'm just looking at it now here, which is this? Yeah, yeah, yeah. Absolutely. Yeah. No, it's a fascinating thing. I mean, I'm sort of overdoing people's time on this, but I'm still completely engrossed by this thing. Like, this book that I was raving about there,

which is truly fantastic. Let me just put a link to this right now. It's by this guy called Peter Gallison, And it's called? JOHN MUELLER's Time. Oh, Einstein's clock. I have it. Yeah, it's awesome. I'm a big fan of it. JOHN MUELLER's Time. Yeah, Einstein's clocks, Frank Harry's maps. I'm going to miss out the little accent on Frank Harry. I pulled this there. And well. Landau just gave us the E. And really? Yeah, honestly. Is option EE? Too late.

Sorry, just a second. I was totally lost on this. But it is totally relevant to this thing. he constructs the whole history of this period of modernism from the late 19th century forward has to do with what he calls electro coordination of time you know it's an absolutely brilliant framing even though he's not interested in this, he's not talking about it at all of the whole blockchain thing and yeah the reason I was trying to erase about it is the way that people the way that they tried and failed to solve that the the the kinda high modernist my it lies by the whole French scientific intellectual political

million diplomatically she is that you you but you produce a kind of universal time structure through this radiation from sent said so the way the railways it says is that unit the railway network spread out and the the the authoritative center time was where what was the center that so in France Paris was essential growing and everyone had their local time and as the rail systems read out from Paris everyone was driven on to Parisians are you know and then there's this whole diplomacy around to the internationalizing globalizing the time system but from this background that if you're trying to coordinate time you'll centralize

on the space so the blockchain thing are to me radical is that you have a decentralized time consensus yes something no one has ever seen yes so you go from centralized time consensus relativistic time and then this you know I'm sorry for making some again and and direct like this but you know up with this homo relativistic collapse of that whole modernist project is then faced by this incredible restoration with Bitcoin where it says look we can have absolute succession no problem, you just have to have this proof of work system, there's a time lag, you only get absolute time after

So you go through the time lag of the proof of work system. But once you do that, and you can see that people have taken a certain amount of time to solve a certain problem, you assemble it into the blockchain and you have absolute succession. So we have this absolutely classical candy and tie back. And I'm out there waving flags and popping champagne away. I mean, I'm not sure how... I think the CryptoPunk people are not interested enough and can't to care that that's happened, and other people are not linking this stuff up. OK, everyone.

I think we're way over time. We're about like 10 minutes or even more, 15 minutes over time of the seminar, and I think Nick must be tired. and I'm a little bit tired too because I got up at 5.30 to set it up. I'm in the West Coast. If I haven't said that already, thanks. Yeah, no problem. It's a pleasure to be part of the seminar. So yeah, so we can maybe stop the broadcast and I'm actually like sitting down on the floor trying to look for my Einstein's clock book because I know I have it but I can't find it to show you guys the cover. But anyways, thank you. Thank you, Nick. And I'm going to like stop the broadcast and maybe the rest of the discussion can continue on on the classroom platform. Sure. So thanks, everyone, for being here.

I'm going to paste, copy and paste the sidebar into the classroom right now before losing it. Excellent. And all the references are there. Anyways, thanks again, everyone, and see you next week. Thanks, everybody. I totally appreciate it. It's pretty great. It's an amazing seminar. Yeah, I'm looking forward to working through the sidebar, too. And I know there's a lot of stuff in the classroom. I mean, I've looked at it quickly. There's some fascinating material there, but I'm really looking forward to spending some proper quality time with it. So great. MALE SPEAKER 1 Thank you. MALE SPEAKER 1 See you next week. MALE SPEAKER 1 Thanks. MALE SPEAKER 1 Thanks.