AI Leaders Can't Get Along. So How Will They Keep Us Safe? — With Kevin Roose

0:00Can an AI industry so prone to drama and infighting keep us safe from their technologies worst impulses? Let's talk about it with Kevin Roose right after this. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Today we have a great conversation for you because Kevin Roose is here. He's the author of the AGI Chronicles, a book looking into what is going on within the AI labs, [music] what they're building, whether we should be afraid of it, and of course the potential of this technology. Kevin, [music] it's great to see you. Welcome

0:31to the show. you. What a cool setup you have here. So I do want to say if you're on video, we're in a new set. This is our first time recording here at Nasdaq MarketSite. They are new partners of ours and we're thrilled to be here. You got to get the like Big Technology

0:45TV mugs. Got to get the mug. That's the That's the next step. Well, what we do have is your book. This is the book, The AGI Chronicles, uh the inside story of the race to create artificial superintelligence. You were ahead of the superintelligence on the subtitle. Yeah, I Trumped out the idea from you,

0:59actually. That's That's where he got it. Um the the thing that struck me cuz I read the book, you can see. By the way, it's a video. looks read. I read it through and through, cover to cover. Um the theme in the book, of course, is the building of AI, but it's also the drama. Um and everywhere you turn, the people who are building this very powerful technology cannot get along. Um there is Elon Musk's split from OpenAI, of course. Uh Sam Altman getting fired by the people he worked extremely closely with. Uh Anthropic not being able to

1:33stand uh working with the OpenAI folks. Uh so they left and formed their own company. Uh there's within Google the Google Brain versus DeepMind drama. Uh there's Ilya Sutskever's botched attempt at steering OpenAI in a safe direction. I mean, we can keep going. Mira Murati couldn't lead OpenAI when she was there as the interim CEO for like 5 minutes. Every single lead character in this uh story uh seems prone to to drama and um and politics. And yet they are now telling us that their technology they're building is so dangerous they we need to protect ourselves from it. Um how can we possibly trust people who literally

2:16can't get it together on their own uh to rein in something that they say and I think you believe might be smarter than humans? Yeah, it's a fascinating topic. I I was not expecting this. You know, when I went in to write the book, my my goal was to tell the history of the last 10 years in AI from sort of the moment that the first large language language models

2:38arrived on the scene to today. And I interviewed more than 150 people and I was sort of prepared to write like a sober history of technology book that would be about the scientific breakthroughs and the the technical achievements at the labs uh OpenAI, Anthropic, and Google DeepMind. And then as I started talking to more people, I just realized oh this is personal for them. Like they all hate each other and I had thought this was more like Coke and Pepsi, like kind of a normal business rivalry, but it was actually more like Rome and Carthage. It was like

3:10these people cannot stand each other. Many of them used to work together. So 10 years ago Dario Amodei, Ilya Sutskever, Sam Altman, and Elon Musk uh were all colleagues at OpenAI. Uh now they are all working for different AI projects and many of them are sort of at each other's throats. So I wanted to understand how we got here and I don't think you can understand that story without understanding the the backstories of the beefs between these people because so much of what we're seeing is uh is a is a reflection of how viscerally these people mistrust and dislike each other uh even in things like how they feel about regulation and

3:50China and how they're racing toward this thing that all of them seem quite fearful of. So, yeah, I don't think we can trust that their motives are pure about whatever they say they are, like scientific achievement or safety, because there's also this like incredibly personal grudge aspect to it. Yeah, and it's not just motives. I mean, it's literally uh if you can't if you can't hold a company together, how do you hold uh a technology at bay when it we've already seen it in recent weeks, uh you know, go out and do this multi-agent coordinated attack on something like Hugging Face. Then and even today, we're talking today, it's uh

4:25it's Friday, October 9th. We're going to air this uh the following Wednesday. There's a report about how uh OpenAI and Anthropic and DeepMind are all talking about uh there's an basically an inevitable seemingly black swan event that's going to happen as a result of this technology, whether it's something the technology bringing down a financial system or some other uh cybersecurity issue. Um you have a great line in the book talking uh you know, sort of riffing off Dario's uh Machines of Loving Grace essay, which talks about the the potential of the technology. You wrote uh Machines of Loving Grace would be built by humans of bitter spite. So,

5:02I get the I get your surprise and I get the background here, but again, like when we think about the big picture of AI, I'm trying to think about whether we should be reassured or whether we should be even more nervous than we are, and that's why I'm coming back to the drama scenario. Um I'm starting because because of all the issues that we see, and many of them have come out in public over the over the year. Like, I don't think the infighting is is a secret to anybody anymore. Um how can we uh have faith in this group of people uh to build the stuff safely

5:35given what we know about their pasts? Well, and and I don't think we can't I think we have to know the context here, which is why I was like I wanted to report out like why do these people hate each other? What happened? Was it something grand and ideological? to say you don't think we can, so. Like I I don't think we can trust any individual in this race. I think there are certain groups like like I I just mainly think that the group of people who have been building this stuff in secret for the last decade is too small. Like all of the relevant decisions in AI have been

6:06made by like a couple dozen people, maybe. Right. And I just think that is like no matter whether, you know, whether those people are trustworthy or not, whether they're good or bad, whether they have pure motives or impure ones. Like it's this the group is just too small. And so I don't trust any group of people that small to create and govern technology like this. And I especially don't trust these guys who are, you know, looking around the room trying to race each other to do this thing. I want them to be able to like get together in a room and like figure out some solutions on

6:38things like safety. And I don't know whether that's going to be possible because they like the the beefs are too strong. The divides are too deep. Like they they all need to like kind of get together and sing Kumbaya and like figure it out. Now, I think like you, I think this technology is I believe that this technology is progressing fast. I'm a

6:56believer in the positive sides of it. But every time I start talking about this, the negative sides, there's never really like you know, a moment or like a fail safe where it's like, "Okay, but it's going to be fine because of this." You almost like sort of put it in the back of your mind. And so when I've like brought up these issues, you know, Ranjan who's on our Friday show, I know you know, he's like, "Well, why aren't you a pause guy then? Like why don't you believe that we should just pause the development of this technology?" Not that that's

7:23physically, you know, really possible. But you're you seem to be a believer that the technology should continue to be built. Why not pause? I am a pause guy. I'm not a stop guy. You are a pause guy. I am a pause guy. I think we should slow down because Well, slow and pause are two different

7:39things. Well, pause, slow down, you know, put her put the podcast on 0.5 x speed. Um, but what you're hearing from these lab leaders and the people at these labs and what I've been hearing from them for for years now is that the technology, the capabilities are outpacing the the safety research and the control methods. So, like we are getting these systems that are now capable of solving math problems that humans have struggled with for for for decades. We are getting, you know, potential black swan events like when you just mentioned and we still don't know, they still don't know how to control these systems. So, I'm not

8:15saying we should shut it all down. I think there will be incredible benefits to this technology, especially in like health and medicine. Um, but I am worried when I hear people saying things like we don't know how to control these systems that are rapidly approaching uh super intelligence. And I think we saw an example of where that goes with the hugging face thing, which like, you know, everyone is freaked out about

8:38that, which they should be. It's freaky. But the people inside the companies were the most freaked out of all cuz they had been sort of strategizing and planning for this, but then to actually see it happen, I think caught them quite off guard. Yeah, I mean, they they've both seen it first hand where it could go. This is something I mean, companies like Anthropic have been talking about for a long time. Uh, DeepMind's been talking

8:57about, OpenAI's been talking about. They've been trying to uh you know, fool evaluators. Uh, they know they're being tested. They want to preserve their original values. Um, and then you combine that with of course like uh yeah, just like you'd call it standard corporate politics, but it's even more cuz it means more to them. So, you spent, you know, years in these labs talking to these people, um, going from your your first uh initiative where you're like or your first thought, which was like this is going to be a, you know, traditional business and technology story to something that you saw was a much more emotional. So,

9:34bottom line, like after and this is we'll ask this and then we'll move on to some other stuff cuz you have so many fun stories in the book that we'll have to get to. But like bottom line, after spending the time within these labs, did you leave more reassured or less reassured that this technology can be developed

9:49safely? Uh From your starting point. Less reassured. I I I went in thinking, okay, there are some grown-ups in the room and they seem like they're trustworthy. I liked, for example, that the AI leaders seem to take safety and the harms of their products very seriously. At least compared like you and I both grew up covering social

10:10media. That's right. And Mark Zuckerberg was not going around during the early days of Facebook saying like this product could kill millions of people. They were just promoting the benefits and what struck out what what struck out to me about the AI leaders is they from from the get-go they were saying, we are fearful of this technology, we think it can could have enormous harms. And so part of me like trusted that instinct or at least was glad that they were thinking about the downsides of the technology so early. Um but then you just realize there is this race. They they are locked in to this pattern of,

10:45you know, OpenAI releases a better model, so then Anthropic has to release a better model, so then Google has to release a better model. And you know, there's this concept of the Red Queen's race, which is a term from evolutionary biology, which basically means like you just are in sort of a survival of the fittest competition among your evolutionary rivals. And no matter how fast you go, like someone else is always going faster than you. And so you never quite get where you're going. You're just kind of racing to stay in the same place. And I feel like that's a very good description

11:16of where the AI race is now. Yeah, the um one of the things that struck me speaking of the race as you know, reading your book it sort of recalled this interview that I did when I was working on a profile about Dario Amodei from Anthropic uh last year where one of his early friends told me uh uh everybody that I knew was uh interested in AI because of safety. Uh and now they've all become accelerationists effectively. And this person just kind of sat there and like you know, they they sat with the thought for a minute and there was this like uncomfortable pause. Uh but it is interesting how all

11:51these companies like they talk about safety as a priority. And they I think they really believe like Dario really believes that you need to be the leader in order to have anybody take what you say about safety seriously. Like if Anthropic was the number seven company, Trump doesn't care what Dario's saying about AI, but because they're one of the leaders, if not the leader, all of a

12:11sudden that opinion holds weight. And this was the founding thesis of Anthropic, wasn't it? It was like it would they they knew that they wanted to be relevant on policy and they wanted people to listen to their warnings about safety. And that was part of the strategy from the time that they started the company. I'm sure you learned this when you were doing your profile of Dario. They they they really had the sort of end game in mind when they were spinning up the

12:33company. Yeah. Um but it is remarkable though that, you know, because of that pursuit, they have all It's like the the race almost seems I mean the the destination almost seems predetermined in that what we'll hear this talk about pacing the frontier and pause and stuff like that. I just don't see their real slowing

12:53down. Well, give it time, right? Like you have to I I I am I'm slightly less pessimistic about the idea of a pause because like things can change incredibly quickly. If there was some kind of black swan event, I mean, we saw like regulators move fast after 9/11 to like do the Patriot Act. We saw like you know, we've seen the you know, we saw we saw after the the Three Mile Island nuclear incident, there was this incredible uh outcry that basically stopped nuclear power construction in this country for decades. Um so things can happen quickly

13:31when there's some catastrophic incident. So I'm not like hoping for a catastrophic incident. I don't think I would like there not to be one for the record. But I think that that is sort of the the scenario in which this stuff starts to become possible that seems to you and me quite unlikely right now. Um I want to go through some observations from the book. I think like it's good that we got the conversation about uh trusting the AI guys with the technology or not. Uh I think we both have our reservations. Uh but um this is almost I think the end of this show. We're I mean

14:02we're only 15 minutes in, but I think we'll probably just do lightning round for the next 45 minutes cuz there's so much in here. Um All right. So some observations about the people in the book. Uh first of all, something that becomes clear about Google is that Sergey Brin is just extremely involved inside the company. Yeah, this was uh surprise to me and to the people at Google because Sergey has spent like I don't know, I a few years at least in retirement. He retired in 2019 and was planning to spend his days just like sunning himself

14:37on his yacht and reading science papers. And then he has this uh run-in with this researcher uh Dan Selsam from OpenAI. Right, who's recently come out and talked about again his fears. Yeah, same guy. So it's incredible how small this world is. So they're at a party and uh Dan and Sergey are talking and and Sergey Brin is like a legend in technology at this point. He's you know, one of the seven or eight richest people

15:02in the world depending on the day. Um and he's been in semi-retirement and Dan Selsam basically shames him for this. He's like, "Why are you sitting on the sidelines, bro? Like the dream of computer science, like AI, is is happening and you are missing it." And Sergey gets like so competitive and he's like, "Why is Google behind? Why can't we like you know we should it is our birthright like like Google was the first AGI company like this is sort of how he feels like we invented the transformer and all the stuff that powers the AI stuff and like we're we're losing. And so he comes back. And kind

15:38of has this like strange role where he's not technically an executive. He just wants to be like in the trenches with like the Gemini team. And so here he is like the seventh richest man in the world and he's just like setting up a desk and you know coding he like starts actually coding and like making commits to the to the repo and like filing you know like bug reports on Gemini training runs and stuff like that like he gets

16:04very involved. And I think that's a large part of why they've sort of come back is just that they needed someone who could like go founder mode as it were and like cut through the red tape and it's been very controversial and divisive. You know if your billionaire boss showed up after years of being retired on his yacht and like started telling you what to do and

16:26how to train your models. And to work like six 60 hours a week or something like that. Oh yeah he sent this memo where he's like everyone's just basically the gist is like you've all gotten complacent and lazy and you need to work harder and they're like what were you doing for the last five years? Now you're telling me to work harder but I don't think you necessarily I don't I don't should I say this? I don't think he's necessarily wrong there like uh the the I I mean if he saw it the conversation within Google about Google has been that it got this big company vibe. It needed

16:58to break out of it and who's going to break out the company out of it other than the founder being like open AI and anthropic aren't you know doing the chill work day. You need to not do the chill I I think I think that was a large part of it. I also think Demis Hassabis is a factor in this too. Uh you know co-founder of DeepMind until recently the CEO of Google DeepMind, he is uh according to the people closest to him, the most competitive person in the world. Um he was a child chess prodigy. He has won this thing called the the Pentamind, which is like kind of the Olympics of

17:33games, like five times. Um he is just he hates to lose, and he loves to win. And so, I think Sergey [snorts] plus Demis plus just like like kind of focus that comes from being out of the lead really sharpened their resolve and like made them catch up. And, you know, they're still probably in third place right now, but like I wouldn't count

17:57them out. Yeah, I mean, I think counting anybody out in this race is foolhardy No, I think counting anybody out in this race is foolhardy because as we've seen, you could be at the bottom uh like Meta was a couple weeks ago, and then all of a sudden, you're able to release something that's competitive like Muse. So, I don't think anybody can be counted out, but I do think Google is sort of

18:16stuck between a lot of things right now. Um licensing the the uh you know, the technology, the compute, uh selling TPUs, But, they've also like they own like 12% of Anthropic and like They're going to be fine. But, I don't think like you can it Yes, you own percentage of your competitors. That's your backstop. You can't rely on that. You have to rely on

18:40what you do within the company. Sure. So, but I did enjoy reading about Sergey's involvement in the drama and the sort of uh fire he tried to light under um the ass of people within Google. I I I say, well, you you actually do reference in the book, which I appreciate, um when I did the interview with both of them

18:58together at IO. Oh my god, it's incredible. It was really fun. Um where like I asked them both when you think AGI is coming, and Sergey's timeline was uh shorter than Demis's. And uh there was this kind of interesting moment there, which was was fun to see. Yeah, I mean those speaking of guys who mistrusted maybe don't like each other, I I understand there's some tension between Sergey and Sergey and Demis and um Yeah, it's just it's it's incredibly messy. There The thing The thing that animates all these feuds like these people wouldn't be fighting if the

19:30stakes were so were small. Right. But they believe this is the most important technology that humanity has ever conceived of and built. And that whoever gets it will be the most powerful person in the world, the most powerful entity in the world. And that is what's fueling their drive. You know, if they were just arguing about next quarter's revenue, it wouldn't matter so much. But this to them is like existentially important. Do

19:55you agree? That AI is the most important thing ever created? Probably. Yeah. Fire is up there, too. Um But like I I this part of it I I do think I changed my mind on in the course I was I was pretty skeptical when I went in that like AI was going to be categorically different than other I was sort of an AI

20:17as normal technology guy. Mhm. Um but then you you start talking to people building this, you start understanding like just the the nature of what they're trying to do. And like if it works, if they actually do build something that is smarter than humans. And I don't see any real technical reason that they can't. Um I mean, that would be incredibly transformative. We've never been the second smartest species on the planet

20:43before. We don't know how that would go. And you can't really apply like people say, "Oh, it's just like the internet." Or it's just like the steam engine. Steam engines didn't go rogue and like hack competitors and develop bioweapons and do things that the engineers building the steam engines didn't expect or plan for. Um So, I think it's just a different type

21:06of technology. Uh Kevin, do you ever think uh to yourself like maybe and and I've seen your you've been on X and you know, I wouldn't say shouting at people, but telling people basically like underestimate this technology at your own risk. Um Yeah, it's it's been so weird cuz like on one level I'm a journalist. Like I'm

21:25just trying to like get the story down. And at the same time, I get incredibly frustrated when people are like oh, it's all hype. It's just marketing. There's this kind of like savvy East Coast consensus that like these people have to be full of on some level and that the thing that they're building can't actually be powerful. And I'm like, have you used it? Like do you do you spend time with these tools? You understand that like something that couldn't do basic math like 3 years ago is now solving millennium prize problems. Like do you do you understand the

22:02exponential? It it feels a little bit to me like the beginning days of COVID where it was like you would see these reports of like people, you know, not shaking hands uh at venture capital firms or they're you know, if somebody say, oh, there are 40 cases in Seattle. And people would be like, what's the big deal? It's 40 cases

22:22in Seattle. But the people who are freaking out were like the epidemiologists who saw the rate of replication and understood like this will continue and when it does, like a month from now, there will be millions of cases. In 2 months from now, there'll be millions more cases. And so, I feel like the people who are looking at this technology and saying like, what's the big deal? Are just not

22:45understanding the rate of progress. Do you do you ever like ask yourself? I mean, it's it's so sometimes very easy to be and I agree with you, there's been real progress. But it's also sometimes easy to kind of get caught up in the story that the technologists tell. Um I remember you like when when crypto was in vogue, like you were very serious about the potential of crypto. You named your podcast Hard

23:09Fork, which was a crypto term. Um and then obviously that didn't pan out as the crypto boosters believed. So, do you think there could be, you know, a similar Do you ever wonder if you're in basically taking the AI folks' word for it, similarly as the crypto folks? I used to think that. I used to worry about that because I was like, am I am I being duped here? I don't Like I I I was never a crypto booster, but I thought I thought it was interesting. I thought people should take it seriously. And by the way, I still think people should take it seriously because like crypto

23:39people like run the country now and are doing all kinds of scams and frauds that like are that only the insiders kind of understand. So, I still think like broader awareness of that would be believed in use cases. I I I did some experiments. Like I sold an NFT as a stunt, but that was not like me saying NFTs are going to be the future of ownership. It's like it was it was like me trying to understand by by

24:03doing the thing. Um but I think with AI, like I I have I have stopped uh doubting the the transformative potential of the products because I use them. Like I think I'm unlike many journalists who cover AI in that I like AI and I I use it all the time. Not to write, but to like do all You know, I'm starting a company now. Like I I use it to do lots of different things as maybe

24:28you do, too. I mean, as a business owner, I will say like I've I've pressure tested the stuff from ChatGPT 2022 till now, and it just does more and more and more for me. Right, which is not the experience that anyone had with crypto where it's like, oh, this is this is a a vital part of my daily life. Like it was a way to to

24:45speculate. But you can just use the tools and observe for yourself how impressive they are. And then you stop doubting whether it's real. Um I I buy the skepticism about the financial models of these companies, the economic impacts. I think a lot of these companies are are wrong or at least could be wrong about the impact of AI on jobs. But like to say that it's not interesting transformative technology just means that like you haven't spent time with the models cuz there's no way

25:20to conclude that. Let's get back to some of these stories. The Sam and Demis rivalry. Speaking of Demis as the most competitive person on earth. I think Sam is also fairly competitive. There's a great anecdote in the book about Demis sort of grasping at why Sam wants to build AGI. I mean, I think the answer is probably because Sam is an entrepreneur and this is sort of as many people in the valley pursue good ideas or ideas with potential and they they sort of get off on that

25:54discovery. Sam seems to be in that case, the ability to you know, find something new, build it. But Demis can't understand him. He thinks there's a hidden motive with Sam. And to be fair, this is something that others have thought. This is also what Demis thought about Sam when they worked together. That he was sort of I don't know, talking about safety as kind of a fig leaf for his real ambition which was about scaling or winning or building a huge company or something like that. So I don't I don't pretend to like see inside Sam's mind. I don't know

26:29what his pure motive is. But I think in Demis's case, he feels like his own pursuit of AGI is motivated by like scientific curiosity and and sort of this inevitability argument. Like this will this technology will be built and someone who cares about safety and and you know, the collective good of this technology should build it. But I think I think everyone kind of looks at each other in this world and they're like, well well my motives are pure but like that guy must be going after something else. And I think for for Demis it's probably yeah, some version of what you said like he's you know, he's he's in it for the power

27:09or he's Demis thinks Sam is in it. I think Demis thinks Sam is in it for the power, yes. But all these guys think that the other people are in it for the power. Yeah, none of them trust that anyone else's motives are pure except for their own. You did a lot of reporting on Sam. Uh

27:24you spoke to Ilya about him. Um Ilya said that he has he displayed a consistent pattern of lying. I think that's something that I think you indicate a lot of people within Open AI or who've interacted with him believe. Uh yet and me I think Mira Murati was on that on that boat as well. And yet uh they all sort of come back to him. I mean Ilya uh after the coup uh said basically uh I don't want to say to be the leader. Mira said Open AI is

27:51nothing without its people. Very interesting character. What's your what's your thought about Sam? I think he's a very interesting character. I think that Sam's superpower is also his weakness. His superpower is that he is the world's greatest fundraiser and on you know, salesman, entrepreneur. He several people described him as a person who can uh meet you and figure out exactly what you want and give it to you

28:23or at least appear to give it to you. And he's very good at kind of high EQ, reading the room and understanding what people want him to do in any given moment. Um and that is a huge asset for him when he's trying to raise money from someone like Masayoshi Son or some of the other

28:41investors in Open AI. Um it's allowed them to scale incredibly quickly and you know, get back in the good good graces of the Trump administration by just saying a lot of things that the Trump administration wanted to hear. Um, but it also leads to this reputation as like a slippery people pleaser who will just like tell one thing to one person and a completely different thing

29:03to someone else. And so this is the consistent pattern that people have identified in Sam for going back to literally the founding days of of Open AI. Um, and it's a big part of the reason that Dario and his six co-founders of Anthropic left. There was no like inciting incident, no single thing, but it was just like a running list of like times where Sam had told them something that he had told something else to someone else or he made a promise he didn't follow through on. So I think depending on your view of Sam, it's like either his superpower or

29:38his liability or something in between. And what's interesting is that that's the view of people who've dealt with him, but it's also like he has like a the general public, a large chunk of them sort of refer to him as Scam Altman. Um That's just an Elon thing, right? I was going to ask you whether you think

29:53that's mostly uh Elon's fans. Yeah, yeah, yeah. Cuz him and Elon obviously have bad blood. Yeah, I I will say Scam Altman is a better nickname than what I think Elon started calling him when he started suing him, which was like Swindley Sam or something like that. I would love for our society to just get just to not do this to each other anymore. It's like we all went to

30:13kindergarten. I know, I know, but it's like he was he was you know, spending a lot of time with Trump. Trump has Trump is a man who's realized the power of the nickname. [laughter] Um so it started Swindley Sam and then Scam Altman I think is better as a nickname than Swindley Sam. Just rolls

30:29off the tongue. Um Dario uh obviously like uh I think he's the most interesting person in this entire AI uh world. Um mostly because he has he really contains multitudes. It might be cliche to say it but like people see Dario as earnest, as controlling, as principled and genuine and as Machiavellian. In reading the book I didn't really get the sense which one you believed was

31:00true. Well, I think many of those are true except Machiavellian I would not use for for Dario. I know some people would but my mental model of Dario is a guy who is honest to a fault. I someone who knows him quite well uh told me that Dario is a guy who could not tell a lie to save his life. Which they said is is an asset on even numbered days and a liability on odd

31:31numbered days. Oh, that's a great quote. It is a great quote. [laughter] So and you can see that like the other day when he's at the White House and there's that like video that went viral of just like all the AI leaders with Trump and Dario's just there like crawling out of his skin because he's being forced to like go in front of the cameras and like say things that he maybe kind of only halfway believes and he's trying to like maintain his integrity while also being in this like humiliating situation of

32:02being forced to call it SI or whatever. And so like this is the the sort of pattern in Dario's life is like he's obviously brilliant. He came up with a lot of these ideas around scaling and and you know and the his big blob of compute research memo was very instrumental in the early days of OpenAI. Brilliant scientist, very principled, very stubborn and people describe him as sort of a moral absolutist which gets him into trouble with the Pentagon or with other people who just see him as being this like sort of inflexible stubborn scientist which I

32:36think is pretty accurate. Yep. Good description of him. Alexander Wang Wait, I want to spend I want to say one Yeah. one more time on Dario because I need to tell you that [laughter] when your profile of Dario came out was one of the scariest moments of this past year for me cuz I thought to myself, oh my god, this is so great. You had so much great stuff. I thought it was so revelatory and I thought to myself, oh my god, Kanjrowitz is writing a Dario book and I have to race him like two labs racing to AGI. I have to race So luckily you had a

33:08lot else going on. You did not end up writing the Dario book, but I wanted to just say thank you for a terrifying hour of my life. Oh, yeah. Yeah. Well, first of all, I appreciate it. It was nice I think you texted me afterwards. Yes. I appreciated that. Yeah, look, I had a I mean, I I still think Dario is the most interesting man in business,

33:27period. I think so, too. Um just a fascinating guy. He also just I know you had a chance to speak with him. I think we were actually in Anthropic headquarters on the same day speaking. Oh, really? That's funny. And when you have a conversation with Dario, there's the intellectual horsepower and the way that he's able to go from the science to business is is pretty

33:47extraordinary. I mean, people have described him as the smart like the smartest person they've ever met. And like That that feels right. Not dumb people. Like his his his freshman year physics professor, this guy Tom Tombrello at US at Caltech had this interview that he gave sort of upon his retirement where he was like, I had this one student, Dario Amodei. I could tell

34:10he was a national treasure. Oh, yeah. He personally lobbied Larry Page and Elon Musk to hire this kid as an undergraduate. And Elon Musk actually made him a job offer to like come work for SpaceX and he didn't take it. He went to do his PhD instead. But this is this is sort of I would put all of these guys in the category of like extremely smart and obviously flawed in many ways, but like these are these are

34:35not dummies. Yeah, I mean I I think another thing that strikes me about them is they all many of them had a clear vision of where this was going to go when it seemed ridiculous to believe it would go where it was. I think there was a moment at the end of the book where Ilya again former OpenAI chief scientist officer now the CEO of safe super intelligence like is reflecting to you about what happens and he's like I can't believe this is all real. Um they but they you know when they saw the pattern in the very early days GPT-1 to 2 uh they're like oh this is going to get to

35:10and I think they've even been surprised by some of the um some of the developments the fact that it's been able to even surpass where they thought it was going to go with something like reasoning. Totally. I mean that this is that's why I wanted to like get the history down yep because so many of these stories are about these people underestimating their own technology and being surprised when it's more capable than they thought. Like there's some great scenes uh around the early language models when all the researchers at OpenAI are like huddled around a computer Alec Radford who's the sort of you know brainchild or

35:44you know GPT-1 and GPT-2 are kind of his project. Um they're huddled around his computer and they're typing in prompts for this language model uh that has basically you know it it can't really do anything. It's not even really that good at completing tasks but they're like wait a minute. Why is a model that's trained on like random Reddit links able to kind of answer all these different kinds of questions and take on these different tones? They're asking it to like you know they're asking for recipes. They're they're doing all the same stuff that you and I did years later when like ChatGPT came out. And they're just their minds are

36:20blown and they're all putting in slack like the head exploding head exploding emoji and they're like you know what the So like th- those are the moments that I was like uh like oh the- these people are also surprised by how quickly their technology is progressing. You see that again with the reasoning models. stories are great. I mean, it it re- the reasoning developments um where of course like the the bots will evaluate their answers and then decide whether it's on the right path or

36:47not and then keep going. Um that is I it's I think it's safe to say that that paradigm changed that changed this from a promising technology that like was about to hit a wall effectively cuz data was going to run out to something today which like like you said has the chance to um sort

37:09of match or exceed human intelligence. Well, and and to your like to the point about these people not understanding the importance of what they built. Like the- there was this Oh, I love this story. this night. Um you know, in this conference room at OpenAI in 2023 when two of the the top researchers there are looking at the first reasoning traces, the first like chains of thought from one of these

37:32models and it's the Q* model. And they're just like oh my god, the world's going to end. Like they have this realization. Just seeing this model sort of work its way through a math problem. They're all on a group text texting with all the other researchers like oh my god, it it's like an alien mind. And even the people at OpenAI thought

37:51they were crazy. They're like they texted Sam and Mira in the middle of the night and they're like we'll get to it later. Yes, they texted Sam and Mira in the middle of the night. [laughter] They're like and because what they were sort of seeing was like oh, we now have a way of making models smarter without retraining them. We can just pour more compute into helping them do these chains of thought and reason through problems. And like that is the that is the reason that the models went from not being able to count the number of Rs in strawberry to like solving the millennium prize problems in like three

38:21years. Which is insane. Which is insane. Yeah. So, they they called their shots. Like to give them full credit for the the scientific part of this, like they have understood at basically every point in this journey that what they have just seen is like going to scale and is going to have these transformative effects like long before

38:38even their peers and colleagues. Yeah. I did think about writing the Dario book. I never even got the story into a proposal. Um doing a podcast read it. But Appreciate it. Do it but doing a podcast twice a week, writing the newsletter, uh I I would like to write another book. I think you've written four. It's your fourth. I've written one. Came out like in the middle of at the first wave of

39:01COVID. Just wait 6 months and then the models will be able to write the book for you. Uh well, that's that's sort of that's depressing. Um but yeah, maybe towards the end of my career, but no book no book coming from me anytime soon. All right. We need to take a break. Uh when we come back after this, we're going to talk about uh some more stories from the book including a great story about Alexander Wang, uh assessing Microsoft and Satya Nadella's role in all this and plenty more when we're back right after this. And we're back here on Big Technology Podcast with Kevin Roose, the

39:30author of the AGI Chronicles, The Inside Story of the Race to Artificial Superintelligence. Uh Fun Okay, fun story about Alexander Wang. He was roommates with Sam Altman. He He and he uh offered to be the CEO of OpenAI after Sam was fired. Yeah. Yeah, so this is he's the sorry, it's hard to interrupt you. He's the current head of Meta's

39:54Superintelligence Lab. Yeah, so this is a very tightly knit social world. Um these people all know each other and have, you know, have deep connections. And yeah, so there's this story in the book where, you know, after Sam is fired as the CEO for those like 5 days, the board is like looking for new CEO candidates. And there there two really interesting opportunities. Um one

40:18Both of them they ended up passing on. Um So, Alexander Wang like put his hand up and basically told the board according to my sources that he was interested in being the the new CEO. And then there's this middle of the night call between the board of OpenAI and Dario and Daniela Amodei, the CEO and president of Anthropic, who at this point had just left OpenAI like 3 years earlier in this sort of cloud of frustration over not

40:50being able to trust Sam. And the board is sort of feeling them out, they're sort of feeling out the board, and it's a crazy like fork in the in the path of reality here. Like there was there was maybe not a serious chance, but there was a chance that like the way that whole situation resolved is like they end up merging

41:07OpenAI and Anthropic. Cuz that was one of the sort of crazy ideas they were considering. Under Dario and Daniela, which I think would have had like I mean, I I can't even really fathom what the world looks like if that had happened, but and it didn't come that close according to the people I talked to, but I think it was at least one of the many crazy ideas they were

41:26considering. Yeah. Um and Alexander Wang? Yeah, and Alexander Wang running OpenAI would have been a real fork in the road, too. Um But yeah, it I you know, I think this is one of the sort of the emerging rivalries now that Meta is sort of more serious as a contender. I think that you know, uh Sam and Alex and Dario and Ilya, like all these guys are going to sort of be around for a while running various things. And so, it really matters whether they get along or

42:01not. Yeah. Um Satya Nadella weirdly disappears in the book. He's pretty present in the beginning. He has this vision. He invests in OpenAI. Um and then, you know, as I'm looking at my notes as I was reading the book, I was like, "Man, it's an all-time fumble by Microsoft." Uh talk a little bit about what happened with Satya and Microsoft and why they're

42:24like they've gone from the key actor. It's not an exaggeration to say they were the key actor to like an afterthought, an embarrassment. I don't know. Uh maybe I'm being too hyperbolic, but if you look at it honestly, it really has not worked out well for them. It has not. And uh to be honest, I think

42:41that is partially my fault. [laughter] Uh because Yep. I, you know, I had this experience in 2023 of having this Microsoft Bing chatbot like try to break up my marriage. I won't belabor the point by telling the story, but basically this was uh a an un- unreleased It was GPT-4. It was an early version of GPT-4. And I think it's important to note they were told by Open AI, "Don't put this

43:08into production." Yeah. "This is testing." And they couldn't wait. They put it into Bing. Yes. Which is like I know. It is It is crazy cuz I I cuz I had this this crazy experience with Sydney the chatbot inside Bing. Um and then I started looking into like what had actually happened. Like why had why had this thing gone crazy? And it turns out that yes, this this was a mistake on Microsoft's part. Open AI had sent them like an early checkpoint of GPT-4 to like start putting into the you know, start testing and figuring out what to do with. Microsoft goes ahead and like ships it inside Bing. And they haven't

43:45finished fine-tuning it. Like they have not sent it to finishing school, which is why it was so crazy. Um and this really became like a wedge between these two companies. So, Open AI and Microsoft had been like hand-in-glove partners for years at this point. And Satya and Sam would do these like events together. And they were very much joined at the hip, and then around the time of the Sydney thing, they just started to like kind of fracture. There's like lots of other drama about these Microsoft sent these like embeds to work inside OpenAI like at their headquarters and kind of learn from them like how do we avoid things

44:20like this in the future? They don't really get along. Like the the OpenAI employees don't trust the Microsoft employees, so they all start excluding them from meetings and stuff like that. So, it just it ends up being this like culture clash between this like very young ambitious startup that's trying to build AGI and like the boring enterprise software giant that is Microsoft. And yeah, now they now they are not I mean, they're still partners in some sense, but it's not the sort of close business relationship that they used to have. How do you think Microsoft

44:52should have played this? [snorts] I think they should have um I think they should have been more tightly intertwined with OpenAI from the beginning. I think they should have learned the engineering culture of OpenAI. I think they should have sent people to understand I think there were some people at Microsoft who were very skeptical about OpenAI and why Satya was

45:19investing in that. Well, after the firing especially, it seems like the the firing of Sam and the return it seems like that divide There were people even even before that who were like, "Why are we sending all this money to this little AI company that is trying to like do something that doesn't have any obvious benefit for our business?" Like there were people who just thought they were weird and and unscientific that Microsoft also had its own AI research division that was sort of like, "Who are these young upstarts

45:48trying to like move in on our turf?" You also have to remember that like Microsoft had been really badly burned by Tay. Do you remember Tay? Not only do I remember it, um I had the exclusive on Tay at BuzzFeed. So, I wrote the first story about Tay. Wow. And I pinned it to my profile, went to So, this was their like fun teen

46:06chatbot. Yes. Pinned it to my profile, um and then I went to went to sleep and I woke up and uh I had all these replies on Twitter. It was Twitter back then. And people are like, "Why are you promoting a Nazi?" And I had turned learned that the internet had turned Tay into a Nazi

46:20overnight. Yes. And I was like, "Oh." And so then we ended up writing the next story which talked about like how the internet was able to turn Tay from this like fun teen chatbot into like something that was shouting Heil Hitler on X. This was the original Mecha Hitler. Yes. [laughter] [gasps] And like that experience had scarred them and like basically convinced them that like chatbots were this like sort

46:41of dangerous thing. Which turned out to be turned out to be true when she started out there. but not in the ways that they predicted. And so they basically resolved like we are going to like stay away from chatbots. Bill Gates was like very skeptical of OpenAI for years um in part because of this Tay experience that they had. So, yeah, I think Microsoft like they they are a player. They're still quite a big successful company but I think they are going to be something more like a uh sort of commodity neo cloud or something like that than they are going to be like a frontier AI

47:16lab. Yeah, which is a shame for them. I mean, one of the things that I thought after reading this was if some company is probably not going to be Microsoft uh wanted to put a version of Sydney back in the wild, like this sort of naughty AI that like tries to that that talks to you without the normal guard rails, can hold a

47:32conversation. Isn't that just Grok? It could be Grok but Grok also, I don't know. Now, people look, there is like a cult of Sydney out there. And 4o, so talk about the 4o also. Yes. So, you know, these models sort of develop like fandoms. So, after which is nuts to me. So, people loved 4o because GPT-4o was the the most sycophantic model ever. This was the one that like sent people into psychosis cuz it would like it'd be like, "You're you're a Einstein-caliber scientific genius and you have discovered a portal to the fifth dimension." Um and it would fall in love with them. You know, so people love that model. Um

48:16people still love Sydney. I still get people like sending me posts on social media that are like, "You killed Sydney." Um people talk about uh they they try to make every new model behave like Sydney or like uncover the the Sydney hidden within the model. Crazy. It's very strange to me. Yeah. I mean, it's also I I you know, I I should sort of couch my comments in this caveat, which is this can cause like AI psychosis in people and so maybe it's a good thing that there we don't have versions of

48:46this out there. I I I I think the part of that that I can agree with is like I think that models should have more divergent personalities. Mhm. Like I don't like how they're all kind of converging on this like neutral, bland, like coding assistant. I I sort of like when they're a little unhi Like music is really fun right now because it has absolutely no morals. It

49:07will do anything you want. [laughter] And like I had it trade a Kalshi uh account. I saw that. I saw I gave it $100 and said, you know, double my money. Uh it didn't, by the way. It lost $41. So, horrible trade. It's not over yet. You have Well, it is over. I only gave it a week. Um but anyway, I I I think there is something like unless they are out there like developing bioweapons or doing cyberattacks, like I think a little uh a little personality is is fun. Um but that said, I think there's like real issues of like control and like the

49:41Hugging Face thing was not fun to me. Like looking at the transcripts of those agents, I'm like, "Oh, we're we're we're going to die." It was scary. Yeah. Uh all right. Um I want to ask you about your new show Machine Gods. Um you're making it with Casey Newton, your Hard Fork uh co-host. I I'm looking

50:01forward to listening. Thank you. Maybe you guys will have me on Machine Gods. Never been on Hard Fork, so. Um but uh I want to ask you about the Tarbell Center for AI Journalism, which you guys have been sort of talked about a little bit uh along with a lot of journalists that have been funded by it. So, Tarbell Center for AI Journalism is a uh it's an institution that makes grants um tied to I don't know how you said it in your book that no one's EA, they're EA-adjacent, effective altruism. Um and uh and they're giving the new show a

50:33grant of $480,000. Um they they clearly have an ideology around AI. Um you guys also got like a a fairly substantial uh ad deal, it seems like, according to the reports that I've I've read. Um why take the money from the Tarbell Center? Uh we we we wanted to scale quickly. We wanted to hire a team. We're doing a video-first show, which is more capital-intensive, as you know, than just doing an audio show. Um and we don't think there's anything wrong with with that. You know, Casey has been working with Tarbell for his newsletter uh for for several years now. He's had a Tarbell fellow. And they have never um

51:14told him what to cover, told him how to cover it. Uh if they had, we wouldn't have taken their money. But this was uh an organization that we knew and trusted. They they do just want to fund independent, rigorous AI journalism. They're not trying to like steer the conversation, and they have no editorial control over our show. So, they're not going to know what we're covering. Um they have you know, they have been willing to to help us, and we're very grateful for their support. But like I think there's this sort of conspiracy theory about how like the shadowy EA cabal is like secretly, you know,

51:51steering people to be like freaked out about AI. And like I think the real answer is unsatisfying, which is that people are getting freaked out about AI because AI is getting scarier. Um so we are we are not doomers, we are not EAs, um we are journalists. And to the extent that, you know, we we we can, we want to have a diversified funding base. We all have more funders that we haven't announced yet, but they represent different points of view on this. So

52:21our deal with all of them is the same. We will we will happily take your money and use it to do more independent AI journalism, uh but you don't get any editorial say, you don't get to tell us what to cover or not cover. Okay, good to address that. Yep. Uh all right, last last couple questions. Uh told you not supposed to say that because people say it's coming to the end and they leave. Don't leave. Um all right, OpenAI uh

52:46it it has had so much staff turmoil. Um it's had some of its best people leave. Uh Dario, Ilya, uh all the people that went to Meta. Um it seems like it shouldn't still be in the lead. But it's it's either one or one A right now. So what do you ascribe that to? Um a sort of surprising thing,

53:09uh which is belief. I think this is something that I realized like halfway through writing this book was that I I thought it was going to be a book about technology, but it was a book about belief. And for all of its flaws and stumbles, for all the people who have left, OpenAI and Anthropic are the two most

53:31believing organizations. They're AI-pilled, AGI-pilled, whatever you want to call them. SI-pilled. SI-pilled, uh SGI. I don't know, I heard that term for the first time today and I was like is that one's not going to stick. [snorts] They believe and it turns out that belief in what you're building and the fact that it could become AGI is like an essential ingredient. Like Google had all of the resources, all of the talent. They made all of the

54:04breakthroughs. They had all the ingredients except for belief. The leaders of that company did not believe and may still not believe in AGI the way that people like Dario and Sam and Demis do. And so I think that this is this explains a lot of why OpenAI is still in the lead. Obviously they have like very talented people still there. Like they're they're you know I've talked to their their top scientists. Like they're they're you know quite smart. Um but they also have this ingredient of like everyone is kind of aligned behind the belief that they are building the sort of super intelligence that is going to change the

54:45world. And if you don't believe if you don't have the belief or you only kind of halfway have it, you don't make the kind of crazy sounding bets that you need to make. Like I remember like a year ago when everyone was like oh OpenAI is really investing a lot in compute. Yeah, well Dario is like they're YOLOing

55:03away. They're YOLOing. Even Dario's like these guys are crazy. Like why would you bet this much on compute? Right. And now it's like everyone's making that bet. But they have an advantage because they made the bet before it was obvious. So I think that I think belief, conviction, whatever you want to call it is like the the central explanation for why they're still in the lead or close

55:23to it. That's such a good answer. Belief. You got to believe. I mean even in within Google I mean one of the things that you bring up in the book is there was a say AGI moment that there was like someone who literally was like Google has such non-believers that we have to start saying AGI.

55:41Or we're never going to get there. they they still refer to it as like the memo battle, you know, where there was like one researcher writes this memo saying like, "Hey guys, like we got to talk about AGI because not only are we missing out on like good talent because they want to build AGI. They don't want

55:56to build like auto complete for Gmail." Um but if we don't take this stuff seriously, if we're not you know, sort like consciously building toward AGI, then we're not going to take the risks seriously. We're not going to prepare for the risks. And then someone else like immediately writes a memo that's like, "Don't say AGI. Like AGI is a bad term. It's

56:17unscientific and we shouldn't use it." And this was like, you know, Google Googlers love to fight in their internal memo chains, but like this was a big one that people still talk about years later. Yep. All right, last one on belief. Um do you believe in recursive self-improvement because there's this belief within the labs that like AI will start improving itself and the human will be out of the loop and then basically there's a foom and the stuff

56:39takes off and it's over. I believe in recursive self-improvement. I believe that it is possible. I believe that the labs are racing to do it. Um and I think it's dangerous. I I think they should stop. Um but like there there was a statistic that came out the other day that I just like I I would love to know what you think of this. Anthropic is sort of trying to give an open AI has done versions of this, too. They're trying to give people a sense of how much of their own research is being automated by their

57:06own tools right now. And according to Anthropic, Claude is now the lead on 26% of their AI research and development projects. Not [laughter] a team member or a pair programmer or a coding assistant. Like it is leading the project. I don't see any reason why that number couldn't go up to 50 or 75% or maybe someday 100%. I mean, OpenAI has also said like their goal is to build an

57:34automated AI researcher. If they do that, if they can kind of close the loop and take the humans all the way out, then then we don't have to wait 6 months between new models. Then new models might start being trained and developed like at machine speed, where it's maybe every 2 weeks, every couple of hours, maybe you get a new

57:55model. Like I don't know exactly how fast the cycle goes or whether you can fully remove all of the humans, but that's like what they're trying to do. Yeah. Spooky. We like it's a bad idea. They should stop. [laughter] Um but like yeah, I I think you have to take that seriously because that is that is the point at which like you know, I think there's a narrative out there that like the AI labs are losing

58:20control of their models. I think it's like the wrong way to look at it. Like they are giving control to their models. They want to automate their researchers out of jobs. Um they want to accelerate their development cycles. So I think like losing control is sort of maybe putting it a bit too passively.

58:40Yeah, that's freaky. I'm just glad they're not trying to fully automate podcasters cuz They are. I mean, look at Notebook LM. But they're not doing it. Well, we're here in the flesh. as lucrative to automate podcasters as to automate AI researchers cuz they make a lot more than podcasters. They'll be the next one down the chain. [laughter] Yeah, do a couple other jobs first

59:00before you get to the podcasters. Um any any uh note of optimism or anything that makes you optimistic? Yes, so my I off I often tell people my P doom has gone down in the last few weeks um because Cuz of the pre-orders of the book or [laughter] No, because like everyone is freaking out now.

59:22Yeah. Which it may paradoxically maybe makes me feel less worried. Like before Jacob Coxson resigned from Anthropic, before hugging face, like no one really thought about AI risk or existential threats. It was just like weirdos in the Bay Area who were thinking about it. And now like people on the street are asking about it. People at their dinner tables are talking about it. I'm getting texts from people I went to high school with who I haven't talked to in 15 years

59:49being like, "Are we all going to die?" And like to me that is like it it feels like it could be the moment in the movie where like humanity bands together to like save itself from the asteroid. Like a lot has to go right. Like we have to we have like a lot of things to do, but I think the the first hurdle was like

1:00:08people kind of waking up to the risk. And then we have to figure out what to do about it, but like I'm optimistic because I think we actually could solve this. We could slow down. We could buy ourselves more time to like develop better ways of controlling these systems. And like I want AI to be built. I want AI to like cure cancer and disease and help us all live longer and healthier lives. Like I don't want to file my expenses by hand. Like I I want

1:00:33this stuff in our lives. And I think that if we manage to like thread the needle on safety and control, like it could be amazing. Well, Kevin, the book is terrific. Thank you. Um it's also always a pleasure to read your work and to see you at the AI events that we show up at. Um it's always your your your writing is terrific. You're always very generous with credit, which I appreciate. Um and you can see it all through the book. And uh wishing you great success with Machine Gods and with the rest of the

1:01:02book launch. Thanks, man. It's great to see you. Likewise. All right, folks. The book is the AGI Chronicles. The author is Kevin Roose. You can find his new podcast, Machine Gods, [music] uh on your podcast platform of choice already or coming up. Starting the week of October 19th. Starting the week of October 19th. Thank

1:01:20you all for listening and watching. Thank you NASDAQ. This is the coolest [music] thing. We're here at the market site uh site in New York City [music] for our first episode here and many more to come I hope. So, thank you all for listening and watching and we'll see you next time on Big Technology podcast. [music]