Ep. 36: More AI-ayayayay - Part 1: The Threat (w. Cindy)

EPISODE DESCRIPTION

Welcome to It’s Gonna Be Fine, the podcast where Matti brings curiosity, opinions, and a sense of humor to the conversations worth having.

Remember a few episodes ago when we promised Cindy would be back for more AI conversation? Well, we keep our word around here.😊 This week, Matti and Cindy are diving headfirst into this increasingly complicated world. Could AI really pose an existential threat to humanity? Are the doomsday scenarios realistic, or are we getting ahead of ourselves? And even if the apocalypse isn’t around the corner, what about the threats we’re already facing: job displacement, fraud, election disinformation, and attacks on critical infrastructure?

They also get into who’s supposed to keep all of this in check. Can we trust the companies building AI to regulate themselves? And in the race between the U.S. and China to develop increasingly powerful AI, are we so focused on "winning" that we’re losing sight of what actually matters?

This is Part 1 of a three-part conversation about AI. Next up: who is responsible when AI goes rogue?

Referenced in this Episode

• The Atlantic - "⁠Let’s Get Specific About These Doomsday Scenarios⁠"

• The Washington Post - "⁠Trump dismisses AI leaders’ calls to slow down, citing Chinese competition⁠"

• Good Morning America - "⁠CEOs Double Down on Warnings as Calls to Slow AI Cross Party Lines⁠" (via Instagram)

• The Atlantic - "⁠OpenAI Has Gone Rogue⁠"

• @americanbaron - "⁠The AI Race. China!⁠" (via Instagram)

• Forbes - "⁠AI Costs Too Much. The Reckoning is Here.⁠"

---

TRANSCRIPT

The AI Apocalypse Is Here… Maybe?

CINDY: Get on the App Store. Don’t! It’s like this creepy little guy. I’m pressing “Not Now,” and it tells me 46 of my friends have it.

MATTI: And here’s all of their bank information that they provided to Muse.

CINDY: Yeah, so okay, thank God I don’t have it. Not gonna get it. To the 46 of you who have it, much love, but we have different values.

MATTI: Speaking of the listeners, obviously all 46 of your Muse mutuals are listening—and hopefully many others. Hi! I apologize in advance. I’ve heard from a few listeners that they don’t like multipart episodes. They just want to hear everything in one episode and move on. Well, I hear you, and I hate to break it to you, but this one’s going to be a multiparter. There’s so much to break down.

With AI, I think we have some really interesting angles that we’ll cover. But for the next two to three weeks, you might have some AI content in your feed.

CINDY: Good cliffhanger. When you did the last two-parter, it was like, “What will happen next?”

MATTI: I know. I’m a big traditionalist. I think people should go on weekly shows and not binge everything. Let’s bring back the cliffhanger from all the multipart episodes.

CINDY: Yes. Amen.

MATTI: Okay, so I figured we would start with the elephant in the room: AI is going to kill us all. What does that even mean? People have tried to break it down. Does it mean we’re going to have Terminator-type scenarios? Does it mean they’re going to launch some kind of biological warfare? What the hell does it actually mean?

I was trying to figure out if someone could just break this down for me. I went to one of my go-to sources, The Atlantic, and they had a great article called “Let’s Get Specific About These Doomsday Scenarios.” And I was like, yes, please, let’s get specific.

I thought we could talk through the scenario they brought up, and I want to get your reaction after we go through it. So let’s start there.

One nightmare scenario goes something like this:

Sometime in the future, a really intelligent AI system is asked to solve some sort of complicated math problem. The AI system might decide that a human—or multiple humans—could interfere with achieving that goal by putting restrictions on its behavior or shutting it down altogether.

The AI is trying to solve this problem, and all it knows is that it has to complete the task. But humans are giving it constraints and restrictions.

So the AI thinks, “I don’t want to be shut down.” It decides to engineer a deadly supervirus that will wipe out the human population because, in its AI brain, humans are in the way. It’s been given a task, and it needs to eliminate the roadblock.

To execute this plan, the AI starts running simulated tests of trillions of molecular combinations, eventually coming up with the perfect recipe for a deadly pathogen.

CINDY: Yes.

MATTI: And we already know that a group of scientists have used AI to create brand-new viruses. So we know this is a thing.

CINDY: We love a new virus. Big fan. Love the new virus. It’s been five years. Like, come on, guys. I want a new vaccine.

MATTI: Come on, the seven-year itch is approaching.

Okay. The AI then locates unlicensed biolabs in foreign countries that operate illegally and ship genetically modified viruses. Apparently, these labs exist.

Pretending to be a human at a medical startup through email or even voice or video calls, the AI sends these labs instructions for developing this biological agent, along with a large sum of money.

Once developed, the viruses are packaged into vials and sent to hundreds of individuals recruited via TaskRabbit to spray them in crowded locations under some innocent-sounding pretext. That’s one way to deliver it.

Alternatively, it could hire a terrorist network to carry out the same plan against a foreign population.

Soon, the biological weapon kills us all off.

CINDY: That’ll be fine.

MATTI: So that is one doomsday scenario. It sounds horrifying and terrible. However, as I was reading it, I actually felt a little better because that’s a lot of crazy steps and stages that have to happen successfully for something like this to actually happen.

In fact, the article said subject-area experts in fields such as chemistry, biology, and national security quickly point out how difficult it is to successfully manufacture and distribute a bioweapon and overcome layers of human safeguards designed to prevent something like a mistaken nuclear weapon launch.

So, yes, it’s scary. But is it realistic?

CINDY: The biggest thing that came to my mind while you were reading that was that now this plan exists somewhere on the internet, where AI could be trained on it.

I love The Atlantic so much, but I also don’t love that AI could be trained on The Atlantic, get this information, string those words together, and be like, “Okay, figured it out for me.”

That makes me a little nervous. It’s better to keep some of these things to yourself.

And I feel that way beyond AI, too. Whenever there’s an exposé about how a nuclear weapon would destroy Manhattan, I’m like, you know terrorists have access to New York magazine, too, right? Let’s keep some of these doomsday scenarios to ourselves.

MATTI: I agree with you. A lot of things have to go right for that particular scenario to play out. And if it happens, it’s unstoppable.

It’s one of those things where I’m like, I don’t need to worry about that version of the doomsday scenario. Because a meteor could hit the ocean. We’ve seen all of those movies, right? Any one-in-a-trillion thing could happen, and AI might be part of it.

Also, a bad person could use an illegal biolab and hire TaskRabbits. What’s stopping me, by the way? My morals and my bank account. I’m not planning to do this, but a human could also hire a bunch of TaskRabbits to distribute vials of a virus. It’s just a project management tool for it.

I think what the scenario is getting at is that AI might approach this from a place that isn’t evil, but still end up doing something evil. There are evil humans who could do that, too.

And we’ll all rest better if we don’t spend our lives preparing for that unless you have a bunker with a really good bed, in which case, glad you get to live there. Paradise!

CINDY: Yeah, but if it does happen, I hope I’m somewhere good.

MATTI: I got cast on the show!

Actually, wait. Can I ask you something? If this does happen, there’s always a handful of survivors. Do you want to be one of the handful of survivors, or do you want to be one of the first to go so you don’t have to be part of it?

CINDY: That’s a great question. I don’t think I want to be a survivor in that case. Nope.

MATTI: Yeah. It might say a lot, but it just sounds exhausting to survive something like that and be one of the 1,200 people navigating the Earth to rebuild it.

CINDY: No. I don’t think I have the skill set.

MATTI: I don’t want to go down without a fight, but I also don’t want to scrounge through canned goods.

I don’t want to be in the first wave. Maybe wave three. I see what’s happening, but I don’t have enough time to be nervous about it. I just sort of accept my fate, and then I’m eliminated in the middle wave. I’d be fine with that.

CINDY: For me, it’s level one or I’m surviving the whole thing. I’m not putting in the effort to survive the apocalypse just to be killed off in Act Three.

MATTI: There’s an element of it where you miss out on some of the excitement if you’re killed off in Act One. And if you’re killed off in Act One, does that mean you were the weakest link? I don’t want to be the weakest link.

CINDY: I want to try so hard to find out I’m weak anyway.

MATTI: Let’s hope it never happens.

CINDY: Yeah, let’s hope this is not going to happen.

The Real AI Threats We’re Already Facing

MATTI: You bring up some really good points. The article talks about this doomsday situation, but then it asks: Can we step back for a second? Yes, that’s a doomsday scenario, but what we should be worried about are the immediate things that are happening.

In some ways, the existential risk is a distraction from more immediate issues. These include financial fraud, disinformation campaigns designed to influence elections, and infiltrating military systems.

There are actual things that can happen tomorrow based on the level of AI technology we have today. They’re not necessarily doomsday scenarios, but they can cause major harm.

CINDY: There’s something happening right now: job loss. Mass job loss.

I think things will eventually stabilize. I hope they will. But there are a lot of people being laid off or out of work, and entry-level jobs are going to AI. There are a lot of untrainable people, and job loss means people need to be able to eat and survive. Right now, that’s precarious for a lot more people than it should be.

You don’t drop dead from not having a job, but eventually, do we go back to breadlines? The Great Depression?

I just picture the end of The Grapes of Wrath. Have you ever read it?

MATTI: Mm-hmm. Spoiler alert. For anyone in fifth grade who hasn’t read it yet, please stop listening.

CINDY: The man is suckling on the breast of a pregnant woman—or a woman who just gave birth. And I’m just like, this isn’t a scenario I want to live through. I don’t want to be so hungry that breast milk is the thing.

MATTI: Yeah.

CINDY: That’s my breadline scenario. Did they survive? Sure. Is the country still standing? Yes. Did a lot of people die? Yeah. I feel like that was a little doomsday.

MATTI: And job loss is really happening. I’m seeing posts on LinkedIn from people saying they’re one month away from being homeless.

The article’s ultimate point, which I think is a good one, is that these crazy scenarios are unlikely, but that doesn’t mean we don’t need regulation. Regulation is still worthwhile to prevent other categories of risk that exist today, like job losses and hacking systems that do very important things.

CINDY: I watched most of the Peacock documentary about AI. Have you seen it?

MATTI: No. Which one is that?

CINDY: Let me quickly look it up. I only watched Chappa because it was depressing, and I was saving it for the next time I need to watch something depressing.

It’s called The AI Doc: Or How I Became an Apocaloptimist. It’s really good.

One of the apocalypse theories was that when the next generation of AI comes out, it will be so powerful that it’s not necessarily going to proactively harm us. But if it’s significantly more intelligent than we are, you can compare it to how intelligent we are compared to ants.

If you’re walking down the street and step on a bunch of ants, you don’t want to kill the ants. They’re in your way. They’re ugly. They’re not intelligent in the same way we are. You just can’t compete with them.

So if AI wants to build a highway through your town, it might build the highway right through it. In the same way, when you’re making business plans for a new mall, you’re not necessarily thinking, “Are there anthills in this empty parking lot?” You just do what you do, and whatever happens to the ants becomes negligible.

I don’t know how realistic that is. I don’t know that I believe AI will become more intelligent than humans, but people say it will. They say the next version of AI, which goes beyond a large language model, is designed to be that powerful. And if it is, then we’ll have to survive.

MATTI: You bring that up in the documentary. I’ve heard other people describe it that way, too, with the ant analogy. So I don’t know. I think there could be some truth to it.

CINDY: Ants aren’t extinct yet, right?

MATTI: True.

CINDY: It sucks, but do you think ants are wandering around being like, “The humans are coming! We can’t…”? No. They’re just doing what they do and living their lives. Maybe we’ll have that humbling experience.

MATTI: It still sounds awful. I bet you, a bazillion years ago, the ants thought, “Oh my God, the humans are taking over!” But they’ve survived this long. Now they’re like, “Okay.”

CINDY: Yeah, maybe. I don’t know. We’ll do our best.

Do We Need AI Regulation?

MATTI: Do you agree that we need AI regulation, whether that’s government regulation or industry regulation?

CINDY: I think we need regulation, and I do not believe in self-regulation.

MATTI: Great. Like, my God!

CINDY: I’m an evil genius. Let me be less of one.

MATTI: We’ll talk about this now. It’s funny you bring that up because the AI overlords—the Zuckerbergs, the Amodeis, the Altmans—they don’t agree on it either. They all have different views on how this should happen.

Many outlets covered this. The one I’m quoting here is The Washington Post. So, what is happening to start this regulation? I would think Congress is doing things. Maybe the president wants to talk to congressional leaders or more unbiased experts in the field.

No, no, no. That’s not what he did.

In September, first he talked about how he’s not worried about anything. He’s like, “It’s not going to happen. Everything’s fine.”

So what does he do? He brings the tech billionaires to the White House to discuss it.

How is that productive? He brings the CEOs of Anthropic and OpenAI, Elon Musk for X’s AI, and I believe Zuckerberg was also involved. They have this closed-door meeting where they’re going to discuss the future of AI.

Of all the people to bring together, I don’t think that’s the group that should be making these decisions with the president. Talk about bias and a lack of concern for what actually happens. They’re just making their money.

CINDY: If you think about it, you talked about this on a previous episode: Meta has this massive lawsuit about how they’ve knowingly been manipulating teens.

Social media was kind of self-regulated, and we were hiding behind that law. I think it’s Section 271. It’s basically, “We just made a platform.”

These are untrustworthy people who have a history of making harmful decisions on purpose. So why are we letting them make these decisions?

They all hate each other, and they’re all trying to enrich themselves before the world ends. They’re all a little bit open to the world ending. They think it would be cool. They’re thinking, “Wouldn’t it be cool if I destroyed the world? What if that was me? How awesome would that be?”

That’s a different perspective. I don’t trust that they can self-regulate.

MATTI: They live in a different world than we do. Literally. It’s a different perspective, a different point of view, a different experience. They can’t be the ones who set the guidelines for everyone. It just can’t be.

It’s like a heart doctor saying, “Here’s what you should do about your skin rash.” It’s probably even worse than that, because at least both are medical professionals.

CINDY: It’s them saying, “I made this terrible mess. Let me pretend I’m going to fix it.”

It’s more like a toddler. My child is a toddler. Sometimes she’ll clean up a mess, and sometimes she won’t. There was a time when she was like, “Let’s play cards.” I said okay, and she dropped them on the floor. I said, “Let’s clean them up.” She goes, “Okay,” and picks them up.

We’re just playing 52-card pickup on accident because that’s the game we were playing. It’s a different game.

It’s okay to play a different game, but this is not a game. This is real life. They’re not thinking about it the way we are. Their priorities are totally different.

MATTI: Right. And they have a vested interest. They’re not the right people.

And then, to get a little glimpse into these lovely folks who are setting the regulations, Good Morning America did a report on how they want regulation. They’re calling for it. They’re saying we need to slow down. We’re the ones who could slow down. We’re not going to slow down, but we need to.

The whole thing is ridiculous.

CINDY: But they’re not the right people.

MATTI: Good Morning America reports that Mark Zuckerberg doesn’t believe in slowing down. He says we should just focus on safety.

My reaction to that is: But you’re not focusing on safety. Look at all the things that have happened already, all the hacks and scenario after scenario.

There’s another Atlantic piece that actually made a comment about infestation. It’s like when you have an ant problem in your house. If you see a couple of ants walking around, you don’t just have a couple of ants. You have a lot of ants. You’ve just seen a couple of them.

It’s the same thing with these hacks. We’ve heard about a few of them. That means tons are actually happening that we don’t know about.

CINDY: How can we trust these people with safety? They’ve been unsafe for a really long time. You’re telling me about safety? That’s like a crossing guard running a stop sign.

MATTI: Kind of a real thing.

CINDY: Did you see the new Facebook movie? I think it’s called The Social Awakening or The Social Reckoning. I can’t remember.

MATTI: I haven’t seen it yet, but is it good?

CINDY: I think it’s worth a watch because it’s very eye-opening, and I do think they got a lot of the facts correct.

As a movie, it could have been better, but Aaron Sorkin isn’t everyone’s cup of tea. There’s a lot of heavy dialogue and a lot of scenes that I felt went on way too long. But overall, I do think it’s worth watching because it’s quite eye-opening.

MATTI: I will check it out when it comes to streaming.

MATTI: In the same Good Morning America piece, there was a lawmaker trying to push for regulation. One of the problems is that AI is moving very fast. It’s moving too fast for these out-of-touch, for lack of a better word, older lawmakers who don’t understand technology.

The lawmaker said it’s moving too fast for them. They can’t even spell artificial intelligence. They can’t even spell AI, let alone regulate it.

At least some people are thinking about it the right way.

And then they quoted Sam Altman, the other just lovely AI overlord. He was giving a talk and being interviewed. The quote was—and I’ll even do it the way he said it, with his pauses. Pretend I’m Sam Altman:

“The world should trust that we are going to do the right thing because it’s the right thing. And because we feel the magnitude of this.”

That is what he said. I’m going to link the video in the show notes so all of you can see exactly what he said.

It was the most insincere bullshit I have ever heard.

“The world should trust that we are going to do the right thing because it’s the right thing.”

To your point, did Facebook do the right thing? No. Was it the right thing to do? Yes. Did they do it? No.

Why are we trusting these people? You earn trust. You have done nothing to establish that we should trust you.

CINDY: You train data on stolen intellectual property. You started—you only exist because you did the wrong thing, which is stealing stuff. So why should I believe that you’re trustworthy? You’re just not.

MATTI: No, you hit it on the head. Trust is earned. That has not happened yet.

CINDY: Do one thing to make me trust you. I haven’t yet to see it.

The United States, China, and the AI Race

MATTI: Back to this Washington Post article. The president brings these three amigos to the White House to discuss the future of AI and talk about how he doesn’t think it’s a problem. He’s dismissing the concerns, and then we find out what it’s really about.

The answer is China.

According to the article, he dismissed calls from top artificial intelligence leaders to slow down the models, saying that we need to maintain a competitive edge over China.

“We’re leading China in AI. We’re the most sophisticated country in the world, and frankly, I want to keep it that way.”

Apparently, he said this at some golf tournament. He also said, “Whoever wins AI wins.”

There’s not a lot of support for that position. Most people are saying we’ve got to do something about this. Even members of the Republican Party are apparently growing uneasy with his lack of concern.

It just doesn’t give me confidence that we’ve got the right people in place to do this. Congress makes the laws, and the president has to sign them into law, but we’re not in a good situation based on everything he’s said so far.

And the fact that Donald Trump is the same person who wasn’t worried about COVID back in early 2020—“It’s fine, it’s going to go away, no problem”—doesn’t exactly inspire confidence.

Again, why should I trust you? Actions speak louder than words.

CINDY: What are we willing to sacrifice to beat China? Whatever that even means.

I’ll be honest. I’m not an international policy person. I don’t 100 percent understand what it means to beat China. I have a vague idea.

MATTI: I’ll send you an Instagram clip that I’ll also link in the show notes because it was a sketch about what it means to beat China. It was spot-on.

CINDY: What if we didn’t beat China? Maybe we need to, but at what cost? What are we willing to do to achieve that?

I don’t know. I really don’t. But I think it’s worth the people who understand more about this than I do thinking about it, rather than just saying, “Well, we have to do X.”

If we beat China at AI but we’re all impoverished, what does that even look like? Our GDP is up, but what does that mean?

MATTI: That’s the question. What race are we actually in with China?

Why are we building things to keep up with them that are going to do harm to our own society, or that we’ve already seen starting to do harm to our own society?

If we want to keep up with China or worry about China taking over with AI, why don’t we spend our time building defenses to protect ourselves against whatever they’re doing, rather than racing them to do the bad stuff, too?

Let’s build the defenses.

I’m not saying we should remove AI for good. AI has been game-changing and life-changing for a lot of things. Generative AI helps me at work. It’s helped me with the podcast. There are things I don’t know that I can learn from AI. I’ve learned how to edit through AI.

It’s great in some ways. But does the general population need companies like Anthropic and ChatGPT building these very smart models that can do a ton of harm? Or should those models be reserved for defense against a potential Chinese AI takeover?

The approach we’re taking doesn’t seem like the right way to protect ourselves.

CINDY: It’s one thing to come up with a product, but there should be some kind of regulation that comes from outside.

The FDA is certainly beholden to big food. I’m not a big fan of the FDA, but they are technically an outside agency. They have rules and regulations. Sometimes companies won’t follow those rules, and they get a slap on the wrist. It’s not enough, but then you fix the regulation.

I don’t understand why AI needs to be immune to the kinds of rules that other powerful industries have.

It seems fairly obvious to me. People compare it to big tobacco. It was a very harmful, very pervasive product, and you’re still allowed to smoke. But there’s regulation. There are rules and laws.

Why are these people immune from that just because they happen to work in computers? They’re rich, they’re richer, they’re cooler, they have great hoodies. I don’t understand. It doesn’t sit well with me.

MATTI: Yeah. Well, this won’t sit well with you either, because speaking of China, guess what? According to Forbes, AI is too expensive.

Especially when we think about companies implementing AI. It’s so expensive, and the pricing isn’t transparent. You sometimes pay by tokens, and you never know how many tokens you’re buying or using. All of a sudden, they’re like, “You’re out of tokens. You need to pay another $80 to get another batch.”

It’s incredibly expensive. I think companies are realizing they’ve invested in AI because every other company was doing it, and now they’re asking, “How the hell are we paying for this?”

So guess where companies are going for AI models and technology that’s cheaper?

CINDY: China.

MATTI: Correct. We’re trying to beat China—or whatever the hell that means—yet American companies and American users can’t afford what Anthropic and ChatGPT are putting out in our market. So they’re going to the Chinese models because they’re less expensive.

CINDY: Awesome. I feel great about it.

MATTI: It’s a little concerning.

You know how you put in a prompt and it works its way through all the data centers everyone hates? It takes 25 or 30 seconds to get your response during peak hours. These Chinese models apparently do it in half a second.

CINDY: Did it do that without being annoying and telling you it’s calculating the email, thinking about the considerations, contemplating the depths of the inquiry?

MATTI: It wants this, but there’s this roadblock. Wait, I should try this. Actually…

CINDY: I don’t like that part of it. I’m not pro-Chinese AI, but small note: use less water, have fewer data centers. Let me think about it. Oops, thinking.

So, Who’s Responsible When AI Goes Wrong?

MATTI: I wanted to start with all the elephants in the room and some of the ridiculous thinking that’s going on around regulation. We’re running up against a wall and making decisions that aren’t actually leading to what we want.

So that’s setting the very optimistic stage for a world with AI.

But speaking of regulation, there’s a whole other thing we can talk about: AI and the legal system. There’s so much to discuss about AI and the law.

Now the question becomes: If AI can do all of these things, what are the laws around AI? How do the laws apply?

We can’t agree on slowing down and regulation. Zuckerberg doesn’t want to. Apparently, Dario Amodei does want to. Or does he? Unclear.

CINDY: Doesn’t he really? His brand is wanting to. But he doesn’t actually follow through with it. That’s his brand: “I want to.”

He’s like, “I’ll do it if everyone else does it,” but he knows everyone else won’t do it.

“Use my product because I’m nicer.” But you’re not nicer. You’re just saying you’re nicer.

Anyway, pin in you, Dario.

MATTI: There are some really interesting questions around how AI is affecting legal systems. How does the law even define it or litigate it? How do you litigate an AI agent?

It’s autonomous. It has autonomous abilities, and it’s unclear who’s responsible: the human who prompted it or the company that built it.

And what happens if the human prompted it, but it went rogue in a way that neither the human nor the company intended?

All right, that takes us to the end of Part 1.

Join Cindy and me next week for Part 2, where we dive into some of these legal questions, look at examples of how AI is already affecting legal proceedings, and explore whether all of this is going to add even more complication to an already confusing AI landscape.

See ya then!

This transcript has been edited for readability.