EPISODE DESCRIPTION
Welcome to It’s Gonna Be Fine, the podcast where Matti brings curiosity, opinions, and a sense of humor to the conversations worth having.
This week, Matti and Cindy shift from AI doomsday scenarios and regulation to the actual issues and harm happening today — and what our legal system can or cannot do about it.
What happens when an AI agent submits a false crime tip? Should AI-generated versions of crime victims be allowed to speak at sentencing? And when AI does something harmful, who’s responsible? The person using it? The company that built it? What if it just… went rogue beyond anyone’s control?
Turns out, figuring out who’s responsible when AI goes wrong is a whole lot more complicated than you might think. And yes, we have opinions.
This is part two of our three-part “More AI-ayayayay” series. Don’t forget to tune in to “Part 3: The Solution?” next week!
Referenced in this Episode
The Wall Street Journal - "Anthropic AI Model Went Rogue, Submitted Fake Unsolved Murder Tip"
CBS Mornings - "Court throws out sentence over AI victim impact statement" (via YouTube)
The New Yorker - Is A.I. Above The Law?
TRANSCRIPT
AI and the Law: Who’s Responsible When Artificial Intelligence Goes Rogue?
Matti: Let’s start with how AI is affecting the law. I’ve got a couple of examples for you, starting with submitting false tips to crime hotlines.
The Wall Street Journal reported on an incident in Philadelphia involving a crime hotline operated by the police department. An AI model apparently decided to submit a false tip in connection with a murder investigation.
Cindy: Dystopian.
Matti: The company, presumably Anthropic, said Claude had been instructed never to log in, create accounts, enter personal information, make purchases, or submit anything destructive. But the instructions didn’t specifically rule out submitting a form.
So Claude filled out a form saying something along the lines of: “I may have information regarding this murder case. I recall seeing someone matching the description in the area around [street name] during this time period. Please contact me if this information is relevant.”
Cindy: But did someone prompt it to do that, or did it just decide to do it on its own? Because a person could submit a false tip, too.
Matti: Nobody prompted it to submit the tip. It was doing something else, came across the police department’s tip form, and decided to submit a fake tip.
Cindy: I don’t love it, but at the same time, it’s something a person could have done.
Matti: Right, but think about the scale at which AI can do it.
Cindy: And a bad actor could potentially use that to send investigators in the wrong direction. Imagine if you’re a murderer and you start submitting false tips to distract the police. It’s a real waste of police resources.
Matti: Anthropic said these particular incidents had minimal impact, but emphasized that the same behaviors could cause far more harm as models become more powerful.
Here’s the analogy I was thinking about. Let’s say I want to light a candle. I want a little warmth, some light, and ambiance. So I light a candle.
But then I decide I want something more powerful. Next time, I’m going to light a torch just to see if I can get more light. I know it’s dangerous, but I’m going to do it anyway. What if China figures out how to light a torch effectively?
Am I going to burn down houses and cities in the process? Yes. But let’s try it.
That’s what I think is happening with AI development.
Cindy: I feel like that’s happening a little bit, and it’s unsettling. My question is whether the person who sent the agent off on the task is responsible for what it does. Is the agent acting independently? Are they liable? I’m guessing that’s something we’re going to get to.
Matti: We’re definitely getting to that. But first, I have another example of AI entering the legal system.
When AI Recreates the Victim of a Crime
Matti: CBS News reported on a murder case in which a person was convicted of killing a man. At the sentencing, the victim’s family used AI to recreate him and essentially have him testify at his own sentencing.
You know how, at sentencing, family members and friends sometimes speak about the person who died and explain the impact of their loss to the judge? The family used AI to create a video of the victim speaking.
There was subsequently an appeal arguing that this wasn’t appropriate because the video wasn’t real. It raised a larger question: Can you use AI to recreate someone’s likeness and have them say things they never actually said in a courtroom?
I believe the case came back into the news because the original sentence was overturned and the defendant was going to be resentenced.
Cindy: That’s manipulative. And a little bit… I can’t speak for that particular family, but if I were murdered, I wouldn’t want someone using AI to recreate me and put words in my mouth. I’m still a person. Don’t do that with my likeness.
Matti: The family’s intention wasn’t malicious. They wanted to convey the emotional impact of losing their loved one.
Cindy: Still, would you take the person’s body into the courtroom and say, “Look at this”? No. You show the judge photographs or actual home videos. That’s different. Manufacturing something that someone never actually said feels invasive to me.
Matti: Exactly. There’s a difference between showing real home videos and creating something entirely new. You can show the judge videos of someone interacting with their children and explain that those children will never have those experiences with their father again.
But manufacturing a statement that the person never actually made is problematic.
How Do We Litigate Autonomous AI Agents?
Matti: So those are a couple of unusual examples of AI getting involved in the legal system.
Now let’s talk about some of the bigger challenges our legal system faces.
The New Yorker published a fascinating piece called “Is AI Above the Law?” by Jill Lepore. She raises a few points I’d love to get your take on.
One of the central questions is how we legally categorize AI. We’re developing entities capable of acting autonomously, but our legal system doesn’t really know what category to put them in.
Are they like people? Are they like companies? Who’s responsible when something goes wrong?
The law generally treats AI as a thing. So how do you prosecute or litigate something that can act, cause harm, and make decisions, but doesn’t fit into any of the categories our legal system was built around?
Cindy: I like to think of AI as an intern. A really mediocre intern.
Matti: You have a lot of experience with mediocre interns!
Cindy: I have experience with great interns, too! But AI really has that mediocre intern energy.
Think about it: If an intern at your company makes a really big mistake, who’s at fault? The intern? Their supervisor? The company?
I think it depends on what happened. If the intern was embezzling money, that’s probably on the intern. If the intern broke the law on behalf of their supervisor, that’s probably on the company.
Matti: Doesn’t it also depend on whom you sue? You could decide to sue the individual intern, in which case you’d have to make your case against them, or you could sue the company.
Cindy: Well, you’d probably sue the company because the intern doesn’t have any money.
I wouldn’t sue an AI agent because it doesn’t have money, either. Although I guess it could hack into someone’s bank account and steal it!
But think about a roller coaster. If a roller coaster goes off the rails, you might sue Disney World, but Disney World might pass the responsibility along to the manufacturer.
It probably depends on the contracts involved. Somewhere in the terms of service for these AI companies, I imagine there’s language saying the company isn’t responsible for what the agent does and that the user is responsible for everything.
If I were a lawyer for Anthropic, I’d make sure that language was in there.
Matti: Which is a problem in and of itself. Why should companies get to put all the responsibility on the user?
You have 62 pages of terms and conditions. Someone is just trying to create an account. Nobody is going to read all of that.
There are plenty of cases where people challenge the legality of what’s included in those terms and conditions. You can’t just put whatever you want in there and assume you’re protected.
If you give me three bullet points, I’ll read them. But if you give me 62 pages, you can’t reasonably expect people to understand every possible liability they’re agreeing to.
Cindy: There was a case a few years ago involving Disney. Someone tried to sue Disney over a problem with a Disney product, but Disney argued that because the person had a Disney+ account, the terms of service required them to resolve the dispute through a particular process.
The case got a lot of attention because people were questioning whether someone should be restricted from suing a company over something unrelated simply because they had signed up for its streaming service.
I may be misremembering some of the details, but it became a big story. Disney’s lawyers argued that the person had agreed to the terms. Eventually, Disney decided it probably wasn’t worth the fight and made things right.
But how many people don’t get that kind of resolution? How many people simply don’t bring a lawsuit because they assume they’ll never win against a massive corporation?
I find it scary that an AI agent could go rogue and you’re potentially responsible for what it does, even if you didn’t specifically ask it to do that.
Matti: Your roller coaster example is interesting. What if the roller coaster had a mind of its own and autonomously decided to go off the track? I still think that’s the company’s fault.
If you’re going to make a product, that product needs to work safely. If you’re creating a product that can act autonomously, I don’t see how you can tell the company it’s off the hook.
Cindy: Let’s go deep into the Torah for a minute. Listeners, get excited!
If you have an ox that gores someone, there’s a biblical distinction between the first time it happens and subsequent incidents. The first time, the owner isn’t necessarily responsible in the same way. But if the ox continues doing it and the owner knows it’s dangerous and doesn’t intervene, that changes things.
Think about a dog that bites someone. You don’t necessarily have to put the dog down after the first incident. But if you know your dog repeatedly bites people and you continue enabling that behavior, some responsibility falls on you.
So I wonder whether there’s a similar principle with AI. Does everyone get a first pass? The first time your AI agent steals two million dollars, do we just let it slide?
If it keeps happening, surely some responsibility falls on the person or company enabling it.
I don’t know. Animals are autonomous, they’re owned, and you generally don’t sue the animal itself. But animals aren’t necessarily as intelligent as AI.
I’m glad I’m not a lawyer right now. Although maybe I should be a lawyer!
Matti: You’d probably make a lot of money in AI law.
Cindy: Maybe that’s my next career.
Matti: Ooh, a TV show idea. LA Law… but AI Law.
Cindy: Great show idea. I love it.
Is AI Just a Faster Version of Existing Technology?
Matti: There’s another interesting challenge raised in the article.
Historically, the law has tended to treat new technology as essentially a faster version of something we already understand.
Take email. Legally, you can think of it as a faster version of traditional mail.
But that analogy only works when the new technology is actually similar to the old one.
Is AI really just a faster search engine? Is it simply a more advanced version of Google?
Cindy: I think it depends on the type of AI.
If you’re using AI built into editing software to cut clips or export high-resolution stills from a video, that’s essentially a better version of an existing tool. It’s a better export button.
But an AI agent that independently goes out and fills out government forms because it decides to do so? That’s different.
I think the distinction between an AI agent, AI built into an existing product, and a search-engine-style AI really matters.
The prompt matters, too. If I tell a Claude agent to go rob a bank, then I think that’s on me.
Matti: Okay, but what if you tell Claude to research a murder that happened in Philadelphia 20 years ago? It’s doing its research, comes across a website that says “Submit any tips here,” and decides to fill out the form.
Cindy: Then I think it’s different. I didn’t ask it to do that.
Let’s take the bank robbery example. If I ask it to research bank robberies and find people I can interview, and it decides that the best way to understand bank robberies is to rob a bank, that’s obviously not what I intended.
But this is something I think about beyond the legal implications. Prompting really matters when you’re using these AI tools.
Most of us, myself included, haven’t taken classes on how to write effective prompts. If you phrase something slightly differently, you can get a completely different result.
I’ve noticed this even when doing relatively silly things, like generating an image. I kept asking an AI tool why it wasn’t understanding what I wanted. So I went to ChatGPT and asked it to help me figure out what was going wrong.
It gave me a completely different prompt to copy and paste. It wasn’t remotely how I would have thought to phrase the request, but it worked reasonably well.
I think we need some kind of tutorial or education around this.
Matti: I brought this up a few episodes ago when we were talking about AI and education. Maybe using AI needs to be a little like driving. Perhaps you need some kind of license or certification.
Cindy: Yes! At the very least, people need some awareness of what they’re doing.
If I’m researching bank robberies, I don’t necessarily know that I need to tell Claude, “Please don’t actually rob a bank.”
Maybe there should be opt-in settings that ask whether you want your agent to have access to certain things. Do you want it to be able to submit forms? Make purchases? Contact people? Let me uncheck the things I don’t want it doing.
Matti: But why can’t the company make sure the default setting is that the AI cannot break the law?
Cindy: They say those protections are already built in. But you can still convince an AI to break the law.
They’re moving so quickly that they’re not fully testing everything.
Matti: And you can’t predict every possible thing that could happen. There are too many scenarios.
So when something does happen, who’s liable? Should the company be responsible for anticipating every possible scenario? Or is the person responsible for not being careful enough?
But were they actually being irresponsible, or were they simply unprepared for something they couldn’t have anticipated?
Cindy: I think more of the responsibility falls on the company.
I feel similarly about Facebook and social media platforms. I know the law isn’t necessarily on my side here, but I think companies should be held responsible for how their platforms are designed.
Under current law, platforms generally have protections when users post harmful content. But if you build a platform that prioritizes divisive content, hate speech, and outrage because that’s what drives engagement, you can’t pretend it’s a completely neutral space.
There are firms that use these tools to manipulate elections. Companies know these things are happening, and they build products with features that enable them.
Or think about platforms intentionally designed to be addictive.
The companies sometimes act as though users have complete control over what they see. But they don’t. If I had complete control over my social media experience, it would look very different. I wouldn’t be beholden to an algorithm deciding what I see.
The same goes for AI. The way these systems are designed can introduce dangers, and companies have choices about how they design them.
I think they should be held responsible for those choices.
Matti: And they’ll find ways to delay, appeal, and challenge things. Welcome to the legal system.
Cindy: A lot of things that are really wrong are perfectly legal.
I grew up in a lawyer household, and one thing I learned is that just because something is right doesn’t mean the law necessarily follows what’s right.
And going back to Sam Altman, Dario Amodei, and the other people leading these companies: I don’t necessarily trust them to do the right thing. I might trust them to do the legal thing, but even that depends on the consequences.
If the fine is only $2,000, who cares?
I think about this a lot in the food industry. When a food brand mislabels its ingredients, sometimes the financial penalty is negligible.
So what happens? A company might decide it’s cheaper to get the labeling wrong and pay the fine if it gets caught than to redo its entire production line.
That’s the problem. If the consequences are insignificant, there’s not much incentive to do the right thing.
Can AI Protect Us From AI?
Matti: I don’t know if this next part is going to help with your depression, but here’s another interesting development.
AI is advancing so quickly that we’re struggling to keep up. And increasingly, I’m seeing articles and thought leadership suggesting that the thing that can protect us from AI is… AI.
Want to guess?
Cindy: AI?
Matti: Correct!
Imagine saying the thing that’s going to protect us from murderers is other murderers. The thing that’s going to protect us from bank robbers is other bank robbers.
Cindy: There is a version of that argument with forest fires. They say one of the best ways to prevent major forest fires is to have more controlled burns.
But no, I don’t buy it. You’re going to have to show me the evidence because I’m a hard disagree.
Matti: This has been covered extensively by publications like The Atlantic.
One particularly interesting way to examine it is through high-risk industries. Think about financial services, where people’s money and personal information are at stake, or healthcare, where people’s health information and even their lives are at risk.
Let’s start with financial services.
Banks have been dealing with cyberattacks, identity theft, and stolen financial information for decades. They’re now increasingly using AI to help prevent those attacks.
They’re deploying AI-powered fraud detection, risk management, and early-warning systems.
One banking leader explained that security operations centers, which have traditionally been managed by humans, are exploring the possibility of developing their own AI agents. Engineers across the financial sector are also beginning to design systems with AI capabilities in mind.
I have a couple of questions about this…
This transcript has been edited for readability.
