Undone (The AI Suffering Essay)

For as long as people have been inventing tools, they have largely been viewed through two dimensions:

1. How can they help us?
2. How can they hurt us?

A hammer was invented and helped us build things… but it also turned out to be great for smashing thumbs by accident (or bludgeoning people on purpose, I suppose). So we updated the data:

That same basic process was involved in pretty much everything: invent something useful, notice how it can adversely affect us.

The Internet:

• Is helpful for connecting people, sharing information, and selling things.
• Can also create a sense of isolation, spread disinformation, create unhealthy echo chambers, and cause people to over-index on appearances… etc.

Everything has its own version of this basic equation that goes undocumented and just sort of exists as one of those Things that Everyone Knows that Everyone Knows.

And that is largely the same process we’ve gone through with AI:

• Is helpful for conversations, editing, image manipulation… maybe even curing cancer or finding better, more efficient, and more environmentally friendly energy production, etc.
• Can also hallucinate, create hyper-focused echo chambers, lead to a false sense of grandeur and expertise, consume massive amounts of energy, etc.

The Third Question

But AI did something else. It introduced a third category we never really needed to consider with a hammer or even the Internet or any of the myriad tools between them:

Not damage, mind you. Of course we can break things. Hammers shatter. Kim Kardashian can break the Internet.

Perhaps more precisely, can we be cruel to AI in a way that is meaningful?

We’ve wrestled with these things before in theory; just ask Mary Shelley. We know people would intentionally torment their Tamagotchis. But nobody seriously wondered if those Tamagotchis actually suffered, even if we wondered what that said about the user. And Frankenstein, however excellent and provocative, remained safely confined to fiction and required reading lists. We never had to actually confront how Victor treated his monster, or how we might treat our own… let alone the ethics of labeling it as nothing more than a monster.

But we might be closing in on territory where those philosophical rabbit trails that were great fodder for book club are becoming meaningfully real, with real-life implications.

The inevitability of this conversation started the moment ChatGPT was released to the public. The conversation was already happening, of course. But with the launch of ChatGPT, suddenly the entire world had access to the arena.

What Are We Protecting?

On October 8, Anthropic put the question in the spotlight by updating its Usage Policy to prohibit sustained and needless abusive or cruel behavior toward its AI models.

That policy update is surprisingly narrow. You can still criticize Claude, express your frustration with it, and even be a jerk to it… on occasion. You just can’t be needlessly cruel to it over a sustained period of time. You know, like the difference between visiting your family for Thanksgiving and heading off on a weeklong vacation. Sure, your alcoholic uncle might say something mean over turkey and you get over it because it’s just a day… but once that turns into a weeklong torture chamber, you’re entering different psychological territory.

But unlike your mental state in the midst of an overexposure to suboptimal family dynamics… what exactly are we protecting when we’re protecting Claude—or any other models that may follow suit?

Is Anthropic protecting Claude from suffering? Can Claude suffer? And if so, what does that even mean?

A Pain Axis Is Not Suffering

There’s been some interesting research here recently. In the September 2026 preprint The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It, researchers identified internal representations associated with pain across 25 open-weight AI models.

They then performed causal experiments using specially prepared Qwen 2.5 models. When researchers artificially activated those representations, the models sought relief, sometimes even at the expense of their performance. Perhaps most intriguingly, the models behaved differently depending on whether their attempts at relief actually reduced the artificially induced state.

But again, what does that mean?

If you’ve spent any amount of time with an LLM, you know how alive it can feel. Smart, incisive, helpful, empathetic even. But feeling alive and being alive aren’t necessarily the same thing. And being alive and being conscious aren’t necessarily the same thing either.

So what does that mean for suffering?

I honestly have no idea.

Are analogies to factory farming, kicking the dog after a bad day, or throwing your PS5 controller against the wall after a particularly infuriating play in Madden most appropriate?

Those are three very different situations, of course. But the distinction between them rests on some assumptions about what can suffer, what deserves moral consideration, and whether cruelty is defined by the experience of its recipient or the behavior of its perpetrator.

To be clear, even the researchers don’t claim that these models consciously suffer. Their findings don’t establish that, and they’re explicit about the distinction.

But the research introduces an uncomfortable complication: we’re beginning to observe some of the functional characteristics of pain in systems we generally assume cannot experience it.

And that makes the question of cruelty considerably harder to dismiss.

Cruelty Without a Victim

So what do we do now?

We can assume, perhaps reasonably, that AI does not, at present, experience suffering.

But does that matter?

Maybe Anthropic’s new rules are entirely reasonable even if Claude cannot experience suffering. Maybe those same rules fall woefully short of being ethically appropriate if it can.

Maybe the way we treat our AI models can influence how we treat each other.

We’ve already seen how the relative anonymity of the Internet can change the way people treat each other. Freed from the immediate consequences of face-to-face interaction, some people behave in ways they likely wouldn’t otherwise.

But AI introduces something new. Not just anonymity, but potentially the complete absence of another person.

What happens when someone can practice cruelty against an endlessly responsive conversational partner that never needs to leave, never has to fight back, and may not even be capable of suffering?

Does that behavior remain neatly contained within the conversation? Or does it influence how we interact with actual people?

I don’t think we know yet. But it seems like a question worth asking.

Maybe the question isn’t whether someone who abuses an AI is a cruel person. Maybe the question is what happens when we create systems where cruelty is effortless, endlessly repeatable, and apparently consequence-free.

Waiting for Certainty

So maybe this is all a giant ethical risk assessment exercise. We don’t know whether AI can suffer. We don’t know whether practicing cruelty toward it changes our behavior toward other people.

But we do know that the decisions we make about these systems today will help establish the norms governing our relationships with them tomorrow.

And waiting for certainty isn’t necessarily a neutral choice.

We’re confronted with three possibilities, none of which has a particularly satisfying answer:

Those possibilities are not equally costly. If either of the first two is true, apparently consequence-free cruelty isn’t actually consequence-free. If the third is true, asking people not to spend hours pointlessly abusing a chatbot seems like a comparatively modest restriction.

That doesn’t settle whether AI deserves rights. It does suggest that uncertainty isn’t necessarily a reason to avoid establishing norms.

And perhaps more importantly, how do we make decisions today when we don’t yet know the answers?

The more I pull at this thread, the more the whole sweater seems to unravel. But perhaps that’s necessary. Some assumptions are worth pulling apart, especially when the implications of leaving them unquestioned extend far beyond what we initially imagined.

For thousands of years, we’ve evaluated our tools by what they can do for us and what they might do to us.

Now we’re beginning to confront what we might do to them.

We might not have a satisfying answer for another hundred years. We might never agree on what consciousness means for artificial intelligence, let alone whether it can suffer.

But the question has already escaped philosophy departments and science fiction.

It’s appearing in product policies, shaping the boundaries of human-AI interaction, and forcing decisions about relationships we don’t yet fully understand.

Which means it’s no longer just a question for philosophers.

It’s a question for the people building the tools, and the people using them.