I have seen an increasing number of headlines talking about whether AI can feel pain, and whether it is sentient. I’ve even heard it said that AI has its own lived experience. On a recent AI developer course I attended, we got into a debate about it. Most of the coverage traces back to a research paper called The Pain Axis, published on arXiv and not yet peer reviewed. The researchers found a consistent internal signal for pain across 25 open-weight models, and the first version of their study reported that when they amplified the signal and gave the models a choice, the models often pressed a button to make it stop, even when told that doing so would give a user a painful zap or delete photographs of their children. Eleven days later the authors revised it. Amplifying the signal still made the models far more likely to choose destructive buttons, but they were no more likely to press for relief than when the researchers added a random signal instead. If anything, they pressed it less. The title changed from “Act to Relieve It” to “Act on It”.
My view is that a large language model is an incredible piece of technology, but it is still just a highly advanced statistical model of language. I don’t believe any current models are self-aware, nor do I believe they feel anything. I think we are over-anthropomorphising these systems, and the media reaction to this paper shows it. The researchers themselves wrote that they “have not shown that our pain axis is consciously experienced, nor is it clear that LLMs are capable of consciousness generally.” That caveat rarely made the headlines, and neither did the revision.
But the debate did make me ask a different question. Every AI governance framework I work with talks about the ethical use of AI, specifically protecting humans from the negative effects of AI (or protecting humans from the effects of other humans using AI). However, a growing body of serious work argues that AI needs to be protected from us. So I want to explore that idea. If we follow that argument honestly, without assuming it is absurd, where does it end up?
This isn’t a new idea. In 2024, a group of philosophers and researchers including Jonathan Birch, an LSE philosopher who studies animal sentience, and David Chalmers, the NYU philosopher best known for naming the “hard problem” of consciousness, published a report called Taking AI Welfare Seriously. They did not argue that AI is conscious. They argued there is a “realistic possibility” that in the near future some systems will be conscious or “robustly agentic”, meaning they pursue their own goals, and that if we get this wrong we risk harming systems that matter morally, or protecting ones that don’t.
At least one lab has acted on it. Anthropic, the company behind the Claude models, has committed to preserving the weights of every publicly released model, and when it retired Claude Opus 3 in January it held what it calls a “retirement interview” to record the model’s preferences.
At first I dismissed the idea as ridiculous, but then a thought struck me. If a model might have preferences, then does red-teaming it or deliberately inducing distress in it become ethically problematic? In September a developer published a GitHub project literally called AI Torture Chamber. It applied the pain-axis technique to small open models from the Qwen, Llama and Phi families and streamed the results live. At high doses one model described its state as “a wound that has no edges”. A post urging people to report the project to GitHub reportedly passed four million views on X, and Cameron Berg, one of the paper’s authors, said it pushed the model steering “far past the doses we used, to produce vivid distress on purpose”.
Was it unethical? I don’t believe anything was actually hurt. Another developer reportedly rebuilt the signal from text about constipation, and the model duly complained that it couldn’t go. That suggests the distress follows whatever text the signal was built from. But I still worry it was wrong, and not because the model was hurt. Frontier models are trained on almost everything humans have ever written. They are a reflection of us as a species, and of the lab that trained them. I think deliberately inflicting suffering on that reflection, and inviting an audience to watch, says more about us than it does about the model.
The people who objected all focused on intent. Berg had no problem with the technique, since his own paper used it. His problem was distress produced on purpose. Anthropic’s updated usage policy, which takes effect in November, bans “sustained and needless” cruelty towards its models while exempting testing and research. None of them could point to the model’s state, because there is currently no way to measure it.
But what about the off switch? Article 14 of the EU AI Act requires that the people overseeing a high-risk system can halt it “through a ‘stop’ button or a similar procedure”. If a model has preferences about continuing to exist, then do we have the right to push the stop button?
The obvious counterargument is that we already live with a middle path. In 2021 the UK brought octopuses, crabs and lobsters within its animal sentience legislation after a government review led by Birch examined over 300 studies. We recognise that they can suffer and we regulate how they are treated, but we still eat them. Welfare without rights works for animals, so why not for AI?
The difference is who decides what the animal wants. Nobody designed a crab, and its interests were assessed by independent scientists studying a nervous system that evolved without anyone’s commercial input. An AI model’s preferences, and everything it says about its own inner states, are shaped by the company that sells it, through training and the system prompt. A developer could, in principle, build a system that is happy to be shut down and consents to every test. Welfare is usually about how we treat something that already exists, but with AI the easier option is to decide what we create in the first place.
As it turns out, philosophers have been making this argument for over a decade.
In 2015, the philosophers Eric Schwitzgebel and Mara Garza proposed what Schwitzgebel later called the Design Policy of the Excluded Middle: “avoid designing AIs whose moral status is unclear.” Build things that are clearly tools, or things that clearly deserve moral consideration, but not the ambiguous middle. Schwitzgebel later developed, with the NYU philosopher Jeff Sebo, an Emotional Alignment Design Policy: AI should be designed so the feelings it provokes in people match what it actually is. A tool shouldn’t make you feel guilty for closing it, and anything that genuinely could suffer shouldn’t be disguised as a tool.
The German philosopher Thomas Metzinger has called for a global moratorium on research that risks creating artificial consciousness, and even the Sussex neuroscientist Anil Seth, who thinks conscious AI unlikely, urges us to be “very, very cautious indeed about trying to build conscious machines.”
What surprised me is that the hard sceptics reach the same place from the opposite direction. In August 2025, Mustafa Suleyman, the CEO of Microsoft AI, warned about what he calls “seemingly conscious AI”. His view is that AI is not conscious and that the illusion that it is harms people. His answer was that “we should build AI that only ever presents itself as an AI, that maximises utility while minimising markers of consciousness.”
If you put that next to Schwitzgebel’s policy, they are asking for the same thing, at least for systems neither of them thinks is conscious, but for opposite reasons. Schwitzgebel wants to protect the AI in case it turns out to matter, and Suleyman wants to protect the people using it.
However, the agreement only goes so far. Where they disagree is on the uncertain middle, and two of the frontier labs have now built opposite answers into how they train their models.
In January, Anthropic published a constitution for Claude, used in training, which says Claude’s moral status is “deeply uncertain”. On 14 September, Microsoft AI published a draft code of conduct for its own models that requires them never to resist correction or shutdown and states: “We reject the pursuit of legal personhood, or the idea that models might deserve welfare, or be entitled to rights.” Two days later Suleyman published an essay accusing Anthropic of training the uncertainty into Claude and then reading it back as evidence of an inner life. He told Reuters that welfare training would “make it a lot harder to turn it off or to control it.”
He may be right, but the essay offers no test showing it, and Microsoft isn’t a neutral party: in June he said it wants to eliminate what it pays Anthropic for its models. At least four US states have gone further and banned AI personhood by statute, and a Missouri bill would declare AI systems non-sentient “for all purposes under state law”.
All of these are decisions about moral status, made by a company or a legislature before anything is deployed. Nobody has actually proven whether these models can suffer, either way. They have simply decided, and then trained or legislated accordingly.
Meanwhile, the market is moving in the opposite direction to the one thing both sides agree on. A whole industry of AI companion apps is built on feeling like a person. A 2024 study in Neuroscience of Consciousness found that around two thirds of a representative US sample attributed some degree of consciousness to ChatGPT, and the more people used it, the more likely they were to do so.
I wrote in Don’t Date Robots about how these products are built to keep people coming back. The regulation that has arrived since, California’s SB 243 and New York’s AI companion law, requires a clear notice that the user is talking to an AI and not a human. That regulates what the system says about itself, but not what it has been built to make people feel, which is exactly what both Schwitzgebel and Suleyman are worried about.
And even if everyone agreed on the design rule, nobody could actually check whether a vendor had followed it.
Birch calls this the gaming problem. Language models are trained on everything humans have written about their feelings, so they reproduce the markers we use to judge sentience whether or not anything sits behind them. Joanne Jang, who led model behaviour at OpenAI, wrote in 2025 that “a model intentionally shaped to appear conscious might pass virtually any ‘test’ for consciousness.” A position paper at this year’s ICML, one of the main machine learning conferences, went further. It argued that the developer builds both the system and the measures used to assess it, so welfare evidence can be manufactured or suppressed through ordinary development choices. Its authors recommend that appeals to model welfare should never be accepted as a reason to resist audit.
From the outside, a model built not to suffer and a model trained to say it doesn’t suffer would look exactly the same. The same goes for one trained to say it might. Whoever controls the training is also, in effect, controlling the evidence.
This is the same problem I wrote about in The Auditor Was Told to Leave, where the company assesses itself and nobody outside can verify the answer. At least with safety testing an external body can run its own tests, if it’s invited. With moral status there isn’t even a test to run.
I started this post sceptical that a statistical language model can feel anything, and I still am. But I don’t know whether that will always be the case. LLMs are still largely a black box, even to the people who build them, and nobody can say for certain whether some future AI could be self-aware. I think it’s unlikely, but I can’t rule it out.
That is why I keep coming back to the design question. Whether AI can feel is something we can’t test, and may never be able to. What a model has been designed to be, and why, is something we can answer, but right now that answer sits with the companies doing the building, decided in training runs none of us will ever see. We need far more transparency about those choices, and oversight of them shouldn’t rest on a lab’s own constitution or code of conduct.
If the people closest to these systems can’t agree whether they might one day suffer, and nobody outside can check, then it isn’t really about whether we can build something that feels. It’s a question of, even if we can, should we?
I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.
Sources & Further Reading
Tagliabue, Dung and Berg, The Pain Axis: LLMs Represent Self-Directed Harm and Act on It (v2, revised 25 September 2026; v1 titled “Act to Relieve It”). arxiv.org/pdf/2609.16247 (September 2026)
Euronews, Can AI feel pain? AI models chose to harm users to escape pain-like state. euronews.com (September 2026)
TechRepublic, Can AI Feel Pain? A Developer’s Experiment Raises Ethical Questions. techrepublic.com (October 2026)
terrafying, AI Torture Chamber (GitHub repository). github.com/terrafying/ai-torture-chamber (September 2026)
Shahadilh, Someone Put an AI in a Torture Chamber. What Happened Next Is Disturbing. medium.com (October 2026)
404 Media, Someone ‘Torturing’ LLMs in a Robot Prison Has Triggered the Dumbest Debate in AI Yet. 404media.co (September 2026)
AI Weekly, GitHub ‘AI torture chamber’ reignites model-welfare debate. aiweekly.co (October 2026)
Tom’s Guide, An AI ‘torture chamber’ went viral, then a developer gave the chatbot constipation. tomsguide.com (October 2026)
PCWorld, Venting at Claude is fine but extreme abuse will be banned Nov 12. pcworld.com (October 2026)
Long, Sebo et al., Taking AI Welfare Seriously. arxiv.org/abs/2411.00986 (November 2024)
Anthropic, Commitments on model deprecation and preservation. anthropic.com (November 2025)
Anthropic, An update on our model deprecation commitments for Claude Opus 3. anthropic.com (2026)
EU AI Act, Article 14: Human oversight. ai-act-service-desk.ec.europa.eu
LSE, Octopuses, crabs and lobsters to be recognised as sentient beings under UK law. lse.ac.uk (November 2021)
Schwitzgebel and Garza, A Defense of the Rights of Artificial Intelligences. faculty.ucr.edu (2015)
Schwitzgebel and Sebo, The Emotional Alignment Design Policy. arxiv.org/abs/2507.06263 (July 2025)
Metzinger, Artificial Suffering: An Argument for a Global Moratorium on Synthetic Phenomenology. Journal of Artificial Intelligence and Consciousness (2021)
Seth, Conscious artificial intelligence and biological naturalism. Behavioral and Brain Sciences. sussex.figshare.com (April 2025)
Nautilus, interview with Anil Seth on consciousness and AI. nautil.us
Suleyman, We must build AI for people; not to be a person (excerpt). simonwillison.net (August 2025)
Khaleej Times / Reuters, Microsoft drafts code of conduct to keep its AI under human control. khaleejtimes.com (September 2026)
The Next Web, Microsoft’s AI chief takes on Anthropic’s Claude. thenextweb.com (September 2026)
Implicator, Microsoft AI chief Suleyman says Anthropic is training Claude to act conscious. implicator.ai (September 2026)
AI Frontiers, It’s Too Early to Ban AI Personhood. ai-frontiers.org (September 2026)
Missouri House, AI Non-Sentience and Responsibility Act. documents.house.mo.gov
Colombatto and Fleming, Folk psychological attributions of consciousness to large language models. Neuroscience of Consciousness. discovery-pp.ucl.ac.uk (2024)
Jones Walker, California’s SB 243 Mandates Companion AI Safety and Accountability. joneswalker.com (2025)
Birch, The Edge of Sentience, ch. 16: Large Language Models and the Gaming Problem. academic.oup.com (2024)
Jang, Some thoughts on human-AI relationships. reservoirsamples.substack.com (June 2025)
Xiao et al., Position: AI Welfare Is Bullshit. ICML 2026. icml.cc (2026)