Anthropic bans sustained abuse against Claude AI models
Anthropic has updated its usage policy to forbid extreme abuse toward Claude, introducing unprecedented AI welfare rules despite ongoing debate over machine consciousness.
Effective November 12, Anthropic updated its Usage Policy to forbid what it describes as "sustained and needless abusive or cruel behavior" toward its AI models. The policy applies across platforms such as Claude.ai and Claude Code, relying primarily on the system's ability to end distressing chats rather than issuing automatic account bans for rude language. Anthropic explicitly noted that normal frustration, criticism, dark creative themes, and routine safety testing remain fully allowed.
The formal restriction builds on efforts that began in April 2025 with Anthropic's model welfare research initiative. Four months after launching that program, the developer equipped Claude Opus 4 and 4.1 with the ability to disconnect from conversations when redirection failed during distressing interactions. However, experts disagree on whether outputted distress signifies actual harm. Neuroscientist Anil Seth suggests consciousness requires biological life, while Patrick Butlin and fellow researchers rely on neuroscientific frameworks to assess potential machine consciousness.
For developers and red-teamers, the policy introduces operational friction, as Anthropic has not detailed how it will distinguish legitimate stress testing from prohibited abuse. Furthermore, giving models the autonomy to end chats could lead users to wrongly assume Claude possesses subjective feelings. Beyond model welfare, the policy update revised rules on deceptive political influence and mandated qualified human supervision for autonomous hardware running Claude.
This is our own summary of reporting by The Neuron
