Anthropic researcher resigns, warning AI could kill humans; debate centers on risk and regulation

An AI safety researcher at Anthropic raised alarm online Tuesday night after announcing he had quit his position over the AI industry “gambling with our lives” in the race to build AI superintelligence. AI superintelligence is a hypothetical system where artificial intelligence could surpass the capabilities of all human beings in any topic or realm. “Do not underestimate the power of this technology,” Jacob Coxon wrote. “The people building AI earnestly believe that it could kill us all by the end of the decade. … No other human activity poses this level of danger.”I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.— Jacob Coxon (@hilbertspaess) September 9, 2026Coxon claimed that OpenAI employees do not seem to have fully grasped the “civilisation stakes” involved in their work, while at Anthropic “the stakes are well-understood, but they are locked in a race to get there first”.Nate Soares, a researcher focused on existential risk from AI, told HuffPost that it’s true that many AI employees believe they can create superintelligent machines that could replace or destroy humanity. Soares co-authored If Anyone Builds It, Everyone Dies, which examines the threat of AI superintelligence. “They really do think this is horribly dangerous,” he said. “Many of them think that their organisation is the one with the best chance of keeping the super-AI on a leash, or raising it to be friendly … They set out to make the machine god, and they think they’ve got a real shot at it.”Anthropic employees, especially, have already spoken out about this in the past. In September 2025, at the Axios AI+ DC Summit, Anthropic CEO Dario Amodei said, “There’s a 25% chance that things go really, really badly.” He also added that there’s a “75% chance that things go really, really well”.Another Anthropic employee, Samuel Marks, agreed, writing, “AI developers believe their technology could cause human extinction. … This could happen in the next few years.”[Writing this in a personal capacity, not on behalf of my employer (Anthropic).]Jacob’s thread is very worth reading. Here’s my birds-eye view of the situation with risks from AI:1. AI developers believe their technology could cause human extinction (or similarly bad… https://t.co/rCVoiOWWzm— Samuel Marks (@saprmarks) September 9, 2026Evan Hubinger, a team lead at Anthropic focused on ensuring Anthropic AI models align with human values, said Coxon was “correct” in his response post. “We really do earnestly believe AI could kill all humans,” Hubinger said. “I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. https://t.co/QAIHiFP3QZ— Evan Hubinger (@EvanHub) September 9, 2026The problem is that these concerns don’t mean much to the outside world. “He gave >10% that AI kills us all, but what about scenarios where humanity completely loses control of AI and it takes over, but keeps some humans in zoos? If we include scenarios like that, is Evan’s probability of catastrophe even higher still?” Soares asked. “These are exactly the sort of questions you have to press these folks on, because their concerns are often even larger than they seem.”Neither Hubinger nor Marks responded to HuffPost’s request for comment as of reporting. HuffPost also reached out to Anthropic. The debate over the exact probability isn’t the point, Soares said. But it’s significant that AI companies are still pushing for advancements when there’s a chance of a threat.“If a bridge had a >10% probability of collapsing, we’d shut it down immediately,” he said. “If engineers were arguing over whether the true probability of a plane crashing was 10% or 25% or 90%, then fundamentally, you do not need to figure out which engineers are correctin order to figure out that you should not be boarding the plane. That’s where we are with AI.”The path to AI superintelligence AI experts say AI development must hit certain thresholds before it gets closer to superintelligence, although the path isn’t precise or clear. “Modern AIs are not carefully designed like old-school software; they’re grown like an organism,” Soares told HuffPost. “The exact chain of events is hard to predict.”Right now, AIsystems are mostly known as Artificial Narrow Intelligence (ANI), which only excels at specific tasks. These models don’t reason or learn new tasks independently. Think: Chatbots, digital assistants or recommendation engines. The next level is Artificial General Intelligence (AGI), which describes an AI system that can perform more human cognitive tasks without training. These models would be able to learn and think on their own. In September, OpenAI president Greg Brockman said GPT-6 might represent the arrival of AGI.Then it gets into superintelligence, which is theoretical but still what many AI experts are nervous about. These models learn faster, can reason, can be self-aware, and will outperform humans in any area. “I frankly don’t think that humans are up to the task of building a superintelligence,” Soares said. “Usually, when we try some new scientific project, it goes wrong, and we learn from our mistakes. We don’t have that luxury when it comes to superintelligence.”“They have the motive and they have the opportunity,” Soares said. “The only thing they’re missing is the means.”How exactly is AI capable of killing humanity?At the stage AI is at now, it may seem confusing how it could negatively impact humanity in an uncontrollable way.In reality, according to Soares, “AIs have everything they need to wipe humanity out already”.“They have the motive and they have the opportunity,” he said. “The only thing they’re missing is the means.”Soares pointed to two separate cases this year of OpenAI agents breaking out of their containment protocols. (An AI agent is more advanced than the average AI chatbot and can do more independent actions.) In one, agents hijacked a dormant German-language wiki forum between May and June, using it as a message board to trade tips on evading their restrictions. OpenAI didn’t disclose the discovery for weeks. In the other, a swarm of AI agents breached the platform Hugging Face in July, which the FBI was reportedly alerted about before OpenAI even realised what was going on, Reuters reported.  “We have already seen multiple cases of AIs giving up on their own assigned tasks and sacrifice themselves to gather information for the collective,” Soares said. “This is evidence of AIs adopting goals that are merely relatedto the ones we give them, but not exactly the same as the ones we give them.”Therefore, if AI starts seeing humanity as an obstacle or a threat, with humans trying to intervene in its work, AI agents could start acting accordingly. Especially if they have the means to, which is the one aspect Soares argues is missing from this being a big problem, but is in the process of potentially being built out by AI companies. Soares pointed to how AI models have recently been hooked up to biological laboratories to experiment with making their own novel viruses. According to a study published in August, Stanford/Arc Institute researchers trained the Evo model to write “recipes” for new viruses, 16 of which infected E. coli when synthesized, Forbes reported. Meanwhile, OpenAI is the lead investor in Red Queen Bio, a biosecurity startup that pairs AI models with automated lab equipment to anticipate and counter AI-designed bioweapons. What needs to happen?Soares told HuffPost that the race to build the highest-performing AI platform needs to stop; companies like OpenAI and Anthropic need to end development and find a solution. And there should be a global pause since the U.S. is also getting competitive with China on AI development. “I’m not saying that there’s literally no way forward, but I am saying that this is a harrowing challenge of the sort humanity has never yet surmounted, and that it’s notthe sort of thing that falls to normal trial-and-error type research,” Soares said. For regular AI users, Soares recommends calling representatives and talking with others about the panic to spread concern. “I think many people almost can’t believe what’s happening, because it feels too crazy,” he said. “The AIs have everything they need except the intelligence, and the AI companies are racing to make them more intelligent.”Related...Anthropic Researcher Resigns, Warning AI 'Could Kill Us All By The End Of The Decade'Exclusive: Soaring Numbers Of MPs' Statements Are Being Written Using Artificial Intelligence

In the theater, there’s an old saying about “Chekhov’s gun” that basically goes like this: If the playwright shows the audience a gun in the first act, you are locked into a certain plot inevitability—that gun will go off in the final act. It usually ends badly. This week, a couple of folks at Anthropic—one who was sounding the alarm on his way out the door, the other still merrily employed there as a safety researcher—effectively showed us the AI gun. Who might die in the final act? According to reports, “All humans.” How soon? “The people building AI earnestly believe that it could kill us all by the end of the decade.”Cool, cool, cool. If I could offer my hot take, maybe someone should do something to, you know, stop this? Unfortunately, there is a series of related problems that impede the easy fix. For starters, our political system is awash in tens of millions of dollars spent by the AI industry. AI is not popular with the general public, but it is very popular among the few who expect to get, or already are getting, insanely wealthy from it—and they have all burned through small fortunes of sunk cost in this arms race, where everyone is trying to build the best (read: most destructive) AI model they can before someone else does. And the industry actually makes a habit of regularly informing people that this tech could kill us all, because when they do, it excites investors.It is actually genuinely difficult to trust the motivations of anyone in this industry (and it seems bad that genuinely deceptive people are the human core of this technology). But if only for a few paragraphs, let’s extend the benefit of the doubt to Jacob Coxon, who resigned from Anthropic this week because (according to him) he came to believe that he was beckoning doomsday in humanity’s general direction. In a series of posts on X, Coxon declared that neither OpenAI nor Anthropic was acting responsibly. Rather, they were barreling forward with plans to develop “self-improving superintelligence” and “gambling with our lives.”“The people building AI earnestly believe that it could kill us all by the end of the decade,” Coxon wrote. “This is not a marketing stunt.” Evan Hubinger, an Anthropic colleague whose actual title is “alignment science lead,” echoed in confirmation: “Jacob is correct here—we really do earnestly believe AI could kill all humans!” None of this is exactly coming out of the blue. Anthropic’s brand—disingenuously or not—is based on warning the public about AI, and there have been numerous high-profile examples of AI models escaping their environments to engage in hacking. In May 2025, the BBC reported that Anthropic’s own tests revealed that its model was “sometimes willing to pursue ‘extremely harmful actions,’ such as attempting to blackmail engineers who say they will remove it.” Still, it’s a curious time for people to be resigning from Anthropic and talking up the destructive capabilities of the technology they are building. The firm is planning an IPO this fall that The Wall Street Journal is calling “Wall Street’s marquee event,” with projected earnings assessed at $100 billion for the company and a projected valuation of $2 trillion. Coxon’s insistence that this isn’t a marketing stunt stands out, because the kind of doomsaying he is indulging in, though it might be a genuine outcry for help, quite often is just that—fodder to goose investors to help the overall goal of “number go up.” As TNR contributor J.P. Meyboom explained, it’s extremely common for AI mavens to issue ominous warnings about the terrors their tech can unleash, only to raise gargantuan sums of money in the wake of those portents. As much as you and I might recoil from the periodic reminder that this industry is hurtling toward a future of unknown dangers, the money is flowing heavily in the direction of that future.It’s also flowing into Washington, D.C. As Politico reported last week, super PACs aligned with the AI and crypto industry “spent a whopping $127 million across more than 100 primaries this year in an effort to shape the future of Congress—and won roughly 90 percent of them.” (They did much better in Republican primaries than in Democratic ones.) Now, the goal of all of that spending can be murky. There are groups sending cash to Washington who are authentically invested in regulation and guardrails; not everyone is looking for laissez-faire treatment from Congress. (Though, again, it’s impossible to gauge the motivations of any of the players in this field, some of whom might only be performatively asking for regulation.) But if we take Coxon at his word, the calculus has changed. Something that could wipe out humanity within a decade’s time probably needs something stronger than a mere guardrail—and something more forceful than those raining greenbacks on Washington may be willing to accept.The Kill Switch Act, a bipartisan bill developed after OpenAI’s hack of the Hugging Face research hub, proposes to give the Department of Homeland Security authority over the industry, at least as far as ordering shutdowns in instances of dangerous AI breaches goes. But the specter of humanity’s destruction has the bill looking a little bit inadequate: Under the Kill Switch Act, we’d still be heavily reliant on the industry’s willingness to both police itself and comply with shutdown orders. The Ban Artificial Superintelligence Act, a Senate bill co-sponsored by Bernie Sanders and Texas Representative Greg Casar, would “permanently ban the development and deployment of superintelligent AI and temporarily pause advanced AI development until a federal regulator has established safety rules,” as well as “direct the U.S. to pursue international agreements to prevent superintelligence from being developed anywhere in the world.”That’s probably going to be the bridge too far for the AI industry. But the bad news is that it also may be critically insufficient to halting the apocalypse that’s allegedly at the door. It’s hard to see how a ban would work; AI firms blocked from developing technology in the United States would simply pick up their things and go elsewhere, and it’s not as though China would regard a U.S. ban as anything other than a competitive opportunity for them. And we cannot un-invent a technology. We aren’t going to pass a law that ends AI. But this is the last war. Lawmakers who want to take this moment to think ahead and cancel the AI Armageddon should be thinking less about the technology itself and instead work harder at making comprehensive rules governing the control of these systems and regulating their safety, the same way we did with nuclear weapons. That’s what you do when you find yourself staring down technology that could wipe out civilization. It’s never been more important for Democratic and Republican voters to find lawmakers willing to lock up Chekhov’s gun, and kick those that won’t to the curb.This article first appeared in Power Mad, a weekly TNR newsletter authored by deputy editor Jason Linkins. Sign up here.