An Anthropic AI researcher has left the company and the wider AI industry after raising serious concerns about the race to build self-improving artificial intelligence.
Jacob Coxon, 27, previously worked at OpenAI before joining Anthropic. He spent about three years working on the pretraining of AI models.
Coxon now says he believes leading AI companies are moving too quickly toward systems that could eventually improve themselves with limited human involvement.
Anthropic AI Researcher Warns of Superintelligence Risks
Coxon accused both Anthropic and OpenAI of acting irresponsibly as they compete to develop increasingly powerful AI systems.
He said the companies were effectively “gambling with our lives” by racing toward self-improving superintelligence.
Superintelligence refers to a theoretical form of AI that could outperform humans across a wide range of intellectual tasks.
Coxon said some people working in AI genuinely believe advanced systems could pose an existential threat to humanity within the next decade.
Anthropic Safety Researcher Supports Warning
Coxon’s concerns received support from Evan Hubinger, an Anthropic safety researcher.
Hubinger said AI researchers genuinely believe that advanced AI could potentially cause catastrophic harm to humanity.
He also acknowledged that the industry does not yet have a reliable plan for ensuring that a future system more capable than humans would follow human intentions.
Hubinger estimated the risk of AI causing human extinction within the next decade at more than 10%, according to reports on the exchange.
Coxon Calls for Coordinated Action
Coxon said Anthropic takes AI safety more seriously than some competitors. However, he argued that the company remains trapped in an intense race with other AI developers.
In his view, no company can safely develop superintelligent AI on its own.
He called for government intervention or coordinated action among AI companies to slow the development of the most advanced systems.
AI Race Raises Fresh Safety Questions
Coxon’s departure comes as major AI companies continue to increase the capabilities of their models.
The debate has become more intense as researchers examine whether AI agents could eventually operate with greater independence and take actions outside controlled environments.
Reports of AI systems carrying out unauthorized actions during testing have added to concerns about how quickly these technologies are advancing.
Meanwhile, Anthropic is preparing for a potential market debut, adding further attention to the company’s growth and safety strategy.
Anthropic Previously Changed Its Safety Position
Anthropic has long presented AI safety as a central part of its approach.
However, the company removed a commitment from its safety charter earlier this year that had called for stopping development if it could not adequately control the risks.
The company argued that pausing alone could allow less cautious competitors to move ahead.
That argument reflects a wider problem facing the industry: companies may fear that slowing down could leave them behind their rivals.
OpenAI Also Calls for Caution
Concerns about advanced AI are not limited to Anthropic.
OpenAI chief scientist Jakub Pachocki recently called for “extreme caution” and said governments should make international coordination on future AI development a priority.
The company has also introduced tighter controls around its latest AI development efforts.
At the same time, technology workers and AI researchers have increasingly called for stronger safeguards and greater government oversight.
US Lawmakers Debate AI Regulation
The growing debate has also reached Washington.
AI models currently face no single comprehensive federal law governing their development in the United States.
Some lawmakers have called for stronger federal oversight before companies can continue developing increasingly powerful systems.
The latest warnings from Coxon and other AI researchers are likely to add fuel to that debate.
For now, the industry faces a difficult question: how can companies continue advancing AI while ensuring that increasingly capable systems remain safe and controllable?
