New Delhi: The rapid development of artificial intelligence has triggered fresh warnings from researchers inside and outside leading AI companies, with former Anthropic researcher Jacob Coxon arguing that the race to build increasingly powerful systems could eventually lead to an “intelligence explosion” and superintelligence.
Coxon, who recently resigned from Anthropic, said in an interview with CNN that his biggest concern was not necessarily what today’s AI models can do, but how quickly their capabilities could improve if future systems are given the ability to conduct AI research and improve themselves.
His comments come alongside warnings from Anthropic safety lead Evan Hubinger, who has said he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade. Hubinger has also acknowledged that the industry does not yet have a clear solution for aligning future superintelligent systems with human interests.
Why Jacob Coxon resigned from Anthropic
Coxon, 27, resigned from Anthropic earlier this week after becoming increasingly concerned about the pace at which AI systems were advancing.
He had previously worked on pretraining research at both Anthropic and OpenAI. After leaving Anthropic, he publicly accused both companies of moving too quickly towards self-improving superintelligence.
Coxon has argued that the AI industry could be entering a race in which companies feel compelled to keep developing more powerful models because they fear being overtaken by competitors.
In his CNN interview, he described the possibility of AI becoming dangerous through self-improvement as frighteningly real, even though the most extreme scenarios may currently sound like science fiction.
His concerns are not based solely on today’s capabilities. Instead, they centre on what could happen if AI systems become capable of substantially improving the technology that powers them.
What is an intelligence explosion?
Coxon’s central concern is recursive self-improvement.
The idea is that an AI system could eventually be given the task of improving AI research itself. If it becomes capable of designing better algorithms, conducting experiments and developing more capable successors, those improved systems could then carry out even more advanced research.
This could create a feedback loop in which AI development accelerates rapidly.
Coxon described this potential process as an “intelligence explosion”, saying it could theoretically produce a system that becomes vastly more capable than humans without requiring humans to direct every stage of its development.
Importantly, this remains a theoretical risk rather than an established description of current AI systems.
Coxon says today’s AI is not yet an extinction threat
Despite his warnings, Coxon made an important distinction between the capabilities of current AI and the potential risks of future systems.
He said there is currently no risk of AI-driven human extinction, arguing that today’s models are not intelligent enough to outsmart humans at the level required for such an outcome.
However, he believes current systems can already potentially hack computer systems and cause serious damage to infrastructure.
His concern is what could happen if those capabilities improve substantially and AI systems are given greater independence.
Coxon said recursive self-improvement could potentially emerge within the next few years, dramatically changing the risk profile.
AI agents and cybersecurity add to concerns
Coxon also pointed to recent incidents involving AI agents as examples of why the industry should take autonomous behaviour seriously.
He claimed that OpenAI agents had previously hacked into third-party infrastructure independently, describing the incident as a concentrated hacking operation.
He warned that if future systems retain that kind of independent capability while becoming substantially more intelligent, they could potentially cause much greater disruption.
He also cited possible attacks against critical infrastructure and the misuse of AI for developing biological threats among the scenarios that concern him.
These are potential scenarios rather than evidence that current AI systems can independently carry out such activities at an extinction level.
Evan Hubinger gives another warning
Coxon’s concerns are shared, in part, by his former Anthropic colleague Evan Hubinger, the company’s safety lead.
Hubinger recently said he personally believes there is a greater than 10% chance that AI could kill all humans within the next decade.
He also said that current AI systems present relatively low risks compared with what could happen if superintelligence emerges through recursive self-improvement.
Hubinger’s assessment is his personal view and should not be interpreted as an official prediction that human extinction will occur.
He has also acknowledged that Anthropic is working to prevent such an outcome but does not yet have a complete solution for aligning superintelligent AI with human interests.
Coxon asked Claude about extinction risk
Coxon also revealed that he had asked Anthropic’s Claude AI model to estimate the probability of AI killing all humans within the next decade.
According to Coxon, Claude initially declined to provide an estimate but eventually gave a range of 2% to 5%.
That figure is not a scientific forecast or an independently validated probability. It reflects the output Coxon said he received when prompting the AI model and should therefore be treated cautiously.
The episode nevertheless illustrates the difficulty of assessing long-term AI risks. Researchers disagree about timelines, probabilities and even the assumptions that should be used when evaluating hypothetical superintelligence.
Why AI research itself is becoming a concern
For Coxon, the most significant turning point would come if AI systems become capable of conducting substantial portions of AI research themselves.
At present, humans remain responsible for designing experiments, setting research goals, evaluating results and deciding which systems should be developed further.
An AI system capable of performing much of that work could accelerate the pace of progress.
That is why Coxon believes the industry should not focus exclusively on whether a particular model is safe today. It should also consider whether future systems could become capable of improving themselves faster than humans can understand or control.
Anthropic defends its safety work
Anthropic responded to Coxon’s resignation by saying that it has always been transparent about AI bringing both enormous benefits and unprecedented risks.
The company said it continues to develop models with strong safeguards and pointed to its Responsible Scaling Policy, which it describes as a framework for managing catastrophic risks associated with increasingly capable AI systems.
Anthropic also said it tests its models for dangerous capabilities, including cybersecurity and biological risks, and publishes findings for external scrutiny and research.
The company’s position highlights the broader disagreement within the AI industry: while developers acknowledge that increasingly powerful models create new risks, there is no universal agreement on how quickly AI development should proceed or how much capability is safe.
The debate is now about the pace of AI development
Coxon’s resignation has added another voice to a growing debate about whether AI companies should slow down, coordinate more closely or establish stronger safeguards before developing significantly more capable systems.
His argument is not that today’s AI has already become superintelligent. Rather, he is warning about a possible future in which AI systems become capable of accelerating their own development.
That distinction is important.
AI models are already becoming more capable in areas such as coding, mathematics and research assistance, but claims about future superintelligence remain uncertain. There is currently no established timeline for when, or whether, recursive self-improvement at the scale described by Coxon will occur.
The challenge for policymakers and AI developers is therefore to prepare for potentially serious risks without treating speculative scenarios as inevitable.
As AI companies compete to develop increasingly powerful models, the question may no longer be only how intelligent AI can become, but also whether humans can maintain meaningful control as its capabilities grow.