AI could threaten human lives in the next decade
AI researcher Jacob Coxon has resigned from Anthropic after three years of work.
Artificial Intelligence company Anthropic researcher Jacob Coxon has decided to leave the company and the entire industry over potential dangers posed by AI. The 27-year-old AI researcher Jacob Coxon has resigned from Anthropic after working there for three years.
The Wall Street Journal reported that instead of issuing a standard farewell message, he raised serious questions about the attitudes of major companies developing AI in a detailed post on the social media platform X.
Coxon stated that,
Top AI companies are in a race to create systems that may be difficult to control in the future. They believe this race could prove disastrous for humanity.
The British researcher Coxon specializes in training new AI models with vast amounts of data. He said on Tuesday that he does not want to be part of the industry-wide race to create AI systems that can improve their own capabilities. He fears that such systems could become uncontrollable and pose a threat to humanity's existence.
Coxon claimed that,
Many people developing AI genuinely believe that this technology could wipe out all of humanity by the end of this decade. In other words, human civilization could end in the next 10 years.
According to Coxon,
Many in the AI industry are now using terms like 'crunch time' and 'endgame' to describe the ongoing development towards self-improving models.
He said,
We are moving towards several extremely aggressive scenarios. By the end of next year, things could be beyond our control. When American companies are competing against each other and against emerging AI companies in China, compromising on safety is almost certain.
Anthropic has not yet commented on Coxon's resignation. The company's CEO Dario Amodei and other senior officials have also repeatedly warned about the dangers of uncontrolled AI models and have appealed to the industry to slow down the pace of development.
Coxon had joined Anthropic earlier this year after leaving OpenAI, as Anthropic is considered a company that pays special attention to AI safety. However, Coxon believes that despite the company's sincere safety efforts, he no longer thinks that any company can responsibly develop Artificial General Intelligence, or AGI, without slowing down the pace of development through government intervention or industry-wide consensus.
AGI refers to AI systems that are capable of outperforming humans in various fields and tasks.
In response to Coxon's post, the head of alignment science at Anthropic, Evan Hubinger, supported his concerns.
Hubinger said,
Jacob Coxon is right. We genuinely believe that AI could take the lives of all humans. Personally, I consider the risk to be more than 10 percent in the next decade. I believe Anthropic is doing its best, but we do not have a solid plan to solve the alignment problem of superintelligence. We are clearly not on the right path in this direction.
He further stated that,
The threat from current AI models is less. Their real concern is about superintelligences that could rapidly become more powerful by continuously improving themselves. According to Hubinger, this process is advancing much faster than previously anticipated.
Coxon’s resignation is one of the early cases of an employee leaving a company due to concerns about AI safety at Anthropic. Previously, a security researcher at the company had left to study poetry, warning that “the world is in danger.”
In recent years, many researchers from OpenAI and other AI companies have also left their jobs expressing similar concerns. Anthropic has established its position among the world's largest AI platforms based on claims of significant investment in responsible AI development and safety. Top officials in the AI industry are also indicating that recent cyberattacks are signs of serious threats in the future. OpenAI stated last week that its latest model is a major achievement towards AGI and is far ahead of previous technologies in terms of capabilities.
OpenAI CEO Sam Altman told G20 officials at a summit held in North Carolina,
If people do not take immediate action, many things could go very wrong in the field of cybersecurity.
OpenAI chief scientist Jacob Pachocki also wrote in a blog post on Sunday,
“This is a time to exercise extreme caution. I am concerned that no one is prepared for the consequences of the rapid growth of machine intelligence.” He called for government intervention and coordination among companies to slow the pace of AI development.
It is noteworthy that there is currently no comprehensive federal law regulating AI in the United States. The administration of President Donald Trump prioritizes a low-government-control policy to maximize the economic benefits of AI. Critics argue that this could increase the risk of major cyberattacks and other serious harms.
Senators Bernie Sanders and Greg Casar introduced a bill last week proposing a permanent ban on superintelligence and halting the development of powerful AI models until new regulations are established by a regulatory body.
Coxon’s resignation comes at a time when Anthropic is preparing for its initial public offering, or IPO, to be listed on the stock market. It is being speculated that it could be among the largest IPOs in the world. The company is expecting a valuation of around ₹2 lakh crore and is presenting responsible AI development as its key feature to attract investors.