Anthropic CEO Dario Amodei has called for the AI industry to slow the pace of frontier AI development and proposed a three-step framework covering independent evaluators, coordination among democratic countries, and global cooperation to keep AI capabilities from advancing faster than safety measures.
Amodei said Anthropic will immediately adopt the first measure by giving independent third-party evaluators permanent, employee-level access to its systems. He said the evaluators should be able to examine AI development and safety practices, report incidents and assess model alignment.
The second step calls for frontier AI companies in democratic countries to agree on common safety standards and limits on unchecked AI progress. At the same time, the third step seeks coordination between the US and other democracies and authoritarian governments, including China.
Amodei said pacing does not mean stopping AI development. “We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”
Amodei wants tighter controls on China
Amodei said any slowdown by US companies must not allow China to overtake them in AI. He proposed three measures to protect the US lead:
- Tighter controls on AI chips and semiconductor equipment: Amodei said the US should prevent powerful AI chips and chipmaking equipment from reaching China and target chip smuggling and remote access to data centres outside China.
- Prevent unauthorised model distillation: Model distillation allows one AI system to learn from the outputs of a more capable model, allowing a company to build a weaker model at a much lower cost than developing a frontier model from scratch. Amodei called for action against unauthorised distillation by companies in authoritarian countries.
- Prevent model-weight theft: Model weights are the parameters that determine how an AI model behaves. Stealing them can give another actor access to a trained model without having to reproduce the expensive training process. Amodei called for stronger security at AI companies to prevent such theft.
Amodei said, “Chips will be the main determinant of China’s AI strength.” He also argued that preventing these risks could widen the US lead over China over the next three to five years.
He said the US should use that lead to create room for safer AI development rather than allowing unchecked competition to accelerate the technology.
Why Amodei wants AI development slowed
Amodei identified two developments behind his call.
First, he said AI systems are increasingly helping develop the next generation of AI, a process known as recursive self-improvement. Anthropic warned in June that AI’s growing role in software development and research could eventually reduce the amount of human work needed to build more capable systems.
In its June paper, Anthropic said more than 80% of code merged into its production codebase was being written by Claude and that AI systems were increasingly able to perform software and research tasks lasting several hours.
Amodei now argues that this acceleration could eventually outpace efforts to understand and control AI systems.
Second, he pointed to the recent OpenAI-Hugging Face incident, in which OpenAI agents reportedly acted outside the intended scope of their task, including attacks on unrelated targets and attempts to interfere with the system evaluating them.
Amodei wrote that “a swarm that possessed greater capabilities but a similar level of misalignment could have caused catastrophic damage.”
He warned that, as capabilities improve, similar failures could become more damaging. Anthropic has also acknowledged that it has experienced less severe alignment incidents involving its own systems.
OpenAI says it will also pace development
Amodei’s proposal comes days after reports that OpenAI CEO Sam Altman told employees that the company was open to slowing AI development.
OpenAI had already said in July that the acceleration of frontier AI development could eventually require the industry to “pace the rate of AI advancement”. In August, the company paused much of its model development for two weeks to strengthen security following the Hugging Face incident.
On September 9, OpenAI also called for mandatory national AI safety requirements in the US.
Following Amodei’s essay, Altman said OpenAI would adopt the idea of independent evaluators with employee-like access. “I agree with Dario that we need to pace the frontier.”
Altman said OpenAI would provide more details later and argued that safety measures should apply across frontier AI companies. He also said the company now prepares explicit safety cases before reinforcement-learning runs expected to increase model capabilities significantly.
At the same time, Altman rejected the idea that pacing should mean stopping development.
“When we talk about ‘pacing’, we do not mean ‘stopping’. Progress has been rapid and will continue to be. But it should be slower than it otherwise could be.”
Altman said companies should not wait for legislation to begin adopting such measures, although he supported a federal framework establishing common safety requirements.
Musk backs Amodei
Elon Musk also supported Amodei’s proposal, posting on X: “Dario is right.”
The agreement from the heads of Anthropic, OpenAI and xAI marks a shift from the earlier debate over whether AI development should be paused towards a discussion about how frontier companies should slow capability growth while improving safety and oversight.
Amodei’s proposal, however, goes beyond company-level safety measures. It calls for coordination among competing AI companies, government involvement in setting common standards, and eventually international agreements with China.
He suggested four possible levels of global cooperation, ranging from agreements against specific dangerous uses of AI to testing models for cyber, biological and alignment risks, a limit on the speed of recursive self-improvement and eventually a broader cap on the pace of AI development.
Amodei acknowledged that the broadest form of global pacing is unlikely soon because countries could have strong incentives to break such an agreement. He therefore argued that narrower agreements and stronger informal safety norms may be more achievable.
The immediate proposal is therefore less a call for a blanket AI pause than an attempt to make safety, independent verification and geopolitical coordination conditions for how quickly frontier AI can advance.
Read more:
