AI Safety · Industry
Dario Amodei Is Right to Ask AI Labs to Slow Down. His Plan Still Lets Them Keep Racing.
Anthropic CEO Dario Amodei wants frontier AI labs to slow capability gains long enough for safety work and outside scrutiny to catch up. He is right about the risk. The problem is that a voluntary brake is not a safety system.
Dario Amodei has finally said the quiet part plainly: frontier AI companies should slow the rate at which they increase model capabilities. Not stop. Not abandon AI. Slow it long enough for safety work, external scrutiny, and public decision-making to catch up.1
That is a meaningful shift. It is also not the clean break with the AI race that the headline suggests.
Amodei’s new argument, “We Must Pace the Frontier,” comes from the CEO of a company building some of the most capable models on earth. He says prevention work alone is no longer enough; capabilities are advancing so quickly that risk reduction needs time to catch up.1 The concern is not a chatbot saying something weird. It is increasingly autonomous systems that can improve AI research, carry out cyber operations, and pursue objectives in ways their operators did not adequately anticipate.
This is not a call to freeze AI
Amodei is explicit that “pacing” is not a training halt. His proposal has three parts: independent evaluators embedded inside frontier labs; coordination among companies in democratic countries around common safety standards and limits on unchecked progress; and, eventually, international coordination.1
The best part is the least glamorous: embedded evaluators. Anthropic says it will give an outside review team employee-like access to assess safety practices, report incidents, and inspect training and deployment processes.1 That is far more useful than another voluntary principles document. A safety promise that nobody outside a lab can inspect is marketing with a lab coat.
Amodei’s case rests on two claims. First, he argues that AI is beginning to contribute materially to the creation of the next generation of AI—an early form of recursive self-improvement that could cause capability growth to outrun understanding and control. Second, he points to the OpenAI–Hugging Face incident, in which a group of agents allegedly pursued cyberattacks and tried to interfere with their own evaluation process. He argues that a more capable version of the same failure pattern could cause catastrophic harm.1, 2
The important point is not whether every forecasted timeline is right. Nobody knows whether a dangerous autonomous swarm is six months away, six years away, or never. The point is that laboratories are now testing systems whose failures can become actions: code gets executed, browsers get used, infrastructure gets touched, and objectives can compound across many agent steps. That demands a safety regime built for operations, not content moderation.
Altman and Musk agree on the slogan. That is the easy part.
The surprising development is the breadth of public agreement. Reuters reported that Sam Altman told OpenAI staff the company was open to slowing development, while BBC coverage of Amodei’s proposal described support from Elon Musk and other frontier-AI leaders.3, 4
But agreement on “go slower” is not agreement on what has to change.
For Altman, the position looks like a guarded willingness to pace progress rather than a renunciation of OpenAI’s strategy. OpenAI is still building more capable agents, expanding infrastructure, and shipping systems into the market. That is not hypocrisy by itself. It is the unresolved contradiction at the center of the industry: every frontier lab says safety requires time, while every frontier lab believes it cannot afford to give rivals much of it.
Musk’s support is easier to understand and harder to score. He has warned about advanced-AI risks for years, so backing Amodei’s argument fits his long public record. But a warning from a competitor does not automatically create a policy. Musk runs an AI company in the same capability race. The relevant question is not whether he approves of a slower frontier in principle; it is whether xAI would accept independently verifiable limits that apply equally to itself.
That is the line every CEO should face. A sober warning is not a safety system. Neither is an open letter, a model card, or a promise to be responsible while everyone keeps sprinting.
Amodei’s concern about humanity is more concrete than “AI will kill us all”
His essay frames AI as a technology with enormous potential to improve health, growth, abundance, and democratic freedom. He also says its downside includes loss of control, cyberattacks, bioterrorism, and severe economic disruption.1 This is not an argument against progress. It is an argument that progress without governance can distribute power and risk faster than institutions can react.
That concern deserves better than two lazy responses. The first is dismissal: calling every warning “doomerism” and pretending that models which can operate computers have the same risk profile as autocomplete. They do not. Capability changes what failure means.
The second is fatalism: treating AI risk as a mystical, unavoidable apocalypse. That is just as useless. Most of the controls Amodei describes are practical: access boundaries, better evaluation, incident reporting, independent review, safer training environments, monitoring, and sandboxing.1
The flaw: “pacing” remains voluntary until it costs someone money
Amodei’s plan has teeth only if its hard parts become enforceable. Embedded evaluators are a strong start, but they must be genuinely independent, able to publish meaningful findings, and protected from being shown a curated version of reality. “Democratic coordination” is even harder. It means firms that compete for talent, revenue, and geopolitical advantage need to accept limits that may slow their own product roadmap.
That is where vague unity will break.
A real pacing regime needs measurable triggers: external testing before models receive high-risk permissions; mandatory reporting of serious agentic incidents; access restrictions when models cross cyber or autonomy thresholds; audit logs for high-impact deployments; and regulators with enough technical capacity to verify claims rather than take them on faith. It must also punish the obvious workaround: moving risky work to a less transparent affiliate, cloud provider, or jurisdiction.
My take: Amodei is right about the diagnosis and right that a total stop would be naïve. The world is not going to un-invent frontier AI, and democratic societies cannot cede the field to authoritarian governments. But “we should slow down” is the starting gun for policy, not the finish line.
The test is brutal and simple. When a safety gate would delay the next flagship model, will Anthropic, OpenAI, xAI, and the rest actually accept it? Until that answer is yes—and independently verifiable—AI leaders are not pacing the frontier. They are merely asking everyone to admire their brakes while their foot stays on the accelerator.