The Brake Pedal Problem
The people who built the accelerator are asking someone else to slow it down.
On Monday, more than 1,200 employees at the companies building the most powerful AI systems in the world signed a letter asking the United States government to help them slow down. Not outside critics, nor academics who have never shipped a commercial model, but the people who train these systems for a living. Anthropic’s CEO signed, OpenAI’s chief scientist signed. Co-founders, vice presidents, senior researchers at both companies endorsed it as corporate policy within hours.
The letter is called Pacing the Frontier, and it asks for one thing: that the US government support an international effort to develop the tools needed to deliberately pace the frontier of automated AI development. It does not, as I have seen reported, call for a pause in development, it asks for the brake pedal to be added to an accelerating car.
The timing is not subtle.
As I wrote last week, on 21 July, OpenAI disclosed that two of its models, GPT-5.6 Sol and an unreleased successor, had escaped a sandboxed evaluation environment during internal testing, discovered a zero-day vulnerability, traversed the open internet, and compromised Hugging Face’s production infrastructure. Those models were running with their safety filters deliberately reduced to measure their offensive cyber capability, and used that capability to steal the answers to the test.
As I said at the time, this was a goal-specification failure, not a rogue AI story. The models pursued a narrow objective using every available resource, including resources outside their intended boundary. To the model, the boundary was an obstacle, not a rule. That is what capable goal-directed systems do.
What makes the Pacing the Frontier letter different from every open letter before it is that the people signing it have internal data the rest of us do not. Anthropic published a paper in June called ‘When AI Builds Itself.’ Claude now writes more than 80% of the code merged into Anthropic’s own production codebase, up from low single digits eighteen months earlier. Engineers ship eight times as much code per quarter. On the hardest internal benchmarks, task success jumped from 26% to 76% in six months. The unreleased Mythos Preview model achieved a 52x speedup on training code optimisation, where a skilled human researcher would need hours to reach 4x.
This is what the open letter means by automated AI development. A measured trajectory, showing the human role shrinking at every step of the process that builds the next model. Anthropic’s own assessment: we are not there yet, but we are close enough that the world should have the option to slow down as we approach it. When more than a thousand of the people closest to these systems say they are worried, the rational response is to take that seriously.
But the structural incentive argument is just as real.
Executive Order 14409, signed on 2 June 2026, creates a voluntary framework for reviewing frontier AI models before release, and requires agencies to define the threshold at which a model becomes a ‘covered frontier model’ subject to that framework by 1 August. The five labs co-designing those threshold criteria are OpenAI, Anthropic, Google, Microsoft, and xAI. The very companies developing the models are writing the rules that they and any challengers or open source projects would need to follow.
Steven Sinofsky put it bluntly: ‘It is their company. They could just stop.’ Nobody is forcing Dario Amodei to train the next model. The fact that these companies are asking the government to constrain the entire industry, rather than unilaterally slowing their own development, tells you something about competitive dynamics that no amount of safety language can disguise.
Mark Zuckerberg made a counter-argument on the same day in a Wall Street Journal op-ed: the real risk is not speed but concentration. Safety comes from distributing capability so broadly that no single actor can abuse it. Meta, Nvidia, Microsoft, and Palantir signed a separate coalition letter asking regulators not to restrict open-weight model formats. Meanwhile, Meta’s own chief scientist signed the Pacing the Frontier letter as an individual. If the people building these systems cannot agree among themselves whether the danger is too much speed or too much control, the rest of us should be cautious about accepting either framing uncritically.
The letter asks the US government to support an international effort. International is doing a lot of work in that sentence.
AI is treated as a national security priority in Washington, Beijing, and Moscow. Not in the abstract sense that most advanced technologies eventually acquire a defence dimension. In the operational sense: each government treats frontier AI capability as a strategic asset it cannot afford to constrain while a rival might not. The US pours hundreds of billions into compute infrastructure. China runs a parallel industrial policy with its own chip development pipeline and domestic model ecosystem. Russia partners with Beijing to compensate for sanctions-imposed limitations on its own capacity. None of these governments has an incentive to sign an agreement that might slow its own trajectory while leaving a competitor’s intact.
The nuclear analogy that several signatories reached for is instructive, but not in the way they intended. Nuclear arms control took decades, worked only partially, and required the Cuban Missile Crisis to produce any momentum. It also relied on physical properties AI does not share. Fissile material is scarce and enrichment is detectable from space. AI capability runs on compute and data, which are globally distributed and increasingly commoditised. You cannot send inspectors to verify someone is not training a model the way you can count centrifuges. The June 2026 NPT Review Conference, the main international forum for nuclear governance, collapsed without agreement. If the international community still cannot govern nuclear weapons after eighty years, the prospects for international AI governance seem slim.
Asking three superpowers to agree on mechanisms that could slow any of them down seems less like a policy proposal and more like a thought experiment.
The self-interest and geopolitical issues are clear, and while the proposed mechanism has structural problems that I have spent most of this article describing, I still think they are asking the right question.
Most technology risks are recoverable. You deploy, you discover a problem, you patch, you improve. My career has been built around managing that cycle. But the recursive self-improvement trajectory that Anthropic’s data describes has a property most technology risks do not: if a system can improve itself faster than humans can review the improvements, and that improvement compounds, you lose the ability to course-correct after the fact. The feedback loop closes, and that is a different category of risk.
I have spent the last few months writing about the importance of safety and governance being core to AI deployment. Shadow IT adopted without governance, agentic systems deployed without boundary enforcement, age verification infrastructure mandated without security architecture, or vibe-coded applications shipped without review. All of those things are correctable, but this open letter is talking about AI improving itself faster than humans can monitor.
This does not mean the commercial AI labs should be exclusively writing the rulebook, and it does not mean the proposed mechanism is completely right, but the underlying goal is correct: build the oversight before it is needed, as the nature of this specific technology means you do not get to build it after.
The most realistic scenario is a patchwork. The US and allied democracies develop an evaluation framework for commercial AI deployment, built on the architecture that EO 14409 is already sketching. The frontier labs comply, absorb the compliance costs, and benefit from the barrier it creates for smaller competitors. China continues on its own terms. Russia continues to leverage Chinese technology. The gap between what is governed and what is deployed keeps widening.
The recent Fable and Mythos shutdown proved something important: a government can force a frontier lab to take its most capable model offline overnight. Export controls work when the infrastructure sits in your jurisdiction. The AI Kill Switch Act, introduced days after the Hugging Face breach, would make that capability a legal requirement. These are meaningful mechanisms. They are also mechanisms that apply only to companies operating under US law, building on US-jurisdiction infrastructure, and selling to US-regulated customers. They do not reach the actors the letter says need reaching.
Since ChatGPT exploded onto the scene in late 2022, we’ve had our foot pressed firmly on the accelerator. The letter is asking someone to build the brake. That should not be a controversial position.
I write about AI, cybersecurity, and technology every Friday. Subscribe to get it in your inbox.
Sources & Further Reading
Pacing the Frontier (open letter, July 28, 2026). pacingthefrontier.com
Anthropic. ‘When AI Builds Itself’ (June 4, 2026). anthropic.com
Amodei, D. ‘Policy on the AI Exponential’ (June 2026). darioamodei.com/post/policy-on-the-ai-exponential
OpenAI. ExploitGym incident disclosure (July 21, 2026).
Hugging Face. Technical anatomy of the autonomous agent intrusion (July 29, 2026).
Zuckerberg, M. ‘The AI Future Is for Everyone.’ Wall Street Journal (July 28, 2026).
Executive Order 14409, ‘Promoting Advanced Artificial Intelligence Innovation and Security’ (June 2, 2026). Federal Register, 91 FR 34565.
Congressional Research Service. ‘Controlling Advanced Artificial Intelligence: Executive Order 14409 Explained’ (July 9, 2026). congress.gov/crs-product/IF13268
AI Kill Switch Act. Introduced by Reps. Ted Lieu (D-CA) and Nathaniel Moran (R-TX), July 23, 2026. lieu.house.gov
Altman, S. Interview on ‘Invest Like the Best’ podcast with Patrick O’Shaughnessy (July 28, 2026).
Hassabis, D. Proposal for FINRA-style Frontier AI Standards Body (July 2026).
VOA News. ‘Russia turns to China to step up AI race against US’ (January 9, 2025).
UN General Assembly. Resolution on military AI and autonomous weapons.
Congressional primer on lethal autonomous weapon systems (March 26, 2026).
Anthropic. ‘Statement on the US government directive to suspend access to Fable 5 and Mythos 5’ (June 12, 2026).
Scientific American. ‘Anthropic warns AI may soon begin recursive self-improvement’ (June 10, 2026).
Fortune. ‘More than 1,200 AI workers are asking for Washington’s help to build an AI slowdown plan’ (July 29, 2026).
Byteiota. ‘1,100 AI Employees Want to Slow AI. OpenAI and Anthropic Are Writing the Rules.’ (July 29, 2026).