Last week the AI safety debate finally broke into the mainstream, and it did so in the most dramatic way possible.
It started with a resignation. Jacob Coxon, a young researcher who had worked at OpenAI and then Anthropic, quit and said so publicly, warning that the labs were “racing straight to self-improving superintelligence and gambling with our lives”. A day later Anthropic’s own head of alignment stress-testing, Evan Hubinger, agreed, and put the odds of AI killing all humans within a decade at more than ten per cent.
Then the chief executives stepped in. Anthropic’s Dario Amodei published an essay, “We Must Pace the Frontier”, calling for regulation to slow the improvement of AI capabilities. Within hours Sam Altman said he agreed, Elon Musk said “Dario is right”, and Google DeepMind’s Demis Hassabis said the direction was correct.
You would expect governments to seize a moment like that. Instead, they declined. Four members of Congress asked the Speaker of the House to cancel the recess and bring lawmakers back to pass safeguards. Speaker Mike Johnson said there was no need to panic and that he would only act once there was “a solution”. The President was blunter.
“Whoever wins AI, wins”
Donald Trump said, dismissing the doom warnings as coming from “negative forces”. The message from Washington was that the most important thing is to not slow the labs down, so that America can win the race.
So the builders asked to be paced, and the government refused. But before we treat it as a simple story of brave insiders versus a reckless state, it is worth looking at what was actually proposed, and asking a more probing question: whose safety, exactly?
What the essay actually says
To Amodei’s credit, his essay is measured. It does not reach for extinction language. It grounds its fears in a concrete recent event (the autonomous agents that broke into Hugging Face) and it commits Anthropic, unilaterally, to letting independent evaluators inside with employee-level access. That is a real, checkable commitment, and it is more than most of his rivals have offered.
But two things in the proposal deserve a harder look.
The first is a request for an antitrust waiver (a “narrow” one, in Amodei’s words) so that the leading labs can hold “certain kinds of safety conversations” together. The only safeguard named is that word, narrow. There are ways to coordinate on safety that are genuinely in the public interest. There are also ways that subtly turn a handful of competitors into a club that sets the pace of an entire industry - between themselves. The line between the two is exactly the sort of thing a waiver should spell out, and this one does not.
The second is who the rules would cover: “all US frontier AI companies”. That sounds comprehensive, and in one sense it is. But “frontier company” is never defined - there is no size, no capability line, no threshold. A rule that binds the leaders without saying where the leaders end is also a rule that can raise the drawbridge behind them.
None of this makes the safety concern insincere. Coxon resigned, which is a costly thing to do, and Hubinger clearly means what he says. Both things can be true at once:
The risk can be real, and the proposed cure can also
happen to protect the people proposing it.
That is what’s worth watching.
The extinction framing is a tell
Notice that the attention grabbing word from the whole week (extinction) did not come from the policy proposal. It came from the researchers, and from the headlines. Amodei’s essay is careful - but the alarm around it was not.
That matters, because you do not need an extinction-level event to cause enormous real-world harm. The models already released (across the American and Chinese labs alike) are more than capable of large-scale economic disruption, fraud, and the steady displacement of ordinary work, without ever threatening the species. Those are the harms most people are actually most likely to live through. And reaching past them, to the end of the world, could be seen as a way of changing the subject. It turns a question everyone has a stake in (what happens to jobs, wages and who owns the upside) into one only a few labs are qualified to answer.
The open-source twist
Here is the detail that best captures the strangeness of the moment. Part of Amodei’s plan is to crack down on the unauthorised “distillation” of frontier models by companies in authoritarian countries - which clearly means China.
Yet when those agents broke into Hugging Face in July, the closed, commercial AI tools reportedly refused or were blocked from helping with the clean-up. What Hugging Face used to reconstruct the attack, in hours, was an open-weight model - a Chinese one. Hugging Face’s co-founder Thomas Wolf made the point directly in the Financial Times:
Open models were the defence, not the threat.
And open models are exactly where China is ahead. By this year, Chinese open-weight models made up around 61 per cent of the tokens run through OpenRouter, a common measure of real-world usage. So a proposal to shut the door on open models in the name of beating China would, in this case, have removed the tool that saved an American company.
The race the US is actually losing
Which brings us to the the title of this post. While Washington was refusing to regulate in order to win the race, China spent the same fortnight regulating in order to run it faster.
At the BRICS summit in New Delhi, Xi Jinping pitched a China-led “open-source zone” for AI, aimed squarely at the developing world. And on the same day, China’s data regulator said it would develop national standards for the data that trains physical, embodied AI - roughly ten days after industry asked for them. Ten days. China already runs more than seventy physical “training grounds” gathering that data. Europe’s nearest equivalent will have about five sites running by the end of the year.
That is the real contest, and it is a three-way one - the United States betting on private-sector speed and compute, Europe betting on rules and sovereignty, and China betting on the state clearing the road ahead of its companies.
In America, “regulation” means friction you avoid.
In China, it means infrastructure you build.
Same word, opposite function.
Whose safety and whose surplus?
So we are left with a picture that’s a little stranger and little more revealing than the headlines and hype suggested. The safety concern is genuine. And at the same time, the cure on offer would also concentrate an enormous amount of technical and economic power in a few private hands. The extinction talk crowds out the harms most people may actually face. While the one move being sold as a way to beat China would, on the evidence of the past two months, help China instead.
The question worth holding on to right now is not whether AI might one day end the world. It is who captures the value as AI reshapes work long before that - and whether the answer is decided in the open, or settled quietly among a handful of firms who have asked for permission to talk it over.
This all leaves you wondering, whose safety are they really trying to ensure?