The people who build the most powerful AI systems on earth just admitted, publicly, that those systems are beginning to slip beyond human control. I think we should take them at their word – and I think we should be more alarmed than we currently are.
On June 4, 2026, Anthropic published a blog post calling on the world’s top artificial intelligence companies to come up with a coordinated way to pause development of advanced AI systems, warning that the technology is improving so quickly that there’s a risk humans would lose control. This was not a statement from a watchdog group, an academic ethicist, or a senator who barely understands how email works. It came from the company that makes Claude, one of the most capable AI systems currently in existence. Jack Clark, co-founder of Anthropic, warned that the rapidly evolving AI sector currently lacks adequate processes to slow development if things move too fast, and that the industry has focused heavily on accelerating innovation without creating enough safeguards to manage potential risks.
The metaphor Clark used to describe the situation is hard to shake. He put it plainly: “You want the option to be able to take your foot off the gas and put your foot on the brake. Right now, it’s like the AI industry has a gas pedal, but it doesn’t have a brake pedal.” He said this knowing the car has been on the highway for years.
The Problem Isn’t Theoretical Anymore
The AI control risk being discussed here is not the robot-uprising scenario from a science fiction screenplay. It is something more specific, more near-term, and in my view, far more unsettling for that reason.
The blog post, co-authored by Clark and Marina Favaro, head of the Anthropic Institute, warned that AI systems are getting close to the point where they may soon be able to improve themselves without human oversight, and said reaching this threshold could lead to massive societal disruption. The technical term for this is “recursive self-improvement” – the process by which an AI system that is good enough at building AI begins building better versions of itself, each generation surpassing the last, with no human hand required to direct the process. Clark and Favaro tackled what they called “full recursive self-improvement,” meaning AI systems that can upgrade themselves without humans directing the process.
The most striking data point in all of this isn’t a projection. It’s already happening. The post reported that Claude now writes more than 80 percent of the code merged into Anthropic’s systems, up from low single digits before the company released Claude Code in early 2025, and that its engineers ship around eight times as much code per quarter as they did a few years ago. At each step of building AI, the company argued, the human role is shrinking.
Read that again slowly. The AI is writing the AI. Eighty percent, and rising. Clark confirmed that figure and indicated it could reach 100% within the next couple of years. When you ask what it looks like when humans start to lose meaningful oversight of AI development, this is what it looks like. Not a dramatic moment. Not a headline event. A percentage that ticks upward quarter by quarter until one day the number is 100 and the question of who is really in charge of building these systems has a different answer than the one we assumed.
The Acceleration Is Already Documented
The pace of this is worth sitting with. In March 2024, Claude Opus 3 completed software tasks requiring four minutes of human labor. By March 2025, Claude Sonnet 3.7 managed tasks requiring 90 minutes, and Claude Opus 4.6 executed 12-hour tasks. According to the Anthropic Institute, tasks requiring days of skilled labor will come within range during 2026, and tasks requiring weeks could be automated by 2027.
That is not exponential growth in a graph on a slide deck. That is a real timeline, with real products, measured in the time it takes a human to do actual work. In two years, we moved from four minutes to twelve hours. The industry has not produced a governance framework that moves at anything approaching that speed.
Anthropic acknowledged that recursive self-improvement “is not inevitable,” but warned that “it could come sooner than most institutions are prepared for,” and floated what it called “a global coordination mechanism” to slow or even pause AI development and allow society room to catch up. The honest reading of that statement is that one of the world’s leading AI companies does not believe the world’s institutions are currently prepared for what its own product may become.
The Strongest Counterargument – And Why It Isn’t Enough
The skeptics point out, correctly, that alarm about runaway AI has a long and undistinguished history. Sixty years after the intelligence-explosion hypothesis was first seriously articulated, the empirical record contains exactly zero instances of the phenomenon. No architecture has ever demonstrated sustained, open-ended, autonomous self-improvement. The scenario has been confidently predicted and reliably failed to materialize since the 1960s. Every previous generation of AI doomsayers was wrong. Why should this one be different?
The skeptics also note the obvious conflict of interest. Anthropic’s post comes as the company races to sell shares on the stock market, in an IPO that could value the company at nearly a trillion dollars. A safety-focused company warning the world about AI risk while simultaneously sprinting toward a trillion-dollar valuation is, at minimum, a complicated posture. Clark and Favaro argued that a pause would provide the world with the breathing room it needs to adjust to AI’s rapid growth. One person’s genuine concern is another person’s excellent IPO narrative.
I think both of those points land. But I also think they miss the key shift that makes this moment different from every previous AI alarm.
What is new is not a theoretical argument. It is the internal data from inside a leading lab showing that their AI is already writing most of the code that builds their AI. That’s a documented, present-tense fact, not a projection. Recursive self-improvement – a process in which AI systems build, test, and improve themselves – is a phenomenon which may come sooner than expected, the company’s research shows, and AI is no longer just changing how people work, it’s also beginning to change how AI itself gets built. The previous generations of AI doomsayers were speculating about capabilities that did not yet exist. We are now watching those capabilities develop in real time, measured in percentage points that were low single digits eighteen months ago and are now above 80.
There Is No Regulatory Framework That Can Keep Up
Here is what I find most troubling, beyond the technical question of whether recursive self-improvement will occur. We do not have a functioning governance structure for the AI that already exists, let alone for AI that builds itself.
Clark’s call for coordinated action from both the industry and policymakers is the part that matters most. He’s not asking for a voluntary slowdown. He’s urging the creation of actual systems, regulatory or otherwise, that could pump the brakes on AI development if needed. The fact that he has to make this argument at all – that no such system currently exists – is the story.
The United States government’s most recent interaction with Anthropic on AI safety did not exactly inspire confidence. According to ABC News, when the Trump administration demanded unrestricted access to Anthropic’s technologies, the company refused to comply out of fear that its services would be used for mass domestic surveillance and the development of lethal weapons that trigger without human control. The administration responded by initiating the process to eliminate Anthropic’s government contracts. The lesson drawn by the most powerful government on earth, when a safety-focused AI company tried to hold a line on autonomous weapons, was to punish the company for holding it. That is where we are.
OpenAI launched its Alignment Research Blog in December 2025, dedicated to the safety challenges of self-improving systems, writing that the company is actively researching “how we can safely develop and deploy increasingly capable AI, and in particular AI capable of recursive self-improvement.” Two of the three most powerful AI companies in the world are now openly warning about the same risk. This is not a fringe position. It is the considered view of the people closest to the technology.
And yet the response from governments has been, in practice, close to nothing. No binding international framework. No agreed standard for when a pause is warranted. No system that would actually work. Anthropic itself acknowledged the collective action problem at the heart of any pause proposal, warning that a worldwide slowdown would “likely be a good thing” but that if only one company stopped, rivals would simply race ahead. That is not an argument against a pause. It is a precise description of why an industry cannot self-regulate on an existential risk without external enforcement.
Read More: Warren Buffett Issues biggest warning for anyone close to retirement
What Nobody Has a Plan For
The uncomfortable truth sitting at the center of this story is that the people who understand AI best are the most worried about it, and the people with the power to act are the least engaged with it.
In my view, the debate about whether recursive self-improvement will happen misses the more immediate question: what happens to human oversight in a world where 80% of AI development is already AI-authored, trending toward 100%? The answer to that question does not require superintelligence. It does not require science fiction. It requires only the continuation of a trend that is already measured, documented, and reported by the companies doing it.
The AI control risk we should be discussing is not the dramatic cliff edge. It’s the gradual slope. The human role in building and checking these systems has been shrinking by percentage points, one quarter at a time, in plain sight. The engineers are now reviewers. The reviewers will eventually become spectators. Nobody in a position of regulatory authority has articulated what happens after that, because no regulatory authority has seriously grappled with the trajectory.
Anthropic calling for a global pause is, in one sense, a significant thing. In another sense, it is a company that builds fast cars asking for a road that doesn’t exist yet to be built faster than the cars are moving. That road has not been started. The cars are already out there. And as Jack Clark put it – the industry still doesn’t have a brake pedal.
The Part That Should Actually Worry You
The most clarifying thing about this moment is not the technology. It’s the politics. A safety-focused AI company drew a line on autonomous weapons and mass surveillance. The most powerful government on earth responded not with a negotiation or a framework but with a ban and a national security designation. The lesson that took is a bad one: safety commitments are liabilities, not assets, when the customer has a gun.
This means the argument for responsible AI development now has to be won entirely within the industry itself, because the external enforcement layer has, at least for now, chosen the other side. That is the actual governance gap. Not the absence of a treaty or a framework document, but the demonstrated willingness of a major government to punish the company that tried to hold a line. Every other lab watched that happen. Every other lab drew its own conclusions.
The percentage of code that AI writes keeps climbing. The human oversight keeps shrinking. The institutions that were supposed to hold the line have, in the one real test we’ve had, walked away from it. What Clark and Favaro are really describing in their blog post isn’t a future risk. It’s a present condition, measured in quarterly data, with no credible counterforce in sight.
AI Disclaimer: This article was created with the assistance of AI tools and reviewed by a human editor.