Anthropic PBC has sounded the alarm on the rapid pace of artificial intelligence development and is calling on the world’s top AI laboratories to consider slamming on the brakes.
In a blog post today, the company warned that AI systems are getting close to the point where they may soon be able to improve themselves without human oversight, and said reaching this threshold could lead to massive societal disruption.
The startup’s solution is for a globally coordinated agreement to temporarily pause or at least slow down the pace at which new frontier models are being developed. In the blog post, Anthropic’s head of internal research Marina Favaro and head of policy Jack Clark argued that a pause would provide the world with the breathing room it needs to adjust to the pace of AI’s rapid growth.
The two authors said model advances appear to be getting closer to a theoretical concept known as “recursive self-improvement,” which refers to AI systems that are capable of independently improving themselves and expanding their own capabilities, simply by writing their own code. They acknowledge that this hasn’t happened yet and isn’t necessarily inevitable, but they also warn it could happen – much sooner than anyone is prepared for. According to Clark, there’s reason to believe that some models could be capable of recursive self-improvement within just two years.
“We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology,” the authors suggested.
There are reasons to listen to Anthropic when it makes such a dire warning. The company recently concluded a massive, $65 billion funding round that brought its valuation close to $1 trillion, and this week it filed paperwork with the Securities and Exchange Commission stating its plans to list on the public market. It has emerged as one of the frontrunners in an intense competition for market supremacy with ChatGPT developer OpenAI Group PBC, which is also expected to launch an initial public offering soon.
However, Anthropic, which has carefully cultivated an image of erring on the side of caution and advocating for public safety, has also been accused of leveraging policy to try to slow down the advances of its competitors. David Sacks, a venture capitalist and an informal adviser to U.S. President Donald Trump, has previously accused the company of running a “regulatory capture agenda.” He argues that the company uses fear-mongering in a calculated way to encourage heavy-handed regulations that would essentially ban lower-cost open-source models in order to boost the popularity of its own proprietary algorithms.
Whether such a slowdown is even feasible is highly debatable. Enforcing a global pause on AI development would likely be near-impossible, Rob Enderle of the Enderle Group told SiliconANGLE. “This would be practically impossible, because the economic and national security stakes are simply too high for any superpower to willingly hit the brakes now,” he said.
Anthropic conceded that it would likely require something similar to the Cold War-era treaties that slowed down the pace of nuclear weapons proliferation, but noted that it’s a lot easier to mask AI training runs than it is to conceal missile silos. Unless some kind of ironclad verification regime is put in place, with cooperation from countries such as China, any pause would risk creating a scenario where competing nations simply ignore the pause and take the lead in AI development.
“Tracking decentralized computing resources, private data centers and algorithmic research globally is far more difficult than monitoring something physical, like nuclear facilities,” Enderle added.
The analyst said Anthropic’s blog post is more about strategic marketing than any concrete initiative to actually rein in AI developers. “The productivity gains it’s reporting are likely to be legitimate, but promoting its progress toward recursive self-improvement is a more calculated move,” Enderle said. “It’s hyping its capabilities to investors to position itself as the absolute bleeding-edge leader in the space, and subtly driving the narrative that its technology is so advanced it requires massive, ongoing funding to manage safely.”
Holger Mueller of Constellation Research said he was encouraged to see Anthropic asking itself important ethical and safety questions, but he too wondered what its motivations really are. “Is it trying to freeze the status quo so it can catch up, or simply retain its lead?” he asked. “A freeze would certainly help Anthropic to maintain its leading position in B2B AI systems and perhaps even expand its market share.”
The analyst also said that the risk of AI using recursive learning to improve itself is only theoretical, and has never been shown to have occurred in the real world. “Moreover, given the lack of AI training capacity at present, it won’t be easy for AI to expand its own capabilities,” he said. “AI definitely needs some level of regulation, and it’s better to have it sooner rather than later, but the difficult is to strike a balance that avoids causing harm without hindering progress.”
Regardless of their real motivations, the authors said they intend to push forward with their idea of holding back AI development. Anthropic Institute, which is the company’s in-house research organization that’s dedicated to understanding and shaping AI’s impact on the world, will continue collaborating with others on ways to “help build systems that a credible slowdown or pause would require.”
The company said it will try to organize a broader debate with policymakers and AI researchers to try and answer questions about the likelihood of self-recursive improvement and what a verification system might look like. “In the absence of a coordinated, global slowdown, we are left with the current situation: powerful technology being developed at breakneck speed by a variety of actors in a variety of countries, locked in a competition with one another where commercial and geopolitical rivalries are drowning out the larger existential-to-the-species aspects of the technology being built,” the authors wrote.
Image: Anthropic
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos, powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network.
About SiliconANGLE Media
SiliconANGLE Media is a recognized leader in digital media innovation, uniting breakthrough technology, strategic insights and real-time audience engagement. As the parent company of SiliconANGLE, theCUBE Network, theCUBE Research, CUBE365, theCUBE AI and theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.
