By Salma El-Sakka
Anthropic has warned in updated regulatory filings that it could imminently reach a milestone known as recursive self-improvement, where AI systems become capable of improving themselves without human intervention, and may not recognize the crossing until after it has occurred.
IPO Filings Reveal New Risks
The company is preparing for an initial public offering that would value it at roughly $1.5 trillion. Its prospectus devotes significant space to business risks, including a warning that its AI could pose a "catastrophic or existential" threat to humanity. The latest update adds that future models might soon show "dramatically accelerated progress attributable to AI," which Anthropic defines as recursive self-improvement (RSI).
Recursive Self-Improvement Defined
RSI describes a scenario in which AI systems are sufficiently powerful to enhance their own capabilities autonomously. Proponents argue this could lead to rapid advances in AI tools, while critics warn of new dangers. Anthropic states it has not yet met its own definition of RSI but continues to assess underlying data on its existing models.
Safety Measures and Delays
The filings caution that capabilities could outpace safety, alignment, and security measures. Anthropic acknowledges it relies on lagging indicators and has only a limited ability to forecast how quickly its systems are advancing. As a precaution, the company has delayed and restricted the release of particularly powerful models, such as Mythos, which excels at discovering cybersecurity vulnerabilities and is available only to approved individuals and organizations.
Human Control Commitment
Anthropic commits to ensuring humans remain in control of AI development. It does not expect its Claude model to safely develop new models without meaningful human supervision, and it is updating policies to keep human oversight central to frontier model development.
Competitive Risks
The company warns investors that a conservative safety approach may increase costs and could cause it to fall behind competitors who do not adopt similar safeguards.
