Why fears of AI self-improvement are causing ‘existential’ concerns at Anthropic and OpenAI

摘要: The Tech Download

Why fears of AI self-improvement are causing ‘existential’ concerns at Anthropic and OpenAI

Published Fri, Sep 11 2026

7:00 AM ED

正文

The Tech Download

Why fears of AI self-improvement are causing ‘existential’ concerns at Anthropic and OpenAI

Published Fri, Sep 11 2026

7:00 AM EDT

Kai Nicol-Schwarz

@in/kains

WATCH LIVE

Key Points

Anthropic and OpenAI researchers raised fresh concerns about recursive self-improvement, the process by which AI helps improve the development of models.

Both OpenAI and Anthropic have in recent months said that autonomous model improvement is happening faster than they expected.

Researchers fear humans could eventually lose control of increasingly capable systems.

In this article

Follow your favorite stocks

CREATE FREE ACCOUNT

This report is from this week's The Tech Download newsletter. Like what you see? You can subscribe

here.

Fears over the safety of AI systems — and their potential to wipe out humanity — gained new, viral traction this week.

Evan Hubinger, an alignment lead at Anthropic, said on X that he thinks there is more than a 10% chance that AI could

kill all humans

within the next decade, after a colleague quit over safety fears.

warnings from researchers

at both Anthropic and OpenAI followed. Cue a social media frenzy.

But it was in Hubinger's reply to his own post that revealed where exactly his concerns lay.

"What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought," he said.

Recursive self-improvement, or RSI, is when AI itself helps improve the process of building new models, potentially leading to spiralling capability as better systems build better systems and so on.

The worry is that if AI takes control of how new models are trained, the very humans who initially built those systems could lose control.

Warnings

Both OpenAI and Anthropic have in recent months said that this autonomous model improvement is happening faster than they thought.

"Our internal data shows Claude is accelerating AI development—a possible

来源: Brave/cnbc.com