Featured image courtesy of Anthropic.

Jacob Coxon, a senior safety researcher at Anthropic, resigned this week, stating there is “more than a 10 percent chance” artificial intelligence could “kill all humans” by the end of the decade. Ars Technica first reported this warning, which adds to internal concerns at a company already developing rapidly advancing AI. Coxon described Anthropic’s development efforts as a “mini Manhattan project” and called this “crunch time for humanity,” according to Wired.

Worries like this have surfaced before at Anthropic. As this outlet reported on August 7, the company was already contending with “rogue AI reports.” In mid-August, Anthropic launched watermarking for Claude-generated text and files. This aimed to address concerns about content origin and potential misuse. Coxon’s departure suggests these internal measures may not calm fears among its researchers.

These warnings come as Anthropic continues aggressive development and pursues significant financial ambitions. Just last week, on September 3, the company announced the release of Claude Fable 5.1 and Mythos 5.1 models. This followed its August 6 decision to hire an AI chip design team, a move towards specialized hardware for its next models. Rapidly increasing capabilities and internal safety alarms create tension within the company.

Adding to this pressure is Anthropic’s planned IPO, which sources now say has shifted towards mid-October. The company’s potential valuation, reported by The Hindu in August, sits at $2 trillion. This immense financial target prioritizes rapid progress and market dominance. This could create friction with safety protocols that might slow development. The departure of a senior safety researcher in this pre-IPO period raises questions about the company’s risk management.

Anthropic faces practical and legal challenges, beyond the existential safety debates. A class action lawsuit, detailed by The Verge, alleges that Anthropic misled power users about the benefits of its top-tier subscription plans. Customers claim they did not receive the promised value, with some popular applications, like OpenClaw, reportedly cut off. This shows a gap between Anthropic’s stated focus on power users and what those users experience.

User trust issues deepened when Anthropic recently warned subscribers about hackers stealing Claude tokens. As TechCrunch reported, a user in August noticed their account consuming tokens despite no activity. Such security vulnerabilities hurt customer confidence and the reliability of Anthropic’s services, especially for the power users it claims to prioritize.

The company also faces complex intellectual property disputes. On September 1, Anthropic was sued by Sony and Warner Music over alleged copyright infringement. This week, TechCrunch reported that authors are pushing back against publishers and agents who seek a share of settlement payments related to these types of claims. Even as Anthropic tries to resolve past issues, the distribution of any settlement remains contentious. This shows the legal and ethical complexities of training large language models on existing content.

What did a senior Anthropic researcher warn about?

A senior Anthropic safety researcher stated there is more than a 10 percent chance artificial intelligence could “kill all humans” by the end of the decade.

When is Anthropic’s IPO expected to launch?

Anthropic’s IPO launch has shifted towards mid-October.

What is the estimated valuation for Anthropic’s IPO?

Anthropic’s IPO is estimated to have a $2 trillion valuation.

Compiled by Launch91 Desk from the sources linked above. More about Launch91.