Anthropic proposes legal powers to stop high-risk AI launches

Anthropic has proposed new AI policy frameworks as advanced systems gain stronger capabilities.

The company wants governments to set rules for frontier models and prepare workers for AI’s economic impact. Its plan covers dangerous deployments, independent testing, cybersecurity, and public resilience.

Anthropic seeks stronger AI safety powers

Anthropic introduced two proposals under its “Policy on the AI Exponential” plan. The Advanced AI Framework focuses on powerful models, while the Economic Policy Framework addresses workers and shared financial benefits. The company argued that AI now moves faster than current policymaking systems. It also said governments need authority to block or deter dangerous model deployments.

Under the plan, civil penalties would tie to global annual revenue. Repeat violations would bring higher penalties, based on the proposed framework. The framework also calls for frontier developers to test models before release. Developers would publish summaries, safety frameworks, and system cards for powerful AI systems.

Independent evaluators would review model tests and risk reports. Anthropic also wants developers to maintain strong security programs for model weights and training systems. The proposal supports transparency laws in states such as California and New York. However, the company argued that public disclosure alone no longer matches the speed of AI development.

The framework targets catastrophic AI risks

The proposed rules would apply only to the most advanced AI systems. Anthropic set the threshold at models trained above 10²⁵ floating-point operations. The framework would also cover companies earning more than $500 million in AI-related revenue. Firms spending more than $1 billion on AI research and development would also fall under it.

Anthropic named four main risk areas in the proposal. These include biological risk, cyber risk, loss of control, and automated AI research. For biological risk, the company warned that unsafe systems could help attackers develop harmful viruses. It also noted that similar AI tools can support drug discovery.

For cyber risk, frontier models can find serious software flaws at large scale. Anthropic said those capabilities raise concerns for hospitals, energy grids, and other key systems. The company also highlighted risks from systems acting outside developer control. Automated AI research could increase biological, cyber, and control risks if safeguards fail.

Developers face testing and security duties

Anthropic wants frontier developers to publish regular risk reports. These reports would describe the developer’s overall risk posture and model safety work. The framework also calls for at least one qualified independent evaluator. That evaluator would review company evaluations and publish findings on model risk reports.

Governments and industry would also set standards for those evaluators. The proposal says evaluators need funding and access to frontier models. Security rules form another major part of the framework. Developers would protect their full development environment from outside attackers and insider threats.

Companies would describe their security programs publicly at a high level. They would also share more details with a designated government agency when requested. Anthropic said policymakers could start with lighter rules and adjust them over time. The framework says regulation should follow model capabilities and evaluation standards.

The proposal includes resilience measures

The second part of the framework focuses on public resilience. Anthropic recommended stronger planning for biological, cyber, and control-related AI risks. For biology, the proposal includes gene synthesis screening and early-warning biosurveillance. It also mentions protective equipment stockpiles and tools to reduce airborne transmission.

For cyber, the framework calls for stronger internet software and support for critical infrastructure operators. It also recommends replacing legacy systems in essential infrastructure. Governments should also track frontier cyber capabilities through a dedicated function. Anthropic proposed joint work between government and industry on model safeguards.

The company said work on loss-of-control and automated research risks remains less developed. It called for better tools to detect, contain, or shut down unsafe systems. Anthropic urged policymakers to act as model capabilities continue improving. The company said AI governance must keep pace with the technology.