Anthropic has reportedly offered Congress a draft AI safety bill. This is reassuring in the way it would be reassuring to buy a powerful machine and discover that the manufacturer has also designed the inspector’s emergency brake.
Politico reports obtaining the proposal, which would limit when government could block an advanced model’s release or use and preempt some state safety laws. Anthropic calls its proposals sensible. It is a draft being pitched, not enacted law. The button has not disappeared. It has acquired a negotiation.
The interesting part of the demonstration is what happens before anyone may use the proposed blocking power. The inspector does not merely need a worried expression. The inspector needs a sufficiently specific account of the approaching emergency. A safety feature has arrived with a forecasting assignment.
Please Demonstrate The Emergency
In Politico’s account, the draft’s threshold is a foreseeable, imminent risk of more than $1 billion in damage, or of killing or seriously injuring more than 50 people. The risk must involve a biological weapon, evasion of the developer’s controls or a cyberattack. These are reported conditions for blocking release or use, not a description of every form of AI oversight.
The figures are serious. The absurdity belongs to the sales arrangement: the company proposes the stopping conditions, while the government gets to work out whether the future will qualify. The vendor supplies a very large machine. The inspector supplies a very small crystal ball, calibrated to statutory specifications.
Here is a fictional sales demonstration, not an Anthropic quotation. Presenter: This will change everything. Inspector: Which thing, how soon, and what is the expected damage? Presenter: We appreciate your interest in our forecasting capabilities.
An extravagant promise about the future can remain pleasantly atmospheric in a product pitch. A prediction offered to justify stopping that product needs considerably better focus. The benefits get a panoramic lens. The brake gets a measuring instrument and a difficult assignment in probability.
None of that establishes that a risk threshold is inherently wrong, or that the proposal would make intervention impossible. It does make the choice of threshold worth more attention than the reassuring existence of a safety button. A conspicuous button photographs well. The conditions governing its use are less likely to make the brochure cover.
Congress can reject or change a company’s draft. The company does not write binding law merely by delivering an opening offer. But admire the opportunity embedded in this particular demonstration: before the government decides whether to stop an advanced model, it may need an exceptionally persuasive prediction about what the model will do.
Somewhere in this imaginary showroom, a second display is practically assembling itself. Not a new safeguard. A sales accessory for understanding whether the first safeguard can be used.
Introducing the entirely imaginary Catastrophe Forecast Pro: AI that predicts whether the government may stop the other AI.