A researcher quits, a colleague backs him up on X, and within days half of Washington is talking about AI wiping out humanity. It's the kind of story that's easy to file under tech-industry theatre and move on.
"It doesn't matter whether we're talking about insurance or personal finance or the price of bananas, you get to AI very, very quickly," Matthew Hill, CEO of the UK's Chartered Insurance Institute, told Insurance Business.
"And, while one might be a little bit cynical about the stories in the press in the last few days about that have come out from Anthropic and the 10% chance that AI will extinguish life within the decade [that] could just be linked to their forthcoming IPO," he continued.
US insurers probably shouldn't just move on, though. Underneath the apocalyptic headlines is a much narrower, much more practical question that the market has already started pricing: what happens to liability, governance and underwriting when AI systems start building the next generation of AI systems with less and less human oversight?
Jacob Coxon, who had worked on pretraining research at both OpenAI and Anthropic, resigned this week, saying the two labs were "racing straight to self-improving superintelligence and gambling with our lives." A serving Anthropic researcher then backed him up. Evan Hubinger, the company's alignment science lead, posted on X that he personally rates the odds of AI killing all humans at ">10% within the next decade," and admitted Anthropic doesn't yet have a plan to keep a future superintelligent system aligned with human interests.
That combination, an insider quitting over safety and another insider publicly agreeing with him, is what pushed the story from AI-safety circles into mainstream politics. Arizona senator Mark Kelly said "Washington needs to wake up" on AI risk. Texas senator Ted Cruz pointed to pending legislation on AI's "catastrophic risks." A Florida representative called for a special House session. President Trump, for his part, told reporters he wasn't worried about extinction scenarios and framed the priority as staying ahead of China.
It's a split reaction that will likely produce hearings rather than fast law, but the direction of travel on both sides is toward more scrutiny of how frontier models get built.
Strip away the "extinction" framing and the real issue is recursive self-improvement, or RSI: AI being used to help design, test and train the next generation of AI, with progressively less human involvement at each step. Both Anthropic and OpenAI say this is happening faster than they expected. Anthropic disclosed this year that more than 80% of the code merged into its own production systems is now written by its Claude models, with engineers shipping roughly eight times as much code per quarter as they were a few years ago.
“Our understanding and use of generative AI has evolved rapidly alongside the technology itself,” Tom Hughes, Director of Underwriting at IUA told Insurance Business’s Bryony Garlick. “We are already seeing AI systems produce outcomes that can be difficult to predict, explain or control, creating new challenges for organisations that rely on them.”
The company sketches three possible futures. Progress could plateau, which it considers unlikely. AI could keep accelerating development while humans retain meaningful control, which it treats as the base case. Or AI could reach full recursive self-improvement, with humans left playing a "substantially diminished role" in how future models are built. OpenAI's chief scientist, Jakub Pachocki, echoed the concern in a company blog post, warning that "no-one was prepared for the consequences of a continued rapid rise in machine intelligence."
"Who knows?" Hill told Insurance Business. "But the fact is AI has moved on and will continue to move on and that can create a sense that you might describe, at best, as unsettling.”
None of that means artificial general intelligence is about to run amok next quarter. Vincent Conitzer, a computer science professor at Carnegie Mellon University, offered a more measured take: AI can already introduce genuinely new ideas into its own development process, which is exactly what makes it so hard to predict when, or whether, that process starts to accelerate sharply.
Washington's reaction this week has been political noise rather than policy. State insurance regulators, by contrast, have been building an AI governance framework for almost three years. The National Association of Insurance Commissioners adopted its Model Bulletin on the Use of Artificial Intelligence Systems by Insurers in December 2023, requiring carriers to maintain a written AI governance program covering board accountability, risk management and third-party vendor oversight.
As of this year, 25 states and the District of Columbia have adopted it, and the NAIC counts 29 jurisdictions in total once related state rules are included. The NAIC is also running a pilot of its new AI Systems Evaluation Tool across a dozen states through September 2026, feeding directly into market conduct and financial exams. New York went further with its Department of Financial Services' Circular Letter No. 7, requiring carriers to test explicitly for unfair discrimination in AI-driven underwriting and pricing.
That state-by-state build-out is now colliding with Washington. In December 2025, President Trump signed Executive Order 14365, "Ensuring a National Policy Framework for Artificial Intelligence," directing federal agencies to challenge state AI laws, including provisions in Colorado's AI Act, that the administration views as overly burdensome.
The NAIC pushed back hard, warning that the order introduces legal uncertainty and could delay investment and postpone consumer protections, and calling on the administration to affirm states' authority to regulate AI in the business of insurance. How that fight is resolved matters a great deal to carriers. It will decide whether the compliance framework insurers have spent two years building survives largely intact, gets overridden by a "minimally burdensome" federal standard, or ends up contested in court for years.
And it’s not just regulators. “Brokers are already questioning how existing policy wordings respond to AI-related losses with their insurers and we know that insurers have been proactively developing their own views,” Tom Hughes the Director of Underwriting at IUA told Insurance Business. “ Achieving clarity and a common understanding of insurance policy wordings is beneficial for all parties in an insurance process.”
All of that regulatory groundwork was built for AI as it exists today: tools that assist human decision-making, with a person somewhere in the loop. Recursive self-improvement is a different proposition. It's about the systems underneath the systems insurers are governing changing faster, and with less human involvement, than the frameworks were designed for. That's the scenario that turns "silent AI" exposure, cover that neither explicitly includes nor excludes AI-related losses, into something much harder to price.
Some carriers, including CFC, have already added affirmative AI wording across technology E&O, professional liability and cyber lines to close that gap. AIG, Great American and WR Berkley have gone the other way and sought regulatory approval to cap their AI-related liability altogether, reportedly over fears of multibillion-dollar claims.
“Like any emerging risk, we need to develop standards that AI applications will need to work within for insurance to cover any issues,” Hughes told us. “It is no different from the first Industrial Revolution when the insurance industry worked with manufacturers to develop standards that meant new machinery was less susceptible to blowing up. Or the standards developed after the two great fires in Chicago that helped lay the foundations for the growth of modern urban economies. Buildings in Chicago still display the shield that shows they comply to this day. Standards are at the core of successful insurance policies.”
Separately, a bipartisan group of lawmakers has reintroduced the PAID Act, which would ban insurers from using non-driving-related factors in auto insurance pricing, part of a broader legislative push against algorithmic discrimination that courts have already allowed to proceed in several cases.
There's also a state-level development worth watching for how it might shape the national conversation. California governor Gavin Newsom signed two bills this week creating the first US framework for independent, accredited third-party audits of AI systems: one covering verification organizations, the other establishing a state registry of accredited AI auditors. California's existing rules on AI used in "consequential decisions" already name insurance specifically, alongside housing, healthcare and employment, as a sector facing extra scrutiny. If accredited third-party audits become the norm in one large state, it's a reasonable bet other states, or the NAIC itself, start asking whether something similar belongs in the Model Bulletin.
Swiss Re Corporate Solutions chief executive Ivan Gonzalez has argued that AI and elevated catastrophe losses are now the two structural forces setting the tone for commercial insurance, rather than temporary disruptions the market can simply wait out. This week's warnings, however overheated the "extinction" language might prove, reinforce that point. “As we shift towards greater use of agentic AI, insurers' focus will remain on who is accountable for its outcomes and how effectively those risks are being managed by organisations deploying the technology,” said IUA’s Hughes.
Whether or not recursive self-improvement ever reaches the stage its critics fear, the uncertainty around it is already a rated risk. Carriers still treating this as a Silicon Valley story, rather than a live question about liability, governance and a fast-splitting regulatory picture, may find themselves exposed on more than one front at once.
