A researcher quits, a colleague backs him up on X, and within days half of Washington is talking about AI wiping out humanity. It's the kind of story that's easy to file under tech-industry theatre and move on.
“It doesn't matter whether we're talking about insurance or personal finance or the price of bananas, you get to AI very, very quickly,” Matthew Hill, the Chartered Insurance Institute’s CEO told Insurance Business.
“And, while one might be a little bit cynical about the stories in the press in the last few days about that have come out from Anthropic and the 10% chance that AI will extinguish life within the decade [that] could just be linked to their forthcoming IPO,” he continued.
However UK insurers probably shouldn't just move on – because underneath the apocalyptic headlines is a much narrower, much more practical question that the market has already started pricing: what happens to liability, governance and underwriting when AI systems start building the next generation of AI systems with less and less human oversight?
Jacob Coxon, who had worked on pretraining research at both OpenAI and Anthropic, resigned this week, saying the two labs were "racing straight to self-improving superintelligence and gambling with our lives." A serving Anthropic researcher then backed him up. Evan Hubinger, the company's head of alignment stress testing, posted on X that he personally rates the odds of AI killing all humans at ">10% within the next decade," and admitted Anthropic doesn't yet have a plan to keep a future superintelligent system aligned with human interests.
That combination, an insider quitting over safety, and another insider agreeing with him publicly, is what pushed the story from AI-safety circles into mainstream politics. Arizona senator Mark Kelly said "Washington needs to wake up" on AI risk; Texas senator Ted Cruz pointed to pending legislation on AI's "catastrophic risks"; and a Florida representative called for a special House session. President Trump, for his part, told reporters he wasn't worried about extinction scenarios and framed the priority as staying ahead of China.
It's a split reaction that will likely produce hearings rather than fast law in the US but the direction of travel, on both sides, is toward more scrutiny of how frontier models get built.
Strip away the "extinction" framing and the real issue is recursive self-improvement, or RSI - AI being used to help design, test and train the next generation of AI, with progressively less human involvement at each step. Both Anthropic and OpenAI say this is happening faster than they expected. Anthropic disclosed this year that more than 80% of the code merged into its own production systems is now written by its Claude models, with engineers shipping roughly eight times as much code per quarter as they were a few years ago.
“Our understanding and use of generative AI has evolved rapidly alongside the technology itself,” said Tom Hughes, Director of Underwriting at IUA. “We are already seeing AI systems produce outcomes that can be difficult to predict, explain or control, creating new challenges for organisations that rely on them.”
The company sketches three possible futures: progress plateaus (unlikely, in its view); AI keeps accelerating development while humans retain meaningful control (its base case); or AI reaches full recursive self-improvement, with humans left playing a "substantially diminished role" in how future models are built. OpenAI's chief scientist, Jakub Pachocki, echoed the concern in a company blog post, warning that "no-one is prepared for the consequences of a continued rapid rise in machine intelligence."
“Who knows?” Hill told Insurance Business. “But the fact is AI has moved on and will continue to move on and that can create a sense that you might describe, at best, as unsettling.”
None of that means artificial general intelligence is about to run amok next quarter. Vincent Conitzer, a computer science professor at Carnegie Mellon University, offered a more measured take: AI can already introduce genuinely new ideas into its own development process, which is exactly what makes it so hard to predict when, or whether, that process starts to accelerate sharply.
Unlike the US, where the response this week has been political noise rather than policy, UK financial regulators have already been here for a while, and have deliberately chosen not to write AI-specific rules. The Bank of England, the FCA and HM Treasury issued a joint statement in May warning that frontier AI models now have cyber capabilities that can exceed a skilled human attacker's, and setting out supervisory expectations for governance and incident response under existing rules rather than new ones.
The Bank's Financial Policy Committee has separately been monitoring AI-linked systemic risk to financial stability since its 2025 report on the subject, while the FCA has stuck to a principles-based, "wait and intervene on egregious failures" approach rather than mirroring the EU's AI Act.
That stance is already being tested in practice. Homeprotect, part of the Avantia Group, is currently the only insurer taking part in the FCA's AI Live Testing programme, working directly with the regulator on governance and live monitoring for AI-driven pricing and claims decisions. It's a useful reminder that "AI regulation" in UK insurance isn't a future hypothetical, it's already a live supervisory relationship for at least one MGA, with more likely to follow as AI moves deeper into underwriting.
The London market has been building its own guardrails too. The Lloyd's Market Association, working with Barnett Waddingham, has published an AI Adoption Toolkit to help managing agents build governance frameworks as AI moves from pilot projects into core underwriting, reserving and claims work. Sanjiv Sharma, the LMA's head of actuarial and exposure management, said the market's focus is now shifting toward how AI "is implemented and governed in practice," rather than whether to adopt it at all. On the composite side, the ABI's AI guide sets out five principles; accountability, transparency, fairness, safety, and contestability and redress that member firms are expected to apply.
“Like any emerging risk, we need to develop standards that AI applications will need to work within for insurance to cover any issues,” agreed Christoper Croft, Chief Executive of LIIBA. “It is no different from the first Industrial Revolution when the insurance industry worked with manufacturers to develop standards that meant new machinery was less susceptible to blowing up.”
All of that UK groundwork was built for AI as it exists today, tools that assist human decision-making, with a person somewhere in the loop. Recursive self-improvement is a different proposition: it's about the systems underneath the systems insurers are governing changing faster, and with less human involvement, than the frameworks were designed for. That's precisely the scenario that turns "silent AI" exposure, cover that neither explicitly includes nor excludes AI-related losses, into something much harder to price.
Some carriers, including CFC, have already added affirmative AI wording across technology E&O, professional liability and cyber lines to close that gap, while AIG, Great American and WR Berkley have gone the other way and sought regulatory approval to cap their AI-related liability altogether, reportedly over fears of multibillion-dollar claims.
There's also a US regulatory development worth watching for its knock-on effects. California's governor, Gavin Newsom, signed two bills this week creating the first US framework for independent, accredited third-party audits of AI systems. Separately, existing California rules on AI used in "consequential decisions" already name insurance specifically, alongside housing, healthcare and employment, as a sector facing extra scrutiny.
It's not UK law, but US frameworks tend to shape the compliance expectations of the AI vendors UK insurers ultimately rely on — and an accredited-auditor model is a plausible template if the FCA or EU regulators eventually move beyond principles-based guidance toward something more prescriptive for high-stakes AI use cases, insurance among them.
Swiss Re Corporate Solutions chief executive Ivan Gonzalez has argued that AI and elevated catastrophe losses are now the two structural forces setting the tone for commercial insurance, rather than temporary disruptions the market can simply wait out. This week's warnings, however overheated the "extinction" language might prove, reinforce that point. Whether or not recursive self-improvement ever reaches the stage its critics fear, the uncertainty around it is already a rated risk. and UK insurers that treat it as Silicon Valley drama rather than an underwriting and governance question may find the gap between their policy wordings and reality widening faster than they'd like.
