A researcher quits, a colleague backs him up on X, and within days half of Washington is talking about AI wiping out humanity. It's the kind of story that's easy to file under tech-industry theatre and move on.
"It doesn't matter whether we're talking about insurance or personal finance or the price of bananas, you get to AI very, very quickly," Matthew Hill, CEO of the UK's Chartered Insurance Institute, told Insurance Business earlier today.
"And, while one might be a little bit cynical about the stories in the press in the last few days about that have come out from Anthropic and the 10% chance that AI will extinguish life within the decade [that] could just be linked to their forthcoming IPO," he continued.
Canadian insurers probably shouldn't just move on, though. Underneath the apocalyptic headlines is a much narrower, much more practical question that the market has already started pricing: what happens to liability, governance and underwriting when AI systems start building the next generation of AI systems with less and less human oversight?
Jacob Coxon, who had worked on pretraining research at both OpenAI and Anthropic, resigned this week, saying the two labs were "racing straight to self-improving superintelligence and gambling with our lives." A serving Anthropic researcher then backed him up. Evan Hubinger, the company's alignment science lead, posted on X that he personally rates the odds of AI killing all humans at ">10% within the next decade," and admitted Anthropic doesn't yet have a plan to keep a future superintelligent system aligned with human interests.
That combination, an insider quitting over safety and another insider publicly agreeing with him, is what pushed the story from AI-safety circles into mainstream US politics. Arizona senator Mark Kelly said "Washington needs to wake up" on AI risk. Texas senator Ted Cruz pointed to pending legislation on AI's "catastrophic risks." A Florida representative called for a special House session. President Trump, for his part, told reporters he wasn't worried about extinction scenarios and framed the priority as staying ahead of China.
It's a split reaction south of the border that will likely produce hearings rather than fast law, but the direction of travel on both sides is toward more scrutiny of how frontier models get built. Canadian insurers don't have the luxury of treating that as someone else's fight, either, because the regulator that actually oversees them has already sounded a similar alarm.
Strip away the "extinction" framing and the real issue is recursive self-improvement, or RSI: AI being used to help design, test and train the next generation of AI, with progressively less human involvement at each step. Both Anthropic and OpenAI say this is happening faster than they expected. Anthropic disclosed this year that more than 80% of the code merged into its own production systems is now written by its Claude models, with engineers shipping roughly eight times as much code per quarter as they were a few years ago.
“Our understanding and use of generative AI has evolved rapidly alongside the technology itself,” Tom Hughes, Director of Underwriting at IUA told Insurance Business’s Bryony Garlick. “We are already seeing AI systems produce outcomes that can be difficult to predict, explain or control, creating new challenges for organisations that rely on them.”
The company sketches three possible futures. Progress could plateau, which it considers unlikely. AI could keep accelerating development while humans retain meaningful control, which it treats as the base case. Or AI could reach full recursive self-improvement, with humans left playing a "substantially diminished role" in how future models are built. OpenAI's chief scientist, Jakub Pachocki, echoed the concern in a company blog post, warning that "no-one was prepared for the consequences of a continued rapid rise in machine intelligence."
"Who knows?" Hill told Insurance Business. "But the fact is AI has moved on and will continue to move on and that can create a sense that you might describe, at best, as unsettling.”
None of that means artificial general intelligence is about to run amok next quarter. Vincent Conitzer, a computer science professor at Carnegie Mellon University, offered a more measured take: AI can already introduce genuinely new ideas into its own development process, which is exactly what makes it so hard to predict when, or whether, that process starts to accelerate sharply.
Canadian insurers have less reason than most to treat this as a foreign story. Their own primary regulator has, in effect, already flagged the exact concern driving this week's headlines, and named Anthropic specifically while doing it. In a private email sent on April 29, 2026 to chief technology, information-security and risk officers across federally regulated financial institutions, the Office of the Superintendent of Financial Institutions warned that advanced AI models "such as Anthropic Claude Mythos" could "significantly compress the timeframe for effective risk mitigation." OSFI flagged frontier models' growing ability to find software vulnerabilities as a factor that could sharply shorten the gap between a flaw being discovered and it being exploited.
OSFI followed that private warning with a public bulletin in July 2026 on generative and agentic AI, setting out sound practices for institutions, insurers among them, to manage AI's effect on their operations. It builds on OSFI's existing guidelines covering technology and cyber risk, operational resilience, and third-party risk. The regulator's central concern is governance: AI adoption can move faster than the frameworks meant to oversee it, and agentic systems are increasingly able to act with little human supervision while leaning on third-party models and data they didn't build. OSFI has also finalized Guideline E-23 on model risk management, effective May 1, 2027, which for the first time extends formal model governance requirements, inventories, independent review, documented performance standards, to AI and machine learning systems across every federally regulated insurer, covering everything from actuarial pricing to claims prediction and catastrophe modelling.
That's a real shift in tone. Less than a year ago, OSFI superintendent Peter Routledge was telling industry the regulator had no plans to rush into AI-specific rules. He argued boards and executives, not OSFI, should work out how to integrate the technology responsibly, and that acting too early risked doing "more harm than good." The gap between that stance and April's warning about compressing risk-mitigation timeframes shows how quickly frontier AI has moved up Canadian regulators' list of concerns.
OSFI isn't working alone on this. It's one of several partners, alongside the Bank of Canada, the Department of Finance, the Financial Consumer Agency of Canada and FINTRAC, in a Global Risk Institute forum that brings senior financial executives together with regulators and academics specifically to examine fast-moving, systemic AI risk across the sector.
And it’s not just regulators. “Brokers are already questioning how existing policy wordings respond to AI-related losses with their insurers and we know that insurers have been proactively developing their own views,” Tom Hughes the Director of Underwriting at IUA told Insurance Business. “ Achieving clarity and a common understanding of insurance policy wordings is beneficial for all parties in an insurance process.”
All of that Canadian regulatory groundwork was built for AI as it exists today: tools that assist human decision-making, with a person somewhere in the loop. Recursive self-improvement is a different proposition. It's about the systems underneath the systems insurers are governing changing faster, and with less human involvement, than the frameworks were designed for. That's the same dynamic OSFI's own bulletin gestures toward when it warns about agentic systems operating with little human supervision, and it's what turns "silent AI" exposure, cover that neither explicitly includes nor excludes AI-related losses, into something much harder to price.
Globally, some carriers, including CFC, have already added affirmative AI wording across technology E&O, professional liability and cyber lines to close that gap. AIG, Great American and WR Berkley have gone the other way in their home US market and sought regulatory approval to cap their AI-related liability altogether, reportedly over fears of multibillion-dollar claims. Canadian insurers operating internationally, or relying on the same reinsurance and MGA relationships as their US and UK counterparts, are unlikely to be insulated from that repricing for long.
“Like any emerging risk, we need to develop standards that AI applications will need to work within for insurance to cover any issues,” Hughes told us. “It is no different from the first Industrial Revolution when the insurance industry worked with manufacturers to develop standards that meant new machinery was less susceptible to blowing up.”
Swiss Re Corporate Solutions chief executive Ivan Gonzalez has argued that AI and elevated catastrophe losses are now the two structural forces setting the tone for commercial insurance, rather than temporary disruptions the market can simply wait out.
“As we shift towards greater use of agentic AI, insurers' focus will remain on who is accountable for its outcomes and how effectively those risks are being managed by organisations deploying the technology,” said IUA’s Hughes.
This week's warnings, however overheated the "extinction" language might prove, reinforce that point, and Canadian insurers have less excuse than most to be caught off guard. Whether or not recursive self-improvement ever reaches the stage its critics fear, their own regulator has already told them, in writing, that the clock for managing frontier AI risk is running faster than it used to.
