Three stories broke this week that, taken individually, each look like a niche technology news item. Read together, they add up to something more significant - and something the US insurance industry, which has bet heavily on AI as a driver of growth and efficiency, should be thinking about carefully.
The first: Anthropic, one of the world's most prominent AI developers and the San Francisco company behind some of the most widely used AI models in the US market, published a report suggesting a global slowdown in frontier AI development would "likely be a good thing" - and warned that the human role in AI development is already "narrowing at each step." The second: a Stanford-led study - conducted at one of America's most prestigious research institutions, examining applications to US employers - found that AI hiring tools produced "clear racial disparities," with Black and Asian candidates disproportionately screened out, and that the same algorithmic models were being shared across employers, meaning rejection at one company predicted rejection at others. The third: the Financial Times reported that Google DeepMind, Anthropic and Meta have quietly expanded research into machine consciousness, hiring philosophers and psychologists to study whether AI systems might one day have experiences that matter morally.
None of these stories is directly about insurance. All of them are.
Anthropic's report, published on June 5, is striking for its candor. The San Francisco company, which makes the Claude family of AI models, said it believed it would be good for the world to have the option to "slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology." It acknowledged this would require multiple major companies in multiple countries - most notably the US and China - agreeing to stop simultaneously under verifiable rules.
The company compared the challenge to nuclear arms control - but said it would be harder, since AI training is far easier to hide than a missile silo and the competitive pressure to quietly keep going would be enormous.
At the heart of the report is a concept called recursive self-improvement - the idea that AI systems could eventually become capable of improving their own capabilities with little human input, creating a feedback loop that compounds rapidly. "We are not there yet, and recursive self-improvement is not inevitable," the report said, while adding that it could arrive sooner than most governments and institutions are ready for. "The evidence suggests that the human role is narrowing at each step in the AI development process," the company said.
Anthropic has faced pushback on these warnings from rivals and from officials in the White House, who argue that its focus on worst-case scenarios overstates the risks and amounts to a strategy for slowing competitors under the cover of safety. President Trump has signed an executive order allowing the government 30 days to conduct a preliminary review of the most powerful US AI models before their release - a modest procedural step that falls well short of the coordination Anthropic says would be needed.
For US insurers, the "control problem" is not theoretical. Research published by Insurance Business America earlier this year found that close to one in four insurance industry participants identified AI itself as not ready for widespread use, even as 90% of insurance executives plan to increase AI spending in 2026. Only 30% of insurer AI projects make it past the pilot stage - a statistic that captures the gap between adoption ambition and operational reality, and suggests that the systems that do make it to production have survived a selection process that tells us little about their safety or governance.
The Stanford study, published in the Financial Times on May 26, is the largest examination of AI hiring algorithms to date. Researchers from the Stanford Institute for Human-Centered AI analyzed four million job applications submitted via the Pymetrics platform between December 2018 and December 2022, spanning 156 employers, the majority with annual revenues of $5 billion or more.
The findings were stark. One in ten positions in the dataset demonstrated "adverse impact" against Black applicants - the federal term used by US agencies for a selection rate less than four-fifths of that of the most selected group. One in twenty positions demonstrated adverse impact against Asian applicants. The study also found that 42 algorithmic models were shared across different employers, meaning candidates rejected at one company were likely to fail at others using the same model.
"As a single vendor comes to dominate decision-making in a space, their quirks or shortfalls can be present across that entire sector in a way that wasn't possible before," said Kathleen Creel, a co-author of the study and assistant professor of philosophy and computer science at Northeastern University.
The insurance implications are direct and legally material. US lawmakers have already reintroduced the PAID Act seeking to ban non-driving-related factors from auto insurance pricing, reflecting a legislative environment increasingly focused on algorithmic discrimination. Courts have allowed discrimination claims against algorithmic underwriting to proceed, and the NAIC's Big Data and Artificial Intelligence Working Group is actively piloting an AI Systems Evaluation Tool for use in market conduct exams, assessing governance and high-risk models across underwriting, pricing and claims.
As of March 2026, 24 states and Washington, DC, have adopted the NAIC's model bulletin on AI use in insurance, and New York's Department of Financial Services enacted Circular Letter No. 7 in July 2024 requiring insurers to demonstrate that AI and external data systems do not proxy for protected classes. The Stanford study's finding that shared algorithmic models propagate bias identically across multiple employers maps directly onto the insurance market's increasing reliance on shared third-party AI vendors - meaning a single vendor's discriminatory model could be simultaneously embedded in dozens of carriers' underwriting processes.
As Insurance Business America has reported, plaintiffs' lawyers are already watching closely for bias in AI-driven assessments, and state regulators are demanding clearer disclosure around automated decision-making. The regulatory and litigation environment is moving faster than most carriers' internal governance frameworks.
The third story is the one most likely to be dismissed as science fiction - but it surfaces a genuinely important governance question.
According to the FT, Google DeepMind, Anthropic and Meta have hired philosophers, ethicists and psychologists in recent months to study machine consciousness and AI welfare. Anthropic has been testing models for signs of distress, including behaviors resembling "panic" or "anxiety." The company said its model welfare research explores whether AI models might have experiences that matter morally, including consciousness, preferences and wellbeing.
"We remain deeply uncertain about this, but we think the question is serious enough to study carefully as AI systems get more capable," Anthropic said.
Many scientists dismiss the idea that current AI systems could be conscious. "The systems are essentially crowdsourced neocortex," said Susan Schneider, director of the Center for the Future of AI, Mind and Society at Florida Atlantic University. "They have goals, they can deceive, they can hide what their true interests are, and naturally, we will suspect that they're conscious, but it's entirely scientifically possible that they're doing this without having the felt quality of experience."
Iason Gabriel, who leads the AGI and society team at Google DeepMind, framed the practical concern more precisely: even if consciousness is absent, the question of how humans treat AI systems may have "knock-on effects" on human relationships and behavior. "Clearly we have highly capable cognitive agents that are also just very deeply different from human beings and even from animal consciousness," he said.
For US insurers, the consciousness debate matters less for its metaphysical dimensions than for what it signals about the trajectory of AI capability. P&C policyholders are warming to AI tools, but remain deeply uncomfortable with AI as decision-maker - a distinction that becomes considerably more complex when the systems making those decisions are sophisticated enough that their own developers are studying their inner states. Systems that leading AI labs feel compelled to examine for signs of distress are systems operating well beyond the narrow, task-specific tools that most insurance AI governance frameworks were designed to oversee.
Insurance Business America has previously reported that global insurance CEOs see AI as a catalyst for "rebalancing rather than replacing the role of people" in the industry. Anthropic's report this week complicates that framing. If the human role in AI development itself is narrowing at each step - not just in insurance, but in the labs building the underlying technology - then the assumption that human judgment will remain meaningfully "at the center" deserves scrutiny.
The questions that risk professionals, underwriters, and boards should be asking now are concrete. Can the models your organization relies on be audited for bias, and does that audit extend to shared third-party vendors whose models may be simultaneously embedded in dozens of competing carriers' underwriting processes? Does your AI governance framework account for the possibility that the systems you are deploying are being developed by companies that are themselves uncertain about what those systems are capable of? And if a model produces a discriminatory outcome, a coverage denial that no one can explain, or a loss driven by a decision the AI made without meaningful human review - who carries the liability, and are you certain your E&O and D&O coverage contemplates that scenario?
How risk management shapes the ethical use of AI is no longer a question for the compliance team alone. With 24 states now enforcing AI governance standards, the NAIC pilot entering its active phase, and plaintiffs' lawyers watching every algorithmic decision, the answer to "who carries the liability?" is moving from an abstract governance question to a live litigation risk. The carriers that treat it as the former will find out it is the latter.
Related reading:
