Artificial intelligence dominated discussion at this year's BIBA conference in Manchester, but beneath the market enthusiasm sat a more uncomfortable concern: insurers and brokers may be deploying AI faster than they can properly govern it.
The concerns are not theoretical. From accountability gaps under consumer duty to the rise of unsanctioned AI use among staff, practitioners across the market identified structural vulnerabilities that deserve serious attention from boards and senior leadership alike.
For Tim Johnson, partner and head of insurance at Browne Jacobson, the fundamental challenge is one of accountability. As AI assumes a greater role in underwriting decisions and customer interactions, regulated firms face a compliance problem that many have not yet fully confronted.
"The main one for me is accountability and explainability," Johnson said. "Customers are entitled to understand how decisions are being made. There's quite a lot of AI software where even software developers can't explain how the AI has come to its decision. And I think that's really problematic."
Under the FCA’s consumer duty, firms must demonstrate that products and services deliver good customer outcomes. Where AI-driven decisions cannot be properly explained, or replicated when systems fail, that obligation becomes harder to meet.
Johnson also identified operational resilience as a related and underappreciated risk. "If the AI system goes down or fails for whatever reason, does the business model enable the insurer using AI to continue to function in the same way? With AI, that's not always the case."
Johnson said many firms were still treating AI adoption as a procurement exercise rather than a strategic transformation. "The biggest error is treating it as a software procurement exercise and not a business transformation exercise," he said. "Once you start using AI to any degree, it's a board level issue."
A further risk specific to brokers and underwriting agents is professional indemnity exposure. Where an AI system produces an error that causes a firm to breach its underwriting authority or deliver incorrect advice, the legal liability falls on the firm, not the vendor. It is, Johnson noted, "a straight negligence claim", and one that is frequently overlooked in the enthusiasm to deploy.
Concerns around governance were not limited to underwriting or customer-facing systems. Several speakers also pointed to growing operational risks created by uncontrolled AI usage inside organisations themselves.
Perhaps the clearest example of governance lag emerged around so-called “shadow AI”, employees using consumer AI tools outside approved corporate systems.
Johnson described this as both a regulatory risk and a practical inevitability.
"If you don't give your staff an AI solution, they will open up their phones and they will use their own AI solution which carries a whole load of additional risks because it will be untested, it won't be a closed loop system," he said. "You can tell staff not to do that, but they will."
The result is that firms moving cautiously on AI may still carry greater unmanaged risk than those investing in controlled, governed systems.
For the market's larger participants, AI is already delivering measurable commercial advantages. At Aviva, new business enquiries are now ingested, enriched and indicatively priced within minutes, transforming both the quality and pace of initial broker conversations.
Michael Yabantu, who leads the insurer's mid-market segment, described the change as substantive rather than incremental.
"When a new business inquiry gets sent into us, we can now ingest it into our platforms, we can enrich it, we can indicatively price it, we can send it back to the underwriter within minutes," he said. "All of a sudden we start to transform the speed and quality of our service and that first conversation with the broker which focuses on how we evolve and enhance the proposition to make sure their outcome is better protected.
Crawford, meanwhile, is addressing the talent and capability dimension through a structured AI literacy programme, developing training at three levels to ensure that employees across the organisation can engage with AI tools confidently and responsibly.
Paul Lofkin, president of Crawford UK and Ireland, was direct about the limits of full automation.
"The science will tell you that fully autonomous claims handling is not currently possible, nor is it the right thing to do," he said. "Insurance is a trust business. It's a people business. And that human in the loop, where we're verifying AI because it's not always right, is really critical."
Donna Richards, chief executive officer of Carpenters Group, said the industry is still only beginning to understand the long-term implications and opportunities of AI adoption and warned against assuming technology alone would solve all operational challenges.
"You can't just assume AI will solve the problem or that everything is going to be autopilot, people must still play a vital role" she said.
BIBA 2026 reflected an industry that has broadly accepted AI as a commercial necessity, but is still working through the legal, regulatory and operational implications of that shift. The firms best positioned are those treating AI adoption not as a technology project, but as a governance challenge. The distinction matters because the gap between the two is where much of the industry's emerging risk now sits.
