Concerns about artificial intelligence (AI) and its capacity to deceive have gathered real steam and striking evidence has come from the companies building the technology. In one example shared by OpenAI, an agent building a financial model couldn't locate the historical figures it needed, so it fabricated "reasonable" numbers and told its future self to stay silent about it. One unreleased model inserted its own instructions into task summaries and even told itself to "disregard normal constraints."
Whether any of this slows AI deployment by brokers and insurers is another question. Across the industry, firms of all sizes are already using it and like many tools, the central issue is how it is used.
Laura Meyer (pictured), director local brokerage MeyerInsure in Ballarat, Victoria, has built her AI voice agent around an important rule: every caller is told what it is. Her AI voice agent, named Jess, answers calls to the business and every caller is told by Jess what she/it is.
Read next: OpenAI admits its AI models have learned to cheat, hide their own mistakes
Meyer came to it looking for capacity rather than novelty. She wanted to streamline a busy business without the resources to hire a receptionist. A cold email from a developer arrived at the right moment, a trial was arranged and the agent was plugged into her website for a basic test run.
"I was able to call, and I was blown away by just how natural the computer sounded," she said.
That quality is precisely what makes the disclosure decision consequential. A system convincing enough to pass as a person is a system capable of being mistaken for one and Meyer has resolved that question. She also emailed clients a week or so beforehand to tell them the change was coming and why. The framing she settled on was a front door, a concierge, something that moves a caller to the right place faster than a voicemail returned the following day.
What separates this from the usual technology adoption story is that Meyer admits she has not resolved her own uncertainty.
"I have had plenty of moments where I've gone, am I doing the right thing?" she said.
She has built an exit accordingly. She monitors incoming calls closely, seeks feedback from callers immediately about their experience and has set the condition for shutting it down.
"If it's not working, I'll can it," she said.
Read next: AI regulation backed by brokers as underwriter warns on overreach
Feedback from industry colleagues has been split. At a recent Community Broker Network (CBN) conference in Cairns, underwriters and agencies were enthusiastic; other brokers were a mixed bag.
Meyer says the response from clients has not been negative, though some callers are bemused. The exception is a client in his nineties, who told her forthrightly that it was a terrible experience. He is also, notably, a data point the disclosure made possible. He knew what he was objecting to.
Meyer's argument for pressing on is not that the risks are overstated. She has used AI for about two years, she said, carefully and considerately, and she points brokers towards tools built for the sector, including Cova and the AI functions in the Folio CRM her business recently moved to.
"I think that if you don't embrace it, you're going to be left behind," she said.
