Study finds 1 in 10 AI pension answers risky
A study by PensionBee finds that more than one in ten AI-generated replies to typical pension questions could cause a saver to lose money or make an irreversible error.
Testing the reliability of chatbots
The analysis showed that although the majority of answers from Copilot, ChatGPT, Gemini and Claude were generally correct, 11%—equating to 57 replies—were classified as potentially harmful. The term refers to any answer that might lead a saver to lose funds or commit an irreversible mistake.
Out of a total of 539 evaluated replies, 89% achieved a minimum accuracy rating of two out of three. According to PensionBee, most of the flagged harmful answers were not outright wrong but left out essential details that could significantly change the guidance. For instance, an answer omitted the requirement that moving a defined benefit (DB) pension exceeding £30,000 must be accompanied by regulated financial advice.
The study examined four AI chatbots by posing 45 queries that spanned nine pension subjects. Testers employed free-level accounts and adhered to a prescribed protocol intended to reduce bias, with every reply evaluated independently by a pair of pension specialists for correctness and possible danger. Each bot received the identical set of 45 questions on three separate occasions, producing a total of 539 answers.
Although 72% of the replies earned top scores, the investigation revealed that any single chatbot had merely a 48% likelihood of delivering a perfect answer to the identical question across all three tries. PensionBee noted that this shows the inconsistency of AI-driven outputs, pointing out that users typically see only one response instead of an aggregated average.
Read Also: Regulator warns schemes on charge focus
Results also differed across subject areas. Queries about pension contributions and withdrawing pension funds reached accuracy levels above 96% with potential-harm incidences at or below 5%. Conversely, questions tied to major life changes or ambiguous user locations showed reduced accuracy and raised risk of harmful advice.
Hidden risks for vulnerable users
These conclusions come after the Financial Conduct Authority’s (FCA) latest Mills Review, which reported that merely 9% of adults obtained pension or investment advice, whereas 29% of individuals who interacted with their pension during the previous year turned to AI.
The review indicated that inquiries about withdrawing pension funds and pension fraud produced no factually inaccurate replies. However, they triggered 12 potential-harm alerts, with dangerous answers outnumbering incorrect ones by roughly three to one in cases where savers faced financial hardship.
Becky O’Connor, head of pensions at PensionBee, said: “Using AI chatbots for pension advice can be a bit like playing Russian roulette with your retirement planning. “While for the most part it gets things technically right; the confident, helpful tone of answers occasionally masks some worrying omissions, it may fail to detect vulnerability, or just straight up get things wrong. “AI can help bridge the advice gap by giving people useful pension information when they might otherwise struggle.
It can make complicated subjects more accessible and help people get started. “But these results also show why consumers need to understand the limits of what an AI chatbot can safely tell them, because according to our research, one in ten times, it could turn out to be a false friend.”
