Study finds AI can use wrong document despite retrieval

Seekr has published a study on how large language models use retrieved documents when answering questions. It found that models could retrieve the correct record but still answer from a different one.

The company tested Llama-3.3-70B and Qwen2.5-72B on 80 questions drawn from US Treasury financial statements. It used three conditions to measure how performance changed when similar records were introduced.

When each model received only the correct statement, Llama reached 95.0% accuracy and Qwen 92.5%. When the correct statement was shown alongside similar statements from other years, accuracy fell to 85.0% for Llama and 87.5% for Qwen, even though the correct source remained available.

In a broader retrieval setting that searched across all statements, accuracy on answered questions dropped to 37.1% for Llama and 30.4% for Qwen.

The results point to two problems in AI systems built on document collections. One is retrieval failure, where the wrong record is selected. The other occurs after retrieval, when a model has the right document but appears to rely on a different, plausible source when forming its answer.

Seekr said it identified that second pattern through attribution, which tracks which source shaped each answer. The study argues that a simple right-or-wrong accuracy score cannot show whether an error came from retrieval, reasoning, or use of the wrong source among similar documents.

Study design

The dataset focused on Treasury financial statements because they are public and can be checked objectively. The documents also resemble the near-identical records often found in business and government settings, such as contract revisions, quarterly filings and regulatory papers.

That matters because many organisations are applying AI systems to internal document sets where multiple versions of the same record sit side by side. In those settings, a system may return an answer that reads clearly and confidently while drawing on the wrong version of a document.

The study did not claim a universal error rate across all model types or business uses. It did report that the same pattern appeared across both model families tested.

A note in the study said most of the performance gap in the full retrieval pipeline came from retrieval selecting the wrong record. It also found that the 5 to 10 percentage point decline in the second test condition happened after retrieval had already succeeded.

Evaluation gap

The findings add to a wider debate over how companies should test AI systems that work with proprietary or regulated data. Many businesses have focused on answer accuracy, but the Seekr study suggests this measure alone can hide a specific class of failure when records are highly similar.

Those conditions are common in finance, legal work, compliance and the public sector, where annual statements, amended contracts and updated filings often differ only in small but important details.

Ben Faircloth, who wrote the paper and leads AI strategy and research at Seekr, was cited in the announcement as available to discuss the methodology behind the study. No direct quotation from him was included.

For companies using retrieval-augmented generation systems, the data suggests that confirming a model has access to the right file may not be enough. A separate check may be needed to confirm that the answer reflects the intended source rather than a similar record from a different period.

The reported figures were 95.0% accuracy for Llama and 92.5% for Qwen with only the correct statement, 85.0% and 87.5% respectively with similar statements added, and 37.1% and 30.4% in full retrieval across all statements on answered questions.