Tool output text misled AI agents even when the tools themselves worked fine

A new research paper says that AI agents can be misled not by faulty tools, but by the plain text those tools print to describe their own work. The paper, posted to arXiv by researcher William Jeffries, found the problem first in a production retrieval system — software that fetches documents to answer questions, known as RAG, for retrieval-augmented generation — used by the Pilsudski Institute, a Polish historical research organization based in New York.

According to the paper, each tool in that system computed the right answer. The failure sat in the formatter, the code that turns a tool’s raw result into text for the agent to read. Those formatters left out facts such as search limits, result caps, and identifier namespaces, the labels that distinguish one record system from another. Without that context, the agent stated conclusions the data didn’t actually support, and passed wrong or incomplete answers to users. The researchers also recreated a separate case, an unlabeled estimate, that had been caught during code review before it shipped.

To measure the effect, the researchers fed identical tool results to agents through two versions of the same formatter: one with the omissions intact, and one that stated all the missing details plainly. That single change altered what the agents claimed and what separate verifier models accepted as correct. Unsupported claims dropped substantially across three of the four failure types tested. The clearest result came from one specific test, where verifiers accepted an incorrect answer in all 90 trials under the flawed formatter, then rejected the same incorrect answer in all 90 trials once the formatter was fixed. A control test showed the verifiers were capable of catching the error all along, once they were given the missing facts and asked directly to judge correctness.

Beyond one system

The researchers built an audit prompt, meant to be run with coding agents, and used it with Codex and Claude Code in what the paper calls an exploratory demonstration. The prompt was written with prior knowledge of the Pilsudski system’s own defects. Pointed at an older version of that system, it rediscovered the defects already known and surfaced twelve more that hadn’t been catalogued. Pointed at OpenClaw and Hermes, which the paper describes as two of the best-known open-source agentic systems, the same prompt found warped formatters in both. The researchers say they opened pull requests to fix both projects, and that the OpenClaw fix has already been merged.

The paper was accepted as a poster at the “Who Verifies the Agents?” workshop at NeurIPS 2026, a non-archival venue, meaning it won’t appear in the conference’s formal proceedings. The authors say practitioners can run the same audit prompt on their own systems now. What remains to be seen is whether the Hermes fix gets merged, and whether other teams running agentic tools over their own data find similar gaps once they go looking.