Sakana AI's agentic reviewer catches 73% of core-claim errors in papers

Sakana AI says a new AI system built to check scientific papers for errors caught 73.43% of core-claim mistakes in testing, far more than any previous automated reviewer managed. The company describes the system, called Multi-Layered Review (MLR), in a paper published in TMLR, the journal Transactions on Machine Learning Research.

MLR uses three separate AI agents, each built on Anthropic’s Claude models, that read a paper and check its central claims against the evidence it presents. Sakana AI says splitting the work across multiple agents, rather than relying on a single model to judge a whole paper at once, helped the system catch mistakes that earlier tools missed.

To measure how well MLR performs, Sakana AI built what it calls the Contradiction Benchmark: a set of 1,164 errors inserted into papers, each one a point where a paper’s central claim does not match its own data or results. Testing systems against planted errors like these gives researchers a way to score how many real mistakes a reviewer would likely catch, since genuine errors in already-published papers are harder to count and confirm.

On that benchmark, MLR identified 73.43% of the core-claim errors. The next-best system the paper compares against caught 14.81% — about one in seven. Sakana AI frames the gap as evidence that organizing multiple AI agents to review different parts of a paper, rather than asking one model to judge the whole thing, makes a meaningful difference for this kind of task.

Peer review in academic publishing relies on human reviewers who already struggle to keep pace with how many papers get submitted each year. Sakana AI’s paper argues that an automated system catching most core-claim errors could serve as a useful check before or alongside human review, flagging the kind of mistakes a rushed reviewer might miss. The paper does not claim MLR replaces human reviewers, only that it shows practical value as a review tool.

Sakana AI’s paper doesn’t specify which Claude models power MLR’s three agents, how the system performs on paper types beyond the benchmark, or whether any journal or conference plans to use it. Watch for Sakana AI to release the Contradiction Benchmark dataset itself, which would let other labs test their own review systems against the same 1,164 planted errors and check whether MLR’s lead holds up outside the paper’s own testing.