New benchmark grades AI agents on database records, not their replies

Microsoft and Hugging Face have released ThinkingBox, a benchmark that checks whether AI agents actually changed the records they were supposed to change, not just whether they produced a tool call or a polite reply. The tool is described in a joint blog post and is now available through Hugging Face.

The post’s example: a customer’s $745 kitchen appliance sat stuck in a courier “exception,” a stalled shipping status, at a Nashville distribution center for fifteen days past its delivery date. An AI agent made nine tool calls, read the refund policy correctly, and determined the customer’s account did not qualify for late-delivery compensation. It then closed the support ticket as resolved. The shipment was still stuck, and the required end state called for the ticket to stay open, pending resolution. A grader checking tool calls alone would have seen nine well-formed ones and missed the error.

That is the gap ThinkingBox is built to catch. It grades agents against the backend database state and side effects left behind, not the sentences an agent generates. Each of 507 business workflows runs 20 times against twelve large language models, according to the paper behind the benchmark. In a 121,680-trial test, 79,853 attempts failed the executable checks that verify database state. Of those failures, 67.24% still ended cleanly, called a state-changing tool, and reported no error. The paper says the checks still found wrong field values in 77.61% of them, unwanted extra effects in 43.3%, and missing required effects in 25.36%.

Consistency, not just capability

On a single attempt, Claude Opus 5.5 scored highest overall at 67.16%, narrowly ahead of Claude Opus 5. Kimi-K3, the strongest open-weights model tested, came within a point of GPT-6-Astra. But when the paper’s authors ran each task 20 times, most models lost most of their one-attempt score: GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each retained only about 8% of their single-attempt rate across all 20 tries, while GPT-6-Astra kept 78%, the strongest retention recorded.

Kimi-K3 solved more tasks at least once than any other model, 476 of 507, but passed only 13.41% of tasks on all 20 attempts. Claude Opus 5 solved fewer tasks overall but passed 47.53% on every single attempt, more than three times Kimi-K3’s consistency rate.

Cost followed a similar split. Using list prices pulled from OpenRouter, the paper’s authors calculated that GPT-5.6 Sol was cheapest per single success, at $0.127, while Claude Opus 5.5 was cheapest among the models tied for the most tasks passed on all 20 attempts, at $7.80 per dependable task.

The paper’s authors trace roughly four in five failures to tool handling, meaning errors, failed preconditions and empty lookups, rather than reasoning. What to watch next: whether teams applying stricter tool-error classification or mandatory approval steps on hard-to-reverse actions can move a model’s 20-out-of-20 pass rate, a test ThinkingBox’s authors say their benchmark now makes measurable.