SpireBench tests AI agents on a full game of Slay the Spire

Researchers have built a new benchmark that tests AI agents on their ability to play a long, complete video game rather than a short, scripted task. Called SpireBench, it has 20 large language models play through the commercial deck-building game Slay the Spire, a card game where choices made early in a run shape whether a player survives dozens of fights later on.

The paper, written by Scott C. Lowe and one co-author and posted to arXiv, argues that most benchmarks for agentic AI, meaning AI systems that take multi-step actions toward a goal, are short and episodic. That makes it hard to tell whether a model succeeds because it reasons well, because it already knows the task, or because it got lucky. A full run of Slay the Spire forces a model to make a long chain of connected decisions, so the researchers can trace exactly where performance breaks down.

At the game’s hardest difficulty setting, the strongest models beat simple rule-based baselines and perform about as well as an average human player, according to the paper. But they remain well behind expert human players, and that gap grows the longer a run lasts. The top models clear the game’s first act about as often as experts do. After that, their results slide. The authors trace the split between stronger and weaker models to how well each one handles individual fights within a run.

Models play too safe

Across the board, the paper reports a consistent pattern: models misjudge risk over the course of a run. They tend to protect their health instead of spending it on choices that would make their deck stronger later, a trade-off that human experts make more readily. The researchers also ran a test where every term in the game was systematically renamed, which strips away a model’s ability to lean on memorized knowledge of Slay the Spire from its training data. That test showed part of the models’ performance comes from having already seen the game before, rather than from reasoning about the situation in front of them.

The paper also looked at whether models improve when they are allowed to write and reuse notes across multiple runs, a memory mechanism agentic systems increasingly rely on. The authors found no such improvement. The notes models wrote for themselves were often inaccurate and pushed the model toward overly cautious play, carrying the same risk-aversion problem into future attempts rather than correcting it.

The authors describe their test harness as the first to give a model the complete, original version of the commercial game, rather than a simplified clone, and they compare it against existing game-playing harnesses used in other AI research. They present SpireBench as a long-horizon benchmark that is not yet saturated, meaning top models are nowhere close to maxing out the score, with tools built in to pinpoint where and why an agent’s performance falls apart during a run.

What to watch next is whether model developers respond to the specific failure the paper identifies, misjudged risk-taking in extended decision chains, and whether later test runs on SpireBench show models closing the gap with expert human play rather than plateauing at the first act.