Benchmark quantifies how AI agents' token costs vary by JS framework

A developer who publishes under nolanlawson.com built the same small web app five times over in each of roughly a dozen JavaScript frameworks, then measured what it cost an AI coding agent to produce working code in every one. The goal, the post says, was to replace “vibes-based” claims about which frameworks suit AI agents with actual numbers on cost, time, and tokens, the units an AI model’s usage is billed in.

The test app was a Pokémon database site with search and filtering, built from the same spec each time using the PokéAPI. Claude Opus 5.5, Anthropic’s model, ran each build with high effort and access to the official MCP or skill tools for that framework, where available. MCP, or Model Context Protocol, lets an agent pull in a tool’s own documentation. The author did not dictate an architecture, such as server rendering versus a single-page app, and the agent had no advance knowledge that its output would later be scored in Google’s Lighthouse testing tool for performance and accessibility. Finished apps were checked against a conformance test suite built on the accessibility tree, chosen so that frameworks using different underlying markup, including web components, could all be evaluated fairly.

According to the analysis, SvelteKit, the Gracile meta-framework paired with the Lit web-component library, and Angular were consistently the costliest frameworks to build in. The post attributes SvelteKit’s expense largely to the agent repeatedly reading a version 3 migration guide, likely because that release was too recent for the model’s training data. For Gracile, the author writes that the agent often didn’t know the framework well enough and resorted to cloning its source repository or reading compiled code directly from its node_modules folder. Angular’s higher cost is linked to the overall size of the code it produced; the framework also failed to generate a measurable app in two of five attempted runs, so some of its metrics are based on three runs instead of five.

The author cautions that costs varied widely between runs of the same framework, calling the results “very stochastic,” a term for behavior driven by randomness. React Router’s median cost ran about $1 higher than Hono paired with htmx, yet React Router’s cheapest run was still cheaper than Hono+htmx’s most expensive one. A plain “vanilla” JavaScript build using the Vite toolchain, with no framework at all, cost more than several frameworks did, which the post says suggests a framework can sometimes be cheaper for an agent than writing from scratch.

Lighthouse scores converge, cost data diverges

On outcomes, Lighthouse’s performance and accessibility scores came out nearly identical across frameworks, the post reports, leaving the clearest differences in the amount of JavaScript shipped to the browser and in load-timing metrics that tend to favor server-rendered, multi-page sites over single-page apps.

The author stresses this covers one type of app, a content-heavy site with limited interactivity, and that agents used each framework’s official tooling inconsistently from run to run. A follow-up test on a more interactive app, or with stricter enforcement of each framework’s MCP tools, would show whether these cost patterns hold.