Buoyant's Kinference cuts vLLM response wait time by 95% in tests

Buoyant has released Kinference, a free, open-source control plane that manages how requests reach vLLM, a popular engine for running large language models, on Kubernetes clusters. The company says its own benchmark found a 95% drop in median time-to-first-token, the delay between a user’s request and the first word of a reply, after switching to Kinference’s routing.

According to MarkTechPost, which reported the release, Kinference’s main contribution is cache-aware load balancing. Instead of spreading requests evenly across available servers, it directs each request to the machine already holding that conversation’s cached data, known as a KV cache, so the model does not have to reprocess earlier context from scratch. MarkTechPost reports that this is the mechanism behind the reported drop in wait time.

The release also adds declarative routing policies, which let operators set rules for how traffic should flow without writing custom scheduling code, and per-tenant metrics, which track usage and performance separately for each customer or team sharing the same cluster. MarkTechPost’s report does not specify how the benchmark was configured, what hardware or model size was used, or how it compares with default Kubernetes load balancing beyond the single percentage figure.

Kinference is licensed under Apache 2.0, a permissive open-source license that allows free commercial use and modification. That licensing choice puts the project in the hands of any team running vLLM on Kubernetes rather than limiting it to Buoyant customers.

What this means

For teams running large language models in production, the time a user waits for the first token is one of the clearest signals of whether an AI product feels responsive. A tool that targets that specific metric, rather than overall throughput, speaks to operators who have already adopted vLLM and Kubernetes as their serving stack but are hitting latency complaints from users. The per-tenant metrics suggest Buoyant is also targeting operators who serve multiple customers or internal teams from one shared cluster and need to see usage broken out by tenant.

What is not yet known, because MarkTechPost’s report does not cover it, is how the 95% figure holds up outside Buoyant’s own test conditions, what workload or cache-hit rate the benchmark assumed, or how Kinference performs against other cache-aware routing approaches already used by some large AI infrastructure teams. Independent benchmarks from users running their own traffic patterns would give a clearer picture of how much of that improvement generalizes.

Buoyant has not said, per the report, whether it plans a paid or managed version of Kinference alongside the open-source release, leaving open how the project will be sustained as it adds features.