Perplexity opens up its GPU serving stack behind pplx-embed search

Perplexity’s engineering team has published a detailed account of how it serves the embedding models behind its search product, describing the infrastructure in a post titled “Fast Embeddings on GPUs.” The write-up covers the systems running pplx-embed and the ranking models used across Perplexity Search, Perplexity Computer and the company’s API Platform.

The team says raw embedding inference has largely converged across different software engines on modern Hopper and Blackwell GPUs, meaning the real gains now come from the runtime and harness surrounding the model rather than the model itself. That includes CUDA graph management, an async result-tracking system, and a request path written in Rust.

The stack splits into three named components. Ivy is a Rust HTTP gateway that handles JSON parsing, tokenization and batch splitting before translating requests into a custom gRPC protocol, also load-balancing chunks across replicas. Tulip is the gRPC inference server, built with Rust, tokio and tonic, which schedules and batches requests before passing them to the engine. ROSE, short for Runtime-Optimized Serving Engine, is mostly Python and handles the actual model inference, managing CUDA graphs and exposing a step() function that Tulip calls.

A central design choice was not building a dedicated embedding engine at all. Because embedding models are small transformers, Perplexity treats batch embedding, used when building or refreshing its vector database, as similar to compute-bound prefill work, and treats short, query-time embedding as similar to memory-bound decode work. Both reuse kernels already built for its large language model stack.

Scheduling and hardware tricks

Tulip schedules sequences first-come, first-served, a simple approach the team says is justified by measurement: at the sequence lengths it serves, the linear cost of dense layers outweighs the quadratic cost of attention, so latency tracks token count rather than sequence count. On a sub-billion-parameter model, around 512 tokens is enough to saturate the GPU, after which packing in more sequences stops helping.

For small batches, launching kernels from the CPU can take longer than the GPU work itself, so Perplexity builds whole-model CUDA graphs that collapse every kernel launch into a single driver call. Because these graphs must be captured separately for each padded token-count configuration, capturing all of them can take minutes per model, so the team uses lazy capture: each configuration runs once eagerly, then captures and replays on its second hit. A related technique, called LazyTensor, tracks a pinned memory buffer and a CUDA event so a Rust async task can wait on one batch’s result while the CPU prepares the next, instead of blocking on the GPU.

ROSE supports several attention backends, including FlashInfer 2, FlashInfer 3 and FlashAttention 4, choosing between them by model and sequence length, and skips the KV cache entirely when serving embeddings since there is no multi-step generation to cache.

Perplexity benchmarked the system against vLLM v0.22.0 in BF16 precision across four test suites covering low-latency and high-throughput scenarios, checking that outputs stayed within 0.1% cosine similarity of the reference. Ivy, Tulip and ROSE remain internal systems; developers can reach the resulting model through Perplexity’s Embeddings API. What is not yet public is a head-to-head cost or latency figure against that vLLM baseline — the post describes the architecture and testing methodology but does not disclose the resulting numbers.