Perplexity open-sources Lily, a Rust-based engine for local AI on Macs

Perplexity has open-sourced Lily, an inference engine — software that runs an AI model and generates text — built specifically for Apple’s M-series chips. The company says Lily powers Hybrid Compute, a feature in its Perplexity Computer product that runs part of a request on the Mac itself rather than in the cloud.

Lily is written in Rust and uses hand-written Metal kernels, low-level code that talks directly to the chip’s graphics processor. Neither PyTorch nor MLX, the two frameworks most Mac AI tools depend on, appears anywhere in the execution path, according to MarkTechPost. Lily supports exactly one model, Qwen3.6-35B-A3B, which stores 35 billion parameters but activates only about 3 billion of them per token, through a router that selects 8 of 256 available experts plus one shared expert every time.

Perplexity says that narrowness is the point. Its default Mac stack, MLX paired with MLX-LM, already ships a tuned Qwen implementation, but that code has to stay reusable across many model architectures. Lily gives up that flexibility, tuning expert routing, attention math and cache reads specifically for this one model on this one family of chips. In Perplexity’s own tests, fusing weight decompression into the matrix-multiplication step raised prompt-processing speed 77.4% at 512 tokens, and moving expert-routing decisions fully onto the GPU added another 89%. During text generation, grouping four query heads so each cache row loads once improved speed 23.8% at 32,000 tokens of context, and a fixed-block attention layout added 40.2% at 128,000 tokens.

On a 40-core, 128GB M5 Max Mac, tested at batch size 1 across ten prompt lengths from 256 to 128,000 tokens, Lily averaged 4,156 tokens per second while processing prompts, against 3,388 for MLX-LM, and 170.0 tokens per second while generating text, against 126.4, Perplexity reported. A separate accuracy check across 192 positions found Lily picked the identical top-ranked word as the original, uncompressed model 96.35% of the time.

Available now, narrowly scoped

A standalone demo server is public in Perplexity’s pplx-garden GitHub repository, serving the model through an API compatible with OpenAI’s chat format. The compressed checkpoint is 19.4GB, roughly a quarter of the model’s uncompressed 70GB size, and Perplexity recommends a Mac with at least 32GB of unified memory, the RAM pool shared by the CPU and GPU on Apple silicon.

Perplexity has not said whether Lily will support additional models or chip families beyond this one pairing. The speed comparisons published so far all come from Perplexity’s own benchmarking; independent tests of Lily against MLX-LM have not yet appeared.