Liquid AI speeds up vision-language model decoding by up to 3.13x

Liquid AI has released LFM2.5-VL-3B-DSpark, an experimental draft model built to speed up decoding, the step where a model generates output tokens, for its LFM2.5-VL-3B vision-language model. The company says the addition cuts decoding time without changing what the model outputs.

The drafter works through speculative decoding, a technique where a small model guesses several tokens ahead and a larger model checks them in one pass. A standard model produces one token per pass. With DSpark attached, Liquid AI reports decoding speedups of up to 3.13x on an Apple M5 Max MacBook Pro and up to 2.66x on a single Nvidia H100 GPU. End-to-end speedups, which include image processing and not just decoding, were smaller: up to 2.62x on the M5 Max and up to 2.27x on the H100.

DSpark adds about 280 million parameters to the model, which Liquid AI says raises the deployed parameter count by 8.9%. The drafter shares its embedding and output layers with the target model rather than carrying its own copy. It reads hidden states, the target model’s internal representations, from several of its layers and predicts the next several tokens at once. Liquid AI says the same drafter design works for both text and images because, by the time information reaches those hidden layers, image patches and text are processed as the same kind of numerical data.

The company trained the drafter on supervised fine-tuning data covering common vision-language tasks for 10 epochs, a full pass through the training data, using AMD hardware for training and testing. It evaluated results using its MMSpec benchmark across six task types, including visual question answering, image captioning and multi-turn conversation, with all runs at 16-bit precision and a single response generated per prompt.

What limits the speedup

Liquid AI says the gains are capped by a bottleneck the company frames using Amdahl’s law, a principle that says total speedup is limited by whatever portion of a process is not accelerated. Speculative decoding only speeds up token generation. It does nothing for encoding an image or prefill, the initial processing of the prompt and image data before decoding starts. On devices with less computing power than data center GPUs, that unaccelerated portion takes up a larger share of total time. Liquid AI points to TextVQA testing on the M5 Max as an example: decoding ran 2.69 times faster, but the full task only sped up by 1.56 times.

Under greedy decoding, a mode that always picks the most likely next token, Liquid AI says output matches the base model exactly, since every token the drafter proposes gets checked by the target model. At other sampling settings, the company says the underlying token distribution is preserved, citing prior published research on speculative decoding. Higher temperature, a setting that widens the range of tokens a model considers, led to more disagreement between drafter and target, and slower results, in Liquid AI’s own tests.

The release is labeled experimental, and the company has not published acceleration results for quantized versions of the model, a compressed format that reduces memory use. Liquid AI has posted integration code as pull requests to the llama.cpp, SGLang and MLX-VLM projects, and developers can watch those repositories for the pace of review and any follow-up support for quantized deployment.