New encoding squeezes ternary AI model weights below the 1.58-bit limit

Ternary large language models store each of their internal numbers, called weights, as one of three values: -1, 0, or +1. This trims the amount of data a model needs and is meant to make AI systems cheaper and faster to run, a process known as quantization: shrinking how precisely numbers are stored. The standard way of packing these models fits five ternary weights into one byte of storage, which works out to 1.625 bits per weight once rounding is accounted for.

Researchers Evangelos Georganas, Alexander Heinecke, and Pradeep Dubey examined how that packing method compares with the numbers actually found inside real models. In a paper posted to arXiv, they measured the mix of -1, 0, and +1 values across 29 published ternary language models. The standard packing scheme assumes the three values show up in roughly equal shares. The researchers found that isn’t true: zeros made up as much as 51.5% of all weights in some models, far more than an even split would predict.

A format built around zeros

Based on that finding, the researchers built a new storage layout called BITCOS. It separates a model’s weights into two pieces: a bitmap marking which weights are zero, and a compacted list of plus/minus signs for the weights that aren’t. Because zeros need only a single bit to mark and nothing more, a model with more zeros needs less total storage. The paper expresses this as a cost of 2 minus z bits per weight, where z is the share of weights that are zero.

Tested against the standard five-weights-per-byte method, BITCOS stored weights more compactly in 26 of the 29 models the researchers examined, and reached as low as 1.485 bits per weight on the model with the most zeros. That is below the 1.585-bit theoretical minimum long used as a reference point for ternary models, though the researchers note that figure assumes equal odds for each of the three values, an assumption their own measurements say doesn’t hold.

The paper also addresses whether the format is practical to use, not just compact. The authors wrote optimized code for unpacking BITCOS-formatted weights on AVX-512 and AVX2, two instruction sets built into many CPUs, as well as on Intel’s Xe2 graphics chips. Compared with existing production kernels for ternary matrix-vector multiplication, a core computation in running these models, the new layout delivered gains of up to 1.28 times. In full end-to-end tests across five different machines, covering client and server CPUs along with integrated and discrete Xe2 GPUs, decode throughput, the rate at which a model generates output tokens, improved by up to 1.18 times on CPUs and 1.27 times on GPUs.

The paper’s DOI registration is still pending, meaning the work has not yet completed formal peer review. What happens next is whether ternary model makers adopt a zero-aware format like BITCOS in place of the current five-trit packing standard, and whether the reported speedups hold up on models the authors didn’t test.