bf16 rounding quietly discards most RL updates in language models

A pair of researchers has identified a rounding flaw in how language models are fine-tuned with reinforcement learning (RL), a training method that rewards good outputs and penalizes bad ones. The flaw, described in a paper posted to arXiv by Andrew Gao and a co-author, causes the vast majority of the model’s intended updates to be thrown away before they ever reach the version generating new training examples.

During RL post-training, the model’s true weights are stored in fp32, a 32-bit number format precise enough to capture small changes. But the copy of the model that actually generates rollouts, the sample outputs used to compute rewards, is stored in bf16, a 16-bit format that rounds each weight to the nearest representable value. The researchers found that at the learning rates commonly used in practice, most individual weight updates are smaller than half the gap between two adjacent bf16 values. Round-to-nearest conversion simply erases them.

The effect is large. On Qwen3-1.7B, a 1.7-billion-parameter model from Alibaba’s Qwen series, only 0.4 to 0.5 percent of the served weights actually change on a given training step, according to the paper. Because so many updates disappear, the version of the model producing training data falls behind the version being optimized by 50 to 150 steps, the researchers report. They note that existing methods for catching mismatches between training and inference miss this problem entirely, because those methods compare two copies that have already been rounded the same way.

A one-line change

The researchers propose a fix they call Fair Cast. Instead of exporting the fp32 master weights to bf16 with standard round-to-nearest conversion, Fair Cast uses an unbiased rounding rule, either stochastic rounding or error feedback, and computes the training loss on the resulting served copy. According to the paper, the change requires editing one line at the export step, adds no new hyperparameters, and costs an extra 22 to 38 milliseconds per export of a 1.7-billion-parameter model.

The researchers tested Fair Cast across 25 matched seed-pairs on Qwen3 models ranging from 0.6 billion to 4 billion parameters, using two different optimizers, AdamW and Muon. Fair Cast outperformed standard rounding in 18 of those pairs, tied in 6, and lost in 1, for a mean gain of 6.1 points, the paper states. On a set of logic puzzles called zebra puzzles, Fair Cast reached a score of 0.57 plus or minus 0.16 across four runs, compared with 0.33 plus or minus 0.10 for round-to-nearest rounding, according to the researchers. They add that Fair Cast also makes the saved fp32 checkpoint match the model that was actually used to generate training data, which round-to-nearest conversion does not guarantee.

The paper does not report results on models larger than 4 billion parameters, and its DOI registration through DataCite is still pending, according to the arXiv listing. The researchers also point out that the lag created by round-to-nearest rounding has been acting as an unintended regularizer, a mechanism that limits how fast a model changes, and they say that effect can be restored on purpose when a task calls for it. Whether major RL training frameworks adopt Fair Cast, and whether the gains hold on larger models, has not yet been established.