Hugging Face releases Olmo-core 3 to train huge mixture-of-experts AI models

Hugging Face has published Olmo-core 3, an open-source system for training large mixture-of-experts models, a type of AI model that activates only a subset of its components for each input. According to the announcement, the framework is meant to scale this approach into the trillion-parameter range without losing the computational savings that make the approach attractive in the first place.

Mixture-of-experts models split their knowledge across many specialized components, called experts, and send each piece of input to only a few of them. That lets a model hold far more parameters, the learned values that store what it knows, without processing all of them for every input. But coordinating which expert handles which input across a cluster of GPUs adds communication overhead, and that overhead can erase the savings as the number of experts grows. In one benchmark described in the release, expanding a model’s expert pool from 8 to 128, while still routing each token to four experts, took total parameter capacity from 4.6 billion to 47 billion. Active parameters per token stayed near 3.2 billion throughout, and training throughput fell by less than 5%. The same system has been benchmarked at over one trillion total parameters.

How the system is built

Olmo-core’s earlier mixture-of-experts setup relied on fully sharded data parallelism, a method that gathers and reassembles model weights for every batch of training data. Olmo-core 3 switches to distributed data parallelism, which keeps each expert resident on its assigned GPU and routes data to it instead. In a preliminary test on eight NVIDIA B300 GPUs, a 47-billion-parameter model reached 52,000 tokens processed per second per GPU under the new system, compared with 19,400 under the earlier one, roughly 2.7 times the throughput.

The framework divides the work three ways: expert parallelism spreads experts across GPUs, pipeline parallelism splits the model’s layers across GPU groups, and a distributed optimizer spreads the extra data needed to calculate training updates across GPUs rather than duplicating it everywhere. Additional techniques, including a method that places routed data directly into expert buffers and one that keeps routing information on the GPU rather than shuttling it to the CPU, are aimed at cutting the overhead of directing data to the right expert.

The release also supports MXFP8, a lower-precision number format. In a controlled test on four B300 GPUs, enabling it raised training throughput by about 21% over the higher-precision BF16 format and reduced peak memory use from 103 GiB to 95 GiB.

At larger scale, the system has been benchmarked on a 1.2-trillion-parameter model running across 512 GPUs, reaching a peak of 858 TFLOP/s per GPU, a measure of useful computation performed each second. A separate test using an alternative communication method, DeepEP v2, reached a configuration of 2.38 trillion total parameters, though the release describes that as a short capacity test rather than a full training run, and notes that both large-scale tests used random routing to measure system speed rather than model quality.

The framework underpins an upcoming Olmo model that will use a mixture-of-experts architecture, trained on a larger dataset with a longer context window than prior versions, according to the announcement. Olmo-core 3 and its accompanying technical report are available on GitHub, and what happens next depends on whether outside researchers can reproduce the throughput and scale claims on their own hardware.