Researchers unveil Seq-Drifting, a one-pass method for AI text generation

Researchers have proposed a new framework called Seq-Drifting that trains a diffusion language model to generate an entire passage of text in a single forward pass, rather than the multiple sampling steps these models normally require. The work is described in a paper titled “One-Step Text Generation by Seq-Drifting,” posted to arXiv by authors Jiayu Liu, Ruizhi Zhao, Tianyun Ji, Yee Whye Teh, Zhenya Huang and Ning Miao.

Diffusion language models, a class of AI systems that build text by gradually refining noisy data over repeated steps, generate tokens in parallel but have still needed several rounds of sampling to produce high-quality output. The authors set out to test whether a model could instead learn to produce a full sequence in one step.

Seq-Drifting works by applying what the authors call drifting to sequences of continuous token representations, learning a direct map from Gaussian noise, a standard random starting point for diffusion models, to finished text. The paper says this differs from existing diffusion language models, which derive training signals at each position directly from the token observed in a training sequence. Instead, Seq-Drifting builds what it calls a drift field, where the target at each position adapts to the context the model has already generated.

According to the authors, this drift field combines two forces. Attraction pulls each output toward a set of token representations that are plausible given the context, while repulsion discourages the model’s samples from collapsing into similar outputs or repeating within a sequence. During training, the generator learns from this drift field, and at inference time it produces all tokens jointly in one pass.

Performance across tasks

The authors report that Seq-Drifting achieves a generative perplexity, a measure of how well a language model predicts text, of 36.01 on the LM1B benchmark using just one generation step. They say the method performs competitively in both unconditional generation, where the model writes freely, and conditional generation, where it responds to a prompt or input.

The paper also describes experiments applying the framework to translation, summarization and reasoning tasks, which the authors say demonstrate that Seq-Drifting’s approach extends beyond open-ended text generation. The authors frame these results as evidence that drifting over token sequences, rather than individual tokens, offers a promising direction for one-step text generation.

The paper does not report comparisons of generation speed or computational cost against standard multi-step diffusion language models, so the degree of any practical speedup is not yet established from the available material. It also does not specify the size of the models used in the experiments beyond the benchmarks named.

The authors say code for Seq-Drifting is available, though the specific repository link was not included in the source material reviewed here. Independent testing of the method on additional benchmarks, and any published comparisons of inference speed against existing diffusion language models, would help clarify how much of a practical difference the one-step approach makes.