NVIDIA and AI4Bharat announce 2.8 trillion-token pretraining dataset for Indic languages

NVIDIA and the AI4Bharat lab at IIT Madras have announced Nemotron-pretraining-Indic-v1, a pretraining dataset of roughly 2.8 trillion tokens spanning 22 Indian languages, due for release alongside the open pipeline used to generate it. The 22 languages are the ones listed in India’s Constitution, first languages for 97% of Indians, or close to 1.2 billion people at the 2011 census, according to the companies.

The corpus covers 34 language-script combinations. It includes native-script text, meaning each language’s own writing system, for all 22 languages, and romanized text, meaning the same languages written in Latin letters, for twelve of them. The companies put the total at 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, counted with the Gemma-4 tokenizer. The data was drawn from 15 subsets of Nemotron-CC-v2, an existing English dataset covering web text, question-and-answer material, mathematics, and code.

To build the corpus, the teams translated English documents using NVIDIA’s NeMo Data Designer framework. For twelve languages, full documents were translated with the Gemma-4-31B-it model, then scored against the English source on a rubric adapted from Multidimensional Quality Metrics, an established framework for grading translation errors by type and severity. Outputs scoring below 75 out of 100 were rewritten using that feedback and rescored, a repair step that recovered 438.4 billion tokens that would otherwise have been discarded. Romanized versions were then generated from the accepted native-script translations rather than directly from English, which the companies say produced fewer hallucinated additions and less leftover English text. The remaining ten languages were translated with Indic Translate, a document-level translation model from the nonprofit Bodhan AI, with adjustments for long documents and for Manipuri’s native script.

NVIDIA’s Nemotron 3.5 Super VL technical report describes using a twelve-language slice of the corpus, combined with naturally written Indic text, during a continued pretraining stage. Of a 100-billion-token budget for that stage, 10 billion tokens were Indic data. Scores rose on all five Indic benchmarks the report tracks, MILU, Indic MMLU-Pro, Indic ARC-Challenge, Indic ARC-Easy, and Indic GSM8K, with gains ranging from 4.98 to 9.82 percentage points. The GSM8K score excluded Odia because the evaluation exceeded the model’s serving context limit.

Mitesh Khapra, who heads AI4Bharat, has said Indian-language technology lags English because “we do not have enough data for Indian languages.” The new corpus and its generation recipe, including the prompts and quality checks used to build it, are meant to let other developers reproduce the work or point the same pipeline at their own source documents.

NVIDIA says future Nemotron models will build on this corpus, and a companion technical report examines how source mixtures and script coverage affect results using an earlier, partial version of the data. Evaluating the full corpus, including the recovered rewrites and the ten additional languages, remains an open task the companies have flagged for follow-up study.