Study finds nearly half of biomedical AI papers face model retirement risk

A new study finds that a large share of biomedical research relying on AI language models is at risk of becoming impossible to verify, because the models themselves keep disappearing. Researchers led by Nathan Wolfrath and colleagues report that 42% of biomedical papers using a named AI model were tied to a model already retired by the time of publication, or scheduled for retirement within two years afterward.

The team searched PubMed, the biomedical literature database, for original research published between 2022 and March 2026 that applied a specific large language model, or LLM, to a biomedical task. An automated extraction system scanned 61,077 article abstracts to identify which models were used, with human reviewers checking a portion of the results for accuracy. The researchers then matched each model to lifecycle data, including its release date and, where applicable, its retirement date, for the 50 most frequently cited models.

After narrowing to those top 50 models, the study identified 8,931 paper-model mentions spanning 5,242 individual publications. Commercial closed-weight models, meaning proprietary systems whose underlying code and parameters are not publicly released, accounted for 77.7% of those mentions. The median time between a paper’s publication and its model’s retirement was 538 days, or about a year and a half.

Why retirement breaks reproducibility

When a commercial vendor retires a model, outside researchers typically lose the ability to rerun the exact system a study used, since the vendor controls access and can shut it down or replace it with a newer version. This differs from open-weight models, whose files can be preserved and rerun independently of any company’s hosting decisions. Because closed-weight systems dominate the biomedical AI literature, the study’s authors argue that reproducibility, the ability to independently rerun a study and reach the same result, is increasingly out of reach for this field once a vendor’s clock runs out on a model.

The authors conclude that model deprecation should be treated as a core reporting and preservation issue in biomedical research, not a side effect of using commercial AI tools. They do not specify in the abstract what preservation steps journals or vendors should adopt, only that the current trajectory points toward widespread computational non-reproducibility after publication.

What happens next depends on decisions outside the study itself: whether journals begin requiring authors to archive model outputs, prompts, and version identifiers at submission time, and whether AI vendors offer any mechanism for researchers to access retired models for verification purposes. The paper does not report on any such policy changes underway, leaving the scale of the problem it documents unresolved for now.