OpenAI builds new benchmarks to improve ChatGPT's coding in COBOL and SQLite

OpenAI says it is working to make ChatGPT more useful for programmers writing in low-resource programming languages, those with far less code and documentation available online than mainstream languages like Python or JavaScript. The company has built new evaluations, or benchmark tests, to measure how well the model performs on real software engineering tasks in these languages, according to OpenAI. Developers have increasingly turned to AI assistants like ChatGPT to speed up coding work, but those tools have historically performed best in widely used languages, leaving a gap for programmers working in rarer ones.

Among the languages covered are SQLite, a lightweight database engine, and COBOL, a programming language that dates back to the 1960s and still runs many banking, insurance, and government systems. OpenAI frames part of the new testing around COBOL migration, the work of moving old COBOL code onto newer systems or languages.

Large language models like ChatGPT are typically trained on vast amounts of publicly available code. Languages such as Python, JavaScript, and Java dominate that pool, because they are widely used in open-source projects. COBOL and other older or more specialized languages appear far less often, which OpenAI says has made it harder for ChatGPT to handle tasks in them as reliably.

Why COBOL still matters

Despite its age, COBOL remains embedded in critical infrastructure. Government agencies and financial institutions have run COBOL systems for decades, and replacing or updating that code is a long-standing challenge for organizations that lack programmers trained in the language. Many of the programmers who originally wrote these systems have since retired, leaving organizations short on in-house expertise to maintain or replace the code. OpenAI’s evaluations target this gap directly, testing how well ChatGPT can read, explain, and help migrate COBOL code rather than just write it from scratch.

OpenAI has not published specific scores, dates, or details about how the benchmark suite is structured beyond saying it spans software engineering tasks, SQLite, and COBOL migration. It is not yet clear how many test cases the evaluations include, how they compare ChatGPT’s performance against other languages, or when any resulting model improvements might reach users. The lack of detail makes it difficult to judge how large an improvement, if any, the new evaluations represent.

What to watch next: whether OpenAI releases the benchmark suite itself, or any accuracy figures showing how ChatGPT’s performance on COBOL and SQLite tasks compares with its performance on more widely used languages.