OpenAI published research on 19 October about building and migrating software in languages such as COBOL and Fortran. The useful question is what survives a rewrite: code can compile successfully while quietly changing a business rule. Its two evaluations approach that problem differently.

Based on 4 reviewed sources · See sources

Building a database and migrating a program are different tests

In OpenAI’s research report, MultiLangBench asks models to build a SQLite-compatible database engine in 20 programming languages. SQLite is a small database system; the task tests whether the replacement handles queries and changes data correctly. OpenAI’s original announcement below introduces the work.

From the source · OpenAI ·

OpenAI introduces its low-resource-language research. The announcement describes evaluations of existing models.

View the original post on X ↗

The separate migration evaluation translates existing COBOL programs into five other languages. A program succeeds only when all its hidden tests reproduce the original’s observable output byte for byte. That catches changes such as rounding or truncation that a clean compilation would miss.

The public test suite has its own conditions

The public repository specifies SQLite 3.40.1 and roughly 24,000 converted tests. Its recommended lower-cost evaluation uses a 272-test subset; a full-suite run is allowed, but should be disclosed. Internet access is prohibited, and evaluators must prevent the candidate from using an existing SQLite implementation.

The repository history also matters: the SQLite evaluation was added on 29 September, the Apache 2.0 licence on 6 October, and the project renamed MultiLangBench on 19 October. The research announcement is new; the underlying public files have an earlier history.

The report does not identify whether its headline database scores use the recommended subset or the full suite. It describes the COBOL migration tests as hidden. Those details matter before treating a headline score as a reproducible comparison.

A passing test still has a boundary

OpenAI itself says production migration needs expert review, dependency checks and tests of business rules. The inspected reactions supplied no independent replication with published conditions and outputs, so the company’s reported gains remain supplier results.

Executing parity tests is also a different check from Claude’s hosted code review, which reads proposed changes without running them. For a rewrite, the question remains concrete: which behaviours were tested, and which still depend on someone knowing why the old system worked that way?

Sources

Written with AI assistance from the sources above. Analysis and practical implications are our interpretation. Our editorial approach.