Legacy code migration evals should measure preserved behavior, not the visual similarity of translated code. A model can produce modern, compiling Java or Rust while silently changing decimal arithmetic, missing-record handling, transaction order, file semantics, or recovery behavior.
OpenAI's new low-resource programming-language evaluations make an important move in the right direction. They separate building new software from migrating existing software and require observable outputs to match. The next step is to turn that benchmark idea into a production migration contract.
OpenAI is testing two different capabilities
The first evaluation, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in 20 programming languages. A candidate receives an initial database and a query, then produces a modified database. The real SQLite implementation runs a validation query against that output. The expected output must match exactly.
OpenAI reports that GPT-6 Astra reaches an average 87.4% test pass rate, compared with 26.8% for GPT-5.6 Sol. Results improved across all 20 languages. COBOL remained the hardest reported language at 72.1%, up from 10.9%.
The second evaluation is closer to enterprise modernization. A model migrates COBOL programs to C++, Java, Kotlin, Python, or Rust. Hidden tests check byte-for-byte parity on observable outputs, and a migration counts as successful only when all tests pass. OpenAI highlights fixed-point behavior as a concrete hazard: COBOL can discard excess decimal digits unless rounding is explicitly requested, so a translation that rounds may run normally while changing financial results.
On this internal migration evaluation, OpenAI reports 86.6% accuracy for GPT-6.1 Sol at low reasoning effort, compared with 5.4% for GPT-5.6 Sol, at 61% of the API cost. Higher reasoning reaches 97.9%.
These are benchmark results, not production migration rates. OpenAI explicitly says real deployments still require expert review and tests against the organization's business rules and dependencies.
The public artifacts establish an evidence boundary
OpenAI released the MultiLangBench setup and tests. Its repository makes several details auditable:
- the target is SQLite 3.40.1;
- roughly 24,000 of SQLite's 51,000 TCL tests were converted into the evaluation format;
- the default evaluation instruction says to run the 272 tests whose names begin with
00,01, or02, with full-suite runs allowed when disclosed; - a crash, mismatch, or 60-second timeout fails a case;
- internet access and installed SQLite implementations are prohibited;
- the score is the pair of API cost and test pass rate.
That public repository is an inspectable test artifact, but not a full reproduction package for the reported scores. It does not include the complete agent harness, prompts, model settings, per-language run artifacts, or cost logs. The official article also does not say whether its reported results used the 272-test default subset or all converted tests. The repository still prevents a common reading error: an average pass rate across 20 language implementations is not evidence that every SQLite behavior, resilience property, or production interface was reproduced.
The COBOL migration evaluation is described in the official article, but OpenAI's linked release is MultiLangBench. The migration dataset, hidden tests, sample composition, and runner are not publicly available through that release. The 86.6% and 97.9% figures therefore sit at a different evidence level: official provider-reported benchmark results without the same public reproduction package.
This distinction matters more as scores approach saturation. A 97.9% score can mean the model nearly solved a representative task distribution. It can also mean the current test distribution no longer separates the remaining production risks. The next evaluation should expand the contract before treating the score as migration readiness.
Behavioral equivalence has several layers
Byte-for-byte output parity is strong when the output bytes fully encode the business behavior. It is insufficient when important behavior lives outside that channel, and it can be unnecessarily strict when timestamps, identifiers, ordering, or formatting are allowed to differ.
A migration contract should define at least five layers.
| Layer | Questions | Evidence |
|---|---|---|
| Functional output | Do the same inputs produce allowed outputs? | Golden cases, differential tests, invariants |
| State and transaction | Are balances, records, ordering, idempotency, and commit boundaries preserved? | State snapshots, transaction traces, replay |
| Integration | Do file formats, schedulers, databases, messages, and downstream consumers still work? | Contract tests, dependency simulation, end-to-end runs |
| Failure behavior | Do missing data, partial files, timeouts, restarts, and duplicate delivery resolve correctly? | Fault injection, restart tests, recovery assertions |
| Operational contract | Are throughput, latency, security, audit, reconciliation, and support limits acceptable? | Load tests, security checks, shadow traffic, runbooks |
IBM's COBOL overview makes the platform boundary explicit. Modernization can involve language translation, but a complete strategy also needs data architecture, runtime replacement, transaction integrity, and other platform concerns. The source language contains only part of the system's contract.
Build an independent behavior oracle
The original program is useful as an oracle, but it cannot be the only authority. Legacy behavior may include old defects, environment-specific quirks, or outputs that violate current policy. Tests generated from the same code or model that performs the migration can reproduce its assumptions instead of challenging them.
Use several independent evidence paths:
- Freeze representative production inputs after privacy review and preserve their provenance.
- Run the old system to capture outputs, state changes, external calls, error paths, and timing envelopes.
- Have domain owners label which behaviors are required, tolerated, intentionally changed, or unknown.
- Add property and metamorphic tests for invariants that examples do not cover.
- Generate adversarial cases from incidents, reconciliation breaks, and historical support tickets.
- Keep protected cases outside the migration agent's context.
- Compare the migrated system with both the old implementation and the approved business contract.
This is where low-resource languages create a special problem. Public code and documentation are limited, but the organization's private execution history can be rich. Production traces, batch files, reports, reconciliation records, and incident logs become the practical specification. They require governance because they can also contain sensitive data and obsolete behavior.
Use an evidence ladder instead of one pass rate
A migration should advance through evidence levels that make overclaiming difficult.
Level 1: Mechanical validity
The target compiles, resolves dependencies, passes static checks, and uses the required language. This removes many local errors and proves little about business parity.
Level 2: Frozen benchmark validity
The candidate passes declared visible and hidden tests under a fixed environment. This is the level measured by most migration benchmarks.
Level 3: Organization-specific parity
The candidate passes differential and contract tests drawn from real workflows, including edge cases and dependency behavior. Domain owners resolve intentional differences.
Level 4: Shadow operation
The old system remains authoritative while the new system processes mirrored or replayed traffic. Outputs and state transitions are reconciled over a declared period and volume.
Level 5: Reversible production cutover
A bounded cohort moves to the new system. The team monitors business reconciliation, performance, security, and operations. Rollback criteria, compatible data state, and decision ownership are tested before expanding traffic.
Each level supports a stronger claim. A public benchmark can justify further evaluation. It cannot by itself justify decommissioning the old system.
Evaluate the migration workflow, not only the model
Model tables report API cost per benchmark sample. Enterprise decisions need the cost per accepted migration unit:
model use + harness compute + test construction + domain review + remediation + shadow operation + cutover + rollback preparation
Track at least:
- percentage of units reaching each evidence level;
- defects found by the model's tests versus independent tests;
- human review and adjudication time;
- parity failures by business-risk class;
- retries and regeneration cost;
- shadow mismatches and production escapes;
- rollback time and state-reconciliation effort;
- cost per unit accepted into production.
This follows the same principle as treating programming languages as agent harnesses: the decisive capability is the feedback surface around generation. It also preserves the distinction from Cursor's MiniSQLite result, where a held-out SQL suite provided semantic evidence but not database production readiness.
For full rewrites, keep intermediate baselines. The AI runtime rewrite ledger separates language, architecture, verification, and total cost so a migration cannot attribute every gain to code translation.
A minimum legacy migration evaluation contract
Before an autonomous migration starts, freeze these fields:
source:
repository_hash: ...
compiler_runtime: ...
dependencies: ...
target:
language_runtime: ...
allowed_architecture_changes: ...
behavior:
required_outputs: ...
state_invariants: ...
allowed_differences: ...
protected_tests: ...
operations:
performance_budgets: ...
failure_scenarios: ...
observability_requirements: ...
release:
shadow_window: ...
canary_scope: ...
rollback_trigger: ...
rollback_state_plan: ...
owners:
business_rules: ...
verification: ...
cutover: ...
The agent may choose implementation paths inside this contract. Changing an invariant, allowed difference, protected test, or rollback criterion requires a separately recorded decision. That keeps optimization from silently rewriting the definition of success.
Frequently asked questions
Can AI migrate legacy code?
AI can accelerate repository analysis, test construction, translation, debugging, and documentation. Production migration still requires an independent behavior oracle, organization-specific validation, dependency testing, domain review, staged cutover, and a tested rollback path.
What should a legacy code migration evaluation measure?
It should measure functional outputs, state and transaction semantics, integrations, failure recovery, operational properties, human review effort, total cost, and rollback readiness against a frozen contract.
Is byte-for-byte parity enough for COBOL migration?
It is strong for outputs where exact bytes define correctness. It does not cover hidden state, external dependencies, performance, security, recovery, or intentionally allowed differences.
Why are low-resource programming languages harder for coding models?
They have less public code and documentation available for training and retrieval. In enterprises, important rules may also exist only in executable code, batch configuration, data layouts, and operator practice.
What is the biggest legacy migration pitfall?
Treating compilation or a single passing test suite as equivalent to system migration. The new code can pass the visible checks while changing an unmeasured business rule or operational dependency.