Real Unreal Bench

Every task is a real change in a complete, private, licensed, human-made Unreal Engine 5 game. We call the tasks large multi-system changes, by the size of each recorded fix and its checks.

The benchmark draws on our private collection of 1,000+ games. The task pool by genre, as a share of its tasks:

We picked this version's tasks from a much larger private pool before any model ran, and fixed the checks and the settings in advance. The pick used one earlier validation run of each task with Codex: tasks it didn't solve, and the ones it took longest to solve. Every game builds and plays on Linux.

The tasks come from eight games in eight genres. The rest of the pool stays private, held out for later versions.

A median task changes 46 game files, but one average hides what matters here: the smallest fix touches 7 files and the largest 1091. Tasks span up to 5 systems. Those counts include generated and re-saved assets.

Each run gets its own Linux VM with one L4 GPU and a working copy of the game. The agent can read and edit the project, build it, launch it and play it, the way a human developer would. It can't see the reference solution, and it's told each task's milestones in words but not how they're measured. Nothing it leaves running on its VM survives into grading.

When a run stops, we throw away the agent's container. A clean copy of the project at its final commit gets our own probe plugin put back, and then it's built and played in a fresh container. The checks read what happens in that play: whether a thing appears, moves, responds, triggers or ends the way the request asked.

So an agent can't talk its way to a pass. Nothing it wrote about its work is read, and nothing it changed in the probes survives the clean copy. A game that doesn't build fails everything that needs it to play.

Each task has 8 to 45 required checks: the build, the start, and measurements of the played game. A task passes only when all of them pass. That is the number on the leaderboard, and it's deliberately unforgiving: nine of ten checks passed is a failed request.

Partial credit is the second number, on the heatmap, so a model that gets most of the way shows it. It counts only the checks the untouched start fails: partial = (checks passed − checks the start passes) ÷ (checks − checks the start passes), floored at 0.

Doing nothing scores 0. Say a task has eight required checks and the start already passes two of them. A run that passes five earns (5 − 2) ÷ (8 − 2) = 50%, and it does not pass.

Before this version ran, every check was tried against real validation runs. Each time one judged a run wrongly, the check was fixed, never the request. These are the rules that came out of that work, and the checks are held to them.

A fix goes in the check, never in the request. A fix can make grading more accurate. It can't make a task easier.

Results, not methods. A check doesn't look for the reference solution's names, positions, fonts or kinds of parts. Any way of producing the asked-for result passes, and each result is read one way however it was made: whether something is lit, whether it changed, what's on screen.

Thresholds are calibrated. A threshold sits between the untouched start and the reference, with room for run-to-run noise. It's never the reference's own number.

The untouched start scores 0, and the reference passes every required check. A check the start already passes only guards that existing features still work, and it carries no weight.

No luck. The same project gets the same result every time, on any machine.

A failed check is the run's failure. A check is called unreadable only on independent evidence, such as frames or logs, from a reviewer who hadn't seen it.

Every task has a reference solution, so it's shown to be solvable before it's scored.

Required checks are deterministic code, not opinions. Where a task also carries a model's review, it's advisory: it never decides a pass and it isn't in the partial score. The reason is simple. A judge that reads the output can be charmed by it, and a running game can't.

Two scripted agents ran the same tasks through the same grading. One changes nothing and passes 0 of 15 tasks. One applies the reference solution and passes 15 of 15 tasks.

One run per model per task, 120 runs in all. A model's own result counts against it: a bad edit, a broken build, the budget running out, a refusal. Another one run was lost to our infrastructure, a VM or a capacity outage. One of those ran again, and none counts against a model.