The one-paragraph version
Code4Scene is a benchmark that asks AI coding agents to actually build and fix 3D worlds inside Unreal Engine, not just write code or generate a pretty picture. It contains 190 tasks made from human-built scenes: 30 open-ended construction tasks where the agent starts empty and follows a text description like “build an industrial harbor,” and 160 editing tasks where the agent gets a broken copy of a scene plus reference photos and must restore it exactly without breaking anything else. The submitted result is judged as a reopened engine file, checking whether the content is present and arranged correctly, whether objects are supported and not intersecting, and — for edits — whether every required fix was made with no side effects. Across 14 agent setups, the best overall score was only about 0.62 out of 1, spatial arrangement was weak for every agent, and even perfect-looking repairs often changed things they should not have touched.
The problem they're solving, and why it matters
Frontier agents can now write code that spawns objects, moves cameras, and screenshots a game engine. Demos look impressive, but a nice screenshot can hide serious errors: a chair floating in air, two walls sunk into each other, a missing dependency that means the scene will not reopen, or an edit that fixed one object but quietly moved ten others.
Existing benchmarks do not separate looking good from being correct. Some score rendered images, some score layout code or JSON, some run in different engines with different tools. That makes it hard to compare general coding agents and hard to know whether they understand 3D structure and can control persistent state.
The paper frames this as two complementary abilities. Construction means scene-level spatial reasoning: turning vague language such as “houses lining a canal” or “a dense town center surrounded by open farmland” into a coherent, physically plausible place, where many answers could be right. Editing means precise control of state: inferring a small change from images — for example, restore the missing globe — and applying it with tight accuracy while leaving a scene of 95 to about 6,700 objects otherwise untouched.
The key idea, in plain words
Instead of grading the agent’s code or its screenshots, grade the saved world itself after reopening it fresh.
Think of the world file, in Unreal called a dot-umap file, as the deliverable. The agent can use any allowed tools to search for furniture and buildings, run Python, place and move things, take pictures to check its work, and save. After time runs out, the evaluator throws away the chat history and opens only that saved file in a clean Unreal process.
That forces three separate questions. First, is the artifact usable — does it load, contain enough actors, and resolve all the assets it refers to? Second, did it do the task — does it satisfy the description, or for editing, match the hidden original? Third, is it physically sane when standing still — are things supported from below and not sunk into each other? A separate navigation test then asks whether a person-sized capsule can actually walk to objects and complete routes.
How it actually works, step by step
Tasks. All tasks start from human-assembled scenes bought from the Fab marketplace that allow AI use. For construction, a vision-language model drafts a description from screenshots, then two human reviewers correct it. The agent sees only the text plus a catalog of allowed assets. Each prompt averages 58 checkable requirements and up to 92. For editing, the authors copy a human scene and deliberately corrupt it with one or more small operations: removal, insertion, duplication, replacement, moving, rotating, rescaling, or mirroring. Indoor edits give two reference views of the original; outdoor edits give one. The original scene, extra views, and exact object states are hidden from the agent.
Reviewers who did not write the tasks check that every task is solvable from what the agent is given. The full 190 cases are split into a 95-case public set for development and a 95-case private set to discourage tuning to the test. The paper reports the public set has 20 construction, 25 indoor-edit, and 50 outdoor-edit cases.
Agent interface. Every agent uses the same 25 tools: searching assets and getting sizes and previews, spawning or deleting actors, setting position and orientation, running editor Python, validating and solving a layout plan, taking screenshots, measuring support gaps and collisions, setting up lighting and ground, and saving. Construction allows up to 800 tool calls and 180 minutes; indoor editing allows 120 calls and 45 minutes; outdoor editing allows 80 calls and 20 minutes. Only the first valid save counts; if there is no valid save the case scores zero.
Evaluation. Every case first faces Candidate Integrity, meaning validity checks that the file reopens, meets a minimum object count, and all referenced packages resolve. Then comes Physical Safety, meaning static checks for support and penetration. Support here means resting on the ground or on another object with at most a 5-centimeter gap. Penetration here means solid objects overlapping beyond a tolerance, roughly 5 centimeters for edits and for construction 5% of the object’s shortest size clamped between 5 and 50 centimeters.
Construction then uses a Semantic Verifier, meaning checks of meaning against the prompt. Detailed Alignment checks individual atomic requirements in four families: Identity and Environment for what kind of place it is, Content and Quantity for what and how many things appear, Attributes and Materials for color, material and style like “weathered facade” or “Victorian-style,” and Spatial Composition for relations like along, between, rows, density, or openness. Some facts can be read from engine state, but most need targeted photos judged by a fixed vision-language judge. Overview Alignment scores four overview photos as a whole, first for how well they match the prompt in alignment, layout, style, and completeness, then discounted if the geometry itself looks stretched, torn, or collapsed.
Editing uses a Ground-Truth-Based Verifier, meaning direct comparison to the hidden original. Repair Recall means what share of required fixes were recovered. Repair Precision means what share of the agent’s counted changes were correct and left unrelated content alone. Their combination, called Repair F1, is high only when the agent both finds everything and adds nothing extra. A restored object must match asset type and be within 5 centimeters in position, 5 degrees in rotation, and 5% in size and bounding-box extent. There is no partial credit for almost fixing an object.
Scores are combined with fixed weights. A construction case gives 20% to detailed requirements, 60% to overall impression, and 20% to physics. An editing case gives 80% to Repair F1 and 20% to physics. The model’s overall score is half construction and half editing.
What they showed — results, honestly, with limitations
Construction and editing correlate but are not the same skill. Across the 14 public setups, the rank correlation between the two settings was a Spearman value of 0.78. Claude Fable 5.1 led construction at 0.788, Gemini 3.8 Flash led editing at 0.581, and GPT-6 Astra led overall at 0.619, just ahead of Gemini at 0.619. The best Repair F1 was only 0.527 by Gemini, so no agent is close to reliable precise editing.
The paper’s clearest finding is that making the right objects is easier than placing them rightly. Spatial Composition was the lowest-scoring detailed family for all 14 agents; even the best, Astra, reached only 0.543. In a focused check where reviewers first confirmed the relevant objects were present, 76 of 369 spatial requirements, or 20.6%, were still judged mismatched. In other words, presence does not imply correct arrangement.
Editing shows different strengths hidden by a single number. Astra led Transform-type repairs that move, rotate, or rescale, while Gemini led Lifecycle operations like adding or removing and Layout rearrangements, and Fable narrowly led Symmetry on only eight cases. More importantly, finishing the fix did not mean a clean fix: across all setups, 76 of 212 complete recoveries, or 35.8%, still contained unintended additions or changes elsewhere. Astra fully recovered 23 indoor cases but only 16 with zero false changes; Gemini recovered 19 but kept 17 clean. The authors show this gap persists under stricter and looser distance tolerances, so it is not just an artifact of the 5-centimeter rule.
The auxiliary navigation score, called Embodied Utility Score, measures whether a fixed walking controller can reach floor-level objects and complete routes. After outdoor edits, Gemini improved over the broken input by 0.059, from 0.264 to 0.323, while seven other agents changed between minus 0.031 and plus 0.014.
Limitations matter here and the authors state many plainly. The benchmark covers static scenes only, no cloth, fluids, or articulated mechanisms, and support is a bounding-box contact proxy, not real stability. Each agent-case pair contributes one saved scene, so we see variation across scenes but not across repeated tries by the same agent. Human agreement was tested only for construction on 16 prompts and seven agents, with four prompts held out retrospectively, and reviewers saw only four overview images while the automatic score also uses engine data and close-ups. Editing was not validated against humans, and the 5-centimeter, 5-degree, 5% tolerances and 80-20 repair-physics weighting are design choices, not human-calibrated optima. Requirement lists were generated automatically with checks but no human review, so an error in the list can propagate into the score, and the public set approximates the full suite well on average but less well for agents unlike those used to select it.
Why this might matter to me
For spatial reasoning in language and vision models, this is a useful stress test that goes beyond naming objects. The conditional result — objects present but relations wrong one-fifth of the time — directly probes whether a model understands along, between, enclosing, rows, and density in 3D, not just in a 2D image. It suggests future work should train and evaluate layout planning, multi-view verification, and explicit relation checking, not just retrieval of the right assets.
For world models, the lesson is that visual plausibility is not simulation readiness. Code4Scene checks persistence, dependency resolution, support, and penetration from engine state, plus whether a walker can actually traverse the scene. That is closer to what a world model for robotics or games needs: a state you can reload, collide with, and navigate, where hidden interpenetration or floating objects would break downstream rollouts.
For text-to-CAD and parametric geometry, the editing track is especially relevant. Real CAD work is iterative repair under tight tolerances with preservation of everything else. The finding that 35.8% of complete recoveries add side effects, and that Transform versus Layout skills diverge across models, points to a need for constraint-aware editing, exact pose regression, and diff-and-verify loops that compare before and after states rather than judging one render.
For 3D generation and scene representation, the paper argues for engine-native, inspectable scenes over pixels or neural fields alone. A dot-umap file with addressable actors, measurable bounds, and collision bodies lets you ask actor-level who moved what, which image similarity cannot answer in a large scene. The open question it leaves is how to combine the strengths of generative priors for style and diversity with symbolic, physics-grounded control for placement and preservation.
Terms worth knowing
Coding agent configuration: the full system tested, meaning a model plus the client that calls tools, such as Claude Code or Codex, plus reasoning setting. A model name in the table is shorthand, not a pure model effect.
Dot-umap file: Unreal Engine’s saved map file. Here it is the persistent, reopenable scene the agent must submit.
Atomic corruption: a single deliberate break applied to make an editing task, such as removing, inserting, duplicating, replacing, moving, rotating, rescaling, or mirroring an object. Real cases can chain several.
Candidate Integrity: validity gate asking whether the saved scene reopens, has minimum content, and all asset dependencies resolve. Failure leaves later checks unevaluated and scores zero.
Physical Safety: static validity from engine measurements, combining the share of evaluated actors that are supported within 5 centimeters and the share free of confirmed solid overlap.
Detailed Alignment: requirement-by-requirement prompt fulfillment, grouped into Identity and Environment, Content and Quantity, Attributes and Materials, and Spatial Composition.
Overview Alignment: holistic judgment from four overview images, with a prompt-aware part for content and style and a prompt-free part that penalizes broken geometry like stretching or collapse.
Repair Recall, Precision, and F1: editing accuracy against hidden ground truth. Recall means fraction of required fixes recovered. Precision means fraction of counted outputs that are correct with no collateral damage. F1 is their harmonic combination, high only when both are high. True positives count fixed targets, false negatives count missed targets, false positives count wrong or extra changes.
Embodied Utility Score: auxiliary usability score combining Approachability, meaning share of floor-level objects connected to the main walkable area, and walk success rate of a fixed capsule follower on selected routes. Withheld means the navigation mesh or route could not be measured reliably, which is reported separately from a certified zero.