All summaries · 2026-11-11

BIBIMBAP: A Benchmark for Instructional BIM-Based Automated Programming

Clemens Kujat, Bharathi Kannan Nithyanantham, Stefan Telgmann, et al.

arXiv:2611.09933 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-11-11

The one-paragraph version

This paper introduces BIBIMBAP, a new test suite for checking whether large language models can actually edit building models from plain English instructions. In this field, a building is not just a 3D shape but a Building Information Model, a structured database where every wall, door, room and floor knows what it is, where it sits, and how it connects to everything else. Until now, language models had mostly been tested on looking up information in those models, not on changing them. The authors created 100 small, carefully controlled tasks where a model is given a short instruction like “rotate this wall” or “tell me this area” plus a tiny example model, and then must write computer code to carry out the change. The code is run and checked automatically for whether the right object was made or deleted, whether it is in the right place with the right size, whether the answer is correct, and whether the underlying database was left consistent. When six current top models were tried in this setup, even the best one, Claude Opus 4.6, averaged only about 50.2% across all checks, doing okay on simple explicit instructions but failing often on geometry, connections between parts, math with areas and volumes, and keeping the model valid.

The problem they're solving, and why it matters

Architectural planning is iterative. A planner does not describe a whole house once and stop. They add a wall, move a door, delete a window, ask “how big is this room now,” undo, and try again, all while respecting functional needs and building rules.

A recent idea is to let a planner do that loop by talking to an AI assistant: the person types an instruction, a large language model writes a small program that edits the building database, the database updates, and the conversation continues. This is sometimes called Text-to-BIM, meaning text language in, Building Information Model out or modified.

The problem is there was no shared, rigorous way to tell whether models are any good at the editing part. Earlier tests mostly covered information retrieval, meaning asking questions like “how many doors are there,” often graded by asking another language model whether the answer looks good, which is expensive and unreliable. Other datasets tested recognizing single objects, rebuilding a model from a scan, or generating a model from a floor plan, but not the full range of everyday editing actions. End-to-end demos that generate a whole building from one prompt also do not tell you which basic step broke.

The authors argue you cannot trust a multi-step design assistant if you cannot trust its atomic steps. An atomic step here means one small, well-defined action, such as creating one wall, moving one element, deleting one door, or returning one quantity. Their goal is a reproducible exam focused on those atoms, so researchers can see exactly where models and their tool connections fail.

The key idea, in plain words

Instead of grading one big “make me a house” prompt, break the job into Create, Read, Update and Delete actions — together called CRUD — and test each one separately under different kinds of thinking.

Create means adding something new, like adding a wall 15 meters long at a given spot. Read means retrieving information, like returning the floor area of a room. Update means changing something that exists, like rotating a wall 45 degrees around its center. Delete means removing something, like deleting all doors on the first floor.

On top of that, vary how hard it is to figure out what to act on. Sometimes the target is named directly by identifier or coordinates. Sometimes it has to be inferred from shape, from position relative to other things, from how parts connect together, from a calculation, or from an abstract architectural idea like “corridor” or “entrance.”

Each task then gets an automatic checker, not a human judge. For edits, the checker looks at the before and after versions of the model file. For questions, it compares the model’s structured answer to a known correct value. That makes the score objective and gives a diagnosis like “you made the object but put it in the wrong place and broke a connection.”

How it actually works, step by step

The input standard for building models in this work is Industry Foundation Classes, usually shortened to IFC. Think of IFC as an open, vendor-neutral file format for buildings. It stores not only 3D geometry but also meaning: this thing is a wall, it belongs to this storey, it contains this opening, that opening is filled by this door. Most commercial design tools can import and export it.

Here is the flow for one task:

  1. Start with a tiny world. The benchmark supplies a task-specific IFC file that contains only what is needed for that task, but with deliberate distractors. For example, a retrieval task might include several similar-looking rooms so the model cannot just guess the first one.

  2. Give a standardized instruction. Every prompt is built from the same grammar. An operator says what to do, like create, rotate, or provide. An operand says what building part or information is wanted, like door or net floor area. Operation specifiers add constraints, like with size 2 by 1 meters or by plus 45 degrees. Context specifiers say which context matters, either explicitly, like with a specific identifier, or implicitly, like which is parallel to another wall or which is enclosed by other walls. Swapping that last piece turns the same basic action from a direct task into a spatial or topological puzzle.

  3. Let the model write and run code. In the baseline setup, the model does not edit IFC text by hand. It is given one tool, a function that runs Python code using IfcOpenShell, a widely used open-source library for reading and changing IFC files. The model can call that tool up to 25 times per task. A typical pattern is: first call to inspect — list walls, check directions and positions — then a later call to apply the change and save it. For questions, the model must also return its answer in a fixed structured format so it can be compared exactly.

  4. Check automatically. Create, Update and Delete tasks are checked by Python test scripts that compare the original and edited files. Retrieval tasks compare the structured output to ground truth. The overall score for a task is described in words as the fraction of relevant checks passed, averaged over all 100 tasks for a final score. The checks used are: required object exists after creation, referenced object no longer exists after deletion, correct location for Create and Update, correct dimension for Create and Update, integrity assured for Create, Update and Delete, and correct answer for retrieval. Integrity here means task-specific consistency: if you insert a door you must also create a hole in the wall that the door fills, if you delete a door you must also delete its hole and clean up relationships like containment in a storey, you must not move other things, clash parts together, or break the floor-by-floor hierarchy.

The full set is 100 tasks: 20 direct tasks, 20 spatial-geometrical reasoning tasks, 20 topological reasoning tasks, 20 numeric-calculative tasks, and 20 conceptual tasks. The first three groups each contain five tasks for each of Create, Read, Update and Delete. The last two groups are retrieval only in this version, in part because defining checkable numeric and conceptual answers was itself part of the contribution.

What they showed — results, honestly, with limitations

No model did well overall. Claude Opus 4.6 was best at about 50.2% average, followed by GPT-5.4 at about 31.3%, Gemini 3 Flash at about 30.8%, GPT-5.2 at about 30.5%, GPT-5 Mini at 25.0%, and GPT-4.1 at 23.9%. The top three were run three times and were fairly consistent across runs, except Gemini showed larger variation on direct tasks.

The pattern by thinking type is clear. Direct tasks with explicit identifiers and numbers were easiest, around 65.8% for the best model. Spatial-geometrical tasks dropped to about 63.9% for the best model but around 25 to 42% for others. Topological tasks were harder still. Numeric-calculative tasks that require pulling measurements, choosing net versus gross values, subtracting openings, handling units, and summing across elements were very hard for everyone: the best model scored only about 38.3% and several models scored near zero. Conceptual tasks like identifying a corridor from proportion and connectivity or an entrance from layout rather than a label were also very hard, around 30% or less.

By action type, Create looked easiest for the best model at about 81.1%, but that partly reflects simple objects with explicit parameters. Update required precise geometry. Delete looked deceptively easy: models often removed the main object but failed integrity because they left behind the opening, the fill relationship, or the containment link. Read was hardest overall because it demands both finding the right elements and computing and formatting the right value.

Across detailed checks, making or removing an object was easier than placing it correctly, sizing it correctly, answering correctly, and especially keeping integrity. The integrity check was the lowest for all models. Tool use also differed: Claude averaged about 4.9 tool calls per task, Gemini about 3.5, and GPT-5.4 only about 1.67, suggesting the winner iterated more to inspect before acting.

Two examples are telling. In a successful spatial case — rotate the wall that is parallel to a referenced wall by 45 degrees around its center — the model extracted directions, found the matching wall, computed its center, and applied the rotation over six tool calls. In a failure case — provide the identifier of the door that is perceived as the entrance — GPT-5.4 looked for a keyword like entrance, found none, and simply picked the first door, which happened to be correct by chance without real reasoning. The paper also notes wrong-unit errors despite explicit units. One detail is unclear: the text labels that successful rotation example as Claude Sonnet 4.5 while the caption says Claude Opus 4.6, so it is not certain which model variant is meant.

The authors are candid about limits. One hundred tasks is small. Retrieval is overrepresented because two whole categories are retrieval-only. The tiny, clean test models do not reflect large, noisy real projects with many storeys and element types. Only single atomic instructions are tested, not multi-turn dialogue like “undo that” or “now make it bigger.” The checks cover hard existence, place, size and relationship rules but not naming conventions, and an Update that deletes and rebuilds an object could pass even if non-geometric properties were lost.

Why this might matter to me

If you care about spatial reasoning in language and vision models, this is a sharp stress test beyond answering “what is left of what.” The model must turn phrases like between, next to, parallel to, enclosed by, or not a perfect square into selections and placements, then compute centers, directions, perimeters, and bounding-box sizes from actual geometry. The low spatial and topological scores suggest current models lack a stable internal map of the scene and often guess rather than construct one.

If you care about world models, think of IFC as a symbolic world model with strong constraints: containment in buildings and storeys, voids filled by doors and windows, walls that must meet to enclose a room. This benchmark shows that generating plausible code is not enough; the agent must preserve world consistency across actions. Integrity failures — leaving dangling openings, broken containments, or clashes — are exactly the kind of world-model violation that matters for robotics and simulation too.

If you care about text-to-CAD and parametric geometry, the pipeline here is instructive. The model never sculpts mesh vertices directly; it writes parametric API calls that create placements, orientations, extrusions, and relationships. Success requires correct parameter grounding, correct units and scales, and correct update of dependent features. The finding that models can create the object but get location, dimension, and dependent opening wrong mirrors common failure modes in text-to-CAD.

If you care about 3D generation and scene representation, the lesson is to evaluate edits, not just pretty outputs, and to check structure, not just appearance. Future scene systems that mix neural fields or meshes with structured graphs could borrow this idea: small atomic edit tasks, structured answers, and automatic checks for existence, pose, extent, and graph integrity, rather than relying on a second model to judge quality.

Terms worth knowing