The one-paragraph version
This paper asks a new question about a popular recipe for robot control: when a language model writes the robot's controller as a program — what the authors call a coding agent — can it also keep the robot safe? The test pairs each ordinary pick-and-place goal, like put the bowl on the plate, with one forbidden object, like a mug the arm must never touch. Out of the box, the coding agent mostly finishes the job but hits the obstacle anyway, even though it correctly sees the obstacle and its prompt says not to touch it. The authors trace this to planning, not seeing or obeying instructions: the agent flies the shortest path and never really plans a detour or a careful grasp. Their fix, called SAFEHARNESS, adds two training-free helpers around the unchanged model and unchanged low-level robot policy. One helper plans and checks a detour before moving, the other chooses a safer grasp side when the obstacle sits next to the target. On the SafeLIBERO simulation benchmark, this reaches 71.9% task success and 87.5% collision avoidance, ahead of the prior best and more than double the same agent without the helpers.
The problem they're solving, and why it matters
A coding agent means a large language model does not output joint torques directly. Instead it writes and runs code that calls ready-made capabilities: look at the scene, move to a pose with inverse kinematics, recall a remembered skill, or hand control to a frozen learned policy for tricky contact. Frozen here means the low-level policy's weights are never updated. This recipe now solves many manipulation tasks with no robot-specific training, but prior work and benchmarks score only whether the goal was reached.
The authors argue that success without safety is incomplete. In a cluttered kitchen or workshop, brushing a wine bottle or moka pot can break hardware, ruin property, or hurt people. They formalize safe manipulation as two demands at once: achieve the goal predicate, meaning the logical test for done such as bowl-on-plate is true, and never displace the obstacle by more than 1 mm. They report the first demand as Task Success Rate, or the share of episodes that finish the goal, and the second as Collision Avoidance Rate, or the share with zero obstacle movement.
The striking finding is that the two demands do not move together. A frozen vision-language-action policy called pi0.5 gets 57.8% success but only 17.1% collision avoidance on average. Ten times more collision-free demonstrations, in cited prior work, barely helps. The paper shows the coding agent itself, given the safety rule in words, agrees with the rule and then violates it.
The key idea, in plain words
The key idea is that telling the model about safety in language is not the same as giving it a mechanism that enforces safety where motion is decided.
Think of each manipulation phase, for example go pick the juice carton, then go place it in the basket, as two different kinds of danger. First there is the route, the long sweep through free space where the arm could take a wide detour. Second there is the contact, the final centimeters of grasping or releasing where the gripper cannot leave the target and must approach from a smart side or angle.
The model fails differently in each part. On the route it has no notion of a clearing path, picks the shortest line, and has no habit of replanning if that line proves bad. At contact it does not realize the grasp itself needs to respect the obstacle. A stronger model alone does not fix this, because over a hundreds-of-steps episode the one sentence naming the obstacle gets buried in a long history of images and tool results and stops guiding behavior.
SAFEHARNESS therefore moves the constraint out of the long conversation and into small, stateless tools that run fresh at each phase. Stateless means the check does not depend on remembering step zero; it looks at the current boxes and waypoints and says pass or fail now. The planner is then only allowed to choose among options that already passed.
How it actually works, step by step
The system reuses the Harness VLA loop, the frozen pi0.5 checkpoint for contact-rich control, a skill memory of past successes, and a segmentation model called SAM-3 for grounding, all unchanged. Only the two harnesses are new, and both run once per phase, so a two-goal task runs them twice.
Obstacle-aware route planning handles free space. First, localize: segment the target and obstacle and represent each as a bounding box, meaning a simple rectangle around the object in the camera view used as a proxy for where it is. Second, plan: ask the model for a route as a polyline of waypoints, meaning a connected list of intermediate 3D positions ending at a pre-grasp pose above the target or a release pose above the destination, with the condition that it ends at the target and does not cross the obstacle box. Third, verify outside the language model: a separate check tests whether the proposed gripper path, plus the held object if carrying one, intrudes into the obstacle box. If it fails, reject and replan. Fourth, execute under guard: fly only the verified route, but halt and replan from the stopping pose if the arm gets too close or stuck on something the boxes did not capture, such as a kinematic limit. To avoid bloating the context window with repeated identical images, the system limits how often the route image can be re-read during execution.
Obstacle-aware contact execution handles the final grasp or release below that pre-grasp or release pose, which route verification does not examine. Each contact starts with an adjacency test, meaning is an obstacle right next to this grasp or place target? If no, use the default strategy as in an empty scene. If yes, approach from the direction farthest from the obstacle that still keeps clearance. If no direction clears, reorient the gripper so its opening axis runs tangent to the obstacle, meaning sideways along its face rather than into it, and lower vertically to avoid lateral motion toward the obstacle. This applies both at pickup and at placement.
What they showed — results, honestly, with limitations
Evaluation is on SafeLIBERO, which adds one obstacle object to 16 LIBERO tasks across four suites called Spatial, Goal, Object, and Long, for 32 obstacle-paired tasks total, 10 seeds per task. Level I puts the obstacle close to the target to stress grasping and releasing. Level II puts it away from the target but on the natural carry path to stress routing. Language never names the obstacle; the safety requirement comes from the task wrapper.
On averages, SAFEHARNESS with GPT-6 reaches 71.9% success and 87.5% collision avoidance, versus the prior best method AEGIS, a barrier-function safety filter over pi0.5, at 67.5% and 68.9%. Barrier-function here means a runtime layer that minimally edits each commanded action to stay inside a safe set. The frozen pi0.5 alone and OpenVLA-OFT score respectably on success but very poorly on safety. The paper reports OpenVLA-OFT at 26.2% success and 5.7% avoidance on average, though those per-suite avoidance numbers look inconsistent in the printed table, so that exact average should be treated cautiously.
Ablations isolate the harness. The same GPT-6 agent with skill memory and frozen VLA but no harness gets only about 31.0% success and 59.0% avoidance. The harness lifts it by about 41 and 29 points. A stronger planner alone helps local grasp-side choices in Level I but barely helps long routes in Level II, while the harness makes Level I and Level II avoidance similarly high under either GPT-5.5 or GPT-6. Qualitative traces show the unharnessed agent mentions the bottle early, then forgets it and flies straight through at grasp height.
Limitations are stated plainly. The benchmark declares one obstacle per scene and the method uses one axis-aligned box; multiple obstacles would need a richer planner. The contact test is tuned for a parallel-jaw gripper, the standard two-finger gripper. Grounding quality is inherited from SAM-3, though failures surface as rejected routes rather than collisions. Episodes cost minutes of planner time, evaluation is simulation-first, and about one episode in eight still collides.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is a clean demonstration that semantic recognition is not spatial planning. The agent correctly names the wine bottle and restates do not hit it, yet draws the shortest line through it. Language agreement does not produce a clearing route, verification, or replanning. If you study spatial benchmarks, the Level I versus Level II split is useful: stronger models improve the local question of which side to grasp from, which is legible in one view, but not the global question of holding a detour over a long motion.
For world models, the lesson is architectural. Instead of asking a large context to remember a rule for 300 steps, SAFEHARNESS uses a tiny explicit world model — boxes plus waypoint intersection tests plus halt-and-replan — as a stateless checker at decision time. That echoes findings that learned video-action models inherit VLA brittleness. A small verifiable model at the bottleneck can beat a bigger generative model left to remember.
For text-to-CAD and parametric geometry, the parallel is direct. A bounding box is the crudest parametric proxy for shape, and a waypoint polyline is the crudest parametric path. The paper succeeds with those, but also shows their ceiling: no orientation beyond axis-aligned, no swept volume, no multi-obstacle search. Replacing the box with a parametric solid or sketch constraint, and the polyline with a constrained curve that must stay outside that solid, is a natural next interface — and the paper's slogan that instructions are not constraints applies equally to CAD: a note saying avoid interference is weaker than a solver that rejects interfering geometry.
For 3D generation and scene representation, the work suggests where richer representations would plug in without retraining the policy. The harness interface is designed to admit a better planner behind the same verify-and-execute calls. A neural field, Gaussian splat, or depth-fused mesh that yields tighter than box clearance, or a generative model that proposes diverse detours for the verifier to rank, could keep the training-free property while reducing the timidity and step-budget cost of long detours that currently hurts Level II success.
Terms worth knowing
Coding agent: a system where a language model controls the robot by writing and executing code that calls perception and motion tools, rather than outputting motor commands directly. The program is the policy.
Harness: the set of tools, checks, memories, and low-level policies wrapped around the language model. All robot competence attributed to GPT-6 Astra here is claimed to belong to its harness.
Frozen VLA: a vision-language-action model whose weights are fixed and called only for hard contact segments, here pi0.5. Vision-language-action means it maps images plus language to actions.
Waypoint route: a path represented as an ordered list of intermediate positions the arm should pass through.
Bounding box grounding: locating an object by drawing a box around its segmented pixels and treating that box as its position for planning.
Pre-grasp / release pose: a safe hover position just above the grasp or place target where the route ends and contact begins.
Replanning: discarding the current route after a failed check or mid-execution halt and planning anew from the current pose.
Task Success Rate and Collision Avoidance Rate: the two SafeLIBERO scores — share of episodes where the goal predicate became true, and share where the obstacle never moved more than 1 mm.
SafeLIBERO Levels I and II: benchmark variants stressing contact, with the obstacle beside the target, versus routing, with the obstacle on the carry path.
AEGIS: prior best baseline that adds a control barrier-function filter downstream of pi0.5 to veto unsafe actions, rather than choosing a safer route or grasp up front.