The one-paragraph version
HapticWorld is a learned robot simulator you can both see and feel. It starts from the Interactive World Simulator, a video-prediction model that lets a person teleoperate a robot inside generated video instead of in the physical lab, and adds a new sense: predicted joint effort. While you move a leader arm, the model generates the next camera image of what the follower robot would see, and at the same time predicts the twisting forces at the robot's motors that would result from contact. That predicted effort is sent back to the motors in your hand controller, so pushing a heavy kettlebell feels heavier than pushing a light one, and pressing an eraser into a whiteboard feels different from waving in air. The authors show this touch channel matters: people collect demonstrations faster with it, the demonstrations carry force labels good enough to train force-aware policies that work on a real robot almost as well as real-world data, and those policies can then be tested inside the simulator with results very close to real-world testing.
The problem they're solving, and why it matters
Contact-rich manipulation means tasks where success is decided by touch, not just by what things look like. Opening a microwave door, wiping a board clean, or pivoting a box upright against a stop all look almost identical whether you are pressing just enough or much too little — a slipping grasp, a hovering eraser, and a box about to fall back can be visually indistinguishable from a good attempt.
Robot policies that also observe force do much better on such tasks, but they need training examples that include force. Getting those examples at scale is hard, and the paper lays out why each current route falls short:
First, real-robot collection captures true forces but every example is tied to physical hardware. Someone must reset the scene after every trial, rebuild setups when switching tasks, and risk breaking the robot or objects during hard contact.
Second, physics engines remove the hardware but require building object models and tuning contact settings for each scene. More importantly for this work, the paper argues that contact forces reported by those engines carry systematic artifacts from simplified contact solvers and do not match real measurements.
Third, learned world simulators — meaning neural networks trained on real images and actions to predict future video given your commands — fix many logistics: resets are instant, setups live inside the model, and hard contact damages nothing. But existing versions are vision-only. The operator feels nothing, so they can unknowingly command motions that would be blocked by contact in reality, drifting outside what the model was trained on and corrupting both the video and the recorded actions. The saved data also has no force labels, so it cannot train or even test a force-aware policy.
HapticWorld aims to keep the convenience of a learned simulator while restoring the missing touch loop.
The key idea, in plain words
Train one shared dynamics model to imagine both what you will see next and what you would feel, and read the feeling directly out of the seeing machinery.
Instead of building a separate force predictor that looks at images and actions on its own, HapticWorld taps into the middle of its video-prediction network. The intuition, in words, is that to predict the next frame correctly during contact — will the door move, will the box rotate or slip — the network must already figure out the contact situation: where the robot is, what it was commanded to do, and what it is touching. That same contact understanding is exactly what determines effort, so a tiny extra head can translate those internal features into joint torque.
Two design choices make this work. One is to condition on commanded joint position, meaning where the operator tells the robot to go, rather than measured joint position, meaning where the robot actually is. Concrete example: when you press an eraser into a board, the real arm stops at the surface while your command keeps going below the surface. The image barely changes as you press harder, so measured position looks the same for very different forces. The gap between command and reality disambiguates how hard you are pushing.
The other is to render that predicted effort back to the human in real time. Feeling resistance keeps the operator inside realistic behavior — they stop pushing when it feels blocked, just as they would on the real robot — which keeps the simulator on distribution and produces better data.
How it actually works, step by step
Data comes in two kinds. Play data means task-free exploration on the real robot: push the box in random directions, start a pivot and drop it halfway, grasp and release the microwave handle, wipe in different directions or miss the board entirely, wave in free space. During play the system records camera images, commanded joint positions for seven arm joints plus the gripper, and measured joint torques read directly from the arm motors with no extra sensors. Demonstration data means complete, successful task runs, but these are later collected inside the simulator, not on the real robot.
Model training happens in two stages. First, an image compressor learns to squeeze each camera image into a compact latent code, meaning a small numerical summary, and rebuild it. Second, that compressor is frozen and a dynamics model learns to predict the next latent code given a window of past codes and commands. That dynamics model is a consistency model, meaning a type of generative model trained to clean up a fully noisy future frame conditioned on history, actions, and noise level. Architecturally it is a stack of 3D convolutional blocks with action conditioning and attention over space and time.
Torque prediction is a readout, meaning a small add-on that reuses existing features. At each step the system takes the widest internal feature map from the middle of the dynamics stack, averages it over space, combines it with an embedding that tells the head how noisy the current frame is, and passes it through a small neural network to output eight torque values. The add-on is tiny — 181,000 parameters, about half a percent of the whole model — and is trained jointly with the video loss using a simple mean-squared error on torque at every frame.
Interaction runs autoregressively, meaning step by step in a loop at 10 times per second. Your newest command enters a sliding window, the dynamics model generates the next latent code, a decoder turns it into an image for display, and the torque head produces a prediction from the same forward pass, so touch adds no extra inference delay. Because the prediction includes gravity holding up the arm itself, not just contact, the system subtracts a model-based gravity term, then filters and clamps the remainder for safety before driving the motors of the leader device on all eight axes. Feeling can be switched off while prediction keeps running, which the authors use to isolate the effect of touch.
What they showed — results, honestly, with limitations
The evaluation uses three tabletop tasks: open the microwave door past 45 degrees, erase all visible marks from a whiteboard, and pivot a box upright to 90 degrees against a fixture. Each task gets 500 play episodes for training plus 50 for validation, about 20 seconds each, with training taking about two days on eight NVIDIA A100 graphics processors. Policies are ACT policies, meaning a popular imitation-learning method that maps observations to action chunks, trained on 50 demonstrations per condition and tested for 20 trials in both simulation and reality.
Torque accuracy comes first, since everything else depends on it. On held-out real episodes, the average prediction error, called RMSE for root-mean-square error, is 0.38 Newton-meters for microwave, 0.26 for wiping, and 0.36 for box flipping. Relative to each joint's peak effort in the same episode, that is 4.8%, 3.6%, and 3.5% average error. In plain terms, predicted traces closely track measured ones.
Touch improves collection. In 10 minutes of teleoperation inside HapticWorld, operators collected 26 versus 21 demos for microwave, 19 versus 8 for wiping, and 22 versus 14 for box flipping with versus without feedback — speedups of 1.2 times, 2.4 times, and 1.6 times. Quality matters more than count: identical force-aware policies trained on 50 demos collected with versus without feeling scored 19 of 20 versus 14 of 20 on microwave, 19 of 20 versus 5 of 20 on wiping, and 16 of 20 versus 6 of 20 in the real world. The paper's explanation is that without feeling, operators both under-push, leaving grasps loose or erasers hovering, and over-push, commanding poses that real contact would have blocked and that the model never saw during training.
Simulator data nearly matches real data. Force-aware policies trained on HapticWorld demos reached 19 of 20, 19 of 20, and 16 of 20 across the three tasks, totaling 54 of 60, versus 19 of 20, 20 of 20, and 17 of 20 for policies trained on real-world demos, totaling 56 of 60. The vision-only baseline — collect in a vision-only simulator and train a vision-only policy — managed only 10 of 20, 1 of 20, and 8 of 20, totaling 19 of 60. Typical vision-only failures were slipping off the handle, leaving marks, and dropping the box back.
The simulator also works as a test bench. Force-aware policies cannot even be run in a vision-only simulator because their required force inputs do not exist. In HapticWorld, simulated success closely tracked real success, differing by at most three trials out of twenty: for example, box flipping policies scored 17 real versus 14 simulated and 16 real versus 14 simulated depending on training source, while microwave and wiping matched exactly or within one trial.
Two extra domains are qualitative only: pushing 5, 10, and 15 pound kettlebells produces correspondingly larger resistance, and pulling a slingshot further increases both draw force and projectile distance into different scoring zones.
Limitations are stated clearly. The system handles slow, smooth contact but not fast impacts or sudden slips, limited by temporal resolution and the 10 Hertz feedback rate. It is trained per task, so each new scene needs its own play data and model. And because touch is read from the same pass that makes the image, errors in visual dynamics flow directly into what you feel. The paper does not claim generalization to unseen objects or scenes.
Why this might matter to me
For world models, this is a concrete step from video predictors toward evaluators you can trust for contact. The close match between in-simulator and real success suggests a path where policy iteration happens largely inside a learned model, with real-robot time reserved for final checks — but only if the model carries the non-visual channels the policy needs.
For spatial reasoning in language- and vision-language models, the lesson is that geometry alone underdetermines action. Whether an eraser touches a board or a box stays in frictional contact cannot be settled from a single image; it lives in the relation between commanded motion, measured resistance, and scene constraints over time. Adding an effort channel, even as an auxiliary training signal, could ground spatial predicates like pressing, seating, or slipping.
For text-to-CAD and parametric geometry plus 3D generation and scene representation, the gap is similar: current pipelines excel at shape and appearance but not at what happens when shapes touch under force. HapticWorld avoids explicit object assets entirely and learns contact behavior from play, which is scalable but per-scene and not parametric or editable. A natural bridge would be hybrid representations — keep explicit geometry and parameters for editability, but learn or distill contact-dependent dynamics and resistance on top, so a generated scene or CAD assembly can answer not just what it looks like but how hard it is to push, wipe, or assemble.
Terms worth knowing
- Joint torque: the twisting effort at each robot motor, here seven arm joints plus gripper. It reflects both gravity and contact with the world.
- World simulator: a neural model that predicts future observations given past observations and actions, used here to teleoperate and collect data inside generated video.
- Play data versus demonstration data: task-free exploration used to train the simulator, versus complete successful task runs used to train policies.
- Leader and follower: the hand controller you move and the robot, real or simulated, that follows your commands. Bilateral teleoperation sends force back to the leader.
- Force-conditioned or torque-conditioned policy: a robot behavior model whose inputs include force or torque in addition to images.
- ACT: Action Chunking Transformer, an imitation-learning policy used in the experiments.
- RMSE: root-mean-square error, an average prediction error magnitude. Lower means closer to measured torque.
- Latent code: a compressed numerical summary of an image that the dynamics model predicts in instead of raw pixels.
- Consistency model: a generative model trained to recover a clean frame from noise in few steps, used here for fast real-time prediction.
- FiLM conditioning: a way to modulate visual features based on another input, here the action commands.