The one-paragraph version
Solaris, from Runway, is a proposed new kind of software called an interface world model. A world model here means a model that generates what you would see next in a visual scene, frame by frame. Instead of building a website or app by writing code that describes buttons and behaviors in advance, Solaris generates the interface itself as video: you click or drag on the current image, and it synthesizes the next image showing what should have happened. It does this live at interactive speed by building on Runway's Gen-4.5 video model, adapted to generate one frame at a time from past frames, sped up with a technique that reduces the number of refinement steps, and stabilized by training on its own outputs. A separate language model — meaning a model that reasons in words — sits alongside it to interpret what you meant and to tell the visual model what kind of change to render. The paper shows demos like trying on clothes by dragging them onto yourself or rearranging a room, plus small user studies arguing that generated interactions look more natural than code recreated by a powerful language model, while openly listing hard limits around text, truthfulness, long sessions, and accessibility.
The problem they're solving, and why it matters
Today every interface goes through translation. A designer imagines a rich visual scene, but for it to run, someone must convert it into an intermediate representation, meaning a middle language like code in JavaScript or CSS that the computer can execute. The paper argues this translation is lossy in two ways. First, visual richness is lost. An image model can draw a beautiful storefront or product page, but once you rebuild it in code you simplify shadows, fabrics, reflections, and layout. Second, behavioral richness is lost. You can only use what a programmer anticipated and implemented: this button goes here, that menu opens that page. Software ships, in their words, as a frozen, compressed-down version of all the interactions you might have wanted. This matters for people too, and for agents. For people, tutorials and shops feel scripted: everyone gets the same pages in the same order. For AI agents that use computers, training on fixed coded layouts teaches them one specific hotel site or office app, so they fail when the layout changes slightly. The authors ask: what if the operating system did not serve fixed apps, but generated a fresh, responsive interface around your intent each time?
The key idea, in plain words
Make the whole video frame the interface, with no code underneath. Imagine a photo of a clothing store that is the store: you grab a red shoe from the rack and drop it on your own photo, and you are now wearing it. Or a living-room picture that is the design tool: you drag the lamp across the room, stretch the painting bigger, and the light and shadows update in place. To do that, Solaris splits the job into two roles. One decider figures out what should happen — did that drag mean "try this on," "move this," or "zoom in"? The other renderer figures out how it should look moment to moment. The decider is a language model that holds the session state and writes short descriptions of the desired behavior. The renderer is a visual world model that turns those descriptions plus your mouse moves into the next frames. Because meaning is described in words at run time rather than programmed in advance, the same gesture can do different things in different contexts. The paper's example: starting from the same hand photo, one drag zooms in to inspect the wrist, another drag measures the hand. Nothing about that had to be coded as a wrist-zoomer or hand-measurer ahead of time.
How it actually works, step by step
The paper describes the system at a high level, without architecture diagrams or training-data details, so some steps are conceptual rather than fully specified. 1. Start from an image. You provide a starting visual state, such as a brand room, a product scene, or a selfie in a store. Solaris begins streaming new frames from there. 2. Act with mouse and keyboard. Your clicks, drags, and typing are treated as a conditioning signal, meaning extra input the model looks at alongside text and images to decide what the next frame should show. Crucially, the model only sees past actions, never future ones, so it learns cause and effect: this kind of drag usually leads to that kind of visual change. 3. Reason about intent. A language model watches the session. It interprets requests like "move the table so I can see how it looks," decides whether to stay in the same scene and deform it or cut to a new scene, invents ambient life like shifting reflections, and writes prompts that steer the visual model. This is the separation of reasoning from rendering. 4. Render the next frame, autoregressively. Autoregressive here means one after another, each new frame depending only on frames that came before. That is different from standard video models that refine a whole clip at once over many denoising steps, meaning repeated clean-up passes starting from noise. Doing it frame-by-frame is what makes interaction possible. 5. Make it fast. The team says they converted the slow teacher model into a real-time engine in three stages: first teach it to generate sequentially, then distill it — meaning train a faster student model to mimic the slow teacher in just a few steps instead of dozens — then train that fast model on its own outputs so small glitches do not snowball over a long drag. The paper does not report frames per second, latency numbers, or model sizes, so the exact speed remains unclear. 6. Keep redefining the controls in words. Because interactions are specified by prompts, any object can become a tool. Click a cat and your next clicks paint with its fur color. Click a painting and you draw in its style. There are no fixed widgets, only described behaviors.
What they showed — results, honestly, with limitations
The paper offers demos, not a standard benchmark, plus two small evaluations. First, the cost of translation. They asked state-of-the-art multimodal models, including Claude Fable 5, to recreate 30 interfaces from a single screenshot by writing code — ranging from plain pages to image-heavy pages to natural images. They measured preservation in two ways: structural similarity, a pixel-by-pixel comparison of layout and appearance in place, and region matching with DINOv3 features, a way of checking whether the visual content survives somewhere even if it moved, using a vision model that describes image patches by what they show. In both measures, every language model lost information, and loss grew with visual complexity. Natural images were hardest. Solaris, by contrast, avoids this round-trip because it never leaves the image — though the paper shows this conceptually rather than with a head-to-head number for Solaris on the same metric. Second, Solaris versus coded interfaces. Starting from the same image and instruction, they compared Solaris output to a website coded by Claude Opus 5. With 250 participants judging 30 examples for about 7,500 pairwise choices, people preferred Solaris 62% of the time versus 25% for code for following the instruction, with 13% tied, and 72% versus 21% for behaving naturally in the scene, with 7% tied. The authors read this as: code can make the requested change, but treats it as an isolated patch, while the world model keeps materials, lighting, and physics coherent. The limits they admit are important. Stable legible text is still very hard for live video models, yet interfaces depend on text. Convincing but wrong outputs are dangerous for shopping or instruction; today grounding comes mainly from the starting frame and reference imagery, not verified live data. Coherence over very long, open-ended sessions still drifts. And a purely generated picture does not yet plug into screen readers or accessibility interfaces. Those are active research areas, not solved problems.
Why this might matter to me
If you care about spatial reasoning in language and vision models, Solaris reframes the test. It is not asking a model to describe left versus right in words; it is asking it to maintain space through action — keep a sofa's identity while you drag it, resize a plant without breaking the room, orbit a car and open its hood. That is an intuitive kind of spatial understanding, but it is appearance-based, not measured. The model may know how a move should look without knowing how far it moved in centimeters. That gap matters for world models more broadly. Here the world model is a renderer of plausible next views, not a simulator with persistent objects, forces, and constraints. It shines at ambient motion and direct manipulation, exactly where learned visual priors help. It does not give you what text-to-CAD and parametric geometry need: editability, precision, and history. In CAD, a wall is not pixels but parameters — length, thickness, relation to other walls — that you can query and change exactly. Solaris can show a wall changing color or a couch changing fabric beautifully, but you cannot select its edge and read a dimension or roll back a feature tree. Similarly for 3D generation and scene representation, Solaris suggests an opposite end of the spectrum from explicit meshes, point clouds, or radiance fields. Instead of building a lasting 3D scene you can rotate and measure, it generates a convincing view stream on demand. That is cheap and flexible for experience, expensive and slippery for truth. The interesting direction implied here is hybrid: use generated video for the living, personalized front end, but anchor it to verified references, product data, and eventually to explicit geometry when precision, persistence, or accessibility is required.
Terms worth knowing
Interface world model: the paper's term for a system that both understands intent and continuously renders an interactive visual world around it, rather than reproducing existing software. Intermediate representation: a middle format, like code, that a design must be translated into before it can run. The paper argues this step loses detail and freezes behavior. Autoregressive generation: creating a sequence one item at a time, where each new frame can only look at previous frames and past actions, which enables live response. Denoising steps and distillation: video diffusion models normally start from noise and clean it up over many passes. Distillation trains a faster model to get a similar result in just a few passes. Conditioning signal: any extra input — text, image, mouse drag — that guides what gets generated next. Structural similarity: a score for how closely two images match in place, pixel structure, brightness, and contrast. DINOv3 features: descriptive numbers from a vision model that capture what an image region shows, used here to check if content survived even after moving. Reasoning versus rendering: the paper's split where a language model decides what an action should mean and a visual model generates how that looks frame by frame.