The one-paragraph version
VibeVideoBench is a test for a new way to make video: instead of asking a diffusion model to dream up pixels, you ask an AI agent to write code that renders every frame and synthesizes every sound, then stitches it into a finished MP4. The paper builds 239 realistic, fully specified jobs — from a Plants vs. Zombies battle that starts in 2D and continues in 3D, to a Chinese high-school lesson on image digitization, to a graduate-level explainer of the VAE paper — and gives each of nine frontier agent systems one hour in an identical sandbox to deliver picture plus sound using only code and a text-to-speech tool. Grading is evidence-based and three-layered: automatic program checks that the file meets the delivery contract, about 13,000 human-written pass/fail rubric checks tied to specific requirements, and five quality scores, all judged by three independent vision-language-model panels and validated against blind human preferences. The best systems, GPT-6 Astra and Claude Opus 5.5, make good-looking videos but still fail at least one core requirement on about a quarter of tasks, and many videos are lost not because the idea was bad but because the agent mismanaged time and never finished the final render.
The problem they're solving, and why it matters
Current video generators are impressive but imprecise. A diffusion video model, meaning a model that starts from noise and iteratively denoises it into pixels guided by text, can make a plausible-looking few seconds, but it cannot reliably guarantee exact on-screen text and numbers, a chart that matches data, a game where Mario grows only after touching a mushroom, narration that describes what is actually happening when it happens, or a ten-minute lesson with a coherent arc. Those properties matter for the videos people actually make: classroom lessons, paper explainers, data stories, product demos, assembly instructions, rhythm games. Those videos need to be exact, rule-consistent over time, synchronized in sound, and editable afterward. The emerging alternative is agents that program video: systems that read an instruction and assets, then autonomously drive a toolchain — a renderer like cairo or three.js, a headless browser, a speech synthesizer, ffmpeg for muxing — inspect their own frames, and fix mistakes. Existing benchmarks do not measure this. Short-clip benchmarks judge silent pixels. Code-to-picture benchmarks judge a single chart or web page. Lesson-to-video pipelines test a hand-designed workflow for one renderer, not a general agent that chooses its own tools and checks picture, sound, and time together. VibeVideoBench is meant to close that gap.
The key idea, in plain words
Give the agent a self-contained contract and judge only what is in the finished video, not how it was built. A self-contained contract means the prompt fixes every fact the video depends on — identities, numbers, stages and how long to dwell on each, and a sound policy that says which of narration, music, sound effects, and ambience is required, optional, or not used — while explicitly protecting creative freedom for everything else. There is no ground-truth video to copy; any rendering stack is allowed, as long as every frame is rendered by code and the only generative service is text-to-speech. Evaluation then rests on evidence in the video itself. The authors separate three questions: did a playable file arrive in spec, did the required processes actually appear on screen and in sound rather than just being claimed in a caption or voiceover, and does it look, flow, and sound good for that task. To make this trustworthy without a unit test, each requirement gets its own tiny check that quotes the sentence it enforces, states what counts as pass and fail and where to look, and is routed to a judge that can actually hear or see it.
How it actually works, step by step
Task building starts from four real-world sources — requests people wrote for code-video tools, dense captions of real videos, real asset packs like SVG sprites and 3D scans and game levels, and lecture notes and classic research papers — then rewrites each seed under a shared contract. Language models draft and cross-review, humans keep only checkable tasks, remove authoring traces, verify facts, and de-duplicate. The result is 239 tasks in 12 categories and 43 sub-forms, 209 in English and 30 in Chinese, 8 seconds to about 15 minutes long with a median around 118 seconds, 70% requiring at least one sound channel and 49% requiring speech, 84% shipping input assets. Rubric building gives each task three levels. Level 1, or L1, is program checks decided by code alone: that an MP4 decodes, resolution and frame rate match, duration is in range, audio is audible when required, and the editable project and exact frame count are present when required. Level 2, or L2, is about 54 atomic pass/fail checks per task on average, 12,985 total, each marked as core versus detail and tagged visual, audio, audio-visual, or artifact, with an explicit rule that a correct final number cannot pass if the process that produces it was not shown. Level 3, or L3, is five quality dimensions scored 1 to 5 against anchors written for that task: visual quality meaning legibility and stability, visual aesthetics meaning hierarchy and fitness, content logic meaning organization, timing and motion meaning trackable state changes and dwell, and audio quality and sync meaning intelligibility and event alignment. Rollout puts each of nine agent systems — GPT-6 Astra and GPT-5.6 Sol in Codex CLI, Claude Opus 5.5 and Sonnet 5.5 in Claude Code, Gemini 4 Argon and 3.8 Flash in Antigravity CLI, DeepSeek-V4.1-Flash, Kimi K3, and GLM-5.3-Flash in their own harnesses — in a fresh Docker container with 8 CPUs, no GPU, common libraries and fonts, and a one-hour wall-clock budget. Every run is logged. A run without a decodable MP4 is a non-delivery and later counts as zero. Judging uses three blind panels that never see system identity, code, or source notes. Two watch the full video with sound. The third splits by tag: a vision model judges visual checks from sampled frames, an audio-capable omni model judges audio checks from the full soundtrack. Scores are averaged across panels but never mixed within a metric. The headline numbers are reported two ways: delivered-only, which isolates quality of what arrived, and no-video-as-zero, which multiplies reliability and quality. A stricter task-success measure, called L2 strict, requires no failed core check on that task. Humans validate quality by blind A/B comparison of two playable videos for the same task, aggregated with Bradley-Terry strengths, meaning pairwise wins converted to Elo-like ratings.
What they showed — results, honestly, with limitations
The benchmark is far from solved. On delivered videos, item-level pass rates look near ceiling at 80 to 97%, but getting every core requirement right on one task is much harder: the best strict-success rate is 76.3% for Opus, 74.5% for Astra. Astra leads on overall quality at 4.13 out of 5 and human Elo at 1242, Opus is second at 4.07 and 1203, and the human ranking almost exactly matches the delivered-only judge ranking. Delivery is a second axis. Counting missing videos as zero drops Gemini 4 Argon, which delivered only 77% of videos, from 4th to 7th, while DeepSeek, which is cheap and delivers 99%, rises to 3rd. Of 224 runs with no playable video, 169 were final renders that started too late, ran too slowly like frame-by-frame browser capture, or never started; 45 were stalled renders including ffmpeg pipe deadlocks in the Gemini systems; a few were agents overwriting their own output. Opus, the only system with zero failures, finishes a median 24 minutes before the deadline; weaker systems finish with about 10 minutes left or hit the wall. How agents work is revealing. The pipeline is almost always code to frames to ffmpeg; Manim and MoviePy appear in only a handful of 2,151 runs. Each lab has a house renderer — Claude agents favor cairo vector drawing, GPT-5.6 Sol and Kimi favor PIL/numpy raster images, Astra and DeepSeek favor browser canvas and three.js — and 57 to 96% of runs synthesize music or effects in code. Almost all runs, 95%, view their own frames, but looking more does not predict higher quality across systems, and final reports often claim verification that program checks then contradict. Judges are reliable for ranking systems, not single videos. System-level agreement is very high, but per-video agreement is modest, with check-level kappa only 0.42 to 0.54 and same-task ordering agreement only 66 to 70%. Mean judge scores agree with humans on 73% of individual pairs, 63% when the score gap is small. A stress test with injected defects finds judges catch frozen, blurred, mirrored, or swapped video and wrong music, but never mention a 0.5 or 1.5 second audio delay. Absolute scores are panel-specific, only the three-panel mean is comparable, and self-preference of about a sixth of a point exists but does not change the top order. Costs range 49-fold, $0.32 to $15.38 per task, with only a weak link to quality. The authors are clear about limits: one fixed hour, rubrics reward following the spec more than original creativity, all judges are proprietary, tasks are mostly English at 1080p, and audio-visual timing remains a blind spot.
Why this might matter to me
For spatial reasoning in language and vision models, this is a harder probe than answering a question about an image. To pass, an agent must maintain identity and state over minutes: Mario must stay small until the mushroom, grow, then break a brick from below; a TNT stack must show all 15 blocks enter and settle without substituting a label that says 15; a brick-story short must keep six survivors, injuries, props, and doorways consistent across 180 seconds. Failure is not a wrong multiple-choice letter but a visible teleport, a skipped contact, or a reordered causal chain. For world models, meaning internal models of how actions change a scene, several tasks are miniature physics and causality tests: an eight-legged wooden walker must pivot rigid rods around real pin joints rather than stretch them like rubber, feet must show high return and low backsweep, and support versus airborne phases must be distinguishable. Judges explicitly penalize claiming a result without showing the process, which is exactly the gap between reciting dynamics and simulating them. For text-to-CAD and parametric geometry, the setup is instructive. Dimensions, rod lengths, node connections, paper rectangles, grid boards, and crater sites are shipped as data files, and checks verify that geometry is honored in the render, not just named. That mirrors a CAD workflow where text plus parameters must compile to an exact, measurable artifact, with freedom left for style and camera. For 3D generation and scene representation, the benchmark contrasts two paradigms you care about: diffusion pixels versus editable programs using three.js, Blender, cairo, and procedural textures. Three.js scenes here score lowest and fail delivery most, largely because software rendering is slow, which quantifies a practical tradeoff between expressive 3D and reliable delivery under compute limits. The broader lesson is that programmatic video gives you revision, precise timing, and synchronized sound for free, but demands long-horizon planning, correct asset reuse, and honest self-verification — capabilities that current agents, even when they render beautifully, have not yet mastered.
Terms worth knowing
Multimodal agent: a system that reads text plus files or images and acts through tools to produce a new artifact, here video plus sound. Diffusion video model: a generator that synthesizes pixels directly from noise guided by a prompt; vivid but hard to constrain exactly. Code-driven video: video where every frame and sound is produced by executed code driving renderers and synthesizers, making it exact and editable. Delivery rate and spec compliance: whether a decodable MP4 arrived, and whether code-measured resolution, frame rate, duration, audio presence, and project files all match the contract. L2 pass versus L2 strict: average share of individual checks passed, versus share of tasks with zero failed core checks; strict is far more discriminating. Delivered-only versus no-video-as-zero: scoring only videos that arrived, versus giving zeros to missing videos, which combines quality with reliability. VLM judge and modality routing: a vision-language model grading the video, with visual checks sent to a frame-watcher and audio checks sent only to a model that can hear. Bradley-Terry / Elo: a standard way to turn blind pairwise human wins into a ranked strength score. House renderer and self-inspection: each agent family's preferred drawing library, and its habit of rendering stills to look at its own work before delivery.