All summaries · 2026-09-24

SurgGraph: Quantitative Laparoscopic Video Understanding via Geometry-Grounded Scene Graphs

Jing-Ying Wang, Rosiana Natalie, Marquise D. Singleterry, et al.

arXiv:2609.25651 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-09-24

The one-paragraph version

SurgGraph is a system that watches gallbladder-removal surgery videos and turns what it sees into a measured map of what is touching what, what is covering what, what has been pulled apart, and what tools are doing. Instead of describing a scene in words like a vision-language model does, it computes numbers for each relationship — for example, how long the shared edge is where fat is stuck to the gallbladder, or how large the cleared gap is between two tiny tubes that must be separated before cutting. It does this without new training by combining two off-the-shelf seeing tools with simple, doctor-informed geometry rules. Those measured maps are then used in SurgGraphQA, a practice tool that finds good and tricky example frames, explains whether safety checklists are met, and draws labels directly on the images. In tests the maps beat a surgery-specific vision-language baseline, and 17 medical students scored much higher after practicing with the tool.

The problem they're solving, and why it matters

Surgical video is how trainees learn anatomy, tool handling, and judgment, but there is far too much video to label by hand. Computers need to understand these videos at the level surgeons think: not just "a grasper is present," but "is the fat mostly cleared off the gallbladder neck?" and "are the cystic duct and cystic artery separated enough to clip safely?"

Current approaches fall short in three ways, according to the paper. General vision-language models, meaning AI systems that take in images and produce text answers, lack surgical vocabulary and fine detail. Surgery-specific versions are better but still unreliable, cited at around 63% on surgical question answering. Task-specific trained models need large labeled datasets that surgery does not have, and do not transfer well. Earlier scene graphs, meaning structured lists of objects and their links, only captured coarse facts like "tool above tissue" or "tool applied to tissue," without measuring how much attachment or separation exists or how it changes as dissection progresses. Without quantities, you cannot retrieve a frame where exposure is almost but not quite enough, explain which safety criterion is unmet, or show a learner what "enough separation" looks like.

The key idea, in plain words

Separate seeing from measuring. Let neural foundation models do what they are good at — outlining every structure and estimating how far each pixel is from the camera — then use transparent, fixed geometry rules to turn those outlines and distances into clinical relationships with numbers attached.

Think of it like tracing a map, then using a ruler. First, the system traces all tissues and tools in each frame and builds a depth picture where brighter or darker means closer or farther. Then it asks doctor-grounded geometric questions: if two tissues share a border and the depth stays smooth across that border, that is physical sticking, which the paper calls attachment. If they share a border in the flat image but depth jumps suddenly, one is just in front of the other, which the paper calls occlusion. If two tubes have a clear pocket between them, the size of that pocket is separation. If a tool tip is near a tissue it is allowed to touch and at the same depth, that is contact and action.

Each result is stored as a four-part tuple, meaning a subject, a verb, an object, and a quantity. A concrete example from the paper is fat attached to gallbladder with value 0.73, where higher means more attached after normalization. Because every relationship is a continuous number tracked over time, progress like gradual fat clearing becomes a falling curve, not just a yes-or-no label.

How it actually works, step by step

The input is ordinary laparoscopic video of laparoscopic cholecystectomy, meaning keyhole gallbladder removal, the most common minimally invasive surgery. The learning goals the authors target are anatomy recognition, tool usage, adequate exposure, and the critical view of safety.

Step 1 — Perceive. For each frame, a segmentation model called SAM 3 produces pixel masks for 19 categories, meaning pixel-level outlines labeled by type, including fat, liver, intestine, gallbladder, gallbladder neck, cystic duct, cystic artery, cystic plate, blood, plus tools like grasper, hook, bipolar, clipper, scissors, and suction. A separate model called Video Depth Anything estimates a depth map, meaning an image where each pixel value encodes relative distance from the camera, normalized to zero to one.

Step 2 — Compute attachment versus occlusion. The system slides a small window along each tissue boundary to find shared edges, meaning border pixels where another tissue is very close. It then measures depth variance along that edge, meaning how bumpy depth is across the shared border. Low variance means the two tissues sit at the same depth and are truly stuck together, so it records attachment with value equal to shared edge length in pixels. High variance means a depth cliff, so one tissue is overlapping the other in view, and it records occlusion with the same length measure. The threshold separating them is set at 0.01 after inspecting an example frame where attached pairs were 0.0001 to 0.0071 and occluding pairs were 0.0215 to 0.0627.

Step 3 — Compute separation. To measure how skeletonized, meaning stripped clean and visibly distinct, the cystic duct and cystic artery are, the system combines their outlines, computes their joint convex hull, meaning the smallest convex shape enclosing both, subtracts the two tissues themselves, and keeps only the gap piece touching both. The pixel area of that in-between gap is the separation value. Larger means more clearly separated.

Step 4 — Detect tool actions. The tooltip, meaning the working end of the instrument, is assumed to be the deepest point on the tool mask because laparoscopic tools enter through the abdominal wall and point inward. The system draws a small circle around that tip, lists which valid target tissues for that tool fall inside — valid pairs come from a hand-built tool-action-target dictionary — then rejects candidates whose average depth differs too much from the tip depth. The threshold is 0.05. The remaining tissue with most overlap wins, producing for example grasper retracting gallbladder.

Step 5 — Normalize and store. Raw pixel lengths grow when the camera zooms in, so each value is first divided by the area of a stable reference structure in the same frame, like gallbladder area, then divided by its maximum across the whole video to scale to zero to one. All values are stored as a four-dimensional tensor, meaning a grid indexed by frame, relation type, subject type, and object type, with five channels for attachment, occlusion, separation, tool action, and area. Phase and safety decisions are then simple threshold rules over those numbers, with no learned parameters.

SurgGraphQA, the teaching demo, queries that tensor in three ways: exact-match retrieval that finds the frame closest to a desired composition; diverse-stage retrieval that groups frames by which requirements are met and samples one per stage to show progression and edge cases; criterion-level checklists that turn thresholds into checkmarks; and automatic labeling that paints attachment zones, gaps, and tool contacts onto images with low, medium, high extent levels.

What they showed — results, honestly, with limitations

On value accuracy, the paper reports tool-action-triplet recognition, meaning correctly naming instrument plus verb plus target, at 79.2% for SurgGraph on 1,129 frames of the CholecT50 test set, versus 31.3% for their re-trained version of SurgVLM, a surgery-specific vision-language model. They note SurgVLM had very high precision but very low recall, meaning it rarely guessed and mostly guessed common actions, while SurgGraph was balanced. For attachment versus occlusion, hand labeling 266 preparation-phase images gave 82.2% classification accuracy, and combined with an cited 92% segmentation accuracy this implies a lower-bound value correctness around 75.6%. For separation, overlap with expert Calot's triangle labels, measured as IoU or intersection-over-union, averaged 0.73 on 74 frames. Thresholds were fairly stable across a wide range around the chosen points.

On downstream reasoning, phase recognition reached 93.8% accuracy versus 91.0% for the re-trained baseline, but the gap in macro-F1, meaning average performance across rare and common phases alike, was much larger: 0.943 versus 0.529. Critical view of safety assessment reached 94.59% overall versus 79.4%, with individual criteria at 91.8%, 90.5%, and 77.0%. The authors explain overall can exceed individual scores because safety is conjunctive — correctly saying "not safe" can still misidentify which part failed — and that the third criterion suffers from rater disagreement in the benchmark itself.

On education, 17 medical students improved from 2.41 to 8.29 out of 10 on exposure and 3.12 to 9.53 on critical view of safety, both reported as statistically significant with very large effect sizes. Interviews praised extent labels, stage comparisons, diverse patients, checklists, and tool examples. Two resident surgeons rated exercises as accurate.

Limitations to keep in mind: the baseline is the authors' re-training, not released SurgVLM weights, so it is not a direct comparison to the original system. The user study has no comparison condition, so we cannot tell how much gain comes from scene graphs versus just practicing with any well-chosen images and feedback. Performance depends entirely on segmentation and depth quality, textures like inflammation are out of scope, each frame is processed independently with no temporal smoothing, tool use is limited to a fixed dictionary, and new cameras or procedures require 5-10 expert-labeled calibration frames and hand engineering. The paper is also unclear on some details like exact SAM 3 prompting and tracking, and all main results are on one operation.

Why this might matter to me

For spatial reasoning in LLMs and VLMs, this is a concrete case where embeddings that describe space in words fail at "how much," while a tiny symbolic program over depth and masks succeeds and stays auditable. It suggests a hybrid pattern: use VLMs for open-ended language, but ground quantitative spatial judgments in explicit geometry you can inspect.

For world models, the tensor of attachment, occlusion, separation, and contact over time is essentially a compact, interpretable world state — a surgical scene evolving along measurable curves like fat attachment falling and duct-artery separation rising. That is a useful alternative to latent video prediction: a state you can query, plot, retrieve by, and write rules over.

For text-to-CAD and parametric geometry, the handbook for adding a relation — define clinically, translate to boundary overlap, depth comparison, area ratio, convex-hull gap, or tip proximity, then normalize by a stable reference — reads like parametric sketching from text. Coverage as mask overlap ratio is directly analogous to CAD constraints, and the zoom-normalization problem mirrors scale invariance in CAD.

For 3D generation and scene representation, SurgGraph shows you can get a useful 3D-aware scene graph without full reconstruction, just monocular depth plus masks plus geometric primitives. Preserving numbers instead of thresholding to labels like "adjacent" keeps information for downstream uses, a design choice worth borrowing for general scene graphs and neural scene representations.

Terms worth knowing