SurgCBVR: A Multi-Domain Benchmark for Content-Based Surgical Video-to-Video Retrieval
Hospitals record thousands of hours of surgical video annually, yet these archives remain difficult to search by content. Surgeons cannot readily retrieve similar operations, complications, or workflow events. Existing surgical computer vision primarily produces labels rather than retrieving relevant videos. We introduce SurgCBVR, the first benchmark for content-based video-to-video retrieval in surgery. It integrates eight public datasets across four domains-ophthalmic, laparoscopic, robotic-assisted, and colonoscopy-covering six procedure types, 679 cases, and 76,489 clips (106 hours). SurgCBVR comprises two tiers: Scene, retrieving the same operative state across cases and sites, and Workflow, retrieving and localizing phase transitions and deviations. Scene includes 74,765 clips spanning 40 classes, three temporal granularities, and three playback speeds; Workflow contains 1,724 transition clips. Fixed clip lengths and frame counts, together with phase-classifier oracle and duration-only baselines, help assess label and timing shortcuts. We evaluate six generic and surgical foundation models, two video retrieval methods, and contrastive heads trained on SurgCBVR, using within-dataset, cross-dataset, and pooled-gallery retrieval restricted to the same procedure type. Frozen representations barely outperform a timing-only baseline, transition retrieval approaches chance, and trained heads lose up to half their accuracy across datasets. These findings expose substantial limitations in current video representations and establish SurgCBVR as a benchmark for advancing surgery-aware retrieval.