Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments

Xusen Hei; Jiali Chen; Jinyu Yang; Yi Cai

Home/Archive/arXiv

arXiv · 2611.02928

Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments

Xusen Hei·Jiali Chen·Jinyu Yang·Yi Cai

View original source ↗Read source document ↗Export metadata ↓

Abstract

Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.

Explore related subjects

cs.MM

Keep this discovery

Explore connections, maps & timelines

BibTeX RIS

Xusen Hei, Jiali Chen, Jinyu Yang, Yi Cai. 2026-11-02. Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments. https://arxiv.org/abs/2611.02928

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

LLMCodec: Adapting Video Codecs for Efficient Weight Compression of Large Language Models

The rapid development of large language models(LLMs) has led to remarkable advances in natural language processing. However, the increasing scale of these models introduces substantial challenges in terms of storage, transmission, and deployment. Though great efforts have been devoted to model compression and quantization, existing methods often rely on fine-tuning or calibration data, which exhibit limited generalization across different tensor types. In this paper, we argue that video codecs offer a promising solution for LLM compression, due to their inherent compatibility with matrix structured data, configurable compression strategies, and the availability of highly optimized, off-the-shelf implementations. Therefore, we present LLMCodec, a video codec-based LLM compression method that integrates affine quantization with the recent VVC/H.266 video codec. Beyond VVC, we further compare a range of video codecs and encoding profiles to evaluate their impact on compression performance. Experiments on different models demonstrate the robustness and generality of LLMCodec. Notably, on LLaMA-3-8B at 2-bit precision, LLMCodec reduces perplexity by over 1.5x and improves downstream task accuracy by 21% compared with the existing method.

cs.MM↗

arXiv2026

Cassette-Editor: An Agent Harness for Editing Video the Way You Do

Coding agents work in an environment built for them: a compiler or a test checks each change and points to the fault, and version control keeps every change in a project people can continue. Video editing agents lack one: an edit's correctness shows only in the costly rendered video, which does not point to the decision at fault, and existing systems offer either editing operations that cover only anticipated edits or a script whose output is a video the user cannot keep editing. We present Cassette-Editor, a complete agent harness for video editing that lets the agent edit the way a person does: knowing what the project allows, seeing what each change affects, and revising from there. The agent writes patches of typed changes to the editor's timeline document. A schema check and a rulebook shared with manual editing check each patch before it commits; a rejected patch leaves the project unchanged and returns each problem located with repair options, and an accepted one becomes a step of the history the user also edits. Reads map source analysis to the frames of the current edit. Because manual gestures are writes in the same language, folded by the same executor and rulebook, the agent can express every change a manual edit makes to the project. On 31 editing requests with the model held fixed, the complete harness passed 89% of trials against 47% for the same model with ten basic editing operations and without the rulebook, at less than half the agent time and cost per successful request; the loss fell on requests at rule boundaries, composite edits, and undo, not on finding content.

cs.MM↗

arXiv2026

MemoCare: An Interactive Multimodal Mobile System for Automated Cognitive Screening

MemoCare is a React Native mobile system for automated multimodal cognitive screening in English and Vietnamese. It integrates speech, temporal and spatial orientation, touchscreen interaction, and visuoconstruction. Google Speech-to-Text transcripts are scored with deterministic, task-specific rules. GPS-grounded current and historical administrative names support location scoring, touch events verify actions, and lightweight convolutional neural networks evaluate drawings. A separate model provides visual interpretation. Software verification passed 151/151 predefined language, spatial-answer, and touch-scoring regression cases and 48/48 fixed-coordinate spatial tests across four countries. Transcript-based tests exclude speech recognition accuracy. On a locked 71-image test set, validation-selected ShuffleNetV2 x1.5 achieved 91.33% mean balanced accuracy and 78.87% exact three-criterion accuracy. Four clinician co-authors rated the workflow (pooled median 4/5 across eight criteria; item medians 3-4.5), highlighting improvements needed in speech handling, response acceptance, and result transparency. The workflow automatically produces item-level and total scores.

cs.MM↗