Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.19931

SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking

Tomáš Čížek·Matej Straka·Illia Volkov·Matěj Kripner·Ervin Macić·Jiri Matas·Klara Janouskova·Martin Schmid
View original source ↗Read source document ↗Export metadata ↓

Abstract

Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.

Explore related subjects

cs.CVcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tomáš Čížek, Matej Straka, Illia Volkov, Matěj Kripner, Ervin Macić, Jiri Matas, Klara Janouskova, Martin Schmid. 2026-10-14. SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking. https://arxiv.org/abs/2610.19931

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

A LoRA is Worth a Thousand Pictures

Low-Rank Adaptation (LoRA) has become a standard approach for parameter-efficient fine-tuning of image diffusion models, with over 200K models trained to replicate artistic styles and shared in online communities. One way to analyze styles of LoRA models is to generate image samples and apply image-based features like CLIP. However, this generation requires additional computation and prompt selection. In this work, we propose analyzing styles directly in the LoRA weight space, bypassing these limitations while achieving better performance even on unseen LoRAs. Since in-the-wild LoRAs lack ground truth model details, we design an experimental setup and show that LoRA weights alone serve as effective feature representations of adapted styles, robust to variations in fine-tuning seeds and image selection. We extensively evaluate using SD1.5 (the most popular base model within online communities) across data domains (fine-art paintings, architectural photos) and diverse prompt settings, with a small-scale SDXL experiment further verifying generalization. These evaluations confirm that our approach, under the selected aspects of variability, preserves fine-grained styles and outperforms image-based methods in clustering and retrieval of unseen LoRAs.

cs.CV↗
arXiv2026

Video Depth Anything: Consistent Depth Estimation for Super-Long Videos

Depth Anything has achieved remarkable success in monocular depth estimation with strong generalization ability. However, it suffers from temporal inconsistency in videos, hindering its practical applications. Various methods have been proposed to alleviate this issue by leveraging video generation models or introducing priors from optical flow and camera poses. Nonetheless, these methods are only applicable to short videos (< 10 seconds) and require a trade-off between quality and computational efficiency. We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos (over several minutes) without sacrificing efficiency. We base our model on Depth Anything V2 and replace its head with an efficient spatial-temporal head. We design a straightforward yet effective temporal consistency loss by constraining the temporal depth gradient, eliminating the need for additional geometric priors. The model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2. Moreover, a novel key-frame-based strategy is developed for long video inference. Experiments show that our model can be applied to arbitrarily long videos without compromising quality, consistency, or generalization ability. Comprehensive evaluations on multiple video benchmarks demonstrate that our approach sets a new state-of-the-art in zero-shot video depth estimation. We offer models of different scales to support a range of scenarios, with our smallest model capable of real-time performance at 30 FPS.

cs.CV↗
arXiv2026

Uncertainty Propagation for Echocardiography Clinical Metric Estimation via Contour Sampling

Echocardiography plays a fundamental role in the extraction of clinical parameters (e.g. left ventricular volume and ejection fraction) required to determine the presence and severity of heart-related conditions. When deploying automated techniques for computing these parameters, uncertainty estimation is highly valuable for assessing their utility. Since clinical parameters are usually derived from segmentation maps, there is no clear path for converting pixel-wise uncertainty values into uncertainty estimates in the downstream clinical metric calculation. In this work, we propose a novel uncertainty estimation method based on contouring rather than segmentation. Our method explicitly predicts contour location uncertainty from which contour samples can be drawn. Finally, the sampled contours can be used to propagate uncertainty to clinical metrics. Our proposed method not only provides accurate uncertainty estimations for the task of contouring but also for the downstream clinical metrics on one public and two private cardiac ultrasound datasets containing a total of 1264 patients. Code is available at: https://github.com/ThierryJudge/contouring-uncertainty.

cs.CV↗