arXiv subjects

Hecheng Wang

Publications and source records attributed to Hecheng Wang.

8 recordsLinked to original sources

arXiv2026

Exponential convergence of Sinkhorn algorithm for entropy martingale optimal transport

We prove the exponential convergence in relative entropy of the Sinkhorn algorithm for the entropy martingale optimal transport. We assume that the marginals have compact supports and are in strict convex order, the terminal marginal support is convex, and the initial marginal support lies in the relative interior of the terminal marginal support; the reference cost is assumed to be Lipschitz in each variable. Under these assumptions we establish uniform bounds on the dual variables modulo affine gauges. This result allows us to obtain the existence and uniqueness of the optimizer, and its exponential representation in terms of dual variables. We then establish a relative-entropy stability estimate for martingale couplings with different terminal marginals. Proving that the constant in the stability estimate stays uniform throughout the Sinkhorn iteration allows us to establish the exponential convergence of the Sinkhorn algorithm.

math.PR↗

arXiv2026

Exponential Convergence of the Sinkhorn Algorithm for the Schrödinger Bridge with Regime Switching

This paper studies the convergence of the Sinkhorn algorithm for the Schrödinger bridge problem with regime switching, as introduced in Zlotchevski and Chen (2025). We consider a class of regime-switching stochastic systems on the hybrid state space $E=\mathbb{R}^d\times\{1,\ldots,m\}$, and construct the Sinkhorn iteration through the associated equivalent entropic optimal transport formulation. The main result of this paper is the exponential convergence of the Sinkhorn algorithm in relative entropy under compactness assumptions. Our proofs are inspired by the arguments recently developed for proving exponential convergence of the classical Schrödinger problem in Chiarini, Conforti, Greco and Tamanini (2024), Eckstein (2025). We perform a similar analysis for the partially observed terminal setting, where only the marginal distribution of the continuous component is prescribed at the terminal time, while the discrete regime is unobserved.

math.PR↗

arXiv2026

VINE: Taming Generative Control Policies for Reinforcement Learning

Flow-matching policies have emerged as an effective policy parameterization for robot learning. They iteratively generate actions from noise, enabling highly expressive modeling of complex and multimodal action distributions. However, prior works observed that scaling these policies with value-gradient reinforcement learning (RL) often leads to training instability. Existing methods attribute this instability to iterative generation and therefore avoid end-to-end value-gradient optimization by sacrificing iterative generation, high expressiveness, or value-gradient optimization. Contrary to prior belief, we show the instability does not stem from iterative generation itself, but from the vanilla sampling strategy originally designed for behavior cloning, which becomes brittle under value-gradient RL. Motivated by this insight, we propose VINE, an RL-oriented sampling method that enables stable end-to-end value-gradient optimization for flow-matching policies. Instead of following a single flow trajectory, VINE reconstructs a new interpolation state at every denoising step, creating a stable differentiable path for value-gradient propagation while remaining compatible with the original flow-matching denoising process. As a result, VINE preserves the expressiveness and iterative generation of flow-matching without sacrificing end-to-end value-gradient optimization. Despite performing end-to-end backpropagation through all ten denoising steps, VINE achieves stable policy improvement and consistently outperforms state-of-the-art RL methods on the OGBench offline RL benchmark and real-world robotic manipulation task. Videos are available on our website: https://agibottech.github.io/vine.

cs.RO↗

arXiv2026

ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training

We study how to improve large foundation vision-language-action (VLA) systems through human-in-the-loop reinforcement learning (RL) in real-world environments. A key challenge is learning reliable value functions from heterogeneous real-world experience, as value estimation provides the primary learning signal for VLA training. In practice, replay buffers contain trajectories collected from historical policies, online rollouts, demonstrations, and intermittent human interventions. Because replay buffers mix trajectories generated by different behaviors, the observed returns can be mismatched with the quality of the current policy. Prior VLA post-training methods often rely on progress-style value signals, which reflect the average quality of historical behaviors, leading to mismatched learning signals for the current policy. In this paper, we propose ALOE, an off-policy evaluation framework whose value function directly evaluates current-policy behavior for each iteration. Specifically, ALOE combines chunked temporal-difference bootstrapping and conservative value aggregation to perform stable current-policy evaluation, then uses these estimates for advantage-weighted policy improvement. This design improves credit assignment to critical action chunks under sparse rewards and supports stable policy improvement. We evaluate ALOE on four real-world manipulation tasks encompassing long-horizon and high-precision scenarios: smartphone packing, laundry folding, multi-object sorting, and phone assembly. Across all tasks, ALOE outperforms other VLA post-training methods, highlighting the benefit of off-policy value estimates for real-world VLA post-training. Videos are available at our project website https://rooshy-yang.github.io/aloe.

cs.RO↗

arXiv2025

Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning

Recent advances in image reasoning methods, particularly "Thinking with Images", have demonstrated remarkable success in Multimodal Large Language Models (MLLMs); however, this dynamic reasoning paradigm has not yet been extended to video reasoning tasks. In this paper, we propose Video-Thinker, which empowers MLLMs to think with videos by autonomously leveraging their intrinsic "grounding" and "captioning" capabilities to generate reasoning clues throughout the inference process. To spark this capability, we construct Video-Thinker-10K, a curated dataset featuring autonomous tool usage within chain-of-thought reasoning sequences. Our training strategy begins with Supervised Fine-Tuning (SFT) to learn the reasoning format, followed by Group Relative Policy Optimization (GRPO) to strengthen this reasoning capability. Through this approach, Video-Thinker enables MLLMs to autonomously navigate grounding and captioning tasks for video reasoning, eliminating the need for constructing and calling external tools. Extensive experiments demonstrate that Video-Thinker achieves significant performance gains on both in-domain tasks and challenging out-of-domain video reasoning benchmarks, including Video-Holmes, CG-Bench-Reasoning, and VRBench. Our Video-Thinker-7B substantially outperforms existing baselines such as Video-R1 and establishes state-of-the-art performance among 7B-sized MLLMs.

cs.CV↗

arXiv2025

RoboRetriever: Single-Camera Robot Object Retrieval via Active and Interactive Perception with Dynamic Scene Graph

Humans effortlessly retrieve objects in cluttered, partially observable environments by combining visual reasoning, active viewpoint adjustment, and physical interaction-with only a single pair of eyes. In contrast, most existing robotic systems rely on carefully positioned fixed or multi-camera setups with complete scene visibility, which limits adaptability and incurs high hardware costs. We present \textbf{RoboRetriever}, a novel framework for real-world object retrieval that operates using only a \textbf{single} wrist-mounted RGB-D camera and free-form natural language instructions. RoboRetriever grounds visual observations to build and update a \textbf{dynamic hierarchical scene graph} that encodes object semantics, geometry, and inter-object relations over time. The supervisor module reasons over this memory and task instruction to infer the target object and coordinate an integrated action module combining \textbf{active perception}, \textbf{interactive perception}, and \textbf{manipulation}. To enable task-aware scene-grounded active perception, we introduce a novel visual prompting scheme that leverages large reasoning vision-language models to determine 6-DoF camera poses aligned with the semantic task goal and geometry scene context. We evaluate RoboRetriever on diverse real-world object retrieval tasks, including scenarios with human intervention, demonstrating strong adaptability and robustness in cluttered scenes with only one RGB-D camera.

cs.RO↗

arXiv2023

Hierarchical Visual Policy Learning for Long-Horizon Robot Manipulation in Densely Cluttered Scenes

In this work, we focus on addressing the long-horizon manipulation tasks in densely cluttered scenes. Such tasks require policies to effectively manage severe occlusions among objects and continually produce actions based on visual observations. We propose a vision-based Hierarchical policy for Cluttered-scene Long-horizon Manipulation (HCLM). It employs a high-level policy and three options to select and instantiate three parameterized action primitives: push, pick, and place. We first train the pick and place options by behavior cloning (BC). Subsequently, we use hierarchical reinforcement learning (HRL) to train the high-level policy and push option. During HRL, we propose a Spatially Extended Q-update (SEQ) to augment the updates for the push option and a Two-Stage Update Scheme (TSUS) to alleviate the non-stationary transition problem in updating the high-level policy. We demonstrate that HCLM significantly outperforms baseline methods in terms of success rate and efficiency in diverse tasks. We also highlight our method's ability to generalize to more cluttered environments with more additional blocks.

cs.RO↗

arXiv2022

A Benchmarking Dataset with 2440 Organic Molecules for Volume Distribution at Steady State

Background: The volume of distribution at steady state (VDss) is a fundamental pharmacokinetics (PK) property of drugs, which measures how effectively a drug molecule is distributed throughout the body. Along with the clearance (CL), it determines the half-life and, therefore, the drug dosing interval. However, the molecular data size limits the generalizability of the reported machine learning models. Objective: This study aims to provide a clean and comprehensive dataset for human VDss as the benchmarking data source, fostering and benefiting future predictive studies. Moreover, several predictive models were also built with machine learning regression algorithms. Methods: The dataset was curated from 13 publicly accessible data sources and the DrugBank database entirely from intravenous drug administration and then underwent extensive data cleaning. The molecular descriptors were calculated with Mordred, and feature selection was conducted for constructing predictive models. Five machine learning methods were used to build regression models, grid search was used to optimize hyperparameters, and ten-fold cross-validation was used to evaluate the model. Results: An enriched dataset of VDss (https://github.com/da-wen-er/VDss) was constructed with 2440 molecules. Among the prediction models, the LightGBM model was the most stable and had the best internal prediction ability with Q2 = 0.837, R2=0.814 and for the other four models, Q2 was higher than 0.79. Conclusions: To the best of our knowledge, this is the largest dataset for VDss, which can be used as the benchmark for computational studies of VDss. Moreover, the regression models reported within this study can be of use for pharmacokinetic related studies.

q-bio.QM↗