Searcharxiv⌕ Search

arXiv subjects

Jiahao Ai

Publications and source records attributed to Jiahao Ai.

7 recordsLinked to original sources

WinDEX: Metric 3D Hand Motion Reconstruction in the Wild for Dexterous Teleoperation and Policy Learning

Human manipulation data are widely used to train robots for human-like dexterity. However, low-cost, large-scale collection in the wild remains challenging. Existing methods remain limited in geometrically grounded metric-scale reconstruction, real-world generalization and inference speed. We present WinDEX, integrating (1) geometric-temporal optimization for metric 3D hand annotation from in-the-wild egocentric stereo videos, combined with visual-inertial SLAM for world-frame reconstruction; (2) WinDEX-Net, an end-to-end, fisheye-native network combining calibrated stereo, local detail and temporal context, with a data-model flywheel of model-assisted annotation and retraining; (3) distilled models for offline reconstruction and causal streaming, supporting reinforcement learning and teleoperation; and (4) low-cost CyberCap stereo-inertial capture, forming the WinDEX-Hand dataset with 1,000 hours of training annotations and held-out human-annotated in-the-wild 2D and independent optical motion-capture evaluation sets. Offline reconstruction runs at 43.61 FPS and causal streaming at 171 FPS. On held-out data, offline reconstruction achieves state-of-the-art accuracy: 14.5 pixels in-the-wild reprojection error and, with visual-inertial odometry, 37.29 mm world-frame joint error. We achieve 7.6 and 3.6 times the task success rates of reinforcement learning policies trained with WiLoR and HaWoR motion references, respectively, and demonstrate real-time arm-hand teleoperation. Project website: https://fruitrabbit.github.io/WinDEX/.

cs.RO↗

Conditional independence testing through marginal permutation

Marginal permutation is exact for testing marginal independence but generally invalid for conditional independence because it breaks the dependence between the exposure ($\bm X$) and covariates ($\bm Z$). In this paper, we show that empirical studentization can nevertheless make this deliberately misspecified resampling scheme asymptotically valid for important conditional independence tests based on scores. The resulting procedures are novel in the sense that they are valid in finite samples when the exposure is marginally independent of the outcome ($\bm Y$) and covariates, and asymptotically valid under the broader conditional independence null, $\bm X\indep \bm Y\mid \bm Z$, subject to method-specific conditions. For generalized linear model scores, separately studentizing the observed and permuted scores yields validity under a correctly specified null mean despite \emph{arbitrary} conditional variance misspecification. Moreover, the permutation calibration also applies to tests based on generalized and projected covariance measures. Their validity accommodates machine learning nuisance estimates under a condition on the stability of refitting after permutation, for which we provide sufficient conditions for common regression learners. Permutation calibration preserves first-order local power while better capturing the finite-sample discreteness and skewness induced by sparse observations. Simulations and single-cell CRISPR screen analysis show substantial calibration gains with our methods when effective sample size is small.

stat.ME↗

Riemannian Optimization on Unknown Manifolds via Stochastic Proximal Refinement

We study Riemannian optimization on an unknown manifold accessed only through feasible samples. The central challenge is to turn feasible samples into optimization steps, i.e., the samples are drawn from the unknown feasible manifold, but do not directly provide the tangent spaces, projections, or retractions required by classic Riemannian methods. Moreover, regions with low objective values may be sparsely represented in the data, so directly sampling from a distribution fitted to these observations can require many draws to find better feasible solutions. We introduce Proximal Refinement on Implicit Smooth Manifolds (PRISM), which turns feasible-sample access into an iterative optimization procedure by repeatedly sampling an objective-aware proximal distribution conditioned on the current iterate. Theoretically, we show that PRISM recovers the effect of Riemannian-gradient refinement, leading to provable objective descent, feasibility (manifold adherence), and convergence toward first-order stationarity. We demonstrate the effectiveness of our approach through synthetic experiments illustrating its geometric properties and applications to data-driven building control and diverse real world optimization tasks, highlighting its potential as a general framework for data-driven optimization across scientific and engineering domains.

eess.SY↗

Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas

The evaluation of large language models (LLMs) is inherently multi-objective, as different models may offer different tradeoffs across performance dimensions such as math, code, and writing. Because such evaluations are often based on costly and time-consuming human feedback, it is important to draw reliable conclusions using as few comparisons as possible. In this paper, we present Vector-Borda Empirical-Gap Elimination (VB-EGE), a fixed-confidence adaptive sampling algorithm for efficiently identifying models on the Pareto frontier in arena-style evaluation, where each comparison evaluates a pair of models on a prompt with respect to a selected performance dimension. VB-EGE progressively eliminates models from an active set once sufficient statistical evidence has been accumulated to certify their Pareto status. Under a multi-objective Bradley-Terry model, we show that VB-EGE returns the correct Pareto frontier with high probability and derive an instance-dependent upper bound on its sample complexity. We further establish a worst-case lower bound showing that VB-EGE is minimax-optimal up to logarithmic factors. Simulations and experiments on three real-world Arena datasets (Arena, SearchArena, and VisionArena) demonstrate VB-EGE's empirical effectiveness.

stat.ML↗

Context-Aware Model Predictive Control for Microgrid Energy Management via LLMs

The optimal operation of modern microgrids, particularly those integrating stochastic renewable generation and battery energy storage system (BESS), relies heavily on load and disturbances forecasting to minimize operational costs. However, in environments with uncertainties in both generation and consumption, traditional numerical forecasting methods often fail to capture generation shifts and event-driven load surges. While contextual information regarding event schedules, system logs, and computational task records is easily obtainable, classic control paradigms lack a formal interface to integrate the unstructured, semantic data into the physical operation loop. This paper addresses this gap by introducing the InstructMPC framework, which utilizes a Large Language Model (LLM) paired with a tunable last layer mapping to translate unstructured operational context into predictive disturbance trajectories for the MPC controller. Unlike conventional forecasting methods, the proposed approach treats the last layer mapping as a tunable component, refined online based on the realized control cost. We establish a theoretical foundation for this closed-loop tuning strategy, proving a regret bound of $O(\sqrt{T \log T})$ for linear systems under a tailored task-aware loss function, together with robustness guarantees against uninformative or noisy textual inputs. The control strategy is experimentally validated on OpenCEM, a real-world microgrid with highly fluctuating generation and consumption. Experimental results demonstrate that the LLM-driven MPC significantly reduces cumulative grid electricity costs compared to classical context-agnostic baselines, validating the efficacy of integrating semantic information directly into physical control loops.

eess.SY↗

InstructMPC: A Human-LLM-in-the-Loop Framework for Context-Aware Control

Model Predictive Control (MPC) is a powerful control strategy widely utilized in domains like energy management, building control, and autonomous systems. However, its effectiveness in real-world settings is challenged by the need to incorporate context-specific predictions and expert instructions, which traditional MPC often neglects. We propose InstructMPC, a novel framework that addresses this gap by integrating real-time human instructions through a Large Language Model (LLM) to produce context-aware predictions for MPC. Our method employs a Language-to-Distribution (L2D) module to translate contextual information into predictive disturbance trajectories, which are then incorporated into the MPC optimization. Unlike existing context-aware and language-based MPC models, InstructMPC enables dynamic human-LLM interaction and fine-tunes the L2D module in a closed loop with theoretical performance guarantees, achieving a regret bound of $O(\sqrt{T\log T})$ for linear dynamics when optimized via advanced fine-tuning methods such as Direct Preference Optimization (DPO) using a tailored loss function.

eess.SY↗

Not all distributional shifts are equal: Fine-grained robust conformal inference

We introduce a fine-grained framework for uncertainty quantification of predictive models under distributional shifts. This framework distinguishes the shift in covariate distributions from that in the conditional relationship between the outcome ($Y$) and the covariates ($X$). We propose to reweight the training samples to adjust for an identifiable covariate shift while protecting against worst-case conditional distribution shift bounded in an $f$-divergence ball. Based on ideas from conformal inference and distributionally robust learning, we present an algorithm that outputs (approximately) valid and efficient prediction intervals in the presence of distributional shifts. As a use case, we apply the framework to sensitivity analysis of individual treatment effects with hidden confounding. The proposed methods are evaluated in simulation studies and four real data applications, demonstrating superior robustness and efficiency compared with existing benchmarks.

stat.ME↗