Searcharxiv⌕ Search

arXiv subjects

Zixin Zhang

Publications and source records attributed to Zixin Zhang.

At least 37 records · Page 2Linked to original sources

DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation

This paper presents DualCamCtrl, a novel end-to-end diffusion model for camera-controlled video generation. Recent works have advanced this field by representing camera poses as ray-based conditions, yet they often lack sufficient scene understanding and geometric awareness. DualCamCtrl specifically targets this limitation by introducing a dual-branch framework that mutually generates camera-consistent RGB and depth sequences. To harmonize these two modalities, we further propose the Semantic Guided Mutual Alignment (SIGMA) mechanism, which performs RGB-depth fusion in a semantics-guided and mutually reinforced manner. These designs collectively enable DualCamCtrl to better disentangle appearance and geometry modeling, generating videos that more faithfully adhere to the specified camera trajectories. Additionally, we analyze and reveal the distinct influence of depth and camera poses across denoising stages and further demonstrate that early and late stages play complementary roles in forming global structure and refining local details. Extensive experiments demonstrate that DualCamCtrl achieves more consistent camera-controlled video generation, with over 40\% reduction in camera motion errors compared with prior methods. Our project page: https://soyouthinkyoucantell.github.io/dualcamctrl-page/

cs.CV↗

Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks

Humans possess spatial reasoning abilities that enable them to understand spaces through multimodal observations, such as vision and sound. Large multimodal reasoning models extend these abilities by learning to perceive and reason, showing promising performance across diverse spatial tasks. However, systematic reviews and publicly available benchmarks for these models remain limited. In this survey, we provide a comprehensive review of multimodal spatial reasoning tasks with large models, categorizing recent progress in multimodal large language models (MLLMs) and introducing open benchmarks for evaluation. We begin by outlining general spatial reasoning, focusing on post-training techniques, explainability, and architecture. Beyond classical 2D tasks, we examine spatial relationship reasoning, scene and layout understanding, as well as visual question answering and grounding in 3D space. We also review advances in embodied AI, including vision-language navigation and action models. Additionally, we consider emerging modalities such as audio and egocentric video, which contribute to novel spatial understanding through new sensors. We believe this survey establishes a solid foundation and offers insights into the growing field of multimodal spatial reasoning. Updated information about this survey, codes and implementation of the open benchmarks can be found at https://github.com/zhengxuJosh/Awesome-Spatial-Reasoning.

cs.CV↗

PhysToolBench: Benchmarking Physical Tool Understanding for MLLMs

The ability to use, understand, and create tools is a hallmark of human intelligence, enabling sophisticated interaction with the physical world. For any general-purpose intelligent agent to achieve true versatility, it must also master these fundamental skills. While modern Multimodal Large Language Models (MLLMs) leverage their extensive common knowledge for high-level planning in embodied AI and in downstream Vision-Language-Action (VLA) models, the extent of their true understanding of physical tools remains unquantified. To bridge this gap, we present PhysToolBench, the first benchmark dedicated to evaluating the comprehension of physical tools by MLLMs. Our benchmark is structured as a Visual Question Answering (VQA) dataset comprising over 1,000 image-text pairs. It assesses capabilities across three distinct difficulty levels: (1) Tool Recognition: Requiring the recognition of a tool's primary function. (2) Tool Understanding: Testing the ability to grasp the underlying principles of a tool's operation. (3) Tool Creation: Challenging the model to fashion a new tool from surrounding objects when conventional options are unavailable. Our comprehensive evaluation of 32 MLLMs-spanning proprietary, open-source, specialized embodied, and backbones in VLAs-reveals a significant deficiency in tool understanding. Furthermore, we provide an in-depth analysis and propose preliminary solutions. Code and dataset are publicly available.

cs.CV↗

Sample-Efficient Online Control Policy Learning with Real-Time Recursive Model Updates

Data-driven control methods need to be sample-efficient and lightweight, especially when data acquisition and computational resources are limited -- such as during learning on hardware. Most modern data-driven methods require large datasets and struggle with real-time updates of models, limiting their performance in dynamic environments. Koopman theory formally represents nonlinear systems as linear models over observables, and Koopman representations can be determined from data in an optimization-friendly setting with potentially rapid model updates. In this paper, we present a highly sample-efficient, Koopman-based learning pipeline: Recursive Koopman Learning (RKL). We identify sufficient conditions for model convergence and provide formal algorithmic analysis supporting our claim that RKL is lightweight and fast, with complexity independent of dataset size. We validate our method on a simulated planar two-link arm and a hybrid nonlinear hardware system with soft actuators, showing that real-time recursive Koopman model updates improve the sample efficiency and stability of data-driven controller synthesis -- requiring only <10% of the data compared to benchmarks. The high-performance C++ codebase is open-sourced. Website: https://www.zixinatom990.com/home/robotics/corl-2025-recursive-koopman-learning.

cs.RO↗

An analytical framework to assess static versus dynamic triggering of fault-slip rockbursts

Fault-slip rockbursts, triggered by seismic rupture of nearby or remote faults, constitute a significant geohazard during deep underground excavations. Although these events occur frequently in underground projects, their underlying mechanisms are not yet fully understood. Most studies tacitly assume dynamic stress waves as the main triggering factor, often disregarding the role of coseismic static stress changes associated with fault slip. This paper introduces a novel analytical framework to diagnose both static and dynamic coseismic stress perturbations and quantify their contributions to fault-slip rockburst around a circular tunnel. Building on linear elastic fracture mechanics, seismic source theory, and the Kirsch solution, the model assesses whether coseismically elevated maximum tangential stress on the tunnel boundary under static and dynamic triggering effects is sufficient to induce failure around the tunnel. We extensively test our framework using synthetic case studies that represent typical fault-slip rockburst scenarios. Our results yield a rockburst hazard map that delineates regions of elevated triggering potential in the near-field and far-field of the seismogenic fault, and classify the triggering types as static, dynamic, or dual. We perform a comprehensive parametric sensitivity analysis to investigate how key factors, including seismic source characteristics, rock mass properties, and in-situ stress conditions, influence the spatial distribution of rockburst susceptibility. The model is further applied to a historical fault-slip rockburst event at the Gotthard Base Tunnel, effectively capturing the triggering mechanism of the observed failure. Our research provides a physically grounded and computationally efficient analytical framework with the results carrying significant implications for rockburst hazard assessments during deep underground excavations.

physics.geo-ph↗

Multi-IMU Sensor Fusion for Legged Robots

This paper presents a state-estimation solution for legged robots that uses a set of low-cost, compact, and lightweight sensors to achieve low-drift pose and velocity estimation under challenging locomotion conditions. The key idea is to leverage multiple inertial measurement units on different links of the robot to correct a major error source in standard proprioceptive odometry. We fuse the inertial sensor information and joint encoder measurements in an extended Kalman filter, then combine the velocity estimate from this filter with camera data in a factor-graph-based sliding-window estimator to form a visual-inertial-leg odometry method. We validate our state estimator through comprehensive theoretical analysis and hardware experiments performed using real-world robot data collected during a variety of challenging locomotion tasks. Our algorithm consistently achieves minimal position deviation, even in scenarios involving substantial ground impact, foot slippage, and sudden body rotations. A C++ implementation, along with a large-scale dataset, is available at https://github.com/ShuoYangRobotics/Cerberus2.0.

cs.RO↗

Step-Audio-AQAA: a Fully End-to-End Expressive Large Audio Language Model

Large Audio-Language Models (LALMs) have significantly advanced intelligent human-computer interaction, yet their reliance on text-based outputs limits their ability to generate natural speech responses directly, hindering seamless audio interactions. To address this, we introduce Step-Audio-AQAA, a fully end-to-end LALM designed for Audio Query-Audio Answer (AQAA) tasks. The model integrates a dual-codebook audio tokenizer for linguistic and semantic feature extraction, a 130-billion-parameter backbone LLM and a neural vocoder for high-fidelity speech synthesis. Our post-training approach employs interleaved token-output of text and audio to enhance semantic coherence and combines Direct Preference Optimization (DPO) with model merge to improve performance. Evaluations on the StepEval-Audio-360 benchmark demonstrate that Step-Audio-AQAA excels especially in speech control, outperforming the state-of-art LALMs in key areas. This work contributes a promising solution for end-to-end LALMs and highlights the critical role of token-based vocoder in enhancing overall performance for AQAA tasks.

cs.SD↗

ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback

With the rapid advancement of generative models, general-purpose generation has gained increasing attention as a promising approach to unify diverse tasks across modalities within a single system. Despite this progress, existing open-source frameworks often remain fragile and struggle to support complex real-world applications due to the lack of structured workflow planning and execution-level feedback. To address these limitations, we present ComfyMind, a collaborative AI system designed to enable robust and scalable general-purpose generation, built on the ComfyUI platform. ComfyMind introduces two core innovations: Semantic Workflow Interface (SWI) that abstracts low-level node graphs into callable functional modules described in natural language, enabling high-level composition and reducing structural errors; Search Tree Planning mechanism with localized feedback execution, which models generation as a hierarchical decision process and allows adaptive correction at each stage. Together, these components improve the stability and flexibility of complex generative workflows. We evaluate ComfyMind on three public benchmarks: ComfyBench, GenEval, and Reason-Edit, which span generation, editing, and reasoning tasks. Results show that ComfyMind consistently outperforms existing open-source baselines and achieves performance comparable to GPT-Image-1. ComfyMind paves a promising path for the development of open-source general-purpose generative AI systems. Project page: https://github.com/LitaoGuo/ComfyMind

cs.AI↗

Modelling lined rock caverns subject to hydrogen embrittlement and cyclic pressurisation in fractured rock masses

The technology of lined rock cavern (LRC) with great geographical flexibility is a promising, cost-effective solution to underground hydrogen storage. However, the air-tight steel tanks used in this technology are susceptible to material degradation due to hydrogen embrittlement (HE), potentially leading to leakage and structural failure, especial for LRCs constructed in complex geological conditions. In this paper, we develop a 2D multiscale numerical model based on the finite element method to assess the impact of HE on the LRC performance in fractured rock masses under cyclic gas pressurisation. Within this framework, a large-scale model is used to simulate the deformation and damage evolution of both fractured rock and an LRC under in-situ stresses and internal gas pressurisation, while a small-scale model captures HE in the steel lining of the LRC. Our simulations reveal that damage in the rock, concrete, and steel degradation is strongly affected by pre-existing fractures and in-situ stresses. Our results also reveal the presence of a strong positive feedback between hydrogen concentration and stress redistribution in the steel lining. Moreover, a comparison between models with and without considering HE illuminates that hydrogen concentration significantly contributes to steel degradation, particularly during the long-term LRC operation, highlighting the critical role of HE in the safety and performance of the LRC. The findings and insights obtained from our work have important implications for the design optimisation and performance assessment of LRCs for sustainable underground hydrogen storage.

physics.comp-ph↗

Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

Real-time speech interaction, serving as a fundamental interface for human-machine collaboration, holds immense potential. However, current open-source models face limitations such as high costs in voice data collection, weakness in dynamic control, and limited intelligence. To address these challenges, this paper introduces Step-Audio, the first production-ready open-source solution. Key contributions include: 1) a 130B-parameter unified speech-text multi-modal model that achieves unified understanding and generation, with the Step-Audio-Chat version open-sourced; 2) a generative speech data engine that establishes an affordable voice cloning framework and produces the open-sourced lightweight Step-Audio-TTS-3B model through distillation; 3) an instruction-driven fine control system enabling dynamic adjustments across dialects, emotions, singing, and RAP; 4) an enhanced cognitive architecture augmented with tool calling and role-playing abilities to manage complex tasks effectively. Based on our new StepEval-Audio-360 evaluation benchmark, Step-Audio achieves state-of-the-art performance in human evaluations, especially in terms of instruction following. On open-source benchmarks like LLaMA Question, shows 9.3% average performance improvement, demonstrating our commitment to advancing the development of open-source multi-modal language technologies. Our code and models are available at https://github.com/stepfun-ai/Step-Audio.

cs.CL↗

Exoplanet Ephemerides Change Observations (ExoEcho). I. Transit Timing Analysis of Thirty-Seven Exoplanets using HST/WFC3 Data

The ExoEcho project is designed to study the photodynamics of exoplanets by leveraging high-precision transit timing data from ground- and space-based telescopes. Some exoplanets are experiencing orbital decay, and transit timing variation (TTV) is a useful technique to study their orbital period variations. In this study, we have obtained transit middle-time data from the Hubble Space Telescope (HST) observations for 37 short-period exoplanets, most of which are hot Jupiters. To search for potential long- and short-term orbital period variations within the sample, we conduct TTV model fitting using both linear and quadratic ephemeris models. Our analysis identifies two hot Jupiters experiencing strong periodic decays. Given the old age of the host stars of the hot Jupiter population, our findings call for a scenario where HJs are continuously being destructed and created. Our study demonstrates the importance of incorporating high-precision transit timing data to TTV study in the future.

astro-ph.EP↗

Comateformer: Combined Attention Transformer for Semantic Sentence Matching

The Transformer-based model have made significant strides in semantic matching tasks by capturing connections between phrase pairs. However, to assess the relevance of sentence pairs, it is insufficient to just examine the general similarity between the sentences. It is crucial to also consider the tiny subtleties that differentiate them from each other. Regrettably, attention softmax operations in transformers tend to miss these subtle differences. To this end, in this work, we propose a novel semantic sentence matching model named Combined Attention Network based on Transformer model (Comateformer). In Comateformer model, we design a novel transformer-based quasi-attention mechanism with compositional properties. Unlike traditional attention mechanisms that merely adjust the weights of input tokens, our proposed method learns how to combine, subtract, or resize specific vectors when building a representation. Moreover, our proposed approach builds on the intuition of similarity and dissimilarity (negative affinity) when calculating dual affinity scores. This allows for a more meaningful representation of relationships between sentences. To evaluate the performance of our proposed model, we conducted extensive experiments on ten public real-world datasets and robustness testing. Experimental results show that our method achieves consistent improvements.

cs.CL↗

Constraining the Presence of Companion Planets in Hot Jupiter Planetary System Using TTV Observation from TESS

The presence of another planetary companion in a transiting exoplanet system can impact its transit light curve, leading to sinusoidal transit timing variations (TTV). By utilizing both $χ^2$ and RMS analysis, we have combined the TESS observation data with an N-body simulation to investigate the existence of an additional planet in the system and put a limit on its mass. We have developed CMAT, an efficient and user-friendly tool for fitting transit light curves and calculating TTV with a theoretical period, based on which we can give a limit on its hidden companion's mass. We use 260 hot Jupiter systems from the complete TESS data set to demonstrate the use of CMAT. Our findings indicate that, for most systems, the upper mass limit of a companion planet can be restricted to several Jupiter masses. This constraint becomes stronger near resonance orbits, such as the 1:2, 2:1, 3:1, and 4:1 mean motion resonance, where the limit is reduced to several Earth masses. These findings align with previous studies suggesting that a lack of companion planets with resonance in hot Jupiter systems could potentially support the high eccentricity migration theory. Additionally, we observed that the choice between $χ^2$ or {root mean square (RMS)} method does not significantly affect the upper limit on companion mass; however, $χ^2$ analysis may result in weaker restrictions but is statistically more robust compared to RMS analysis in most cases.

astro-ph.EP↗

Elite-EvGS: Learning Event-based 3D Gaussian Splatting by Distilling Event-to-Video Priors

Event cameras are bio-inspired sensors that output asynchronous and sparse event streams, instead of fixed frames. Benefiting from their distinct advantages, such as high dynamic range and high temporal resolution, event cameras have been applied to address 3D reconstruction, important for robotic mapping. Recently, neural rendering techniques, such as 3D Gaussian splatting (3DGS), have been shown successful in 3D reconstruction. However, it still remains under-explored how to develop an effective event-based 3DGS pipeline. In particular, as 3DGS typically depends on high-quality initialization and dense multiview constraints, a potential problem appears for the 3DGS optimization with events given its inherent sparse property. To this end, we propose a novel event-based 3DGS framework, named Elite-EvGS. Our key idea is to distill the prior knowledge from the off-the-shelf event-to-video (E2V) models to effectively reconstruct 3D scenes from events in a coarse-to-fine optimization manner. Specifically, to address the complexity of 3DGS initialization from events, we introduce a novel warm-up initialization strategy that optimizes a coarse 3DGS from the frames generated by E2V models and then incorporates events to refine the details. Then, we propose a progressive event supervision strategy that employs the window-slicing operation to progressively reduce the number of events used for supervision. This subtly relives the temporal randomness of the event frames, benefiting the optimization of local textural and global structural details. Experiments on the benchmark datasets demonstrate that Elite-EvGS can reconstruct 3D scenes with better textural and structural details. Meanwhile, our method yields plausible performance on the captured real-world data, including diverse challenging conditions, such as fast motion and low light scenes.

cs.CV↗

Robots with Attitude: Singularity-Free Quaternion-Based Model-Predictive Control for Agile Legged Robots

We present a model-predictive control (MPC) framework for legged robots that avoids the singularities associated with common three-parameter attitude representations like Euler angles during large-angle rotations. Our method parameterizes the robot's attitude with singularity-free unit quaternions and makes modifications to the iterative linear-quadratic regulator (iLQR) algorithm to deal with the resulting geometry. The derivation of our algorithm requires only elementary calculus and linear algebra, deliberately avoiding the abstraction and notation of Lie groups. We demonstrate the performance and computational efficiency of quaternion MPC in several experiments on quadruped and humanoid robots.

cs.RO↗

Long-term Orbital Period Variation of Hot Jupiters from Transiting Time Analysis using TESS Survey Data

Many hot Jupiters may experience orbital decays, which are manifested as long-term transit timing variations. We have analyzed 7068 transits from the Transiting Exoplanet Survey Satellite (TESS) for a sample of 326 hot Jupiters. These new mid-transit time data allow us to update ephemerides for these systems. By combining the new TESS transit timing data with archival data, we search for possible long-term orbital period variations in these hot Jupiters using a linear and a quadratic ephemeris model. We identified 26 candidates that exhibit possible long-term orbital period variations, including 18 candidates with decreasing orbital periods and 8 candidates with increasing orbital periods. Among them, 12 candidates have failed in our leave-one-out cross-validation (LOOCV) test and thus should be considered as marginal candidates. In addition to tidal interaction, alternative mechanisms such as apsidal precession, Rømer effect, and Applegate effect could also contribute to the observed period variations. The ephemerides derived in this work are useful for scheduling follow-up observations for these hot Jupiters in the future. The Python code used to generate the ephemerides is made available online.

astro-ph.EP↗

PlankAssembly: Robust 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs

In this paper, we develop a new method to automatically convert 2D line drawings from three orthographic views into 3D CAD models. Existing methods for this problem reconstruct 3D models by back-projecting the 2D observations into 3D space while maintaining explicit correspondence between the input and output. Such methods are sensitive to errors and noises in the input, thus often fail in practice where the input drawings created by human designers are imperfect. To overcome this difficulty, we leverage the attention mechanism in a Transformer-based sequence generation model to learn flexible mappings between the input and output. Further, we design shape programs which are suitable for generating the objects of interest to boost the reconstruction accuracy and facilitate CAD modeling applications. Experiments on a new benchmark dataset show that our method significantly outperforms existing ones when the inputs are noisy or incomplete.

cs.CV↗

Cerberus: Low-Drift Visual-Inertial-Leg Odometry For Agile Locomotion

We present an open-source Visual-Inertial-Leg Odometry (VILO) state estimation solution, Cerberus, for legged robots that estimates position precisely on various terrains in real time using a set of standard sensors, including stereo cameras, IMU, joint encoders, and contact sensors. In addition to estimating robot states, we also perform online kinematic parameter calibration and contact outlier rejection to substantially reduce position drift. Hardware experiments in various indoor and outdoor environments validate that calibrating kinematic parameters within the Cerberus can reduce estimation drift to lower than 1% during long distance high speed locomotion. Our drift results are better than any other state estimation method using the same set of sensors reported in the literature. Moreover, our state estimator performs well even when the robot is experiencing large impacts and camera occlusion. The implementation of the state estimator, along with the datasets used to compute our results, are available at https://github.com/ShuoYangRobotics/Cerberus.

cs.RO↗