arXiv subjects

Tianyang Zhao

Publications and source records attributed to Tianyang Zhao.

8 recordsLinked to original sources

arXiv2026

PanoVGGT: Feed-Forward 3D Reconstruction from Panoramic Imagery

Panoramic imagery offers a full 360° field of view and is increasingly common in consumer devices. However, it introduces non-pinhole distortions that challenge joint pose estimation and 3D reconstruction. Existing feed-forward models, built for perspective cameras, generalize poorly to this setting. We propose PanoVGGT, a permutation-equivariant Transformer framework that jointly predicts camera poses, depth maps, and 3D point clouds from one or multiple panoramas in a single forward pass. The model incorporates spherical-aware positional embeddings and a panorama-specific three-axis SO(3) rotation augmentation, enabling effective geometric reasoning in the spherical domain. To resolve inherent global-frame ambiguity, we further introduce a stochastic anchoring strategy during training. In addition, we contribute PanoCity, a large-scale outdoor panoramic dataset with dense depth and 6-DoF pose annotations. Extensive experiments on PanoCity and standard benchmarks demonstrate that PanoVGGT achieves competitive accuracy, strong robustness, and improved cross-domain generalization. Code and dataset will be released.

cs.CV↗

arXiv2026

The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Models

Semantic information in embodied AI is inherently multi-source and multi-stage, making it challenging to fully leverage for achieving stable perception-to-action loops in real-world environments. Early studies have combined manual engineering with deep neural networks, achieving notable progress in specific semantic-related embodied tasks. However, as embodied agents encounter increasingly complex environments and open-ended tasks, the demand for more generalizable and robust semantic processing capabilities has become imperative. Recent advances in foundation models (FMs) address this challenge through their cross-domain generalization abilities and rich semantic priors, reshaping the landscape of embodied AI research. In this survey, we propose the Semantic Lifecycle as a unified framework to characterize the evolution of semantic knowledge within embodied AI driven by foundation models. Departing from traditional paradigms that treat semantic processing as isolated modules or disjoint tasks, our framework offers a holistic perspective that captures the continuous flow and maintenance of semantic knowledge. Guided by this embodied semantic lifecycle, we further analyze and compare recent advances across three key stages: acquisition, representation, and storage. Finally, we summarize existing challenges and outline promising directions for future research.

cs.CV↗

arXiv2022

Energy-Based Continuous Inverse Optimal Control

The problem of continuous inverse optimal control (over finite time horizon) is to learn the unknown cost function over the sequence of continuous control variables from expert demonstrations. In this article, we study this fundamental problem in the framework of energy-based model, where the observed expert trajectories are assumed to be random samples from a probability density function defined as the exponential of the negative cost function up to a normalizing constant. The parameters of the cost function are learned by maximum likelihood via an "analysis by synthesis" scheme, which iterates (1) synthesis step: sample the synthesized trajectories from the current probability density using the Langevin dynamics via back-propagation through time, and (2) analysis step: update the model parameters based on the statistical difference between the synthesized trajectories and the observed trajectories. Given the fact that an efficient optimization algorithm is usually available for an optimal control problem, we also consider a convenient approximation of the above learning method, where we replace the sampling in the synthesis step by optimization. Moreover, to make the sampling or optimization more efficient, we propose to train the energy-based model simultaneously with a top-down trajectory generator via cooperative learning, where the trajectory generator is used to fast initialize the synthesis step of the energy-based model. We demonstrate the proposed methods on autonomous driving tasks, and show that they can learn suitable cost functions for optimal control.

cs.LG↗

arXiv2021

Trajectory Prediction with Latent Belief Energy-Based Model

Human trajectory prediction is critical for autonomous platforms like self-driving cars or social robots. We present a latent belief energy-based model (LB-EBM) for diverse human trajectory forecast. LB-EBM is a probabilistic model with cost function defined in the latent space to account for the movement history and social context. The low-dimensionality of the latent space and the high expressivity of the EBM make it easy for the model to capture the multimodality of pedestrian trajectory distributions. LB-EBM is learned from expert demonstrations (i.e., human trajectories) projected into the latent space. Sampling from or optimizing the learned LB-EBM yields a belief vector which is used to make a path plan, which then in turn helps to predict a long-range trajectory. The effectiveness of LB-EBM and the two-step approach are supported by strong empirical results. Our model is able to make accurate, multi-modal, and social compliant trajectory predictions and improves over prior state-of-the-arts performance on the Stanford Drone trajectory prediction benchmark by 10.9% and on the ETH-UCY benchmark by 27.6%.

cs.LG↗

arXiv2020

Resilient Load Restoration in Microgrids Considering Mobile Energy Storage Fleets: A Deep Reinforcement Learning Approach

Mobile energy storage systems (MESSs) provide mobility and flexibility to enhance distribution system resilience. The paper proposes a Markov decision process (MDP) formulation for an integrated service restoration strategy that coordinates the scheduling of MESSs and resource dispatching of microgrids. The uncertainties in load consumption are taken into account. The deep reinforcement learning (DRL) algorithm is utilized to solve the MDP for optimal scheduling. Specifically, the twin delayed deep deterministic policy gradient (TD3) is applied to train the deep Q-network and policy network, then the well trained policy can be deployed in on-line manner to perform multiple actions simultaneously. The proposed model is demonstrated on an integrated test system with three microgrids connected by Sioux Falls transportation network. The simulation results indicate that mobile and stationary energy resources can be well coordinated to improve system resilience.

math.OC↗

arXiv2019

Resilient Unit Commitment for Day-ahead Market Considering Probabilistic Impacts of Hurricanes

In the face of extreme events, e.g., hurricanes, the transmission systems, especially the transmission lines, are affected across time and space. To mitigate these impacts on the day-ahead market from a probabilistic perspective, a resilient unit commitment (UC) problem is formulated as a two-stage robust optimization (RO) problem. In the first stage, the status, energy, and reserves of generators are pre-scheduled to minimize the operational cost, responding to the worst line failure scenario in the operating day. The failure, operation status, and repair of transmission lines are depicted by a novel robust uncertainty set with chance constraint considering the repair of failed lines. This chance constraint is reformulated to its deterministic equivalence. Using both load shedding and generation curtailment, a recourse problem is formulated in the second stage considering the time-varying transmission lines operation status. The formulated RO problem is solved using a column-and-constraint generation scheme. Simulations are conducted on IEEE-24 and two-area IEEE reliability test system-1996 under hurricanes and results verify the effectiveness of the proposed method on the conservation of uncertainty set, worst-case line failure scenario detection and repair preparedness.

math.OC↗

arXiv2019

Rolling Optimization of Mobile Energy Storage Fleets for Resilient Service Restoration

Mobile energy storage systems (MESSs) provide promising solutions to enhance distribution system resilience in terms of mobility and flexibility. This paper proposes a rolling integrated service restoration strategy to minimize the total system cost by coordinating the scheduling of MESS fleets, resource dispatching of microgrids and network reconfiguration of distribution systems. The integrated strategy takes into account damage and repair to both the roads in transportation networks and the branches in distribution systems. The uncertainties in load consumption and the status of roads and branches are modeled as scenario trees using Monte Carlo simulation method. The operation strategy of MESSs is modeled by a stochastic multi-layer time-space network technique. A rolling optimization framework is adopted to dynamically update system damage, and the coordinated scheduling at each time interval over the prediction horizon is formulated as a two-stage stochastic mixed-integer linear program with temporal-spatial and operation constraints. The proposed model is verified on two integrated test systems, one is with Sioux Falls transportation network and four 33-bus distribution systems, and the other is the Singapore transportation network-based test system connecting six 33-bus distribution systems. The results demonstrate the effectiveness of MESS mobility to enhance distribution system resilience due to the coordination of mobile and stationary resources.

math.OC↗

arXiv2019

Multi-Agent Tensor Fusion for Contextual Trajectory Prediction

Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context. The model decodes recurrently to multiple agents' future trajectories, using adversarial loss to learn stochastic predictions. Experiments on both highway driving and pedestrian crowd datasets show that the model achieves state-of-the-art prediction accuracy.

cs.CV↗