arXiv subjects

Chao Yu

Publications and source records attributed to Chao Yu.

At least 127 records · Page 7Linked to original sources

arXiv2024

LAGOON: Language-Guided Motion Control

We aim to control a robot to physically behave in the real world following any high-level language command like "cartwheel" or "kick". Although human motion datasets exist, this task remains particularly challenging since generative models can produce physically unrealistic motions, which will be more severe for robots due to different body structures and physical properties. Deploying such a motion to a physical robot can cause even greater difficulties due to the sim2real gap. We develop LAnguage-Guided mOtion cONtrol (LAGOON), a multi-phase reinforcement learning (RL) method to generate physically realistic robot motions under language commands. LAGOON first leverages a pretrained model to generate a human motion from a language command. Then an RL phase trains a control policy in simulation to mimic the generated human motion. Finally, with domain randomization, our learned policy can be deployed to a quadrupedal robot, leading to a quadrupedal robot that can take diverse behaviors in the real world under natural language commands

cs.RO↗

arXiv2024

A Dual Curriculum Learning Framework for Multi-UAV Pursuit-Evasion in Diverse Environments

This paper addresses multi-UAV pursuit-evasion, where a group of drones cooperates to capture a fast evader in a confined environment with obstacles. Existing heuristic algorithms, which simplify the pursuit-evasion problem, often lack expressive coordination strategies and struggle to capture the evader in extreme scenarios, such as when the evader moves at high speeds. In contrast, reinforcement learning (RL) has been applied to this problem and has the potential to obtain highly cooperative capture strategies. However, RL-based methods face challenges in training for complex 3-dimensional scenarios with diverse task settings due to the vast exploration space. The dynamics constraints of drones further restrict the ability of reinforcement learning to acquire high-performance capture strategies. In this work, we introduce a dual curriculum learning framework, named DualCL, which addresses multi-UAV pursuit-evasion in diverse environments and demonstrates zero-shot transfer ability to unseen scenarios. DualCL comprises two main components: the Intrinsic Parameter Curriculum Proposer, which progressively suggests intrinsic parameters from easy to hard to improve the capture capability of drones, and the External Environment Generator, tasked with exploring unresolved scenarios and generating appropriate training distributions of external environment parameters. The simulation experimental results show that DualCL significantly outperforms baseline methods, achieving over 90% capture rate and reducing the capture timestep by at least 27.5% in the training scenarios. Additionally, it exhibits the best zero-shot generalization ability in unseen environments. Moreover, we demonstrate the transferability of our pursuit strategy from simulation to real-world environments. Further details can be found on the project website at https://sites.google.com/view/dualcl.

cs.LG↗

arXiv2024

CASTER: A Computer-Vision-Assisted Wireless Channel Simulator for Gesture Recognition

In this paper, a computer-vision-assisted simulation method is proposed to address the issue of training dataset acquisition for wireless hand gesture recognition. In the existing literature, in order to classify gestures via the wireless channel estimation, massive training samples should be measured in a consistent environment, consuming significant efforts. In the proposed CASTER simulator, however, the training dataset can be simulated via existing videos. Particularly, a gesture is represented by a sequence of snapshots, and the channel impulse response of each snapshot is calculated via tracing the rays scattered off a primitive-based hand model. Moreover, CASTER simulator relies on the existing videos to extract the motion data of gestures. Thus, the massive measurements of wireless channel can be eliminated. The experiments demonstrate a 90.8% average classification accuracy of simulation-to-reality inference.

eess.SP↗

arXiv2024

Off-Policy Primal-Dual Safe Reinforcement Learning

Primal-dual safe RL methods commonly perform iterations between the primal update of the policy and the dual update of the Lagrange Multiplier. Such a training paradigm is highly susceptible to the error in cumulative cost estimation since this estimation serves as the key bond connecting the primal and dual update processes. We show that this problem causes significant underestimation of cost when using off-policy methods, leading to the failure to satisfy the safety constraint. To address this issue, we propose conservative policy optimization, which learns a policy in a constraint-satisfying area by considering the uncertainty in cost estimation. This improves constraint satisfaction but also potentially hinders reward maximization. We then introduce local policy convexification to help eliminate such suboptimality by gradually reducing the estimation uncertainty. We provide theoretical interpretations of the joint coupling effect of these two ingredients and further verify them by extensive experiments. Results on benchmark tasks show that our method not only achieves an asymptotic performance comparable to state-of-the-art on-policy methods while using much fewer samples, but also significantly reduces constraint violation during training. Our code is available at https://github.com/ZifanWu/CAL.

cs.LG↗

arXiv2024

Miniaturized time-correlated single-photon counting module for time-of-flight non-line-of-sight imaging applications

Single-photon time-of-flight (TOF) non-line-of-sight (NLOS) imaging enables the high-resolution reconstruction of objects outside the field of view. The compactness of TOF NLOS imaging systems, entailing the miniaturization of key components within such systems is crucial for practical applications. Here, we present a miniaturized four-channel time-correlated single-photon counting module dedicated to TOF NLOS imaging applications. The module achieves excellent performance with a 10 ps bin size and 27.4 ps minimum root-mean-square time resolution. We present the results of TOF NLOS imaging experiment using an InGaAs/InP single-photon detector and the time-correlated single-photon counting module, and show that a 6.3 cm lateral resolution and 2.3 cm depth resolution can be achieved under the conditions of 5 m imaging distance and 1 ms pixel dwell time.

physics.ins-det↗

arXiv2024

Multi-Agent Reinforcement Learning with a Hierarchy of Reward Machines

In this paper, we study the cooperative Multi-Agent Reinforcement Learning (MARL) problems using Reward Machines (RMs) to specify the reward functions such that the prior knowledge of high-level events in a task can be leveraged to facilitate the learning efficiency. Unlike the existing work that RMs have been incorporated into MARL for task decomposition and policy learning in relatively simple domains or with an assumption of independencies among the agents, we present Multi-Agent Reinforcement Learning with a Hierarchy of RMs (MAHRM) that is capable of dealing with more complex scenarios when the events among agents can occur concurrently and the agents are highly interdependent. MAHRM exploits the relationship of high-level events to decompose a task into a hierarchy of simpler subtasks that are assigned to a small group of agents, so as to reduce the overall computational complexity. Experimental results in three cooperative MARL domains show that MAHRM outperforms other MARL methods using the same prior knowledge of high-level events.

cs.AI↗

arXiv2024

Safe Offline Reinforcement Learning with Real-Time Budget Constraints

Aiming at promoting the safe real-world deployment of Reinforcement Learning (RL), research on safe RL has made significant progress in recent years. However, most existing works in the literature still focus on the online setting where risky violations of the safety budget are likely to be incurred during training. Besides, in many real-world applications, the learned policy is required to respond to dynamically determined safety budgets (i.e., constraint threshold) in real time. In this paper, we target at the above real-time budget constraint problem under the offline setting, and propose Trajectory-based REal-time Budget Inference (TREBI) as a novel solution that models this problem from the perspective of trajectory distribution and solves it through diffusion model planning. Theoretically, we prove an error bound of the estimation on the episodic reward and cost under the offline setting and thus provide a performance guarantee for TREBI. Empirical results on a wide range of simulation tasks and a real-world large-scale advertising application demonstrate the capability of TREBI in solving real-time budget constraint problems under offline settings.

cs.LG↗

arXiv2024

The prominent and heterogeneous gender disparities in scientific novelty: evidence from biomedical doctoral theses

Scientific novelty is the essential driving force for research breakthroughs and innovation. However, little is known about how early-career scientists pursue novel research paths, and the gender disparities in this process. To address this research gap, this study investigates a comprehensive dataset of 279,424 doctoral theses in biomedical sciences authored by US Ph.D. graduates. Spanning from 1980 to 2016, the data originates from the ProQuest Dissertations & Theses Database. This study aims to shed light on Ph.D. students' pursuit of scientific novelty in their doctoral theses and assess gender-related differences in this process. Using a combinatorial approach and a pre-trained Bio-BERT model, we quantify the scientific novelty of doctoral theses based on bio-entities. Applying fractional logistic and quantile regression models, this study reveals a decreasing trend in scientific novelty over time and heterogeneous gender disparities in doctoral theses. Specifically, female students consistently exhibited lower scientific novelty levels than their male peers. When supervised by female advisors, students' theses are found to be less novel than those under male advisors. The significant interaction effect of female students and female advisors suggests that female advisors may amplify the gender disparity in scientific novelty. Moreover, heterogeneous gender disparities in scientific novelty are identified, with non-top-tier universities displaying more pronounced disparities, while the differences at higher percentile ranges were comparatively more minor. These findings indicate a potential underrepresentation of female scientists pursuing novel research during the early stages of their careers. Notably, the outcomes of this study hold significant policy implications for advancing the careers of female scientists.

physics.soc-ph↗

arXiv2024

On the Theory of Solid-State Harmonic Generation Governed by Crystal Symmetry

The solid-state harmonic generation (SSHG) derives from photocurrent coherence. The crystal symmetry, including point-group symmetry and time-reversal symmetry, constrains the amplitude and phase of the photocurrent, thus manipulates the coherent processes in SSHG. We revisit the expression of photocurrent under the electric dipole approximation and give an unambiguous picture of non-equilibrium dynamics of photocarriers on laser-dressed effective bands. In addition to the dynamical phase, we reveal the indispensable roles of the phase difference of transition dipole moments and the phase induced by shift vector in the photocurrent coherence. Microscopic mechanism of the selection rule, orientation dependence, polarization characteristics, time-frequency analysis and ellipticity dependence of harmonics governed by symmetries is uniformly clarified in our theoretical framework. This work integrates non-equilibrium electronic dynamics of condensed matter in strong laser fields, and paves a way to explore more nonlinear optical phenomena governed by crystal symmetry.

physics.optics↗

arXiv2024

LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordination

AI agents powered by Large Language Models (LLMs) have made significant advances, enabling them to assist humans in diverse complex tasks and leading to a revolution in human-AI coordination. LLM-powered agents typically require invoking LLM APIs and employing artificially designed complex prompts, which results in high inference latency. While this paradigm works well in scenarios with minimal interactive demands, such as code generation, it is unsuitable for highly interactive and real-time applications, such as gaming. Traditional gaming AI often employs small models or reactive policies, enabling fast inference but offering limited task completion and interaction abilities. In this work, we consider Overcooked as our testbed where players could communicate with natural language and cooperate to serve orders. We propose a Hierarchical Language Agent (HLA) for human-AI coordination that provides both strong reasoning abilities while keeping real-time execution. In particular, HLA adopts a hierarchical framework and comprises three modules: a proficient LLM, referred to as Slow Mind, for intention reasoning and language interaction, a lightweight LLM, referred to as Fast Mind, for generating macro actions, and a reactive policy, referred to as Executor, for transforming macro actions into atomic actions. Human studies show that HLA outperforms other baseline agents, including slow-mind-only agents and fast-mind-only agents, with stronger cooperation abilities, faster responses, and more consistent language communications.

cs.AI↗

arXiv2024

Policy-regularized Offline Multi-objective Reinforcement Learning

In this paper, we aim to utilize only offline trajectory data to train a policy for multi-objective RL. We extend the offline policy-regularized method, a widely-adopted approach for single-objective offline RL problems, into the multi-objective setting in order to achieve the above goal. However, such methods face a new challenge in offline MORL settings, namely the preference-inconsistent demonstration problem. We propose two solutions to this problem: 1) filtering out preference-inconsistent demonstrations via approximating behavior preferences, and 2) adopting regularization techniques with high policy expressiveness. Moreover, we integrate the preference-conditioned scalarized update method into policy-regularized offline RL, in order to simultaneously learn a set of policies using a single policy network, thus reducing the computational cost induced by the training of a large number of individual policies for various preferences. Finally, we introduce Regularization Weight Adaptation to dynamically determine appropriate regularization weights for arbitrary target preferences during deployment. Empirical results on various multi-objective datasets demonstrate the capability of our approach in solving offline MORL problems.

cs.LG↗

arXiv2023

Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning

Learning Nash equilibrium (NE) in complex zero-sum games with multi-agent reinforcement learning (MARL) can be extremely computationally expensive. Curriculum learning is an effective way to accelerate learning, but an under-explored dimension for generating a curriculum is the difficulty-to-learn of the subgames -- games induced by starting from a specific state. In this work, we present a novel subgame curriculum learning framework for zero-sum games. It adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. Building upon this framework, we derive a subgame selection metric that approximates the squared distance to NE values and further adopt a particle-based state sampler for subgame generation. Integrating these techniques leads to our new algorithm, Subgame Automatic Curriculum Learning (SACL), which is a realization of the subgame curriculum learning framework. SACL can be combined with any MARL algorithm such as MAPPO. Experiments in the particle-world environment and Google Research Football environment show SACL produces much stronger policies than baselines. In the challenging hide-and-seek quadrant environment, SACL produces all four emergent stages and uses only half the samples of MAPPO with self-play. The project website is at https://sites.google.com/view/sacl-rl.

cs.LG↗

arXiv2023

Passive Handwriting Tracking via Weak mmWave Communication Signals

In this letter, a cooperative sensing framework based on millimeter wave (mmWave) communication systems is proposed to detect tiny motions with a millimeter-level resolution. Particularly, the cooperative sensing framework is facilitated with one transmitter and two receivers. There are two radio frequency (RF) chains at each receiver. Hence, the Doppler effect due to the tiny motions can be detected via passive sensing respectively at the receivers, and the velocities of the motions can be estimated by integrating the Doppler frequencies. It is demonstrated that the proposed cooperative sensing system is able to track the handwriting with 90% error below 6 mm. Moreover, the proposed cooperative sensing is robust to the strength of received signal. For example, it works even without the line-of-sight paths from the transmitter to the receivers or the sensing target, where the received signal strength is not sufficient for timing synchronization or demodulation.

eess.SY↗

arXiv2023

Active Neural Topological Mapping for Multi-Agent Exploration

This paper investigates the multi-agent cooperative exploration problem, which requires multiple agents to explore an unseen environment via sensory signals in a limited time. A popular approach to exploration tasks is to combine active mapping with planning. Metric maps capture the details of the spatial representation, but are with high communication traffic and may vary significantly between scenarios, resulting in inferior generalization. Topological maps are a promising alternative as they consist only of nodes and edges with abstract but essential information and are less influenced by the scene structures. However, most existing topology-based exploration tasks utilize classical methods for planning, which are time-consuming and sub-optimal due to their handcrafted design. Deep reinforcement learning (DRL) has shown great potential for learning (near) optimal policies through fast end-to-end inference. In this paper, we propose Multi-Agent Neural Topological Mapping (MANTM) to improve exploration efficiency and generalization for multi-agent exploration tasks. MANTM mainly comprises a Topological Mapper and a novel RL-based Hierarchical Topological Planner (HTP). The Topological Mapper employs a visual encoder and distance-based heuristics to construct a graph containing main nodes and their corresponding ghost nodes. The HTP leverages graph neural networks to capture correlations between agents and graph nodes in a coarse-to-fine manner for effective global goal selection. Extensive experiments conducted in a physically-realistic simulator, Habitat, demonstrate that MANTM reduces the steps by at least 26.40% over planning-based baselines and by at least 7.63% over RL-based competitors in unseen scenarios.

cs.RO↗

arXiv2023

Compact free-running InGaAs/InP single-photon detector with 40% detection efficiency and 2.3 kcps dark count rate

Free-running InGaAs/InP single-photon detectors (SPDs) based on negative-feedback avalanche diodes (NFADs) are the key components for applications requiring asynchronous single-photon detection in the near-infrared region. From the perspective of practical applications, the features of SPDs in terms of high photon detection efficiency (PDE), low noise, large sensitive area, and compactness are highly desired for system integration and performance enhancement. Here, we present the implementation of a compact four-channel multimode fiber coupling free-running InGaAs/InP SPD, with the best overall performance to date. On the one hand, we design and fabricate structure-optimized InGaAs/InP NFAD devices with 25 $μ$m diameter active area and integrated thin film resistors to enhance the maximum achievable PDE. On the other hand, we apply a compact thermoacoustic cryocooler to regulate the operating temperature of NFADs within a large range, and design a dedicated readout circuit with minimized parasitic parameters and tunable settings of hold-off time to suppress the afterpulsing effect. The SPD is then characterized to achieve remarkable overall performance simultaneously at 1550 nm, i.e., 40% PDE, 2.3 kcps dark count rate, 8% afterpulse probability and 49 ps timing jitter (full width at half maximum) under the conditions of 5.9 V excess bias voltage, 10 $μ$s hold-off time and 213 K operation temperature. Such performance and the results of the long-term stability tests indicate that the SPD could be a favorable solution for practical applications.

physics.ins-det↗

arXiv2023

Fictitious Cross-Play: Learning Global Nash Equilibrium in Mixed Cooperative-Competitive Games

Self-play (SP) is a popular multi-agent reinforcement learning (MARL) framework for solving competitive games, where each agent optimizes policy by treating others as part of the environment. Despite the empirical successes, the theoretical properties of SP-based methods are limited to two-player zero-sum games. However, for mixed cooperative-competitive games where agents on the same team need to cooperate with each other, we can show a simple counter-example where SP-based methods cannot converge to a global Nash equilibrium (NE) with high probability. Alternatively, Policy-Space Response Oracles (PSRO) is an iterative framework for learning NE, where the best responses w.r.t. previous policies are learned in each iteration. PSRO can be directly extended to mixed cooperative-competitive settings by jointly learning team best responses with all convergence properties unchanged. However, PSRO requires repeatedly training joint policies from scratch till convergence, which makes it hard to scale to complex games. In this work, we develop a novel algorithm, Fictitious Cross-Play (FXP), which inherits the benefits from both frameworks. FXP simultaneously trains an SP-based main policy and a counter population of best response policies. The main policy is trained by fictitious self-play and cross-play against the counter population, while the counter policies are trained as the best responses to the main policy's past versions. We validate our method in matrix games and show that FXP converges to global NEs while SP methods fail. We also conduct experiments in a gridworld domain, where FXP achieves higher Elo ratings and lower exploitabilities than baselines, and a more challenging football game, where FXP defeats SOTA models with over 94% win rate.

cs.AI↗

arXiv2023

OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control

In this work, we introduce OmniDrones, an efficient and flexible platform tailored for reinforcement learning in drone control, built on Nvidia's Omniverse Isaac Sim. It employs a bottom-up design approach that allows users to easily design and experiment with various application scenarios on top of GPU-parallelized simulations. It also offers a range of benchmark tasks, presenting challenges ranging from single-drone hovering to over-actuated system tracking. In summary, we propose an open-sourced drone simulation platform, equipped with an extensive suite of tools for drone learning. It includes 4 drone models, 5 sensor modalities, 4 control modes, over 10 benchmark tasks, and a selection of widely used RL baselines. To showcase the capabilities of OmniDrones and to support future research, we also provide preliminary results on these benchmark tasks. We hope this platform will encourage further studies on applying RL to practical drone systems.

cs.RO↗

arXiv2023

AlphaZero Gomoku

In the past few years, AlphaZero's exceptional capability in mastering intricate board games has garnered considerable interest. Initially designed for the game of Go, this revolutionary algorithm merges deep learning techniques with the Monte Carlo tree search (MCTS) to surpass earlier top-tier methods. In our study, we broaden the use of AlphaZero to Gomoku, an age-old tactical board game also referred to as "Five in a Row." Intriguingly, Gomoku has innate challenges due to a bias towards the initial player, who has a theoretical advantage. To add value, we strive for a balanced game-play. Our tests demonstrate AlphaZero's versatility in adapting to games other than Go. MCTS has become a predominant algorithm for decision processes in intricate scenarios, especially board games. MCTS creates a search tree by examining potential future actions and uses random sampling to predict possible results. By leveraging the best of both worlds, the AlphaZero technique fuses deep learning from Reinforcement Learning with the balancing act of MCTS, establishing a fresh standard in game-playing AI. Its triumph is notably evident in board games such as Go, chess, and shogi.

cs.AI↗