arXiv subjects

Chen Wang

Publications and source records attributed to Chen Wang.

At least 361 records · Page 20Linked to original sources

arXiv2024

Map It Anywhere (MIA): Empowering Bird's Eye View Mapping using Large-scale Public Data

Top-down Bird's Eye View (BEV) maps are a popular representation for ground robot navigation due to their richness and flexibility for downstream tasks. While recent methods have shown promise for predicting BEV maps from First-Person View (FPV) images, their generalizability is limited to small regions captured by current autonomous vehicle-based datasets. In this context, we show that a more scalable approach towards generalizable map prediction can be enabled by using two large-scale crowd-sourced mapping platforms, Mapillary for FPV images and OpenStreetMap for BEV semantic maps. We introduce Map It Anywhere (MIA), a data engine that enables seamless curation and modeling of labeled map prediction data from existing open-source map platforms. Using our MIA data engine, we display the ease of automatically collecting a dataset of 1.2 million pairs of FPV images & BEV maps encompassing diverse geographies, landscapes, environmental factors, camera models & capture scenarios. We further train a simple camera model-agnostic model on this data for BEV map prediction. Extensive evaluations using established benchmarks and our dataset show that the data curated by MIA enables effective pretraining for generalizable BEV map prediction, with zero-shot performance far exceeding baselines trained on existing datasets by 35%. Our analysis highlights the promise of using large-scale public maps for developing & testing generalizable BEV perception, paving the way for more robust autonomous navigation. Website: https://mapitanywhere.github.io/

cs.CV↗

arXiv2024

Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction

Large language models (LLMs) have been driving a new wave of interactive AI applications across numerous domains. However, efficiently serving LLM inference requests is challenging due to their unpredictable execution times originating from the autoregressive nature of generative models. Existing LLM serving systems exploit first-come-first-serve (FCFS) scheduling, suffering from head-of-line blocking issues. To address the non-deterministic nature of LLMs and enable efficient interactive LLM serving, we present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths. Our open-source SSJF implementation does not require changes to memory management or batching strategies. Evaluations on real-world datasets and production workload traces show that SSJF reduces average job completion times by 30.5-39.6% and increases throughput by 2.2-3.6x compared to FCFS schedulers, across no batching, dynamic batching, and continuous batching settings.

cs.DC↗

arXiv2024

A Layered Architecture for Developing and Enhancing Capabilities in Large Language Model-based Software Systems

Significant efforts has been made to expand the use of Large Language Models (LLMs) beyond basic language tasks. While the generalizability and versatility of LLMs have enabled widespread adoption, evolving demands in application development often exceed their native capabilities. Meeting these demands may involve a diverse set of methods, such as enhancing creativity through either inference temperature adjustments or creativity-provoking prompts. Selecting the right approach is critical, as different methods lead to trade-offs in engineering complexity, scalability, and operational costs. This paper introduces a layered architecture that organizes LLM software system development into distinct layers, each characterized by specific attributes. By aligning capabilities with these layers, the framework encourages the systematic implementation of capabilities in effective and efficient ways that ultimately supports desired functionalities and qualities. Through practical case studies, we illustrate the utility of the framework. This work offers developers actionable insights for selecting suitable technologies in LLM-based software system development, promoting robustness and scalability.

cs.SE↗

arXiv2024

Membership Inference Attack against Long-Context Large Language Models

Recent advances in Large Language Models (LLMs) have enabled them to overcome their context window limitations, and demonstrate exceptional retrieval and reasoning capacities on longer context. Quesion-answering systems augmented with Long-Context Language Models (LCLMs) can automatically search massive external data and incorporate it into their contexts, enabling faithful predictions and reducing issues such as hallucinations and knowledge staleness. Existing studies targeting LCLMs mainly concentrate on addressing the so-called lost-in-the-middle problem or improving the inference effiencicy, leaving their privacy risks largely unexplored. In this paper, we aim to bridge this gap and argue that integrating all information into the long context makes it a repository of sensitive information, which often contains private data such as medical records or personal identities. We further investigate the membership privacy within LCLMs external context, with the aim of determining whether a given document or sequence is included in the LCLMs context. Our basic idea is that if a document lies in the context, it will exhibit a low generation loss or a high degree of semantic similarity to the contents generated by LCLMs. We for the first time propose six membership inference attack (MIA) strategies tailored for LCLMs and conduct extensive experiments on various popular models. Empirical results demonstrate that our attacks can accurately infer membership status in most cases, e.g., 90.66% attack F1-score on Multi-document QA datasets with LongChat-7b-v1.5-32k, highlighting significant risks of membership leakage within LCLMs input contexts. Furthermore, we examine the underlying reasons why LCLMs are susceptible to revealing such membership information.

cs.CL↗

arXiv2024

Non-Hermitian Effects in Dicke models

The Dicke model, which describes the collective interaction between an ensemble of atoms and a single-mode photon field, serves as a fundamental framework for studying light-matter interactions and quantum electrodynamic phenomena. In this work, we investigate the manifestation of non-Hermitian effects in a generalized Dicke model, where two dissipative atom ensembles interact with a single-mode photon field. By applying the Holstein-Primakoff transformation, we explore the system in the semiclassical limit as a non-Hermitian Dicke model, revealing rich exceptional points (EPs) and diabolic points in such a system. We find that, by introducing the nonlinear saturation gain into an atomic ensemble, higher-order EP can be induced, leading to intriguing properties. Furthermore, if the system is extended to a one-dimensional chain, then the band topology will interplay with the non-Hermitian effect. In the quantum regime, we explore the quantum signature of EPs, noting that the conditions for their emergence are influenced by discrete photon numbers. We further study the transition from photon anti-bunching to bunching at a steady state, driven by non-Hermitian dynamics. Our findings deepen the understanding of non-Hermitian physics in light-matter interaction which is instructive for the design of advanced photonic and quantum systems.

quant-ph↗

arXiv2024

ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep is expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observations. We present system implementations on a wheeled single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models. Website at https://rekep-robot.github.io/.

cs.RO↗

arXiv2024

Object-Centric Dexterous Manipulation from Human Motion Data

Manipulating objects to achieve desired goal states is a basic but important skill for dexterous manipulation. Human hand motions demonstrate proficient manipulation capability, providing valuable data for training robots with multi-finger hands. Despite this potential, substantial challenges arise due to the embodiment gap between human and robot hands. In this work, we introduce a hierarchical policy learning framework that uses human hand motion data for training object-centric dexterous robot manipulation. At the core of our method is a high-level trajectory generative model, learned with a large-scale human hand motion capture dataset, to synthesize human-like wrist motions conditioned on the desired object goal states. Guided by the generated wrist motions, deep reinforcement learning is further used to train a low-level finger controller that is grounded in the robot's embodiment to physically interact with the object to achieve the goal. Through extensive evaluation across 10 household objects, our approach not only demonstrates superior performance but also showcases generalization capability to novel object geometries and goal states. Furthermore, we transfer the learned policies from simulation to a real-world bimanual dexterous robot system, further demonstrating its applicability in real-world scenarios. Project website: https://cypypccpy.github.io/obj-dex.github.io/.

cs.RO↗

arXiv2024

24 days-stable CNOT-gate on fluxonium qubits with over 99.9% fidelity

Fluxonium qubit is a promising building block for quantum information processing due to its long coherence time and strong anharmonicity. In this paper, we realize a 60 ns direct CNOT-gate on two inductively-coupled fluxonium qubits using selective darkening approach, resulting in a gate fidelity as high as 99.94%. The fidelity remains above 99.9% for 24 days without any recalibration between randomized benchmarking measurements. Compared with the 99.96% fidelity of a 60 ns identity gate, our data brings the investigation of the non-decoherence-related errors during gate operations down to $2 \times 10^{-4}$. The present result adds a simple and robust two-qubit gate into the still relatively small family of "the beyond three nines" demonstrations on superconducting qubits.

quant-ph↗

arXiv2024

Revisit Many-body Interaction Heat Current and Thermal Conductivity Calculation in Moment Tensor Potential/LAMMPS Interface

The definition of heat current operator for systems for non-pairwise additive interactions and its impact on related lattice thermal conductivity ($κ_{L}$) via molecular dynamics simulation (MD) are ambiguous and controversial when migrating from conventional empirical potential models to machine learning potential (MLP) models. Empirical model descriptions are often limited to three- to four-body interaction while a sophisticated representation of the many-body physics could be resembled in MLPs. Herein, we study and compare the significance of many-body interaction to the heat current computation in one of the most popular MLP models, the Moment Tensor Potential (MTP). Non-equilibrium MD simulations and equilibrium MD simulations among four different materials, $PbTe$, amorphous $Sc_{0.2}Sb_{2}Te_{3}$, graphene, and $BAs$, were performed. We found inconsistency between the simulation thermostat and its implemented heat current operator in our non-equilibrium MD results which violate law of energy conservation and suggest a need for revision. We revisit the virial stress tensor expression within the calculator and identified the lack of a generalised many-body heat current description in it. We uncover the influence of the modified heat current formula that could alter the $κ_{L}$ results 29% to 64% using the equilibrium MD computational approach. Our work demonstrates the importance of a many-body description during thermal analysis in MD simulations when MLPs are in concern. This work sheds light on a better understanding of the relationship between interatomic interaction and its heat transport mechanism.

cond-mat.mtrl-sci↗

arXiv2024

EgoMimic: Scaling Imitation Learning via Egocentric Video

The scale and diversity of demonstration data required for imitation learning is a significant challenge. We present EgoMimic, a full-stack framework which scales manipulation via human embodiment data, specifically egocentric human videos paired with 3D hand tracking. EgoMimic achieves this through: (1) a system to capture human embodiment data using the ergonomic Project Aria glasses, (2) a low-cost bimanual manipulator that minimizes the kinematic gap to human data, (3) cross-domain data alignment techniques, and (4) an imitation learning architecture that co-trains on human and robot data. Compared to prior works that only extract high-level intent from human videos, our approach treats human and robot data equally as embodied demonstration data and learns a unified policy from both data sources. EgoMimic achieves significant improvement on a diverse set of long-horizon, single-arm and bimanual manipulation tasks over state-of-the-art imitation learning methods and enables generalization to entirely new scenes. Finally, we show a favorable scaling trend for EgoMimic, where adding 1 hour of additional hand data is significantly more valuable than 1 hour of additional robot data. Videos and additional information can be found at https://egomimic.github.io/

cs.RO↗

arXiv2024

Observation of Anderson localization transitions in a two-dimensional conjugated metal-organic framework

Anderson localization transitions are a universal quantum phenomenon sensitive to the disorder and dimensionality of electronic systems. Over the past decades, this intriguing topic has inspired overwhelmingly more theoretical studies than experimental verifications due to the difficulty of controlling a material's disorder or dimensionality without modifying its fundamental electronic properties. Organic crystals with their rich disorders would be terrific playgrounds to investigate such disorder-driven phase transitions except for their low conductivities which usually prohibit low-temperature measurements. Here, we conduct systematic transport experiments in mesoscopic devices made with copper benzenehexathiol thin films across a wide range of thicknesses. We find metal-insulator transitions both among three-dimensional samples with different disorder strengths and between three-dimensional and quasi-two-dimensional samples. Temperature-dependence analysis of the conductivities corroborates the dimensionality crossover. Moreover, our theoretical modeling provides a basis for understanding both types of metal-insulator transitions within the framework of Anderson localization transitions. Our findings establish for the first time that organic crystals such as conductive metal-organic frameworks can exhibit such quantum interference effects. With organic materials' versatile chemical designs and crystalline structures, our work opens new avenues to search for novel quantum phenomena in organic material platforms.

cond-mat.mes-hall↗

arXiv2024

PhysORD: A Neuro-Symbolic Approach for Physics-infused Motion Prediction in Off-road Driving

Motion prediction is critical for autonomous off-road driving, however, it presents significantly more challenges than on-road driving because of the complex interaction between the vehicle and the terrain. Traditional physics-based approaches encounter difficulties in accurately modeling dynamic systems and external disturbance. In contrast, data-driven neural networks require extensive datasets and struggle with explicitly capturing the fundamental physical laws, which can easily lead to poor generalization. By merging the advantages of both methods, neuro-symbolic approaches present a promising direction. These methods embed physical laws into neural models, potentially significantly improving generalization capabilities. However, no prior works were evaluated in real-world settings for off-road driving. To bridge this gap, we present PhysORD, a neural-symbolic approach integrating the conservation law, i.e., the Euler-Lagrange equation, into data-driven neural models for motion prediction in off-road driving. Our experiments showed that PhysORD can accurately predict vehicle motion and tolerate external disturbance by modeling uncertainties. The learned dynamics model achieves 46.7% higher accuracy using only 3.1% of the parameters compared to data-driven methods, demonstrating the data efficiency and superior generalization ability of our neural-symbolic method.

cs.RO↗

arXiv2024

Automated Creation of Digital Cousins for Robust Policy Learning

Training robot policies in the real world can be unsafe, costly, and difficult to scale. Simulation serves as an inexpensive and potentially limitless source of training data, but suffers from the semantics and physics disparity between simulated and real-world environments. These discrepancies can be minimized by training in digital twins, which serve as virtual replicas of a real scene but are expensive to generate and cannot produce cross-domain generalization. To address these limitations, we propose the concept of digital cousins, a virtual asset or scene that, unlike a digital twin, does not explicitly model a real-world counterpart but still exhibits similar geometric and semantic affordances. As a result, digital cousins simultaneously reduce the cost of generating an analogous virtual environment while also facilitating better robustness during sim-to-real domain transfer by providing a distribution of similar training scenes. Leveraging digital cousins, we introduce a novel method for their automated creation, and propose a fully automated real-to-sim-to-real pipeline for generating fully interactive scenes and training robot policies that can be deployed zero-shot in the original scene. We find that digital cousin scenes that preserve geometric and semantic affordances can be produced automatically, and can be used to train policies that outperform policies trained on digital twins, achieving 90% vs. 25% success rates under zero-shot sim-to-real transfer. Additional details are available at https://digital-cousins.github.io/.

cs.RO↗

arXiv2024

Revisiting the Velocity Dispersion-Size Relation in Molecular Cloud Structures

Structures in molecular ISM are observed to follow a power-law relation between the velocity dispersion and spatial size, known as Larson's first relation, which is often attributed to the turbulent nature of molecular ISM and imprints the dynamics of molecular cloud structures. Using the ${}^{13}\mathrm{CO}~(J=1-0)$ data from the Milky Way Imaging Scroll Painting survey, we built a sample with 360 structures having relatively accurate distances obtained from either the reddened background stars with Gaia parallaxes or associated maser parallaxes, spanning from $0.4$ to $\sim 15~\mathrm{kpc}$. Using this sample and about 0.3 million pixels, we analyzed the correlations between velocity dispersion, surface/column density, and spatial scales. Our structure-wise results show power-law indices smaller than 0.5 in both the $σ_v$-$R_{\mathrm{eff}}$ and $σ_v$-$R_{\mathrm{eff}} \cdot Σ$ relations. In the pixel-wise results, the $σ_v^{\mathrm{pix}}$ is statistically scaling with the beam physical size ($R_{\mathrm{s}} \equiv ΘD/2$) in form of $σ_v^{\mathrm{pix}} \propto R_{\mathrm{s}}^{0.43 \pm 0.03}$. Meanwhile, $σ_v^{\mathrm{pix}}$ in the inner Galaxy is statistically larger than the outer side. We also analyzed correlations between $σ_v^{\mathrm{pix}}$ and the $\mathrm{H_2}$ column density $N(\mathrm{H_2})$, finding that $σ_v^{\mathrm{pix}}$ stops increasing with $N(\mathrm{H_2})$ after $\gtrsim 10^{22}~{\mathrm{cm^{-2}}}$. The structures with and without high-column-density ($> 10^{22}~\mathrm{cm^{-2}}$) pixels show different $σ_v^{\mathrm{pix}} \propto N(\mathrm{H_2})^ξ$ relations, where the mean (std) $ξ$ values are $0.38~(0.14)$ and $0.62~(0.27)$, respectively.

astro-ph.GA↗

arXiv2024

Stripped helium-star and compact object binaries in coeval populations -- predictions based on detailed binary evolution models

Massive stars mainly form in close binaries, where their mutual interactions can profoundly alter their evolutionary paths. Evolved binaries consisting of a massive OB-type main-sequence star with a stripped helium star or a compact companion represent a crucial stage in the evolution towards double compact objects, whose mergers are (potentially) detectable via gravitational waves. The recent detection of X-ray quiet OB+black hole binaries and OB+stripped helium star binaries has set the stage for discovering more of these systems in the near future. In this work, based on 3670 detailed binary-evolution models and using empirical distributions of initial binary parameters, we compute the expected population of such evolved massive binaries in coeval stellar populations, including stars in star clusters and in galaxies with starburst activities, for ages up to 100 Myr. Our results are vividly illustrated in an animation that shows the evolution of these binaries in the color-magnitude diagram over time. We find that the number of OB+black hole binaries peaks around 10 Myr, and OB+neutron star binaries are most abundant at approximately 20 Myr. Both black holes and neutron stars can potentially be found in populations with ages up to 90 Myr. Additionally, we analyze the properties of such binaries at specific ages. We find that OB+helium stars and OB+black hole binaries are likely to be identifiable as single-lined spectroscopic binaries. Our research serves as a guide for future observational efforts to discover such binaries in young star clusters and starburst environments.

astro-ph.SR↗

arXiv2024

Do LLMs Have the Generalization Ability in Conducting Causal Inference?

In causal inference, generalization capability refers to the ability to conduct causal inference methods on new data to estimate the causal-effect between unknown phenomenon, which is crucial for expanding the boundaries of knowledge. Studies have evaluated the causal inference capabilities of Large Language Models (LLMs) concerning known phenomena, yet the generalization capabilities of LLMs concerning unseen phenomena remain unexplored. In this paper, we selected four tasks: Causal Path Discovery (CP), Backdoor Adjustment (BA), Factual Inference (FI), and Counterfactual Inference (CI) as representatives of causal inference tasks. To generate evaluation questions about previously unseen phenomena in new data on the four tasks, we propose a benchmark generation framework, which employs randomly generated graphs and node names to formulate questions within hypothetical new causal scenarios. Based on this framework, we compile a benchmark dataset of varying levels of question complexity. We extensively tested the generalization capabilities of five leading LLMs across four tasks. Experiment results reveal that while LLMs exhibit good generalization performance in solving simple CP, FI, and complex CI questions, they encounter difficulties when tackling BA questions and face obvious performance fluctuations as the problem complexity changes. Furthermore, when the names of phenomena incorporate existing terms, even if these names are entirely novel, their generalization performance can still be hindered by interference from familiar terms.

cs.CL↗

arXiv2024

TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction

Learning in simulation and transferring the learned policy to the real world has the potential to enable generalist robots. The key challenge of this approach is to address simulation-to-reality (sim-to-real) gaps. Previous methods often require domain-specific knowledge a priori. We argue that a straightforward way to obtain such knowledge is by asking humans to observe and assist robot policy execution in the real world. The robots can then learn from humans to close various sim-to-real gaps. We propose TRANSIC, a data-driven approach to enable successful sim-to-real transfer based on a human-in-the-loop framework. TRANSIC allows humans to augment simulation policies to overcome various unmodeled sim-to-real gaps holistically through intervention and online correction. Residual policies can be learned from human corrections and integrated with simulation policies for autonomous execution. We show that our approach can achieve successful sim-to-real transfer in complex and contact-rich manipulation tasks such as furniture assembly. Through synergistic integration of policies learned in simulation and from humans, TRANSIC is effective as a holistic approach to addressing various, often coexisting sim-to-real gaps. It displays attractive properties such as scaling with human effort. Videos and code are available at https://transic-robot.github.io/

cs.RO↗

arXiv2024

Learning-on-the-Drive: Self-supervised Adaptation of Visual Offroad Traversability Models

Autonomous offroad driving is essential for applications like emergency rescue, military operations, and agriculture. Despite progress, systems struggle with high-speed vehicles exceeding 10m/s due to the need for accurate long-range (> 50m) perception for safe navigation. Current approaches are limited by sensor constraints; LiDAR-based methods offer precise short-range data but are noisy beyond 30m, while visual models provide dense long-range measurements but falter with unseen scenarios. To overcome these issues, we introduce ALTER, a learning-on-the-drive perception framework that leverages both sensor types. ALTER uses a self-supervised visual model to learn and adapt from near-range LiDAR measurements, improving long-range prediction in new environments without manual labeling. It also includes a model selection module for better sensor failure response and adaptability to known environments. Testing in two real-world settings showed on average 43.4% better traversability prediction than LiDAR-only and 164% over non-adaptive state-of-the-art (SOTA) visual semantic methods after 45 seconds of online learning.

cs.RO↗