arXiv subjects

Jianxiong Li

Publications and source records attributed to Jianxiong Li.

At least 37 records · Page 2Linked to original sources

arXiv2024

Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learning

Multimodal task specification is essential for enhanced robotic performance, where \textit{Cross-modality Alignment} enables the robot to holistically understand complex task instructions. Directly annotating multimodal instructions for model training proves impractical, due to the sparsity of paired multimodal data. In this study, we demonstrate that by leveraging unimodal instructions abundant in real data, we can effectively teach robots to learn multimodal task specifications. First, we endow the robot with strong \textit{Cross-modality Alignment} capabilities, by pretraining a robotic multimodal encoder using extensive out-of-domain data. Then, we employ two Collapse and Corrupt operations to further bridge the remaining modality gap in the learned multimodal representation. This approach projects different modalities of identical task goal as interchangeable representations, thus enabling accurate robotic operations within a well-aligned multimodal latent space. Evaluation across more than 130 tasks and 4000 evaluations on both simulated LIBERO benchmark and real robot platforms showcases the superior capabilities of our proposed framework, demonstrating significant advantage in overcoming data constraints in robotic learning. Website: zh1hao.wang/Robo_MUTUAL

cs.RO↗

arXiv2024

Query-Policy Misalignment in Preference-Based Reinforcement Learning

Preference-based reinforcement learning (PbRL) provides a natural way to align RL agents' behavior with human desired outcomes, but is often restrained by costly human feedback. To improve feedback efficiency, most existing PbRL methods focus on selecting queries to maximally improve the overall quality of the reward model, but counter-intuitively, we find that this may not necessarily lead to improved performance. To unravel this mystery, we identify a long-neglected issue in the query selection schemes of existing PbRL studies: Query-Policy Misalignment. We show that the seemingly informative queries selected to improve the overall quality of reward model actually may not align with RL agents' interests, thus offering little help on policy learning and eventually resulting in poor feedback efficiency. We show that this issue can be effectively addressed via near on-policy query and a specially designed hybrid experience replay, which together enforce the bidirectional query-policy alignment. Simple yet elegant, our method can be easily incorporated into existing approaches by changing only a few lines of code. We showcase in comprehensive experiments that our method achieves substantial gains in both human feedback and RL sample efficiency, demonstrating the importance of addressing query-policy misalignment in PbRL tasks.

cs.LG↗

arXiv2024

DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learning

Multimodal pretraining is an effective strategy for the trinity of goals of representation learning in autonomous robots: 1) extracting both local and global task progressions; 2) enforcing temporal consistency of visual representation; 3) capturing trajectory-level language grounding. Most existing methods approach these via separate objectives, which often reach sub-optimal solutions. In this paper, we propose a universal unified objective that can simultaneously extract meaningful task progression information from image sequences and seamlessly align them with language instructions. We discover that via implicit preferences, where a visual trajectory inherently aligns better with its corresponding language instruction than mismatched pairs, the popular Bradley-Terry model can transform into representation learning through proper reward reparameterizations. The resulted framework, DecisionNCE, mirrors an InfoNCE-style objective but is distinctively tailored for decision-making tasks, providing an embodied representation learning framework that elegantly extracts both local and global task progression features, with temporal consistency enforced through implicit time contrastive learning, while ensuring trajectory-level instruction grounding via multimodal joint encoding. Evaluation on both simulated and real robots demonstrates that DecisionNCE effectively facilitates diverse downstream policy learning tasks, offering a versatile solution for unified representation and reward learning. Project Page: https://2toinf.github.io/DecisionNCE/

cs.RO↗

arXiv2024

Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Model

Safe offline RL is a promising way to bypass risky online interactions towards safe policy learning. Most existing methods only enforce soft constraints, i.e., constraining safety violations in expectation below thresholds predetermined. This can lead to potentially unsafe outcomes, thus unacceptable in safety-critical scenarios. An alternative is to enforce the hard constraint of zero violation. However, this can be challenging in offline setting, as it needs to strike the right balance among three highly intricate and correlated aspects: safety constraint satisfaction, reward maximization, and behavior regularization imposed by offline datasets. Interestingly, we discover that via reachability analysis of safe-control theory, the hard safety constraint can be equivalently translated to identifying the largest feasible region given the offline dataset. This seamlessly converts the original trilogy problem to a feasibility-dependent objective, i.e., maximizing reward value within the feasible region while minimizing safety risks in the infeasible region. Inspired by these, we propose FISOR (FeasIbility-guided Safe Offline RL), which allows safety constraint adherence, reward maximization, and offline policy learning to be realized via three decoupled processes, while offering strong safety performance and stability. In FISOR, the optimal policy for the translated optimization problem can be derived in a special form of weighted behavior cloning. Thus, we propose a novel energy-guided diffusion model that does not require training a complicated time-dependent classifier to extract the policy, greatly simplifying the training. We compare FISOR against baselines on DSRL benchmark for safe offline RL. Evaluation results show that FISOR is the only method that can guarantee safety satisfaction in all tasks, while achieving top returns in most tasks.

cs.LG↗

arXiv2023

A Fully Data-Driven Approach for Realistic Traffic Signal Control Using Offline Reinforcement Learning

The optimization of traffic signal control (TSC) is critical for an efficient transportation system. In recent years, reinforcement learning (RL) techniques have emerged as a popular approach for TSC and show promising results for highly adaptive control. However, existing RL-based methods suffer from notably poor real-world applicability and hardly have any successful deployments. The reasons for such failures are mostly due to the reliance on over-idealized traffic simulators for policy optimization, as well as using unrealistic fine-grained state observations and reward signals that are not directly obtainable from real-world sensors. In this paper, we propose a fully Data-Driven and simulator-free framework for realistic Traffic Signal Control (D2TSC). Specifically, we combine well-established traffic flow theory with machine learning to construct a reward inference model to infer the reward signals from coarse-grained traffic data. With the inferred rewards, we further propose a sample-efficient offline RL method to enable direct signal control policy learning from historical offline datasets of real-world intersections. To evaluate our approach, we collect historical traffic data from a real-world intersection, and develop a highly customized simulation environment that strictly follows real data characteristics. We demonstrate through extensive experiments that our approach achieves superior performance over conventional and offline RL baselines, and also enjoys much better real-world applicability.

cs.AI↗

arXiv2023

PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning

Offline-to-online reinforcement learning (RL), by combining the benefits of offline pretraining and online finetuning, promises enhanced sample efficiency and policy performance. However, existing methods, effective as they are, suffer from suboptimal performance, limited adaptability, and unsatisfactory computational efficiency. We propose a novel framework, PROTO, which overcomes the aforementioned limitations by augmenting the standard RL objective with an iteratively evolving regularization term. Performing a trust-region-style update, PROTO yields stable initial finetuning and optimal final performance by gradually evolving the regularization term to relax the constraint strength. By adjusting only a few lines of code, PROTO can bridge any offline policy pretraining and standard off-policy RL finetuning to form a powerful offline-to-online RL pathway, birthing great adaptability to diverse methods. Simple yet elegant, PROTO imposes minimal additional computation and enables highly efficient online finetuning. Extensive experiments demonstrate that PROTO achieves superior performance over SOTA baselines, offering an adaptable and efficient offline-to-online RL framework.

cs.LG↗

arXiv2023

A Policy-Guided Imitation Approach for Offline Reinforcement Learning

Offline reinforcement learning (RL) methods can generally be categorized into two types: RL-based and Imitation-based. RL-based methods could in principle enjoy out-of-distribution generalization but suffer from erroneous off-policy evaluation. Imitation-based methods avoid off-policy evaluation but are too conservative to surpass the dataset. In this study, we propose an alternative approach, inheriting the training stability of imitation-style methods while still allowing logical out-of-distribution generalization. We decompose the conventional reward-maximizing policy in offline RL into a guide-policy and an execute-policy. During training, the guide-poicy and execute-policy are learned using only data from the dataset, in a supervised and decoupled manner. During evaluation, the guide-policy guides the execute-policy by telling where it should go so that the reward can be maximized, serving as the \textit{Prophet}. By doing so, our algorithm allows \textit{state-compositionality} from the dataset, rather than \textit{action-compositionality} conducted in prior imitation-style methods. We dumb this new approach Policy-guided Offline RL (\texttt{POR}). \texttt{POR} demonstrates the state-of-the-art performance on D4RL, a standard benchmark for offline RL. We also highlight the benefits of \texttt{POR} in terms of improving with supplementary suboptimal data and easily adapting to new tasks by only changing the guide-poicy.

cs.LG↗

arXiv2023

Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization

Most offline reinforcement learning (RL) methods suffer from the trade-off between improving the policy to surpass the behavior policy and constraining the policy to limit the deviation from the behavior policy as computing $Q$-values using out-of-distribution (OOD) actions will suffer from errors due to distributional shift. The recently proposed \textit{In-sample Learning} paradigm (i.e., IQL), which improves the policy by quantile regression using only data samples, shows great promise because it learns an optimal policy without querying the value function of any unseen actions. However, it remains unclear how this type of method handles the distributional shift in learning the value function. In this work, we make a key finding that the in-sample learning paradigm arises under the \textit{Implicit Value Regularization} (IVR) framework. This gives a deeper understanding of why the in-sample learning paradigm works, i.e., it applies implicit value regularization to the policy. Based on the IVR framework, we further propose two practical algorithms, Sparse $Q$-learning (SQL) and Exponential $Q$-learning (EQL), which adopt the same value regularization used in existing works, but in a complete in-sample manner. Compared with IQL, we find that our algorithms introduce sparsity in learning the value function, making them more robust in noisy data regimes. We also verify the effectiveness of SQL and EQL on D4RL benchmark datasets and show the benefits of in-sample learning by comparing them with CQL in small data regimes.

cs.LG↗

arXiv2023

When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning

In offline reinforcement learning (RL), one detrimental issue to policy learning is the error accumulation of deep Q function in out-of-distribution (OOD) areas. Unfortunately, existing offline RL methods are often over-conservative, inevitably hurting generalization performance outside data distribution. In our study, one interesting observation is that deep Q functions approximate well inside the convex hull of training data. Inspired by this, we propose a new method, DOGE (Distance-sensitive Offline RL with better GEneralization). DOGE marries dataset geometry with deep function approximators in offline RL, and enables exploitation in generalizable OOD areas rather than strictly constraining policy within data distribution. Specifically, DOGE trains a state-conditioned distance function that can be readily plugged into standard actor-critic methods as a policy constraint. Simple yet elegant, our algorithm enjoys better generalization compared to state-of-the-art methods on D4RL benchmarks. Theoretical analysis demonstrates the superiority of our approach to existing methods that are solely based on data distribution or support constraints.

cs.LG↗

arXiv2023

Mind the Gap: Offline Policy Optimization for Imperfect Rewards

Reward function is essential in reinforcement learning (RL), serving as the guiding signal to incentivize agents to solve given tasks, however, is also notoriously difficult to design. In many cases, only imperfect rewards are available, which inflicts substantial performance loss for RL agents. In this study, we propose a unified offline policy optimization approach, \textit{RGM (Reward Gap Minimization)}, which can smartly handle diverse types of imperfect rewards. RGM is formulated as a bi-level optimization problem: the upper layer optimizes a reward correction term that performs visitation distribution matching w.r.t. some expert data; the lower layer solves a pessimistic RL problem with the corrected rewards. By exploiting the duality of the lower layer, we derive a tractable algorithm that enables sampled-based learning without any online interactions. Comprehensive experiments demonstrate that RGM achieves superior performance to existing methods under diverse settings of imperfect rewards. Further, RGM can effectively correct wrong or inconsistent rewards against expert preference and retrieve useful information from biased rewards.

cs.LG↗

arXiv2021

Offline Reinforcement Learning with Soft Behavior Regularization

Most prior approaches to offline reinforcement learning (RL) utilize \textit{behavior regularization}, typically augmenting existing off-policy actor critic algorithms with a penalty measuring divergence between the policy and the offline data. However, these approaches lack guaranteed performance improvement over the behavior policy. In this work, we start from the performance difference between the learned policy and the behavior policy, we derive a new policy learning objective that can be used in the offline setting, which corresponds to the advantage function value of the behavior policy, multiplying by a state-marginal density ratio. We propose a practical way to compute the density ratio and demonstrate its equivalence to a state-dependent behavior regularization. Unlike state-independent regularization used in prior approaches, this \textit{soft} regularization allows more freedom of policy deviation at high confidence states, leading to better performance and stability. We thus term our resulting algorithm Soft Behavior-regularized Actor Critic (SBAC). Our experimental results show that SBAC matches or outperforms the state-of-the-art on a set of continuous control locomotion and manipulation tasks.

cs.LG↗

arXiv2021

Strong-field control of plasmonic properties in core-shell nanoparticles

The strong-field control of plasmonic nanosystems opens up new perspectives for nonlinear plasmonic spectroscopy and petahertz electronics. Questions, however, remain regarding the nature of nonlinear light-matter interactions at sub-wavelength spatial and ultrafast temporal scales. Addressing this challenge, we investigated the strong-field control of the plasmonic response of Au nanoshells with a SiO$_2$ core to an intense laser pulse. We show that the photoelectron energy spectrum from these core-shell nanoparticles displays a striking transition between the weak and strong-field regime. This observed transition agrees with the prediction of our modified Mie-theory simulation that incorporates the nonlinear dielectric nanoshell response. The demonstrated intensity-dependent optical control of the plasmonic response in prototypical core-shell nanoparticles paves the way towards ultrafast switching and opto-electronic signal modulation with more complex nanostructures.

physics.optics↗

arXiv2021

Electrically-controlled digital metasurface device for light projection displays

Light projection displays play an increasingly important role in our modern life. Core projection systems including liquid crystal displays and digital micromirror devices can impose spatial light modulation and actively shape light waves. Recently, the advent of metasurfaces has revolutionized design concepts in nanophotonics, enabling a new family of optical elements with exceptional degrees of freedom. Here, we demonstrate a light projection display technology based on optical metasurfaces, called digital metasurface device (DMSD). Each metasurface pixel in a DMSD is electrically reconfigurable with well-controlled programmability and addressability. The DMSD can not only continuously modulate the intensity of light with high contrast, but also shape the wavefront of light generated by each metasurface pixel and dynamically switch between arbitrary holographic patterns. Our approach will pave an avenue towards the development of customized light projection devices. It will also drive the field of dynamic optical metasurfaces with fresh momentum and inspiring concepts.

physics.optics↗

arXiv2021

A Reusable Metasurface Template

Metasurfaces have revolutionized the design concepts for opticalcomponents, fostering an excitingfield offlat optics. Thanks to theflat and ultrathinnature, metasurfaces possess unique advantages over conventional optical components,such as lightweight, high compatibility, among others. To comply with potential appli-cations, further research endeavors need to be exerted for advancing the reusability andpracticality of metasurfaces. In this work, we demonstrate a reusable template approachto achieve optical multifunctionality with metasurfaces, utilizing both the geometric andpropagation phases to shape light waves. In principle, such a metasurface template can beemployed infinite times to enable a large variety of optical functions. As proof-of-conceptexperiments, we demonstrate metalensing, holography, and vortex beam shaping. Ourwork will leverage the high scalability aspects of metasurface devices for practical applications.

physics.optics↗

arXiv2021

Dynamic Janus Metasurfaces in the Visible Spectral Region

Janus monolayers have long been captivated as a popular notion for breaking in-plane and out-of-plane structural symmetry. Originated from chemistry and materials science, the concept of Janus functions have been recently extended to ultrathin metasurfaces by arranging meta-atoms asymmetrically with respect to the propagation or polarization direction of the incident light. However, such metasurfaces are intrinsically static and the information they carry can be straightforwardly decrypted by scanning the incident light directions and polarization states once the devices are fabricated. In this Letter, we present a dynamic Janus metasurface scheme in the visible spectral region. In each super unit cell, three plasmonic pixels are categorized into two sets. One set contains a magnesium nanorod and a gold nanorod that are orthogonally oriented with respect to each other, working as counter pixels. The other set only contains a magnesium nanorod. The effective pixels on the Janus metasurface can be reversibly regulated by hydrogenation/dehydrogenation of the magnesium nanorods. Such dynamic controllability at visible frequencies allows for flat optical elements with novel functionalities including beam steering, bifocal lensing, holographic encryption, and dual optical function switching.

physics.optics↗

arXiv2021

Magnesium-Based Metasurfaces for Dual-Function Switching between Dynamic Holography and Dynamic Color Display

Metasurface-based color display and holography have greatly advanced the state of the art display technologies. To further enrich the metasurface functionalities, recently a lot of research endeavors have been made to combine these two display functions within a single device. However, so far such metasurfaces have remained static and lack tunability once the devices are fabricated. In this work, we demonstrate a dynamic dual-function metasurface device at visible frequencies. It allows for switching between dynamic holography and dynamic color display, taking advantage of the reversible phase transition of magnesium through hydrogenation and dehydrogenation. Spatially arranged stepwise nanocavity pixels are employed to accurately control the amplitude and phase of light, enabling the generation of high-quality color prints and holograms. Our work represents a paradigm toward compact and multifunctional optical elements for future display technologies.

physics.optics↗

arXiv2021

Generation of Switchable Singular Beams with Dynamic Metasurfaces

Singular beams have attracted great attention due to their optical properties and broad applications from light manipulation to optical communications. However, there has been a lack of practical schemes with which to achieve switchable singular beams with sub-wavelength resolution using ultrathin and flat optical devices. In this work, we demonstrate the generation of switchable vector and vortex beams utilizing dynamic metasurfaces at visible frequencies. The dynamic functionality of the metasurface pixels is enabled by the utilization of magnesium nanorods, which possess plasmonic reconfigurability upon hydrogenation and dehydrogenation. We show that switchable vector beams of different polarization states and switchable vortex beams of different topological charges can be implemented through simple hydrogenation and dehydrogenation of the same metasurfaces. Furthermore, we demonstrate a two-cascade metasurface scheme for holographic pattern switching, taking inspiration from orbital angular momentum-shift keying. Our work provides an additional degree of freedom to develop high-security optical elements for anti-counterfeiting applications.

physics.optics↗

arXiv2021

Addressable metasurfaces for dynamic holography and optical information encryption

Metasurfaces enable manipulation of light propagation at an unprecedented level, benefitting from a number of merits unavailable to conventional optical elements, such as ultracompactness, precise phase and polarization control at deep subwavelength scale, and multifunctionalities. Recent progress in this field has witnessed a plethora of functional metasurfaces, ranging from lenses and vortex beam generation to holography. However, research endeavors have been mainly devoted to static devices, exploiting only a glimpse of opportunities that metasurfaces can offer. We demonstrate a dynamic metasurface platform, which allows independent manipulation of addressable subwavelength pixels at visible frequencies through controlled chemical reactions. In particular, we create dynamic metasurface holograms for advanced optical information processing and encryption. Plasmonic nanorods tailored to exhibit hierarchical reaction kinetics upon hydrogenation/dehydrogenation constitute addressable pixels in multiplexed metasurfaces. The helicity of light, hydrogen, oxygen, and reaction duration serve as multiple keys to encrypt the metasurfaces. One single metasurface can be deciphered into manifold messages with customized keys, featuring a compact data storage scheme as well as a high level of information security. Our work suggests a novel route to protect and transmit classified data, where highly restricted access of information is imposed.

physics.optics↗