arXiv subjects

Grace Zhang

Publications and source records attributed to Grace Zhang.

11 recordsLinked to original sources

arXiv2026

Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning

Inverse reinforcement learning (IRL) provides a powerful framework for learning from demonstrations. However, real-world tasks often exhibit substantial natural variations (e.g., picking up mugs with varying shapes), making it impractical to collect demonstrations that fully specify a new task under every possible scenario. In practice, while demonstrations for the target task are limited, it is often easier to obtain datasets of heterogeneous but related behaviors. This motivates the problem of few-shot IRL with multi-task demonstrations (FM-IRL), where an agent must learn a new task with substantial variations from only a limited number of target-task demonstrations, together with sufficient demonstrations of related tasks and online agent experience. To do so, we must both recover the expert distribution of the new task and provide guidance when the agent deviates from it. We introduce Multitask discriminator Proximity-Guided IRL (MPG), which learns two complementary reward components: (1) a generalizable discriminator that transfers shared structure across related tasks to identify expert behavior in a new task, and (2) a proximity function that measures how far a state deviates from expert behavior and provides corrective guidance during exploration. We demonstrate the effectiveness of our method on multiple challenging navigation and manipulation tasks under significant variations (e.g., object configurations, table layouts, and initial robot poses), achieving an average success rate of 81.2%, outperforming the strongest per-task baseline by an average of 24.7 percentage points.

cs.LG↗

arXiv2026

Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini

We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage the multimodal capabilities of Gemini to produce embeddings for arbitrary combinations of interleaved inputs across all these modalities that generalize well across a wide variety of tasks. Applying large-scale contrastive learning in a multi-task multi-stage training setup, we achieve state-of-the-art performance on key embedding benchmarks including unimodal, cross-modal, and multimodal retrieval spanning a diverse set of tasks. We show that our embedding model demonstrates strong performance (with a score of 62.9 R@1 on MSCOCO, 68.8 NDCG@10 on Vatex, 69.9 on MTEB multilingual and 84.0 on MTEB Code) across a variety of tasks surpassing the performance of specialized models. These unified capabilities make Gemini Embedding 2 a promising candidate for downstream use cases such as RAG, recommendation and search. Furthermore, its robust zero-shot performance across distinct fields - from astronomy and bioscience to fine arts and the culinary arts - establishes it as a highly reliable, out-of-the-box representation even for specialized domains.

cs.CV↗

arXiv2024

CushSense: Soft, Stretchable, and Comfortable Tactile-Sensing Skin for Physical Human-Robot Interaction

Whole-arm tactile feedback is crucial for robots to ensure safe physical interaction with their surroundings. This paper introduces CushSense, a fabric-based soft and stretchable tactile-sensing skin designed for physical human-robot interaction (pHRI) tasks such as robotic caregiving. Using stretchable fabric and hyper-elastic polymer, CushSense identifies contacts by monitoring capacitive changes due to skin deformation. CushSense is cost-effective ($\sim$US\$7 per taxel) and easy to fabricate. We detail the sensor design and fabrication process and perform characterization, highlighting its high sensing accuracy (relative error of 0.58%) and durability (0.054% accuracy drop after 1000 interactions). We also present a user study underscoring its perceived safety and comfort for the assistive task of limb manipulation. We open source all sensor-related resources on https://emprise.cs.cornell.edu/cushsense.

cs.RO↗

arXiv2023

An Application of Optimal Control Theory to R-Tipping

An application of optimal control theory results in a lower bound on the speed $|\dot{\lambda}(t)|$ that must be attained at least once by any external forcing function that induces tipping in the asymptotically autonomous scalar ODE $\dot{x} = f(x+\lambda(t))$. The value of this critical speed depends on the total arclength $\int_{-\infty}^{\infty} |\dot{\lambda}(t)| dt$ of forcing, and may be interpreted as a safe threshold rate associated to each given arclength, such that if the speed of forcing remains everywhere slower than this, tipping cannot occur. The bound is tight in the sense that there exists a forcing function (continuous but non-smooth) which induces tipping, possesses the required arclength, and never exceeds the threshold speed. Further, the threshold speed is a strictly decreasing function of arclength, thus capturing the abstract trade off between how fast and how far of a minimal disturbance characterizes tipping.

math.DS↗

arXiv2023

QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharing

Multi-task reinforcement learning (MTRL) aims to learn several tasks simultaneously for better sample efficiency than learning them separately. Traditional methods achieve this by sharing parameters or relabeled data between tasks. In this work, we introduce a new framework for sharing behavioral policies across tasks, which can be used in addition to existing MTRL methods. The key idea is to improve each task's off-policy data collection by employing behaviors from other task policies. Selectively sharing helpful behaviors acquired in one task to collect training data for another task can lead to higher-quality trajectories, leading to more sample-efficient MTRL. Thus, we introduce a simple and principled framework called Q-switch mixture of policies (QMP) that selectively shares behavior between different task policies by using the task's Q-function to evaluate and select useful shareable behaviors. We theoretically analyze how QMP improves the sample efficiency of the underlying RL algorithm. Our experiments show that QMP's behavioral policy sharing provides complementary gains over many popular MTRL algorithms and outperforms alternative ways to share behaviors in various manipulation, locomotion, and navigation environments. Videos are available at https://qmp-mtrl.github.io.

cs.LG↗

arXiv2021

CoMPS: Continual Meta Policy Search

We develop a new continual meta-learning method to address challenges in sequential multi-task learning. In this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinforcement learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quickly. We introduce a new method, continual meta-policy search (CoMPS), that removes this limitation by meta-training in an incremental fashion, over each task in a sequence, without revisiting prior tasks. CoMPS continuously repeats two subroutines: learning a new task using RL and using the experience from RL to perform completely offline meta-learning to prepare for subsequent task learning. We find that CoMPS outperforms prior continual learning and off-policy meta-reinforcement methods on several sequences of challenging continuous control tasks.

cs.LG↗

arXiv2021

Policy Transfer across Visual and Dynamics Domain Gaps via Iterative Grounding

The ability to transfer a policy from one environment to another is a promising avenue for efficient robot learning in realistic settings where task supervision is not available. This can allow us to take advantage of environments well suited for training, such as simulators or laboratories, to learn a policy for a real robot in a home or office. To succeed, such policy transfer must overcome both the visual domain gap (e.g. different illumination or background) and the dynamics domain gap (e.g. different robot calibration or modelling error) between source and target environments. However, prior policy transfer approaches either cannot handle a large domain gap or can only address one type of domain gap at a time. In this paper, we propose a novel policy transfer method with iterative "environment grounding", IDAPT, that alternates between (1) directly minimizing both visual and dynamics domain gaps by grounding the source environment in the target environment domains, and (2) training a policy on the grounded source environment. This iterative training progressively aligns the domains between the two environments and adapts the policy to the target environment. Once trained, the policy can be directly executed on the target environment. The empirical results on locomotion and robotic manipulation tasks demonstrate that our approach can effectively transfer a policy across visual and dynamics domain gaps with minimal supervision and interaction with the target environment. Videos and code are available at https://clvrai.com/idapt .

cs.RO↗

arXiv2020

Sudden collapse of magnetic order in oxygen deficient nickelate films

Oxygen vacancies play a crucial role in the control of the electronic, magnetic, ionic, and transport properties of functional oxide perovskites. Rare earth nickelates (RENiO$_{3-x}$) have emerged over the years as a rich platform to study the interplay between the lattice, the electronic structure, and ordered magnetism. In this study, we investigate the evolution of the electronic and magnetic structure in thin films of RENiO$_{3-x}$, using a combination of X-ray absorption spectroscopy and imaging, resonant X-ray scattering, and extended multiplet ligand field theory modeling. We find that oxygen vacancies modify the electronic configuration within the Ni-O orbital manifolds, leading to a dramatic evolution of long-range electronic transport pathways despite the absence of nanoscale phase separation. Remarkably, magnetism is robust to substantial levels of carrier doping, and only a moderate weakening of the $(1/4, 1/4, 1/4)_{pc}$ antiferromagnetic order parameter is observed, whereas the magnetic transition temperature is largely unchanged. Only at a certain point long-range magnetism is abruptly erased without an accompanying structural transition. We propose the progressive disruption of the 3D magnetic superexchange pathways upon introduction of point defects as the mechanism behind the sudden collapse of magnetic order in oxygen-deficient nickelates. Our work demonstrates that, unlike most other oxides, ordered magnetism in RENiO$_{3-x}$ is mostly insensitive to carrier doping. The sudden collapse of ordered magnetism upon oxygen removal may provide a new mechanism for solid-state magneto-ionic switching and new applications in antiferromagnetic spintronics.

cond-mat.str-el↗

arXiv2019

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning

In this paper, we aim to develop a simple and scalable reinforcement learning algorithm that uses standard supervised learning methods as subroutines. Our goal is an algorithm that utilizes only simple and convergent maximum likelihood loss functions, while also being able to leverage off-policy data. Our proposed approach, which we refer to as advantage-weighted regression (AWR), consists of two standard supervised learning steps: one to regress onto target values for a value function, and another to regress onto weighted target actions for the policy. The method is simple and general, can accommodate continuous and discrete actions, and can be implemented in just a few lines of code on top of standard supervised learning methods. We provide a theoretical motivation for AWR and analyze its properties when incorporating off-policy data from experience replay. We evaluate AWR on a suite of standard OpenAI Gym benchmark tasks, and show that it achieves competitive performance compared to a number of well-established state-of-the-art RL algorithms. AWR is also able to acquire more effective policies than most off-policy algorithms when learning from purely static datasets with no additional environmental interactions. Furthermore, we demonstrate our algorithm on challenging continuous control tasks with highly complex simulated characters.

cs.LG↗

arXiv2019

MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies

Humans are able to perform a myriad of sophisticated tasks by drawing upon skills acquired through prior experience. For autonomous agents to have this capability, they must be able to extract reusable skills from past experience that can be recombined in new ways for subsequent tasks. Furthermore, when controlling complex high-dimensional morphologies, such as humanoid bodies, tasks often require coordination of multiple skills simultaneously. Learning discrete primitives for every combination of skills quickly becomes prohibitive. Composable primitives that can be recombined to create a large variety of behaviors can be more suitable for modeling this combinatorial explosion. In this work, we propose multiplicative compositional policies (MCP), a method for learning reusable motor skills that can be composed to produce a range of complex behaviors. Our method factorizes an agent's skills into a collection of primitives, where multiple primitives can be activated simultaneously via multiplicative composition. This flexibility allows the primitives to be transferred and recombined to elicit new behaviors as necessary for novel tasks. We demonstrate that MCP is able to extract composable skills for highly complex simulated characters from pre-training tasks, such as motion imitation, and then reuse these skills to solve challenging continuous control tasks, such as dribbling a soccer ball to a goal, and picking up an object and transporting it to a target location.

cs.LG↗

arXiv2018

Stable Cluster Variables

Richard Eager and Sebastian Franco introduced a change of basis transformation on the F-polynomials of Fomin and Zelevinsky, corresponding to rewriting them in the basis given by fractional brane charges rather than quiver gauge groups. This transformation seems to display a surprising stabilization property, apparently causing the first few terms of the polynomials at each step of the mutation sequence to coincide. Eager and Franco conjecture that this transformation will always cause the polynomials to converge to a formal power series as the number of mutations goes to infinity, at least for quivers possessing certain symmetries and along periodic mutation sequences respecting such symmetries. In this paper, we verify this convergence in the case of the Kronecker and Conifold quivers. We also investigate convergence in the $F_0$ quiver. We provide a combinatorial interpretation for the stable cluster variables in each appropriate case.

math.CO↗