Searcharxiv⌕ Search

arXiv subjects

Tian Xu

Publications and source records attributed to Tian Xu.

At least 37 records · Page 2Linked to original sources
arXiv2024

On the Bär-Hijazi-Lott invariant for the Dirac operator and a spinorial proof of the Yamabe problem

Let $M$ be a closed spin manifold of dimension $m\geq6$ equipped with a Riemannian metric $\ig$ and a spin structure $\sa$. Let $\lm_1^+(\tilde\ig)$ be the smallest positive eigenvalue of the Dirac operator $D_{\tilde\ig}$ on $M$ with respect to a metric $\tilde\ig$ conformal to $\ig$. The Bär-Hijazi-Lott invariant is defined by $\lm_{min}^+(M,\ig,\sa)=\inf_{\tilde\ig\in[\ig]}\lm_1^+(\tilde\ig)\Vol(M,\tilde\ig)^\frac{1}{m}$. In this paper, we show that \[ \lm_{min}^+(M,\ig,\sa)<\lm_{min}^+(S^m,\ig_{S^m},\sa_{S^m})=\frac m2\Vol(S^m,\ig_{S^m})^{\frac1m} \] provided that $\ig$ is not locally conformally flat. This estimate is a spinorial analogue to an estimate by T. Aubin, solving the Yamabe problem in this setting.

math.DG↗
arXiv2024

ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models

Reinforcement Learning from Human Feedback (RLHF) is key to aligning Large Language Models (LLMs), typically paired with the Proximal Policy Optimization (PPO) algorithm. While PPO is a powerful method designed for general reinforcement learning tasks, it is overly sophisticated for LLMs, leading to laborious hyper-parameter tuning and significant computation burdens. To make RLHF efficient, we present ReMax, which leverages 3 properties of RLHF: fast simulation, deterministic transitions, and trajectory-level rewards. These properties are not exploited in PPO, making it less suitable for RLHF. Building on the renowned REINFORCE algorithm, ReMax does not require training an additional value model as in PPO and is further enhanced with a new variance reduction technique. ReMax offers several benefits over PPO: it is simpler to implement, eliminates more than 4 hyper-parameters in PPO, reduces GPU memory usage, and shortens training time. ReMax can save about 46% GPU memory than PPO when training a 7B model and enables training on A800-80GB GPUs without the memory-saving offloading technique needed by PPO. Applying ReMax to a Mistral-7B model resulted in a 94.78% win rate on the AlpacaEval leaderboard and a 7.739 score on MT-bench, setting a new SOTA for open-source 7B models. These results show the effectiveness of ReMax while addressing the limitations of PPO in LLMs.

cs.LG↗
arXiv2024

ShennongAlpha: an AI-driven sharing and collaboration platform for intelligent curation, acquisition, and translation of natural medicinal material knowledge

Natural Medicinal Materials (NMMs) have a long history of global clinical applications and a wealth of records and knowledge. Although NMMs are a major source for drug discovery and clinical application, the utilization and sharing of NMM knowledge face crucial challenges, including the standardized description of critical information, efficient curation and acquisition, and language barriers. To address these, we developed ShennongAlpha, an AI-driven sharing and collaboration platform for intelligent knowledge curation, acquisition, and translation. For standardized knowledge curation, the platform introduced a Systematic Nomenclature to enable accurate differentiation and identification of NMMs. More than fourteen thousand Chinese NMMs have been curated into the platform along with their knowledge. Furthermore, the platform pioneered chat-based knowledge acquisition, standardized machine translation, and collaborative knowledge updating. Together, our study represents the first major advance in leveraging AI to empower NMM knowledge sharing, which not only marks a novel application of AI for Science, but also will significantly benefit the global biomedical, pharmaceutical, physician, and patient communities.

cs.AI↗
arXiv2024

Singular Solutions for the Conformal Dirac-Einstein Problem on the Sphere

In this paper we investigate the existence of singular solutions to the conformal Dirac-Einstein system. Because of its conformal invariance, there are many similarities with the classical construction of singular solutions for the Yamabe problem. We construct here a family of singular solutions, on the three-dimensional sphere, having exactly two singularities.

math.DG↗
arXiv2024

Policy Optimization in RLHF: The Impact of Out-of-preference Data

Aligning intelligent agents with human preferences and values is important. This paper examines two popular alignment methods: Direct Preference Optimization (DPO) and Reward-Model-Based Policy Optimization (RMB-PO). A variant of RMB-PO, referred to as RMB-PO+ is also considered. These methods, either explicitly or implicitly, learn a reward model from preference data and differ in the data used for policy optimization to unlock the generalization ability of the reward model. In particular, compared with DPO, RMB-PO additionally uses policy-generated data, and RMB-PO+ further leverages new, preference-free data. We examine the impact of such out-of-preference data. Our study, conducted through controlled and synthetic experiments, demonstrates that DPO performs poorly, whereas RMB-PO+ performs the best. In particular, even when providing the policy model with a good feature representation, we find that policy optimization with adequate out-of-preference data significantly improves performance by harnessing the reward model's generalization capabilities.

cs.LG↗
arXiv2023

Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning

Learning a precise dynamics model can be crucial for offline reinforcement learning, which, unfortunately, has been found to be quite challenging. Dynamics models that are learned by fitting historical transitions often struggle to generalize to unseen transitions. In this study, we identify a hidden but pivotal factor termed dynamics reward that remains consistent across transitions, offering a pathway to better generalization. Therefore, we propose the idea of reward-consistent dynamics models: any trajectory generated by the dynamics model should maximize the dynamics reward derived from the data. We implement this idea as the MOREC (Model-based Offline reinforcement learning with Reward Consistency) method, which can be seamlessly integrated into previous offline model-based reinforcement learning (MBRL) methods. MOREC learns a generalizable dynamics reward function from offline data, which is subsequently employed as a transition filter in any offline MBRL method: when generating transitions, the dynamics model generates a batch of transitions and selects the one with the highest dynamics reward value. On a synthetic task, we visualize that MOREC has a strong generalization ability and can surprisingly recover some distant unseen transitions. On 21 offline tasks in D4RL and NeoRL benchmarks, MOREC improves the previous state-of-the-art performance by a significant margin, i.e., 4.6% on D4RL tasks and 25.9% on NeoRL tasks. Notably, MOREC is the first method that can achieve above 95% online RL performance in 6 out of 12 D4RL tasks and 3 out of 9 NeoRL tasks.

cs.LG↗
arXiv2023

Provably Efficient Adversarial Imitation Learning with Unknown Transitions

Imitation learning (IL) has proven to be an effective method for learning good policies from expert demonstrations. Adversarial imitation learning (AIL), a subset of IL methods, is particularly promising, but its theoretical foundation in the presence of unknown transitions has yet to be fully developed. This paper explores the theoretical underpinnings of AIL in this context, where the stochastic and uncertain nature of environment transitions presents a challenge. We examine the expert sample complexity and interaction complexity required to recover good policies. To this end, we establish a framework connecting reward-free exploration and AIL, and propose an algorithm, MB-TAIL, that achieves the minimax optimal expert sample complexity of $\widetilde{O} (H^{3/2} |S|/\varepsilon)$ and interaction complexity of $\widetilde{O} (H^{3} |S|^2 |A|/\varepsilon^2)$. Here, $H$ represents the planning horizon, $|S|$ is the state space size, $|A|$ is the action space size, and $\varepsilon$ is the desired imitation gap. MB-TAIL is the first algorithm to achieve this level of expert sample complexity in the unknown transition setting and improves upon the interaction complexity of the best-known algorithm, OAL, by $O(H)$. Additionally, we demonstrate the generalization ability of MB-TAIL by extending it to the function approximation setting and proving that it can achieve expert sample and interaction complexity independent of $|S|$

cs.LG↗
arXiv2023

Non-compactness results for the spinorial Yamabe-type problems with non-smooth geometric data

Let $(M,\textit{g},σ)$ be an $m$-dimensional closed spin manifold, with a fixed Riemannian metric $\textit{g}$ and a fixed spin structure $σ$; let $\mathbb{S}(M)$ be the spinor bundle over $M$. The spinorial Yamabe-type problems address the solvability of the following equation \[ D_{\textit{g}}ψ= f(x)|ψ|_{\textit{g}}^{\frac2{m-1}}ψ, \quad ψ:M\to\mathbb{S}(M), \ x\in M \] where $D_{\textit{g}}$ is the associated Dirac operator and $f:M\to\mathbb{R}$ is a given function. The study of such nonlinear equation is motivated by its important applications in Spin Geometry: when $m=2$, a solution corresponds to a conformal isometric immersion of the universal covering $\widetilde M$ into $\mathbb{R}^3$ with prescribed mean curvature $f$; meanwhile, for general dimensions and $f\equiv constant\neq0$, a solution provides an upper bound estimate for the Bär-Hijazi-Lott invariant. The aim of this paper is to establish non-compactness results related to the spinorial Yamabe-type problems. Precisely, concrete analysis is made for two specific models on the manifold $(S^m,\textit{g})$ where the solution set of the spinorial Yamabe-type problem is not compact: $1).$ the geometric potential $f$ is constant (say $f\equiv1$) with the background metric $\textit{g}$ being a $C^k$ perturbation of the canonical round metric $\textit{g}_{S^m}$, which is not conformally flat somewhere on $S^m$; $2).$ $f$ is a perturbation from constant and is of class $C^2$, while the background metric $\textit{g}\equiv\textit{g}_{S^m}$.

math.DG↗
arXiv2023

Solutions of Spinorial Yamabe-type Problems on $S^m$: Perturbations and Applications

This paper is part of a program to establish the existence theory for the conformally invariant Dirac equation \[ D_{\textit{g}}ψ=f(x)|ψ|_{\textit{g}}^{\frac2{m-1}}ψ\] on a closed spin manifold $(M,\textit{g})$ of dimension $m\geq2$ with a fixed spin structure, where $f:M\to\mathbb{R}$ is a given function. The study on such nonlinear equation is motivated by its important applications in Spin Geometry: when $m=2$, a solution corresponds to an isometric immersion of the universal covering $\widetilde M$ into $\mathbb{R}^3$ with prescribed mean curvature $f$; meanwhile, for general dimensions and $f\equiv constant$, a solution provides an upper bound estimate for the Bär-Hijazi-Lott invariant.

math.DG↗
arXiv2023

Theoretical Analysis of Offline Imitation With Supplementary Dataset

Behavioral cloning (BC) can recover a good policy from abundant expert data, but may fail when expert data is insufficient. This paper considers a situation where, besides the small amount of expert data, a supplementary dataset is available, which can be collected cheaply from sub-optimal policies. Imitation learning with a supplementary dataset is an emergent practical framework, but its theoretical foundation remains under-developed. To advance understanding, we first investigate a direct extension of BC, called NBCU, that learns from the union of all available data. Our analysis shows that, although NBCU suffers an imitation gap that is larger than BC in the worst case, there exist special cases where NBCU performs better than or equally well as BC. This discovery implies that noisy data can also be helpful if utilized elaborately. Therefore, we further introduce a discriminator-based importance sampling technique to re-weight the supplementary data, proposing the WBCU method. With our newly developed landscape-based analysis, we prove that WBCU can outperform BC in mild conditions. Empirical studies show that WBCU simultaneously achieves the best performance on two challenging tasks where prior state-of-the-art methods fail.

cs.LG↗
arXiv2023

Curvature effect in the spinorial Yamabe problem on product manifolds

Let $(M_1,\textit{g}^{(1)})$, $(M_2,\textit{g}^{(2)})$ be closed Riemannian spin manifolds. We study the existence of solutions of the spinorial Yamabe problem on the product $M_1\times M_2$ equipped with a family of metrics $\varepsilon^{-2}\textit{g}^{(1)}\oplus\textit{g}^{(2)}$, $\varepsilon>0$. Via variational methods and blow-up techniques, we prove the existence of solutions which depend only on the factor $M_1$, and which exhibit a spike layer as $\varepsilon\to0$. Moreover, we locate the asymptotic position of the peak points of the solutions in terms of the curvature tensor on $(M_1,\textit{g}^{(1)})$.

math.DG↗
arXiv2023

Constructions of Delaunay-type solutions for the spinorial Yamabe equation on spheres

In this paper we construct singular solutions to the critical Dirac equation on spheres. More precisely, first we construct solutions admitting two points singularities that we call Delaunay-type solutions because of their similarities with the Delaunay solutions constructed for the singular Yamabe problem in \cite{MP1 , Schoen1989}. Then we construct another kind of singular solutions admitting a great circle as a singular set. These solutions are the building blocks for singular solutions on a general Spin manifold.

math.AP↗
arXiv2022

A Survey on Model-based Reinforcement Learning

Reinforcement learning (RL) solves sequential decision-making problems via a trial-and-error process interacting with the environment. While RL achieves outstanding success in playing complex video games that allow huge trial-and-error, making errors is always undesired in the real world. To improve the sample efficiency and thus reduce the errors, model-based reinforcement learning (MBRL) is believed to be a promising direction, which builds environment models in which the trial-and-errors can take place without real costs. In this survey, we take a review of MBRL with a focus on the recent progress in deep RL. For non-tabular environments, there is always a generalization error between the learned environment model and the real environment. As such, it is of great importance to analyze the discrepancy between policy training in the environment model and that in the real environment, which in turn guides the algorithm design for better model learning, model usage, and policy training. Besides, we also discuss the recent advances of model-based techniques in other forms of RL, including offline RL, goal-conditioned RL, multi-agent RL, and meta-RL. Moreover, we discuss the applicability and advantages of MBRL in real-world tasks. Finally, we end this survey by discussing the promising prospects for the future development of MBRL. We think that MBRL has great potential and advantages in real-world applications that were overlooked, and we hope this survey could attract more research on MBRL.

cs.LG↗
arXiv2022

Model Generation with Provable Coverability for Offline Reinforcement Learning

Model-based offline optimization with dynamics-aware policy provides a new perspective for policy learning and out-of-distribution generalization, where the learned policy could adapt to different dynamics enumerated at the training stage. But due to the limitation under the offline setting, the learned model could not mimic real dynamics well enough to support reliable out-of-distribution exploration, which still hinders policy to generalize well. To narrow the gap, previous works roughly ensemble randomly initialized models to better approximate the real dynamics. However, such practice is costly and inefficient, and provides no guarantee on how well the real dynamics could be approximated by the learned models, which we name coverability in this paper. We actively address this issue by generating models with provable ability to cover real dynamics in an efficient and controllable way. To that end, we design a distance metric for dynamic models based on the occupancy of policies under the dynamics, and propose an algorithm to generate models optimizing their coverage for the real dynamics. We give a theoretical analysis on the model generation process and proves that our algorithm could provide enhanced coverability. As a downstream task, we train a dynamics-aware policy with minor or no conservative penalty, and experiments demonstrate that our algorithm outperforms prior offline methods on existing offline RL benchmarks. We also discover that policies learned by our method have better zero-shot transfer performance, implying their better generalization.

cs.LG↗
arXiv2022

Rethinking ValueDice: Does It Really Improve Performance?

Since the introduction of GAIL, adversarial imitation learning (AIL) methods attract lots of research interests. Among these methods, ValueDice has achieved significant improvements: it beats the classical approach Behavioral Cloning (BC) under the offline setting, and it requires fewer interactions than GAIL under the online setting. Are these improvements benefited from more advanced algorithm designs? We answer this question by the following conclusions. First, we show that ValueDice could reduce to BC under the offline setting. Second, we verify that overfitting exists and regularization matters in the low-data regime. Specifically, we demonstrate that with weight decay, BC also nearly matches the expert performance as ValueDice does. The first two claims explain the superior offline performance of ValueDice. Third, we establish that ValueDice does not work when the expert trajectory is subsampled. Instead, the mentioned success of ValueDice holds when the expert trajectory is complete, in which ValueDice is closely related to BC that performs well as mentioned. Finally, we discuss the implications of our research for imitation learning studies beyond ValueDice.

cs.LG↗
arXiv2022

A Note on Target Q-learning For Solving Finite MDPs with A Generative Oracle

Q-learning with function approximation could diverge in the off-policy setting and the target network is a powerful technique to address this issue. In this manuscript, we examine the sample complexity of the associated target Q-learning algorithm in the tabular case with a generative oracle. We point out a misleading claim in [Lee and He, 2020] and establish a tight analysis. In particular, we demonstrate that the sample complexity of the target Q-learning algorithm in [Lee and He, 2020] is $\widetilde{\mathcal O}(|\mathcal S|^2|\mathcal A|^2 (1-γ)^{-5}\varepsilon^{-2})$. Furthermore, we show that this sample complexity is improved to $\widetilde{\mathcal O}(|\mathcal S||\mathcal A| (1-γ)^{-5}\varepsilon^{-2})$ if we can sequentially update all state-action pairs and $\widetilde{\mathcal O}(|\mathcal S||\mathcal A| (1-γ)^{-4}\varepsilon^{-2})$ if $γ$ is further in $(1/2, 1)$. Compared with the vanilla Q-learning, our results conclude that the introduction of a periodically-frozen target Q-function does not sacrifice the sample complexity.

cs.LG↗
arXiv2022

On Generalization of Adversarial Imitation Learning and Beyond

Despite massive empirical evaluations, one of the fundamental questions in imitation learning is still not fully settled: does AIL (adversarial imitation learning) provably generalize better than BC (behavioral cloning)? We study this open problem with tabular and episodic MDPs. For vanilla AIL that uses the direct maximum likelihood estimation, we provide both negative and positive answers under the known transition setting. For some MDPs, we show that vanilla AIL has a worse sample complexity than BC. The key insight is that the state-action distribution matching principle is weak so that AIL may generalize poorly even on visited states from the expert demonstrations. For another class of MDPs, vanilla AIL is proved to generalize well even on non-visited states. Interestingly, its sample complexity is horizon-free, which provably beats BC by a wide margin. Finally, we establish a framework in the unknown transition scenario, which allows AIL to explore via reward-free exploration strategies. Compared with the best-known online apprenticeship learning algorithm, the resulting algorithm improves the sample complexity and interaction complexity.

cs.LG↗