arXiv subjects

Tianyu Li

Publications and source records attributed to Tianyu Li.

At least 163 records · Page 9Linked to original sources

arXiv2022

FastMimic: Model-based Motion Imitation for Agile, Diverse and Generalizable Quadrupedal Locomotion

Robots operating in human environments need various skills, like slow and fast walking, turning, side-stepping, and many more. However, building robot controllers that can exhibit such a large range of behaviors is a challenging problem that requires tedious investigation for every task. We present a unified model-based control algorithm for imitating different animal gaits without expensive simulation training or real-world fine-tuning. Our method consists of stance and swing leg controllers using a centroidal dynamics model augmented with online adaptation techniques. We also develop a whole-body trajectory optimization procedure to fix the kinematic infeasibility of the reference animal motions. We demonstrate that our universal data-driven model-based controller can seamlessly imitate various motor skills, including trotting, pacing, turning, and side-stepping. It also shows better tracking capabilities in simulation and the real world against several baselines, including another model-based imitation controller and a learning-based motion imitation technique.

cs.RO↗

arXiv2022

UserBERT: Modeling Long- and Short-Term User Preferences via Self-Supervision

E-commerce platforms generate vast amounts of customer behavior data, such as clicks and purchases, from millions of unique users every day. However, effectively using this data for behavior understanding tasks is challenging because there are usually not enough labels to learn from all users in a supervised manner. This paper extends the BERT model to e-commerce user data for pre-training representations in a self-supervised manner. By viewing user actions in sequences as analogous to words in sentences, we extend the existing BERT model to user behavior data. Further, our model adopts a unified structure to simultaneously learn from long-term and short-term user behavior, as well as user attributes. We propose methods for the tokenization of different types of user behavior sequences, the generation of input representation vectors, and a novel pretext task to enable the pre-trained model to learn from its own input, eliminating the need for labeled training data. Extensive experiments demonstrate that the learned representations result in significant improvements when transferred to three different real-world tasks, particularly compared to task-specific modeling and multi-task representation learning

cs.LG↗

arXiv2022

Connecting Weighted Automata, Tensor Networks and Recurrent Neural Networks through Spectral Learning

In this paper, we present connections between three models used in different research fields: weighted finite automata~(WFA) from formal languages and linguistics, recurrent neural networks used in machine learning, and tensor networks which encompasses a set of optimization techniques for high-order tensors used in quantum physics and numerical analysis. We first present an intrinsic relation between WFA and the tensor train decomposition, a particular form of tensor network. This relation allows us to exhibit a novel low rank structure of the Hankel matrix of a function computed by a WFA and to design an efficient spectral learning algorithm leveraging this structure to scale the algorithm up to very large Hankel matrices.We then unravel a fundamental connection between WFA and second-orderrecurrent neural networks~(2-RNN): in the case of sequences of discrete symbols, WFA and 2-RNN with linear activationfunctions are expressively equivalent. Leveraging this equivalence result combined with the classical spectral learning algorithm for weighted automata, we introduce the first provable learning algorithm for linear 2-RNN defined over sequences of continuous input vectors.This algorithm relies on estimating low rank sub-blocks of the Hankel tensor, from which the parameters of a linear 2-RNN can be provably recovered. The performances of the proposed learning algorithm are assessed in a simulation study on both synthetic and real-world data.

cs.LG↗

arXiv2021

Engineering Dissipative Quasicrystals

We discuss the systematic engineering of quasicrystals in open quantum systems where quasiperiodicity is introduced through purely dissipative processes. While the resulting short-time dynamics is governed by non-Hermitian variants of the Aubry-Andre-Harper model, we demonstrate how phases and phase transitions pertaining to the non-Hermitian quasicrystals fundamentally change the long-time, steady-state-approaching dynamics under the Lindblad master equation. Our schemes are based on an exact mapping between the eigenspectrum of the Liouvillian superoperator with that of the non-Hermitian Hamiltonian, under the condition of quadratic fermionic systems subject to linear dissipation. Our work suggests a systematic route toward engineering exotic quantum dynamics in open systems, based on insights of non-Hermitian physics.

cond-mat.quant-gas↗

arXiv2021

Learning Navigation Skills for Legged Robots with Learned Robot Embeddings

Recent work has shown results on learning navigation policies for idealized cylinder agents in simulation and transferring them to real wheeled robots. Deploying such navigation policies on legged robots can be challenging due to their complex dynamics, and the large dynamical difference between cylinder agents and legged systems. In this work, we learn hierarchical navigation policies that account for the low-level dynamics of legged robots, such as maximum speed, slipping, contacts, and learn to successfully navigate cluttered indoor environments. To enable transfer of policies learned in simulation to new legged robots and hardware, we learn dynamics-aware navigation policies across multiple robots with robot-specific embeddings. The learned embedding is optimized on new robots, while the rest of the policy is kept fixed, allowing for quick adaptation. We train our policies across three legged robots in simulation - 2 quadrupeds (A1, AlienGo) and a hexapod (Daisy). At test time, we study the performance of our learned policy on two new legged robots in simulation (Laikago, 4-legged Daisy), and one real-world quadrupedal robot (A1). Our experiments show that our learned policy can sample-efficiently generalize to previously unseen robots, and enable sim-to-real transfer of navigation policies for legged robots.

cs.RO↗

arXiv2021

Ramsey numbers of quadrilateral versus books

A book $B_n$ is a graph which consists of $n$ triangles sharing a common edge. In this paper, we study Ramsey numbers of quadrilateral versus books. Previous results give the exact value of $r(C_4,B_n)$ for $1\le n\le 14$. We aim to show the exact value of $r(C_4,B_n)$ for infinitely many $n$. To achieve this, we first prove that $r(C_4,B_{(m-1)^2+(t-2)})\le m^2+t$ for $m\ge4$ and $0 \leq t \leq m-1$. This improves upon a result by Faudree, Rousseau and Sheehan (1978) which states that \begin{align*} r(C_4,B_n)\le g(g(n)), \;\;\text{where}\;\;g(n)=n+\lfloor\sqrt{n-1}\rfloor+2. \end{align*} Combining the new upper bound and constructions of $C_4$-free graphs, we are able to determine the exact value of $r(C_4,B_n)$ for infinitely many $n$. As a special case, we show $r(C_4,B_{q^2-q-2}) = q^2+q-1$ for all prime power $q\ge4$.

math.CO↗

arXiv2021

Observation of non-Hermitian topological Anderson insulator in quantum dynamics

Disorder and non-Hermiticity dramatically impact the topological and localization properties of a quantum system, giving rise to intriguing quantum states of matter. The rich interplay of disorder, non-Hermiticity, and topology is epitomized by the recently proposed non-Hermitian topological Anderson insulator that hosts a plethora of exotic phenomena. Here we experimentally simulate the non-Hermitian topological Anderson insulator using disordered photonic quantum walks, and characterize its localization and topological properties. In particular, we focus on the competition between Anderson localization induced by random disorder, and the non-Hermitian skin effect under which all eigenstates are squeezed toward the boundary. The two distinct localization mechanisms prompt a non-monotonous change in profile of the Lyapunov exponent, which we experimentally reveal through dynamic observables. We then probe the disorder-induced topological phase transitions, and demonstrate their biorthogonal criticality. Our experiment further advances the frontier of synthetic topology in open systems.

cond-mat.mes-hall↗

arXiv2021

Detecting non-Bloch topological invariants in quantum dynamics

Non-Bloch topological invariants preserve the bulk-boundary correspondence in non-Hermitian topological systems, and are a key concept in the contemporary study of non-Hermitian topology. Here we report the dynamic detection of non-Bloch topological invariants in single-photon quantum walks, revealed through the biorthogonal chiral displacement, and crosschecked with the dynamic spin textures in the generalized quasimomentum-time domain following a quantum quench. Both detection schemes are robust against symmetry-preserving disorders, and yield consistent results with theoretical predictions. Our experiments are performed far away from any boundaries, and therefore underline non-Bloch topological invariants as intrinsic properties of the system that persist in the thermodynamic limit. Our work sheds new light on the experimental investigation of non-Hermitian topology.

cond-mat.mes-hall↗

arXiv2021

Planning in Learned Latent Action Spaces for Generalizable Legged Locomotion

Hierarchical learning has been successful at learning generalizable locomotion skills on walking robots in a sample-efficient manner. However, the low-dimensional "latent" action used to communicate between two layers of the hierarchy is typically user-designed. In this work, we present a fully-learned hierarchical framework, that is capable of jointly learning the low-level controller and the high-level latent action space. Once this latent space is learned, we plan over continuous latent actions in a model-predictive control fashion, using a learned high-level dynamics model. This framework generalizes to multiple robots, and we present results on a Daisy hexapod simulation, A1 quadruped simulation, and Daisy robot hardware. We compare a range of learned hierarchical approaches from literature, and show that our framework outperforms baselines on multiple tasks and two simulations. In addition to learning approaches, we also compare to inverse-kinematics (IK) acting on desired robot motion, and show that our fully-learned framework outperforms IK in adverse settings on both A1 and Daisy simulations. On hardware, we show the Daisy hexapod achieve multiple locomotion tasks, in an unstructured outdoor setting, with only 2000 hardware samples, reinforcing the robustness and sample-efficiency of our approach.

cs.RO↗

arXiv2021

Two-dimensional quantum walk with non-Hermitian skin effects

We construct a two-dimensional, discrete-time quantum walk exhibiting non-Hermitian skin effects under open-boundary conditions. As a confirmation of the non-Hermitian bulk-boundary correspondence, we show that the emergence of topological edge states are consistent with Floquet winding numbers calculated using a non-Bloch band theory invoking time-dependent generalized Billouin zones. Further, the non-Bloch topological invariants associated with quasienergy bands are captured by a non-Hermitian local Chern marker in real space, defined through local biorthogonal eigen wave functions of the non-unitary Floquet operator. Our work would stimulate further studies of non-Hermitian Floquet topological phases where skin effects play a key role.

cond-mat.mes-hall↗

arXiv2021

Non-Bloch quench dynamics

We study the quench dynamics of non-Hermitian topological models with non-Hermitian skin effects. Adopting the non-Bloch band theory and projecting quench dynamics onto the generalized Brillouin zone, we find that emergent topological structures, in the form of dynamic skyrmions, exist in the generalized momentum-time domain, and are correlated with the non-Bloch topological invariants of the static Hamiltonians. The skyrmion structures anchor on the fixed points of dynamics whose existence are conditional on the coincidence of generalized Brillouin zones of the pre- and post-quench Hamiltonians. Global signatures of dynamic skyrmions, however, persist well beyond such a condition, thus offering a general dynamic detection scheme for non-Bloch topology in the presence of non-Hermitian skin effects. Applying our theory to an experimentally relevant, non-unitary quantum walk, we explicitly demonstrate how the non-Bloch topological invariants can be revealed through the non-Bloch quench dynamics.

cond-mat.quant-gas↗

arXiv2020

Representation of Reinforcement Learning Policies in Reproducing Kernel Hilbert Spaces

We propose a general framework for policy representation for reinforcement learning tasks. This framework involves finding a low-dimensional embedding of the policy on a reproducing kernel Hilbert space (RKHS). The usage of RKHS based methods allows us to derive strong theoretical guarantees on the expected return of the reconstructed policy. Such guarantees are typically lacking in black-box models, but are very desirable in tasks requiring stability. We conduct several experiments on classic RL domains. The results confirm that the policies can be robustly embedded in a low-dimensional space while the embedded policy incurs almost no decrease in return.

cs.LG↗

arXiv2020

Neural Representations in Hybrid Recommender Systems: Prediction versus Regularization

Autoencoder-based hybrid recommender systems have become popular recently because of their ability to learn user and item representations by reconstructing various information sources, including users' feedback on items (e.g., ratings) and side information of users and items (e.g., users' occupation and items' title). However, existing systems still use representations learned by matrix factorization (MF) to predict the rating, while using representations learned by neural networks as the regularizer. In this paper, we define the neural representation for prediction (NRP) framework and apply it to the autoencoder-based recommendation systems. We theoretically analyze how our objective function is related to the previous MF and autoencoder-based methods and explain what it means to use neural representations as the regularizer. We also apply the NRP framework to a direct neural network structure which predicts the ratings without reconstructing the user and item information. We conduct extensive experiments on two MovieLens datasets and two real-world e-commerce datasets. The results confirm that neural representations are better for prediction than regularization and show that the NRP framework, combined with the direct neural network structure, outperforms the state-of-the-art methods in the prediction task, with less training time and memory.

cs.IR↗

arXiv2020

Learning to Profile: User Meta-Profile Network for Few-Shot Learning

Meta-learning approaches have shown great success in vision and language domains. However, few studies discuss the practice of meta-learning for large-scale industrial applications. Although e-commerce companies have spent many efforts on learning representations to provide a better user experience, we argue that such efforts cannot be stopped at this step. In addition to learning a strong profile, the challenging question about how to effectively transfer the learned representation is raised simultaneously. This paper introduces the contributions that we made to address these challenges from three aspects. 1) Meta-learning model: In the context of representation learning with e-commerce user behavior data, we propose a meta-learning framework called the Meta-Profile Network, which extends the ideas of matching network and relation network for knowledge transfer and fast adaptation; 2) Encoding strategy: To keep high fidelity of large-scale long-term sequential behavior data, we propose a time-heatmap encoding strategy that allows the model to encode data effectively; 3) Deep network architecture: A multi-modal model combined with multi-task learning architecture is utilized to address the cross-domain knowledge learning and insufficient label problems. Moreover, we argue that an industrial model should not only have good performance in terms of accuracy, but also have better robustness and uncertainty performance under extreme conditions. We evaluate the performance of our model with extensive control experiments in various extreme scenarios, i.e. out-of-distribution detection, data insufficiency and class imbalance scenarios. The Meta-Profile Network shows significant improvement in the model performance when compared to baseline models.

cs.LG↗

arXiv2020

Learning Classifiers on Positive and Unlabeled Data with Policy Gradient

Existing algorithms aiming to learn a binary classifier from positive (P) and unlabeled (U) data generally require estimating the class prior or label noises ahead of building a classification model. However, the estimation and classifier learning are normally conducted in a pipeline instead of being jointly optimized. In this paper, we propose to alternatively train the two steps using reinforcement learning. Our proposal adopts a policy network to adaptively make assumptions on the labels of unlabeled data, while a classifier is built upon the output of the policy network and provides rewards to learn a better strategy. The dynamic and interactive training between the policy maker and the classifier can exploit the unlabeled data in a more effective manner and yield a significant improvement on the classification performance. Furthermore, we present two different approaches to represent the actions sampled from the policy. The first approach considers continuous actions as soft labels, while the other uses discrete actions as hard assignment of labels for unlabeled examples.We validate the effectiveness of the proposed method on two benchmark datasets as well as one e-commerce dataset. The result shows the proposed method is able to consistently outperform state-of-the-art methods in various settings.

cs.LG↗

arXiv2020

Mainlining Databases: Supporting Fast Transactional Workloads on Universal Columnar Data File Formats

The proliferation of modern data processing tools has given rise to open-source columnar data formats. The advantage of these formats is that they help organizations avoid repeatedly converting data to a new format for each application. These formats, however, are read-only, and organizations must use a heavy-weight transformation process to load data from on-line transactional processing (OLTP) systems. We aim to reduce or even eliminate this process by developing a storage architecture for in-memory database management systems (DBMSs) that is aware of the eventual usage of its data and emits columnar storage blocks in a universal open-source format. We introduce relaxations to common analytical data formats to efficiently update records and rely on a lightweight transformation process to convert blocks to a read-optimized layout when they are cold. We also describe how to access data from third-party analytical tools with minimal serialization overhead. To evaluate our work, we implemented our storage engine based on the Apache Arrow format and integrated it into the DB-X DBMS. Our experiments show that our approach achieves comparable performance with dedicated OLTP DBMSs while enabling orders-of-magnitude faster data exports to external data science and machine learning tools than existing methods.

cs.DB↗

arXiv2019

Efficient Planning under Partial Observability with Unnormalized Q Functions and Spectral Learning

Learning and planning in partially-observable domains is one of the most difficult problems in reinforcement learning. Traditional methods consider these two problems as independent, resulting in a classical two-stage paradigm: first learn the environment dynamics and then plan accordingly. This approach, however, disconnects the two problems and can consequently lead to algorithms that are sample inefficient and time consuming. In this paper, we propose a novel algorithm that combines learning and planning together. Our algorithm is closely related to the spectral learning algorithm for predicitive state representations and offers appealing theoretical guarantees and time complexity. We empirically show on two domains that our approach is more sample and time efficient compared to classical methods.

cs.AI↗

arXiv2019

Learning Generalizable Locomotion Skills with Hierarchical Reinforcement Learning

Learning to locomote to arbitrary goals on hardware remains a challenging problem for reinforcement learning. In this paper, we present a hierarchical learning framework that improves sample-efficiency and generalizability of locomotion skills on real-world robots. Our approach divides the problem of goal-oriented locomotion into two sub-problems: learning diverse primitives skills, and using model-based planning to sequence these skills. We parametrize our primitives as cyclic movements, improving sample-efficiency of learning on a 18 degrees of freedom robot. Then, we learn coarse dynamics models over primitive cycles and use them in a model predictive control framework. This allows us to learn to walk to arbitrary goals up to 12m away, after about two hours of training from scratch on hardware. Our results on a Daisy hexapod hardware and simulation demonstrate the efficacy of our approach at reaching distant targets, in different environments and with sensory noise.

cs.RO↗