arXiv subjects

Yiming Chen

Publications and source records attributed to Yiming Chen.

At least 163 records · Page 9Linked to original sources

arXiv2021

DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training

The scale of deep learning nowadays calls for efficient distributed training algorithms. Decentralized momentum SGD (DmSGD), in which each node averages only with its neighbors, is more communication efficient than vanilla Parallel momentum SGD that incurs global average across all computing nodes. On the other hand, the large-batch training has been demonstrated critical to achieve runtime speedup. This motivates us to investigate how DmSGD performs in the large-batch scenario. In this work, we find the momentum term can amplify the inconsistency bias in DmSGD. Such bias becomes more evident as batch-size grows large and hence results in severe performance degradation. We next propose DecentLaM, a novel decentralized large-batch momentum SGD to remove the momentum-incurred bias. The convergence rate for both non-convex and strongly-convex scenarios is established. Our theoretical results justify the superiority of DecentLaM to DmSGD especially in the large-batch scenario. Experimental results on a variety of computer vision tasks and models demonstrate that DecentLaM promises both efficient and high-quality training.

cs.LG↗

arXiv2021

IMU Data Processing For Inertial Aided Navigation: A Recurrent Neural Network Based Approach

In this work, we propose a novel method for performing inertial aided navigation, by using deep neural networks (DNNs). To date, most DNN inertial navigation methods focus on the task of inertial odometry, by taking gyroscope and accelerometer readings as input and regressing for integrated IMU poses (i.e., position and orientation). While this design has been successfully applied on a number of applications, it is not of theoretical performance guarantee unless patterned motion is involved. This inevitably leads to significantly reduced accuracy and robustness in certain use cases. To solve this problem, we design a framework to compute observable IMU integration terms using DNNs, followed by the numerical pose integration and sensor fusion to achieve the performance gain. Specifically, we perform detailed analysis on the motion terms in IMU kinematic equations, propose a dedicated network design, loss functions, and training strategies for the IMU data processing, and conduct extensive experiments. The results show that our method is generally applicable and outperforms both traditional and DNN methods by wide margins.

cs.RO↗

arXiv2021

Learned Gradient Compression for Distributed Deep Learning

Training deep neural networks on large datasets containing high-dimensional data requires a large amount of computation. A solution to this problem is data-parallel distributed training, where a model is replicated into several computational nodes that have access to different chunks of the data. This approach, however, entails high communication rates and latency because of the computed gradients that need to be shared among nodes at every iteration. The problem becomes more pronounced in the case that there is wireless communication between the nodes (i.e. due to the limited network bandwidth). To address this problem, various compression methods have been proposed including sparsification, quantization, and entropy encoding of the gradients. Existing methods leverage the intra-node information redundancy, that is, they compress gradients at each node independently. In contrast, we advocate that the gradients across the nodes are correlated and propose methods to leverage this inter-node redundancy to improve compression efficiency. Depending on the node communication protocol (parameter server or ring-allreduce), we propose two instances of the LGC approach that we coin Learned Gradient Compression (LGC). Our methods exploit an autoencoder (i.e. trained during the first stages of the distributed training) to capture the common information that exists in the gradients of the distributed nodes. We have tested our LGC methods on the image classification and semantic segmentation tasks using different convolutional neural networks (ResNet50, ResNet101, PSPNet) and multiple datasets (ImageNet, Cifar10, CamVid). The ResNet101 model trained for image classification on Cifar10 achieved an accuracy of 93.57%, which is lower than the baseline distributed training with uncompressed gradients only by 0.18%.

cs.LG↗

arXiv2021

Large-Scale Training System for 100-Million Classification at Alibaba

In the last decades, extreme classification has become an essential topic for deep learning. It has achieved great success in many areas, especially in computer vision and natural language processing (NLP). However, it is very challenging to train a deep model with millions of classes due to the memory and computation explosion in the last output layer. In this paper, we propose a large-scale training system to address these challenges. First, we build a hybrid parallel training framework to make the training process feasible. Second, we propose a novel softmax variation named KNN softmax, which reduces both the GPU memory consumption and computation costs and improves the throughput of training. Then, to eliminate the communication overhead, we propose a new overlapping pipeline and a gradient sparsification method. Furthermore, we design a fast continuous convergence strategy to reduce total training iterations by adaptively adjusting learning rate and updating model parameters. With the help of all the proposed methods, we gain 3.9$\times$ throughput of our training system and reduce almost 60\% of training iterations. The experimental results show that using an in-house 256 GPUs cluster, we could train a classifier of 100 million classes on Alibaba Retail Product Dataset in about five days while achieving a comparable accuracy with the naive softmax training process.

cs.LG↗

arXiv2020

Signatures of global symmetry violation in relative entropies and replica wormholes

It is widely believed that exact global symmetries do not exist in theories that admit quantum black holes. Here we propose a way to quantify the degree of global symmetry violation in the Hawking radiation of a black hole by using certain relative entropies. While the violations of global symmetry that we consider are non-perturbative effects, they nevertheless give $\mathcal{O}(1)$ contributions to the relative entropy after the Page time. Furthermore, using "island" formulas, these relative entropies can be computed within semi-classical gravity, which we demonstrate with explicit examples. These formulas give a rather precise operational sense to the statement that a global charge thrown into an old black hole will be lost after a scrambling time. The relative entropies considered here may also be computed using a replica trick. At integer replica index, the global symmetry violating effects manifest themselves as charge flowing through the replica wormhole.

hep-th↗

arXiv2020

Pose Estimation for Ground Robots: On Manifold Representation, Integration, Re-Parameterization, and Optimization

In this paper, we focus on motion estimation dedicated for non-holonomic ground robots, by probabilistically fusing measurements from the wheel odometer and exteroceptive sensors. For ground robots, the wheel odometer is widely used in pose estimation tasks, especially in applications under planar-scene based environments. However, since the wheel odometer only provides 2D motion estimates, it is extremely challenging to use that for performing accurate full 6D pose (3D position and 3D orientation) estimation. Traditional methods on 6D pose estimation either approximate sensor or motion models, at the cost of accuracy reduction, or rely on other sensors, e.g., inertial measurement unit (IMU), to provide complementary measurements. By contrast, in this paper, we propose a novel method to utilize the wheel odometer for 6D pose estimation, by modeling and utilizing motion manifold for ground robots. Our approach is probabilistically formulated and only requires the wheel odometer and an exteroceptive sensor (e.g., a camera). Specifically, our method i) formulates the motion manifold of ground robots by parametric representation, ii) performs manifold based 6D integration with the wheel odometer measurements only, and iii) re-parameterizes manifold equations periodically for error reduction. To demonstrate the effectiveness and applicability of the proposed algorithmic modules, we integrate that into a sliding-window pose estimator by using measurements from the wheel odometer and a monocular camera. By conducting extensive simulated and real-world experiments, we show that the proposed algorithm outperforms competing state-of-the-art algorithms by a significant margin in pose estimation accuracy, especially when deployed in complex large-scale real-world environments.

cs.RO↗

arXiv2020

Replica wormhole and information retrieval in the SYK model coupled to Majorana chains

Motivated by recent studies of the information paradox in (1+1)-D anti-de Sitter spacetime with a bath described by a (1+1)-D conformal field theory, we study the dynamics of second Rényi entropy of the Sachdev-Ye-Kitaev (SYK) model ($χ$) coupled to a Majorana chain bath ($ψ$). The system is prepared in the thermofield double (TFD) state and then evolved by $H_L+H_R$. For small system-bath coupling, we find that the second Rényi entropy $S^{(2)}_{χ_L, χ_R}$ of the SYK model undergoes a first order transition during the evolution. In the sense of holographic duality, the long-time solution corresponds to a "replica wormhole". The transition time corresponds to the Page time of a black hole coupled to a thermal bath. We further study the information scrambling and retrieval by introducing a classical control bit, which controls whether or not we add a perturbation in the SYK system. The mutual information between the bath and the control bit shows a positive jump at the Page time, indicating that the entanglement wedge of the bath includes an island in the holographic bulk.

hep-th↗

arXiv2020

Pulling Out the Island with Modular Flow

Recent works have suggested that the entanglement wedge of Hawking radiation coming from an AdS black hole, will include an island inside the black hole interior after the Page time. In this paper, we propose a concrete way to extract the information from the island by acting only on the radiation degrees of freedom, building on the equivalence between the boundary and bulk modular flow. We consider examples with black holes in JT gravity coupled to baths. In the case that the bulk conformal fields contain free massless fermion field, we provide explicit bulk picture of the information extraction process, where we find that one can almost pull out an operator from the island to the bath with modular flow.

hep-th↗

arXiv2020

A Lightweight and Accurate Localization Algorithm Using Multiple Inertial Measurement Units

This paper proposes a novel inertial-aided localization approach by fusing information from multiple inertial measurement units (IMUs) and exteroceptive sensors. IMU is a low-cost motion sensor which provides measurements on angular velocity and gravity compensated linear acceleration of a moving platform, and widely used in modern localization systems. To date, most existing inertial-aided localization methods exploit only one single IMU. While the single-IMU localization yields acceptable accuracy and robustness for different use cases, the overall performance can be further improved by using multiple IMUs. To this end, we propose a lightweight and accurate algorithm for fusing measurements from multiple IMUs and exteroceptive sensors, which is able to obtain noticeable performance gain without incurring additional computational cost. To achieve this, we first probabilistically map measurements from all IMUs onto a virtual IMU. This step is performed by stochastic estimation with least-square estimators and probabilistic marginalization of inter-IMU rotational accelerations. Subsequently, the propagation model for both state and error state of the virtual IMU is also derived, which enables the use of the classical filter-based or optimization-based sensor fusion algorithms for localization. Finally, results from both simulation and real-world tests are provided, which demonstrate that the proposed algorithm outperforms competing algorithms by noticeable margins.

cs.RO↗

arXiv2019

Visual-Inertial Localization for Skid-Steering Robots with Kinematic Constraints

While visual localization or SLAM has witnessed great progress in past decades, when deploying it on a mobile robot in practice, few works have explicitly considered the kinematic (or dynamic) constraints of the real robotic system when designing state estimators. To promote the practical deployment of current state-of-the-art visual-inertial localization algorithms, in this work we propose a low-cost kinematics-constrained localization system particularly for a skid-steering mobile robot. In particular, we derive in a principle way the robot's kinematic constraints based on the instantaneous centers of rotation (ICR) model and integrate them in a tightly-coupled manner into the sliding-window bundle adjustment (BA)-based visual-inertial estimator. Because the ICR model parameters are time-varying due to, for example, track-to-terrain interaction and terrain roughness, we estimate these kinematic parameters online along with the navigation state. To this end, we perform in-depth the observability analysis and identify motion conditions under which the state/parameter estimation is viable. The proposed kinematics-constrained visual-inertial localization system has been validated extensively in different terrain scenarios.

cs.RO↗

arXiv2019

Random Forest Models for Accurate Identification of Coordination Environments from X-ray Absorption Near-Edge Structure

Analyzing coordination environments using X-ray absorption spectroscopy has broad applications ranging from solid-state physics to material chemistry. Here, we show that random forest models can identify the main coordination environment from K-edge X-ray absorption near edge structure (XANES) with a high accuracy of 85.4% and all associated coordination environments with a high Jaccard score of 81.8% for 33 cation elements in oxides, significantly outperforming other machine learning (ML) models. In a departure from prior works, we used a robust description of the coordination environment as a distribution over 25 distinct coordination motifs with coordination numbers ranging from 1-12. The random forest models were trained on the world's largest database of ~190,000 computed K-edge XANES spectra. Furthermore, the random forest models can be applied to predict the coordination environment from experimental K-edge XANES with minimal loss in accuracy (82.1%) due to the use of data augmentation. A drop-out feature importance analysis highlights the key roles that the pre-edge and main-peak regions play in coordination environment identification, with the post-peak region becoming increasingly important at higher coordination numbers. This work provides a general strategy to identify the coordination environment from K-edge XANES across broad chemistries, paving the way for future advancements in the application of ML to spectroscopy.

cond-mat.mtrl-sci↗

arXiv2019

A Performance and Cost Assessment of Machine Learning Interatomic Potentials

Machine learning of the quantitative relationship between local environment descriptors and the potential energy surface of a system of atoms has emerged as a new frontier in the development of interatomic potentials (IAPs). Here, we present a comprehensive evaluation of ML-IAPs based on four local environment descriptors --- Behler-Parrinello symmetry functions, smooth overlap of atomic positions (SOAP), the Spectral Neighbor Analysis Potential (SNAP) bispectrum components, and moment tensors --- using a diverse data set generated using high-throughput density functional theory (DFT) calculations. The data set comprising bcc (Li, Mo) and fcc (Cu, Ni) metals and diamond group IV semiconductors (Si, Ge) is chosen to span a range of crystal structures and bonding. All descriptors studied show excellent performance in predicting energies and forces far surpassing that of classical IAPs, as well as predicting properties such as elastic constants and phonon dispersion curves. We observe a general trade-off between accuracy and the degrees of freedom of each model, and consequently computational cost. We will discuss these trade-offs in the context of model selection for molecular dynamics and other applications.

physics.comp-ph↗

arXiv2019

FoxNet: A Multi-face Alignment Method

Multi-face alignment aims to identify geometry structures of multiple faces in an image, and its performance is essential for the many practical tasks, such as face recognition, face tracking, and face animation. In this work, we present a fast bottom-up multi-face alignment approach, which can simultaneously localize multi-person facial landmarks with high precision.In more detail, our bottom-up architecture maps the landmarks to the high-dimensional space with which landmarks of all faces are represented. By clustering the features belonging to the same face, our approach can align the multi-person facial landmarks synchronously.Extensive experiments show that our method can achieve high performance in the multi-face landmark alignment task while our model is extremely fast. Moreover, we propose a new multi-face dataset to compare the speed and precision of bottom-up face alignment method with top-down methods. Our dataset is publicly available at https://github.com/AISAResearch/FoxNet

cs.CV↗

arXiv2019

Entanglement Entropy of Two Coupled SYK Models and Eternal Traversable Wormhole

In this paper, we study the entanglement entropy between two SYK systems with bilinear coupling. We use the replica trick to calculate the entanglement entropy in the ground state. In parallel, we calculate the entanglement entropy through the Ryu-Takayanagi formula in gravity. For the ground state that is dual to an eternal traversable wormhole in AdS_2, the bulk quantum correction to the entanglement entropy is of the same order as the minimal surface area. The ground state of the coupled system is close to a thermofield double state with particular temperature and they have the same entanglement entropy. From the gravity point of view, we explain why the two states have the same entanglement entropy. We also study a case with time-dependent coupling, which involves finding the quantum extremal surface in the bulk.

hep-th↗

arXiv2018

Modular Flow as a Disentangler

In holographic duality, the entanglement entropy of a boundary region is proposed to be dual to the area of an extremal codimension-2 surface that is homologous to the boundary region, known as the Hubeny-Rangamani-Takayanagi (HRT) surface. In this paper, we study when the HRT surfaces of two boundary subregions R, A are in the same Cauchy slice. This condition is necessary for the subregion-subregion mapping to be local for both subregions and for states to have a tensor network description. To quantify this, we study the area of a surface that is homologous to A and is extremal except at possible intersections with the HRT surface of R (minimizing over all such possible surfaces), which we call the constrained area. We give a boundary proposal for an upper bound of this quantity, a bound which is saturated when the constrained surface intersects the HRT surface of R at a constant angle. This boundary quantity is the minimum entropy of region A in a modular evolved state -- a state that has been evolved unitarily with the modular Hamiltonian of R. We can prove this formula in two boundary dimensions or when the modular Hamiltonian is local. This modular minimal entropy is a boundary quantity that probes bulk causality and, from this quantity, we can extract whether two HRT surfaces are in the future or past of each other. These entropies satisfy some inequalities reminiscent of strong subadditivity and can be used to remove certain corner divergences.

hep-th↗

arXiv2017

Automated Generation and Ensemble-Learned Matching of X-ray Absorption Spectra

We report the development of XASdb, a large database of computed reference X-ray absorption spectra (XAS), and a novel Ensemble-Learned Spectra IdEntification (ELSIE) algorithm for the matching of spectra. XASdb currently hosts more than 300,000 K-edge X-ray absorption near-edge spectra (XANES) for over 30,000 materials from the open-science Materials Project database. We discuss a high-throughput automation framework for FEFF calculations, built on robust, rigorously benchmarked parameters. We will demonstrate that the ELSIE algorithm, which combines 33 weak "learners" comprising a set of preprocessing steps and a similarity metric, can achieve up to 84.2% accuracy in identifying the correct oxidation state and coordination environment of a test set of 19 K-edge XANES spectra encompassing a diverse range of chemistries and crystal structures. The XASdb with the ELSIE algorithm has been integrated into a web application in the Materials Project, providing an important new public resource for the analysis of XAS to all materials researchers. Finally, the ELSIE algorithm itself has been made available as part of Veidt, an open source machine learning library for materials science.

cond-mat.mtrl-sci↗

arXiv2017

Tunable Quantum Chaos in the Sachdev-Ye-Kitaev Model Coupled to a Thermal Bath

The Sachdev-Ye-Kitaev (SYK) model describes Majorana fermions with random interaction, which displays many interesting properties such as non-Fermi liquid behavior, quantum chaos, emergent conformal symmetry and holographic duality. Here we consider a SYK model or a chain of SYK models with $N$ Majorana fermion modes coupled to another SYK model with $N^2$ Majorana fermion modes, in which the latter has many more degrees of freedom and plays the role as a thermal bath. For a single SYK model coupled to the thermal bath, we show that although the Lyapunov exponent is still proportional to temperature, it monotonically decreases from $2π/β$ ($β=1/(k_BT)$, $T$ is temperature) to zero as the coupling strength to the thermal bath increases. For a chain of SYK models, when they are uniformly coupled to the thermal bath, we show that the butterfly velocity displays a crossover from a $\sqrt{T}$-dependence at relatively high temperature to a linear $T$-dependence at low temperature, with the crossover temperature also controlled by the coupling strength to the thermal bath. If only the end of the SYK chain is coupled to the thermal bath, the model can introduce a spatial dependence of both the Lyapunov exponent and the butterfly velocity. Our models provide canonical examples for the study of thermalization within chaotic models.

hep-th↗

arXiv2017

Competition between Chaotic and Non-Chaotic Phases in a Quadratically Coupled Sachdev-Ye-Kitaev Model

The Sachdev-Ye-Kitaev (SYK) model is a concrete solvable model to study non-Fermi liquid properties, holographic duality and maximally chaotic behavior. In this work, we consider a generalization of the SYK model that contains two SYK models with different number of Majorana modes coupled by quadratic terms. This model is also solvable, and the solution shows a zero-temperature quantum phase transition between two non-Fermi liquid chaotic phases. This phase transition is driven by tuning the ratio of two mode numbers, and a Fermi liquid non-chaotic phase sits at the critical point with equal mode number. At finite temperature, the Fermi liquid phase expands to a finite regime. More intriguingly, a different non-Fermi liquid phase emerges at finite temperature. We characterize the phase diagram in term of the spectral function, the Lyapunov exponent and the entropy. Our results illustrate a concrete example of quantum phase transition and critical regime between two non-Fermi liquid phases.

cond-mat.str-el↗