Searcharxiv⌕ Search

arXiv subjects

Shen Li

Publications and source records attributed to Shen Li.

At least 91 records · Page 5Linked to original sources
arXiv2021

PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers

The size of Transformer models is growing at an unprecedented pace. It has only taken less than one year to reach trillion-level parameters after the release of GPT-3 (175B). Training such models requires both substantial engineering efforts and enormous computing resources, which are luxuries most research teams cannot afford. In this paper, we propose PipeTransformer, which leverages automated and elastic pipelining and data parallelism for efficient distributed training of Transformer models. PipeTransformer automatically adjusts the pipelining and data parallelism by identifying and freezing some layers during the training, and instead allocates resources for training of the remaining active layers. More specifically, PipeTransformer dynamically excludes converged layers from the pipeline, packs active layers into fewer GPUs, and forks more replicas to increase data-parallel width. We evaluate PipeTransformer using Vision Transformer (ViT) on ImageNet and BERT on GLUE and SQuAD datasets. Our results show that PipeTransformer attains a 2.4 fold speedup compared to the state-of-the-art baseline. We also provide various performance analyses for a more comprehensive understanding of our algorithmic and system-wise design. We also develop open-sourced flexible APIs for PipeTransformer, which offer a clean separation among the freeze algorithm, model definitions, and training accelerations, hence allowing it to be applied to other algorithms that require similar freezing strategies.

cs.LG↗
arXiv2020

PyTorch Distributed: Experiences on Accelerating Data Parallel Training

This paper presents the design, implementation, and evaluation of the PyTorch distributed data parallel module. PyTorch is a widely-adopted scientific computing package used in deep learning research and applications. Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability. In general, the technique of distributed data parallelism replicates the model on every computational resource to generate gradients independently and then communicates those gradients at each iteration to keep model replicas consistent. Despite the conceptual simplicity of the technique, the subtle dependencies between computation and communication make it non-trivial to optimize the distributed training efficiency. As of v1.5, PyTorch natively provides several techniques to accelerate distributed data parallel, including bucketing gradients, overlapping computation with communication, and skipping gradient synchronization. Evaluations show that, when configured appropriately, the PyTorch distributed data parallel module attains near-linear scalability using 256 GPUs.

cs.DC↗
arXiv2020

Identifying through Flows for Recovering Latent Representations

Identifiability, or recovery of the true latent representations from which the observed data originates, is de facto a fundamental goal of representation learning. Yet, most deep generative models do not address the question of identifiability, and thus fail to deliver on the promise of the recovery of the true latent sources that generate the observations. Recent work proposed identifiable generative modelling using variational autoencoders (iVAE) with a theory of identifiability. Due to the intractablity of KL divergence between variational approximate posterior and the true posterior, however, iVAE has to maximize the evidence lower bound (ELBO) of the marginal likelihood, leading to suboptimal solutions in both theory and practice. In contrast, we propose an identifiable framework for estimating latent representations using a flow-based model (iFlow). Our approach directly maximizes the marginal likelihood, allowing for theoretical guarantees on identifiability, thereby dispensing with variational approximations. We derive its optimization objective in analytical form, making it possible to train iFlow in an end-to-end manner. Simulations on synthetic data validate the correctness and effectiveness of our proposed method and demonstrate its practical advantages over other existing methods.

cs.LG↗
arXiv2020

A robust fiber-based quantum thermometer coupled with nitrogen-vacancy centers

The nitrogen-vacancy center in diamond has been broadly applied in quantum sensing since it is sensitive to different physical quantities. Meanwhile, it is difficult to isolate disturbances from unwanted physical quantities in practical applications. Here, we present a robust fiber-based quantum thermometer which can significantly isolate the magnetic field noise and microwave power shift. With a frequency modulation scheme, we realize the temperature measurement by detecting the variation of the sharp-dip in the zero-field optically detected magnetic resonance spectrum in a high-density nitrogen-vacancy ensemble. Thanks to its simplicity and compatibility in implementation and robustness in the isolation of magnetic and microwave noise, this quantum thermometer is then applied to the surface temperature imaging of an electronic chip with a sensitivity of $18$ $\rm{mK}/\sqrt{\rm{Hz}}$. It paves the way to high sensitive temperature measurement in ambiguous environments.

physics.app-ph↗
arXiv2020

Quantum Inspired Word Representation and Computation

Word meaning has different aspects, while the existing word representation "compresses" these aspects into a single vector, and it needs further analysis to recover the information in different dimensions. Inspired by quantum probability, we represent words as density matrices, which are inherently capable of representing mixed states. The experiment shows that the density matrix representation can effectively capture different aspects of word meaning while maintaining comparable reliability with the vector representation. Furthermore, we propose a novel method to combine the coherent summation and incoherent summation in the computation of both vectors and density matrices. It achieves consistent improvement on word analogy task.

cs.CL↗
arXiv2020

Planning With Uncertain Specifications (PUnS)

Reward engineering is crucial to high performance in reinforcement learning systems. Prior research into reward design has largely focused on Markovian functions representing the reward. While there has been research into expressing non-Markov rewards as linear temporal logic (LTL) formulas, this has focused on task specifications directly defined by the user. However, in many real-world applications, task specifications are ambiguous, and can only be expressed as a belief over LTL formulas. In this paper, we introduce planning with uncertain specifications (PUnS), a novel formulation that addresses the challenge posed by non-Markovian specifications expressed as beliefs over LTL formulas. We present four criteria that capture the semantics of satisfying a belief over specifications for different applications, and analyze the qualitative implications of these criteria within a synthetic domain. We demonstrate the existence of an equivalent Markov decision process (MDP) for any instance of PUnS. Finally, we demonstrate our approach on the real-world task of setting a dinner table automatically with a robot that inferred task specifications from human demonstrations.

cs.RO↗
arXiv2019

Generalize Symbolic Knowledge With Neural Rule Engine

As neural networks have dominated the state-of-the-art results in a wide range of NLP tasks, it attracts considerable attention to improve the performance of neural models by integrating symbolic knowledge. Different from existing works, this paper investigates the combination of these two powerful paradigms from the knowledge-driven side. We propose Neural Rule Engine (NRE), which can learn knowledge explicitly from logic rules and then generalize them implicitly with neural networks. NRE is implemented with neural module networks in which each module represents an action of a logic rule. The experiments show that NRE could greatly improve the generalization abilities of logic rules with a significant increase in recall. Meanwhile, the precision is still maintained at a high level.

cs.CL↗
arXiv2019

Self-Balanced Dropout

Dropout is known as an effective way to reduce overfitting via preventing co-adaptations of units. In this paper, we theoretically prove that the co-adaptation problem still exists after using dropout due to the correlations among the inputs. Based on the proof, we further propose Self-Balanced Dropout, a novel dropout method which uses a trainable variable to balance the influence of the input correlation on parameter update. We evaluate Self-Balanced Dropout on a range of tasks with both simple and complex models. The experimental results show that the mechanism can effectively solve the co-adaption problem to some extent and significantly improve the performance on all tasks.

cs.CL↗
arXiv2019

Algebras with finite relative dominant dimension and almost n-precluster tilting modules

In this paper, we investigate the relative dominant dimension with respect to an injective module and characterize the algebras with finite relative dominant dimension. As an application, we introduce the almost n-precluster tilting module and establish a correspondence between almost n-precluster tilting modules and almost n-minimal Auslander-Gorenstein algebras. Moreover, we give a description of the Gorenstein projective modules over almost n-minimal Auslander-Gorenstein algebras in terms of the corresponding almost n-precluster tilting modules.

math.RT↗
arXiv2019

Thermal-demagnetization-enhanced hybrid fiber-based thermometer coupled with nitrogen-vacancy centers

Nitrogen-vacancy centers in diamond are attractive as quantum sensors owing to their remarkable optical and spin properties under ambient conditions. Here we experimentally demonstrated a hybrid fiber-based thermometer coupled with nitrogen-vacancy center ensemble and a permanent magnet, where the temperature sensitivity was improved by converting the temperature variation to the magnetic field change based on the thermal-demagnetization of the permanent magnet. We have achieved both large temperature working range (room temperature to 373 K) and millikelvin sensitivity (1.6 mK/sqrt(Hz)), nearly 6-fold improvement compared with conventional technique. This stable and compact hybrid thermometer will enable a wide range of applications for large-area detection and imaging with high temperature sensitivity.

physics.ins-det↗
arXiv2018

High contrast quantum imaging with time-gated fluorescence detection

Optical detection of spin state has been widely used for the solid state spin qubit in the application of quantum information processing. The signal contrast determines the accuracy of quantum state manipulation, sensitivity of quantum sensing and resolution of quantum imaging. Here, we demonstrated a time-gated fluorescence detection method for enhancing the spin state signal contrast of nitrogen vacancy (NV) center in diamond. By adjusting the delay between time gate and the excitation laser pulse, we improved both the signal contrast and signal-to-noise ratio for NV spin detection. An enhancement ratio of 1.86 times was reached for the signal contrast. Utilizing the time-gated fluorescence detection, we further demonstrated a high contrast quantum imaging of nanoparticle's stray magnetic field. Without any additional manipulation of the quantum state, we expect that this method can be used to improve the performance of various applications with NV center.

physics.optics↗
arXiv2018

From Random to Supervised: A Novel Dropout Mechanism Integrated with Global Information

Dropout is used to avoid overfitting by randomly dropping units from the neural networks during training. Inspired by dropout, this paper presents GI-Dropout, a novel dropout method integrating with global information to improve neural networks for text classification. Unlike the traditional dropout method in which the units are dropped randomly according to the same probability, we aim to use explicit instructions based on global information of the dataset to guide the training process. With GI-Dropout, the model is supposed to pay more attention to inapparent features or patterns. Experiments demonstrate the effectiveness of the dropout with global information on seven text classification tasks, including sentiment analysis and topic classification.

cs.CL↗
arXiv2018

Percolation Phase Transition from Ionic Liquids to Ionic Liquid Crystals

As typical complex liquids, ionic liquids (ILs) exhibit phases beyond the description of simple liquid theories. In particular, with an intermediate cationic side-chain length, ILs can form the nanoscale segregated liquid (NSL) phase, which will eventually transform into the ionic liquid crystal (ILC) phase when the side chains are adequately long. However, the microscopic mechanism of this transformation is still unclear. In this work, by means of coarse-grained molecular dynamics simulation, we show that, with increasing cationic side-chain length, some local pieces of non-polar domains are gradually formed by side chains aligned in parallel inside the NSL phase, before an abrupt percolation phase transition happens when the system transforms into the ILC phase, manifesting that it is a critical phenomenon. Percolation phase transition is applied to ILs, providing new insights into many recent observations both in experiments and simulations.

cond-mat.soft↗
arXiv2018

Cooperative Control of Heterogeneous Connected Vehicles with Directed Acyclic Interactions

Cooperation of multiple connected vehicles has the potential to benefit the road traffic greatly. In this paper, we consider analysis and synthesis problems of the cooperative control of a platoon of heterogeneous connected vehicles with directed acyclic interactions (characterized by directed acyclic graphs). In contrast to previous works that view heterogeneity as a type of uncertainty, this paper directly takes heterogeneity into account in the problem formulation, allowing us to develop a deeper understanding of the influence of heterogeneity on the collective behavior of a platoon of connected vehicles. Our major strategies include an application of the celebrated internal model principle and an exploitation of lower-triangular structures for platoons with directed acyclic interactions. The major findings include: 1) we explicitly highlight the tracking ability of heterogeneous platoons, showing that the followers can only track the leader's spacing and velocity; 2) we analytically derive a stability region of feedback gains for platoons with directed acyclic interactions; 3) and consequently we propose a synthesis method based on the solution to an algebraic Riccati equation that shares the dimension of single vehicle dynamics. Numerical experiments are carried out to validate the effectiveness of our results.

math.OC↗
arXiv2018

Analogical Reasoning on Chinese Morphological and Semantic Relations

Analogical reasoning is effective in capturing linguistic regularities. This paper proposes an analogical reasoning task on Chinese. After delving into Chinese lexical knowledge, we sketch 68 implicit morphological relations and 28 explicit semantic relations. A big and balanced dataset CA8 is then built for this task, including 17813 questions. Furthermore, we systematically explore the influences of vector representations, context features, and corpora on analogical reasoning. With the experiments, CA8 is proved to be a reliable benchmark for evaluating Chinese word embeddings.

cs.CL↗
arXiv2018

Gorenstein projective dimensions of modules over minimal Auslander-Gorenstein algebras

In this article we investigate the relations between the Gorenstein projective dimensions of $Λ$-modules and their socles for minimal n-Auslander-Gorenstein algebras $Λ$ in the sense of Iyama and Solberg \cite{IS}. First we give a description of projective-injective $Λ$-modules in terms of their socles. Then we prove that a $Λ$-module $N$ has Gorenstein projective dimension at most n iff its socle has Gorenstein projective dimension at most n iff $N$ is cogenerated by a projective $Λ$-module. Furthermore, we show that minimal n-Auslander-Gorenstein algebras can be characterised by the relations between the Gorenstein projective dimensions of modules and their socles.

math.RT↗