arXiv subjects

Yutong Lu

Publications and source records attributed to Yutong Lu.

62 records · Page 4Linked to original sources

arXiv2022

EnergonAI: An Inference System for 10-100 Billion Parameter Transformer Models

Large transformer models display promising performance on a wide range of natural language processing (NLP) tasks. Although the AI community has expanded the model scale to the trillion parameter level, the practical deployment of 10-100 billion parameter models is still uncertain due to the latency, throughput, and memory constraints. In this paper, we proposed EnergonAI to solve the challenges of the efficient deployment of 10-100 billion parameter transformer models on single- or multi-GPU systems. EnergonAI adopts a hierarchy-controller system architecture to coordinate multiple devices and efficiently support different parallel patterns. It delegates the execution of sub-models to multiple workers in the single-controller style and applies tensor parallelism and pipeline parallelism among the workers in a multi-controller style. Upon the novel architecture, we propose three techniques, i.e. non-blocking pipeline parallelism, distributed redundant computation elimination, and peer memory pooling. EnergonAI enables the users to program complex parallel code the same as a serial one. Compared with the FasterTransformer, we have proven that EnergonAI has superior performance on latency and throughput. In our experiments, EnergonAI can achieve 37% latency reduction in tensor parallelism, 10% scalability improvement in pipeline parallelism, and it improves the model scale inferred on a single GPU by using a larger heterogeneous memory space at cost of limited performance reduction.

cs.LG↗

arXiv2022

LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose \textbf{LayoutLMv3} to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. The code and models are publicly available at \url{https://aka.ms/layoutlmv3}.

cs.CL↗

arXiv2021

Leveraging Large-scale Computational Database and Deep Learning for Accurate Prediction of Material Properties

Accurately predicting the physical and chemical properties of materials remains one of the most challenging tasks in material design, and one effective strategy is to construct a reliable data set and use it for training a machine learning model. In this study, we constructed a large-scale material genome database (Matgen) containing 76,463 materials collected from experimentally-observed database, and computed their bandgap properties through the Density functional theory (DFT) method with Perdew-Burke-Ernzehof (PBE) functional. We verified the computation method by comparing part of our results with those from the open Material Project (MP) and Open Quantum Materials Database (OQMD), all with PBE computations, and found that Matgen achieved the same computation accuracy based on both measured and computed bandgap properties. Based on the computed properties of our comprehensive dataset, we have developed a new graph-based deep learning model, namely CrystalNet, through our recently developed Communicative Message Passing Neural Network (CMPNN) framework. The model was shown to outperform other state-of-the-art prediction models. A further fine-tuning on 1716 experimental bandgap values (CrystalNet-TL) achieved a superior performance with mean absolute error (MAE) of 0.77 eV on independent test, which has outperformed the pure PBE (1.14~1.45 eV). Moreover, the model was proven applicable to hypothetical materials with MAE of 0.77 eV as referred by computations from HSE, a highly accurate quantum mechanics (QM) method, consist better than PBE (MAE=1.13eV). We also made material structures, computed properties by PBE, and the CrystalNet models publically available at https://matgen.nscc-gz.cn.

cond-mat.mtrl-sci↗

arXiv2021

A Picture is Worth a Thousand Words: A Unified System for Diverse Captions and Rich Images Generation

A creative image-and-text generative AI system mimics humans' extraordinary abilities to provide users with diverse and comprehensive caption suggestions, as well as rich image creations. In this work, we demonstrate such an AI creation system to produce both diverse captions and rich images. When users imagine an image and associate it with multiple captions, our system paints a rich image to reflect all captions faithfully. Likewise, when users upload an image, our system depicts it with multiple diverse captions. We propose a unified multi-modal framework to achieve this goal. Specifically, our framework jointly models image-and-text representations with a Transformer network, which supports rich image creation by accepting multiple captions as input. We consider the relations among input captions to encourage diversity in training and adopt a non-autoregressive decoding strategy to enable real-time inference. Based on these, our system supports both diverse captions and rich images generations. Our code is available online.

cs.CV↗

arXiv2021

Unifying Multimodal Transformer for Bi-directional Image and Text Generation

We study the joint learning of image-to-text and text-to-image generations, which are naturally bi-directional tasks. Typical existing works design two separate task-specific models for each task, which impose expensive design efforts. In this work, we propose a unified image-and-text generative framework based on a single multimodal model to jointly study the bi-directional tasks. We adopt Transformer as our unified architecture for its strong performance and task-agnostic design. Specifically, we formulate both tasks as sequence generation tasks, where we represent images and text as unified sequences of tokens, and the Transformer learns multimodal interactions to generate sequences. We further propose two-level granularity feature representations and sequence-level training to improve the Transformer-based unified framework. Experiments show that our approach significantly improves previous Transformer-based model X-LXMERT's FID from 37.0 to 29.9 (lower is better) for text-to-image generation, and improves CIDEr-D score from 100.9% to 122.6% for fine-tuned image-to-text generation on the MS-COCO dataset. Our code is available online.

cs.CV↗

arXiv2020

A parallel structured divide-and-conquer algorithm for symmetric tridiagonal eigenvalue problems

In this paper, a parallel structured divide-and-conquer (PSDC) eigensolver is proposed for symmetric tridiagonal matrices based on ScaLAPACK and a parallel structured matrix multiplication algorithm, called PSMMA. Computing the eigenvectors via matrix-matrix multiplications is the most computationally expensive part of the divide-and-conquer algorithm, and one of the matrices involved in such multiplications is a rank-structured Cauchy-like matrix. By exploiting this particular property, PSMMA constructs the local matrices by using generators of Cauchy-like matrices without any communication, and further reduces the computation costs by using a structured low-rank approximation algorithm. Thus, both the communication and computation costs are reduced. Experimental results show that both PSMMA and PSDC are highly scalable and scale to 4096 processes at least. PSDC has better scalability than PHDC that was proposed in [J. Comput. Appl. Math. 344 (2018) 512--520] and only scaled to 300 processes for the same matrices. Comparing with \texttt{PDSTEDC} in ScaLAPACK, PSDC is always faster and achieves $1.4$x--$1.6$x speedup for some matrices with few deflations. PSDC is also comparable with ELPA, with PSDC being faster than ELPA when using few processes and a little slower when using many processes.

cs.MS↗

arXiv2019

Decoupling Localization and Classification in Single Shot Temporal Action Detection

Video temporal action detection aims to temporally localize and recognize the action in untrimmed videos. Existing one-stage approaches mostly focus on unifying two subtasks, i.e., localization of action proposals and classification of each proposal through a fully shared backbone. However, such design of encapsulating all components of two subtasks in one single network might restrict the training by ignoring the specialized characteristic of each subtask. In this paper, we propose a novel Decoupled Single Shot temporal Action Detection (Decouple-SSAD) method to mitigate such problem by decoupling the localization and classification in a one-stage scheme. Particularly, two separate branches are designed in parallel to enable each component to own representations privately for accurate localization or classification. Each branch produces a set of action anchor layers by applying deconvolution to the feature maps of the main stream. Each branch produces a set of feature maps by applying deconvolution to the feature maps of the main stream. High-level semantic information from deeper layers is thus incorporated to enhance the feature representations. We conduct extensive experiments on THUMOS14 dataset and demonstrate superior performance over state-of-the-art methods. Our code is available online.

cs.CV↗

arXiv2016

Differential Neutrino Condensation onto Cosmic Structure

Astrophysical techniques have pioneered the discovery of neutrino mass properties. Current cosmological observations give an upper bound on neutrino masses by attempting to disentangle the small neutrino contribution from the sum of all matter using precise theoretical models. We discover the differential neutrino condensation effect in our TianNu N-body simulation. Neutrino masses can be inferred using this effect by comparing galaxy properties in regions of the universe with different neutrino relative abundance (i.e. the local neutrino to cold dark matter density ratio). In "neutrino-rich"' regions, more neutrinos can be captured by massive halos compared to "neutrino-poor" regions. This effect differentially skews the halo mass function and opens up the path to independent neutrino mass measurements in current or future galaxy surveys.

astro-ph.CO↗