arXiv subjects

Zhuohan Li

Publications and source records attributed to Zhuohan Li.

36 records · Page 2Linked to original sources

arXiv2024

LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset

Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.

cs.CL↗

arXiv2023

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Evaluating large language model (LLM) based chat assistants is challenging due to their broad capabilities and the inadequacy of existing benchmarks in measuring human preferences. To address this, we explore using strong LLMs as judges to evaluate these models on more open-ended questions. We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases, as well as limited reasoning ability, and propose solutions to mitigate some of them. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Chatbot Arena, a crowdsourced battle platform. Our results reveal that strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans. Hence, LLM-as-a-judge is a scalable and explainable way to approximate human preferences, which are otherwise very expensive to obtain. Additionally, we show our benchmark and traditional benchmarks complement each other by evaluating several variants of LLaMA and Vicuna. The MT-bench questions, 3K expert votes, and 30K conversations with human preferences are publicly available at https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge.

cs.CL↗

arXiv2023

Efficient Memory Management for Large Language Model Serving with PagedAttention

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4$\times$ with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm

cs.LG↗

arXiv2023

AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving

Model parallelism is conventionally viewed as a method to scale a single large deep learning model beyond the memory limits of a single device. In this paper, we demonstrate that model parallelism can be additionally used for the statistical multiplexing of multiple devices when serving multiple models, even when a single model can fit into a single device. Our work reveals a fundamental trade-off between the overhead introduced by model parallelism and the opportunity to exploit statistical multiplexing to reduce serving latency in the presence of bursty workloads. We explore the new trade-off space and present a novel serving system, AlpaServe, that determines an efficient strategy for placing and parallelizing collections of large deep learning models across a distributed cluster. Evaluation results on production workloads show that AlpaServe can process requests at up to 10x higher rates or 6x more burstiness while staying within latency constraints for more than 99% of requests.

cs.LG↗

arXiv2023

FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU

The high computational and memory requirements of large language model (LLM) inference make it feasible only with multiple high-end accelerators. Motivated by the emerging demand for latency-insensitive tasks with batched processing, this paper initiates the study of high-throughput LLM inference using limited resources, such as a single commodity GPU. We present FlexGen, a high-throughput generation engine for running LLMs with limited GPU memory. FlexGen can be flexibly configured under various hardware resource constraints by aggregating memory and computation from the GPU, CPU, and disk. By solving a linear programming problem, it searches for efficient patterns to store and access tensors. FlexGen further compresses the weights and the attention cache to 4 bits with negligible accuracy loss. These techniques enable FlexGen to have a larger space of batch size choices and thus significantly increase maximum throughput. As a result, when running OPT-175B on a single 16GB GPU, FlexGen achieves significantly higher throughput compared to state-of-the-art offloading systems, reaching a generation throughput of 1 token/s for the first time with an effective batch size of 144. On the HELM benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21 hours. The code is available at https://github.com/FMInference/FlexGen

cs.LG↗

arXiv2023

Unsupervised machine learning for identifying phase transition using two-times clustering

In recent years, developing unsupervised machine learning for identifying phase transition is a research direction. In this paper, we introduce a two-times clustering method that can help select perfect configurations from a set of degenerate samples and assign the configuration with labels in a manner of unsupervised machine learning. These perfect configurations can then be used to train a neural network to classify phases. The derivatives of the predicted classification in the phase diagram, show peaks at the phase transition points. The effectiveness of our method is tested for the Ising, Potts, and Blume-Capel models. By using the ordered configuration from two-times clustering, our method can provide a useful way to obtain phase diagrams.

cond-mat.dis-nn↗

arXiv2022

The stellar parameters and elemental abundances from low-resolution spectra I: 1.2 million giants from LAMOST DR8

As a typical data-driven method, deep learning becomes a natural choice for analysing astronomical data nowadays. In this study, we built a deep convolutional neural network to estimate basic stellar parameters $T\rm{_{eff}}$, log g, metallicity ([M/H] and [Fe/H]) and [$α$/M] along with nine individual elemental abundances ([C/Fe], [N/Fe], [O/Fe], [Mg/Fe], [Al/Fe], [Si/Fe], [Ca/Fe], [Mn/Fe], [Ni/Fe]). The neural network is trained using common stars between the APOGEE survey and the LAMOST survey. We used low-resolution spectra from LAMOST survey as input, and measurements from APOGEE as labels. For stellar spectra with the signal-to-noise ratio in g band larger than 10 in the test set, the mean absolute error (MAE) is 29 K for $T\rm{_{eff}}$, 0.07 dex for log g, 0.03 dex for both [Fe/H] and [M/H], and 0.02 dex for [$α$/M]. The MAE of most elements is between 0.02 dex and 0.04 dex. The trained neural network was applied to 1,210,145 giants, including sub-giants, from LAMOST DR8 within the range of stellar parameters 3500 K < $T\rm{_{eff}}$ < 5500 K, 0.0 dex < log g < 4.0 dex, -2.5 dex < [Fe/H] < 0.5 dex. The distribution of our results in the chemical spaces is highly consistent with APOGEE labels and stellar parameters show consistency with external high-resolution measurements from GALAH. The results in this study allow us to further studies based on LAMOST data and deepen our understanding of the accretion and evolution history of the Milky Way. The electronic version of the value added catalog is available at http://www.lamost.org/dr8/v1.1/doc/vac.

astro-ph.GA↗

arXiv2022

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Alpa automates model-parallel training of large deep learning (DL) models by generating execution plans that unify data, operator, and pipeline parallelism. Existing model-parallel training systems either require users to manually create a parallelization plan or automatically generate one from a limited space of model parallelism configurations. They do not suffice to scale out complex DL models on distributed compute devices. Alpa distributes the training of large DL models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive efficient parallel execution plans at each parallelism level. Alpa implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Our evaluation shows Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans. Alpa's source code is publicly available at https://github.com/alpa-projects/alpa

cs.LG↗

arXiv2021

Hoplite: Efficient and Fault-Tolerant Collective Communication for Task-Based Distributed Systems

Task-based distributed frameworks (e.g., Ray, Dask, Hydro) have become increasingly popular for distributed applications that contain asynchronous and dynamic workloads, including asynchronous gradient descent, reinforcement learning, and model serving. As more data-intensive applications move to run on top of task-based systems, collective communication efficiency has become an important problem. Unfortunately, traditional collective communication libraries (e.g., MPI, Horovod, NCCL) are an ill fit, because they require the communication schedule to be known before runtime and they do not provide fault tolerance. We design and implement Hoplite, an efficient and fault-tolerant collective communication layer for task-based distributed systems. Our key technique is to compute data transfer schedules on the fly and execute the schedules efficiently through fine-grained pipelining. At the same time, when a task fails, the data transfer schedule adapts quickly to allow other tasks to keep making progress. We apply Hoplite to a popular task-based distributed framework, Ray. We show that Hoplite speeds up asynchronous stochastic gradient descent, reinforcement learning, and serving an ensemble of machine learning models that are difficult to execute efficiently with traditional collective communication by up to 7.8x, 3.9x, and 3.3x, respectively.

cs.DC↗

arXiv2021

TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models

Model parallelism has become a necessity for training modern large-scale deep language models. In this work, we identify a new and orthogonal dimension from existing model parallel approaches: it is possible to perform pipeline parallelism within a single training sequence for Transformer-based language models thanks to its autoregressive property. This enables a more fine-grained pipeline compared with previous work. With this key idea, we design TeraPipe, a high-performance token-level pipeline parallel algorithm for synchronous model-parallel training of Transformer-based language models. We develop a novel dynamic programming-based algorithm to calculate the optimal pipelining execution scheme given a specific model and cluster configuration. We show that TeraPipe can speed up the training by 5.0x for the largest GPT-3 model with 175 billion parameters on an AWS cluster with 48 p3.16xlarge instances compared with state-of-the-art model-parallel methods. The code for reproduction can be found at https://github.com/zhuohan123/terapipe

cs.LG↗

arXiv2021

Physical Origin of the Mechanochemical Coupling at Interfaces

We used density functional theory (DFT) calculations to investigate the physical origin of the mechano-chemical response of materials interfaces. Our results show that the mechano-chemical response can be decomposed into the contribution from the interface itself (deformation of interfacial bonds) and a contribution from the underlying solid. The relative contributions depend on the stiffness of these regions and the contact geometry, which affects the stress distribution within the bulk region. We demonstrate that, contrary to what is commonly assumed, the contribution to the activation volume from the elastic deformation of the surrounding bulk is significant and, in some case, may be dominant. We also show that the activation volume and the mechanochemical response of interfaces should be finite due to the effects on the stiffness and stress distribution within the near-surface bulk region. Our results indicate that the large range of activation volumes measured in the previous experiments even for the same material system might originate from the different degrees of contributions probed from the bulk vs. interface.

cond-mat.mtrl-sci↗

arXiv2020

Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers

Since hardware resources are limited, the objective of training deep learning models is typically to maximize accuracy subject to the time and memory constraints of training and inference. We study the impact of model size in this setting, focusing on Transformer models for NLP tasks that are limited by compute: self-supervised pretraining and high-resource machine translation. We first show that even though smaller Transformer models execute faster per iteration, wider and deeper models converge in significantly fewer steps. Moreover, this acceleration in convergence typically outpaces the additional computational overhead of using larger models. Therefore, the most compute-efficient training strategy is to counterintuitively train extremely large models but stop after a small number of iterations. This leads to an apparent trade-off between the training efficiency of large Transformer models and the inference efficiency of small Transformer models. However, we show that large models are more robust to compression techniques such as quantization and pruning than small models. Consequently, one can get the best of both worlds: heavily compressed, large models achieve higher accuracy than lightly compressed, small models.

cs.CL↗

arXiv2020

Fast Structured Decoding for Sequence Models

Autoregressive sequence models achieve state-of-the-art performance in domains like machine translation. However, due to the autoregressive factorization nature, these models suffer from heavy latency during inference. Recently, non-autoregressive sequence models were proposed to reduce the inference time. However, these models assume that the decoding process of each token is conditionally independent of others. Such a generation process sometimes makes the output sentence inconsistent, and thus the learned non-autoregressive models could only achieve inferior accuracy compared to their autoregressive counterparts. To improve then decoding consistency and reduce the inference cost at the same time, we propose to incorporate a structured inference module into the non-autoregressive models. Specifically, we design an efficient approximation for Conditional Random Fields (CRF) for non-autoregressive sequence models, and further propose a dynamic transition technique to model positional contexts in the CRF. Experiments in machine translation show that while increasing little latency (8~14ms), our model could achieve significantly better translation performance than previous non-autoregressive models on different translation datasets. In particular, for the WMT14 En-De dataset, our model obtains a BLEU score of 26.80, which largely outperforms the previous non-autoregressive baselines and is only 0.61 lower in BLEU than purely autoregressive models.

cs.LG↗

arXiv2019

Hint-Based Training for Non-Autoregressive Machine Translation

Due to the unparallelizable nature of the autoregressive factorization, AutoRegressive Translation (ART) models have to generate tokens sequentially during decoding and thus suffer from high inference latency. Non-AutoRegressive Translation (NART) models were proposed to reduce the inference time, but could only achieve inferior translation accuracy. In this paper, we proposed a novel approach to leveraging the hints from hidden states and word alignments to help the training of NART models. The results achieve significant improvement over previous NART models for the WMT14 En-De and De-En datasets and are even comparable to a strong LSTM-based ART baseline but one order of magnitude faster in inference.

cs.CL↗

arXiv2019

Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View

The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Ordinary Differential Equation (ODE) solver for a convection-diffusion equation in a multi-particle dynamic system. In particular, how words in a sentence are abstracted into contexts by passing through the layers of the Transformer can be interpreted as approximating multiple particles' movement in the space using the Lie-Trotter splitting scheme and the Euler's method. Given this ODE's perspective, the rich literature of numerical analysis can be brought to guide us in designing effective structures beyond the Transformer. As an example, we propose to replace the Lie-Trotter splitting scheme by the Strang-Marchuk splitting scheme, a scheme that is more commonly used and with much lower local truncation errors. The Strang-Marchuk splitting scheme suggests that the self-attention and position-wise feed-forward network (FFN) sub-layers should not be treated equally. Instead, in each layer, two position-wise FFN sub-layers should be used, and the self-attention sub-layer is placed in between. This leads to a brand new architecture. Such an FFN-attention-FFN layer is "Macaron-like", and thus we call the network with this new architecture the Macaron Net. Through extensive experiments, we show that the Macaron Net is superior to the Transformer on both supervised and unsupervised learning tasks. The reproducible codes and pretrained models can be found at https://github.com/zhuohan123/macaron-net

cs.LG↗

arXiv2018

Chemical aging of large-scale randomly rough frictional contacts

It has been shown that contact aging due to chemical reactions in single asperity contacts can have a significant effect on friction. However, it is currently unknown how chemically-induced contact aging of friction depends on roughness that is typically encountered in macroscopic rough contacts. Here, we develop an approach that brings together a kinetic Monte Carlo model of chemical aging with a contact mechanics model of rough surfaces based on the boundary element method to determine the magnitude of chemical aging in silica/silica contacts with random roughness. Our multi-scale model predicts that chemical aging for randomly rough contacts has a logarithmic dependence on time. It also shows that friction aging switches from a linear to a non-linear dependence on the applied load as the load increase. We discover that surface roughness affects the aging behavior primarily by modifying the real contact area and the local contact pressure, whereas the effect of contact morphology is relatively small. Our results demonstrate how understanding of chemical aging can be translated from studies of single asperity contacts to macroscopic rough contacts.

cond-mat.mtrl-sci↗

arXiv2018

Multiphysics model of chemical aging in frictional contacts

An increase of static friction during stationary contacts of two solids due to interfacial chemical bonding has been reported in multiple experiments. However, the physics underlying such frictional aging is still not fully understood because it involves multiple physical and chemical effects coupled with each other, making direct interpretation of experimental results difficult. Here, we develop a multiphysics chemical aging model that combines contact mechanics, mechanochemistry, and interfacial chemical reaction kinetics. Our model predicts that aging is proportional to normal loads in a low-load regime and becomes nonlinear at higher loads. We also discovered a nonmonotonic temperature dependence of aging with a peak near room temperature. In addition, our simulations provide insights into contributions from specific physical/chemical effects on the overall aging. Our model shows quantitative agreement with available single-asperity experiments on silica-silica interfaces, and it provides a framework for building a chemical aging model for other material systems with arbitrary types of physical and chemical effects involved.

cond-mat.soft↗

arXiv2018

Towards Binary-Valued Gates for Robust LSTM Training

Long Short-Term Memory (LSTM) is one of the most widely used recurrent structures in sequence modeling. It aims to use gates to control information flow (e.g., whether to skip some information or not) in the recurrent computations, although its practical implementation based on soft gates only partially achieves this goal. In this paper, we propose a new way for LSTM training, which pushes the output values of the gates towards 0 or 1. By doing so, we can better control the information flow: the gates are mostly open or closed, instead of in a middle state, which makes the results more interpretable. Empirical studies show that (1) Although it seems that we restrict the model capacity, there is no performance drop: we achieve better or comparable performances due to its better generalization ability; (2) The outputs of gates are not sensitive to their inputs: we can easily compress the LSTM unit in multiple ways, e.g., low-rank approximation and low-precision approximation. The compressed models are even better than the baseline models without compression.

cs.LG↗