arXiv subjects

Lijuan Jiang

Publications and source records attributed to Lijuan Jiang.

4 recordsLinked to original sources

arXiv2025

A Flexible Programmable Pipeline Parallelism Framework for Efficient DNN Training

Pipeline parallelism is an essential distributed parallelism method. Increasingly complex and diverse DNN models necessitate meticulously customized pipeline schedules for performance. However, existing practices typically rely on predefined schedules, each with strengths, but fail to adapt automatically to the emerging model architectures. Exploring novel high-efficiency schedules is daunting due to the enormous and varying schedule space. Besides, manually implementing schedules can be challenging due to the onerous coding burdens and constantly changing needs. Unfortunately, existing frameworks have limitations in automated schedule exploration and lack flexibility and controllability. This paper presents FlexPipe, a programmable pipeline parallelism framework with enhanced productivity, programmability, debuggability, and ease of tuning. FlexPipe has two main components: a succinct domain-specific language (DSL) and an automated scheduler. FlexPipe enables automated schedule exploration for various parallel scenarios within a broad spectrum of schedule types at a small search cost. Besides, users can swiftly develop and customize schedules using the FlexPipe DSL, which embodies flexible controllability in the pipeline order of micro-batch computations over stages. It also provides convenient mechanisms to include new operations in schedules to meet changing demands. Our evaluation results demonstrate that FlexPipe achieves up to 2.28X performance speedup compared to the popular large-scale parallel framework Megtron-LM, and gains up to 1.49X performance speedup compared to the state-of-the-art automated pipeline parallelism framework.

cs.DC↗

arXiv2024

Efficient Training of Large Language Models on Distributed Infrastructures: A Survey

Large Language Models (LLMs) like GPT and LLaMA are revolutionizing the AI industry with their sophisticated capabilities. Training these models requires vast GPU clusters and significant computing time, posing major challenges in terms of scalability, efficiency, and reliability. This survey explores recent advancements in training systems for LLMs, including innovations in training infrastructure with AI accelerators, networking, storage, and scheduling. Additionally, the survey covers parallelism strategies, as well as optimizations for computation, communication, and memory in distributed LLM training. It also includes approaches of maintaining system reliability over extended training periods. By examining current innovations and future directions, this survey aims to provide valuable insights towards improving LLM training systems and tackling ongoing challenges. Furthermore, traditional digital circuit-based computing systems face significant constraints in meeting the computational demands of LLMs, highlighting the need for innovative solutions such as optical computing and optical networks.

cs.DC↗

arXiv2024

ZeroPP: Unleashing Exceptional Parallelism Efficiency through Tensor-Parallelism-Free Methodology

Large-scale models rely heavily on 3D parallelism for distributed training, which utilizes tensor parallelism (TP) as the intra-operator parallelism to partition model states across GPUs. However, TP introduces significant communication overheads and complexity in modifying single-GPU code. In this paper, we propose a TP-free distributed framework ZeroPP, which leverages the hybrid of scalable inter-operator pipeline parallelism and intra-operator fully sharded data parallelism to train models at scale, reducing memory consumption and enabling high training efficiency. Through extensive experimentation, we demonstrate that ZeroPP achieves significant performance gains of up to 33% compared to conventional 3D parallelism while maintaining comparable GPU memory consumption.

cs.DC↗

arXiv2019

Unilateral Left-Tail Anderson Darling Test Based Spectrum Sensing with Laplacian Noise

This paper focuses on spectrum sensing under Laplacian noise. To remit the negative effects caused by heavy-tailed behavior of Laplacian noise, the fractional lower order moments (FLOM) technology is employed to pre-process the received samples before spectrum sensing. Via exploiting the asymmetrical difference between the distribution for the FLOM of received samples in the absence and presence of primary users, we formulate the spectrum sensing problem under Laplacian noise as a unilateral goodness-of-fit (GoF) test problem. Based on this test problem, we propose a new GoF-based detector which is called unilateral left-tail Anderson Darling (ULAD) detector. The analytical expressions for the theoretical performance, in terms of false-alarm and detection probabilities, of the ULAD are derived. Moreover, a closed-form expression for the optimal detection threshold is also derived to minimize the total error rate. Simulation results are provided to validate the theoretical analyses and to demonstrate the superior performance of the proposed detector than others.

cs.IT↗