arXiv subjects

Yingtao Li

Publications and source records attributed to Yingtao Li.

5 recordsLinked to original sources

arXiv2026

DisDP: Disaggregating Compute, Network, and Storage for Model-Sharded Data-Parallel Training

Model-sharded data parallelism (MSDP), e.g., ZeRO, evenly shards the model states across all GPUs, and thus has been widely adopted by LLM pre-training, such as Llama and DeepSeek, due to its low GPU memory capacity requirement. However, MSDP introduces severe overhead from additional network communication collectives (i.e., AllGather and ReduceScatter). Although the collectives themselves only occupy fewer than 10% of GPU SMs, their execution time increases by 41% due to the serial execution of aggregated CPU/GPU-managed compute (i.e., GEMM), network (i.e., NCCL), and storage (i.e., optimizer states). To this end, we present DisDP, a fully disaggregated distributed data-parallel architecture that first fully disaggregates compute, network, and storage for MSDP, such that GPUs only focus on the computing part, and thus the GPU utilization is maximized. The key idea is 1) fully offloading collectives to SmartNICs and SmartSwitch to avoid interference between GEMM kernels and collective kernels, and 2) fully offloading storage to a SmartSwitch-enhanced parameter server that allows a single PS to serve massive workers with linear scalability. DisDP on 8 distributed GPUs outperforms the state-of-the-art training systems by 3.98x when training on a 175B model, validating the efficiency of disaggregation.

cs.DC↗

arXiv2025

Breaking the Vicious Cycle: Coherent 3D Gaussian Splatting from Sparse and Motion-Blurred Views

3D Gaussian Splatting (3DGS) has emerged as a state-of-the-art method for novel view synthesis. However, its performance heavily relies on dense, high-quality input imagery, an assumption that is often violated in real-world applications, where data is typically sparse and motion-blurred. These two issues create a vicious cycle: sparse views ignore the multi-view constraints necessary to resolve motion blur, while motion blur erases high-frequency details crucial for aligning the limited views. Thus, reconstruction often fails catastrophically, with fragmented views and a low-frequency bias. To break this cycle, we introduce CoherentGS, a novel framework for high-fidelity 3D reconstruction from sparse and blurry images. Our key insight is to address these compound degradations using a dual-prior strategy. Specifically, we combine two pre-trained generative models: a specialized deblurring network for restoring sharp details and providing photometric guidance, and a diffusion model that offers geometric priors to fill in unobserved regions of the scene. This dual-prior strategy is supported by several key techniques, including a consistency-guided camera exploration module that adaptively guides the generative process, and a depth regularization loss that ensures geometric plausibility. We evaluate CoherentGS through both quantitative and qualitative experiments on synthetic and real-world scenes, using as few as 3, 6, and 9 input views. Our results demonstrate that CoherentGS significantly outperforms existing methods, setting a new state-of-the-art for this challenging task. The code and video demos are available at https://potatobigroom.github.io/CoherentGS/.

cs.CV↗

arXiv2025

FpgaHub: Fpga-centric Hyper-heterogeneous Computing Platform for Big Data Analytics

Modern data analytics requires a huge amount of computing power and processes a massive amount of data. At the same time, the underlying computing platform is becoming much more heterogeneous on both hardware and software. Even though specialized hardware, e.g., FPGA- or GPU- or TPU-based systems, often achieves better performance than a CPU-only system due to the slowing of Moore's law, such systems are limited in what they can do. For example, GPU-only approaches suffer from severe IO limitations. To truly exploit the potential of hardware heterogeneity, we present FpgaHub, an FPGA-centric hyper-heterogeneous computing platform for big data analytics. The key idea of FpgaHub is to use reconfigurable computing to implement a versatile hub complementing other processors (CPUs, GPUs, DPUs, programmable switches, computational storage, etc.). Using an FPGA as the basis, we can take advantage of its highly reconfigurable nature and rich IO interfaces such as PCIe, networking, and on-board memory, to place it at the center of the architecture and use it as a data and control plane for data movement, scheduling, pre-processing, etc. FpgaHub enables architectural flexibility to allow exploring the rich design space of heterogeneous computing platforms.

cs.DC↗

arXiv2023

P4SGD: Programmable Switch Enhanced Model-Parallel Training on Generalized Linear Models on Distributed FPGAs

Generalized linear models (GLMs) are a widely utilized family of machine learning models in real-world applications. As data size increases, it is essential to perform efficient distributed training for these models. However, existing systems for distributed training have a high cost for communication and often use large batch sizes to balance computation and communication, which negatively affects convergence. Therefore, we argue for an efficient distributed GLM training system that strives to achieve linear scalability, while keeping batch size reasonably low. As a start, we propose P4SGD, a distributed heterogeneous training system that efficiently trains GLMs through model parallelism between distributed FPGAs and through forward-communication-backward pipeline parallelism within an FPGA. Moreover, we propose a light-weight, latency-centric in-switch aggregation protocol to minimize the latency of the AllReduce operation between distributed FPGAs, powered by a programmable switch. As such, to our knowledge, P4SGD is the first solution that achieves almost linear scalability between distributed accelerators through model parallelism. We implement P4SGD on eight Xilinx U280 FPGAs and a Tofino P4 switch. Our experiments show P4SGD converges up to 6.5X faster than the state-of-the-art GPU counterpar.

cs.DC↗

arXiv2016

A Unified Framework for Information Consumption Based on Markov Chains

This paper establishes a Markov chain model as a unified framework for understanding information consumption processes in complex networks, with clear implications to the Internet and big-data technologies. In particular, the proposed model is the first one to address the formation mechanism of the "trichotomy" in observed probability density functions from empirical data of various social and technical networks. Both simulation and experimental results demonstrate a good match of the proposed model with real datasets, showing its superiority over the classical power-law models.

cs.SI↗