arXiv subjects

Weinan Li

Publications and source records attributed to Weinan Li.

3 recordsLinked to original sources

arXiv2025

Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching

Large Language Models (LLMs) exhibit pronounced memory-bound characteristics during inference due to High Bandwidth Memory (HBM) bandwidth constraints. In this paper, we propose an L2 Cache-oriented asynchronous KV Cache prefetching method to break through the memory bandwidth bottleneck in LLM inference through computation-load overlap. By strategically scheduling idle memory bandwidth during active computation windows, our method proactively prefetches required KV Cache into GPU L2 cache, enabling high-speed L2 cache hits for subsequent accesses and effectively hiding HBM access latency within computational cycles. Extensive experiments on NVIDIA H20 GPUs demonstrate that the proposed method achieves 2.15x improvement in attention kernel efficiency and up to 1.97x end-to-end throughput enhancement, surpassing state-of-the-art baseline FlashAttention-3. Notably, our solution maintains orthogonality to existing optimization techniques and can be integrated with current inference frameworks, providing a scalable latency-hiding solution for next-generation LLM inference engines.

cs.LG↗

arXiv2023

DxPU: Large Scale Disaggregated GPU Pools in the Datacenter

The rapid adoption of AI and convenience offered by cloud services have resulted in the growing demands for GPUs in the cloud. Generally, GPUs are physically attached to host servers as PCIe devices. However, the fixed assembly combination of host servers and GPUs is extremely inefficient in resource utilization, upgrade, and maintenance. Due to these issues, the GPU disaggregation technique has been proposed to decouple GPUs from host servers. It aggregates GPUs into a pool, and allocates GPU node(s) according to user demands. However, existing GPU disaggregation systems have flaws in software-hardware compatibility, disaggregation scope, and capacity. In this paper, we present a new implementation of datacenter-scale GPU disaggregation, named DxPU. DxPU efficiently solves the above problems and can flexibly allocate as many GPU node(s) as users demand. In order to understand the performance overhead incurred by DxPU, we build up a performance model for AI specific workloads. With the guidance of modeling results, we develop a prototype system, which has been deployed into the datacenter of a leading cloud provider for a test run. We also conduct detailed experiments to evaluate the performance overhead caused by our system. The results show that the overhead of DxPU is less than 10%, compared with native GPU servers, in most of user scenarios.

cs.DC↗

arXiv2023

Time-variant Seismic Resilience Analysis Model for Water Distribution Systems

Water distribution systems (WDS) sustained severe damage in the past earthquakes. While previous studies investigated the seismic performance of buried water pipelines, the effects of corrosion on the pipeline seismic performance were ignored. The presence of corrosion on metallic pipeline walls aggravates seismic damage level because corrosion significantly reduces the pipeline strength. Most of the existing buried pipelines in the United States are aged and non-ductile metallic pipelines, which are vulnerable to seismic loading. To ensure continuous and smooth water supply to communities during and after earthquakes, it is necessary to evaluate the system-level seismic performance of WDS considering the aging effect in corroded pipelines. The current study develops a new framework of estimating the seismic resilience of WDS considering the time-variant effect of corrosion. The study formulates an approach that : (1) determines the seismic failure probability of pipeline using an extended American Lifelines Alliance (ALA) model that account for the effects of time-dependent corrosion; and (2) estimates system-level seismic performance based on pipeline reliability and edge betweenness centrality. The proposed approach is illustrated with a scenario earthquake hazard for mid-size WDS. The outcomes of the study reveal that the presence of corrosion on pipelines significantly reduces the system-level seismic performance of WDS. Most cast iron pipes have 100 years lifetime, system-level seismic resilience may decrease by 81% at high seismic wave intensity.

physics.geo-ph↗