arXiv subjects

Euijun Chung

Publications and source records attributed to Euijun Chung.

6 recordsLinked to original sources

arXiv2026

Characterizing CPU-Induced Slowdowns in Multi-GPU LLM Inference

Large-scale machine learning workloads increasingly rely on multi-GPU systems, yet their performance is often limited by an overlooked component: the CPU. Through a detailed study of modern large language model (LLM) serving workloads, we find that multi-GPU performance often degrades not because GPUs are saturated, but because CPUs fail to keep them busy. Under limited CPU allocations, systems exhibit symptoms such as delayed kernel launch, stalled communication, and increased tokenization latency, leading to severe GPU underutilization even when ample GPU resources are available. The problem becomes more severe in agentic LLM serving, where long accumulated contexts increase CPU-side tokenization work while high prefix-cache reuse across multi-turn interactions reduces GPU-side prefill work. These bottlenecks persist even in serving stacks that employ process-level separation and modern GPU-side optimizations such as CUDA Graphs. Since CPU cores cost orders of magnitude less than GPUs, provisioning additional cores is a highly cost-effective mitigation. Under moderate serving load, we observe that CPU-starved configurations frequently time out, while providing adequate CPU resources restores responsiveness and reduces time-to-first-token (TTFT) latency by 1.47-7.11x across configurations, all without requiring additional GPUs.

cs.AR↗

arXiv2026

A Fast Locality Simulator for GEMM Design-Space Exploration on Multi-Chiplet GPUs

In multi-chiplet GPUs, memory accesses that cross the silicon interposer to a remote chiplet's high-bandwidth memory (HBM) incur extra latency and energy, making remote-traffic reduction crucial for efficiency. For general matrix multiply (GEMM), the dominant operator in LLMs, inter-chiplet traffic depends strongly on design knobs such as per-operand memory layout, cooperative thread array (CTA) traversal order, and data placement. The optimal combination is difficult to identify analytically, as locality depends strongly on CTA traversal and its interaction with the L2 cache. To this end, we present a fast, tile-level locality simulator that models data placement and CTA-to-chiplet mapping, CTA traversal, per-chiplet L2 caches, and local/remote HBM accesses. This enables rapid evaluation of locality, performance, and energy efficiency under various GEMM configurations. Using the simulator, we find that the best locality-aware configuration for each GEMM reduces remote traffic by up to 18.3x and improves energy efficiency by up to 17% over 4 KB-interleaved data with round-robin CTA-to-chiplet mapping. Moreover, using the simulator output as feedback, an agentic AI adopts a 2D block-swizzle CTA traversal that improves mean energy efficiency by 15.2% for Qwen and 6.9% for Llama relative to the best 1D traversal under 4 KB-interleaved data placement. Overall, our simulator enables fast exploration of the GEMM locality design space on multi-chiplet GPUs and is available at https://github.com/gthparch/chiplet_locality_simulator.

cs.AR↗

arXiv2026

TileLens: Efficiently Using Large-Granularity Memory Systems with Transparent Two-Dimensional Memory Layout

Large Language Model (LLM) inference is bottlenecked by the capacity and bandwidth of GPU High-Bandwidth Memory (HBM). Recent proposals, such as High-Bandwidth Flash (HBF) and RoMe, offer higher capacity or bandwidth than HBM, but require a minimum access granularity of kilobytes. We show that these Large-Granularity Memory Systems (LGMS) can degrade the performance of tiled matrix-multiplication, which is the dominant operation in LLM inference, by up to an order of magnitude. The root cause of the slowdown is read amplification, where memory requests fetch far more data than the tile actually needs. This waste stems from a fundamental mismatch between the two-dimensional nature of compute tiles and the one-dimensional memory layout, leading to each request spilling well beyond the tile boundaries. To mitigate read amplification, we propose to use tile-major layout for LGMS. Rather than storing data as a one-dimensional strip, tile-major layout reshapes each contiguous memory block into a two-dimensional rectangle, aligning memory granularity with tile boundaries. To ease the adoption of tile-major layout on GPUs, we propose TileLens, lightweight software and hardware extensions that collectively cover major classes of GPU kernels. TileLens-SW extends GPU DSLs so that DSL-based kernels can adopt tile-major in global memory by changing only the layout descriptor. TileLens-HW extends the Tensor Memory Accelerator (TMA) for transparent tile-major support in TMA-based kernels without code changes. We evaluate TileLens on a cycle-level simulator using matrix-multiplication kernels from Qwen-3 30B and Llama-3.1 70B. Combining a tile-major layout with an adaptive hardware prefetcher, TileLens achieves near-HBM performance on HBF-augmented GPUs with a 5us HBF NAND read latency, reducing the geomean slowdown from 1.61-6.49x with conventional layouts to within 1% of an HBM-only baseline.

cs.AR↗

arXiv2026

Making Locality-aware GEMM Compatible with Page-Granularity Placement on Chiplet GPUs

Multi-chiplet GPUs scale compute throughput and high-bandwidth memory (HBM) capacity, but their non-uniform memory system makes locality between chiplets and their data critical to the GPU's performance and energy efficiency. Locality-aware scheduling and data placement identify which data should reside near each chiplet. However, in general matrix multiplication (GEMM), locality-aware data placement often becomes incompatible with a fixed page-granularity data interleaving, since the optimal granularity for mapping data across chiplets varies widely across workloads. We propose Chiplet-Contiguous Layout, a global memory layout that stores chiplet-local data contiguously. Chiplet-Contiguous Layout enables locality-aware placement compatible with page-granularity placement across diverse large language model (LLM) GEMM shapes, without changes to the operating system or hardware. On representative LLM inference and training GEMMs from Qwen 3 30B and Llama 3.1 70B, Chiplet-Contiguous Layout on average reduces remote HBM traffic by 13.0x on Qwen and 20.7x on Llama over 4\,KB interleaving, and by 3.3x and 3.7x over coarse locality-aware placement.

cs.AR↗

arXiv2025

Hardware vs. Software Implementation of Warp-Level Features in Vortex RISC-V GPU

RISC-V GPUs present a promising path for supporting GPU applications. Traditionally, GPUs achieve high efficiency through the SPMD (Single Program Multiple Data) programming model. However, modern GPU programming increasingly relies on warp-level features, which diverge from the conventional SPMD paradigm. In this paper, we explore how RISC-V GPUs can support these warp-level features both through hardware implementation and via software-only approaches. Our evaluation shows that a hardware implementation achieves up to 4 times geomean IPC speedup in microbenchmarks, while software-based solutions provide a viable alternative for area-constrained scenarios.

cs.AR↗

arXiv2024

Towards Performance-Aware Allocation for Accelerated Machine Learning on GPU-SSD Systems

The exponential growth of data-intensive machine learning workloads has exposed significant limitations in conventional GPU-accelerated systems, especially when processing datasets exceeding GPU DRAM capacity. We propose MQMS, an augmented in-storage GPU architecture and simulator that is aware of internal SSD states and operations, enabling intelligent scheduling and address allocation to overcome performance bottlenecks caused by CPU-mediated data access patterns. MQMS introduces dynamic address allocation to maximize internal parallelism and fine-grained address mapping to efficiently handle small I/O requests without incurring read-modify-write overheads. Through extensive evaluations on workloads ranging from large language model inference to classical machine learning algorithms, MQMS demonstrates orders-of-magnitude improvements in I/O request throughput, device response time, and simulation end time compared to existing simulators.

cs.AR↗