arXiv subjects

Chenpeng Wu

Publications and source records attributed to Chenpeng Wu.

4 recordsLinked to original sources

arXiv2026

HIERA: Workload-Aware Planning Across Implementation Spaces for GPU Kernel Optimization

High-performance GPU kernels underpin modern deep learning and scientific computing. As workloads become increasingly diverse and GPU hardware evolves rapidly, developing efficient methods for automated GPU kernel generation and optimization has become increasingly important. Existing LLM-based methods typically optimize within a fixed implementation space, limiting either optimization flexibility or search efficiency. We propose \textsc{HIERA}, a hierarchical search-space planning framework for GPU kernel optimization. \textsc{HIERA} constructs contract-augmented task specifications, selects an appropriate implementation space across PyTorch operators, CUDA libraries, and custom CUDA kernels, and uses profiling feedback and expert knowledge to guide structured iterative refinement. Experiments on KernelBench across multiple various workload levels and base LLMs show that \textsc{HIERA} delivers stronger overall implementation validity, sample efficiency, and optimization performance than existing training-free methods, while remaining competitive with the training-based CUDA-L1 without additional model training. A case study on a specialized stencil operator from scientific computing further achieves a $1.53\times$ speedup over cuDNN, demonstrating the potentiality of the general framework beyond standard machine-learning workloads.

cs.DC↗

arXiv2026

Do We Need Tensor Cores for Stencil Computations?

Stencil computation constitutes a cornerstone of scientific computing, serving as a critical kernel in domains ranging from fluid dynamics to weather simulation. While stencil computations are conventionally regarded as memory-bound and thus unsuitable for compute-centric Tensor Cores, recent empirical studies have demonstrated significant speedups after applying Tensor Cores, forming an apparent contradiction. This paper resolves this contradiction by conducting a systematic performance analysis of stencil computations on Tensor Cores. We begin by revisiting the adaptation of stencils onto Tensor Cores, quantifying the computational redundancy introduced by the transformations required to satisfy hardware constraints. These metrics are subsequently integrated into an enhanced performance model that explicitly accounts for the arithmetic intensity shifts driven by temporal fusion. Guided by this formulation, we derive analytical criteria to determine the suitability of Tensor Cores for varying stencil workloads. By classifying operational regions, we identify the specific \textit{sweet spot} for effective acceleration and further demonstrate how Sparse Tensor Cores expand this profitable design space. Extensive evaluations on NVIDIA GPUs across SOTA implementations, including DRStencil, EBISU, ConvStencil, and SPIDER, validate our performance model and analytical criteria. These results demonstrate the effectiveness of our approach in guiding stencil performance optimization.

cs.DC↗

arXiv2025

SPIDER: Unleashing Sparse Tensor Cores for Stencil Computation via Strided Swapping

Recent research has focused on accelerating stencil computations by exploiting emerging hardware like Tensor Cores. To leverage these accelerators, the stencil operation must be transformed to matrix multiplications. However, this transformation introduces undesired sparsity into the kernel matrix, leading to significant redundant computation. In this paper, we present SPIDER, the first system to turn this unresolved sparsity into an optimization opportunity by exploring the potential of Sparse Tensor Cores (SpTCs) for stencil acceleration. Specifically, SPIDER introduces an efficient and elegant transformation method that integrates two cooperative techniques: an ahead-of-time strided swapping transformation for kernel matrices and an on-the-fly row-swapping mechanism for inputs. This rule-based approach effectively transforms stencil computation into operations compatible with SpTCs, introducing only slight compile-time overhead and zero runtime overhead. Additionally, SPIDER incorporates multiple optimizations to maximize computational efficiency. Experimental evaluations demonstrate that SPIDER outperforms vendor library cuDNN by 6.20$\times$ and state-of-the-art (SOTA) Tensor Core-based approaches (ConvStencil, FlashFFTStencil, etc.) by 2.00$\times$ on average.

cs.DC↗

arXiv2025

Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Cores

The escalating size of Mixture-of-Experts (MoE) based Large Language Models (LLMs) presents significant computational and memory challenges, necessitating innovative solutions to enhance efficiency without compromising model accuracy. Structured sparsity emerges as a compelling strategy to address these challenges by leveraging the emerging sparse computing hardware. Prior works mainly focus on the sparsity in model parameters, neglecting the inherent sparse patterns in activations. This oversight can lead to additional computational costs associated with activations, potentially resulting in suboptimal performance. This paper presents Samoyeds, an innovative acceleration system for MoE LLMs utilizing Sparse Tensor Cores (SpTCs). Samoyeds is the first to apply sparsity simultaneously to both activations and model parameters. It introduces a bespoke sparse data format tailored for MoE computation and develops a specialized sparse-sparse matrix multiplication kernel. Furthermore, Samoyeds incorporates systematic optimizations specifically designed for the execution of dual-side structured sparse MoE LLMs on SpTCs, further enhancing system performance. Evaluations show that Samoyeds outperforms SOTA works by up to 1.99$\times$ at the kernel level and 1.58$\times$ at the model level. Moreover, it enhances memory efficiency, increasing maximum supported batch sizes by 4.41$\times$ on average. Additionally, Samoyeds surpasses existing SOTA structured sparse solutions in both model accuracy and hardware portability.

cs.LG↗