arXiv subjects

An Zou

Publications and source records attributed to An Zou.

At least 19 recordsLinked to original sources

arXiv2026

CUDA-Harness: Harnessing Agentic CUDA Kernel Generation and Optimization from Natural Language

Developing high-performance CUDA kernels demands specialized knowledge in algorithm implementation, correctness validation, and hardware-aware parallel optimization, creating a substantial expertise barrier and making generating CUDA kernels directly from natural language (Text2CUDA) essential. Meanwhile, the general-purpose code generation capability of Large Language Models (LLMs) prompts a series of works exploring LLM-based CUDA kernel generation. They mainly focus on transpilation from high-level frameworks such as PyTorch to CUDA (Torch2CUDA) rather than Text2CUDA, where models must understand the high-level input semantics and handle low-level kernel implementation and validation. Additionally, these methods are vulnerable to reward hacking due to reliance on predefined test inputs. In this paper, we propose CUDA-Harness, a framework for harnessing agentic CUDA kernel generation and optimization from natural language. Specifically, we introduce Intermediate-Structured Generation to connect high-level semantic understanding with low-level kernel generation. To dilute reward hacking in Text2CUDA, we construct Synthesis-Based Verification to provide isolated test data and progressive validation. Furthermore, we propose Feedback-Adaptive Evolution, a kernel evolution strategy that prioritizes correctness while optimizing performance. Finally, through extensive experiments, we demonstrate the effectiveness of CUDA-Harness, with further evaluations illustrating generalization across LLMs, hardware platforms, and to C-to-CUDA transpilation.

cs.CL↗

arXiv2026

From Tokens to Regions: CUDA-Sensitive Instruction Tuning for GPU Kernel Generation

High-performance CUDA kernels are essential for scalable AI systems, while Large Language Models (LLMs) still struggle to generate correct kernels due to strict and implicit execution constraints. Existing LLM-based approaches either rely on costly agentic or reinforcement-learning (RL) pipelines, or adopt supervised fine-tuning (SFT) objectives that fail to explicitly model CUDA sensitivity, namely code tokens or regions tightly coupled with execution constraints. In this work, we investigate CUDA sensitivity from the perspective of token confidence patterns, showing that CUDA sensitivity appears at both token and region levels, where most CUDA-sensitive tokens are predicted with high confidence, while a smaller low-confidence subset forms regions corresponding to execution-critical structures. These findings suggest that effective CUDA kernel generation should both leverage high-confidence CUDA-sensitive tokens and preserve low-confidence CUDA-sensitive regions. Building on these insights, we propose \textbf{\underline{CU}DA-\underline{Se}nsitive Instruction \underline{T}uning (CuSeT)}, a low-cost post-training method within a simple SFT framework. CuSeT follows the principle of ``from tokens to regions'' by combining \emph{adaptive token-level masking} with \emph{region-aware sample reweighting}. Experiments show that CuSeT consistently improves functional correctness across multiple model families and scales, outperforming standard SFT and advanced SFT variants, while achieving competitive performance against frontier CUDA kernel generation models with substantially lower inference cost.

cs.LG↗

arXiv2026

River-LLM: Large Language Model Seamless Exit Based on KV Share

Large Language Models (LLMs) have demonstrated exceptional performance across diverse domains but are increasingly constrained by high inference latency. Early Exit has emerged as a promising solution to accelerate inference by dynamically bypassing redundant layers. However, in decoder-only architectures, the efficiency of Early Exit is severely bottlenecked by the KV Cache Absence problem, where skipped layers fail to provide the necessary historical states for subsequent tokens. Existing solutions, such as recomputation or masking, either introduce significant latency overhead or incur severe precision loss, failing to bridge the gap between theoretical layer reduction and practical wall-clock speedup. In this paper, we propose River-LLM, a training-free framework that enables seamless token-level Early Exit. River-LLM introduces a lightweight KV-Shared Exit River that allows the backbone's missing KV cache to be naturally generated and preserved during the exit process, eliminating the need for costly recovery operations. Furthermore, we utilize state transition similarity within decoder blocks to predict cumulative KV errors and guide precise exit decisions. Extensive experiments on mathematical reasoning and code generation tasks demonstrate that River-LLM achieves 1.53 to 2.16 times of practical speedup while maintaining high generation quality.

cs.CL↗

arXiv2026

CUDABench: Benchmarking LLMs for Text-to-CUDA Generation

Recent studies have demonstrated the potential of Large Language Models (LLMs) in generating GPU Kernels. Current benchmarks focus on the translation of high-level languages into CUDA, overlooking the more general and challenging task of text-to-CUDA generation. Furthermore, given the hardware-specific and performance-critical features of GPU programming, accurately assessing the performance of LLM-generated GPU programs is nontrivial. In this work, we introduce CUDABench, a comprehensive benchmark designed to evaluate the text-to-CUDA capabilities of LLMs. First, we construct CUDABench-Set, which covers Breadth-Depth-Difficulty evaluation space in diverse application domains, including artificial intelligence, scientific computing, and data analytics, etc. Furthermore, we propose CUDABench-Score and Generative Verification Pipeline that assess (1) compilation correctness, (2) functional consistency through execution-based verification, and (3) a novel roofline-based metric, Performance-Score. Benchmarking state-of-the-art LLMs reveals insightful findings and challenges of text-to-CUDA, such as a notable mismatch between high compilation success rates and low functional correctness, a lack of domain-specific algorithmic knowledge, and suboptimal utilization of GPU hardware resources. Our benchmark is available at https://github.com/CUDA-Bench/CUDABench.

cs.LG↗

arXiv2025

TimeBill: Time-Budgeted Inference for Large Language Models

Large Language Models (LLMs) are increasingly deployed in time-critical systems, such as robotics, autonomous driving, embodied intelligence, and industrial automation, where generating accurate responses within a given time budget is crucial for decision-making, control, or safety-critical tasks. However, the auto-regressive generation process of LLMs makes it challenging to model and estimate the end-to-end execution time. Furthermore, existing efficient inference methods based on a fixed key-value (KV) cache eviction ratio struggle to adapt to varying tasks with diverse time budgets, where an improper eviction ratio may lead to incomplete inference or a drop in response performance. In this paper, we propose TimeBill, a novel time-budgeted inference framework for LLMs that balances the inference efficiency and response performance. To be more specific, we propose a fine-grained response length predictor (RLP) and an execution time estimator (ETE) to accurately predict the end-to-end execution time of LLMs. Following this, we develop a time-budgeted efficient inference approach that adaptively adjusts the KV cache eviction ratio based on execution time prediction and the given time budget. Finally, through extensive experiments, we demonstrate the advantages of TimeBill in improving task completion rate and maintaining response performance under various overrun strategies.

cs.CL↗

arXiv2025

CUDA-LLM: LLMs Can Write Efficient CUDA Kernels

Large Language Models (LLMs) have demonstrated strong capabilities in general-purpose code generation. However, generating the code which is deeply hardware-specific, architecture-aware, and performance-critical, especially for massively parallel GPUs, remains a complex challenge. In this work, we explore the use of LLMs for the automated generation and optimization of CUDA programs, with the goal of producing high-performance GPU kernels that fully exploit the underlying hardware. To address this challenge, we propose a novel framework called \textbf{Feature Search and Reinforcement (FSR)}. FSR jointly optimizes compilation and functional correctness, as well as the runtime performance, which are validated through extensive and diverse test cases, and measured by actual kernel execution latency on the target GPU, respectively. This approach enables LLMs not only to generate syntactically and semantically correct CUDA code but also to iteratively refine it for efficiency, tailored to the characteristics of the GPU architecture. We evaluate FSR on representative CUDA kernels, covering AI workloads and computational intensive algorithms. Our results show that LLMs augmented with FSR consistently guarantee correctness rates. Meanwhile, the automatically generated kernels can outperform general human-written code by a factor of up to 179$\times$ in execution speeds. These findings highlight the potential of combining LLMs with performance reinforcement to automate GPU programming for hardware-specific, architecture-sensitive, and performance-critical applications.

cs.LG↗

arXiv2025

A Survey of Real-time Scheduling on Accelerator-based Heterogeneous Architecture for Time Critical Applications

Accelerator-based heterogeneous architectures, such as CPU-GPU, CPU-TPU, and CPU-FPGA systems, are widely adopted to support the popular artificial intelligence (AI) algorithms that demand intensive computation. When deployed in real-time applications, such as robotics and autonomous vehicles, these architectures must meet stringent timing constraints. To summarize these achievements, this article presents a comprehensive survey of real-time scheduling techniques for accelerator-based heterogeneous platforms. It highlights key advancements from the past ten years, showcasing how proposed solutions have evolved to address the distinct challenges and requirements of these systems. This survey begins with an overview of the hardware characteristics and common task execution models used in accelerator-based heterogeneous systems. It then categorizes the reviewed works based on soft and hard deadline constraints. For soft real-time approaches, we cover real-time scheduling methods supported by hardware vendors and strategies focusing on timing-critical scheduling, energy efficiency, and thermal-aware scheduling. For hard real-time approaches, we first examine support from processor vendors. We then discuss scheduling techniques that guarantee hard deadlines (with strict response time analysis). After reviewing general soft and hard real-time scheduling methods, we explore application- or scenario-driven real-time scheduling techniques for accelerator-enabled heterogeneous computing platforms. Finally, the article concludes with a discussion of open issues and challenges within this research area.

cs.DC↗

arXiv2024

Path-Consistency with Prefix Enhancement for Efficient Inference in LLMs

To enhance the reasoning capabilities of large language models (LLMs), self-consistency has become a popular approach, combining multiple samplings with majority voting. However, current methods are computationally expensive and time-consuming due to the need for numerous samplings. To address this, this paper introduces path-consistency, which leverages the confidence of earlier-generated answers to identify the most promising prefix and guide the generation of subsequent branches. By dynamically guiding the generation of subsequent branches based on this prefix, path-consistency mitigates both the errors and redundancies from random or less useful sampling in self-consistency. This approach reduces errors and redundancies from random sampling, significantly accelerating inference by minimizing token consumption. Our extensive empirical results demonstrate that path-consistency improves inference latency by up to 40.5\%, while maintaining task accuracy across various tasks, including mathematical reasoning, commonsense reasoning, and symbolic reasoning.

cs.CL↗

arXiv2024

Order of Compression: A Systematic and Optimal Sequence to Combinationally Compress CNN

Model compression has gained significant popularity as a means to alleviate the computational and memory demands of machine learning models. Each compression technique leverages unique features to reduce the size of neural networks. Although intuitively combining different techniques may enhance compression effectiveness, we find that the order in which they are combined significantly influences performance. To identify the optimal sequence for compressing neural networks, we propose the Order of Compression, a systematic and optimal sequence to apply multiple compression techniques in the most effective order. We start by building the foundations of the orders between any two compression approaches and then demonstrate inserting additional compression between any two compressions will not break the order of the two compression approaches. Based on the foundations, an optimal order is obtained with topological sorting. Validated on image-based regression and classification networks across different datasets, our proposed Order of Compression significantly reduces computational costs by up to 859 times on ResNet34, with negligible accuracy loss (-0.09% for CIFAR10) compared to the baseline model. We believe our simple yet effective exploration of the order of compression will shed light on the practice of model compression.

cs.LG↗

arXiv2024

ONE-SA: Enabling Nonlinear Operations in Systolic Arrays for Efficient and Flexible Neural Network Inference

The computation and memory-intensive nature of DNNs limits their use in many mobile and embedded contexts. Application-specific integrated circuit (ASIC) hardware accelerators employ matrix multiplication units (such as the systolic arrays) and dedicated nonlinear function units to speed up DNN computations. A close examination of these ASIC accelerators reveals that the designs are often specialized and lack versatility across different networks, especially when the networks have different types of computation. In this paper, we introduce a novel systolic array architecture, which is capable of executing nonlinear functions. By encompassing both inherent linear and newly enabled nonlinear functions within the systolic arrays, the proposed architecture facilitates versatile network inferences, substantially enhancing computational power and energy efficiency. Experimental results show that employing this systolic array enables seamless execution of entire DNNs, incurring only a negligible loss in the network inference accuracy. Furthermore, assessment and evaluation with FPGAs reveal that integrating nonlinear computation capacity into a systolic array does not introduce extra notable (less than 1.5%) block memory memories (BRAMs), look-up-tables (LUTs), or digital signal processors (DSPs) but a mere 13.3% - 24.1% more flip flops (FFs). In comparison to existing methodologies, executing the networks with the proposed systolic array, which enables the flexibility of different network models, yields up to 25.73x, 5.21x, and 1.54x computational efficiency when compared to general-purpose CPUs, GPUs, and SoCs respectively, while achieving comparable (83.4% - 135.8%) performance with the conventional accelerators which are designed for specific neural network models.

cs.AR↗

arXiv2023

Mechanical transistors for logic-with-memory computing

As a potential revolutionary topic in future information processing, mechanical computing has gained tremendous attention for replacing or supplementing conventional electronics vulnerable to power outages, security attacks, and harsh environments. Despite its potential for constructing intelligent matter towards nonclassical computing systems beyond the von Neumann architecture, most works on mechanical computing demonstrated that the ad hoc design of simple logic gates cannot fully realize a universal mechanical processing framework involving interconnected arithmetic logic components and memory. However, such a logic-with-memory computing architecture is critical for complex and persistent state-dependent computations such as sequential logic. Here we propose a mechanical transistor (M-Transistor), abstracting omnipresent temperatures as the input-output mechanical bits, which consists of a metamaterial thermal channel as the gate terminal driving a nonlinear bistable soft actuator to selectively connect the output terminal to two other variable thermal sources. This M-Transistor is an elementary unit to modularly form various combinational and sequential circuits, such as complex logic gates, registers (volatile memory), and long-term memories (non-volatile memory) with much fewer units than the electronic counterparts. Moreover, they can establish a universal processing core comprising an arithmetic circuit and a register in a compact, reprogrammable network involving periodic read, write, memory, and logic operations of the mechanical bits. Our work contributes to realizing a non-electric universal mechanical computing architecture that combines multidisciplinary engineering with structural mechanics, materials science, thermal engineering, physical intelligence, and computational science.

physics.app-ph↗

arXiv2023

NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference

The inherent diversity of computation types within the deep neural network (DNN) models often requires a variety of specialized units in hardware processors, which limits computational efficiency, increasing both inference latency and power consumption, especially when the hardware processor needs to support and execute different neural networks. In this study, we introduce NeuralMatrix, which elastically transforms the computations of entire DNNs into linear matrix operations. This transformation allows seamless execution of various DNN models all with matrix operations and paves the way for running versatile DNN models with a single General Matrix Multiplication (GEMM) accelerator.Extensive experiments with both CNN and transformer-based models demonstrate the potential of NeuralMatrix to accurately and efficiently execute a wide range of DNN models, achieving 2.17-38.72 times computation efficiency (i.e., throughput per power) compared to CPUs, GPUs, and SoC platforms. This level of efficiency is usually only attainable with the accelerator designed for a specific neural network.

cs.LG↗

arXiv2022

Predictive Exit: Prediction of Fine-Grained Early Exits for Computation- and Energy-Efficient Inference

By adding exiting layers to the deep learning networks, early exit can terminate the inference earlier with accurate results. The passive decision-making of whether to exit or continue the next layer has to go through every pre-placed exiting layer until it exits. In addition, it is also hard to adjust the configurations of the computing platforms alongside the inference proceeds. By incorporating a low-cost prediction engine, we propose a Predictive Exit framework for computation- and energy-efficient deep learning applications. Predictive Exit can forecast where the network will exit (i.e., establish the number of remaining layers to finish the inference), which effectively reduces the network computation cost by exiting on time without running every pre-placed exiting layer. Moreover, according to the number of remaining layers, proper computing configurations (i.e., frequency and voltage) are selected to execute the network to further save energy. Extensive experimental results demonstrate that Predictive Exit achieves up to 96.2% computation reduction and 72.9% energy-saving compared with classic deep learning networks; and 12.8% computation reduction and 37.6% energy-saving compared with the early exit under state-of-the-art exiting strategies, given the same inference accuracy and latency.

cs.LG↗

arXiv2021

Disjoining Pressure Driven Transpiration of Water in a Simulated Tree

We present an investigation of transpiration of water in a 100 m tall tree using continuum simulations. Disjoining pressure is found to induce absolute negative pressures as high as -23.5 atm at the liquid-vapor meniscus during evaporation, thus presenting a sufficient stand-alone explanation of the transpiration mechanism. In this work, we begin by first developing an expression of disjoining pressure in a water film as a function of distance from the surface from prior experimental findings. The expression is then implemented in a commercial computational fluid dynamics solver and the disjoining pressure effect on water wicking in nanochannels of height varying from 59 nm to 1 micron is simulated. The simulation results are in excellent agreement with experimental data, thus demonstrating and validating that near-surface molecular interactions can be integrated in continuum numerical simulations through the disjoining pressure term. Following the implementation, we simulate the transpiration process of passive water transport over a height of 100 m by using a domain comprising of nanopore connected to a tube with a ground-based water tank, thus mimicking the stomata-xylem-soil pathway in trees. By varying the evaporation rate from liquid-vapor interface in the nanopore, effects on naturally-created pressure difference and liquid flow velocity are estimated. Further, kinetic theory analysis is performed to study the combined effect of absolute negative liquid-film pressure and accommodation coefficient on the maximum mass flux feasible during transpiration. Continuum simulations coupled with kinetic theory reiterate the existence of an upper limit to height of trees. The numerical model developed here is adept to be employed to design and advance several other nanofluidics-based natural and engineering systems.

physics.flu-dyn↗

arXiv2021

Implementation of Near-Surface Disjoining Pressure Effect in Continuum Simulations

An expression of disjoining pressure in a water film, as a function of distance from the surface, is developed from prior experimental findings. The expression is implemented in commercial computational fluid dynamics solver and disjoining pressure effect on water wicking in nanochannels of height varying from 59 nm to 1 micron is simulated. The simulation results are in excellent agreement with experimental data, thus demonstrating and validating that near-surface molecular interactions can be integrated in continuum numerical simulations through the disjoining pressure term. Such an implementation can be used to design and advance nanofluidics-based engineering systems.

physics.flu-dyn↗

arXiv2021

Thermal Management of Photovoltaics using Porous Nanochannels

The photoelectric conversion efficiency of a solar cell is dependent on its temperature. When the solar radiation is incident on the photovoltaics (PV) panel, a large portion of it is absorbed by the underlying material which increases its internal energy leading to the generation of heat. An overheated PV panel results in a decline in its performance which calls for an efficient cooling mechanism that can offer an optimum output of the electrical power. In the present numerical work, thermal management with a porous nanochannels device capable to dissipate high heat flux is employed to regulate the temperature of a commercial PV panel by integrating the device on the back face of the panel. The spatial and temporal variation of the PV surface temperature is obtained by solving the energy balance equation numerically. By evaluating the steady-state PV surface temperature with and without thermal management, the extent of cooling and the resulting enhancement in the electrical power output is studied in detail. The nanochannels device is found to reduce the PV surface temperature significantly with an average cooling of 31.5 oC. Additionally, the enhancement in the electrical power output by ~33% and the reduction in the response time to 1/8th highlight the potential of using porous nanochannels as a thermal management device. Furthermore, the numerical method is used to develop a universal curve which can predict the extent of PV cooling for any generic thermal management device.

physics.app-ph↗

arXiv2021

RTGPU: Real-Time GPU Scheduling of Hard Deadline Parallel Tasks with Fine-Grain Utilization

Many emerging cyber-physical systems, such as autonomous vehicles and robots, rely heavily on artificial intelligence and machine learning algorithms to perform important system operations. Since these highly parallel applications are computationally intensive, they need to be accelerated by graphics processing units (GPUs) to meet stringent timing constraints. However, despite the wide adoption of GPUs, efficiently scheduling multiple GPU applications while providing rigorous real-time guarantees remains a challenge. In this paper, we propose RTGPU, which can schedule the execution of multiple GPU applications in real-time to meet hard deadlines. Each GPU application can have multiple CPU execution and memory copy segments, as well as GPU kernels. We start with a model to explicitly account for the CPU and memory copy segments of these applications. We then consider the GPU architecture in the development of a precise timing model for the GPU kernels and leverage a technique known as persistent threads to implement fine-grained kernel scheduling with improved performance through interleaved execution. Next, we propose a general method for scheduling parallel GPU applications in real time. Finally, to schedule multiple parallel GPU applications, we propose a practical real-time scheduling algorithm based on federated scheduling and grid search (for GPU kernel segments) with uniprocessor fixed priority scheduling (for multiple CPU and memory copy segments). Our approach provides superior schedulability compared with previous work, and gives real-time guarantees to meet hard deadlines for multiple GPU applications according to comprehensive validation and evaluation on a real NVIDIA GTX1080Ti GPU system.

cs.DC↗

arXiv2020

Nano-Confinement Effects on Liquid Pressure

In this work, molecular dynamics simulations are performed to estimate the equilibrium pressure of liquid confined in nanopores. The simulations show that the pressure is highly sensitive to the pore size and can significantly change from absolute positive to negative values for a very small (0.1 nm) change in pore size. The contribution from the solid-liquid interaction always dominates the pressure in the first liquid layer adjacent to the surface and the sensitiveness of pressure on the pore size is due to the atom distribution in the liquid layers. A surface influence number S is introduced to quantitatively characterize the degree of the confinement. The S number decreases with increasing pore size based on a power law function at constant system temperature. In nanopores with large S number, the pore liquid pressure is found to be independent of bulk liquid pressure while the pore pressure increases with bulk pressure in nanopores with small S number.

cond-mat.soft↗