arXiv subjects

Jiawei Yi

Publications and source records attributed to Jiawei Yi.

5 recordsLinked to original sources

arXiv2025

LiteCache: A Query Similarity-Driven, GPU-Centric KVCache Subsystem for Efficient LLM Inference

During LLM inference, KVCache memory usage grows linearly with sequence length and batch size and often exceeds GPU capacity. Recent proposals offload KV states to host memory and reduce transfers using top-k attention. But their CPU-centric management of the on-GPU cache and CPU-GPU data movement incurs high overhead and fragments the bulk GPU execution that CUDA Graph relies on. To close this gap, we observe that adjacent queries within the same attention head exhibit strong directional similarity and retrieve highly overlapping top-k KV states. This insight enables a simple head granularity cache algorithm, QSAC, in which each head reuses its previously cached KV states whenever the current query is sufficiently similar to the prior one. QSAC further simplifies cache management primitives and cuts CPU involvement almost entirely. We develop LiteCache, a KVCache subsystem that incorporates QSAC. LiteCache introduces a GPU-centric synchronization controller and speculative sparse prefetching, enabling fully overlapped data movement and computation. These mechanisms produce a stable and predictable execution pattern that remains compatible with the bulk execution mode required by CUDA Graphs. Evaluation on two widely-used LLMs indicates that LiteCache achieves comparable accuracy to baselines, while sharply minimizing CPU overhead, fully utilizing PCIe bandwidth, thus improving decoding throughput by 10.7-224.2% on both H100 and A40 GPUs and easily supporting sequence lengths beyond 1M. We opensource LiteCache at https://anonymous.4open.science/r/LiteCache-888D.

cs.LG↗

arXiv2025

CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering

Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks. However, the substantial Key-Value (KV) cache required for long-context LLMs poses significant deployment challenges. Existing approaches either discard potentially critical information needed for future generations or offer limited efficiency gains due to high computational overhead. In this paper, we introduce CentroidKV, a simple yet effective framework for online KV cache clustering. Our approach is based on the observation that key states exhibit high similarity along the sequence dimension. To enable efficient clustering, we divide the sequence into chunks and propose Chunked Soft Matching, which employs an alternating partition strategy within each chunk and identifies clusters based on similarity. CentroidKV then merges the KV cache within each cluster into a single centroid. Additionally, we provide a theoretical analysis of the computational complexity and the optimality of the intra-chunk partitioning strategy. Extensive experiments across various models and long-context benchmarks demonstrate that CentroidKV achieves up to 75% reduction in KV cache memory usage while maintaining comparable model performance. Moreover, with minimal computational overhead, CentroidKV accelerates the decoding stage of inference by up to $1.92\times$ and increases the serving throughput by up to $4\times$.

cs.CL↗

arXiv2025

HATA: Trainable and Hardware-Efficient Hash-Aware Top-k Attention for Scalable Large Model Inference

Large Language Models (LLMs) have emerged as a pivotal research area, yet the attention module remains a critical bottleneck in LLM inference, even with techniques like KVCache to mitigate redundant computations. While various top-$k$ attention mechanisms have been proposed to accelerate LLM inference by exploiting the inherent sparsity of attention, they often struggled to strike a balance between efficiency and accuracy. In this paper, we introduce HATA (Hash-Aware Top-$k$ Attention), a novel approach that systematically integrates low-overhead learning-to-hash techniques into the Top-$k$ attention process. Different from the existing top-k attention methods which are devoted to seeking an absolute estimation of qk score, typically with a great cost, HATA maps queries and keys into binary hash codes, and acquires the relative qk score order with a quite low cost, which is sufficient for realizing top-k attention. Extensive experiments demonstrate that HATA achieves up to 7.2$\times$ speedup compared to vanilla full attention while maintaining model accuracy. In addition, HATA outperforms the state-of-the-art top-$k$ attention methods in both accuracy and efficiency across multiple mainstream LLM models and diverse tasks. HATA is open source at https://github.com/gpzlx1/HATA.

cs.LG↗

arXiv2025

Intelligent bear deterrence system based on computer vision: Reducing human-bear conflicts in remote areas

Human-bear conflicts on the Tibetan Plateau threaten both local livelihoods and the conservation of Tibetan brown bears (Ursus arctos pruinosus). To address this challenge, we developed a low-power, network-independent deterrence system that combines computer vision with Internet of Things (IoT) hardware. The system integrates a YOLOv5-MobileNet detection model deployed on a low-power edge artificial intelligence (AI) board with a solar-powered bear spray device. We compiled a data set of 1,243 wildlife images (including 795 bears with 100 infrared captures for nighttime detection, plus other common objects and animals such as mastiffs, yaks, humans, and vehicles), from which 80% were used for training and 20% for validation. Validation showed robust performance (mean average precision = 91.4%, recall = 93.6%). In 100 controlled activation tests involving simulated approaches by bears, humans, and other animals, the spray deployed within 0.2 seconds of detection with 97.2% accuracy, confirming timely and reliable responses. A 30-day field trial in Zadoi County, Qinghai Province, China, recorded 3 successful deterrence events without false activations. By using energy-efficient components and ensuring continuous and stable system operation, this solution provides a practical, sustainable, and scalable approach to mitigating human-bear conflicts, effectively enhancing human safety and bear conservation in remote areas without network or grid coverage.

cs.CV↗

arXiv2022

Two-dimensional half Chern-Weyl semimetal with multiple screw axes

Half topological states of matter and two-dimensional (2D) magnetism have gained much attention recently. In this paper, we propose a special topological semimetal phase called a 2D half Chern-Weyl semimetal (HCWS), which is a 2D magnetic Weyl semimetal bound to the half Chern insulator phase by symmetry, and the two phases can be converted to each other by manipulating the magnetization direction. We provide the symmetry conditions to realize this state in 2D systems with multiple screw axes. Tight-binding models with multiple basis and a predicted 2D material, monolayer TiTe, are shown as the concrete examples for HCWSs. The TiTe monolayer was shown to have a high ferremagnetic Curie temperature (~966 K) as well as a Coulomb correlation-enhanced spin-orbit coupling (SOC), and further demonstrates the effect of correlation-enhanced SOC on magnetocrystalline anisotropy energy and energy gap opening. Our work reveals a state with switchable and spin-resolved half body charge currents as well as half boundary charge currents, and will provide a platform for novel and high-performance topological spintronics devices.

cond-mat.mes-hall↗