Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.29820

UlaCache: Unified Logical Addressing for Sparse LLM Serving

Isaac Zhu·Alan Cao·Hugh Yin·Rocky Wang·Orion Wang·Hscos Zhang·Jack Li·Jason Zhao
View original source ↗Read source document ↗Export metadata ↓

Abstract

Long contexts increase the memory cost of the key-value (KV) cache in large language model (LLM) inference. Sparse attention reduces attention computation and memory traffic by selecting historical tokens, but the retained KV cache must remain available for later decoding steps. We present UlaCache, a KV cache management system built around a unified logical address space spanning GPU KV pools and optional CPU memory. Stable logical references let the inference engine allocate and select KV independently of physical placement. Common allocation and sparse-access interfaces support data-parallel (DP) and tensor-parallel (TP) attention. For distributed storage, local GPU buffers retain selected KV across decoding steps, so only missing entries require transfers. When CPU storage is enabled, Parallel Bucketed LRU (PB-LRU) manages a GPU pool shared across requests. It updates recency metadata and selects eviction victims in parallel. UlaCache transfers KV and then updates the address mappings. With speculative decoding enabled under matched CPU and GPU memory budgets, UlaCache achieves peak decoding throughput speedups of 1.16 $\times$ (DP) and 1.70 $\times$ (TP) on H200, and 1.73 $\times$ (DP) and 2.51 $\times$ (TP) on B300, compared with the state-of-the-art baseline.

Explore related subjects

cs.DC

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Isaac Zhu, Alan Cao, Hugh Yin, Rocky Wang, Orion Wang, Hscos Zhang, Jack Li, Jason Zhao. 2026-10-21. UlaCache: Unified Logical Addressing for Sparse LLM Serving. https://arxiv.org/abs/2610.29820

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

NanoCP: Taming Load Imbalance in Data-Expert Parallel Decoding under Variable-Length Workloads

Modern serving systems for Mixture-of-Experts (MoE) models adopt hybrid data-expert parallelism: expert parallelism (EP) shards experts across GPUs to scale capacity, while data parallelism (DP) replicates attention layers across instances to process independent requests. Existing systems bind each request's attention, MoE communication, and KV cache to a single instance. Because attention latency scales with KV cache size while MoE communication latency scales with batch size, this binding cannot balance both simultaneously, producing EP stragglers; it also fragments KV memory across instances, inflating tail latency under long contexts. While existing context parallelism (CP) mitigates these constraints, existing group-level and uniform CP designs incur prohibitive communication and attention-side overheads. We present NanoCP, which decouples MoE communication from KV cache placement and achieves dual balance through dynamic context parallelism (DCP). DCP assigns each request a context-parallel degree sized to its KV footprint: long requests distribute attention across multiple instances; short requests remain local. This dynamic parallelism effectively liquefies the KV cache across DP instances within each node, balancing both the per-instance KV cache occupancy and batch sizes without unnecessary load-balancing costs. To bridge DCP with static execution, NanoCP introduces an ahead-of-time (AOT) graph engine paired with a custom routing-based communication backend. Experimental results show that, under 99% TPOT-SLO attainment, NanoCP achieves $3.27\times$ and $2.11\times$ the maximum sustainable request rate of the best-performing vLLM baseline on the DeepSeek-V3 Issue 1% and Issue 5% workloads, respectively, while reducing P99 TPOT by $1.94\times$ and $2.01\times$ on the same two workloads, respectively.

cs.DC↗
arXiv2026

EP-NCO: Latency-Aware Service Placement using Neural Combinatorial Optimisers for Edge-Cloud Systems

Latency-sensitive Internet of Things (IoT) applications are increasingly deployed as interdependent microservices across heterogeneous edge-cloud infrastructures. Placing them under resource and network constraints is an NP-hard combinatorial optimisation problem. This study proposes EP-NCO, a graph-based neural combinatorial optimisation framework for service placement in compute continuum platforms. The objective accounts for execution latency, direct-link communication delay, and an approximate sharing of bandwidth among dependency flows that coexist in the resulting placement. Across held-out graph sizes generated by the same synthetic framework, the EP-NCO variant trained on Large instances with hard decoding achieved the lowest mean aggregate latency cost at every scale. It was 6.0%-12.5% lower than a genetic algorithm (GA), 14.4%-20.8% lower than particle swarm optimisation (PSO), and 1.2%-9.4% lower than the strongest reinforcement learning ablation. On Small instances, its mean cost was 10.5% below the best Gurobi solutions found and close to Gurobi's mean lower bound. The observed inference time of the EP-NCO variants was 0.88-1.03 s per instance, compared with 85-97 s for GA and PSO. As a result, the one-off training cost of the EP-NCO variant trained on Large instances with hard decoding is recovered after roughly 1,000 placements.

cs.DC↗
arXiv2026

Weight Placement Is Not Enough: Synchronization in Multi-Node CPU Decoding of Ternary LLMs

Ternary LLMs read only a few gigabytes per decoded token, yet on dual-socket servers with four NUMA nodes the fastest ternary runtime we found uses less than half of the memory bandwidth. Placing each weight on the node of the thread that reads it is necessary but not enough: in a decode engine with node-local weights that reuses ik_llama's kernels, threads on two generations of Xeon servers still spend 20-48% of each token waiting at synchronization points. Per-thread barrier traces show that most of the wait is for the slowest threads of each node; the barriers' own latency (microseconds each, but up to 12% of the token for the smallest model) and serial per-node work are smaller parts. On identical kernels, a Megatron-style column split beats a row-owned split without cross-node reductions in every configuration we measured, and isolating the number of barriers shows that most of its gain comes from its threads waiting less for each other and for their node leaders, and only a minority from having fewer barriers. Where decoding is DRAM-bound, the resulting engine is 1.5-2.1x faster than ik_llama with all cores and 1.5-1.9x faster than ik_llama at its fastest thread count (3.2x and 2.8x for a small model that runs from the L3 cache), at the same accuracy and with less energy per token.

cs.DC↗