Heying Zhang | Searcharxiv

arXiv2026

AnchorKV: Lossless Key-Value Cache Reconstruction from Layer-Sparse Residual Anchors

Residual-stream caching can reconstruct transformer key-value (KV) caches losslessly, but its memory advantage depends on architecture. We present AnchorKV, a storage-replay spectrum connecting per-layer residual caching and full recomputation through configurable residual anchors. An architecture-dependent break-even rule identifies the first storage-saving spacing, while deployment selects an operating point from the spectrum and cascade replay reduces reconstruction from O(Ls) to O(L), with L the model depth and s the anchor spacing. For interleaved sliding-window architectures, exact dynamic programming places anchors under non-uniform reconstruction demand. Across CPU, CUDA, and Ascend NPU backends, six architecture families, and fp32/bf16 arithmetic, evaluations verify bit-exact KV reconstruction and unchanged continuation logits and tokens. At 4k context, cascade reconstruction is up to 16.3 times faster than naive replay; across eight applicable global-attention checkpoints at 32k context, deepest spacing yields 5.78-54.0 times full-KV reactivatable session capacity. Compared with seven external cache methods on a common model, AnchorKV preserves full-cache retrieval and question-answering scores, whereas lossy methods change output fidelity and task scores. The break-even spacing is computable from a model's configuration file alone.

cs.AR↗

arXiv2026

The Residual Stream Is All You Need: On the Redundancy of the KV Cache in Transformer Inference

The key-value (KV) cache is widely treated as essential state in transformer inference, and a large body of work engineers policies to compress, evict, or approximate its entries. We prove that this state is entirely redundant: keys and values at every layer are deterministic projections of the residual stream, and recomputing them from a single residual vector per token incurs exactly zero reconstruction error, not approximately, but bit-identically. We verify this across six models from four architecture families (135M to 4B parameters). Cross-task residual patching at every layer produces D_KL = 0 between patched and original output distributions, confirming that the residual stream satisfies a Markov property and is the sole information-carrying state. Removing the cache entirely and recomputing from scratch yields token-identical output under greedy decoding on all models tested. We build on this result with KV-Direct, a bounded-memory inference scheme that checkpoints residual vectors (5 KB per token on Gemma 3-4B) instead of full KV pairs (136 KB), recomputing keys and values on demand. Over 20 conversation turns, KV-Direct holds peak memory at 42 MB while the standard cache grows past 103 MB. Against five eviction baselines (H2O, StreamingLLM, SnapKV, TOVA, window-only), KV-Direct maintains 100% token match at every cache budget; all baselines degrade to 5-28%. A per-operation latency analysis shows recomputation runs up to 5x faster than reading cached tensors at moderate batch sizes. Code is available at https://github.com/Kaleemullahqasim/KV-Direct.

cs.LG↗