arXiv subjects

Zeyu Xing

Publications and source records attributed to Zeyu Xing.

3 recordsLinked to original sources

arXiv2026

Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoning

KV caches, typically used only to speed up autoregressive decoding, encode contextual information that can be reused for downstream tasks at no extra cost. We propose treating the KV cache as a lightweight representation, eliminating the need to recompute or store full hidden states. Despite being weaker than dedicated embeddings, KV-derived representations are shown to be sufficient for two key applications: \textbf{(i) Chain-of-Embedding}, where they achieve competitive or superior performance on Llama-3.1-8B-Instruct and Qwen2-7B-Instruct; and \textbf{(ii) Fast/Slow Thinking Switching}, where they enable adaptive reasoning on Qwen3-8B and DeepSeek-R1-Distil-Qwen-14B, reducing token generation by up to $5.7\times$ with minimal accuracy loss. Our findings establish KV caches as a free, effective substrate for sampling and reasoning, opening new directions for representation reuse in LLM inference. Code: https://github.com/cmd2001/ICLR2026_KV-Embedding.

cs.CL↗

arXiv2025

KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference

KV cache quantization can improve Large Language Models (LLMs) inference throughput and latency in long contexts and large batch-size scenarios while preserving LLMs effectiveness. However, current methods have three unsolved issues: overlooking layer-wise sensitivity to KV cache quantization, high overhead of online fine-grained decision-making, and low flexibility to different LLMs and constraints. Therefore, we theoretically analyze the inherent correlation of layer-wise transformer attention patterns to KV cache quantization errors and study why key cache is generally more important than value cache for quantization error reduction. We further propose a simple yet effective framework KVTuner to adaptively search for the optimal hardware-friendly layer-wise KV quantization precision pairs for coarse-grained KV cache with multi-objective optimization and directly utilize the offline searched configurations during online inference. To reduce the computational cost of offline calibration, we utilize the intra-layer KV precision pair pruning and inter-layer clustering to reduce the search space. Experimental results show that we can achieve nearly lossless 3.25-bit mixed precision KV cache quantization for LLMs like Llama-3.1-8B-Instruct and 4.0-bit for sensitive models like Qwen2.5-7B-Instruct on mathematical reasoning tasks. The maximum inference throughput can be improved by 21.25\% compared with KIVI-KV8 quantization over various context lengths. Our code and searched configurations are available at https://github.com/cmd2001/KVTuner.

cs.LG↗

arXiv2020

Photonic integrated quantum key distribution receiver for multiple users

Integrated photonics has the advantages of miniaturization, low cost, and CMOS compatibility, and it provides a stable, highly integrated, and practical platform for quantum key distribution (QKD). While photonic integration of optical components has greatly reduced the overall cost of QKD systems, single-photon detectors (SPDs) have become the most expensive part of a practical QKD system. In order to circumvent this obstacle and make full use of SPDs, we have designed and fabricated a QKD receiver chip for multiple users. Our chip is based on a time-division multiplexing technique and makes use of a single set of SPDs to support up to four users' QKD. Our proof-of-principle chip-based QKD system is capable of producing an average secret key rate of 13.68 kbps for four users with a quantum bit error rate (QBER) as low as 0.51% over a simulated distance of 20 km in fiber. Our result clearly demonstrates the feasibility of multiplexing SPDs for setting QKD channels with different users using photonic integrated chip and may find applications in the commercialization of quantum communication technology.

quant-ph↗