Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2609.37062

vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains

Wei Da·Yavuz Ferhatosmanoglu·Evangelia Kalyvianaki
View original source ↗Read source document ↗Export metadata ↓

Abstract

Dynamic layer skipping reduces LLM computation by allowing each token to execute only a subset of the model's layers. However, existing skippers rely on specialized generation loops and do not integrate with modern serving engines. As a result, fewer executed layers do not necessarily translate into lower serving latency: FlexiDepth skips 8 of Llama-3-8B's 32 layers on average, yet its standard generation loop decodes 14.6--21.0% more slowly than the base model. We present vSkipper, a virtualization layer that makes dynamic layer skippers pluggable in serving engines while preserving continuous batching, fixed-shape batches, paged KV caching, and captured decode graphs. At each routed layer, vSkipper groups tokens by the skipper's decision and uses routed execution only when predicted to be profitable. We implement vSkipper in SGLang and evaluate the released FlexiDepth checkpoint against upstream SGLang under identical prompts, arrivals, output lengths, and launch settings. At the knee of upstream's load curve, vSkipper reduces mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH. Under saturation, it increases request throughput by 11.3% and 7.4%. Serving adds no statistically resolved quality loss beyond the checkpoint's own. Across synthetic skip policies, two Qwen3 skippers, and three GPUs, we demonstrate reuse without workload-specific tuning. To our knowledge, vSkipper is the first system to realize serving-efficiency gains from per-token interior layer skipping within a modern LLM serving engine. The code is open-sourced as an SGLang fork at https://github.com/AKafakA/sglang-vskipper/tree/vskipper-ref

Explore related subjects

cs.DCcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Wei Da, Yavuz Ferhatosmanoglu, Evangelia Kalyvianaki. 2026-09-29. vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains. https://arxiv.org/abs/2609.37062

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

ZenFlow: Enabling Stall-Free Offloading for LLM Training

Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle, waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients. ZenFlow performs in-place updates of important gradients on GPU while offloading and accumulating less important ones on CPU without blocking training iterations, fully overlapping CPU work with GPU computation. To make this importance-aware design scalable, ZenFlow leverages a novel spatio-temporal locality of important gradients, concentrated in specific positions that remain stable across iterations, to make important gradient identification efficient and lightweight while preserving fidelity. Evaluation shows that ZenFlow achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy.

cs.DC↗
arXiv2026

Asynchronous Indulgent Gathering of Opaque Robots

We consider the benchmarking problem of gathering in an $(N,f)$-fault system consisting of $N$ robots, of which at most $f$ may fail at any execution. The {\em traditional} version asks $N-f$ non-faulty robots to gather at a point not known a priori. The recently studied {\em indulgent} version asks $N-f$ non-faulty robots to gather at the location of a faulty one. Indulgent gathering is impossible in a $(2,1)$-crash system in the $LUMI$ model under semi-synchrony, even when infinitely many colors and coordinate axis agreement are available. In this paper, we establish four results for indulgent gathering under asynchrony in $LUMI$, considering a {\em mobility fault} model in which a robot crash only impacts its mobility but not the operation of the light. We first show that indulgent gathering is impossible in a $(2,1)$-mobility system using 2-colored lights, and then give an optimal solution w.r.t. the number of colors using 3-colored lights. We then give two deterministic algorithms for an $(N,f)$-mobility system, $f<N$, $N,f$ unknown, that exhibit a nice time-color trade-off: One with time $O(N)$ using 7-colored lights and another with time $O(\max\{\ell,f\})$ using 26-colored lights, where $\ell< N$ is the number of distinct convex layers of robot positions in the initial configuration. Our algorithms for an $(N,f)$-mobility system are the first to be analysed time complexity, can withstand obstructed visibility (opaque robot model) and asynchronous scheduling. Interestingly, our algorithms also solve traditional gathering.

cs.DC↗
arXiv2026

TeraAgent: A Distributed Agent-Based Simulation Engine for Simulating Half a Trillion Agents

Agent-based simulation is a method for studying highly complex systems in disciplines as diverse as biology, medicine, sociology, economics, finance, and more. In this method, an agent represents a small part of the system, such as a human cell, and, through interactions among many agents, the system's behavior emerges, like the growth of cancer in the human body. Real-world complex systems, such as a human organ, comprise billions of agents. However, the state-of-the-art agent-based simulation platform, BioDynaMo, does not support the distributed processing of a simulation across servers. This limitation restricts the scalability of agent counts to the computational resources available within a single server, hindering users from conducting larger and more meaningful simulations. In this paper, we introduce TeraAgent, a new distributed agent-based simulation engine. To overcome BioDynaMo's scalability limitation, TeraAgent enables efficient exchange of agent information across servers by implementing two solutions: 1) a tailored serialization mechanism that allows agents to be accessed and mutated directly at the receive buffer, and 2) delta encoding that significantly reduces data transfer across servers by leveraging the iterative nature of agent-based simulations. Built on our solutions, TeraAgent 1) enables extreme-scale simulations with half a trillion agents (292x that of BioDynaMo) on 84,096 CPU cores, 2) reduces time-to-result leveraging parallelism across compute nodes, 3) improves interoperability with third-party tools, and 4) provides users with more hardware flexibility. TeraAgent serializes agents with a median speedup of 110x (37x for deserialization) over the state-of-the-art ROOT I/O baseline, delivers 8x the agent update rate per CPU core of Biocellion, and improves in-situ visualization performance by 39x.

cs.DC↗