Searcharxiv⌕ Search

arXiv · 2611.14848

Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices

Abstract

Large language model (LLM) agents are rapidly becoming a dominant paradigm for AI applications. Recently, there is a growing demand to deploy large-scale Mixture-of-Experts (MoE) models locally on consumer PCs to balance data privacy, API costs, and inference efficiency. However, unlike datacenter servers with abundant DRAM, consumer devices are tightly constrained in both GPU VRAM and host RAM, forcing massive MoE weights to spill across VRAM, host RAM, and SSD. Under such spilling, we observe that existing systems fail to fully utilize this memory hierarchy, treating the tiers as independent caches and thrashing in inference. We present TIDEWAY, a system for efficient inference of large MoE models on memory-constrained consumer devices. Our core insight is to manage host RAM and GPU VRAM as a unified memory pool. Within this pool, TIDEWAY couples phase-aware memory layouts, which adapt to the distinct access patterns of prefill and decode, with an execution engine that optimizes CUDA Graph execution under expert paging. Our evaluations on representative consumer hardware with frontier MoE models demonstrate that TIDEWAY significantly outperforms state-of-the-art systems. On average, TIDEWAY achieves speedups of 1.97x in end-to-end latency and 2.40x in Time-to-First-Token (TTFT), along with 1.61x higher decode throughput against the fastest baseline. Our source code is publicly available at https://github.com/Afterglow231/Tideway .

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yifei Wang, Tsun-yu Yang, Yechen Xu, Hancheng Ye, Qinsi Wang, Baijun Zhou, Tingjun Chen, Hai "Helen" Li, Danyang Zhuo, Yiran Chen. 2026-11-10. Tideway: Phase-Aware Tiered Memory Management for MoE Inference on Consumer Devices. https://searcharxiv.com/abs/2611.14848

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

HyCoInfer: Cloud-Edge Collaborative Inference for Large Language Models on Heterogeneous GPU-FPGA Architectures

Large language model (LLM) serving relies predominantly on cloud GPUs, yet autoregressive decoding is memory-bound and often leaves their arithmetic units underutilized. Edge FPGAs offer attractive decoding efficiency, but are ill suited to sustaining a high-concurrency serving workload alone. This paper presents HyCoInfer, a heterogeneous cloud-edge serving system that assigns prefill to cloud GPUs and distributes decoding across cloud GPUs and edge FPGAs. The design addresses three barriers to practical offloading: capacity balancing across heterogeneous accelerators, the large KV cache crossing a constrained wide-area link, and the latency added by heterogeneous execution. HyCoInfer incorporates FPGA decode capacity into GPU prefill/decode role partitioning and places requests according to queue load, prompt length, and network state. It further combines layer-wise pipelined transfer, decoder-side precomputation, and bandwidth-adaptive KV cache compression based on residual vector quantization. We implement HyCoInfer using two NVIDIA V100 GPUs and an AMD Xilinx Alveo U280 and evaluate it over 100-1000 Mbps links. At 500 Mbps, HyCoInfer delivers up to 4.11 times the request goodput of cloud-only vLLM and reduces mean end-to-end latency by up to 61.1%. Across different network conditions and workloads, the system improves overall serving capacity, with the largest gains on communication-intensive tasks, and reduces per-request energy.

cs.OS↗

Bounded Local Generator Classes for Deterministic State Evolution

Persistent memory must retain history without revisiting it at every transition. Bounded Local Generator Classes (BLGC) formalize carried numerical state, selected context and ordered local updates. The $\ell^\infty$ formulation permits countably many uniformly bounded coordinates without square summability. Projected maps preserve admissibility, Lipschitz estimates control finite-horizon sensitivity, and bounded acquisition and evaluation give a cardinality-independent transition-work contract. OPAL supplies a working persistent-state engine and controlled production measurements. After six successful million-node workloads, 18 matched fresh Release processes compared original and incremental receipt indexing at 10,000, 100,000 and 1,000,000 nodes. Degree, record shape, state dimension, active-target count and event schedule were fixed. Repaired resident-update P50 remained 0.083-0.084 microseconds; subsequent receipt-event P50 ranged from 4.458 to 6.459 microseconds across all scales. The original receipt path took 177.700-182.297 milliseconds at one million nodes. Reliability was tested alongside performance. Baseline and repaired engines produced byte-identical checkpoints, including continuation after fresh-process recovery. Replay, conflict, rejection, concurrent-event and allocation-failure checks passed; all 18 matched scaling processes passed their correctness gates. The repair preserves the state law and durable receipt format. These results connect the mathematical construction to a traced production transition with verified state continuity and measured scaling. Capacity-growth costs and formal scope are reported separately.

cs.OS↗

The Compute ICE-AGE: OPAL ONE, Continuity That Persists While Reasoning Is Idle

The Compute ICE-AGE separates persistent application memory from probabilistic reasoning. We present OPAL, a CPU-resident C++17 graph engine that preserves record identities, relationships, and accepted numerical state between reasoning sessions. Applications evolve that state through defined updates, save canonical checkpoints, and recover continuity after restart. Bounded Local Generator Classes (BLGC) provide the mathematical foundation for this bounded local evolution. On Apple Silicon M2 Pro hardware, the current-engine resident-update P50 measured 0.083 microseconds at 100,000, 1,000,000, and 5,000,000 retained Atomic Memory Vertices (AMVs). A same-session 1M-5M-1M sequence reproduced that median using one frozen binary and 1,024 prepared targets, with every process passing its correctness checks. These measurements cover complete resident-update calls, including validation, indexed lookup, and numerical state mutation. Initialization and storage-growth costs are reported separately. Reliability tests verified event replay, conflict rejection, concurrent calls, allocation-failure handling, and fresh-process recovery with continued operation. Engine-version comparisons produced byte-identical checkpoints and continuation files. The installed PYAMV Python package passed 17 SDK contract checks and 20 workload checks. KINDRED's Model Context Protocol interface passed 22 distinct focused checks covering memory operations, authorization, and transport. Together, these results demonstrate persistent continuity with low recurring local-update costs through five million retained AMVs, providing working interfaces for applications and developers. The architecture keeps accepted memory available while reasoning is idle, allowing applications to invoke models only when tasks require interpretation, planning, or generation.

cs.OS↗