Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.36564

DeepInfer-Mac: Running a 750B-Class MoE on a 24 GB Laptop

Haorui Li
View original source ↗Read source document ↗Export metadata ↓

Abstract

DeepInfer-Mac runs the 750B-class DeepSeek-V4.1-Flash model on a MacBook Air with 24 GB of unified memory. A mixed low-bit conversion occupies 338 GiB on the internal SSD. About 7.6 GiB of weights remain resident; routed experts and Engram rows are loaded when needed. This separation makes the model executable, but it leaves each generated token waiting for a succession of storage reads. We organize expert weights so that reads fill reusable GPU buffers directly, and couple cache allocation with the lifetime of those buffers. In end-to-end runs, the system generates English, Chinese and code at 0.21-0.72 tokens per second, with a largest sampled process footprint of 16.73 GiB across the tested requests. Matched decode comparisons show a 21.0% throughput gain from next-layer prefetching over resident-expert overlap alone, at the cost of additional reads. Longer prompts remain costly. We describe the design, provide the implementation and measurement scripts, and identify the limits of this operating point: a model far larger than memory can run locally, while storage and prefill still govern the user's wait.

Explore related subjects

cs.PF

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Haorui Li. 2026-10-27. DeepInfer-Mac: Running a 750B-Class MoE on a 24 GB Laptop. https://arxiv.org/abs/2610.36564

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

Measuring the Energy Cost of Completing Tasks with Quantized Language-Model Agents

Quantization reduces the memory and energy needed to serve large language models (LLMs) and AI agents for coding and data analysis can now run on locally hosted LLMs. In practice, this benefit is commonly benchmarked per output token. Yet an agent calls its model repeatedly, runs tools, and revises its work, and quantization can change both how much the agent generates and whether it succeeds. Existing energy studies either compare quantized models on single requests or measure agents at one fixed precision, overlooking how a change in precision plays out across a complete agent task. To address this gap, this paper presents the first characterization of the task-level energy trade-offs of model quantization in locally hosted agents. We develop 29 agent tasks with automatic pass/fail checks in six families, ranging from single-answer SQL analytics to multi-step repository bug fixing, and measure CPU and GPU energy per output token and per completed task for 4-bit and 8-bit versions of two open-weight models on bare-metal NVIDIA A100 nodes with llama$.$cpp. Our findings reveal that 4-bit precision generally saves energy per token, though whether it saves energy per completed task is task dependent and changes with the model and the execution budget (i.e., limits on turns, tokens, and time). We trace this discrepancy to the extra tokens needed to complete a task in full, which translates into using up to 3.5 times as much energy per completed task. These findings offer actionable guidance for developers and operators choosing model precision for agents, showing how tokens per completed task, set against the per-token saving, reveal whether lower precision completes the work with less energy.

cs.PF↗
arXiv2026

One Megakernel is Not Enough: Predicting Where to Cut for Streaming Inference

We address a fundamental question: How far should kernel fusion extend and which boundaries are worth preserving? Unlike the general convention of using one monolith megakernel at lower batch sizes or supporting only a selective model component as megakernel and rest of the model as conventional graph, we show that the best configuration is a composition of multiple megakernels with no overlapping conventional graph component, or a fully conventional graph. We present a framework that selects megakernel regions in a streaming inference model from conventional-graph profiles and standalone kernel benchmarks, without running any candidate design end to end. It simulates each candidate's tile and event graphs to predict time to first output and selects the measured best design in all 13 combinations of model and batch size we tested. Across NH-TTS, Qwen3-TTS and Llama-1B, the best composition ranges from two to four megakernels to fully conventional execution and none is a single megakernel. We introduce four model categories (megakernel-stable/ receding/ advancing/ averse) that capture how a model's best megakernel composition changes with batch size. Using NH-TTS model on H100 NVL GPU, we demonstrate that preserving three specialized megakernels can improve both latency and throughput despite introducing additional kernel boundaries. At batch size 32, this design delivers 21.4% lower mean batch-p95 time to first audio (TTFA) and 28% higher throughput than the evaluated monolith megakernel. Against conventional TensorRT graph execution, it reduces engine-level mean batch TTFA p95 by 4.05x and increases throughput by 19.7%.

cs.PF↗
arXiv2026

Managing Shared Hospital Capacity for Medical Tourism: Patient Behavior and Hybrid Simulation

Hospitals that serve medical tourists alongside local patients share beds, specialists, and appointment slots between two groups for whom waiting has different consequences. Operations research models are widely used for hospital capacity and scheduling, but often do not explicitly capture patient behavior; agent-based simulation can represent such behavior alongside discrete-event patient flow, while medical-tourism research still mainly treats traveling patients as cases to be scheduled. We develop a hybrid simulation that represents channel choice, physician switching, treatment abandonment, and medication-related deterioration in a multi-specialty international patient department. A 256-run fractional factorial experiment screens 16 controllable factors across six performance measures, followed by a replicated comparison with a discrete-event-only version without these behaviors. At a bed-constrained configuration, mean inpatient admission waiting time is 42% lower in the hybrid model (14.8 versus 25.8 days in the discrete-event-only model, p < 0.001) and 25% to 64% lower under two further conditions. Lacking these behaviors, the simpler model shows no abandonment and fewer emergency escalations. The comparison shows how results depend on model structure; neither variant is validated against observed waiting times. A bed-sharing routine in both variants adds no beds as implemented, so bed sharing is not evaluated. The screening identifies bed capacity as the most consistent driver of waiting; priority rules, channel design, and appointment timing also matter, while specialist capacity has less effect on waiting. Managers should review bed capacity and admission rules before adding specialists or tightening schedules.

cs.PF↗