Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.21162

Coordination-Free Workload Regularization for Variable-Cost Distributed Training

Tao Huang·Chen Zhu·Antonio Alonso·Daming Li·Adarsh Goura·Jaideep Ray·Yitong Zhou
View original source ↗Read source document ↗Export metadata ↓

Abstract

Synchronous data-parallel training usually assigns every rank the same number of records, even when record costs vary by orders of magnitude. For variable-length models, this converts input variation into idle time because each step waits for its most expensive rank-local batch. We present Batch Cost Lookahead (BCL), a coordination-free data-loading policy that buffers a bounded rank-local window and repartitions it into fixed-cardinality packs with balanced estimated cost. BCL preserves the trainer's record-count contract and requires neither cross-rank metadata exchange nor runtime modification. In simulation over public workload distributions, enabling BCL reduces per-rank estimated batch-cost variability by up to 82.8%. Across 13 NeMo-RL SFT corpora on eight H200 GPUs, rank-local BCL runs 1.050 to 1.167 times faster than pooled MFFD. Compiled H200 measurements on an internal sequence workload show speedups of 1.197 to 1.343 times over packed execution from one to four nodes. We also identify a memory-feasibility boundary: a p99 cap for high-variance or OOM-prone data bounds outliers before BCL balances the rest. These results show that substantial variable-cost imbalance can be removed at the data-loader boundary, while clarifying when data preparation or broader coordination is still necessary.

Explore related subjects

cs.DCcs.LG

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Tao Huang, Chen Zhu, Antonio Alonso, Daming Li, Adarsh Goura, Jaideep Ray, Yitong Zhou. 2026-10-15. Coordination-Free Workload Regularization for Variable-Cost Distributed Training. https://arxiv.org/abs/2610.21162

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

ZenFlow: Enabling Stall-Free Offloading for LLM Training

Fine-tuning large language models (LLMs) often exceeds GPU memory limits, prompting systems to offload model states to CPU memory. However, existing offloaded training frameworks like ZeRO-Offload treat all parameters equally and update the full model on the CPU, causing severe GPU stalls, where fast, expensive GPUs sit idle, waiting for slow CPU updates and limited-bandwidth PCIe transfers. We present ZenFlow, a new offloaded training framework that decouples updates between GPU and CPU by prioritizing important gradients. ZenFlow performs in-place updates of important gradients on GPU while offloading and accumulating less important ones on CPU without blocking training iterations, fully overlapping CPU work with GPU computation. To make this importance-aware design scalable, ZenFlow leverages a novel spatio-temporal locality of important gradients, concentrated in specific positions that remain stable across iterations, to make important gradient identification efficient and lightweight while preserving fidelity. Evaluation shows that ZenFlow achieves up to 5x end-to-end speedup, 2x lower PCIe traffic, 85% less GPU stalls, and up to 72% lower cloud GPU cost, all while preserving accuracy.

cs.DC↗
arXiv2026

Asynchronous Indulgent Gathering of Opaque Robots

We consider the benchmarking problem of gathering in an $(N,f)$-fault system consisting of $N$ robots, of which at most $f$ may fail at any execution. The {\em traditional} version asks $N-f$ non-faulty robots to gather at a point not known a priori. The recently studied {\em indulgent} version asks $N-f$ non-faulty robots to gather at the location of a faulty one. Indulgent gathering is impossible in a $(2,1)$-crash system in the $LUMI$ model under semi-synchrony, even when infinitely many colors and coordinate axis agreement are available. In this paper, we establish four results for indulgent gathering under asynchrony in $LUMI$, considering a {\em mobility fault} model in which a robot crash only impacts its mobility but not the operation of the light. We first show that indulgent gathering is impossible in a $(2,1)$-mobility system using 2-colored lights, and then give an optimal solution w.r.t. the number of colors using 3-colored lights. We then give two deterministic algorithms for an $(N,f)$-mobility system, $f<N$, $N,f$ unknown, that exhibit a nice time-color trade-off: One with time $O(N)$ using 7-colored lights and another with time $O(\max\{\ell,f\})$ using 26-colored lights, where $\ell< N$ is the number of distinct convex layers of robot positions in the initial configuration. Our algorithms for an $(N,f)$-mobility system are the first to be analysed time complexity, can withstand obstructed visibility (opaque robot model) and asynchronous scheduling. Interestingly, our algorithms also solve traditional gathering.

cs.DC↗
arXiv2026

TeraAgent: A Distributed Agent-Based Simulation Engine for Simulating Half a Trillion Agents

Agent-based simulation is a method for studying highly complex systems in disciplines as diverse as biology, medicine, sociology, economics, finance, and more. In this method, an agent represents a small part of the system, such as a human cell, and, through interactions among many agents, the system's behavior emerges, like the growth of cancer in the human body. Real-world complex systems, such as a human organ, comprise billions of agents. However, the state-of-the-art agent-based simulation platform, BioDynaMo, does not support the distributed processing of a simulation across servers. This limitation restricts the scalability of agent counts to the computational resources available within a single server, hindering users from conducting larger and more meaningful simulations. In this paper, we introduce TeraAgent, a new distributed agent-based simulation engine. To overcome BioDynaMo's scalability limitation, TeraAgent enables efficient exchange of agent information across servers by implementing two solutions: 1) a tailored serialization mechanism that allows agents to be accessed and mutated directly at the receive buffer, and 2) delta encoding that significantly reduces data transfer across servers by leveraging the iterative nature of agent-based simulations. Built on our solutions, TeraAgent 1) enables extreme-scale simulations with half a trillion agents (292x that of BioDynaMo) on 84,096 CPU cores, 2) reduces time-to-result leveraging parallelism across compute nodes, 3) improves interoperability with third-party tools, and 4) provides users with more hardware flexibility. TeraAgent serializes agents with a median speedup of 110x (37x for deserialization) over the state-of-the-art ROOT I/O baseline, delivers 8x the agent update rate per CPU core of Biocellion, and improves in-situ visualization performance by 39x.

cs.DC↗