Communication-Semantic-Aware RDMA Loss Recovery for QP-scalable Hyperscale AI Training

Abstract

Current artificial intelligence (AI) infrastructures widely adopt Remote Direct Memory Access (RDMA) to support high-performance communication. Training trillion-parameter models involves frequent collective communication operations, such as All-Reduce and All-to-All, which generate intensive RDMA traffic. Existing RDMA deployments predominantly use the reliable connection (RC) model, where each process pair requires a dedicated queue pair (QP). This leads to poor scalability: since the RDMA-capable network interface card (RNIC) can cache only a few thousand QPs, excess entries trigger PCIe round-trip penalties. Meanwhile, global synchronization makes training sensitive to tail latency, where a few packet losses can delay iteration completion. To address these challenges, we propose Communication-Semantic-Aware Unreliable Datagram (CSA-UD), a novel RDMA loss recovery mechanism that combines scalability and reliability. CSA-UD decouples data transmission from loss recovery and dynamically adjusts the loss detection interval, accelerating tail recovery and exploiting the synchronization semantics of distributed training. It further supports multipath transmission and bitmap-guided reassembly, enabling high throughput without requiring lossless fabrics. Testbed experiments and ns-3 simulations show that CSA-UD significantly reduces tail latency under large-scale collective communication. Under high network load, it achieves better scalability than RC and over 30% lower 99th percentile flow completion times compared with counterparts.