SHIFT: Exploring the Boundary of RDMA Network Fault Tolerance

Abstract

Under gang scheduling for large-scale distributed large language model (LLM) training, a single network anomaly can stall or abort an entire job. Current network fault tolerance mechanisms typically adopt a ``fallback and bypass'' approach within the switching fabric and at the access layer, tolerating in-network and access-layer failures. We explore whether RDMA fault tolerance can be extended to the cross-NIC level by failing over traffic to intra-host backup NICs. For the first time, we prove a fundamental Trilemma: it is impossible to have Cross-NIC RDMA failover that simultaneously preserves Exactly-Once Execution, Receiver-NIC Opacity, and a Zero-Copy datapath. Fortunately, we observe that dominant training frameworks (e.g., NCCL) rely on idempotent bulk transfers that tolerate relaxed memory ordering, as long as notification ordering is preserved. Leveraging this insight, we present SHIFT, a user-space RDMA layer that provides cross-NIC fault tolerance while preserving correct memory semantics. We implement SHIFT in \texttt{rdma-core} and evaluate it with PyTorch distributed training. Results show that SHIFT incurs negligible overhead during normal operation and successfully masks fatal NIC failures and link anomalies, allowing training to continue without costly restarts.