Searcharxiv⌕ Search

arXiv subjects

Assaf Ravid

Publications and source records attributed to Assaf Ravid.

2 recordsLinked to original sources

Release Sooner, Expose Less: Offloading MoE Dispatch in NCCL-EP under Dual-Batch Overlap

Dual-batch overlap (DBO) hides mixture-of-experts (MoE) communication behind another microbatch's computation, but only after dispatch send releases the GPU. The time dispatch send holds the GPU therefore stays exposed, while network progress after release can be hidden. We evaluate GPUDirect Async - Kernel Initiated (GDA-KI) Free Flow, an NCCL-EP research configuration that leaves WQE construction on the GPU but moves send progression to a host thread. On a GB300 NVL72 system, GDA-KI Free Flow reduces dispatch-send time relative to default GDA-KI at every operating point in the evaluated parameter sweep. An experiment that inserts a GPU delay between dispatch send and receive shows that offload increases the in-flight interval. Depending on the load, 32-96 microseconds of computation or other independent work between send and receive hides the deferred progress, leaving less exposed time than default GDA-KI. In disaggregated vLLM serving with DBO, GDA-KI Free Flow lowers median TPOT by 15-25% relative to NCCL-EP default GDA-KI and by 16-42% relative to DeepEP v1.

cs.DC↗

NCCL EP: Towards a Unified Expert Parallel Communication API for NCCL

Mixture-of-Experts (MoE) architectures have become essential for scaling large language models, driving the development of specialized device-initiated communication libraries such as DeepEP, Hybrid-EP, and others. These libraries demonstrate the performance benefits of GPU-initiated RDMA for MoE dispatch and combine operations. This paper presents NCCL EP (Expert Parallelism), a ground-up MoE communication library built entirely on NCCL's Device API. NCCL EP provides unified ncclEpDispatch and ncclEpCombine primitives with both C and Python interfaces, supporting Low-Latency (LL) mode for inference decoding and High-Throughput (HT) mode for training and inference prefill. LL targets small batch sizes (1-128 tokens) using direct all-to-all RDMA+NVLink mesh connectivity with double-buffered communication for overlapping dispatch and combine phases. HT targets large batches (4096+ tokens) using hierarchical communication that aggregates tokens within NVLink domains before inter-node RDMA transmission. Both modes leverage Device API for both intra- and inter-node communications, taking advantage of its topology awareness and optimized GPU-initiated implementation. We evaluate NCCL EP on an H100-based cluster across multi-node configurations, demonstrating competitive LL kernel performance and presenting end-to-end results with vLLM integration. By building MoE communication natively within NCCL, NCCL EP provides a supported path for expert parallelism on current and emerging NVIDIA platforms.

cs.DC↗