arXiv subjects

Raj Joshi

Publications and source records attributed to Raj Joshi.

4 recordsLinked to original sources

arXiv2025

ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching

Distributed prefix caching accelerates long-context LLM serving by reusing KV cache entries for common context prefixes. However, KV cache fetches can become a bottleneck when network bandwidth is limited. Compression mitigates the bandwidth issue, but can degrade overall performance when decompression interferes with model computation. We present ShadowServe, the first SmartNIC-accelerated, interference-free prefix caching system for LLM serving. ShadowServe separates a control plane on the host and a data plane fully offloaded to the SmartNIC, which eliminates interference to both host GPU and CPU. To overcome the SmartNIC's limited compute and memory resources, we design a chunked pipeline that parallelizes data plane operations across the SmartNIC's compute resources, and a minimal-copy memory management scheme that reduces memory pressure on the SmartNIC. Compared to state-of-the-art solutions, ShadowServe achieves up to 2.2x lower loaded time-per-output-token (TPOT), and reduces time-to-first-token (TTFT) by up to 1.38x in low-bandwidth scenarios (<= 20 Gbps), translating to up to 1.35x higher throughput.

cs.DC↗

arXiv2024

Unlocking Diversity of Fast-Switched Optical Data Center Networks with Unified Routing

Optical data center networks (DCNs) are emerging as a promising solution for cloud infrastructure in the post-Moore's Law era, particularly with the advent of 'fast-switched' optical architectures capable of circuit reconfiguration at microsecond or even nanosecond scales. However, frequent reconfiguration of optical circuits introduces a unique challenge: in-flight packets risk loss during these transitions, hindering the deployment of many mature optical hardware designs due to the lack of suitable routing solutions. In this paper, we present Unified Routing for Optical networks (URO), a general routing framework designed to support fast-switched optical DCNs across various hardware architectures. URO combines theoretical modeling of this novel routing problem with practical implementation on programmable switches, enabling precise, time-based packet transmission. Our prototype on Intel Tofino2 switches achieves a minimum circuit duration of 2us, ensuring end-to-end, loss-free application performance. Large-scale simulations using production DCN traffic validate URO's generality across different hardware configurations, demonstrating its effectiveness and efficient system resource utilization.

cs.NI↗

arXiv2024

OpenOptics: An Open Research Framework for Optical Data Center Networks

Optical data center networks (DCNs) are emerging as a promising design for cloud infrastructure. However, existing optical DCN architectures operate as closed ecosystems, tying software solutions to specific optical hardware. We introduce OpenOptics, an open research framework that decouples software from hardware, allowing them to evolve independently. OpenOptics features: (1) a time-flow table abstraction as a common interface between optical hardware and software, (2) a unified workflow and user-friendly API for implementing various optical DCNs with simple Python scripts, and (3) a backend system that re-architects queue management to support the time-flow tables and provides rich infrastructure services for diverse applications. Built on programmable switches, OpenOptics achieves a record-breaking minimum optical circuit duration of 2 $\mu$s using commodity devices. We validate OpenOptics' generality by implementing six optical architectures and seven routing schemes on an optical testbed and conducting benchmarks on a 108-ToR setup, showcasing its efficiency. Additionally, case studies highlight novel research opportunities enabled by OpenOptics.

cs.NI↗

arXiv2024

Nanosecond Precision Time Synchronization for Optical Data Center Networks

Optical data center networks (DCNs) are renovating the infrastructure design for the cloud in the post Moore's law era. The fact that optical DCNs rely on optical circuits of microsecond-scale durations makes nanosecond-precision time synchronization essential for the correct functioning of routing on the network fabric. However, current studies on optical DCNs neglect the fundamental need for accurate time synchronization. In this paper, we bridge the gap by developing Nanosecond Optical Synchronization (NOS), the first nanosecond-precision synchronization solution for optical DCNs general to various optical hardware. NOS builds clock propagation trees on top of the dynamically reconfigured circuits in optical DCNs, allowing switches to seek better sync parents throughout time. It predicts drifts in the tree-building process, which enables minimization of sync errors. We also tailor today's sync protocols to the needs of optical DCNs, including reducing the number of sync messages to fit into short circuit durations and correcting timestamp errors for higher sync accuracy. Our implementation on programmable switches shows 28ns sync accuracy in a 192-ToR setting.

cs.NI↗