arXiv subjects

Jiwu Shu

Publications and source records attributed to Jiwu Shu.

28 records · Page 2Linked to original sources

arXiv2023

From RDMA to RDCA: Toward High-Speed Last Mile of Data Center Networks Using Remote Direct Cache Access

In this paper, we conduct systematic measurement studies to show that the high memory bandwidth consumption of modern distributed applications can lead to a significant drop of network throughput and a large increase of tail latency in high-speed RDMA networks.We identify its root cause as the high contention of memory bandwidth between application processes and network processes. This contention leads to frequent packet drops at the NIC of receiving hosts, which triggers the congestion control mechanism of the network and eventually results in network performance degradation. To tackle this problem, we make a key observation that given the distributed storage service, the vast majority of data it receives from the network will be eventually written to high-speed storage media (e.g., SSD) by CPU. As such, we propose to bypass host memory when processing received data to completely circumvent this performance bottleneck. In particular, we design Lamda, a novel receiver cache processing system that consumes a small amount of CPU cache to process received data from the network at line rate. We implement a prototype of Lamda and evaluate its performance extensively in a Clos-based testbed. Results show that for distributed storage applications, Lamda improves network throughput by 4.7% with zero memory bandwidth consumption on storage nodes, and improves network throughput by up 17% and 45% for large block size and small size under the memory bandwidth pressure, respectively. Lamda can also be applied to latency-sensitive HPC applications, which reduces their communication latency by 35.1%.

cs.NI↗

arXiv2022

RIO: Order-Preserving and CPU-Efficient Remote Storage Access

Modern NVMe SSDs and RDMA networks provide dramatically higher bandwidth and concurrency. Existing networked storage systems (e.g., NVMe over Fabrics) fail to fully exploit these new devices due to inefficient storage ordering guarantees. Severe synchronous execution for storage order in these systems stalls the CPU and I/O devices and lowers the CPU and I/O performance efficiency of the storage system. We present Rio, a new approach to the storage order of remote storage access. The key insight in Rio is that the layered design of the software stack, along with the concurrent and asynchronous network and storage devices, makes the storage stack conceptually similar to the CPU pipeline. Inspired by the CPU pipeline that executes out-of-order and commits in-order, Rio introduces the I/O pipeline that allows internal out-of-order and asynchronous execution for ordered write requests while offering intact external storage order to applications. Together with merging consecutive ordered requests, these design decisions make for write throughput and CPU efficiency close to that of orderless requests. We implement Rio in Linux NVMe over RDMA stack, and further build a file system named RioFS atop Rio. Evaluations show that Rio outperforms Linux NVMe over RDMA and a state-of-the-art storage stack named Horae by two orders of magnitude and 4.9 times on average in terms of throughput of ordered write requests, respectively. RioFS increases the throughput of RocksDB by 1.9 times and 1.5 times on average, against Ext4 and HoraeFS, respectively.

cs.OS↗

arXiv2022

Replicating Persistent Memory Key-Value Stores with Efficient RDMA Abstraction

Combining persistent memory (PM) with RDMA is a promising approach to performant replicated distributed key-value stores (KVSs). However, existing replication approaches do not work well when applied to PM KVSs: 1) Using RPC induces software queueing and execution at backups, increasing request latency; 2) Using one-sided RDMA WRITE causes many streams of small PM writes, leading to severe device-level write amplification (DLWA) on PM. In this paper, we propose Rowan, an efficient RDMA abstraction to handle replication writes in PM KVSs; it aggregates concurrent remote writes from different servers, and lands these writes to PM in a sequential (thus low DLWA) and one-sided (thus low latency) manner. We realize Rowan with off-the-shelf RDMA NICs. Further, we build Rowan-KV, a log-structured PM KVS using Rowan for replication. Evaluation shows that under write-intensive workloads, compared with PM KVSs using RPC and RDMA WRITE for replication, Rowan-KV boosts throughput by 1.22X and 1.39X as well as lowers median PUT latency by 1.77X and 2.11X, respectively, while largely eliminating DLWA.

cs.DC↗

arXiv2021

Sherman: A Write-Optimized Distributed B+Tree Index on Disaggregated Memory

Memory disaggregation architecture physically separates CPU and memory into independent components, which are connected via high-speed RDMA networks, greatly improving resource utilization of databases. However, such an architecture poses unique challenges to data indexing in databases due to limited RDMA semantics and near-zero computation power at memory-side. Existing indexes supporting disaggregated memory either suffer from low write performance, or require hardware modification. This paper presents Sherman, a write-optimized distributed B+Tree index on disaggregated memory that delivers high performance with commodity RDMA NICs. Sherman combines RDMA hardware features and RDMA-friendly software techniques to boost index write performance from three angles. First, to reduce round trips, Sherman coalesces dependent RDMA commands by leveraging in-order delivery property of RDMA. Second, to accelerate concurrent accesses, Sherman introduces a hierarchical lock that exploits on-chip memory of RDMA NICs. Finally, to mitigate write amplification, Sherman tailors the data structure layout of B+Tree with a two-level version mechanism. Our evaluation shows that, Sherman is one order of magnitude faster in terms of both throughput and 99th percentile latency on typical write-intensive workloads, compared with state-of-the-art designs.

cs.DC↗

arXiv2020

Sapphire: Automatic Configuration Recommendation for Distributed Storage Systems

Modern distributed storage systems come with aplethora of configurable parameters that controlmodule behavior and affect system performance. Default settings provided by developers are often suboptimal for specific user cases. Tuning parameters can provide significant performance gains but is a difficult task requiring profound experience and expertise, due to the immense number of configurable parameters, complex inner dependencies and non-linearsystem behaviors. To overcome these difficulties, we propose an automatic simulation-based approach, Sapphire, to recommend optimal configurations by leveraging machine learning and black-box optimization techniques. We evaluate Sapphire on Ceph. Results show that Sapphire significantly boosts Ceph performance to 2.2x compared to the default configuration.

cs.DC↗

arXiv2019

Kernel/User-level Collaborative Persistent Memory File System with Efficiency and Protection

Emerging high performance non-volatile memories recall the importance of efficient file system design. To avoid the virtual file system (VFS) and syscall overhead as in these kernel-based file systems, recent works deploy file systems directly in user level. Unfortunately, a userlevel file system can easily be corrupted by a buggy program with misused pointers, and is hard to scale on multi-core platforms which incorporates a centralized coordination service. In this paper, we propose KucoFS, a Kernel and user-level collaborative file system. It consists of two parts: a user-level library with direct-access interfaces, and a kernel thread, which performs metadata updates and enforces write protection by toggling the permission bits in the page table. Hence, KucoFS achieves both direct-access of user-level designs and fine-grained write protection of kernel-level ones. We further explore its scalability to multicores: For metadata scalability, KucoFS rebalances the pathname resolution overhead between the kernel and userspace, by adopting the index offloading technique. For data access efficiency, it coordinates the data allocation between kernel and userspace, and uses range-lock write and lock-free read to improve concurrency. Experiments on Optane DC persistent memory show that KucoFS significantly outperforms existing file systems and shows better scalability.

cs.OS↗

arXiv2017

Improving the Performance and Endurance of Persistent Memory with Loose-Ordering Consistency

Persistent memory provides high-performance data persistence at main memory. Memory writes need to be performed in strict order to satisfy storage consistency requirements and enable correct recovery from system crashes. Unfortunately, adhering to such a strict order significantly degrades system performance and persistent memory endurance. This paper introduces a new mechanism, Loose-Ordering Consistency (LOC), that satisfies the ordering requirements at significantly lower performance and endurance loss. LOC consists of two key techniques. First, Eager Commit eliminates the need to perform a persistent commit record write within a transaction. We do so by ensuring that we can determine the status of all committed transactions during recovery by storing necessary metadata information statically with blocks of data written to memory. Second, Speculative Persistence relaxes the write ordering between transactions by allowing writes to be speculatively written to persistent memory. A speculative write is made visible to software only after its associated transaction commits. To enable this, our mechanism supports the tracking of committed transaction ID and multi-versioning in the CPU cache. Our evaluations show that LOC reduces the average performance overhead of memory persistence from 66.9% to 34.9% and the memory write traffic overhead from 17.1% to 3.4% on a variety of workloads.

cs.AR↗

arXiv2017

Sky: Opinion Dynamics Based Consensus for P2P Network with Trust Relationships

Traditional Byzantine consensus does not work in P2P network due to Sybil attack while the most prevalent Sybil-proof consensus at present still can't resist adversary with dominant compute power. This paper proposed opinion dynamics based consensus for P2P network with trust relationships, consisting of the sky framework and the sky model. With the sky framework, opinion dynamics can be applied in P2P network for consensus which is Sybil-proof through trust relationships and emerges from local interactions of each node with its direct contacts without topology, global information or even sample of the network involved. The sky model has better performance of convergence than existing models including MR, voter and Sznajd, and its lower bound of fault tolerance performance is also analyzed and proved. Simulations show that our approach can tolerant failures by at least 13% random nodes or 2% top influential nodes while over 96% correct nodes still make correct decision within 70 seconds on the SNAP Wikipedia who-votes-on-whom network for initial configuration of convergence>0.5 with reasonable latencies. Comparing to compute power based consensus, our approach can resist any faulty or malicious nodes by unfollowing them. To the best of our knowledge, it's the first work to bring opinion dynamics to P2P network for consensus.

cs.DC↗

arXiv2017

SkyHash: a Hash Opinion Dynamics Model

This paper proposes the first hash opinion dynamics model, named SkyHash, that can help a P2P network quickly reach consensus on hash opinion. The model consists of a bit layer and a hash layer, each time when a node shapes its new opinion, the bit layer is to determine each bit of a pseudo hash, and the hash layer is to choose a hash opinion with minimum Hamming distance to the pseudo hash. With simulations, we conducted a comprehensive study on the convergence speed of the model by taking into account impacts of various configurations such as network size, node degree, hash size, and initial hash density. Evaluation demonstrates that using our model, consensus can be quickly reached even in large networks. We also developed a denial-of-service (DoS) proof extension for our model. Experiments on the SNAP dataset of the Wikipedia who-votes-on-whom network demonstrate that besides the ability to refuse known ill-behaved nodes, the DoS-proof extended model also outperforms Bitcoin by producing consensus in 45 seconds, and tolerating DoS attack committed by up to 0.9% top influential nodes.

cs.DC↗

arXiv2012

C-Codes: Cyclic Lowest-Density MDS Array Codes Constructed Using Starters for RAID 6

The distance-3 cyclic lowest-density MDS array code (called the C-Code) is a good candidate for RAID 6 because of its optimal storage efficiency, optimal update complexity, optimal length, and cyclic symmetry. In this paper, the underlying connections between C-Codes (or quasi-C-Codes) and starters in group theory are revealed. It is shown that each C-Code (or quasi-C-Code) of length $2n$ can be constructed using an even starter (or even multi-starter) in $(Z_{2n},+)$. It is also shown that each C-Code (or quasi-C-Code) has a twin C-Code (or quasi-C-Code). Then, four infinite families (three of which are new) of C-Codes of length $p-1$ are constructed, where $p$ is a prime. Besides the family of length $p-1$, C-Codes for some sporadic even lengths are also presented. Even so, there are still some even lengths (such as 8) for which C-Codes do not exist. To cover this limitation, two infinite families (one of which is new) of quasi-C-Codes of length $2(p-1)$ are constructed for these even lengths.

cs.IT↗