Space-Optimal, Computation-Optimal, Topology-Agnostic, Throughput-Scalable Causal Delivery through Hybrid Buffering
Message delivery respecting causal ordering (causal delivery) is one of the most classic and widely useful abstraction for inter-process communication in a distributed system. Most approaches tag messages with causality information and buffer them at the receiver until they can be safely delivered. Except for specific approaches that exploit communication topology, therefore not generally applicable, they incur a metadata overhead which is prohibitive for a large number of processes. Much less used are the approaches that enforce causal order by buffering messages at the sender, until it is safe to release them to the network, as the classic algorithm has too many drawbacks. In this paper, first we discuss the limitations of sender-only buffering approaches and introduce the Sender Permission to Send (SPS) enforcement strategy, showing that SPS + FIFO implies Causal. We analyze a recent sender-buffering algorithm, Cykas, which follows SPS + FIFO, albeit very conservatively, pointing out throughput scalability and liveness issues. Then, we introduce a novel SPS + FIFO based algorithm, which adopts a new hybrid approach: enforcing causality by combining sender-buffering to enforce SPS and receiver-buffering to enforce FIFO. The algorithm overcomes limitations of sender-only buffering, and achieves effectively constant metadata size per message. By a careful choice of data-structures, the algorithm is also computationally-optimal, with amortized effectively constant processing overhead. As far as we know, there is no other topology-agnostic causal delivery algorithm with these properties.