Fundamental Limits of Hypergraph Edge Partitioning under Independent Edge Sampling

Abstract

Hypergraph edge partitioning is a central problem in theoretical and applied computer science, with broad impact on distributed computation, communications, optimization, and machine learning. In this setting, one is given a collection of hyperedges -- each consisting of up to $d$ vertices from a ground set of size $n$ -- and seeks to assign these hyperedges across $N$ partitions so as to minimize, for example, the vertex footprint, i.e., the maximum number of vertices that appear in any partition. We here identify the fundamental limits of hypergraph edge partitioning -- optimized over all conceivable algorithms -- for a broad class of probabilistic hypergraph models where each hyperedge may appear independently with \emph{its own} probability; a model sufficiently general to encompass well-known models such as the Degree-Corrected or Mixed-Membership models, the Hypergraph Stochastic Block model, the Latent-Space/Geometric or Kernel Models, and others. By pairing our deterministic partitioner with a new converse, we first show that, for any $n,d$, and under the very mild condition of $N \leq \binom{\lfloor\sqrt{\frac{nd}{2}}\rfloor}{d}$, as long as the hyperedge set $\mathbf{X}$ satisfies $|\mathbf{X}| \gtrsim n N \log N$, then with probability at least $1-2/3n^z$, no algorithm can provide a footprint $\pi_{\mathbf{X}}$ less than $$\pi^{\bigstar}_{\mathbf{X}} = \frac{1}{2\sqrt{2}}\frac{n}{N^{1/d}}. $$ We then show that our hypergraph partitioner comes to within a small constant factor from $\pi^{\bigstar}_{\mathbf{X}}$, for each $\mathbf{X}$. This optimality captures dense and sparse hypergraphs alike (with sizes down to linear in $n$), and it additionally entails a near-optimally balanced allocation of hyperedges across partitions.