EAIFD: A Fast and Scalable Algorithm for Incremental Functional Dependency Discovery

Abstract

Functional dependencies (FDs) are fundamental integrity constraints in relational databases, but discovering them under incremental updates remains challenging. While static algorithms are inefficient due to full re-execution, incremental algorithms suffer from severe performance and memory bottlenecks. To address these challenges, this paper proposes EAIFD, a novel algorithm for incremental FD discovery. EAIFD maintains the partial hypergraph of difference sets and reframes the incremental FD discovery problem into minimal hitting set enumeration on hypergraph, avoiding full re-runs. EAIFD introduces two key innovations. First, a multi-attribute hash table ($MHT$) is devised for high-frequency key-value mappings of valid FDs, whose memory consumption is proven to be independent of the dataset size. Second, two-step validation strategy is developed to efficiently validate the enumerated candidates, which leverages $MHT$ to effectively reduce the validation space and then selectively loads data blocks for batch validation of remaining candidates, effectively avoiding repeated I/O operations. Experimental results on real-world datasets demonstrate the significant advantages of EAIFD. Compared to existing algorithms, EAIFD achieves up to an order-of-magnitude speedup in runtime while reducing memory usage by over two orders-of-magnitude, establishing it as a highly efficient and scalable solution for incremental FD discovery.