arXiv subjects

Xixian Han

Publications and source records attributed to Xixian Han.

9 recordsLinked to original sources

arXiv2026

Efficient discovery of unique column combinations on disk-resident data with limited memory

The discovery of unique column combinations (UCCs) is a core task in data profiling, describing the key constraints of a table. The existing algorithms cannot deal with large-scale disk-resident data well due to high memory consumption and computational cost. In this paper, a novel DUD algorithm is developed to efficiently discover UCCs on disk-resident data with limited memory, which is inspired by the relationship between UCC discovery and transversal hypergraph. Rather than complete difference set generation of quadratic complexity, DUD only generates partial difference sets for hypergraph construction, followed by minimal hitting set enumeration to generate candidates and a validation process. DUD devises a strategy to generate full useful difference sets by pairwise comparisons of tuples having the same values with respect to some selected attributes. A novel theorem is developed and proved in this paper to report the candidates including the selected attributes as true UCCs directly without validation, which reduces the number of candidates to be validated significantly. A hash-based batch validation strategy is devised to validate a set of candidates on the relation instance, which only needs to maintain a small number of tuples in memory at a time. The extensive experimental results, conducted on synthetic and real-life data sets, show that DUD can discover UCCs on disk-resident data with high efficiency and low memory consumption.

cs.DB↗

arXiv2026

EviDC: A Violation-Guided Algorithm for Incremental Denial Constraint Discovery

Denial Constraints (DCs) are an important class of integrity constraints and have been widely used in data quality management. In dynamic datasets, newly inserted tuples may invalidate existing DCs and require the constraint set to be updated. Existing incremental DC discovery methods still generate a large amount of intermediate evidence because they do not exploit the structural information of existing DCs during evidence construction. We propose EviDC, a violation-guided incremental DC discovery algorithm. EviDC organizes existing DCs into a prefix tree structure called DCTrie, in which each path from the root to a leaf represents a potential violation path. During incremental processing, evidence is expanded only along reachable violation paths, while irrelevant branches are pruned as early as possible. We evaluate EviDC on real-world and synthetic datasets. The results show that EviDC reduces intermediate evidence and improves runtime efficiency in most scenarios. The performance gain becomes more pronounced as the insertion ratio and dataset size increase, showing the effectiveness and scalability of violation-guided evidence construction.

cs.DB↗

arXiv2026

Not Every Dependency Is Worth Discovering: Toward Value-Driven Data Dependency Discovery

Data dependency discovery has traditionally focused on identifying dependencies that hold in the data or are statistically strong. Yet a dependency may be valid without being valuable: it may be irrelevant to the governance task, redundant given existing knowledge, or too costly to discover, validate, maintain, and apply. We call for a shift from validity-driven to value-driven dependency discovery. We define dependency use value decision-theoretically as the expected reduction in task-specific loss from incorporating a dependency into the governance process, and define net value by further accounting for lifecycle costs. Building on this framework, we outline principles for value-aware search, validation, dependency-set selection, and maintenance, and identify a research agenda spanning value estimation before full discovery, loss and cost learning, budgeted set selection, lifecycle monitoring, and benchmarking.

cs.DB↗

arXiv2026

Top-k Approximate Functional Dependency Discovery

Approximate functional dependencies (AFDs) relax exact functional dependencies by tolerating a bounded degree of violation, making them suited for data quality auditing. Threshold-based discovery returns all dependencies above a user-specified cutoff, but output size is uncontrollable, the right threshold varies across datasets, and widely used measures are sensitive to LHS dimensionality. We study global top-$k$ AFD discovery, where neither the LHS nor the RHS is fixed and the $k$ strongest dependencies under $\mu^+$ are returned directly. The cross-attribute comparability of $\mu^+$ makes such a global ranking well-defined. We prove a Triangle Incompatibility Theorem showing that minimality, global top-$k$ ranking, and exact-$k$ output cannot simultaneously hold under any non-monotonic scoring function, justifying the removal of the minimality requirement. We present two algorithms: TALE-Base, which returns the exact global top-$k$ result by exhaustive level-wise evaluation, and TALE-Opt, which reduces computation through Apriori-style candidate generation, LHS computation reuse, and two complementary pruning rules exploiting exact FD monotonicity and an optimistic upper bound on $\mu^+$. Experiments on 41 real-world datasets show that TALE-Opt achieves pruning ratios up to 99.81\% and speedups over TALE-Base up to 78.81$\times$.

cs.DB↗

arXiv2026

Topology-Aware Subset Repair via Entropy-Guided Density and Graph Decomposition

Subset repair is an important data cleaning technique that enforces integrity constraints by deleting a minimal number of conflicting tuples, yet multiple minimal repairs often exist. Density-based methods address this ambiguity by favoring repairs that preserve dense, high-quality data regions; however, their effectiveness is limited by density bias from dirty clusters, high computational cost, and uniform attribute weighting. We propose a topology-aware approximate subset repair framework based on a joint density-conflict penalty model. The framework integrates three key components. First, a two-layer conflict detection strategy combines attribute inverted indexes with CFD rule grouping to efficiently identify violations. Second, we introduce EntroCFDensity, a density metric that incorporates information entropy and CFD weights to dynamically adjust attribute importance and reduce homogeneity bias. Third, a conflict degree measure is defined to complement local density, enabling a topology-adaptive penalty mechanism with dynamic weight allocation guided by the coefficient of variation. The conflict graph is further decomposed into independent subgraphs, transforming global repair into tractable local subproblems. Based on this framework, we develop two algorithms: PPIS, a scalable heuristic, and MICO, a mixed-integer programming method with theoretical guarantees. Experimental results show that our approach improves repair accuracy and robustness while effectively preserving high-quality data.

cs.DB↗

arXiv2026

EAIFD: A Fast and Scalable Algorithm for Incremental Functional Dependency Discovery

Functional dependencies (FDs) are fundamental integrity constraints in relational databases, but discovering them under incremental updates remains challenging. While static algorithms are inefficient due to full re-execution, incremental algorithms suffer from severe performance and memory bottlenecks. To address these challenges, this paper proposes EAIFD, a novel algorithm for incremental FD discovery. EAIFD maintains the partial hypergraph of difference sets and reframes the incremental FD discovery problem into minimal hitting set enumeration on hypergraph, avoiding full re-runs. EAIFD introduces two key innovations. First, a multi-attribute hash table ($MHT$) is devised for high-frequency key-value mappings of valid FDs, whose memory consumption is proven to be independent of the dataset size. Second, two-step validation strategy is developed to efficiently validate the enumerated candidates, which leverages $MHT$ to effectively reduce the validation space and then selectively loads data blocks for batch validation of remaining candidates, effectively avoiding repeated I/O operations. Experimental results on real-world datasets demonstrate the significant advantages of EAIFD. Compared to existing algorithms, EAIFD achieves up to an order-of-magnitude speedup in runtime while reducing memory usage by over two orders-of-magnitude, establishing it as a highly efficient and scalable solution for incremental FD discovery.

cs.DB↗

arXiv2026

Redundancy-Driven Top-$k$ Functional Dependency Discovery

Functional dependencies (FDs) are basic constraints in relational databases and are used for many data management tasks. Most FD discovery algorithms find all valid dependencies, but this causes two problems. First, the computational cost is prohibitive: computational complexity grows quadratically with the number of tuples and exponentially with the number of attributes, making discovery slow on large-scale and high-dimensional data. Second, the result set can be huge, making it hard to identify useful dependencies. We propose SDP (Selective-Discovery-and-Prune), which discovers the top-$k$ FDs ranked by redundancy count. Redundancy count measures how much duplicated information an FD explains and connects directly to storage overhead and update anomalies. SDP uses an upper bound on redundancy to prune the search space. It is proved that this upper bound is monotone: adding attributes refines partitions and thus decreases the bound. Once the bound falls below the top-$k$ threshold, the entire branch can be skipped. We improve SDP with three optimizations: ordering attributes by partition cardinality, using pairwise statistics in a Partition Cardinality Matrix to tighten bounds, and a global scheduler to explore promising branches first. Experiments on over 40 datasets show that SDP is much faster and uses less memory than exhaustive methods.

cs.DB↗

arXiv2025

PAT: Pattern-Perceptive Transformer for Error Detection in Relational Databases

Error detection in relational databases is critical for maintaining data quality and is fundamental to tasks such as data cleaning and assessment. Current error detection studies mostly employ the multi-detector approach to handle heterogeneous attributes in databases, incurring high costs. Additionally, their data preprocessing strategies fail to leverage the variable-length characteristic of data sequences, resulting in reduced accuracy. In this paper, we propose an attribute-wise PAttern-perceptive Transformer (PAT) framework for error detection in relational databases. First, PAT introduces a learned pattern module that captures attribute-specific data distributions through learned embeddings during model training. Second, the Quasi-Tokens Arrangement (QTA) tokenizer is designed to divide the cell sequence based on its length and word types, and then generate the word-adaptive data tokens, meanwhile providing compact hyperparameters to ensure efficiency. By interleaving data tokens with the attribute-specific pattern tokens, PAT jointly learns shared data features across different attributes and pattern features that are distinguishable and unique in each specified attribute. Third, PAT visualizes the attention map to interpret its error detection mechanism. Extensive experiments show that PAT achieves excellent F1 scores compared to state-of-the-art data error detection methods. Moreover, PAT significantly reduces the model parameters and FLOPs when applying the compact QTA tokenizer.

cs.DB↗

arXiv2025

Efficient Semi-External Breadth-First Search

Breadth-first search (BFS) is known as a basic search strategy for learning graph properties. As the scales of graph databases have increased tremendously in recent years, large-scale graphs G are often disk-resident. Obtaining the BFS results of G in semi-external memory model is inevitable, because the in-memory BFS algorithm has to maintain the entire G in the main memory, and external BFS algorithms consume high computational costs. As a good trade-off between the internal and external memory models, semi-external memory model assumes that the main memory can at least reside a spanning tree of G. Nevertheless, the semi-external BFS problem is still an open issue due to its difficulty. Therefore, this paper presents a comprehensive study for processing BFS in semi-external memory model. After discussing the naive solutions based on the basic framework of semi-external graph algorithms, this paper presents an efficient algorithm, named EP-BFS, with a small minimum memory space requirement, which is an important factor for evaluating semi-external algorithms. Extensive experiments are conducted on both real and synthetic large-scale graphs, where graph WDC-2014 contains over 1.7 billion nodes, and graph eu-2015 has over 91 billion edges. Experimental results confirm that EP-BFS can achieve up to 10 times faster.

cs.DS↗