arXiv subjects

Jianzhong Li

Publications and source records attributed to Jianzhong Li.

At least 37 records · Page 2Linked to original sources

arXiv2021

Impacts of Dirty Data: and Experimental Evaluation

Data quality issues have attracted widespread attention due to the negative impacts of dirty data on data mining and machine learning results. The relationship between data quality and the accuracy of results could be applied on the selection of the appropriate algorithm with the consideration of data quality and the determination of the data share to clean. However, rare research has focused on exploring such relationship. Motivated by this, this paper conducts an experimental comparison for the effects of missing, inconsistent and conflicting data on classification and clustering algorithms. Based on the experimental findings, we provide guidelines for algorithm selection and data cleaning.

cs.DB↗

arXiv2020

Efficient Approximate Nearest Neighbor Search for Multiple Weighted $l_{p\leq2}$ Distance Functions

Nearest neighbor search is fundamental to a wide range of applications. Since the exact nearest neighbor search suffers from the "curse of dimensionality", approximate approaches, such as Locality-Sensitive Hashing (LSH), are widely used to trade a little query accuracy for a much higher query efficiency. In many scenarios, it is necessary to perform nearest neighbor search under multiple weighted distance functions in high-dimensional spaces. This paper considers the important problem of supporting efficient approximate nearest neighbor search for multiple weighted distance functions in high-dimensional spaces. To the best of our knowledge, prior work can only solve the problem for the $l_2$ distance. However, numerous studies have shown that the $l_p$ distance with $p\in(0,2)$ could be more effective than the $l_2$ distance in high-dimensional spaces. We propose a novel method, WLSH, to address the problem for the $l_p$ distance for $p\in(0,2]$. WLSH takes the LSH approach and can theoretically guarantee both the efficiency of processing queries and the accuracy of query results while minimizing the required total number of hash tables. We conduct extensive experiments on synthetic and real data sets, and the results show that WLSH achieves high performance in terms of query efficiency, query accuracy and space consumption.

cs.DB↗

arXiv2020

Sampling Based Approximate Skyline Calculation on Big Data

The existing algorithms for processing skyline queries cannot adapt to big data. This paper proposes two approximate skyline algorithms based on sampling. The first algorithm obtains a fixed size sample and computes the approximate skyline on the sample. The error of the first algorithm is relatively small in most cases, and is almost independent of the input relation size. The second algorithm returns an $(ε,δ)$-approximation for the exact skyline. The size of sample required by the second algorithm can be regarded as a constant relative to the input relation size, so is the running time. Experiments verify the error analysis of the first algorithm and show that the second algorithm is much faster than the existing skyline algorithms.

cs.DB↗

arXiv2020

A Sub-linear Time Algorithm for Approximating k-Nearest-Neighbor with Full Quality Guarantee

In this paper we propose an algorithm for the approximate k-Nearest-Neighbors problem. According to the existing researches, there are two kinds of approximation criterion. One is the distance criteria, and the other is the recall criteria. All former algorithms suffer the problem that there are no theoretical guarantees for the two approximation criterion. The algorithm proposed in this paper unifies the two kinds of approximation criterion, and has full theoretical guarantees. Furthermore, the query time of the algorithm is sub-linear. As far as we know, it is the first algorithm that achieves both sub-linear query time and full theoretical approximation guarantee.

cs.CG↗

arXiv2020

PHOTOPiC: Calculate photo-ionization functions and model coefficients for gas discharge simulations

A program to compute photo-ionization functions and fitting parameters for an efficient photo-ionization model is presented. The code integrates the product of spectrum emission intensity, the photo-ionization yield and the absorption coefficient to calculate the photo-ionization function of each gas and the total photo-ionization function of the mixture. The coefficients of Helmholtz photo-ionization model is obtained by fitting the total photo-ionization function. A database consisting $\rm N_2$, $\rm O_2$, $\rm CO_2$ and $\rm H_2O$ molecules are included and can be modified by the users. The program provides more accurate photo-ionization functions and source terms for plasma fluid models.

physics.plasm-ph↗

arXiv2020

Auto-Model: Utilizing Research Papers and HPO Techniques to Deal with the CASH problem

In many fields, a mass of algorithms with completely different hyperparameters have been developed to address the same type of problems. Choosing the algorithm and hyperparameter setting correctly can promote the overall performance greatly, but users often fail to do so due to the absence of knowledge. How to help users to effectively and quickly select the suitable algorithm and hyperparameter settings for the given task instance is an important research topic nowadays, which is known as the CASH problem. In this paper, we design the Auto-Model approach, which makes full use of known information in the related research paper and introduces hyperparameter optimization techniques, to solve the CASH problem effectively. Auto-Model tremendously reduces the cost of algorithm implementations and hyperparameter configuration space, and thus capable of dealing with the CASH problem efficiently and easily. To demonstrate the benefit of Auto-Model, we compare it with classical Auto-Weka approach. The experimental results show that our proposed approach can provide superior results and achieves better performance in a short time.

cs.AI↗

arXiv2020

Complexity and Efficient Algorithms for Data Inconsistency Evaluating and Repairing

Data inconsistency evaluating and repairing are major concerns in data quality management. As the basic computing task, optimal subset repair is not only applied for cost estimation during the progress of database repairing, but also directly used to derive the evaluation of database inconsistency. Computing an optimal subset repair is to find a minimum tuple set from an inconsistent database whose remove results in a consistent subset left. Tight bound on the complexity and efficient algorithms are still unknown. In this paper, we improve the existing complexity and algorithmic results, together with a fast estimation on the size of optimal subset repair. We first strengthen the dichotomy for optimal subset repair computation problem, we show that it is not only APXcomplete, but also NPhard to approximate an optimal subset repair with a factor better than $17/16$ for most cases. We second show a $(2-0.5^{\tinyσ-1})$-approximation whenever given $σ$ functional dependencies, and a $(2-η_k+\frac{η_k}{k})$-approximation when an $η_k$-portion of tuples have the $k$-quasi-Tur$\acute{\text{a}}$n property for some $k>1$. We finally show a sublinear estimator on the size of optimal \textit{S}-repair for subset queries, it outputs an estimation of a ratio $2n+εn$ with a high probability, thus deriving an estimation of FD-inconsistency degree of a ratio $2+ε$. To support a variety of subset queries for FD-inconsistency evaluation, we unify them as the $\subseteq$-oracle which can answer membership-query, and return $p$ tuples uniformly sampled whenever given a number $p$. Experiments are conducted on range queries as an implementation of $\subseteq$-oracle, and results show the efficiency of our FD-inconsistency degree estimator.

cs.DB↗

arXiv2019

Recognizing the Tractability in Big Data Computing

Due to the limitation on computational power of existing computers, the polynomial time does not works for identifying the tractable problems in big data computing. This paper adopts the sublinear time as the new tractable standard to recognize the tractability in big data computing, and the random-access Turing machine is used as the computational model to characterize the problems that are tractable on big data. First, two pure-tractable classes are first proposed. One is the class $\mathrm{PL}$ consisting of the problems that can be solved in polylogarithmic time by a RATM. The another one is the class $\mathrm{ST}$ including all the problems that can be solved in sublinear time by a RATM. The structure of the two pure-tractable classes is deeply investigated and they are proved $\mathrm{PL^i} \subsetneq \mathrm{PL^{i+1}}$ and $\mathrm{PL} \subsetneq \mathrm{ST}$. Then, two pseudo-tractable classes, $\mathrm{PTR}$ and $\mathrm{PTE}$, are proposed. $\mathrm{PTR}$ consists of all the problems that can solved by a RATM in sublinear time after a PTIME preprocessing by reducing the size of input dataset. $\mathrm{PTE}$ includes all the problems that can solved by a RATM in sublinear time after a PTIME preprocessing by extending the size of input dataset. The relations among the two pseudo-tractable classes and other complexity classes are investigated and they are proved that $\mathrm{PT} \subseteq \mathrm{P}$, $\sqcap'\mathrm{T^0_Q} \subsetneq \mathrm{PTR^0_Q}$ and $\mathrm{PT_P} = \mathrm{P}$.

cs.CC↗

arXiv2019

Autoregressive-Model-Based Methods for Online Time Series Prediction with Missing Values: an Experimental Evaluation

Time series prediction with missing values is an important problem of time series analysis since complete data is usually hard to obtain in many real-world applications. To model the generation of time series, autoregressive (AR) model is a basic and widely used one, which assumes that each observation in the time series is a noisy linear combination of some previous observations along with a constant shift. To tackle the problem of prediction with missing values, a number of methods were proposed based on various data models. For real application scenarios, how do these methods perform over different types of time series with different levels of data missing remains to be investigated. In this paper, we focus on online methods for AR-model-based time series prediction with missing values. We adapted five mainstream methods to fit in such a scenario. We make detailed discussion on each of them by introducing their core ideas about how to estimate the AR coefficients and their different strategies to deal with missing values. We also present algorithmic implementations for better understanding. In order to comprehensively evaluate these methods and do the comparison, we conduct experiments with various configurations of relative parameters over both synthetic and real data. From the experimental results, we derived several noteworthy conclusions and shows that imputation is a simple but reliable strategy to handle missing values in online prediction tasks.

stat.ML↗

arXiv2019

A True $O(n \log{n}) $ Algorithm for the All-k-Nearest-Neighbors Problem

In this paper we examined an algorithm for the All-k-Nearest-Neighbor problem proposed in 1980s, which was claimed to have an $O(n\log{n})$ upper bound on the running time. We find the algorithm actually exceeds the so claimed upper bound, and prove that it has an $Ω(n^2)$ lower bound on the time complexity. Besides, we propose a new algorithm that truly achieves the $O(n\log{n})$ bound. Detailed and rigorous theoretical proofs are provided to show the proposed algorithm runs exactly in $$O(n\log{n})$$ time.

cs.CG↗

arXiv2019

Regular Expression Matching on billion-nodes Graphs

In many applications, it is necessary to retrieve pairs of vertices with the path between them satisfying certain constraints, since regular expression is a powerful tool to describe patterns of a sequence. To meet such requirements, in this paper, we define regular expression (RE) query on graphs to use regular expression to represent the constraints between vertices. To process RE queries on large graphs such as social networks, we propose the RE query processing method with the index size sublinear to the graph size. Considering that large graphs may be randomly distributed in multiple machines, the parallel RE processing algorithms are presented without the assumption of graph distribution. To achieve high efficiency for complex RE query processing, we develop cost-based query optimization strategies with only a small size statistical information which is suitable for querying large graphs. Comprehensive experimental results show that this approach works scale well for large graphs.

cs.DB↗

arXiv2019

An Iterative Scheme for Leverage-based Approximate Aggregation

The current data explosion poses great challenges to the approximate aggregation with an efficiency and accuracy. To address this problem, we propose a novel approach to calculate the aggregation answers with a high accuracy using only a small portion of the data. We introduce leverages to reflect individual differences in the samples from a statistical perspective. Two kinds of estimators, the leverage-based estimator, and the sketch estimator (a "rough picture" of the aggregation answer), are in constraint relations and iteratively improved according to the actual conditions until their difference is below a threshold. Due to the iteration mechanism and the leverages, our approach achieves a high accuracy. Moreover, some features, such as not requiring recording the sampled data and easy to extend to various execution modes (e.g., the online mode), make our approach well suited to deal with big data. Experiments show that our approach has an extraordinary performance, and when compared with the uniform sampling, our approach can achieve high-quality answers with only 1/3 of the same sample size.

cs.DB↗

arXiv2018

An Algorithm for Reducing Approximate Nearest Neighbor to Approximate Near Neighbor with O(logn) Query Time

This paper proposes a new algorithm for reducing Approximate Nearest Neighbor problem to Approximate Near Neighbor problem. The advantage of this algorithm is that it achieves O(log n) query time. As a reduction problem, the uery time complexity is the times of invoking the algorithm for Approximate Near Neighbor problem. All former algorithms for the same reduction need polylog(n) query time. A box split method proposed by Vaidya is used in our paper to achieve the O(log n) query time complexity.

cs.DS↗

arXiv2018

Schema Integration on Massive Data Sources

As the fundamental phrase of collecting and analyzing data, data integration is used in many applications, such as data cleaning, bioinformatics and pattern recognition. In big data era, one of the major problems of data integration is to obtain the global schema of data sources since the global schema could be hardly derived from massive data sources directly. In this paper, we attempt to solve such schema integration problem. For different scenarios, we develop batch and incremental schema integration algorithms. We consider the representation difference of attribute names in various data sources and propose ED Join and Semantic Join algorithms to integrate attributes with different representations. Extensive experimental results demonstrate that the proposed algorithms could integrate schemas efficiently and effectively.

cs.DB↗

arXiv2018

Mining CFD Rules on Big Data

Current conditional functional dependencies (CFDs) discovery algorithms always need a well-prepared training data set. This makes them difficult to be applied on large datasets which are always in low-quality. To handle the volume issue of big data, we develop the sampling algorithms to obtain a small representative training set. For the low-quality issue of big data, we then design the fault-tolerant rule discovery algorithm and the conflict resolution algorithm. We also propose parameter selection strategy for CFD discovery algorithm to ensure its effectiveness. Experimental results demonstrate that our method could discover effective CFD rules on billion-tuple data within reasonable time.

cs.DB↗

arXiv2018

On the Fairness of Quality-based Data Markets

For data pricing, data quality is a factor that must be considered. To keep the fairness of data market from the aspect of data quality, we proposed a fair data market that considers data quality while pricing. To ensure fairness, we first design a quality-driven data pricing strategy. Then based on the strategy, a fairness assurance mechanism for quality-driven data marketplace is proposed. In this mechanism, we ensure that savvy consumers cannot cheat the system and users can verify each consumption with Trusted Third Party (TTP) that they are charged properly. Based on this mechanism, we develop a fair quality-driven data market system. Extensive experiments are performed to verify the effectiveness of proposed techniques. Experimental results show that our quality-driven data pricing strategy could assign a reasonable price to the data according to data quality and the fairness assurance mechanism could effectively protect quality-driven data pricing from potential cheating.

cs.DB↗

arXiv2018

Diversification on Big Data in Query Processing

Recently, in the area of big data, some popular applications such as web search engines and recommendation systems, face the problem to diversify results during query processing. In this sense, it is both significant and essential to propose methods to deal with big data in order to increase the diversity of the result set. In this paper, we firstly define a set's diversity and an element's ability to improve the set's overall diversity. Based on these definitions, we propose a diversification framework which has good performance in terms of effectiveness and efficiency. Also, this framework has theoretical guarantee on probability of success. Secondly, we design implementation algorithms based on this framework for both numerical and string data. Thirdly, for numerical and string data respectively, we carry out extensive experiments on real data to verify the performance of our proposed framework, and also perform scalability experiments on synthetic data.

cs.DB↗

arXiv2018

Improve3C: Data Cleaning on Consistency and Completeness with Currency

Data quality plays a key role in big data management today. With the explosive growth of data from a variety of sources, the quality of data is faced with multiple problems. Motivated by this, we study the multiple data quality improvement on completeness, consistency and currency in this paper. For the proposed problem, we introduce a 4-step framework, named Improve3C, for detection and quality improvement on incomplete and inconsistent data without timestamps. We compute and achieve a relative currency order among records derived from given currency constraints, according to which inconsistent and incomplete data can be repaired effectively considering the temporal impact. For both effectiveness and efficiency consideration, we carry out inconsistent repair ahead of incomplete repair. Currency-related consistency distance is defined to measure the similarity between dirty records and clean ones more accurately. In addition, currency orders are treated as an important feature in the training process of incompleteness repair. The solution algorithms are introduced in detail with examples. A thorough experiment on one real-life data and a synthetic one verifies that the proposed method can improve the performance of dirty data cleaning with multiple quality problems which are hard to be cleaned by the existing approaches effectively.

cs.DB↗