arXiv subjects

Christoph Koch

Publications and source records attributed to Christoph Koch.

At least 55 records · Page 3Linked to original sources

arXiv2015

Solving the Join Ordering Problem via Mixed Integer Linear Programming

We transform join ordering into a mixed integer linear program (MILP). This allows to address query optimization by mature MILP solver implementations that have evolved over decades and steadily improved their performance. They offer features such as anytime optimization and parallel search that are highly relevant for query optimization. We present a MILP formulation for searching left-deep query plans. We use sets of binary variables to represent join operands and intermediate results, operator implementation choices or the presence of interesting orders. Linear constraints restrict value assignments to the ones representing valid query plans. We approximate the cost of scan and join operations via linear functions, allowing to increase approximation precision up to arbitrary degrees. Our experimental results are encouraging: we are able to find optimal plans for joins between 60 tables; a query size that is beyond the capabilities of prior exhaustive query optimization methods.

cs.DB↗

arXiv2015

Parallelizing Query Optimization on Shared-Nothing Architectures

Data processing systems offer an ever increasing degree of parallelism on the levels of cores, CPUs, and processing nodes. Query optimization must exploit high degrees of parallelism in order not to gradually become the bottleneck of query evaluation. We show how to parallelize query optimization at a massive scale. We present algorithms for parallel query optimization in left-deep and bushy plan spaces. At optimization start, we divide the plan space for a given query into partitions of equal size that are explored in parallel by worker nodes. At the end of optimization, each worker returns the optimal plan in its partition to the master which determines the globally optimal plan from the partition-optimal plans. No synchronization or data exchange is required during the actual optimization phase. The amount of data sent over the network, at the start and at the end of optimization, as well as the complexity of serial steps within our algorithms increase only linearly in the number of workers and in the query size. The time and space complexity of optimization within one partition decreases uniformly in the number of workers. We parallelize single- and multi-objective query optimization over a cluster with 100 nodes in our experiments, using more than 250 concurrent worker threads (Spark executors). Despite high network latency and task assignment overheads, parallelization yields speedups of up to one order of magnitude for large queries whose optimization takes minutes on a single node.

cs.DB↗

arXiv2015

Probably Approximately Optimal Query Optimization

Evaluating query predicates on data samples is the only way to estimate their selectivity in certain scenarios. Finding a guaranteed optimal query plan is not a reasonable optimization goal in those cases as it might require an infinite number of samples. We therefore introduce probably approximately optimal query optimization (PAO) where the goal is to find a query plan whose cost is near-optimal with a certain probability. We will justify why PAO is a suitable formalism to model scenarios in which predicate sampling and optimization need to be interleaved. We present the first algorithm for PAO. Our algorithm is non-intrusive and uses standard query optimizers and sampling components as sub-functions. It is generic and can be applied to a wide range of scenarios. Our algorithm is iterative and calculates in each iteration a query plan together with a region in the selectivity space where the plan has near-optimal cost. It determines the confidence that the true selectivity values fall within the aforementioned region and chooses the next samples to take based on the current state if the confidence does not reach the threshold specified as problem input. We devise different algorithm variants and analyze their complexity. We experimentally compare them in terms of the number of optimizer invocations, samples, and iterations over many different query classes.

cs.DB↗

arXiv2015

Multiple Query Optimization on the D-Wave 2X Adiabatic Quantum Computer

The D-Wave adiabatic quantum annealer solves hard combinatorial optimization problems leveraging quantum physics. The newest version features over 1000 qubits and was released in August 2015. We were given access to such a machine, currently hosted at NASA Ames Research Center in California, to explore the potential for hard optimization problems that arise in the context of databases. In this paper, we tackle the problem of multiple query optimization (MQO). We show how an MQO problem instance can be transformed into a mathematical formula that complies with the restrictive input format accepted by the quantum annealer. This formula is translated into weights on and between qubits such that the configuration minimizing the input formula can be found via a process called adiabatic quantum annealing. We analyze the asymptotic growth rate of the number of required qubits in the MQO problem dimensions as the number of qubits is currently the main factor restricting applicability. We experimentally compare the performance of the quantum annealer against other MQO algorithms executed on a traditional computer. While the problem sizes that can be treated are currently limited, we already find a class of problem instances where the quantum annealer is three orders of magnitude faster than other approaches.

cs.DB↗

arXiv2015

The phase transition in the multi-type binomial random graph $G(\mathbf{n},P)$

We determine the asymptotic size of the largest component in the $2$-type binomial random graph $G(\mathbf{n},P)$ near criticality using a refined branching process approach. In $G(\mathbf{n},P)$ every vertex has one of two types, the vector $\mathbf{n}$ describes the number of vertices of each type, and any edge $\{u,v\}$ is present independently with a probability that is given by an entry of the probability matrix $P$ according to the types of $u$ and $v.$ We prove that in the weakly supercritical regime, i.e. if the distance to the critical point of the phase transition is given by an $\varepsilon=\varepsilon(\mathbf{n})\to0,$ with probability $1-o(1),$ the largest component in $G(\mathbf{n},P)$ contains asymptotically $2\varepsilon \|\mathbf{n}\|_1$ vertices and all other components are of size $o(\varepsilon \|\mathbf{n}\|_1).$

math.PR↗

arXiv2015

Threshold and hitting time for high-order connectivity in random hypergraphs

We consider the following definition of connectivity in $k$-uniform hypergraphs: Two $j$-sets are $j$-connected if there is a walk of edges between them such that two consecutive edges intersect in at least $j$ vertices. We determine the threshold at which the random $k$-uniform hypergraph with edge probability $p$ becomes $j$-connected with high probability. We also deduce a hitting time result for the random hypergraph process -- the hypergraph becomes $j$-connected at exactly the moment when the last isolated $j$-set disappears. This generalises well-known results for graphs.

math.CO↗

arXiv2015

Properties of stochastic Kronecker graphs

The stochastic Kronecker graph model introduced by Leskovec et al. is a random graph with vertex set $\mathbb Z_2^n$, where two vertices $u$ and $v$ are connected with probability $α^{{u}\cdot{v}}γ^{(1-{u})\cdot(1-{v})}β^{n-{u}\cdot{v}-(1-{u})\cdot(1-{v})}$ independently of the presence or absence of any other edge, for fixed parameters $0<α,β,γ<1$. They have shown empirically that the degree sequence resembles a power law degree distribution. In this paper we show that the stochastic Kronecker graph a.a.s. does not feature a power law degree distribution for any parameters $0<α,β,γ<1$. In addition, we analyze the number of subgraphs present in the stochastic Kronecker graph and study the typical neighborhood of any given vertex.

math.CO↗

arXiv2015

The size of the giant component in random hypergraphs

The phase transition in the size of the giant component in random graphs is one of the most well-studied phenomena in random graph theory. For hypergraphs, there are many possible generalisations of the notion of a component, and for all but the simplest example, the phase transition phenomenon was first proved by Cooley, Kang and Person. In this paper we build on this and determine the asymptotic size of the unique giant component.

math.CO↗

arXiv2015

The Homeostasis Protocol: Avoiding Transaction Coordination Through Program Analysis

Datastores today rely on distribution and replication to achieve improved performance and fault-tolerance. But correctness of many applications depends on strong consistency properties - something that can impose substantial overheads, since it requires coordinating the behavior of multiple nodes. This paper describes a new approach to achieving strong consistency in distributed systems while minimizing communication between nodes. The key insight is to allow the state of the system to be inconsistent during execution, as long as this inconsistency is bounded and does not affect transaction correctness. In contrast to previous work, our approach uses program analysis to extract semantic information about permissible levels of inconsistency and is fully automated. We then employ a novel homeostasis protocol to allow sites to operate independently, without communicating, as long as any inconsistency is governed by appropriate treaties between the nodes. We discuss mechanisms for optimizing treaties based on workload characteristics to minimize communication, as well as a prototype implementation and experiments that demonstrate the benefits of our approach on common transactional benchmarks.

cs.DB↗

arXiv2014

LINVIEW: Incremental View Maintenance for Complex Analytical Queries

Many analytics tasks and machine learning problems can be naturally expressed by iterative linear algebra programs. In this paper, we study the incremental view maintenance problem for such complex analytical queries. We develop a framework, called LINVIEW, for capturing deltas of linear algebra programs and understanding their computational cost. Linear algebra operations tend to cause an avalanche effect where even very local changes to the input matrices spread out and infect all of the intermediate results and the final view, causing incremental view maintenance to lose its performance benefit over re-evaluation. We develop techniques based on matrix factorizations to contain such epidemics of change. As a consequence, our techniques make incremental view maintenance of linear algebra practical and usually substantially cheaper than re-evaluation. We show, both analytically and experimentally, the usefulness of these techniques when applied to standard analytics tasks. Our evaluation demonstrates the efficiency of LINVIEW in generating parallel incremental programs that outperform re-evaluation techniques by more than an order of magnitude.

cs.DB↗

arXiv2014

Approximation Schemes for Many-Objective Query Optimization

The goal of multi-objective query optimization (MOQO) is to find query plans that realize a good compromise between conflicting objectives such as minimizing execution time and minimizing monetary fees in a Cloud scenario. A previously proposed exhaustive MOQO algorithm needs hours to optimize even simple TPC-H queries. This is why we propose several approximation schemes for MOQO that generate guaranteed near-optimal plans in seconds where exhaustive optimization takes hours. We integrated all MOQO algorithms into the Postgres optimizer and present experimental results for TPC-H queries; we extended the Postgres cost model and optimize for up to nine conflicting objectives in our experiments. The proposed algorithms are based on a formal analysis of typical cost functions that occur in the context of MOQO. We identify properties that hold for a broad range of objectives and can be exploited for the design of future MOQO algorithms.

cs.DB↗

arXiv2013

Sparse projections onto the simplex

Most learning methods with rank or sparsity constraints use convex relaxations, which lead to optimization with the nuclear norm or the $\ell_1$-norm. However, several important learning applications cannot benefit from this approach as they feature these convex norms as constraints in addition to the non-convex rank and sparsity constraints. In this setting, we derive efficient sparse projections onto the simplex and its extension, and illustrate how to use them to solve high-dimensional learning problems in quantum tomography, sparse density estimation and portfolio selection with non-convex constraints.

cs.LG↗

arXiv2012

DBToaster: Higher-order Delta Processing for Dynamic, Frequently Fresh Views

Applications ranging from algorithmic trading to scientific data analysis require realtime analytics based on views over databases that change at very high rates. Such views have to be kept fresh at low maintenance cost and latencies. At the same time, these views have to support classical SQL, rather than window semantics, to enable applications that combine current with aged or historical data. In this paper, we present viewlet transforms, a recursive finite differencing technique applied to queries. The viewlet transform materializes a query and a set of its higher-order deltas as views. These views support each other's incremental maintenance, leading to a reduced overall view maintenance cost. The viewlet transform of a query admits efficient evaluation, the elimination of certain expensive query operations, and aggressive parallelization. We develop viewlet transforms into a workable query execution technique, present a heuristic and cost-based optimization framework, and report on experiments with a prototype dynamic data management system that combines viewlet transforms with an optimizing compilation technique. The system supports tens of thousands of complete view refreshes a second for a wide range of queries.

cs.DB↗

arXiv2009

Cooperative Update Exchange in the Youtopia System

Youtopia is a platform for collaborative management and integration of relational data. At the heart of Youtopia is an update exchange abstraction: changes to the data propagate through the system to satisfy user-specified mappings. We present a novel change propagation model that combines a deterministic chase with human intervention. The process is fundamentally cooperative and gives users significant control over how mappings are repaired. An additional advantage of our model is that mapping cycles can be permitted without compromising correctness. We investigate potential harmful interference between updates in our model; we introduce two appropriate notions of serializability that avoid such interference if enforced. The first is very general and related to classical final-state serializability; the second is more restrictive but highly practical and related to conflict-serializability. We present an algorithm to enforce the latter notion. Our algorithm is an optimistic one, and as such may sometimes require updates to be aborted. We develop techniques for reducing the number of aborts and we test these experimentally.

cs.DB↗

arXiv2008

Dynamic Approaches to In-Network Aggregation

Collaboration between small-scale wireless devices hinges on their ability to infer properties shared across multiple nearby nodes. Wireless-enabled mobile devices in particular create a highly dynamic environment not conducive to distributed reasoning about such global properties. This paper addresses a specific instance of this problem: distributed aggregation. We present extensions to existing unstructured aggregation protocols that enable estimation of count, sum, and average aggregates in highly dynamic environments. With the modified protocols, devices with only limited connectivity can maintain estimates of the aggregate, despite \textit{unexpected} peer departures and arrivals. Our analysis of these aggregate maintenance extensions demonstrates their effectiveness in unstructured environments despite high levels of node mobility.

cs.DC↗

arXiv2008

A Compositional Query Algebra for Second-Order Logic and Uncertain Databases

World-set algebra is a variable-free query language for uncertain databases. It constitutes the core of the query language implemented in MayBMS, an uncertain database system. This paper shows that world-set algebra captures exactly second-order logic over finite structures, or equivalently, the polynomial hierarchy. The proofs also imply that world-set algebra is closed under composition, a previously open problem.

cs.DB↗

arXiv2008

Conditioning Probabilistic Databases

Past research on probabilistic databases has studied the problem of answering queries on a static database. Application scenarios of probabilistic databases however often involve the conditioning of a database using additional information in the form of new evidence. The conditioning problem is thus to transform a probabilistic database of priors into a posterior probabilistic database which is materialized for subsequent query processing or further refinement. It turns out that the conditioning problem is closely related to the problem of computing exact tuple confidence values. It is known that exact confidence computation is an NP-hard problem. This has led researchers to consider approximation techniques for confidence computation. However, neither conditioning nor exact confidence computation can be solved using such techniques. In this paper we present efficient techniques for both problems. We study several problem decomposition methods and heuristics that are based on the most successful search techniques from constraint satisfaction, such as the Davis-Putnam algorithm. We complement this with a thorough experimental evaluation of the algorithms proposed. Our experiments show that our exact algorithms scale well to realistic database sizes and can in some scenarios compete with the most efficient previous approximation algorithms.

cs.DB↗

arXiv2008

10^(10^6) Worlds and Beyond: Efficient Representation and Processing of Incomplete Information

Current systems and formalisms for representing incomplete information generally suffer from at least one of two weaknesses. Either they are not strong enough for representing results of simple queries, or the handling and processing of the data, e.g. for query evaluation, is intractable. In this paper, we present a decomposition-based approach to addressing this problem. We introduce world-set decompositions (WSDs), a space-efficient formalism for representing any finite set of possible worlds over relational databases. WSDs are therefore a strong representation system for any relational query language. We study the problem of efficiently evaluating relational algebra queries on sets of worlds represented by WSDs. We also evaluate our technique experimentally in a large census data scenario and show that it is both scalable and efficient.

cs.DB↗