arXiv subjects

Travis Gagie

Publications and source records attributed to Travis Gagie.

At least 127 records · Page 7Linked to original sources

arXiv2012

Grammar-Based Construction of Indexes for Binary Jumbled Pattern Matching

We show how, given a straight-line program with $g$ rules for a binary string $B$ of length $n$, in $O(g^{2 / 3} n^{4 / 3})$ time we can build a linear-space index such that, given $m$ and $c$, in O(1) time we can determine whether there is a substring of $B$ with length $m$ containing exactly $c$ copies of 1. If we use $O(n \log n)$ space for the index, then we can list all such substrings using $O(m)$ time per substring.

cs.DS↗

arXiv2012

Faster Approximate Pattern Matching in Compressed Repetitive Texts

Motivated by the imminent growth of massive, highly redundant genomic databases, we study the problem of compressing a string database while simultaneously supporting fast random access, substring extraction and pattern matching to the underlying string(s). Bille et al. (2011) recently showed how, given a straight-line program with $r$ rules for a string $s$ of length $n$, we can build an $\Oh{r}$-word data structure that allows us to extract any substring of length $m$ in $\Oh{\log n + m}$ time. They also showed how, given a pattern $p$ of length $m$ and an edit distance (k \leq m), their data structure supports finding all \occ approximate matches to $p$ in $s$ in $\Oh{r (\min (m k, k^4 + m) + \log n) + \occ}$ time. Rytter (2003) and Charikar et al. (2005) showed that $r$ is always at least the number $z$ of phrases in the LZ77 parse of $s$, and gave algorithms for building straight-line programs with $\Oh{z \log n}$ rules. In this paper we give a simple $\Oh{z \log n}$-word data structure that takes the same time for substring extraction but only $\Oh{z \min (m k, k^4 + m) + \occ}$ time for approximate pattern matching.

cs.DS↗

arXiv2012

A Faster Grammar-Based Self-Index

To store and search genomic databases efficiently, researchers have recently started building compressed self-indexes based on grammars. In this paper we show how, given a straight-line program with $r$ rules for a string (S [1..n]) whose LZ77 parse consists of $z$ phrases, we can store a self-index for $S$ in $\Oh{r + z \log \log n}$ space such that, given a pattern (P [1..m]), we can list the $\occ$ occurrences of $P$ in $S$ in $\Oh{m^2 + \occ \log \log n}$ time. If the straight-line program is balanced and we accept a small probability of building a faulty index, then we can reduce the $\Oh{m^2}$ term to $\Oh{m \log m}$. All previous self-indexes are larger or slower in the worst case.

cs.DS↗

arXiv2012

Sequential-Access FM-Indexes

Previous authors have shown how to build FM-indexes efficiently in external memory, but querying them efficiently remains an open problem. Searching naïvely for a pattern $P$ requires (Θ(|P|)) random access. In this paper we show how, by storing a few small auxiliary tables, we can access data only in the order in which they appear on disk, which should be faster.

cs.DS↗

arXiv2012

On the Value of Multiple Read/Write Streams for Data Compression

We study whether, when restricted to using polylogarithmic memory and polylogarithmic passes, we can achieve qualitatively better data compression with multiple read/write streams than we can with only one. We first show how we can achieve universal compression using only one pass over one stream. We then show that one stream is not sufficient for us to achieve good grammar-based compression. Finally, we show that two streams are necessary and sufficient for us to achieve entropy-only bounds.

cs.DS↗

arXiv2012

Efficient Fully-Compressed Sequence Representations

We present a data structure that stores a sequence $s[1..n]$ over alphabet $[1..σ]$ in $n\Ho(s) + o(n)(\Ho(s){+}1)$ bits, where $\Ho(s)$ is the zero-order entropy of $s$. This structure supports the queries \access, \rank\ and \select, which are fundamental building blocks for many other compressed data structures, in worst-case time $\Oh{\lg\lgσ}$ and average time $\Oh{\lg \Ho(s)}$. The worst-case complexity matches the best previous results, yet these had been achieved with data structures using $n\Ho(s)+o(n\lgσ)$ bits. On highly compressible sequences the $o(n\lgσ)$ bits of the redundancy may be significant compared to the the $n\Ho(s)$ bits that encode the data. Our representation, instead, compresses the redundancy as well. Moreover, our average-case complexity is unprecedented. Our technique is based on partitioning the alphabet into characters of similar frequency. The subsequence corresponding to each group can then be encoded using fast uncompressed representations without harming the overall compression ratios, even in the redundancy. The result also improves upon the best current compressed representations of several other data structures. For example, we achieve $(i)$ compressed redundancy, retaining the best time complexities, for the smallest existing full-text self-indexes; $(ii)$ compressed permutations $π$ with times for $π()$ and $\pii()$ improved to loglogarithmic; and $(iii)$ the first compressed representation of dynamic collections of disjoint sets. We also point out various applications to inverted indexes, suffix arrays, binary relations, and data compressors. ...

cs.DS↗

arXiv2012

Linear-Space Substring Range Counting over Polylogarithmic Alphabets

Bille and Gørtz (2011) recently introduced the problem of substring range counting, for which we are asked to store compactly a string $S$ of $n$ characters with integer labels in ([0, u]), such that later, given an interval ([a, b]) and a pattern $P$ of length $m$, we can quickly count the occurrences of $P$ whose first characters' labels are in ([a, b]). They showed how to store $S$ in $\Oh{n \log n / \log \log n}$ space and answer queries in $\Oh{m + \log \log u}$ time. We show that, if $S$ is over an alphabet of size (\polylog (n)), then we can achieve optimal linear space. Moreover, if (u = n \polylog (n)), then we can also reduce the time to $\Oh{m}$. Our results give linear space and time bounds for position-restricted substring counting and the counting versions of indexing substrings with intervals, indexing substrings with gaps and aligned pattern matching.

cs.DS↗

arXiv2011

A Compressed Self-Index for Genomic Databases

Advances in DNA sequencing technology will soon result in databases of thousands of genomes. Within a species, individuals' genomes are almost exact copies of each other; e.g., any two human genomes are 99.9% the same. Relative Lempel-Ziv (RLZ) compression takes advantage of this property: it stores the first genome uncompressed or as an FM-index, then compresses the other genomes with a variant of LZ77 that copies phrases only from the first genome. RLZ achieves good compression and supports fast random access; in this paper we show how to support fast search as well, thus obtaining an efficient compressed self-index.

cs.DS↗

arXiv2011

Pattern Kits

Suppose we have just performed searches in a self-index for two patterns $A$ and $B$ and now we want to search for their concatenation \A B); how can we best make use of our previous computations? In this paper we consider this problem and, more generally, how we can store a dynamic library of patterns that we can easily manipulate in interesting ways. We give a space- and time-efficient data structure for this problem that is compatible with many of the best self-indexes.

cs.DS↗

arXiv2010

Bounds from a Card Trick

We describe a new variation of a mathematical card trick, whose analysis leads to new lower bounds for data compression and estimating the entropy of Markov sources.

cs.IT↗

arXiv2010

New Algorithms on Wavelet Trees and Applications to Information Retrieval

Wavelet trees are widely used in the representation of sequences, permutations, text collections, binary relations, discrete points, and other succinct data structures. We show, however, that this still falls short of exploiting all of the virtues of this versatile data structure. In particular we show how to use wavelet trees to solve fundamental algorithmic problems such as {\em range quantile} queries, {\em range next value} queries, and {\em range intersection} queries. We explore several applications of these queries in Information Retrieval, in particular {\em document retrieval} in hierarchical and temporal documents, and in the representation of {\em inverted lists}.

cs.DS↗

arXiv2010

Counting Colours in Compressed Strings

Suppose we are asked to preprocess a string $s [1..n]$ such that later, given a substring's endpoints, we can quickly count how many distinct characters it contains. In this paper we give a data structure for this problem that takes $n H_0 (s) + \Oh{n} + \oh{n H_0 (s)}$ bits, where $H_0 (s)$ is the 0th-order empirical entropy of $s$, and answers queries in $\Oh{\log^{1 + ε} n}$ time for any constant $ε> 0$. We also show how our data structure can be made partially dynamic.

cs.DS↗

arXiv2010

Competitive Boolean Function Evaluation: Beyond Monotonicity, and the Symmetric Case

We study the extremal competitive ratio of Boolean function evaluation. We provide the first non-trivial lower and upper bounds for classes of Boolean functions which are not included in the class of monotone Boolean functions. For the particular case of symmetric functions our bounds are matching and we exactly characterize the best possible competitiveness achievable by a deterministic algorithm. Our upper bound is obtained by a simple polynomial time algorithm.

cs.DS↗

arXiv2010

Range Quantile Queries: Another Virtue of Wavelet Trees

We show how to use a balanced wavelet tree as a data structure that stores a list of numbers and supports efficient {\em range quantile queries}. A range quantile query takes a rank and the endpoints of a sublist and returns the number with that rank in that sublist. For example, if the rank is half the sublist's length, then the query returns the sublist's median. We also show how these queries can be used to support space-efficient {\em coloured range reporting} and {\em document listing}.

cs.DS↗

arXiv2010

Grammar-Based Compression in a Streaming Model

We show that, given a string $s$ of length $n$, with constant memory and logarithmic passes over a constant number of streams we can build a context-free grammar that generates $s$ and only $s$ and whose size is within an $\Oh{\min (g \log g, \sqrt{n \log g})}$-factor of the minimum $g$. This stands in contrast to our previous result that, with polylogarithmic memory and polylogarithmic passes over a single stream, we cannot build such a grammar whose size is within any polynomial of $g$.

cs.DS↗

arXiv2009

A Lower Bound on the Complexity of Approximating the Entropy of a Markov Source

Suppose that, for any (k \geq 1), (ε> 0) and sufficiently large $σ$, we are given a black box that allows us to sample characters from a $k$th-order Markov source over the alphabet (\{0, ..., σ- 1\}). Even if we know the source has entropy either 0 or at least (\log (σ- k)), there is still no algorithm that, with probability bounded away from (1 / 2), guesses the entropy correctly after sampling at most ((σ- k)^{k / 2 - ε}) characters.

cs.IT↗

arXiv2009

Lightweight Data Indexing and Compression in External Memory

In this paper we describe algorithms for computing the BWT and for building (compressed) indexes in external memory. The innovative feature of our algorithms is that they are lightweight in the sense that, for an input of size $n$, they use only ${n}$ bits of disk working space while all previous approaches use $\Th{n \log n}$ bits of disk working space. Moreover, our algorithms access disk data only via sequential scans, thus they take full advantage of modern disk features that make sequential disk accesses much faster than random accesses. We also present a scan-based algorithm for inverting the BWT that uses $\Th{n}$ bits of working space, and a lightweight {\em internal-memory} algorithm for computing the BWT which is the fastest in the literature when the available working space is $\os{n}$ bits. Finally, we prove {\em lower} bounds on the complexity of computing and inverting the BWT via sequential scans in terms of the classic product: internal-memory space $\times$ number of passes over the disk data.

cs.DS↗

arXiv2009

Tight Bounds for Online Stable Sorting

Although many authors have considered how many ternary comparisons it takes to sort a multiset $S$ of size $n$, the best known upper and lower bounds still differ by a term linear in $n$. In this paper we restrict our attention to online stable sorting and prove upper and lower bounds that are within (o (n)) not only of each other but also of the best known upper bound for offline sorting. Specifically, we first prove that if the number of distinct elements (σ= o (n / \log n)), then ((H + 1) n + o (n)) comparisons are sufficient, where $H$ is the entropy of the distribution of the elements in $S$. We then give a simple proof that ((H + 1) n - o (n)) comparisons are necessary in the worst case.

cs.DS↗