arXiv subjects

Ioannis Kontoyiannis

Publications and source records attributed to Ioannis Kontoyiannis.

67 records · Page 4Linked to original sources

arXiv2009

Lossy Compression in Near-Linear Time via Efficient Random Codebooks and Databases

The compression-complexity trade-off of lossy compression algorithms that are based on a random codebook or a random database is examined. Motivated, in part, by recent results of Gupta-Verdú-Weissman (GVW) and their underlying connections with the pattern-matching scheme of Kontoyiannis' lossy Lempel-Ziv algorithm, we introduce a non-universal version of the lossy Lempel-Ziv method (termed LLZ). The optimality of LLZ for memoryless sources is established, and its performance is compared to that of the GVW divide-and-conquer approach. Experimental results indicate that the GVW approach often yields better compression than LLZ, but at the price of much higher memory requirements. To combine the advantages of both, we introduce a hybrid algorithm (HYB) that utilizes both the divide-and-conquer idea of GVW and the single-database structure of LLZ. It is proved that HYB shares with GVW the exact same rate-distortion performance and implementation complexity, while, like LLZ, requiring less memory, by a factor which may become unbounded, depending on the choice or the relevant design parameters. Experimental results are also presented, illustrating the performance of all three methods on data generated by simple discrete memoryless sources. In particular, the HYB algorithm is shown to outperform existing schemes for the compression of some simple discrete sources with respect to the Hamming distortion criterion.

cs.IT↗

arXiv2008

Computable exponential bounds for screened estimation and simulation

Suppose the expectation $E(F(X))$ is to be estimated by the empirical averages of the values of $F$ on independent and identically distributed samples $\{X_i\}$. A sampling rule called the "screened" estimator is introduced, and its performance is studied. When the mean $E(U(X))$ of a different function $U$ is known, the estimates are "screened," in that we only consider those which correspond to times when the empirical average of the $\{U(X_i)\}$ is sufficiently close to its known mean. As long as $U$ dominates $F$ appropriately, the screened estimates admit exponential error bounds, even when $F(X)$ is heavy-tailed. The main results are several nonasymptotic, explicit exponential bounds for the screened estimates. A geometric interpretation, in the spirit of Sanov's theorem, is given for the fact that the screened estimates always admit exponential error bounds, even if the standard estimates do not. And when they do, the screened estimates' error probability has a significantly better exponent. This implies that screening can be interpreted as a variance reduction technique. Our main mathematical tools come from large deviations techniques. The results are illustrated by a detailed simulation example.

math.PR↗

arXiv2008

On the entropy and log-concavity of compound Poisson measures

Motivated, in part, by the desire to develop an information-theoretic foundation for compound Poisson approximation limit theorems (analogous to the corresponding developments for the central limit theorem and for simple Poisson approximation), this work examines sufficient conditions under which the compound Poisson distribution has maximal entropy within a natural class of probability measures on the nonnegative integers. We show that the natural analog of the Poisson maximum entropy property remains valid if the measures under consideration are log-concave, but that it fails in general. A parallel maximum entropy result is established for the family of compound binomial measures. The proofs are largely based on ideas related to the semigroup approach introduced in recent work by Johnson for the Poisson family. Sufficient conditions are given for compound distributions to be log-concave, and specific examples are presented illustrating all the above results.

cs.IT↗

arXiv2008

From the entropy to the statistical structure of spike trains

We use statistical estimates of the entropy rate of spike train data in order to make inferences about the underlying structure of the spike train itself. We first examine a number of different parametric and nonparametric estimators (some known and some new), including the ``plug-in'' method, several versions of Lempel-Ziv-based compression algorithms, a maximum likelihood estimator tailored to renewal processes, and the natural estimator derived from the Context-Tree Weighting method (CTW). The theoretical properties of these estimators are examined, several new theoretical results are developed, and all estimators are systematically applied to various types of synthetic data and under different conditions. Our main focus is on the performance of these entropy estimators on the (binary) spike trains of 28 neurons recorded simultaneously for a one-hour period from the primary motor and dorsal premotor cortices of a monkey. We show how the entropy estimates can be used to test for the existence of long-term structure in the data, and we construct a hypothesis test for whether the renewal process model is appropriate for these spike trains. Further, by applying the CTW algorithm we derive the maximum a posterior (MAP) tree model of our empirical data, and comment on the underlying structure it reveals.

q-bio.NC↗

arXiv2007

Some information-theoretic computations related to the distribution of prime numbers

We illustrate how elementary information-theoretic ideas may be employed to provide proofs for well-known, nontrivial results in number theory. Specifically, we give an elementary and fairly short proof of the following asymptotic result: The sum of (log p)/p, taken over all primes p not exceeding n, is asymptotic to log n as n tends to infinity. We also give finite-n bounds refining the above limit. This result, originally proved by Chebyshev in 1852, is closely related to the celebrated prime number theorem.

cs.IT↗

arXiv2005

Mismatched codebooks and the role of entropy-coding in lossy data compression

We introduce a universal quantization scheme based on random coding, and we analyze its performance. This scheme consists of a source-independent random codebook (typically_mismatched_ to the source distribution), followed by optimal entropy-coding that is_matched_ to the quantized codeword distribution. A single-letter formula is derived for the rate achieved by this scheme at a given distortion, in the limit of large codebook dimension. The rate reduction due to entropy-coding is quantified, and it is shown that it can be arbitrarily large. In the special case of "almost uniform" codebooks (e.g., an i.i.d. Gaussian codebook with large variance) and difference distortion measures, a novel connection is drawn between the compression achieved by the present scheme and the performance of "universal" entropy-coded dithered lattice quantizers. This connection generalizes the "half-a-bit" bound on the redundancy of dithered lattice quantizers. Moreover, it demonstrates a strong notion of universality where a single "almost uniform" codebook is near-optimal for_any_ source and_any_ difference distortion measure.

cs.IT↗

arXiv2005

Large deviations asymptotics and the spectral theory of multiplicatively regular Markov processes

We continue the investigation of the spectral theory and exponential asymptotics of Markov processes, following Kontoyiannis and Meyn (2003). We introduce a new family of nonlinear Lyapunov drift criteria, characterizing distinct subclasses of geometrically ergodic Markov processes in terms of inequalities for the nonlinear generator. We concentrate on the class of "multiplicatively regular" Markov processes, characterized via conditions similar to (but weaker than) those of Donsker-Varadhan. For any such process {Phi(t)} with transition kernel P on a general state space, the following are obtained. 1. SPECTRAL THEORY: For a large class of functionals F, the kernel Phat(x,dy) = e^{F(x)}P(x,dy) has a discrete spectrum in an appropriately defined Banach space. There exists a "maximal" solution to the "multiplicative Poisson equation," defined as the eigenvalue problem for Phat. Regularity properties are established for Λ(F) = \log(λ), where λis the maximal eigenvalue, and for its convex dual. 2. MULTIPLICATIVE MEAN ERGODIC THEOREM: The normalized mean E_x[\exp(S_t)] of the exponential of the partial sums {S_t} of the process with respect to any one of the above functionals F, converges to the maximal eigenfunction. 3. MULTIPLICATIVE REGULARITY: The drift criterion under which our results are derived is equivalent to the existence of regeneration times with finite exponential moments for {S_t}. 4. LARGE DEVIATIONS: The sequence of empirical measures of {Phi(t)} satisfies an LDP in a topology finer than the τ-topology. The rate function is Λ^* and it coincides with the Donsker-Varadhan rate function. 5. EXACTR LARGE DEVIATIONS: The partial sums {S_t} satisfy an exact LD expansion, analogous to that obtained for independent random variables.

math.PR↗

arXiv2004

Entropy and the Law of Small Numbers

Two new information-theoretic methods are introduced for establishing Poisson approximation inequalities. First, using only elementary information-theoretic techniques it is shown that, when $S_n=\sum_{i=1}^nX_i$ is the sum of the (possibly dependent) binary random variables $X_1,X_2,...,X_n$, with $E(X_i)=p_i$ and $E(S_n)=\la$, then \ben D(P_{S_n}\|\Pol)\leq \sum_{i=1}^n p_i^2 + \Big[\sum_{i=1}^nH(X_i) - H(X_1,X_2,..., X_n)\Big], \een where $D(P_{S_n}\|{Po}(\la))$ is the relative entropy between the distribution of $S_n$ and the Poisson($\la$) distribution. The first term in this bound measures the individual smallness of the $X_i$ and the second term measures their dependence. A general method is outlined for obtaining corresponding bounds when approximating the distribution of a sum of general discrete random variables by an infinitely divisible distribution. Second, in the particular case when the $X_i$ are independent, the following sharper bound is established, \ben D(P_{S_n}\|\Pol)\leq \frac{1}λ \sum_{i=1}^n \frac{p_i^3}{1-p_i}, % \label{eq:abs2} \een and it is also generalized to the case when the $X_i$ are general integer-valued random variables. Its proof is based on the derivation of a subadditivity property for a new discrete version of the Fisher information, and uses a recent logarithmic Sobolev inequality for the Poisson distribution.

math.PR↗

arXiv2002

A remark on unified error exponents: Hypothesis testing, data compression and measure concentration

Let A be finite set equipped with a probability distribution P, and let M be a "mass" function on A. A characterization is given for the most efficient way in which A^n can be covered using spheres of a fixed radius. A covering is a subset C_n of A^n with the property that most of the elements of A^n are within some fixed distance from at least one element of C_n, and "most of the elements" means a set whose probability is exponentially close to one (with respect to the product distribution P^n). An efficient covering is one with small mass M^n(C_n). With different choices for the geometry on A, this characterization gives various corollaries as special cases, including Marton's error-exponents theorem in lossy data compression, Hoeffding's optimal hypothesis testing exponents, and a new sharp converse to some measure concentration inequalities on discrete spaces.

math.PR↗

arXiv2002

Steady state analysis of balanced-allocation routing

We compare the long-term, steady-state performance of a variant of the standard Dynamic Alternative Routing (DAR) technique commonly used in telephone and ATM networks, to the performance of a path-selection algorithm based on the "balanced-allocation" principle; we refer to this new algorithm as the Balanced Dynamic Alternative Routing (BDAR) algorithm. While DAR checks alternative routes sequentially until available bandwidth is found, the BDAR algorithm compares and chooses the best among a small number of alternatives. We show that, at the expense of a minor increase in routing overhead, the BDAR algorithm gives a substantial improvement in network performance, in terms both of network congestion and of bandwidth requirement.

math.PR↗

arXiv2002

Spectral Theory and Limit Theorems for Geometrically Ergodic Markov Processes

Consider the partial sums {S_t} of a real-valued functional F(Phi(t)) of a Markov chain {Phi(t)} with values in a general state space. Assuming only that the Markov chain is geometrically ergodic and that the functional F is bounded, the following conclusions are obtained: 1. Spectral theory: Well-behaved solutions can be constructed for the ``multiplicative Poisson equation''. 2. A ``multiplicative'' mean ergodic theorem: For all complex αin a neighborhood of the origin, the normalized mean of \exp(αS_t) converges exponentially fast to a solution of the multiplicative Poisson equation. 3. Edgeworth Expansions: Rates are obtained for the convergence of the distribution function of the normalized partial sums S_t to the standard Gaussian distribution. 4. Large Deviations: The partial sums are shown to satisfy a large deviations principle in a neighborhood of the mean. This result, proved under geometric ergodicity alone, cannot in general be extended to the whole real line. 5. Exact Large Deviations Asymptotics: Rates of convergence are obtained for the large deviations estimates above. Extensions of these results to continuous-time Markov processes are also given.

math.PR↗

arXiv2000

Efficient sphere-covering and converse measure concentration via generalized coding theorems

Suppose A is a finite set equipped with a probability measure P and let M be a ``mass'' function on A. We give a probabilistic characterization of the most efficient way in which A^n can be almost-covered using spheres of a fixed radius. An almost-covering is a subset C_n of A^n, such that the union of the spheres centered at the points of C_n has probability close to one with respect to the product measure P^n. An efficient covering is one with small mass M^n(C_n); n is typically large. With different choices for M and the geometry on A our results give various corollaries as special cases, including Shannon's data compression theorem, a version of Stein's lemma (in hypothesis testing), and a new converse to some measure concentration inequalities on discrete spaces. Under mild conditions, we generalize our results to abstract spaces and non-product measures.

math.PR↗

arXiv2000

Critical Behavior in Lossy Source Coding

The following critical phenomenon was recently discovered. When a memoryless source is compressed using a variable-length fixed-distortion code, the fastest convergence rate of the (pointwise) compression ratio to the optimal $R(D)$ bits/symbol is either $O(\sqrt{n})$ or $O(\log n)$. We show it is always $O(\sqrt{n})$, except for discrete, uniformly distributed sources.

math.PR↗