arXiv subjects

Ioannis Kontoyiannis

Publications and source records attributed to Ioannis Kontoyiannis.

At least 19 recordsLinked to original sources

arXiv2026

Fine asymptotics for approximate simulation of symmetric channels

We study the approximate channel simulation problem in the normal approximation regime. Our starting point is the fact established by Cao et al.~\cite{Cao_channel_simulation}, that the minimal codebook size $M_ε^*(W)$ for a fixed error tolerance $ε>0$ is closely related to the $ε$-smooth channel max-information, $I_{\max}^ε(W)$. Using this, we refine earlier large-blocklength approximations to $M_ε^*(W^{\otimes n})$ for discrete memoryless channels. Specifically, we provide tight third-order expansions for Gallager-symmetric channels and demonstrate that the third-order term is channel-dependent. As in corresponding channel coding expansions, we identify a substantial rate gap between singular and nonsingular channels. Notably, the direction of this gap is reversed relative to channel coding: The third-order term is $-\frac{1}{2}\log n$ for singular channels but $O(\log\log n)$ for nonsingular channels.

cs.IT↗

arXiv2026

Fast PAC Global Optimization via Restarted Langevin: Exploration, Exploitation, and Degenerate Cooling

This paper addresses a fundamental question in non-convex optimization: \textit{How should a stochastic optimizer allocate computation between global exploration and local exploitation?} We study this question in continuous time, using Langevin dynamics for global exploration and deterministic gradient flow for local exploitation. The simplest algorithm in this class is the best-state Langevin--gradient method: several Langevin trajectories explore the objective landscape, the best state encountered is retained, and a single gradient trajectory then refines this state to high terminal accuracy. Our main objective is to characterize the computational work required to achieve a prescribed accuracy with prescribed confidence. We develop low-temperature approximations for this probably approximately correct (PAC) work--accuracy tradeoff, both for the best-state method and for related Langevin schemes. Global exploration is governed by energy barriers, through spectral-gap and Eyring--Kramers asymptotics. The resulting approximations expose the competing effects of temperature, computational budget, and the number of exploratory trajectories, and quantify their dependence on the confidence parameter. In contrast, terminal accuracy is largely separated from global exploration: once a suitable region of attraction has been reached, gradient flow requires only $O(\log(1/\varepsilon))$ additional work to reach accuracy at least $\varepsilon$. Numerical experiments on challenging objectives serve to compare algorithms and to test the quantitative predictions of the asymptotic theory. They identify regimes in which the low-temperature approximations provide useful guidance, as well as regimes exhibiting substantial temperature sensitivity and potentially significant high-dimensional limitations.

math.OC↗

arXiv2026

The Sample Complexity of Lossless Data Compression

A new framework is introduced for examining and evaluating the fundamental limits of lossless data compression, that emphasizes genuinely non-asymptotic results. The {\em sample complexity} of compressing a given source is defined as the smallest blocklength at which it is possible to compress that source at a specifically constrained rate and to within a specified excess-rate probability. This formulation parallels corresponding developments in statistics and computer science, and it facilitates the use of existing results on the sample complexity of various hypothesis testing problems. For arbitrary sources, the sample complexity of general variable-length compressors is shown to be tightly coupled with the sample complexity of prefix-free codes and fixed-length codes. For memoryless sources, it is shown that the sample complexity is characterized not by the source entropy, but by its Rényi entropy of order~$1/2$. Nonasymptotic bounds on the sample complexity are obtained, with explicit constants. Generalizations to Markov sources are established, showing that the sample complexity is determined by the source's Rényi entropy rate of order~$1/2$. Finally, bounds on the sample complexity of universal data compression are developed for families of memoryless sources. There, the sample complexity is characterized by the minimum Rényi divergence of order~$1/2$ between elements of the family and the uniform distribution. The connection of this problem with identity testing and with the associated separation rates is explored and discussed.

cs.IT↗

arXiv2026

Entropy lower bounds and sum-product phenomena

Various lower bounds are established for the entropy of sums, products and their combinations. First, we derive a prime-field analogue of a version of the entropy power inequality established by Tao over torsion-free groups. Next, we prove an entropy sum-product statement: For independent and identically distributed random variables $X,X'$, the maximum of ${\bf H}(X+X')$ and ${\bf H}(XX')$ is bounded below by a linear combination of the entropy and the min-entropy (Rényi entropy of order~$\infty$) of $X$. This result, obtained by bounding entropies of the form ${\bf H}\bigl( X(Y+Z)\bigr)$ from above and below, is valid over arbitrary fields $F$. Over $F={\bf R}$, a slightly stronger inequality is derived. Finally, a weak version of a purely Shannon-entropic sum-product result is developed: If the entropic additive doubling of a random variable $X$ over an arbitrary field is $O(1)$, then its multiplicative doubling is at least proportional to ${\bf H}(X)$.

math.CO↗

arXiv2026

Entropy power inequalities in compact groups

Suppose $X,Y$ are independent random variables with values in a compact abelian group $(G,+)$. We examine the following two entropy power-type inequalities: $h(X+Y)\geq \frac{1}{2}h(X)+\frac{1}{2}h(Y)$ and $h(X+Y)\geq \max\{h(X),h(Y)\}$, where the entropy $h(Z)$ of a $G$-valued random variable $Z$ is defined in terms of its density with respect to Haar measure on $G$. For groups that are either connected or finite with no nontrivial subgroups, we precisely characterize the cases of equality and establish explicit, quantitative stability estimates in terms of relative entropy for these two inequalities. The main tools are a generalization of an entropic inequality obtained by Green, Manners and Tao (2023) for discrete entropy, and a harmonic-analytic estimate for the chi-squared contraction coefficient in connected compact groups. As an application, we derive exponential convergence rates to the uniform distribution in relative entropy for random walks on connected compact abelian groups.

math.PR↗

arXiv2026

ConRAD: Conformal Risk-Aware Neural Databases

Querying incomplete knowledge graphs with neural predictors is powerful but dangerous. Errors compound across multi-hop pipelines with no formal bound on the completeness of results. We introduce ConRAD, the first framework to enforce declarative recall guarantees natively within a neural graph database query engine. Given a user-specified risk budget, ConRAD automatically derives per-operator prediction thresholds that satisfy the recall target with finite-sample, distribution-free statistical validity via Conformal Risk Control, while maximizing end-to-end precision. To scale calibration across multi-operator query topologies, we introduce a quantile-space scalarization that reduces intractable high-dimensional threshold searches to a single parameter. We further design the conformal gate, a novel physical operator that dynamically bypasses neural inference when local graph evidence suffices, eliminating unnecessary model inferences in dense graph regions. Evaluated across three benchmarks and three query topologies, ConRAD strictly satisfies all risk budgets, with empirical recall falling below the target by at most 0.046 across all settings. It reduces neural invocations to zero in near-complete graph regions, and achieves precision that matches or exceeds best-case static baselines that offer no guarantees and require manual threshold search.

cs.DB↗

arXiv2026

The relative value of interventional and observational samples in Bayesian Causal Linear Gaussian Models

We investigate the asymptotic properties of Bayesian bivariate causal discovery for Gaussian Linear Structural Equation Models (SEMs) with heteroscedastic noise. We demonstrate that with purely observational data, the posterior distribution over the models fails to consistently identify the true causal structure - a consequence of the fundamental non-identifiability within the Markov Equivalence Class. Specifically, if the true generating mechanism corresponds to a connected graph (A -> B or B -> A), the asymptotic behavior of the posterior is given by the ratio between the prior on the true model and the push-forward prior of the alternative. In contrast, for the independence model, we establish that the posterior concentrates at a stochastic polynomial rate of O_p(n^{-1/2}). To resolve this non-identifiability, we incorporate m interventional samples and characterize the concentration rates as a function of the observational-to-total sample ratio, η. We identify a sharp concentration dichotomy: while the independence graph maintains a polynomial O_p(N^{-1/2}) rate (where N = n+m), connected graphs undergo a phase transition to exponentially fast convergence. This highlights an exponential relative importance between the two data types, as altering the amount of one data type directly changes the exponent governing the concentration speed. We derive explicit formulae for the exponential decay rates and provide precise conditions under which mixing observational and interventional data optimizes concentration speed. Finally, our theoretical findings are validated through empirical simulations in Bayesian Gaussian equivalent (BGe)-style prior specifications offering a principled foundation for experimental design in Bayesian causal discovery.

math.ST↗

arXiv2025

Shakespeare, Entropy and Educated Monkeys

It has often been said, correctly, that a monkey forever randomly typing on a keyboard would eventually produce the complete works of William Shakespeare. Almost just as often it has been pointed out that this "eventually" is well beyond any conceivably relevant time frame. We point out that an educated monkey that still types at random but is constrained to only write "statistically typical" text, would produce any given passage in a much shorter time. Information theory gives a very simple way to estimate that time. For example, Shakespeare's phrase, Better three hours too soon than a minute too late, from The Merry Wives of Windsor, would take the educated monkey only 73 thousand years to produce, compared to the beyond-astronomical $2.7 \times 10^{63}$ years for the randomly typing one. Despite the obvious improvement, it would still take the educated monkey an unimaginably long $10^{42,277}$ years to produce all of Hamlet.

math.HO↗

arXiv2025

Entropic additive energy and entropy inequalities for sums and products

Following a growing number of studies that, over the past 15 years, have established entropy inequalities via ideas and tools from additive combinatorics, in this work we obtain a number of new bounds for the differential entropy of sums, products, and sum-product combinations of continuous random variables. Partly motivated by recent work by Goh on the discrete entropic version of the notion of "additive energy", we introduce the additive energy of pairs of continuous random variables and prove various versions of the statement that "the additive energy is large if and only if the entropy of the sum is small", along with a version of the Balog-Szemerédi-Gowers theorem for differential entropy. Then, motivated in part by recent work by Máthé and O'Regan, we establish a series of new differential entropy inequalities for products and sum-product combinations of continuous random variables. In particular, we prove a new, general, ring Plünnecke-Ruzsa entropy inequality. We briefly return to the case of discrete entropy and provide a characterization of discrete random variables with "large doubling", analogous to Tao's Freiman-type inverse sumset theory for the case of small doubling. Finally, we consider the natural entropic analog of the Erdös-Szemerédi sum-product phenomenon for integer-valued random variables. We show that, if it does hold, then the range of parameters for which it does would necessarily be significantly more restricted than its anticipated combinatorial counterpart.

cs.IT↗

arXiv2025

Pragmatic lossless compression: Fundamental limits and universality

The problem of variable-rate lossless data compression is considered, for codes with and without prefix constraints. Sharp bounds are derived for the best achievable compression rate of memoryless sources, when the excess-rate probability is required to be exponentially small in the blocklength. Accurate nonasymptotic expansions with explicit constants are obtained for the optimal rate, using tools from large deviations and Gaussian approximation. When the source distribution is unknown, a universal achievability result is obtained with an explicit ''price for universality'' term. This is based on a fine combinatorial estimate on the number of sequences with small empirical entropy, which might be of independent interest. Examples are shown indicating that, in the small excess-rate-probability regime, the approximation to the fundamental limit of the compression rate suggested by these bounds is significantly more accurate than the approximations provided by either normal approximation or error exponents. The new bounds reinforce the crucial operational conclusion that, in applications where the blocklength is relatively short and where stringent guarantees are required on the rate, the best achievable rate is no longer close to the entropy. Rather, it is an appropriate, more pragmatic rate, determined via the inverse error exponent function and the blocklength.

cs.IT↗

arXiv2025

The case for and against fixed step-size: Stochastic approximation algorithms in optimization and machine learning

Theory and application of stochastic approximation (SA) have become increasingly relevant due in part to applications in optimization and reinforcement learning. This paper takes a new look at SA with constant step-size $α>0$, defined by the recursion, $$θ_{n+1} = θ_{n}+ αf(θ_n,Φ_{n+1})$$ in which $θ_n\in\mathbb{R}^d$ and $\{Φ_{n}\}$ is a Markov chain. The goal is to approximately solve root finding problem $\bar{f}(θ^*)=0$, where $\bar{f}(θ)=\mathbb{E}[f(θ,Φ)]$ and $Φ$ has the steady-state distribution of $\{Φ_{n}\}$. The following conclusions are obtained under an ergodicity assumption on the Markov chain, compatible assumptions on $f$, and for $α>0$ sufficiently small: $\textbf{1.}$ The pair process $\{(θ_n,Φ_n)\}$ is geometrically ergodic in a topological sense. $\textbf{2.}$ For every $1\le p\le 4$, there is a constant $b_p$ such that $\limsup_{n\to\infty}\mathbb{E}[\|θ_n-θ^*\|^p]\le b_p α^{p/2}$ for each initial condition. $\textbf{3.}$ The Polyak-Ruppert-style averaged estimates $θ^{\text{PR}}_n=n^{-1}\sum_{k=1}^{n}θ_k$ converge to a limit $θ^{\text{PR}}_\infty$ almost surely and in mean square, which satisfies $θ^{\text{PR}}_\infty=θ^*+α\barΥ^*+O(α^2)$ for an identified non-random $\barΥ^*\in\mathbb{R}^d$. Moreover, the covariance is approximately optimal: The limiting covariance matrix of $θ^{\text {PR}}_n$ is approximately minimal in a matricial sense. The two main take-aways for practitioners are application-dependent. It is argued that, in applications to optimization, constant gain algorithms may be preferable even when the objective has multiple local minima; while a vanishing gain algorithm is preferable in applications to reinforcement learning due to the presence of bias.

math.ST↗

arXiv2025

Conditions for equality and stability in Shannon's and Tao's entropy power inequalities

We show that there is equality in Shannon's Entropy Power Inequality (EPI) if and only if the random variables involved are Gaussian, assuming nothing beyond the existence of differential entropies. This is done by justifying de Bruijn's identity without a second moment assumption. Part of the proof also relies on a re-examination of an example of Bobkov and Chistyakov (2015), which shows that there exists a random variable $X$ with finite differential entropy $h(X),$ such that $h(X+Y) = \infty$ for any independent random variable $Y$ with finite entropy. We prove that either $X$ has this property, or $h(X+Y)$ is finite for any independent $Y$ that does not have this property. Using this, we prove the continuity of $t \mapsto h(X+\sqrt{t}Z)$ at $t=0$, where $Z \sim \mathcal{N}(0,1)$ is independent of $X$, under minimal assumptions. We then establish two stability results: A qualitative stability result for Shannon's EPI in terms of weak convergence under very mild moment conditions, and a quantitative stability result in Tao's discrete analogue of the EPI under log-concavity. The proof for the first stability result is based on a compactness argument, while the proof of the second uses the Cheeger inequality and leverages concentration properties of discrete log-concave distributions.

math.PR↗

arXiv2025

The Bayesian Context Trees State Space Model for time series modelling and forecasting

A hierarchical Bayesian framework is introduced for developing tree-based mixture models for time series, partly motivated by applications in finance and forecasting. At the top level, meaningful discrete states are identified as appropriately quantised values of some of the most recent samples. At the bottom level, a different, arbitrary base model is associated with each state. This defines a very general framework that can be used in conjunction with any existing model class to build flexible and interpretable mixture models. We call this the Bayesian Context Trees State Space Model, or the BCT-X framework. Appropriate algorithmic tools are described, which allow for effective and efficient Bayesian inference and learning; these algorithms can be updated sequentially, facilitating online forecasting. The utility of the general framework is illustrated in the particular instances when AR or ARCH models are used as base models. The latter results in a mixture model that offers a powerful way of modelling the well-known volatility asymmetries in financial data, revealing a novel, important feature of stock market index data, in the form of an enhanced leverage effect. In forecasting, the BCT-X methods are found to outperform several state-of-the-art techniques, both in terms of accuracy and computational requirements.

stat.ME↗

arXiv2025

Bayesian causal discovery: Posterior concentration and optimal detection

We consider the problem of Bayesian causal discovery for the standard model of linear structural equations with equivariant Gaussian noise. A uniform prior is placed on the space of directed acyclic graphs (DAGs) over a fixed set of variables and, given the graph, independent Gaussian priors are placed on the associated linear coefficients of pairwise interactions. We show that the rate at which the posterior on model space concentrates on the true underlying DAG depends critically on its nature: If it is maximal, in the sense that adding any one new edge would violate acyclicity, then its posterior probability converges to 1 exponentially fast (almost surely) in the sample size $n$. Otherwise, it converges at a rate no faster than $1/\sqrt{n}$. This sharp dichotomy is an instance of the important general phenomenon that avoiding overfitting is significantly harder than identifying all of the structure that is present in the model. We also draw a new connection between the posterior distribution on model space and recent results on optimal hypothesis testing in the related problem of edge detection. Our theoretical findings are illustrated empirically through simulation experiments.

math.ST↗

arXiv2025

Change-point Detection and Segmentation of Discrete Data using Bayesian Context Trees

A new Bayesian modelling framework is introduced for piece-wise homogeneous variable-memory Markov chains, along with a collection of effective algorithmic tools for change-point detection and segmentation of discrete time series. Building on the recently introduced Bayesian Context Trees (BCT) framework, the distributions of different segments in a discrete time series are described as variable-memory Markov chains. Inference for the presence and location of change-points is then performed via Markov chain Monte Carlo sampling. The key observation that facilitates effective sampling is that, using one of the BCT algorithms, the prior predictive likelihood of the data can be computed exactly, integrating out all the models and parameters in each segment. This makes it possible to sample directly from the posterior distribution of the number and location of the change-points, leading to accurate estimates and providing a natural quantitative measure of uncertainty in the results. Estimates of the actual model in each segment can also be obtained, at essentially no additional computational cost. Results on both simulated and real-world data indicate that the proposed methodology performs better than or as well as state-of-the-art techniques.

stat.ME↗

arXiv2024

The ODE Method for Asymptotic Statistics in Stochastic Approximation and Reinforcement Learning

The paper concerns the $d$-dimensional stochastic approximation recursion, $$ θ_{n+1}= θ_n + α_{n + 1} f(θ_n, Φ_{n+1}) $$ where $ \{ Φ_n \}$ is a stochastic process on a general state space, satisfying a conditional Markov property that allows for parameter-dependent noise. The main results are established under additional conditions on the mean flow and a version of the Donsker-Varadhan Lyapunov drift condition known as (DV3): (i) An appropriate Lyapunov function is constructed that implies convergence of the estimates in $L_4$. (ii) A functional central limit theorem (CLT) is established, as well as the usual one-dimensional CLT for the normalized error. Moment bounds combined with the CLT imply convergence of the normalized covariance $\textsf{E}[ z_n z_n^T ]$ to the asymptotic covariance in the CLT, where $z_n =: (θ_n-θ^*)/\sqrt{α_n}$. (iii) The CLT holds for the normalized version $z^{\text{PR}}_n =: \sqrt{n} [θ^{\text{PR}}_n -θ^*]$, of the averaged parameters $θ^{\text{PR}}_n =:n^{-1} \sum_{k=1}^nθ_k$, subject to standard assumptions on the step-size. Moreover, the covariance in the CLT coincides with the minimal covariance of Polyak and Ruppert. (iv) An example is given where $f$ and $\bar{f}$ are linear in $θ$, and $Φ$ is a geometrically ergodic Markov chain but does not satisfy (DV3). While the algorithm is convergent, the second moment of $θ_n$ is unbounded and in fact diverges. This arXiv version represents a major extension of the results in prior versions.The main results now allow for parameter-dependent noise, as is often the case in applications to reinforcement learning.

math.ST↗

arXiv2024

The entropic doubling constant and robustness of Gaussian codebooks for additive-noise channels

Entropy comparison inequalities are obtained for the differential entropy $h(X+Y)$ of the sum of two independent random vectors $X,Y$, when one is replaced by a Gaussian. For identically distributed random vectors $X,Y$, these are closely related to bounds on the entropic doubling constant, which quantifies the entropy increase when adding an independent copy of a random vector to itself. Consequences of both large and small doubling are explored. For the former, lower bounds are deduced on the entropy increase when adding an independent Gaussian, while for the latter, a qualitative stability result for the entropy power inequality is obtained. In the more general case of non-identically distributed random vectors $X,Y$, a Gaussian comparison inequality with interesting implications for channel coding is established: For additive-noise channels with a power constraint, Gaussian codebooks come within a $\frac{\sf snr}{3{\sf snr}+2}$ factor of capacity. In the low-SNR regime this improves the half-a-bit additive bound of Zamir and Erez (2004). Analogous results are obtained for additive-noise multiple access channels, and for linear, additive-noise MIMO channels.

cs.IT↗

arXiv2024

Finite de Finetti bounds in relative entropy

We review old and recent finite de Finetti theorems in total variation distance and in relative entropy, and we highlight their connections with bounds on the difference between sampling with and without replacement. We also establish two new finite de Finetti theorems for exchangeable random vectors taking values in arbitrary spaces. These bounds are tight, and they are independent of the size and the dimension of the underlying space.

math.PR↗