arXiv subjects

Varun Jog

Publications and source records attributed to Varun Jog.

33 records · Page 2Linked to original sources

arXiv2018

Optimal Rates for Community Estimation in the Weighted Stochastic Block Model

Community identification in a network is an important problem in fields such as social science, neuroscience, and genetics. Over the past decade, stochastic block models (SBMs) have emerged as a popular statistical framework for this problem. However, SBMs have an important limitation in that they are suited only for networks with unweighted edges; in various scientific applications, disregarding the edge weights may result in a loss of valuable information. We study a weighted generalization of the SBM, in which observations are collected in the form of a weighted adjacency matrix and the weight of each edge is generated independently from an unknown probability density determined by the community membership of its endpoints. We characterize the optimal rate of misclustering error of the weighted SBM in terms of the Renyi divergence of order 1/2 between the weight distributions of within-community and between-community edges, substantially generalizing existing results for unweighted SBMs. Furthermore, we present a computationally tractable algorithm based on discretization that achieves the optimal error rate. Our method is adaptive in the sense that the algorithm, without assuming knowledge of the weight densities, performs as well as the best algorithm that knows the weight densities.

math.ST↗

arXiv2018

Convexity of mutual information along the Ornstein-Uhlenbeck flow

We study the convexity of mutual information as a function of time along the flow of the Ornstein-Uhlenbeck process. We prove that if the initial distribution is strongly log-concave, then mutual information is eventually convex, i.e., convex for all large time. In particular, if the initial distribution is sufficiently strongly log-concave compared to the target Gaussian measure, then mutual information is always a convex function of time. We also prove that if the initial distribution is either bounded or has finite fourth moment and Fisher information, then mutual information is eventually convex. Finally, we provide counterexamples to show that mutual information can be nonconvex at small time.

cs.IT↗

arXiv2018

Convexity of mutual information along the heat flow

We study the convexity of mutual information along the evolution of the heat equation. We prove that if the initial distribution is log-concave, then mutual information is always a convex function of time. We also prove that if the initial distribution is either bounded, or has finite fourth moment and Fisher information, then mutual information is eventually convex, i.e., convex for all large time. Finally, we provide counterexamples to show that mutual information can be nonconvex at small time.

cs.IT↗

arXiv2018

Graph-Based Ascent Algorithms for Function Maximization

We study the problem of finding the maximum of a function defined on the nodes of a connected graph. The goal is to identify a node where the function obtains its maximum. We focus on local iterative algorithms, which traverse the nodes of the graph along a path, and the next iterate is chosen from the neighbors of the current iterate with probability distribution determined by the function values at the current iterate and its neighbors. We study two algorithms corresponding to a Metropolis-Hastings random walk with different transition kernels: (i) The first algorithm is an exponentially weighted random walk governed by a parameter $γ$. (ii) The second algorithm is defined with respect to the graph Laplacian and a smoothness parameter $k$. We derive convergence rates for the two algorithms in terms of total variation distance and hitting times. We also provide simulations showing the relative convergence rates of our algorithms in comparison to an unbiased random walk, as a function of the smoothness of the graph function. Our algorithms may be categorized as a new class of "descent-based" methods for function maximization on the nodes of a graph.

cs.SI↗

arXiv2018

Generalization Error Bounds for Noisy, Iterative Algorithms

In statistical learning theory, generalization error is used to quantify the degree to which a supervised machine learning algorithm may overfit to training data. Recent work [Xu and Raginsky (2017)] has established a bound on the generalization error of empirical risk minimization based on the mutual information $I(S;W)$ between the algorithm input $S$ and the algorithm output $W$, when the loss function is sub-Gaussian. We leverage these results to derive generalization error bounds for a broad class of iterative algorithms that are characterized by bounded, noisy updates with Markovian structure. Our bounds are very general and are applicable to numerous settings of interest, including stochastic gradient Langevin dynamics (SGLD) and variants of the stochastic gradient Hamiltonian Monte Carlo (SGHMC) algorithm. Furthermore, our error bounds hold for any output function computed over the path of iterates, including the last iterate of the algorithm or the average of subsets of iterates, and also allow for non-uniform sampling of data in successive updates of the algorithm.

cs.LG↗

arXiv2018

An entropy inequality for symmetric random variables

We establish a lower bound on the entropy of weighted sums of (possibly dependent) random variables $(X_1, X_2, \dots, X_n)$ possessing a symmetric joint distribution. Our lower bound is in terms of the joint entropy of $(X_1, X_2, \dots, X_n)$. We show that for $n \geq 3$, the lower bound is tight if and only if $X_i$'s are i.i.d.\ Gaussian random variables. For $n=2$ there are numerous other cases of equality apart from i.i.d.\ Gaussians, which we completely characterize. Going beyond sums, we also present an inequality for certain linear transformations of $(X_1, \dots, X_n)$. Our primary technical contribution lies in the analysis of the equality cases, and our approach relies on the geometry and the symmetry of the problem.

cs.IT↗

arXiv2017

Information and estimation in Fokker-Planck channels

We study the relationship between information- and estimation-theoretic quantities in time-evolving systems. We focus on the Fokker-Planck channel defined by a general stochastic differential equation, and show that the time derivatives of entropy, KL divergence, and mutual information are characterized by estimation-theoretic quantities involving an appropriate generalization of the Fisher information. Our results vastly extend De Bruijn's identity and the classical I-MMSE relation.

cs.IT↗

arXiv2017

Intrinsic entropies of log-concave distributions

The entropy of a random variable is well-known to equal the exponential growth rate of the volumes of its typical sets. In this paper, we show that for any log-concave random variable $X$, the sequence of the $\lfloor nθ\rfloor^{\text{th}}$ intrinsic volumes of the typical sets of $X$ in dimensions $n \geq 1$ grows exponentially with a well-defined rate. We denote this rate by $h_X(θ)$, and call it the $θ^{\text{th}}$ intrinsic entropy of $X$. We show that $h_X(θ)$ is a continuous function of $θ$ over the range $[0,1]$, thereby providing a smooth interpolation between the values 0 and $h(X)$ at the endpoints 0 and 1, respectively.

cs.IT↗

arXiv2016

Analysis of centrality in sublinear preferential attachment trees via the CMJ branching process

We investigate centrality and root-inference properties in a class of growing random graphs known as sublinear preferential attachment trees. We show that a continuous time branching processes called the Crump-Mode-Jagers (CMJ) branching process is well-suited to analyze such random trees, and prove that almost surely, a unique terminal tree centroid emerges, having the property that it becomes more central than any other fixed vertex in the limit of the random growth process. Our result generalizes and extends previous work establishing persistent centrality in uniform and linear preferential attachment trees. We also show that centrality may be utilized to generate a finite-sized $1-ε$ confidence set for the root node, for any $ε> 0$ in a certain subclass of sublinear preferential attachment trees.

math.PR↗

arXiv2015

Persistence of centrality in random growing trees

We investigate properties of node centrality in random growing tree models. We focus on a measure of centrality that computes the maximum subtree size of the tree rooted at each node, with the most central node being the tree centroid. For random trees grown according to a preferential attachment model, a uniform attachment model, or a diffusion processes over a regular tree, we prove that a single node persists as the tree centroid after a finite number of steps, with probability 1. Furthermore, this persistence property generalizes to the top $K \ge 1$ nodes with respect to the same centrality measure. We also establish necessary and sufficient conditions for the size of an initial seed graph required to ensure persistence of a particular node with probability $1-ε$, as a function of $ε$: In the case of preferential and uniform attachment models, we derive bounds for the size of an initial hub constructed around the special node. In the case of a diffusion process over a regular tree, we derive bounds for the radius of an initial ball centered around the special node. Our necessary and sufficient conditions match up to constant factors for preferential attachment and diffusion tree models.

math.PR↗

arXiv2015

Information-theoretic bounds for exact recovery in weighted stochastic block models using the Renyi divergence

We derive sharp thresholds for exact recovery of communities in a weighted stochastic block model, where observations are collected in the form of a weighted adjacency matrix, and the weight of each edge is generated independently from a distribution determined by the community membership of its endpoints. Our main result, characterizing the precise boundary between success and failure of maximum likelihood estimation when edge weights are drawn from discrete distributions, involves the Renyi divergence of order $\frac{1}{2}$ between the distributions of within-community and between-community edges. When the Renyi divergence is above a certain threshold, meaning the edge distributions are sufficiently separated, maximum likelihood succeeds with probability tending to 1; when the Renyi divergence is below the threshold, maximum likelihood fails with probability bounded away from 0. In the language of graphical channels, the Renyi divergence pinpoints the information-theoretic capacity of discrete graphical channels with binary inputs. Our results generalize previously established thresholds derived specifically for unweighted block models, and support an important natural intuition relating the intrinsic hardness of community estimation to the problem of edge classification. Along the way, we establish a general relationship between the Renyi divergence and the probability of success of the maximum likelihood estimator for arbitrary edge weight distributions. Finally, we discuss consequences of our bounds for the related problems of censored block models and submatrix localization, which may be seen as special cases of the framework developed in our paper.

cs.IT↗

arXiv2015

A Geometric Analysis of the AWGN channel with a $(σ, ρ)$-Power Constraint

In this paper, we consider the AWGN channel with a power constraint called the $(σ, ρ)$-power constraint, which is motivated by energy harvesting communication systems. Given a codeword, the constraint imposes a limit of $σ+ k ρ$ on the total power of any $k\geq 1$ consecutive transmitted symbols. Such a channel has infinite memory and evaluating its exact capacity is a difficult task. Consequently, we establish an $n$-letter capacity expression and seek bounds for the same. We obtain a lower bound on capacity by considering the volume of ${\cal S}_n(σ, ρ) \subseteq \mathbb{R}^n$, which is the set of all length $n$ sequences satisfying the $(σ, ρ)$-power constraints. For a noise power of $ν$, we obtain an upper bound on capacity by considering the volume of ${\cal S}_n(σ, ρ) \oplus B_n(\sqrt{nν})$, which is the Minkowski sum of ${\cal S}_n(σ, ρ)$ and the $n$-dimensional Euclidean ball of radius $\sqrt{nν}$. We analyze this bound using a result from convex geometry known as Steiner's formula, which gives the volume of this Minkowski sum in terms of the intrinsic volumes of ${\cal S}_n(σ, ρ)$. We show that as the dimension $n$ increases, the logarithm of the sequence of intrinsic volumes of $\{{\cal S}_n(σ, ρ)\}$ converges to a limit function under an appropriate scaling. The upper bound on capacity is then expressed in terms of this limit function. We derive the asymptotic capacity in the low and high noise regime for the $(σ, ρ)$-power constrained AWGN channel, with strengthened results for the special case of $σ= 0$, which is the amplitude constrained AWGN channel.

cs.IT↗

arXiv2015

On model misspecification and KL separation for Gaussian graphical models

We establish bounds on the KL divergence between two multivariate Gaussian distributions in terms of the Hamming distance between the edge sets of the corresponding graphical models. We show that the KL divergence is bounded below by a constant when the graphs differ by at least one edge; this is essentially the tightest possible bound, since classes of graphs exist for which the edge discrepancy increases but the KL divergence remains bounded above by a constant. As a natural corollary to our KL lower bound, we also establish a sample size requirement for correct model selection via maximum likelihood estimation. Our results rigorize the notion that it is essential to estimate the edge structure of a Gaussian graphical model accurately in order to approximate the true distribution to close precision.

cs.IT↗

arXiv2012

The Entropy Power Inequality and Mrs. Gerber's Lemma for Abelian Groups of Order 2^n

Shannon's Entropy Power Inequality can be viewed as characterizing the minimum differential entropy achievable by the sum of two independent random variables with fixed differential entropies. The entropy power inequality has played a key role in resolving a number of problems in information theory. It is therefore interesting to examine the existence of a similar inequality for discrete random variables. In this paper we obtain an entropy power inequality for random variables taking values in an abelian group of order 2^n, i.e. for such a group G we explicitly characterize the function f_G(x,y) giving the minimum entropy of the sum of two independent G-valued random variables with respective entropies x and y. Random variables achieving the extremum in this inequality are thus the analogs of Gaussians in this case, and these are also determined. It turns out that f_G(x,y) is convex in x for fixed y and, by symmetry, convex in y for fixed x. This is a generalization to abelian groups of order 2^n of the result known as Mrs. Gerber's Lemma.

cs.IT↗

arXiv2009

An information inequality for the BSSC channel

We establish an information theoretic inequality concerning the binary skew-symmetric broadcast channel that was conjectured by one of the authors. This inequality helps to quantify the gap between the sum rate obtained by the inner bound and outer bound for the binary skew-symmetric broadcast channel.

cs.IT↗