arXiv subjects

Andrew Thangaraj

Publications and source records attributed to Andrew Thangaraj.

At least 19 recordsLinked to original sources

arXiv2026

Semantic Smoothing for Language Models via Distribution Estimation and Embeddings

We propose semantic smoothing, a smoothing method for language models that uses embeddings to share statistical observations across semantically similar contexts. The starting point is a decomposition of log-perplexity that motivates smoothing as a collection of distribution-estimation problems under Kullback-Leibler (KL) loss. We then show that, under a Lipschitz-logit model for embedding-based language generation, proximity of context embeddings implies proximity of the corresponding next-word distributions in KL divergence. Combining these observations, we formulate semantic smoothing as distribution estimation in KL loss with KL-proximity side information. For $n$ samples on a $d$-symbol alphabet with a side-information distribution at KL distance $\Delta$, we give an interpolation estimator with worst-case KL risk $O(\min\{\Delta,d/n\})$, and prove a matching-order lower bound for uniform side information. We extend the estimator to multiple and empirically estimated synonymous distributions. Experiments on synthetic Markov data and WikiText-103 bigram models using Word2Vec, GloVe, and GPT-2 embeddings show that semantic smoothing consistently reduces test perplexity when applied to add-constant and Kneser-Ney estimates.

cs.IT↗

arXiv2026

Finite-Length Empirical Comparison of Polar, PAC, and Invertible-Extractor Secrecy Codes over the Wiretap BSC

We compare three secrecy-coding schemes for the degraded wiretap binary symmetric channel (BSC) in the finite-blocklength regime: (i) polar wiretap coset codes, (ii) PAC codes used as wiretap coset codes, and (iii) the invertible-extractor (IE) framework of Bellare-Tessaro. Our comparison is empirical and uses a common semantic-secrecy metric (distinguishing advantage). For polar coset codes, we compute Eve's polarized bit-channel capacities (via the Tal-Vardy construction) to obtain explicit finite-length upper bounds on mutual-information leakage, yielding strong secrecy bounds. For PAC coset codes, we prove that Eve's synthesized bit-channels are equivalent to those of polar codes (up to a permutation), so the same leakage bounds apply; we then convert these strong-secrecy bounds into semantic-secrecy guarantees for symmetric wiretap channels. For the IE scheme, we use the closed-form semantic-secrecy bounds given in the reference work. Finally, we report finite-length results that jointly characterize (a) semantic-secrecy guarantees against Eve and (b) frame-error-rate performance at Bob, illustrating that PAC codes can significantly improve reliability without changing the secrecy bounds inherited from polar coding. Moreover, under the finite-length bounds considered in this work, polar/PAC secrecy codes provide tighter security guarantees than the invertible-extractor framework.

cs.IT↗

arXiv2026

Distribution Estimation with Side Information

We consider the classical problem of discrete distribution estimation using i.i.d. samples in a novel scenario where additional side information is available on the distribution. In large alphabet datasets such as text corpora, such side information arises naturally through word semantics/similarities that can be inferred by closeness of vector word embeddings, for instance. We consider two specific models for side information--a local model where the unknown distribution is in the neighborhood of a known distribution, and a partial ordering model where the alphabet is partitioned into known higher and lower probability sets. In both models, we theoretically characterize the improvement in a suitable squared-error risk because of the available side information. Simulations over natural language and synthetic data illustrate these gains.

cs.IT↗

arXiv2024

Rate of Model Collapse in Recursive Training

Given the ease of creating synthetic data from machine learning models, new models can be potentially trained on synthetic data generated by previous models. This recursive training process raises concerns about the long-term impact on model quality. As models are recursively trained on generated data from previous rounds, their ability to capture the nuances of the original human-generated data may degrade. This is often referred to as \emph{model collapse}. In this work, we ask how fast model collapse occurs for some well-studied distribution families under maximum likelihood (ML or near ML) estimation during recursive training. Surprisingly, even for fundamental distributions such as discrete and Gaussian distributions, the exact rate of model collapse is unknown. In this work, we theoretically characterize the rate of collapse in these fundamental settings and complement it with experimental evaluations. Our results show that for discrete distributions, the time to forget a word is approximately linearly dependent on the number of times it occurred in the original corpus, and for Gaussian models, the standard deviation reduces to zero roughly at $n$ iterations, where $n$ is the number of samples at each iteration. Both of these findings imply that model forgetting, at least in these simple distributions under near ML estimation with many samples, takes a long time.

cs.LG↗

arXiv2024

Just Wing It: Near-Optimal Estimation of Missing Mass in a Markovian Sequence

We study the problem of estimating the stationary mass -- also called the unigram mass -- that is missing from a single trajectory of a discrete-time, ergodic Markov chain. This problem has several applications -- for example, estimating the stationary missing mass is critical for accurately smoothing probability estimates in sequence models. While the classical Good--Turing estimator from the 1950s has appealing properties for i.i.d. data, it is known to be biased in the Markovian setting, and other heuristic estimators do not come equipped with guarantees. Operating in the general setting in which the size of the state space may be much larger than the length $n$ of the trajectory, we develop a linear-runtime estimator called Windowed Good--Turing (WingIt) and show that its risk decays as $\widetilde{O}(\mathsf{T_{mix}}/n)$, where $\mathsf{T_{mix}}$ denotes the mixing time of the chain in total variation distance. Notably, this rate is independent of the size of the state space and minimax-optimal up to a logarithmic factor in $n / \mathsf{T_{mix}}$. We also present an upper bound on the variance of the missing mass random variable, which may be of independent interest. We extend our estimator to approximate the stationary mass placed on elements occurring with small frequency in the trajectory. Finally, we demonstrate the efficacy of our estimators both in simulations on canonical chains and on sequences constructed from natural language text.

stat.ML↗

arXiv2023

How good is Good-Turing for Markov samples?

The Good-Turing (GT) estimator for the missing mass (i.e., total probability of missing symbols) in $n$ samples is the number of symbols that appeared exactly once divided by $n$. For i.i.d. samples, the bias and squared-error risk of the GT estimator can be shown to fall as $1/n$ by bounding the expected error uniformly over all symbols. In this work, we study convergence of the GT estimator for missing stationary mass (i.e., total stationary probability of missing symbols) of Markov samples on an alphabet $\mathcal{X}$ with stationary distribution $[π_x:x \in \mathcal{X}]$ and transition probability matrix (t.p.m.) $P$. This is an important and interesting problem because GT is widely used in applications with temporal dependencies such as language models assigning probabilities to word sequences, which are modelled as Markov. We show that convergence of GT depends on convergence of $(P^{\sim x})^n$, where $P^{\sim x}$ is $P$ with the $x$-th column zeroed out. This, in turn, depends on the Perron eigenvalue $λ^{\sim x}$ of $P^{\sim x}$ and its relationship with $π_x$ uniformly over $x$. For randomly generated t.p.ms and t.p.ms derived from New York Times and Charles Dickens corpora, we numerically exhibit such uniform-over-$x$ relationships between $λ^{\sim x}$ and $π_x$. This supports the observed success of GT in language models and practical text data scenarios. For Markov chains with rank-2, diagonalizable t.p.ms having spectral gap $β$, we show minimax rate upper and lower bounds of $1/(nβ^5)$ and $1/(nβ)$, respectively, for the estimation of stationary missing mass. This theoretical result extends the $1/n$ minimax rate for i.i.d. or rank-1 t.p.ms to rank-2 Markov, and is a first such minimax rate result for missing mass of Markov samples.

cs.IT↗

arXiv2023

Missing $g$-mass: Investigating the Missing Parts of Distributions

Estimating the underlying distribution from \textit{iid} samples is a classical and important problem in statistics. When the alphabet size is large compared to number of samples, a portion of the distribution is highly likely to be unobserved or sparsely observed. The missing mass, defined as the sum of probabilities $\text{Pr}(x)$ over the missing letters $x$, and the Good-Turing estimator for missing mass have been important tools in large-alphabet distribution estimation. In this article, given a positive function $g$ from $[0,1]$ to the reals, the missing $g$-mass, defined as the sum of $g(\text{Pr}(x))$ over the missing letters $x$, is introduced and studied. The missing $g$-mass can be used to investigate the structure of the missing part of the distribution. Specific applications for special cases such as order-$α$ missing mass ($g(p)=p^α$) and the missing Shannon entropy ($g(p)=-p\log p$) include estimating distance from uniformity of the missing distribution and its partial estimation. Minimax estimation is studied for order-$α$ missing mass for integer values of $α$ and exact minimax convergence rates are obtained. Concentration is studied for a class of functions $g$ and specific results are derived for order-$α$ missing mass and missing Shannon entropy. Sub-Gaussian tail bounds with near-optimal worst-case variance factors are derived. Two new notions of concentration, named strongly sub-Gamma and filtered sub-Gaussian concentration, are introduced and shown to result in right tail bounds that are better than those obtained from sub-Gaussian concentration.

math.ST↗

arXiv2023

Generalizations and Extensions to Lifting Constructions for Coded Caching

Coded caching is a technique for achieving increased throughput in cached networks during peak hours. Placement delivery arrays (PDAs) capture both placement and delivery scheme requirements in coded caching in a single array. Lifting is a method of constructing PDAs, where entries in a small base PDA are replaced with constituent PDAs that satisfy a property called Blackburn-compatibility. We propose two new constructions for Blackburn-compatible PDAs including a novel method for lifting Blackburn-compatible PDAs to obtain new sets of Blackburn-compatible PDAs. Both of these constructions improve upon previous tradeoffs between rate, memory and subpacketization. We generalize lifting constructions by defining partial Blackburn-compatibility between two PDAs w.r.t. a third PDA. This is a wider notion of Blackburn-compatibility making the original definition a special case. We show that some popular coded caching schemes can be defined as lifting constructions in terms of this extended notion.

cs.IT↗

arXiv2023

Capacity Achieving Codes for an Erasure Queue-Channel

We consider a queue-channel model that captures the waiting time-dependent degradation of information bits as they wait to be transmitted. Such a scenario arises naturally in quantum communications, where quantum bits tend to decohere rapidly. Trailing the capacity results obtained recently for certain queue-channels, this paper aims to construct practical channel codes for the erasure queue-channel (EQC) -- a channel characterized by highly correlated erasures, governed by the underlying queuing dynamics. Our main contributions in this paper are twofold: (i) We propose a generic `wrapper' based on interleaving across renewal blocks of the queue to convert any capacity-achieving block code for a memoryless erasure channel to a capacity-achieving code for the EQC. Next, due to the complexity involved in implementing interleaved systems, (ii) we study the performance of LDPC and Polar codes without any interleaving. We show that standard Arıkan's Polar transform polarizes the EQC for certain restricted class of erasure probability functions. We also highlight some possible approaches and the corresponding challenges involved in proving polarization of a general EQC.

cs.IT↗

arXiv2022

Lifting Constructions of PDAs for Coded Caching with Linear Subpacketization

Coded caching is a technique where multicasting and coding opportunities are utilized to achieve better rate-memory tradeoff in cached networks. A crucial parameter in coded caching is subpacketization, which is the number of parts a file is to be split into for coding purposes. The original Maddah-Ali-Niesen scheme has order-optimal rate at a subpacketization growing exponentially with the number of users. In contrast, placement and delivery schemes in coded caching, designed using placement delivery arrays (PDAs), can have linear subpacketization with a penalty in rate. In this work, we propose several constructions of efficient PDAs through lifting, where a base PDA is expanded by replacing each entry by another PDA. By proposing and using the notion of Blackburn-compatibility of PDAs, we provide multiple lifting constructions with increasing coding gains. We compare the constructed coded caching schemes with other existing schemes for moderately high number of users and show that the proposed constructions are versatile and achieve a good rate-memory tradeoff at low subpacketizations.

cs.IT↗

arXiv2022

Missing Mass Estimation from Sticky Channels

Distribution estimation under error-prone or non-ideal sampling modelled as "sticky" channels have been studied recently motivated by applications such as DNA computing. Missing mass, the sum of probabilities of missing letters, is an important quantity that plays a crucial role in distribution estimation, particularly in the large alphabet regime. In this work, we consider the problem of estimation of missing mass, which has been well-studied under independent and identically distributed (i.i.d) sampling, in the case when sampling is "sticky". Precisely, we consider the scenario where each sample from an unknown distribution gets repeated a geometrically-distributed number of times. We characterise the minimax rate of Mean Squared Error (MSE) of estimating missing mass from such sticky sampling channels. An upper bound on the minimax rate is obtained by bounding the risk of a modified Good-Turing estimator. We derive a matching lower bound on the minimax rate by extending the Le Cam method.

math.ST↗

arXiv2020

Convergence of Chao Unseen Species Estimator

Support size estimation and the related problem of unseen species estimation have wide applications in ecology and database analysis. Perhaps the most used support size estimator is the Chao estimator. Despite its wide spread use, little is known about its theoretical properties. We analyze the Chao estimator and show that its worst case mean squared error (MSE) is smaller than the MSE of the plug-in estimator by a factor of $\mathcal{O} ((k/n)^4)$, where $k$ is the maximum support size and $n$ is the number of samples. Our main technical contribution is a new method to analyze rational estimators for discrete distribution properties, which may be of independent interest.

math.ST↗

arXiv2019

Subpacketization in Coded Caching with Demand Privacy

Coded caching is a technique where we utilize multi-casting opportunities to reduce rate in cached networks. One limitation of coded caching schemes is that they reveal the demands of all users to their peers. In this work, we consider coded caching schemes that assure privacy for user demands with a particular focus on reducing subpacketization. For the 2-user, 2-file case, we present a new linear demand-private scheme with the lowest possible subpacketization. This is done by presenting the scheme explicitly and proving impossibility results under lower subpacketization. Additionally, when only partial privacy is required, we show that subpacketization can be significantly reduced when there are a large number of files.

cs.IT↗

arXiv2018

Protograph LDPC Codes with Block Thresholds: Extension to Degree-1 and Generalized Nodes

Protograph low-density-parity-check (LDPC) are considered to design near-capacity low-rate codes over the binary erasure channel (BEC) and binary additive white Gaussian noise (BIAWGN) channel. For protographs with degree-one variable nodes and doubly-generalized LDPC (DGLDPC) codes, conditions are derived to ensure equality of bit-error threshold and block-error threshold. Using this condition, low-rate codes with block-error threshold close to capacity are designed and shown to have better error rate performance than other existing codes.

cs.IT↗

arXiv2017

Irregular Recovery and Unequal Locality for Locally Recoverable Codes with Availability

A code is said to be a Locally Recoverable Code (LRC) with availability if every coordinate can be recovered from multiple disjoint sets of other coordinates called recovering sets. The vector of sizes of recovering sets of a coordinate is called its recovery profile. In this work, we consider LRCs with availability under two different settings: (1) irregular recovery: non-constant recovery profile that remains fixed for all coordinates, (2) unequal locality: regular recovery profile that can vary with coordinates. For each setting, we derive bounds for the minimum distance that generalize previously known bounds to the cases of irregular or varying recovery profiles. For the case of regular and fixed recovery profile, we show that a specific Tamo-Barg polynomial-evaluation construction is optimal for all-symbol locality, and we provide parity-check matrix constructions for information locality with availability.

cs.IT↗

arXiv2017

Minimax Risk for Missing Mass Estimation

The problem of estimating the missing mass or total probability of unseen elements in a sequence of $n$ random samples is considered under the squared error loss function. The worst-case risk of the popular Good-Turing estimator is shown to be between $0.6080/n$ and $0.6179/n$. The minimax risk is shown to be lower bounded by $0.25/n$. This appears to be the first such published result on minimax risk for estimation of missing mass, which has several practical and theoretical applications.

cs.IT↗

arXiv2016

Dual Capacity Upper Bounds for Noisy Runlength Constrained Channels

Binary-input memoryless channels with a runlength constrained input are considered. Upper bounds to the capacity of such noisy runlength constrained channels are derived using the dual capacity method with Markov test distributions satisfying the Karush-Kuhn-Tucker (KKT) conditions for the capacity-achieving output distribution. Simplified algebraic characterizations of the bounds are presented for the binary erasure channel (BEC) and the binary symmetric channel (BSC). These upper bounds are very close to achievable rates, and improve upon previously known feedback-based bounds for a large range of channel parameters. For the binary-input Additive White Gaussian Noise (AWGN) channel, the upper bound is simplified to a small-scale numerical optimization problem. These results provide some of the simplest upper bounds for an open capacity problem that has theoretical and practical relevance.

cs.IT↗

arXiv2015

Capacity Bounds for Discrete-Time, Amplitude-Constrained, Additive White Gaussian Noise Channels

The capacity-achieving input distribution of the discrete-time, additive white Gaussian noise (AWGN) channel with an amplitude constraint is discrete and seems difficult to characterize explicitly. A dual capacity expression is used to derive analytic capacity upper bounds for scalar and vector AWGN channels. The scalar bound improves on McKellips' bound and is within 0.1 bits of capacity for all signal-to-noise ratios (SNRs). The two-dimensional bound is within 0.15 bits of capacity provably up to 4.5 dB, and numerical evidence suggests a similar gap for all SNRs.

cs.IT↗