arXiv subjects

Aslan Tchamkerten

Publications and source records attributed to Aslan Tchamkerten.

At least 19 recordsLinked to original sources

arXiv2026

Effective Context in Transformers: An Analysis of Fragmentation and Tokenization

Transformers predict over a representation of a sequence. The same data can be written as bytes, characters, or subword tokens, and these representations may be lossless. Yet, under a fixed context window, they need not expose the same information to the model. This raises a basic question: how does the choice of representation change what a finite-context predictor can achieve? We study this question on Markov sources and uncover two complementary phenomena. First, we observe that moving to smaller representation units can hurt prediction even when the context window is enlarged to cover the relevant source history. To explain this, we introduce fragmentation: a lossless recoding that replaces each source symbol by several smaller units. We prove that fragmentation can strictly increase the optimal finite-context log-loss, showing that the gap is not merely an optimization or capacity issue, but can be intrinsic to the representation. This gives a theoretical account of the finite-context gap observed in byte- and character-level models such as ByT5 and CANINE relative to subword-tokenized models. Second, we study the opposite direction: greedy tokenization -- BPE, WordPiece, and related methods -- which groups source symbols into larger units. We show that tokenization can make a short token window behave like a longer source-context window, and we give a loss guarantee describing when this is achievable. The guarantee depends on how reliably token windows span the needed source history, together with the compression rate of the tokenizer. This also yields a simple diagnostic for real tokenizers: measuring how much source context a fixed token window reliably contains. Together, the two directions establish a finite-context information-theoretic framework for reasoning about representation choices in Transformers.

cs.LG↗

arXiv2025

Compression with Privacy-Preserving Random Access

We show that an i.i.d.\ binary source sequence $X_1,\ldots,X_n$ can be losslessly compressed at any rate above entropy while ensuring that the decoding of any $X_i$ reveals no information about the remaining symbols $\{X_j : j \neq i\}$. This problem reduces to a marginal consistency problem induced by the simultaneous privacy and reliability constraints. To address it, we develop a technique based on a geometric representation of codeword distributions, which may be of independent interest.

cs.IT↗

arXiv2023

Feedback Increases the Capacity of Queues with Bounded Service Times

In the "Bits Through Queues" paper, it was hypothesized that full feedback always increases the capacity of first-in-first-out queues, except when the service time distribution is memoryless. More recently, a non-explicit sufficient condition under which feedback increases capacity was provided, along with simple examples of service times meeting this condition. While this condition yields examples where feedback is beneficial, it does not offer explicit structural properties of such service times. In this paper, we show that full feedback increases capacity whenever the service time has bounded support. This is achieved by investigating a generalized notion of feedback, with full feedback and weak feedback as particular cases.

cs.IT↗

arXiv2022

Local Decoding in Distributed Compression

It was recently shown that the lossless compression of a single source $X^n$ is achievable with a notion of strong locality; any $X_i$ can be decoded from a constant number of compressed bits, with a vanishing in $n$ probability of error. By contrast, we show that for two separately encoded sources $(X^n,Y^n)$, lossless compression and strong locality is generally not possible. Specifically, we show that for the class of ``confusable'' sources, strong locality cannot be achieved whenever one of the sources is compressed below its entropy. Irrespective of $n$, for some index $i$ the probability of error of decoding $(X_i,Y_i)$ is lower bounded by $2^{-O(d)}$, where $d$ denotes the number of compressed bits accessed by the local decoder. Conversely, if the source is not confusable, strong locality is possible even if one of the sources is compressed below its entropy. Results extend to an arbitrary number of sources.

cs.IT↗

arXiv2021

Reconfigurable Adaptive Channel Sensing

Channel sensing consists of probing the channel from time to time to check whether or not it is active - say, because of an incoming message. When communication is sparse with information being sent once in a long while, channel sensing becomes a significant source of energy consumption. How to reliably detect messages while minimizing the receiver energy consumption? This paper addresses this problem through a reconfigurable scheme, referred to as AdaSense, which exploits the dependency between the receiver noise figure (i.e., the receiver added noise) and the receiver power consumption; a higher power typically translates into less noisy channel observations. AdaSense begins in a low power low reliability mode and makes a first tentative decision based on a few channel observations. If a message is declared, it switches to a high power high reliability mode to confirm the decision, else it sleeps for the entire duration of the second phase. Compared to prominent detection schemes such as the BMAC protocol, AdaSense provides relative energy gains that grow unbounded in the small probability of false-alarm regime, as communication gets sparser. In the non-asymptotic regime energy gains are 30% to 75% for communication scenarios typically found in the context of wake-up receivers.

cs.IT↗

arXiv2021

Zero-Error Sum Modulo Two with a Common Observation

This paper investigates the classical modulo two sum problem in source coding, but with a common observation: a transmitter observes $(X,Z)$, the other transmitter observes $(Y,Z)$, and the receiver wants to compute $X \oplus Y$ without error. Through a coupling argument, this paper establishes a new lower bound on the sum-rate when $X-Z-Y$ forms a Markov chain.

cs.IT↗

arXiv2021

Error-Correction for Sparse Support Recovery Algorithms

Consider the compressed sensing setup where the support $s^*$ of an $m$-sparse $d$-dimensional signal $x$ is to be recovered from $n$ linear measurements with a given algorithm. Suppose that the measurements are such that the algorithm does not guarantee perfect support recovery and that true features may be missed. Can they efficiently be retrieved? This paper addresses this question through a simple error-correction module referred to as LiRE. LiRE takes as input an estimate $s_{in}$ of the true support $s^*$, and outputs a refined support estimate $s_{out}$. In the noiseless measurement setup, sufficient conditions are established under which LiRE is guaranteed to recover the entire support, that is $s_{out}$ contains $s^*$. These conditions imply, for instance, that in the high-dimensional regime LiRE can correct a sublinear in $m$ number of errors made by Orthogonal Matching Pursuit (OMP). The computational complexity of LiRE is $O(mnd)$. Experimental results with random Gaussian design matrices show that LiRE substantially reduces the number of measurements needed for perfect support recovery via Compressive Sampling Matching Pursuit, Basis Pursuit (BP), and OMP. Interestingly, adding LiRE to OMP yields a support recovery procedure that is more accurate and significantly faster than BP. This observation carries over in the noisy measurement setup. Finally, as a standalone support recovery algorithm with a random initialization, experiments show that LiRE's reconstruction performance lies between OMP and BP. These results suggest that LiRE may be used generically, on top of any suboptimal baseline support recovery algorithm, to improve support recovery or to operate with a smaller number of measurements, at the cost of a relatively small computational overhead. Alternatively, LiRE may be used as a standalone support recovery algorithm that is competitive with respect to OMP.

cs.IT↗

arXiv2021

Approximating Probability Distributions by ReLU Networks

How many neurons are needed to approximate a target probability distribution using a neural network with a given input distribution and approximation error? This paper examines this question for the case when the input distribution is uniform, and the target distribution belongs to the class of histogram distributions. We obtain a new upper bound on the number of required neurons, which is strictly better than previously existing upper bounds. The key ingredient in this improvement is an efficient construction of the neural nets representing piecewise linear functions. We also obtain a lower bound on the minimum number of neurons needed to approximate the histogram distributions.

cs.LG↗

arXiv2020

$O(\log \log n)$ Worst-Case Local Decoding and Update Efficiency for Data Compression

This paper addresses the problem of data compression with local decoding and local update. A compression scheme has worst-case local decoding $d_{wc}$ if any bit of the raw file can be recovered by probing at most $d_{wc}$ bits of the compressed sequence, and has update efficiency of $u_{wc}$ if a single bit of the raw file can be updated by modifying at most $u_{wc}$ bits of the compressed sequence. This article provides an entropy-achieving compression scheme for memoryless sources that simultaneously achieves $ O(\log\log n) $ local decoding and update efficiency. Key to this achievability result is a novel succinct data structure for sparse sequences which allows efficient local decoding and local update. Under general assumptions on the local decoder and update algorithms, a converse result shows that $d_{wc}$ and $u_{wc}$ must grow as $ Ω(\log\log n) $.

cs.IT↗

arXiv2019

Local Decode and Update for Big Data Compression

This paper investigates data compression that simultaneously allows local decoding and local update. The main result is a universal compression scheme for memoryless sources with the following features. The rate can be made arbitrarily close to the entropy of the underlying source, contiguous fragments of the source can be recovered or updated by probing or modifying a number of codeword bits that is on average linear in the size of the fragment, and the overall encoding and decoding complexity is quasilinear in the blocklength of the source. In particular, the local decoding or update of a single message symbol can be performed by probing or modifying a constant number of codeword bits. This latter part improves over previous best known results for which local decodability or update efficiency grows logarithmically with blocklength.

cs.IT↗

arXiv2019

Somatic mutations render human exome and pathogen DNA more similar

Immunotherapy has recently shown important clinical successes in a substantial number of oncology indications. Additionally, the tumor somatic mutation load has been shown to associate with response to these therapeutic agents, and specific mutational signatures are hypothesized to improve this association, including signatures related to pathogen insults. We sought to study in silico the validity of these observations and how they relate to each other. We first addressed whether somatic mutations typically involved in cancer may increase, in a statistically meaningful manner, the similarity between common pathogens and the human exome. Our study shows that common mutagenic processes increase, in the upper range of biologically plausible frequencies, the similarity between cancer exomes and pathogen DNA at a scale of 12-16 nucleotide sequences and established that this increased similarity is due to the specific mutation distribution of the considered mutagenic processes. Next, we studied the impact of mutation rate and showed that increasing mutation rate generally results in an increased similarity between the cancer exome and pathogen DNA, at a scale of 4-5 amino acids. Finally, we investigated whether the considered mutational processes result in amino-acid changes with functional relevance that are more likely to be immunogenic. We showed that functional tolerance to mutagenic processes across species generally suggests more resilience to mutagenic processes that are due to exposure to elements of nature than to mutagenic processes that are due to exposure to cancer-causing artificial substances. These results support the idea that recognition of pathogen sequences as well as differential functional tolerance to mutagenic processes may play an important role in the immune recognition process involved in tumor infiltration by lymphocytes.

q-bio.GN↗

arXiv2018

Capacity-Achieving Input Distribution in Per-Sample Zero-Dispersion Model of Optical Fiber

The per-sample zero-dispersion channel model of the optical fiber is considered. It is shown that capacity is uniquely achieved by an input probability distribution that has continuous uniform phase and discrete amplitude that takes on finitely many values. This result holds when the channel is subject to general input cost constraints, that include a peak amplitude constraint and a joint average and peak amplitude constraint.

cs.IT↗

arXiv2018

Bounds on the Approximation Power of Feedforward Neural Networks

The approximation power of general feedforward neural networks with piecewise linear activation functions is investigated. First, lower bounds on the size of a network are established in terms of the approximation error and network depth and width. These bounds improve upon state-of-the-art bounds for certain classes of functions, such as strongly convex functions. Second, an upper bound is established on the difference of two neural networks with identical weights but different activation functions.

cs.LG↗

arXiv2017

Sampling Constrained Asynchronous Communication: How to Sleep Efficiently

The minimum energy, and, more generally, the minimum cost, to transmit one bit of information has been recently derived for bursty communication when information is available infrequently at random times at the transmitter. Furthermore, it has been shown that even if the receiver is constrained to sample only a fraction $ρ\in (0,1]$ of the channel outputs, there is no capacity penalty. That is, for any strictly positive sampling rate $ρ>0$, the asynchronous capacity per unit cost is the same as under full sampling, i.e., when $ρ=1$. Moreover, there is no penalty in terms of decoding delay. The above results are asymptotic in nature, considering the limit as the number $B$ of bits to be transmitted tends to infinity, while the sampling rate $ρ$ remains fixed. A natural question is then whether the sampling rate $ρ(B)$ can drop to zero without introducing a capacity (or delay) penalty compared to full sampling. We answer this question affirmatively. The main result of this paper is an essentially tight characterization of the minimum sampling rate. We show that any sampling rate that grows at least as fast as $ω(1/B)$ is achievable, while any sampling rate smaller than $o(1/B)$ yields unreliable communication. The key ingredient in our improved achievability result is a new, multi-phase adaptive sampling scheme for locating transient changes, which we believe may be of independent interest for certain change-point detection problems.

cs.IT↗

arXiv2017

Second Order Asymptotics for Communication under Strong Asynchronism

The capacity under strong asynchronism was recently shown to be essentially unaffected by the imposed output sampling rate $ρ$ and decoding delay $d$---the elapsed time between when information is available at the transmitter and when it is decoded. This paper examines this result in the finite blocklength regime and shows that, by contrast with capacity, the second order term in the rate expansion is sensitive to both parameters. When the receiver must exactly locate the sent codeword, that is $d=n$ where $n$ denotes blocklength, the second order term in the rate expansion is of order $Θ(1/ρ)$ for any $ρ=O(1/\sqrt{n})$---and $ρ=ω(1/n)$ for otherwise reliable communication is impossible. However, if $ρ=ω(1/\sqrt{n})$ then the second order term is the same as under full sampling and is given by a standard $O(\sqrt{n})$ term whose dispersion constant only depends on the level of asynchronism. This second order term also corresponds to the case of the slightly relaxed delay constraint $d\leq n(1+o(1))$ for any $ρ=ω(1/n)$.

cs.IT↗

arXiv2017

Bits through queues with feedback

In their $1996$ paper Anantharam and Verdú showed that feedback does not increase the capacity of a queue when the service time is exponentially distributed. Whether this conclusion holds for general service times has remained an open question which this paper addresses. Two main results are established for both the discrete-time and the continuous-time models. First, a sufficient condition on the service distribution for feedback to increase capacity under FIFO service policy. Underlying this condition is a notion of weak feedback wherein instead of the queue departure times the transmitter is informed about the instants when packets start to be served. Second, a condition in terms of output entropy rate under which feedback does not increase capacity. This condition is general in that it depends on the output entropy rate of the queue but explicitly depends neither on the queue policy nor on the service time distribution. This condition is satisfied, for instance, by queues with LCFS service policies and bounded service times.

cs.IT↗

arXiv2015

On Cooperation in Multi-Terminal Computation and Rate Distortion

A receiver wants to compute a function of two correlated sources separately observed by two transmitters. One of the transmitters may send a possibly private message to the other transmitter in a cooperation phase before both transmitters communicate to the receiver. For this network configuration this paper investigates both a function computation setup, wherein the receiver wants to compute a given function of the sources exactly, and a rate distortion setup, wherein the receiver wants to compute a given function within some distortion. For the function computation setup, a general inner bound to the rate region is established and shown to be tight in a number of cases: partially invertible functions, full cooperation between transmitters, one-round point-to-point communication, two-round point-to-point communication, and the cascade setup where the transmitters and the receiver are aligned. In particular it is shown that the ratio of the total number of transmitted bits without cooperation and the total number of transmitted bits with cooperation can be arbitrarily large. Furthermore, one bit of cooperation suffices to arbitrarily reduce the amount of information both transmitters need to convey to the receiver. For the rate distortion version, an inner bound to the rate region is exhibited which always includes, and sometimes strictly, the convex hull of Kaspi-Berger's related inner bounds. The strict inclusion is shown via two examples.

cs.IT↗

arXiv2015

Distributed Function Computation Over a Rooted Directed Tree

This paper establishes the rate region for a class of source coding function computation setups where sources of information are available at the nodes of a tree and where a function of these sources must be computed at the root. The rate region holds for any function as long as the sources' joint distribution satisfies a certain Markov criterion. This criterion is met, in particular, when the sources are independent. This result recovers the rate regions of several function computation setups. These include the point-to-point communication setting with arbitrary sources, the noiseless multiple access network with "conditionally independent sources," and the cascade network with Markovian sources.

cs.IT↗