arXiv subjects

Vicente Opazo

Publications and source records attributed to Vicente Opazo.

3 recordsLinked to original sources

arXiv2026

Polynomial-Time Mistake-Bounded Language Generation

In this paper, we introduce a polynomial-time version of the mistake-bounded language generation (MBLG) framework due to Kleinberg, Peale, and Reingold (2026). We obtain upper and lower bounds for a number of simple families of Boolean functions. Namely, we first observe that families of parities of variables, symmetric functions and 2CNFs are polynomial-time MBLG. We then show that the family of monotone functions with polynomially-many maxterms is polynomial-time MBLG. For instance, disjunctions of literals are monotone Boolean functions with 1 maxterms, and thus polynomial-time MBLG. Under the strong RSA assumption, we show that disjunctions of literals are not polynomial-time MBLG. From the latter result, we deduce that polynomial-time MBLG families are not closed under union, and that there are families that are polynomial-time PAC learnable but not polynomial-time MBLG (again, under the strong RSA assumption). Finally, assuming existence of injective one-way functions, we show that there are polynomial-time MBLG families that are not polynomial-time PAC learnable.

cs.CC↗

arXiv2026

Three Tokens Force Exponential Feature Rank in Nonnegative Kernel Attention

Full attention exposes every token pair, whereas kernel attention compresses a sequence into a fixed-dimensional sketch. We show that this distinction becomes exponential at the first context length containing two competing candidates. On Min-IP over Boolean inputs, rank-one normalized kernel attention solves every sequence of length at most two exactly. In contrast, any single normalized nonnegative kernel-attention head that succeeds on all three-token sequences with error strictly below $1/2$ requires $2^{Ω(m)}$ features, even with arbitrary finite-dimensional tokenwise values and an arbitrary query-dependent affine readout. Dense softmax solves the same task with $m$-dimensional scores and constant temperature. The conclusion survives position-dependent token maps and a causal final query. As context length grows, the lower bound approaches the exact $2^m$-feature realization. Separately, for deterministic multihead, multilayer sketch models whose cross-token channels have finite alphabets, we prove a transcript lower bound linear in the number of independent answers and logarithmic in their alphabet size.

cs.LG↗

arXiv2026

Indexing: the Beginning and the End

We study information bottlenecks in modern deep-learning architectures -- RNNs, softmax transformers, linear-attention transformers and state-space models -- through the lens of the indexing primitive. In this primitive, the input consists of $n$ bits and one integer $i$ from $1$ to $n$ called the index, and the output equals the value of the $i$-th bit. We introduce causal complexity for masked architectures. We show that architectures with low causal complexity cannot solve the indexing primitive in any constant number of layers when the index appears at the end of the input. In particular, this limitation applies to low-parameter RNNs, SSMs and masked linear-attention transformers. In contrast, small softmax transformers can solve it in one layer, while non-masked linear-attention transformers can solve it in 2, which separates them from their masked counterparts. In turn, when the index appears at the beginning, we show that small RNNs are capable of solving this task in 1 layer, while all the other architectures require 2. All our impossibility results are unconditional and apply even to models that employ infinite-precision real arithmetic. Moreover, experiments for up to $n=64$ qualitatively align with our theory: configurations with low-parameter theoretical solutions learn the indexing task easily, while configurations that do not admit such theoretical solutions struggle to learn as the sequence length grows.

cs.LG↗