arXiv subjects

Vsevolod Kapatsinski

Publications and source records attributed to Vsevolod Kapatsinski.

2 recordsLinked to original sources

arXiv2026

Exemplars in Disguise: Pure Exemplar Models Mimic Abstraction-First Learning

Whether idiosyncratic, item-specific knowledge is learned before abstract class-level generalizations, or vice versa, is a central question in language learning, with exemplar and abstraction-based theories making opposite predictions. Recent methods have claimed to show that, at least for large language models, abstract knowledge is learned first. We show that these methods fall short: pure memorizer models with no abstract representations can appear, by the same criteria, to learn either item-specific or class-level knowledge first, depending on their sensitivity to individual observations, with the transition point governed by the distributional properties of the input. We further argue that the distinction between item-specific and abstract knowledge may be ill-defined for distributed representations, as a word's class-level properties may not be separable from its item-specific properties.

cs.CL↗

arXiv2026

Transformers perform adaptive partial pooling

Any language model must decide what to say in novel contexts based on information from similar contexts. But what about contexts that are not novel but merely infrequent? In hierarchical regression, the model's predictions for behavior in a context are affected by observations from similar contexts to the extent that 1) the current context is infrequent and 2) different contexts behave similarly. This is called adaptive partial pooling. This paper shows that next-word predictions of a transformer (GPT2) are affected by observations from outside the current context, but this pooling reduces with more training. Pooling is affected by context frequency, context number (type frequency) and context variability in a qualitatively similar way to hierarchical regression. However, there is a "sweet spot" in training at which the transformer best matches the behavior of hierarchical regression. This is the point at which the effect of context frequency on pooling is at maximum.

cs.CL↗