Distributional Semantics, Holism, and the Instability of Meaning

Abstract

Large Language Models are built on the so-called distributional semantic approach to linguistic meaning that has the distributional hypothesis at its core. The distributional hypothesis involves a holistic conception of word meaning: the meaning of a word depends upon its relations to other words in the model. A standard objection to holism is the charge of instability: any change in the meaning properties of a linguistic system (a human speaker, for example) would lead to many changes or a complete change in the entire system. We examine whether the instability objection poses a problem for distributional models of meaning. First, we distinguish between distinct forms of instability that these models could exhibit, and argue that only one such form is relevant for understanding the relation between instability and communication: what we call differential instability. Differential instability is variation in the relative distances between points in a space, rather than variation in the absolute position of those points. We distinguish differential and absolute instability by constructing two of our own smaller language models. We demonstrate the two forms of instability by showing these models change as the corpora they are constructed from increase in size. We argue that the instability that these models display is constrained by the structure and scale of relationships between words, such that the resistance to change for a word is roughly proportional to its frequent and consistent use within the language system. The differential instability that language models exhibit allows for productive forms of meaning change while not leading to the problems raised by the instability objection.