Leyi Lao | Searcharxiv

arXiv2026

Modeling the Arrival Time of Information-Bearing Events in Natural Language

Understanding how information is distributed over time is fundamental to modeling natural language. While existing empirical studies primarily rely on post-hoc analysis of information-theoretic metrics, such as surprisal, little is known about the generation processes under the surface. We propose a novel approach for addressing this gap using the tool of neural temporal point processes (TPPs) by modeling the key information-bearing events in text sequences with learnable event intensity functions. Our model takes the history of event intervals and the contextual semantic representations from LLMs as input, and predicts the future arrival times for two types of events: new-word and high-surprisal-token. The experimental results reveal an interesting distinction in event intensities: LLMs tend to produce new words at significantly higher rates than humans, while high-surprisal tokens do not exhibit the difference. A decaying pattern of intensity over time is found in new-word events for both humans and LLMs, but not in high-surprisal-token events. Our findings revisit and extend previous studies that use simpler methods; provide supplementary evidence for theories about constrained context representation inherent in human language production; and find a new angle to view human and LLM generation differences.

cs.CL↗

arXiv2026

Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models

Backchannels and fillers are important linguistic expressions in dialogue, but often treated as 'noise' to be bypassed in modern transformer-based language models (LMs). Here, we study how they are represented in LMs using three fine-tuning strategies on three dialogue corpora in English and Japanese, in which backchannels and fillers are both preserved and annotated. This allows us to investigate how fine-tuning can help LMs learn these representations. We first apply clustering analysis to the learnt representation of backchannels and fillers, and find increased silhouette scores in representations from fine-tuned models, which suggests that fine-tuning enables LMs to distinguish the nuanced semantic variation in different backchannel and filler use. We also employ natural language generation metrics and qualitative analyses to verify that utterances produced by fine-tuned LMs resemble those produced by humans more closely. Our findings suggest the potential for transforming general LMs into conversational LMs that can produce human-like language more adequately.

cs.CL↗