Searcharxiv⌕ Search

arXiv · 2611.10565

Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition

Abstract

Computational models of cognition have traditionally relied on abstract symbolic inputs that approximate sensory experiences with simplified vector representations. This simplification reflects historical computational constraints and a long-held assumption that the details of sensory realism are incidental to the study of cognitive processes. Recent advances have enabled the development of sensory-realistic models capable of performing diverse cognitive tasks, yet whether sensory realism meaningfully shapes cognition has not been formally established. We address this gap through a large-scale empirical comparison of abstract and sensory-realistic neural network models trained on a battery of memory-dependent decision-making tasks. Our results show that sensory-realistic models mirror human behavioral response patterns significantly more than their abstract counterparts, though a substantial gap to human internal behavioral consistency remains even for the best tested models. By analyzing the internal dynamics of these systems, we find that the learned recurrent weights are not the main source of this divergence. In fact, frozen abstract-trained weights driven by sensory-realistic inputs recover the human consistency of sensory-realistic models. The resulting dynamics are also within the seed-to-seed variability of sensory-realistic models. Together, our findings argue that across a broad set of visual working-memory tasks, sensory realism accounts for a substantial portion of the model-human behavioral agreement, more than the choice of sequence architecture and comparable to the choice of vision encoder. These results suggest that findings obtained from abstract task models in neuroscience should be more carefully interpreted in light of their potential behavioral limitations.

Explore related subjects

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Hao Yuan Bai, Xiaoxuan Lei, Zihan Weng, Motahareh Pourrahimi, Lucas Gomez, Pouya Bashivan. 2026-11-07. Pixels over Symbols: Sensory Realism Improves Behavioral Alignment in Models of Cognition. https://searcharxiv.com/abs/2611.10565

Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Searcharxiv by Indiciae: BibTeX · RIS.

KEEP EXPLORING

Related papers

Overlooked weak structural connections support human cognition under nonlinear connectome scaling

Human cognition depends on large scale communication constrained by white matter architecture. Although weak connections are abundant in mammalian connectomes, they have long been treated as noise and downweighted because of tractography uncertainty in the human brain, and their relevance to human cognition and large scale functional organization remains unresolved. Across multiple datasets and tractography pipelines, we show that, when tractography derived connection weights are interpreted through a nonlinear weighting framework, weak connections make measurable contributions to cognitive predictions, functional connectivity simulation, and structure function coupling. These effects are selective, nonlinear weighting improves the prediction of general cognitive ability and memory more than that of crystallized intelligence or processing speed, consistent with the notion that weak connections preferentially expand the modal repertoire of brain structural networks to enhance both large scale integration and fine grained segregation, thereby supporting the functional balance essential for diverse cognitive abilities. Importantly, these effects are replicated in a reliability aware connectome generated by integrating two post tractography filtering methods, in which preserving weak links consistently outperforms conventional thresholding strategy in structure function coupling and model based simulation. Finally, we show that weak connections contain functional information organized along system level and gene expression gradients. In particular, weak connections with negative gene coexpression predominantly link visual, motor with limbic regions and exert a disproportionately large influence on brain function than those with positive gene coexpression.

q-bio.NC↗

Quantum-Like Models of Cognition and Decision Making: Open-Systems and Gorini--Kossakowski--Sudarshan--Lindblad Dynamics

This paper starts with surveying the evolution of quantum-like models of cognition and decision making, transitioning from static kinematic representations to a robust dynamical framework based on open quantum systems. We provide a comprehensive analysis of the Gorini-Kossakowski-Sudarshan-Lindblad (GKSL) master equation's application in cognitive psychology and decision making, illustrating how it models mental state evolution as a dissipative process influenced by an informational environment. We categorize dynamical regimes into Passive and Active Hamiltonians, demonstrating how non-commutation with projections on decision basis serves as a mathematical signature of cognitive agency and Quantum Escape from classical equilibria. The utility of this framework is further explored through its ability to stabilize non-Nash outcomes in strategic games, such as the Prisoner's Dilemma. Building upon this dynamical foundation, we identify ``cognitive beats'' as a signature of the internal struggle between competing ``flows of mind'' deliberated at approximately equal frequencies. Distinct from the damped oscillations of simple interference, these beats emerge from a structural tension between Liouvillian channels that generates a secondary, slow-scale modulation of conviction. This beat envelope dictates the timing of peak readiness and hesitation, providing a mathematical map of the transition between conflicting cognitive states. By resolving these nested time scales, we provide a new spectral diagnostic for the depth of cognitive agency and the complexity of the underlying deliberation process. This paper develops a theoretical framework linking GKSL dynamics with quantum-like cognition and decision-making (QCDM), highlighting how dissipative quantum models can capture features of human thought and decision processes.

q-bio.NC↗

A 1000-hour EEG-EMG-audio dataset of Japanese speech production

We present a multimodal dataset of 1020 hours of simultaneously recorded scalp electroencephalography (EEG), facial electromyography (EMG), and speech audio from three healthy native Japanese speakers during open-vocabulary overt speech, covert repetition and speech listening. Recordings were acquired in many sessions over several months with three EEG systems of 61-128 channels, an ultra-high-density system (g.Pangolin) and two cap-type systems (g.SCARABEO and eego sports). Each session provides time-synchronized EEG, facial EMG, and audio, together with speech-event annotations and transcriptions. Although collected with speech decoding as a primary motivation, the dataset also supports work on multimodal signal processing, artifact modeling, longitudinal and cross-device adaptation, and EEG representation learning. Technical validation used power spectral density and event-related potential analyses to characterise spectral profiles and time-locked responses across participants, devices, and tasks. The dataset is available in Brain Imaging Data Structure (BIDS) format from Harvard Dataverse under controlled access for research use. Its speech audio is voice-masked.

q-bio.NC↗