Audio Outperforms Text for Visual Decoding

Abstract

Decoding visual semantic representations from human brain activity is a significant challenge. While recent zero-shot decoding approaches have improved performance by leveraging aligned image-text datasets, they overlook a fundamental aspect of human cognition: semantic understanding is inherently anchored in the auditory modality of speech, not text. To address this, our study introduces the first comparative framework for evaluating auditory versus textual semantic modalities in zero-shot visual neural decoding. We propose a novel brain-visual-auditory multimodal alignment model that directly utilizes auditory representations to encapsulate semantics, serving as a substitute for traditional textual descriptors. Our experimental results demonstrate that the auditory modality not only surpasses the textual modality in decoding accuracy but also achieves higher computational efficiency. These findings indicate that auditory semantic representations are more closely aligned with neural activity patterns during visual processing. This work reveals the critical and previously underestimated role of auditory semantics in decoding visual cognition and provides new insights for developing brain-computer interfaces that are more congruent with natural human cognitive mechanisms.