Marks, Channels, and Dead Ends: Stop Running Graphical Perception Studies and Start Modeling Visualizations as Images

Abstract

Graphical perception studies are the visualization community's preferred tool for evaluating visualizations. By measuring how accurately people interpret arrangements of visual marks and channels, they aim to establish best practices for visual encoding. We argue that this model is fundamentally flawed, and no amount of additional empirical studies will fix it. Visualization theory frames effectiveness at the level of the encoder: which data-to-visual mappings work best in a given context. Human perception, however, operates as a fundamentally different decoder at the level of retinal images. This encoder-decoder asymmetry means that experimental results and guidelines can be poor predictors of perceptual performance. Moreover, the image reaching the visual system emerges from interactions among encoding rules, input data, and micro-design parameters--factors largely invisible to encoding theory. Consequently, small changes in data distributions or design variations can substantially alter perception even when the nominal encoding remains unchanged. We argue that visualizations should instead be studied as images and evaluated using computational models of human vision that take pixels as input. Such models capture the perceptual representations the visual system actually constructs, shifting evaluation toward the decoder rather than abstract encoding specifications. This approach is scalable, human-grounded, and sensitive to emergent image properties that both encoding theory and graphical perception studies miss. We first describe weaknesses of the current paradigm and propose a theory of visualization perception grounded in summary-statistical accounts of vision. We then show how image-based vision models can predict visualization discriminability in scatterplots while reproducing established results. We close by outlining a research agenda for vision-based visualization evaluation.