Perceptual distances, not categories, encoded along the human ventral visual pathway

Josh Wilson · January 16, 2026

Neural population representations in ventral visual cortex support a wide range of perceptual behaviors, some of which ostensibly impose competing demands on those representations. For example, it seems that categorization favors representations that are insensitive to fine-grained visual variation and instead emphasize category membership, whereas perceptual difference judgments favor the opposite — sensitivity to detailed visual features while largely ignoring category identity. Here, we ask how visual cortical representations are organized with respect to these opposing perceptual demands. To address this question, we used diffusion-based image synthesis to construct continuous interpolated images between naturalistic objects, yielding stimulus sets that can give rise to distinct perceptual geometries: one characterized by sharp categorical boundaries and another reflecting graded variation in visual features. Using these stimuli, we measured both perceptual distance and category choices and compared the resulting perceptual geometries to representational geometries measured in human ventral visual cortex and deep neural network models. We find that across all stages of the ventral visual hierarchy, cortical representational geometry aligns with perceptual distance judgments rather than with categorical decisions, even in high-level visual cortex. Despite this alignment with perceptual distance, we show that representations in late ventral visual cortex are sufficient to construct both perceptual geometries via different readout mechanisms. Finally, we show that ImageNet-trained neural network models reflect the geometry of ventral visual cortex, where late-stage representations encode perceptual distance, and categorical perceptual geometry emerges only at the final decision stage via readout and softmax operations. Together, our results suggest that representations across ventral visual cortex preserve graded visual featural structure, and that categorization arises primarily through readout from, rather than representation in, sensory cortex.