Morimoto, T.; Akbarinia, A.; Storrs, K. R.; Cheeseman, J. R.; Smithson, H. E.; Gegenfurtner, K. R.; Fleming, R. W.

A key goal of vision science is to uncover the computations involved in perceiving visual properties like colour, curvature, or glossiness. Here, we used machine learning as a data-driven tool to identify potential computations of gloss perception. We generated thousands of object images using computer graphics, varying lighting, shapes, and viewpoints, and experimentally measured perceived glossiness for each image. Observers showed curious patterns of agreement and disagreement with the physical reflectance in their gloss estimates, yet their judgments remained highly consistent both within and across individuals. We then compared two sets of neural networks: one set trained to mimic the human responses (human-like networks) and another trained on physical labels to approximate physical reality (ground-truth networks). We progressively reduced the size of the networks to identify the minimum computations capable of meeting the two objective functions. While quite deep networks were required to estimate physical reflectance, we found that surprisingly shallow networks, with as few as three convolutional layers, could accurately replicate human gloss judgments. Indeed, even a miniscule network with just a single filter could predict human judgments better than the best ground-truth network. The human-like networks also successfully predicted some known perceptual gloss effects beyond the training range. Our findings suggest that humans do not judge material properties through complex computations such as inverse optics; instead, gloss perception arises from simpler computations useful for other visual tasks.