Neurons and Cognition (q-bio.NC)

Abstract: Large language models (LLMs), such as the General Pre-trained Transformer (GPT), have shown remarkable performance in various cognitive tasks. However, it remains unclear whether these models have the ability to accurately infer human perceptual representations. Previous research has addressed this question by quantifying correlations between similarity response patterns from humans and LLMs. Although it has been shown that the correlation between humans and LLMs is reasonably high, simple correlation analysis is inadequate to reveal the degree of detailed structural correspondence between humans and LLMs. Here, we use an unsupervised alignment method based on Gromov-Wasserstein optimal transport to assess the equivalence of similarity structures between humans and LLMs in a more detailed manner. As a tractable study, we compare the color similarity structures of humans (color-neurotypical and color-atypical participants) and two GPT models (GPT-3.5 and GPT-4) by examining the similarity structures of 93 colors. Our results show that the similarity structures of color-neurotypical humans can be remarkably well-aligned to that of GPT-4 and, to a lesser extent, to that of GPT-3.5. These results contribute to our understanding of the ability of LLMs to accurately infer human perception, and highlight the potential of unsupervised alignment methods to reveal detailed structural equivalence or differences that cannot be detected by simple correlation analysis.