Crossmodal semantics in memory: scoping review and meta-analyses of multisensory effects in short-term and episodic memory systems

Abstract

The human brain represents objects and events in the environment by binding together their defining semantic attributes across the senses (e.g., vision, hearing, touch). Semantic relationships between these attributes in different senses, or crossmodal semantic relationships, are fundamental to carving out meaningful categories and to encode and store experiences in the form of memories for later retrieval. Unsurprisingly, the subject of crossmodal semantic interactions in human memory has been on the agenda of researchers interested in multisensory processes for several decades now and there appears to be a renewed wave of interest in the field currently. By and large, the central question has been whether or not memories for events with crossmodally congruent semantic attributes are better remembered. Nevertheless, this research area has been characterized by mixed methodological approaches, inconsistent outcomes, and alternative theoretical interpretations, with few attempts at synthesis. Here, we examine the past 30 years of research on the topic, covering short-term as well as episodic memory systems. First, we garner existing evidence in a systematic scoping review of studies, complemented by meta-analyses. Then, we provide a synthesis highlighting outstanding empirical questions and potential contradictions between competing theoretical interpretations. With some exceptions, there is abundant support for the hypothesis that crossmodally congruent events are better remembered than single-modality or crossmodal but incongruent events. Nevertheless, the mechanisms underlying this multisensory benefit and its theoretical interpretation are still the subject of substantial debates. We propose avenues to resolve these issues and advance current knowledge in this burgeoning research area.

Paul Verhaeghen served as action editor.The data sets and scripts used in this review are available on the Open Science Framework project at https://osf.io/35qsf, including (a) a spreadsheet with the detailed Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for scoping reviews article selection process, from initial search output to the final article sample, and (b) the datafiles and scripts used for the meta-analyses.The preparation of the article was supported by Generalitat de Catalunya, Agència de Gestió d'Ajuts Universitaris i de Recerca Grant AGAUR 2021 SGR 00911, Institució Catalana de Recerca i Estudis Avançats, Agencia Española de Investigación (AEI) Grant PID2022-137277NB-I00 AEI/FEDER awarded to Salvador Soto-Faraco, and Arts and Humanities Research Council (AHRC) Grant AH/L007053/1 awarded to Charles Spence. The authors are grateful to Miguel Vadillo for precious advice on conducting meta-analyses and feedback on an earlier version of this article.Salvador Soto-Faraco played a lead role in data curation, methodology, project administration, software, visualization, and writing-original draft and an equal role in conceptualization, funding acquisition, investigation, resources, and writing-review and editing. Charles Spence played a supporting role in visualization and an equal role in conceptualization, funding acquisition, and writing-review and editing.