Multimodal development in children's narrative speech: Evidence for tight gesture-speech temporal alignment patterns as early as 5 years old

Abstract

Purpose: This study aims to analyze the development of gesture-speech temporal alignment patterns in children's narrative speech from a longitudinal perspective and, specifically, the potential differences between different gesture types, namely, gestures that imagistically portray or refer to semantic content in speech (i.e., referential gestures) and those that lack semantic content (i.e., non-referential gestures). Method: This study uses an audiovisual corpus of narrative productions (n = 332) from 83 children (43 girls, 40 boys) who participated in a narrative retelling task at two time points in development (at 5-6 and 7-9 years of age). The 332 narratives were coded for both manual co-speech gesture types and prosody. Gestural annotations included gesture phasing (i.e., preparation, stroke, hold, and recovery) and gesture types (in terms of referentiality, i.e., referential and non-referential), whereas prosodic annotations included pitch-accented syllables. Results: Results revealed that by ages 5-6 years, children already temporally aligned the stroke of both referential and non-referential gestures with pitch-accented syllables, showing no significant differences between these two gesture types. Conclusions: The results of the present study contribute to the view that both referential and non-referential gestures are aligned with pitch accentuation, and therefore, this is not only a characteristic of non-referential gestures. Our results also add support to McNeill's phonological synchronization rule from a developmental perspective and indirectly back up recent theories about the biomechanics of gesture-speech alignment, suggesting that this is an inherent ability of oral communication

The authors would like to acknowledge the financial support awarded by the Spanish Ministry of Science, Innovation and Universities, Agencia Estatal de Investigación and Fondo Europeo de Desarrollo Regional (PGC2018- 097007-B-100: “Multimodal Language Learning: Prosodic and Gestural Integration in Pragmatic and Phonological Development,” and PID2021-123823NB-I00: “Multimodal Communication: The Integration of Prosody and Gesture in Human Communication and in Language Learning”); by the Generalitat de Catalunya (2017 SGR_971 and 2021 SGR_922); and by the GEstures and Head Movements in language research network, funded by the Independent Research Fund Denmark (9055-00004B)