Enhancement of gesture-speech integration

0.Rating_studies

The program code and results of the two cloze probability rating studies and the gesture rating study.

1.Testing

The program code and participant information for the EEG experiment.

2.Data

The behavioral and the raw EEG data collected during the experiment. It also contains history files summarizing the pre-processing steps in Brain Vision Analyzer.

3.Analyses

The analysis scripts for the planned N400 analysis and the exploratory cluster-based permutation analysis. The latter is divided into: 'seg_bl' (exported data after pre-processing in Brain Vision Analyzer), 'preproc' (same data converted to Fieldtrip format, 'stats' (containing 1stlevel and 2ndlevel), 'materials' (info on EEG montage)

ReadMe

Tree diagram that lists the contents of the archive.

Details

Title: Enhancement of gesture-speech integration
Contributor: Florian Hintz
Genre: Psycholinguistics
Neurobiology
Persistent Identifier: https://hdl.handle.net/1839/95a7a1ab-eed5-4dca-83ef-e64c19097734
Description: In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non-predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.
Description: In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non-predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.

Detailed Metadata

expand all

MPI_Collection
- Title : Enhancement of gesture-speech integration
- DataType : Audio recordings
- DataType : Behavioural data
- DataType : Demographic data
- DataType : EEG data
- DataType : Stimuli
- DataType : Video recordings
- ResearchField : Psycholinguistics
- ResearchField : Neurobiology
- Date : 2023-01-30
- Description
  Description : In face-to-face discourse, listeners exploit cues in the input to generate predictions about upcoming words. Moreover, in addition to speech, speakers produce a multitude of visual signals, such as iconic gestures, which listeners readily integrate with incoming words. Previous studies have shown that processing of target words is facilitated when these are embedded in predictable compared to non-predictable discourses and when accompanied by iconic compared to meaningless gestures. In the present study, we investigated the interaction of both factors. We recorded electroencephalogram from 60 Dutch adults while they were watching videos of an actress producing short discourses. The stimuli consisted of an introductory and a target sentence; the latter contained a target noun. Depending on the preceding discourse, the target noun was either predictable or not. Each target noun was paired with an iconic gesture and a gesture that did not convey meaning. In both conditions, gesture presentation in the video was timed such that the gesture stroke slightly preceded the onset of the spoken target by 130 ms. Our ERP analyses revealed independent facilitatory effects for predictable discourses and iconic gestures. However, the interactive effect of both factors demonstrated that target processing (i.e., gesture-speech integration) was facilitated most when targets were part of predictable discourses and accompanied by an iconic gesture. Our results thus suggest a strong intertwinement of linguistic predictability and non-verbal gesture processing where listeners exploit predictive discourse cues to pre-activate verbal and non-verbal representations of upcoming target words.
- Contributor
  Name : Florian Hintz
  
  Role : author
  
  Role : depositor
  
  Role : researcher
  
  EmailAddress : florian.hintz@uni-marburg.de
- Institution
  Organisation : Max Planck Institute for Psycholinguistics
  
  Organisation : Philipps University of Marburg
- Project
  ProjectName : Enhancement of gesture-speech integration
  
  ProjectDescription : This project investigated the interaction between predictive discourse processing and iconic and non-iconic hand movements (i.e., gestures).