Spatial orienting in complex audiovisual environments

Davide Nardo, Valerio Santangelo, Emiliano Macaluso

Research output: Contribution to journalArticle

29 Citations (Scopus)

Abstract

Previous studies on crossmodal spatial orienting typically used simple and stereotyped stimuli in the absence of any meaningful context. This study combined computational models, behavioural measures and functional magnetic resonance imaging to investigate audiovisual spatial interactions in naturalistic settings. We created short videos portraying everyday life situations that included a lateralised visual event and a co-occurring sound, either on the same or on the opposite side of space. Subjects viewed the videos with or without eye-movements allowed (overt or covert orienting). For each video, visual and auditory saliency maps were used to index the strength of stimulus-driven signals, and eye-movements were used as a measure of the efficacy of the audiovisual events for spatial orienting. Results showed that visual salience modulated activity in higher-order visual areas, whereas auditory salience modulated activity in the superior temporal cortex. Auditory salience modulated activity also in the posterior parietal cortex, but only when audiovisual stimuli occurred on the same side of space (multisensory spatial congruence). Orienting efficacy affected activity in the visual cortex, within the same regions modulated by visual salience. These patterns of activation were comparable in overt and covert orienting conditions. Our results demonstrate that, during viewing of complex multisensory stimuli, activity in sensory areas reflects both stimulus-driven signals and their efficacy for spatial orienting; and that the posterior parietal cortex combines spatial information about the visual and the auditory modality.

Original languageEnglish
Pages (from-to)1597-1614
Number of pages18
JournalHuman Brain Mapping
Volume35
Issue number4
DOIs
Publication statusPublished - 2014

Fingerprint

Parietal Lobe
Eye Movements
Auditory Cortex
Visual Cortex
Temporal Lobe
Magnetic Resonance Imaging

Keywords

  • Attention
  • Auditory
  • Ecological
  • Eye movements
  • Multisensory
  • Orienting
  • Posterior parietal cortex
  • Saliency
  • Space
  • Visual

ASJC Scopus subject areas

  • Clinical Neurology
  • Anatomy
  • Neurology
  • Radiology Nuclear Medicine and imaging
  • Radiological and Ultrasound Technology
  • Medicine(all)

Cite this

Spatial orienting in complex audiovisual environments. / Nardo, Davide; Santangelo, Valerio; Macaluso, Emiliano.

In: Human Brain Mapping, Vol. 35, No. 4, 2014, p. 1597-1614.

Research output: Contribution to journalArticle

Nardo, Davide ; Santangelo, Valerio ; Macaluso, Emiliano. / Spatial orienting in complex audiovisual environments. In: Human Brain Mapping. 2014 ; Vol. 35, No. 4. pp. 1597-1614.
@article{b93e96156f25436396e1762b0ffbb732,
title = "Spatial orienting in complex audiovisual environments",
abstract = "Previous studies on crossmodal spatial orienting typically used simple and stereotyped stimuli in the absence of any meaningful context. This study combined computational models, behavioural measures and functional magnetic resonance imaging to investigate audiovisual spatial interactions in naturalistic settings. We created short videos portraying everyday life situations that included a lateralised visual event and a co-occurring sound, either on the same or on the opposite side of space. Subjects viewed the videos with or without eye-movements allowed (overt or covert orienting). For each video, visual and auditory saliency maps were used to index the strength of stimulus-driven signals, and eye-movements were used as a measure of the efficacy of the audiovisual events for spatial orienting. Results showed that visual salience modulated activity in higher-order visual areas, whereas auditory salience modulated activity in the superior temporal cortex. Auditory salience modulated activity also in the posterior parietal cortex, but only when audiovisual stimuli occurred on the same side of space (multisensory spatial congruence). Orienting efficacy affected activity in the visual cortex, within the same regions modulated by visual salience. These patterns of activation were comparable in overt and covert orienting conditions. Our results demonstrate that, during viewing of complex multisensory stimuli, activity in sensory areas reflects both stimulus-driven signals and their efficacy for spatial orienting; and that the posterior parietal cortex combines spatial information about the visual and the auditory modality.",
keywords = "Attention, Auditory, Ecological, Eye movements, Multisensory, Orienting, Posterior parietal cortex, Saliency, Space, Visual",
author = "Davide Nardo and Valerio Santangelo and Emiliano Macaluso",
year = "2014",
doi = "10.1002/hbm.22276",
language = "English",
volume = "35",
pages = "1597--1614",
journal = "Human Brain Mapping",
issn = "1065-9471",
publisher = "John Wiley and Sons Inc.",
number = "4",

}

TY - JOUR

T1 - Spatial orienting in complex audiovisual environments

AU - Nardo, Davide

AU - Santangelo, Valerio

AU - Macaluso, Emiliano

PY - 2014

Y1 - 2014

N2 - Previous studies on crossmodal spatial orienting typically used simple and stereotyped stimuli in the absence of any meaningful context. This study combined computational models, behavioural measures and functional magnetic resonance imaging to investigate audiovisual spatial interactions in naturalistic settings. We created short videos portraying everyday life situations that included a lateralised visual event and a co-occurring sound, either on the same or on the opposite side of space. Subjects viewed the videos with or without eye-movements allowed (overt or covert orienting). For each video, visual and auditory saliency maps were used to index the strength of stimulus-driven signals, and eye-movements were used as a measure of the efficacy of the audiovisual events for spatial orienting. Results showed that visual salience modulated activity in higher-order visual areas, whereas auditory salience modulated activity in the superior temporal cortex. Auditory salience modulated activity also in the posterior parietal cortex, but only when audiovisual stimuli occurred on the same side of space (multisensory spatial congruence). Orienting efficacy affected activity in the visual cortex, within the same regions modulated by visual salience. These patterns of activation were comparable in overt and covert orienting conditions. Our results demonstrate that, during viewing of complex multisensory stimuli, activity in sensory areas reflects both stimulus-driven signals and their efficacy for spatial orienting; and that the posterior parietal cortex combines spatial information about the visual and the auditory modality.

AB - Previous studies on crossmodal spatial orienting typically used simple and stereotyped stimuli in the absence of any meaningful context. This study combined computational models, behavioural measures and functional magnetic resonance imaging to investigate audiovisual spatial interactions in naturalistic settings. We created short videos portraying everyday life situations that included a lateralised visual event and a co-occurring sound, either on the same or on the opposite side of space. Subjects viewed the videos with or without eye-movements allowed (overt or covert orienting). For each video, visual and auditory saliency maps were used to index the strength of stimulus-driven signals, and eye-movements were used as a measure of the efficacy of the audiovisual events for spatial orienting. Results showed that visual salience modulated activity in higher-order visual areas, whereas auditory salience modulated activity in the superior temporal cortex. Auditory salience modulated activity also in the posterior parietal cortex, but only when audiovisual stimuli occurred on the same side of space (multisensory spatial congruence). Orienting efficacy affected activity in the visual cortex, within the same regions modulated by visual salience. These patterns of activation were comparable in overt and covert orienting conditions. Our results demonstrate that, during viewing of complex multisensory stimuli, activity in sensory areas reflects both stimulus-driven signals and their efficacy for spatial orienting; and that the posterior parietal cortex combines spatial information about the visual and the auditory modality.

KW - Attention

KW - Auditory

KW - Ecological

KW - Eye movements

KW - Multisensory

KW - Orienting

KW - Posterior parietal cortex

KW - Saliency

KW - Space

KW - Visual

UR - http://www.scopus.com/inward/record.url?scp=84896393752&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84896393752&partnerID=8YFLogxK

U2 - 10.1002/hbm.22276

DO - 10.1002/hbm.22276

M3 - Article

C2 - 23616340

AN - SCOPUS:84896393752

VL - 35

SP - 1597

EP - 1614

JO - Human Brain Mapping

JF - Human Brain Mapping

SN - 1065-9471

IS - 4

ER -