The impact of cross-modal correspondences on working memory performance

Riccardo Brunetti, Allegra Indraccolo, Serena Mastroberardino, Charles Spence, Valerio Santangelo

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

Cross-modal correspondences influence perceptual performance in adults, infants, and even nonhuman primates across a variety of different sensory modalities, including tasks involving speeded detection and categorization. However, to date, it is still unclear whether and how correspondences could modulate post-perceptual processes, such as working memory (WM). We investigated this issue using an audiovisual two-back task. In Experiment 1, 3 kinds of correspondences were used: audio/visual numerosity, pitch/shape, and pitch/elevation, each presented congruently (e.g., for numerosity: 3 tones along with 3 shapes) or incongruently (3 tones/2 shapes). Participants attended to the visual or auditory modalities, or both, simultaneously. The results revealed faster target-detection latencies following congruent as compared to incongruent stimulation, especially for numerosity congruence. In Experiment 2, we focused on numerosity, varying the correspondence of the unattended modality, thus having correspondences at both sample (e.g., 3 tones/3 shapes) and target (e.g., 3 tones/3 shapes), only at sample (sample: 3 tones/3 shapes; target: 3 tones/2 shapes), only at target (sample: 3 tones/2 shapes; target: 3 tones/3 shapes), or never. To investigate the information format we included "symbolic" quantities (i.e., visually/auditorily presented digits). The results confirmed the congruence effects, specifically when the correspondence operates at the target display, thus affecting response selection. The experiment revealed modal effects, showing how task-irrelevant digits affect performance only in the auditory modality, while task-irrelevant quantities affect it only when presented visually. Overall, these findings highlight the impact of cross-modal correspondences on WM, adding new light on the link between perceptual and postperceptual stages of human information processing.

Original languageEnglish
Pages (from-to)819-831
Number of pages13
JournalJournal of Experimental Psychology: Human Perception and Performance
Volume43
Issue number4
DOIs
Publication statusPublished - Apr 1 2017

Fingerprint

Short-Term Memory
Mental Processes
Primates
Light
Working Memory
Numerosity

Keywords

  • Bimodal n-back
  • Cross-modal binding
  • Cross-modal correspondences
  • Modal/amodal representations
  • Multisensory integration

ASJC Scopus subject areas

  • Experimental and Cognitive Psychology
  • Arts and Humanities (miscellaneous)
  • Behavioral Neuroscience

Cite this

The impact of cross-modal correspondences on working memory performance. / Brunetti, Riccardo; Indraccolo, Allegra; Mastroberardino, Serena; Spence, Charles; Santangelo, Valerio.

In: Journal of Experimental Psychology: Human Perception and Performance, Vol. 43, No. 4, 01.04.2017, p. 819-831.

Research output: Contribution to journalArticle

Brunetti, Riccardo ; Indraccolo, Allegra ; Mastroberardino, Serena ; Spence, Charles ; Santangelo, Valerio. / The impact of cross-modal correspondences on working memory performance. In: Journal of Experimental Psychology: Human Perception and Performance. 2017 ; Vol. 43, No. 4. pp. 819-831.
@article{fec85b949c1a499eb71a2bbeb0ddf676,
title = "The impact of cross-modal correspondences on working memory performance",
abstract = "Cross-modal correspondences influence perceptual performance in adults, infants, and even nonhuman primates across a variety of different sensory modalities, including tasks involving speeded detection and categorization. However, to date, it is still unclear whether and how correspondences could modulate post-perceptual processes, such as working memory (WM). We investigated this issue using an audiovisual two-back task. In Experiment 1, 3 kinds of correspondences were used: audio/visual numerosity, pitch/shape, and pitch/elevation, each presented congruently (e.g., for numerosity: 3 tones along with 3 shapes) or incongruently (3 tones/2 shapes). Participants attended to the visual or auditory modalities, or both, simultaneously. The results revealed faster target-detection latencies following congruent as compared to incongruent stimulation, especially for numerosity congruence. In Experiment 2, we focused on numerosity, varying the correspondence of the unattended modality, thus having correspondences at both sample (e.g., 3 tones/3 shapes) and target (e.g., 3 tones/3 shapes), only at sample (sample: 3 tones/3 shapes; target: 3 tones/2 shapes), only at target (sample: 3 tones/2 shapes; target: 3 tones/3 shapes), or never. To investigate the information format we included {"}symbolic{"} quantities (i.e., visually/auditorily presented digits). The results confirmed the congruence effects, specifically when the correspondence operates at the target display, thus affecting response selection. The experiment revealed modal effects, showing how task-irrelevant digits affect performance only in the auditory modality, while task-irrelevant quantities affect it only when presented visually. Overall, these findings highlight the impact of cross-modal correspondences on WM, adding new light on the link between perceptual and postperceptual stages of human information processing.",
keywords = "Bimodal n-back, Cross-modal binding, Cross-modal correspondences, Modal/amodal representations, Multisensory integration",
author = "Riccardo Brunetti and Allegra Indraccolo and Serena Mastroberardino and Charles Spence and Valerio Santangelo",
year = "2017",
month = "4",
day = "1",
doi = "10.1037/xhp0000348",
language = "English",
volume = "43",
pages = "819--831",
journal = "Journal of Experimental Psychology: Human Perception and Performance",
issn = "0096-1523",
publisher = "American Psychological Association Inc.",
number = "4",

}

TY - JOUR

T1 - The impact of cross-modal correspondences on working memory performance

AU - Brunetti, Riccardo

AU - Indraccolo, Allegra

AU - Mastroberardino, Serena

AU - Spence, Charles

AU - Santangelo, Valerio

PY - 2017/4/1

Y1 - 2017/4/1

N2 - Cross-modal correspondences influence perceptual performance in adults, infants, and even nonhuman primates across a variety of different sensory modalities, including tasks involving speeded detection and categorization. However, to date, it is still unclear whether and how correspondences could modulate post-perceptual processes, such as working memory (WM). We investigated this issue using an audiovisual two-back task. In Experiment 1, 3 kinds of correspondences were used: audio/visual numerosity, pitch/shape, and pitch/elevation, each presented congruently (e.g., for numerosity: 3 tones along with 3 shapes) or incongruently (3 tones/2 shapes). Participants attended to the visual or auditory modalities, or both, simultaneously. The results revealed faster target-detection latencies following congruent as compared to incongruent stimulation, especially for numerosity congruence. In Experiment 2, we focused on numerosity, varying the correspondence of the unattended modality, thus having correspondences at both sample (e.g., 3 tones/3 shapes) and target (e.g., 3 tones/3 shapes), only at sample (sample: 3 tones/3 shapes; target: 3 tones/2 shapes), only at target (sample: 3 tones/2 shapes; target: 3 tones/3 shapes), or never. To investigate the information format we included "symbolic" quantities (i.e., visually/auditorily presented digits). The results confirmed the congruence effects, specifically when the correspondence operates at the target display, thus affecting response selection. The experiment revealed modal effects, showing how task-irrelevant digits affect performance only in the auditory modality, while task-irrelevant quantities affect it only when presented visually. Overall, these findings highlight the impact of cross-modal correspondences on WM, adding new light on the link between perceptual and postperceptual stages of human information processing.

AB - Cross-modal correspondences influence perceptual performance in adults, infants, and even nonhuman primates across a variety of different sensory modalities, including tasks involving speeded detection and categorization. However, to date, it is still unclear whether and how correspondences could modulate post-perceptual processes, such as working memory (WM). We investigated this issue using an audiovisual two-back task. In Experiment 1, 3 kinds of correspondences were used: audio/visual numerosity, pitch/shape, and pitch/elevation, each presented congruently (e.g., for numerosity: 3 tones along with 3 shapes) or incongruently (3 tones/2 shapes). Participants attended to the visual or auditory modalities, or both, simultaneously. The results revealed faster target-detection latencies following congruent as compared to incongruent stimulation, especially for numerosity congruence. In Experiment 2, we focused on numerosity, varying the correspondence of the unattended modality, thus having correspondences at both sample (e.g., 3 tones/3 shapes) and target (e.g., 3 tones/3 shapes), only at sample (sample: 3 tones/3 shapes; target: 3 tones/2 shapes), only at target (sample: 3 tones/2 shapes; target: 3 tones/3 shapes), or never. To investigate the information format we included "symbolic" quantities (i.e., visually/auditorily presented digits). The results confirmed the congruence effects, specifically when the correspondence operates at the target display, thus affecting response selection. The experiment revealed modal effects, showing how task-irrelevant digits affect performance only in the auditory modality, while task-irrelevant quantities affect it only when presented visually. Overall, these findings highlight the impact of cross-modal correspondences on WM, adding new light on the link between perceptual and postperceptual stages of human information processing.

KW - Bimodal n-back

KW - Cross-modal binding

KW - Cross-modal correspondences

KW - Modal/amodal representations

KW - Multisensory integration

UR - http://www.scopus.com/inward/record.url?scp=85016242920&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85016242920&partnerID=8YFLogxK

U2 - 10.1037/xhp0000348

DO - 10.1037/xhp0000348

M3 - Article

VL - 43

SP - 819

EP - 831

JO - Journal of Experimental Psychology: Human Perception and Performance

JF - Journal of Experimental Psychology: Human Perception and Performance

SN - 0096-1523

IS - 4

ER -