Fast mental states decoding in mixed reality

Daniele De Massari, Daniel Pacheco, Rahim Malekshahi, Alberto Betella, Paul F M J Verschure, Niels Birbaumer, Andrea Caria

Research output: Contribution to journalArticle


The combination of Brain-Computer Interface (BCI) technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality (MR) systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. Brain states discrimination during mixed reality experience is thus critical for adapting specific data features to contingent brain activity. In this study we recorded electroencephalographic (EEG) data while participants experienced MR scenarios implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the user’s actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in MR, using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5stime-windowshiftingevery200ms.Ourresultsindicateoverallbetterperformanceof LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled MR scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in MR.

Original languageEnglish
Article number415
JournalFrontiers in Behavioral Neuroscience
Issue numberNOV
Publication statusPublished - Nov 27 2014



  • EEG
  • Mental states decoding
  • Mixed reality
  • XIM

ASJC Scopus subject areas

  • Behavioral Neuroscience
  • Cognitive Neuroscience
  • Neuropsychology and Physiological Psychology

Cite this

Massari, D. D., Pacheco, D., Malekshahi, R., Betella, A., Verschure, P. F. M. J., Birbaumer, N., & Caria, A. (2014). Fast mental states decoding in mixed reality. Frontiers in Behavioral Neuroscience, 8(NOV), [415].