Considering patient clinical history impacts performance of machine learning models in predicting course of multiple sclerosis

Ruggiero Seccia, Daniele Gammelli, Fabio Dominici, Silvia Romano, Anna Chiara Landi, Marco Salvetti, Andrea Tacchella, Andrea Zaccaria, Andrea Crisanti, Francesca Grassi, Laura Palagi

Research output: Contribution to journalArticlepeer-review

Abstract

Multiple Sclerosis (MS) progresses at an unpredictable rate, but predictions on the disease course in each patient would be extremely useful to tailor therapy to the individual needs. We explore different machine learning (ML) approaches to predict whether a patient will shift from the initial Relapsing-Remitting (RR) to the Secondary Progressive (SP) form of the disease, using only "real world" data available in clinical routine. The clinical records of 1624 outpatients (207 in the SP phase) attending the MS service of Sant'Andrea hospital, Rome, Italy, were used. Predictions at 180, 360 or 720 days from the last visit were obtained considering either the data of the last available visit (Visit-Oriented setting), comparing four classical ML methods (Random Forest, Support Vector Machine, K-Nearest Neighbours and AdaBoost) or the whole clinical history of each patient (History-Oriented setting), using a Recurrent Neural Network model, specifically designed for historical data. Missing values were handled by removing either all clinical records presenting at least one missing parameter (Feature-saving approach) or the 3 clinical parameters which contained missing values (Record-saving approach). The performances of the classifiers were rated using common indicators, such as Recall (or Sensitivity) and Precision (or Positive predictive value). In the visit-oriented setting, the Record-saving approach yielded Recall values from 70% to 100%, but low Precision (5% to 10%), which however increased to 50% when considering only predictions for which the model returned a probability above a given "confidence threshold". For the History-oriented setting, both indicators increased as prediction time lengthened, reaching values of 67% (Recall) and 42% (Precision) at 720 days. We show how "real world" data can be effectively used to forecast the evolution of MS, leading to high Recall values and propose innovative approaches to improve Precision towards clinically useful values.

Original languageEnglish
Pages (from-to)e0230219
JournalPLoS One
Volume15
Issue number3
DOIs
Publication statusPublished - 2020

Keywords

  • Adolescent
  • Adult
  • Algorithms
  • Child
  • Disease Progression
  • Female
  • Humans
  • Machine Learning
  • Male
  • Middle Aged
  • Multiple Sclerosis/pathology
  • Neural Networks, Computer
  • Probability
  • Rome
  • Support Vector Machine
  • Young Adult

Fingerprint Dive into the research topics of 'Considering patient clinical history impacts performance of machine learning models in predicting course of multiple sclerosis'. Together they form a unique fingerprint.

Cite this