Tractography reproducibility challenge with empirical data (TraCED): The 2017 ISMRM diffusion study group challenge

Vishwesh Nath, Kurt G Schilling, Prasanna Parvathaneni, Yuankai Huo, Justin A Blaber, Allison E Hainline, Muhamed Barakovic, David Romascano, Jonathan Rafael-Patino, Matteo Frigo, Gabriel Girard, Jean-Philippe Thiran, Alessandro Daducci, Matt Rowe, Paulo Rodrigues, Vesna Prčkovska, Dogu B Aydogan, Wei Sun, Yonggang Shi, William A ParkerAbdol A Ould Ismail, Ragini Verma, Ryan P Cabeen, Arthur W Toga, Allen T Newton, Jakob Wasserthal, Peter Neher, Klaus Maier-Hein, Giovanni Savini, Fulvia Palesi, Enrico Kaden, Ye Wu, Jianzhong He, Yuanjing Feng, Michael Paquette, Francois Rheault, Jasmeen Sidhu, Catherine Lebel, Alexander Leemans, Maxime Descoteaux, Tim B Dyrby, Hakmook Kang, Bennett A Landman

Research output: Contribution to journalArticle

Abstract

BACKGROUND: Fiber tracking with diffusion-weighted MRI has become an essential tool for estimating in vivo brain white matter architecture. Fiber tracking results are sensitive to the choice of processing method and tracking criteria.

PURPOSE: To assess the variability for an algorithm in group studies reproducibility is of critical context. However, reproducibility does not assess the validity of the brain connections. Phantom studies provide concrete quantitative comparisons of methods relative to absolute ground truths, yet do no capture variabilities because of in vivo physiological factors. The ISMRM 2017 TraCED challenge was created to fulfill the gap.

STUDY TYPE: A systematic review of algorithms and tract reproducibility studies.

SUBJECTS: Single healthy volunteers.

FIELD STRENGTH/SEQUENCE: 3.0T, two different scanners by the same manufacturer. The multishell acquisition included b-values of 1000, 2000, and 3000 s/mm2 with 20, 45, and 64 diffusion gradient directions per shell, respectively.

ASSESSMENT: Nine international groups submitted 46 tractography algorithm entries each consisting 16 tracts per scan. The algorithms were assessed using intraclass correlation (ICC) and the Dice similarity measure.

STATISTICAL TESTS: Containment analysis was performed to assess if the submitted algorithms had containment within tracts of larger volume submissions. This also serves the purpose to detect if spurious submissions had been made.

RESULTS: The top five submissions had high ICC and Dice >0.88. Reproducibility was high within the top five submissions when assessed across sessions or across scanners: 0.87-0.97. Containment analysis shows that the top five submissions are contained within larger volume submissions. From the total of 16 tracts as an outcome relatively the number of tracts with high, moderate, and low reproducibility were 8, 4, and 4.

DATA CONCLUSION: The different methods clearly result in fundamentally different tract structures at the more conservative specificity choices. Data and challenge infrastructure remain available for continued analysis and provide a platform for comparison.

LEVEL OF EVIDENCE: 5 Technical Efficacy Stage: 1 J. Magn. Reson. Imaging 2019.

Original languageEnglish
JournalJournal of Magnetic Resonance Imaging
DOIs
Publication statusE-pub ahead of print - Jun 9 2019

Fingerprint

Diffusion Magnetic Resonance Imaging
Brain
Healthy Volunteers
Direction compound
White Matter

Cite this

Tractography reproducibility challenge with empirical data (TraCED) : The 2017 ISMRM diffusion study group challenge. / Nath, Vishwesh; Schilling, Kurt G; Parvathaneni, Prasanna; Huo, Yuankai; Blaber, Justin A; Hainline, Allison E; Barakovic, Muhamed; Romascano, David; Rafael-Patino, Jonathan; Frigo, Matteo; Girard, Gabriel; Thiran, Jean-Philippe; Daducci, Alessandro; Rowe, Matt; Rodrigues, Paulo; Prčkovska, Vesna; Aydogan, Dogu B; Sun, Wei; Shi, Yonggang; Parker, William A; Ould Ismail, Abdol A; Verma, Ragini; Cabeen, Ryan P; Toga, Arthur W; Newton, Allen T; Wasserthal, Jakob; Neher, Peter; Maier-Hein, Klaus; Savini, Giovanni; Palesi, Fulvia; Kaden, Enrico; Wu, Ye; He, Jianzhong; Feng, Yuanjing; Paquette, Michael; Rheault, Francois; Sidhu, Jasmeen; Lebel, Catherine; Leemans, Alexander; Descoteaux, Maxime; Dyrby, Tim B; Kang, Hakmook; Landman, Bennett A.

In: Journal of Magnetic Resonance Imaging, 09.06.2019.

Research output: Contribution to journalArticle

Nath, V, Schilling, KG, Parvathaneni, P, Huo, Y, Blaber, JA, Hainline, AE, Barakovic, M, Romascano, D, Rafael-Patino, J, Frigo, M, Girard, G, Thiran, J-P, Daducci, A, Rowe, M, Rodrigues, P, Prčkovska, V, Aydogan, DB, Sun, W, Shi, Y, Parker, WA, Ould Ismail, AA, Verma, R, Cabeen, RP, Toga, AW, Newton, AT, Wasserthal, J, Neher, P, Maier-Hein, K, Savini, G, Palesi, F, Kaden, E, Wu, Y, He, J, Feng, Y, Paquette, M, Rheault, F, Sidhu, J, Lebel, C, Leemans, A, Descoteaux, M, Dyrby, TB, Kang, H & Landman, BA 2019, 'Tractography reproducibility challenge with empirical data (TraCED): The 2017 ISMRM diffusion study group challenge', Journal of Magnetic Resonance Imaging. https://doi.org/10.1002/jmri.26794
Nath, Vishwesh ; Schilling, Kurt G ; Parvathaneni, Prasanna ; Huo, Yuankai ; Blaber, Justin A ; Hainline, Allison E ; Barakovic, Muhamed ; Romascano, David ; Rafael-Patino, Jonathan ; Frigo, Matteo ; Girard, Gabriel ; Thiran, Jean-Philippe ; Daducci, Alessandro ; Rowe, Matt ; Rodrigues, Paulo ; Prčkovska, Vesna ; Aydogan, Dogu B ; Sun, Wei ; Shi, Yonggang ; Parker, William A ; Ould Ismail, Abdol A ; Verma, Ragini ; Cabeen, Ryan P ; Toga, Arthur W ; Newton, Allen T ; Wasserthal, Jakob ; Neher, Peter ; Maier-Hein, Klaus ; Savini, Giovanni ; Palesi, Fulvia ; Kaden, Enrico ; Wu, Ye ; He, Jianzhong ; Feng, Yuanjing ; Paquette, Michael ; Rheault, Francois ; Sidhu, Jasmeen ; Lebel, Catherine ; Leemans, Alexander ; Descoteaux, Maxime ; Dyrby, Tim B ; Kang, Hakmook ; Landman, Bennett A. / Tractography reproducibility challenge with empirical data (TraCED) : The 2017 ISMRM diffusion study group challenge. In: Journal of Magnetic Resonance Imaging. 2019.
@article{f0fc342e30b34c31958cb0eef858d448,
title = "Tractography reproducibility challenge with empirical data (TraCED): The 2017 ISMRM diffusion study group challenge",
abstract = "BACKGROUND: Fiber tracking with diffusion-weighted MRI has become an essential tool for estimating in vivo brain white matter architecture. Fiber tracking results are sensitive to the choice of processing method and tracking criteria.PURPOSE: To assess the variability for an algorithm in group studies reproducibility is of critical context. However, reproducibility does not assess the validity of the brain connections. Phantom studies provide concrete quantitative comparisons of methods relative to absolute ground truths, yet do no capture variabilities because of in vivo physiological factors. The ISMRM 2017 TraCED challenge was created to fulfill the gap.STUDY TYPE: A systematic review of algorithms and tract reproducibility studies.SUBJECTS: Single healthy volunteers.FIELD STRENGTH/SEQUENCE: 3.0T, two different scanners by the same manufacturer. The multishell acquisition included b-values of 1000, 2000, and 3000 s/mm2 with 20, 45, and 64 diffusion gradient directions per shell, respectively.ASSESSMENT: Nine international groups submitted 46 tractography algorithm entries each consisting 16 tracts per scan. The algorithms were assessed using intraclass correlation (ICC) and the Dice similarity measure.STATISTICAL TESTS: Containment analysis was performed to assess if the submitted algorithms had containment within tracts of larger volume submissions. This also serves the purpose to detect if spurious submissions had been made.RESULTS: The top five submissions had high ICC and Dice >0.88. Reproducibility was high within the top five submissions when assessed across sessions or across scanners: 0.87-0.97. Containment analysis shows that the top five submissions are contained within larger volume submissions. From the total of 16 tracts as an outcome relatively the number of tracts with high, moderate, and low reproducibility were 8, 4, and 4.DATA CONCLUSION: The different methods clearly result in fundamentally different tract structures at the more conservative specificity choices. Data and challenge infrastructure remain available for continued analysis and provide a platform for comparison.LEVEL OF EVIDENCE: 5 Technical Efficacy Stage: 1 J. Magn. Reson. Imaging 2019.",
author = "Vishwesh Nath and Schilling, {Kurt G} and Prasanna Parvathaneni and Yuankai Huo and Blaber, {Justin A} and Hainline, {Allison E} and Muhamed Barakovic and David Romascano and Jonathan Rafael-Patino and Matteo Frigo and Gabriel Girard and Jean-Philippe Thiran and Alessandro Daducci and Matt Rowe and Paulo Rodrigues and Vesna Prčkovska and Aydogan, {Dogu B} and Wei Sun and Yonggang Shi and Parker, {William A} and {Ould Ismail}, {Abdol A} and Ragini Verma and Cabeen, {Ryan P} and Toga, {Arthur W} and Newton, {Allen T} and Jakob Wasserthal and Peter Neher and Klaus Maier-Hein and Giovanni Savini and Fulvia Palesi and Enrico Kaden and Ye Wu and Jianzhong He and Yuanjing Feng and Michael Paquette and Francois Rheault and Jasmeen Sidhu and Catherine Lebel and Alexander Leemans and Maxime Descoteaux and Dyrby, {Tim B} and Hakmook Kang and Landman, {Bennett A}",
note = "{\circledC} 2019 International Society for Magnetic Resonance in Medicine.",
year = "2019",
month = "6",
day = "9",
doi = "10.1002/jmri.26794",
language = "English",
journal = "Journal of Magnetic Resonance Imaging",
issn = "1053-1807",
publisher = "John Wiley and Sons Inc.",

}

TY - JOUR

T1 - Tractography reproducibility challenge with empirical data (TraCED)

T2 - The 2017 ISMRM diffusion study group challenge

AU - Nath, Vishwesh

AU - Schilling, Kurt G

AU - Parvathaneni, Prasanna

AU - Huo, Yuankai

AU - Blaber, Justin A

AU - Hainline, Allison E

AU - Barakovic, Muhamed

AU - Romascano, David

AU - Rafael-Patino, Jonathan

AU - Frigo, Matteo

AU - Girard, Gabriel

AU - Thiran, Jean-Philippe

AU - Daducci, Alessandro

AU - Rowe, Matt

AU - Rodrigues, Paulo

AU - Prčkovska, Vesna

AU - Aydogan, Dogu B

AU - Sun, Wei

AU - Shi, Yonggang

AU - Parker, William A

AU - Ould Ismail, Abdol A

AU - Verma, Ragini

AU - Cabeen, Ryan P

AU - Toga, Arthur W

AU - Newton, Allen T

AU - Wasserthal, Jakob

AU - Neher, Peter

AU - Maier-Hein, Klaus

AU - Savini, Giovanni

AU - Palesi, Fulvia

AU - Kaden, Enrico

AU - Wu, Ye

AU - He, Jianzhong

AU - Feng, Yuanjing

AU - Paquette, Michael

AU - Rheault, Francois

AU - Sidhu, Jasmeen

AU - Lebel, Catherine

AU - Leemans, Alexander

AU - Descoteaux, Maxime

AU - Dyrby, Tim B

AU - Kang, Hakmook

AU - Landman, Bennett A

N1 - © 2019 International Society for Magnetic Resonance in Medicine.

PY - 2019/6/9

Y1 - 2019/6/9

N2 - BACKGROUND: Fiber tracking with diffusion-weighted MRI has become an essential tool for estimating in vivo brain white matter architecture. Fiber tracking results are sensitive to the choice of processing method and tracking criteria.PURPOSE: To assess the variability for an algorithm in group studies reproducibility is of critical context. However, reproducibility does not assess the validity of the brain connections. Phantom studies provide concrete quantitative comparisons of methods relative to absolute ground truths, yet do no capture variabilities because of in vivo physiological factors. The ISMRM 2017 TraCED challenge was created to fulfill the gap.STUDY TYPE: A systematic review of algorithms and tract reproducibility studies.SUBJECTS: Single healthy volunteers.FIELD STRENGTH/SEQUENCE: 3.0T, two different scanners by the same manufacturer. The multishell acquisition included b-values of 1000, 2000, and 3000 s/mm2 with 20, 45, and 64 diffusion gradient directions per shell, respectively.ASSESSMENT: Nine international groups submitted 46 tractography algorithm entries each consisting 16 tracts per scan. The algorithms were assessed using intraclass correlation (ICC) and the Dice similarity measure.STATISTICAL TESTS: Containment analysis was performed to assess if the submitted algorithms had containment within tracts of larger volume submissions. This also serves the purpose to detect if spurious submissions had been made.RESULTS: The top five submissions had high ICC and Dice >0.88. Reproducibility was high within the top five submissions when assessed across sessions or across scanners: 0.87-0.97. Containment analysis shows that the top five submissions are contained within larger volume submissions. From the total of 16 tracts as an outcome relatively the number of tracts with high, moderate, and low reproducibility were 8, 4, and 4.DATA CONCLUSION: The different methods clearly result in fundamentally different tract structures at the more conservative specificity choices. Data and challenge infrastructure remain available for continued analysis and provide a platform for comparison.LEVEL OF EVIDENCE: 5 Technical Efficacy Stage: 1 J. Magn. Reson. Imaging 2019.

AB - BACKGROUND: Fiber tracking with diffusion-weighted MRI has become an essential tool for estimating in vivo brain white matter architecture. Fiber tracking results are sensitive to the choice of processing method and tracking criteria.PURPOSE: To assess the variability for an algorithm in group studies reproducibility is of critical context. However, reproducibility does not assess the validity of the brain connections. Phantom studies provide concrete quantitative comparisons of methods relative to absolute ground truths, yet do no capture variabilities because of in vivo physiological factors. The ISMRM 2017 TraCED challenge was created to fulfill the gap.STUDY TYPE: A systematic review of algorithms and tract reproducibility studies.SUBJECTS: Single healthy volunteers.FIELD STRENGTH/SEQUENCE: 3.0T, two different scanners by the same manufacturer. The multishell acquisition included b-values of 1000, 2000, and 3000 s/mm2 with 20, 45, and 64 diffusion gradient directions per shell, respectively.ASSESSMENT: Nine international groups submitted 46 tractography algorithm entries each consisting 16 tracts per scan. The algorithms were assessed using intraclass correlation (ICC) and the Dice similarity measure.STATISTICAL TESTS: Containment analysis was performed to assess if the submitted algorithms had containment within tracts of larger volume submissions. This also serves the purpose to detect if spurious submissions had been made.RESULTS: The top five submissions had high ICC and Dice >0.88. Reproducibility was high within the top five submissions when assessed across sessions or across scanners: 0.87-0.97. Containment analysis shows that the top five submissions are contained within larger volume submissions. From the total of 16 tracts as an outcome relatively the number of tracts with high, moderate, and low reproducibility were 8, 4, and 4.DATA CONCLUSION: The different methods clearly result in fundamentally different tract structures at the more conservative specificity choices. Data and challenge infrastructure remain available for continued analysis and provide a platform for comparison.LEVEL OF EVIDENCE: 5 Technical Efficacy Stage: 1 J. Magn. Reson. Imaging 2019.

U2 - 10.1002/jmri.26794

DO - 10.1002/jmri.26794

M3 - Article

C2 - 31179595

JO - Journal of Magnetic Resonance Imaging

JF - Journal of Magnetic Resonance Imaging

SN - 1053-1807

ER -