Training improves the interobserver agreement of the expert positron emission tomography review panel in primary mediastinal B-cell lymphoma

Interim analysis in the ongoing International Extranodal Lymphoma Study Group-37 study

Luca Ceriani, Sally Barrington, Alberto Biggi, Bogdan Malkowski, Ur Metser, Annibale Versari, Maurizio Martelli, Andrew Davies, Peter W. Johnson, Emanuele Zucca, Stéphane Chauvie

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

The International Extranodal Lymphoma Study Group (IELSG)-37 is a prospective randomized trial assessing the role of consolidation mediastinal radiotherapy after immunochemotherapy to patients with newly diagnosed primary mediastinal large B-cell lymphoma (PMBCL). It is a positron emission tomography (PET) response-guided study where patients obtaining a complete metabolic response on an end-of-therapy PET-computed tomography (CT) scan evaluated by a central review are randomized to receive radiotherapy or no further treatment. The aims of this study were to measure agreement between reviewers reporting PET-CT scans for this trial and to determine the effect of training upon concordance rates. The review panel comprised 6 experienced nuclear physicians who read PET-CT scans using the 5-point Deauville scale. Interobserver agreement (IOA) was measured at 4 time points: after a blinded review of a "training set" of 20 patients with PMBCL from the previous IELSG-26 study (phase 1); after the first 10 clinical cases enrolled in the IELSG-37 (phase 2); and after 2 further groups of 50 (phase 3) and 40 clinical cases (phase 4). After feedback from the training set and the first 10 cases, a meeting was held to discuss interpretation, and a detailed set of instructions for the review procedure was agreed and acted upon. Between 2012 and 2014, the first 100 patients were reviewed. Using Deauville score 3 as the cutoff for a complete metabolic response, the overall IOA among the reviewers was good (Krippendorff α = 0.72.) The binary concordance between pairs of reviewers (Cohen κ) ranged from 0.60 to 0.78. The IOA, initially moderate, improved progressively from phase 1 to 4 (Krippendorff α from 0.53 to 0.81; Cohen κ from 0.35-0.72 to 0.77-0.87). Our experience indicates that the agreement among "expert" nuclear physicians reporting PMBCL, even using standardized criteria, was only moderate when the study began. However, agreement improved using a harmonization process, which included a training exercise with discussion of points leading to disagreement and compiling practical rules to sit alongside commonly adopted interpretation criteria.

Original languageEnglish
JournalHematological Oncology
DOIs
Publication statusAccepted/In press - 2016

Fingerprint

B-Cell Lymphoma
Positron-Emission Tomography
Lymphoma
Radiotherapy
Physicians
Exercise
Therapeutics
Positron Emission Tomography Computed Tomography

Keywords

  • F-18-FDG PET-CT
  • Interobserver agreement
  • Lugano classification
  • Primary mediastinal B-cell lymphoma
  • Reading criteria

ASJC Scopus subject areas

  • Hematology
  • Oncology
  • Cancer Research

Cite this

Training improves the interobserver agreement of the expert positron emission tomography review panel in primary mediastinal B-cell lymphoma : Interim analysis in the ongoing International Extranodal Lymphoma Study Group-37 study. / Ceriani, Luca; Barrington, Sally; Biggi, Alberto; Malkowski, Bogdan; Metser, Ur; Versari, Annibale; Martelli, Maurizio; Davies, Andrew; Johnson, Peter W.; Zucca, Emanuele; Chauvie, Stéphane.

In: Hematological Oncology, 2016.

Research output: Contribution to journalArticle

Ceriani, Luca ; Barrington, Sally ; Biggi, Alberto ; Malkowski, Bogdan ; Metser, Ur ; Versari, Annibale ; Martelli, Maurizio ; Davies, Andrew ; Johnson, Peter W. ; Zucca, Emanuele ; Chauvie, Stéphane. / Training improves the interobserver agreement of the expert positron emission tomography review panel in primary mediastinal B-cell lymphoma : Interim analysis in the ongoing International Extranodal Lymphoma Study Group-37 study. In: Hematological Oncology. 2016.
@article{587fd7890f0c498b86374a1b3805af00,
title = "Training improves the interobserver agreement of the expert positron emission tomography review panel in primary mediastinal B-cell lymphoma: Interim analysis in the ongoing International Extranodal Lymphoma Study Group-37 study",
abstract = "The International Extranodal Lymphoma Study Group (IELSG)-37 is a prospective randomized trial assessing the role of consolidation mediastinal radiotherapy after immunochemotherapy to patients with newly diagnosed primary mediastinal large B-cell lymphoma (PMBCL). It is a positron emission tomography (PET) response-guided study where patients obtaining a complete metabolic response on an end-of-therapy PET-computed tomography (CT) scan evaluated by a central review are randomized to receive radiotherapy or no further treatment. The aims of this study were to measure agreement between reviewers reporting PET-CT scans for this trial and to determine the effect of training upon concordance rates. The review panel comprised 6 experienced nuclear physicians who read PET-CT scans using the 5-point Deauville scale. Interobserver agreement (IOA) was measured at 4 time points: after a blinded review of a {"}training set{"} of 20 patients with PMBCL from the previous IELSG-26 study (phase 1); after the first 10 clinical cases enrolled in the IELSG-37 (phase 2); and after 2 further groups of 50 (phase 3) and 40 clinical cases (phase 4). After feedback from the training set and the first 10 cases, a meeting was held to discuss interpretation, and a detailed set of instructions for the review procedure was agreed and acted upon. Between 2012 and 2014, the first 100 patients were reviewed. Using Deauville score 3 as the cutoff for a complete metabolic response, the overall IOA among the reviewers was good (Krippendorff α = 0.72.) The binary concordance between pairs of reviewers (Cohen κ) ranged from 0.60 to 0.78. The IOA, initially moderate, improved progressively from phase 1 to 4 (Krippendorff α from 0.53 to 0.81; Cohen κ from 0.35-0.72 to 0.77-0.87). Our experience indicates that the agreement among {"}expert{"} nuclear physicians reporting PMBCL, even using standardized criteria, was only moderate when the study began. However, agreement improved using a harmonization process, which included a training exercise with discussion of points leading to disagreement and compiling practical rules to sit alongside commonly adopted interpretation criteria.",
keywords = "F-18-FDG PET-CT, Interobserver agreement, Lugano classification, Primary mediastinal B-cell lymphoma, Reading criteria",
author = "Luca Ceriani and Sally Barrington and Alberto Biggi and Bogdan Malkowski and Ur Metser and Annibale Versari and Maurizio Martelli and Andrew Davies and Johnson, {Peter W.} and Emanuele Zucca and St{\'e}phane Chauvie",
year = "2016",
doi = "10.1002/hon.2339",
language = "English",
journal = "Hematological Oncology",
issn = "0278-0232",
publisher = "wiley",

}

TY - JOUR

T1 - Training improves the interobserver agreement of the expert positron emission tomography review panel in primary mediastinal B-cell lymphoma

T2 - Interim analysis in the ongoing International Extranodal Lymphoma Study Group-37 study

AU - Ceriani, Luca

AU - Barrington, Sally

AU - Biggi, Alberto

AU - Malkowski, Bogdan

AU - Metser, Ur

AU - Versari, Annibale

AU - Martelli, Maurizio

AU - Davies, Andrew

AU - Johnson, Peter W.

AU - Zucca, Emanuele

AU - Chauvie, Stéphane

PY - 2016

Y1 - 2016

N2 - The International Extranodal Lymphoma Study Group (IELSG)-37 is a prospective randomized trial assessing the role of consolidation mediastinal radiotherapy after immunochemotherapy to patients with newly diagnosed primary mediastinal large B-cell lymphoma (PMBCL). It is a positron emission tomography (PET) response-guided study where patients obtaining a complete metabolic response on an end-of-therapy PET-computed tomography (CT) scan evaluated by a central review are randomized to receive radiotherapy or no further treatment. The aims of this study were to measure agreement between reviewers reporting PET-CT scans for this trial and to determine the effect of training upon concordance rates. The review panel comprised 6 experienced nuclear physicians who read PET-CT scans using the 5-point Deauville scale. Interobserver agreement (IOA) was measured at 4 time points: after a blinded review of a "training set" of 20 patients with PMBCL from the previous IELSG-26 study (phase 1); after the first 10 clinical cases enrolled in the IELSG-37 (phase 2); and after 2 further groups of 50 (phase 3) and 40 clinical cases (phase 4). After feedback from the training set and the first 10 cases, a meeting was held to discuss interpretation, and a detailed set of instructions for the review procedure was agreed and acted upon. Between 2012 and 2014, the first 100 patients were reviewed. Using Deauville score 3 as the cutoff for a complete metabolic response, the overall IOA among the reviewers was good (Krippendorff α = 0.72.) The binary concordance between pairs of reviewers (Cohen κ) ranged from 0.60 to 0.78. The IOA, initially moderate, improved progressively from phase 1 to 4 (Krippendorff α from 0.53 to 0.81; Cohen κ from 0.35-0.72 to 0.77-0.87). Our experience indicates that the agreement among "expert" nuclear physicians reporting PMBCL, even using standardized criteria, was only moderate when the study began. However, agreement improved using a harmonization process, which included a training exercise with discussion of points leading to disagreement and compiling practical rules to sit alongside commonly adopted interpretation criteria.

AB - The International Extranodal Lymphoma Study Group (IELSG)-37 is a prospective randomized trial assessing the role of consolidation mediastinal radiotherapy after immunochemotherapy to patients with newly diagnosed primary mediastinal large B-cell lymphoma (PMBCL). It is a positron emission tomography (PET) response-guided study where patients obtaining a complete metabolic response on an end-of-therapy PET-computed tomography (CT) scan evaluated by a central review are randomized to receive radiotherapy or no further treatment. The aims of this study were to measure agreement between reviewers reporting PET-CT scans for this trial and to determine the effect of training upon concordance rates. The review panel comprised 6 experienced nuclear physicians who read PET-CT scans using the 5-point Deauville scale. Interobserver agreement (IOA) was measured at 4 time points: after a blinded review of a "training set" of 20 patients with PMBCL from the previous IELSG-26 study (phase 1); after the first 10 clinical cases enrolled in the IELSG-37 (phase 2); and after 2 further groups of 50 (phase 3) and 40 clinical cases (phase 4). After feedback from the training set and the first 10 cases, a meeting was held to discuss interpretation, and a detailed set of instructions for the review procedure was agreed and acted upon. Between 2012 and 2014, the first 100 patients were reviewed. Using Deauville score 3 as the cutoff for a complete metabolic response, the overall IOA among the reviewers was good (Krippendorff α = 0.72.) The binary concordance between pairs of reviewers (Cohen κ) ranged from 0.60 to 0.78. The IOA, initially moderate, improved progressively from phase 1 to 4 (Krippendorff α from 0.53 to 0.81; Cohen κ from 0.35-0.72 to 0.77-0.87). Our experience indicates that the agreement among "expert" nuclear physicians reporting PMBCL, even using standardized criteria, was only moderate when the study began. However, agreement improved using a harmonization process, which included a training exercise with discussion of points leading to disagreement and compiling practical rules to sit alongside commonly adopted interpretation criteria.

KW - F-18-FDG PET-CT

KW - Interobserver agreement

KW - Lugano classification

KW - Primary mediastinal B-cell lymphoma

KW - Reading criteria

UR - http://www.scopus.com/inward/record.url?scp=84994493293&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84994493293&partnerID=8YFLogxK

U2 - 10.1002/hon.2339

DO - 10.1002/hon.2339

M3 - Article

JO - Hematological Oncology

JF - Hematological Oncology

SN - 0278-0232

ER -