Calibration belt for quality-of-care assessment based on dichotomous outcomes

Stefano Finazzi, Daniele Poole, Davide Luciani, Paola E. Cogo, Guido Bertolini

Research output: Contribution to journalArticle

34 Citations (Scopus)

Abstract

Prognostic models applied in medicine must be validated on independent samples, before their use can be recommended. The assessment of calibration, i.e., the model's ability to provide reliable predictions, is crucial in external validation studies. Besides having several shortcomings, statistical techniques such as the computation of the standardized mortality ratio (SMR) and its confidence intervals, the Hosmer-Lemeshow statistics, and the Cox calibration test, are all non-informative with respect to calibration across risk classes. Accordingly, calibration plots reporting expected versus observed outcomes across risk subsets have been used for many years. Erroneously, the points in the plot (frequently representing deciles of risk) have been connected with lines, generating false calibration curves. Here we propose a methodology to create a confidence band for the calibration curve based on a function that relates expected to observed probabilities across classes of risk. The calibration belt allows the ranges of risk to be spotted where there is a significant deviation from the ideal calibration, and the direction of the deviation to be indicated. This method thus offers a more analytical view in the assessment of quality of care, compared to other approaches.

Original languageEnglish
Article numbere16110
JournalPLoS One
Volume6
Issue number2
DOIs
Publication statusPublished - 2011

Fingerprint

Quality of Health Care
Calibration
calibration
Validation Studies
Set theory
Medicine
confidence interval
medicine
statistics
methodology
Statistics
Confidence Intervals
prediction
Mortality

ASJC Scopus subject areas

  • Agricultural and Biological Sciences(all)
  • Biochemistry, Genetics and Molecular Biology(all)
  • Medicine(all)

Cite this

Calibration belt for quality-of-care assessment based on dichotomous outcomes. / Finazzi, Stefano; Poole, Daniele; Luciani, Davide; Cogo, Paola E.; Bertolini, Guido.

In: PLoS One, Vol. 6, No. 2, e16110, 2011.

Research output: Contribution to journalArticle

Finazzi, Stefano ; Poole, Daniele ; Luciani, Davide ; Cogo, Paola E. ; Bertolini, Guido. / Calibration belt for quality-of-care assessment based on dichotomous outcomes. In: PLoS One. 2011 ; Vol. 6, No. 2.
@article{f8d66b46283d403f9d8fbb911b958e5a,
title = "Calibration belt for quality-of-care assessment based on dichotomous outcomes",
abstract = "Prognostic models applied in medicine must be validated on independent samples, before their use can be recommended. The assessment of calibration, i.e., the model's ability to provide reliable predictions, is crucial in external validation studies. Besides having several shortcomings, statistical techniques such as the computation of the standardized mortality ratio (SMR) and its confidence intervals, the Hosmer-Lemeshow statistics, and the Cox calibration test, are all non-informative with respect to calibration across risk classes. Accordingly, calibration plots reporting expected versus observed outcomes across risk subsets have been used for many years. Erroneously, the points in the plot (frequently representing deciles of risk) have been connected with lines, generating false calibration curves. Here we propose a methodology to create a confidence band for the calibration curve based on a function that relates expected to observed probabilities across classes of risk. The calibration belt allows the ranges of risk to be spotted where there is a significant deviation from the ideal calibration, and the direction of the deviation to be indicated. This method thus offers a more analytical view in the assessment of quality of care, compared to other approaches.",
author = "Stefano Finazzi and Daniele Poole and Davide Luciani and Cogo, {Paola E.} and Guido Bertolini",
year = "2011",
doi = "10.1371/journal.pone.0016110",
language = "English",
volume = "6",
journal = "PLoS One",
issn = "1932-6203",
publisher = "Public Library of Science",
number = "2",

}

TY - JOUR

T1 - Calibration belt for quality-of-care assessment based on dichotomous outcomes

AU - Finazzi, Stefano

AU - Poole, Daniele

AU - Luciani, Davide

AU - Cogo, Paola E.

AU - Bertolini, Guido

PY - 2011

Y1 - 2011

N2 - Prognostic models applied in medicine must be validated on independent samples, before their use can be recommended. The assessment of calibration, i.e., the model's ability to provide reliable predictions, is crucial in external validation studies. Besides having several shortcomings, statistical techniques such as the computation of the standardized mortality ratio (SMR) and its confidence intervals, the Hosmer-Lemeshow statistics, and the Cox calibration test, are all non-informative with respect to calibration across risk classes. Accordingly, calibration plots reporting expected versus observed outcomes across risk subsets have been used for many years. Erroneously, the points in the plot (frequently representing deciles of risk) have been connected with lines, generating false calibration curves. Here we propose a methodology to create a confidence band for the calibration curve based on a function that relates expected to observed probabilities across classes of risk. The calibration belt allows the ranges of risk to be spotted where there is a significant deviation from the ideal calibration, and the direction of the deviation to be indicated. This method thus offers a more analytical view in the assessment of quality of care, compared to other approaches.

AB - Prognostic models applied in medicine must be validated on independent samples, before their use can be recommended. The assessment of calibration, i.e., the model's ability to provide reliable predictions, is crucial in external validation studies. Besides having several shortcomings, statistical techniques such as the computation of the standardized mortality ratio (SMR) and its confidence intervals, the Hosmer-Lemeshow statistics, and the Cox calibration test, are all non-informative with respect to calibration across risk classes. Accordingly, calibration plots reporting expected versus observed outcomes across risk subsets have been used for many years. Erroneously, the points in the plot (frequently representing deciles of risk) have been connected with lines, generating false calibration curves. Here we propose a methodology to create a confidence band for the calibration curve based on a function that relates expected to observed probabilities across classes of risk. The calibration belt allows the ranges of risk to be spotted where there is a significant deviation from the ideal calibration, and the direction of the deviation to be indicated. This method thus offers a more analytical view in the assessment of quality of care, compared to other approaches.

UR - http://www.scopus.com/inward/record.url?scp=79952013335&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79952013335&partnerID=8YFLogxK

U2 - 10.1371/journal.pone.0016110

DO - 10.1371/journal.pone.0016110

M3 - Article

C2 - 21373178

AN - SCOPUS:79952013335

VL - 6

JO - PLoS One

JF - PLoS One

SN - 1932-6203

IS - 2

M1 - e16110

ER -