TY - JOUR
T1 - Double-blind evaluation and benchmarking of survival models in a multi-centre study
AU - Taktak, A.
AU - Antolini, L.
AU - Aung, M.
AU - Boracchi, P.
AU - Campbell, I.
AU - Damato, B.
AU - Ifeachor, E.
AU - Lama, N.
AU - Lisboa, P.
AU - Setzkorn, C.
AU - Stalbovskaya, V.
AU - Biganzoli, E.
PY - 2007/8
Y1 - 2007/8
N2 - Accurate modelling of time-to-event data is of particular importance for both exploratory and predictive analysis in cancer, and can have a direct impact on clinical care. This study presents a detailed double-blind evaluation of the accuracy in out-of-sample prediction of mortality from two generic non-linear models, using artificial neural networks benchmarked against a partial logistic spline, log-normal and COX regression models. A data set containing 2880 samples was shared over the Internet using a purpose-built secure environment called GEOCONDA (www.geoconda.com). The evaluation was carried out in three parts. The first was a comparison between the predicted survival estimates for each of the four survival groups defined by the TNM staging system, against the empirical estimates derived by the Kaplan-Meier method. The second approach focused on the accurate prediction of survival over time, quantified with the time dependent C index (Ctd). Finally, calibration plots were obtained over the range of follow-up and tested using a generalization of the Hosmer-Lemeshow test. All models showed satisfactory performance, with values of Ctd of about 0.7. None of the models showed a systematic tendency towards over/under estimation of the observed survival at τ = 3 and 5 years. At τ = 10 years, all models underestimated the observed survival, except for COX regression which returned an overestimate. The study presents a robust and unbiased benchmarking methodology using a bespoke web facility. It was concluded that powerful, recent flexible modelling algorithms show a comparative predictive performance to that of more established methods from the medical and biological literature, for the reference data set.
AB - Accurate modelling of time-to-event data is of particular importance for both exploratory and predictive analysis in cancer, and can have a direct impact on clinical care. This study presents a detailed double-blind evaluation of the accuracy in out-of-sample prediction of mortality from two generic non-linear models, using artificial neural networks benchmarked against a partial logistic spline, log-normal and COX regression models. A data set containing 2880 samples was shared over the Internet using a purpose-built secure environment called GEOCONDA (www.geoconda.com). The evaluation was carried out in three parts. The first was a comparison between the predicted survival estimates for each of the four survival groups defined by the TNM staging system, against the empirical estimates derived by the Kaplan-Meier method. The second approach focused on the accurate prediction of survival over time, quantified with the time dependent C index (Ctd). Finally, calibration plots were obtained over the range of follow-up and tested using a generalization of the Hosmer-Lemeshow test. All models showed satisfactory performance, with values of Ctd of about 0.7. None of the models showed a systematic tendency towards over/under estimation of the observed survival at τ = 3 and 5 years. At τ = 10 years, all models underestimated the observed survival, except for COX regression which returned an overestimate. The study presents a robust and unbiased benchmarking methodology using a bespoke web facility. It was concluded that powerful, recent flexible modelling algorithms show a comparative predictive performance to that of more established methods from the medical and biological literature, for the reference data set.
KW - Double-blind study
KW - Evaluation studies
KW - Multi-centre studies
KW - Survival analysis
KW - Uveal neoplasms
UR - http://www.scopus.com/inward/record.url?scp=34347230846&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=34347230846&partnerID=8YFLogxK
U2 - 10.1016/j.compbiomed.2006.10.001
DO - 10.1016/j.compbiomed.2006.10.001
M3 - Article
C2 - 17184760
AN - SCOPUS:34347230846
VL - 37
SP - 1108
EP - 1120
JO - Computers in Biology and Medicine
JF - Computers in Biology and Medicine
SN - 0010-4825
IS - 8
ER -