Detection accuracy of collective intelligence assessments for skin cancer diagnosis

Ralf H J M Kurvers, Jens Krause, Giuseppe Argenziano, Iris Zalaudek, Max Wolf

Research output: Contribution to journalArticle

20 Citations (Scopus)

Abstract

IMPORTANCE Incidence rates of skin cancer are increasing globally, and the correct classification of skin lesions (SLs) into benign and malignant tissue remains a continuous challenge. A collective intelligence approach to skin cancer detection may improve accuracy. OBJECTIVE To evaluate the performance of 2 well-known collective intelligence rules (majority rule and quorum rule) that combine the independent conclusions of multiple decision makers into a single decision. DESIGN, SETTING, AND PARTICIPANTS Evaluationswere obtained from 2 large and independent data sets. The first data set consisted of 40 experienced dermoscopists, each of whom independently evaluated 108 images of SLs during the Consensus Net Meeting of 2000. The second data set consisted of 82 medical professionals with varying degrees of dermatology experience, each of whom evaluated a minimum of 110 SLs. All SLs were evaluated via the Internet. Image selection of SLs was based on high image quality and the presence of histopathologic information. Data were collected from July through October 2000 for study 1 and from February 2003 through January 2004 for study 2 and evaluated from January 5 through August 7, 2015. MAIN OUTCOMES AND MEASURES For both collective intelligence rules,we determined the true-positive rate (ie, the hit rate or specificity) and the false-positive rate (ie, the false-alarm rate or 1 - sensitivity) and compared these rates with the performance of single decision makers. Furthermore, we evaluated the effect of group size on true- and false-positive rates. RESULTS One hundred twenty-two medical professionals performed 16 029 evaluations. Use of either collective intelligence rule consistently outperformed single decision makers. The groups achieved an increased true-positive rate and a decreased false-positive rate. For example, individual decision makers in study 1, using the pattern analysis as diagnostic algorithm, achieved a true-positive rate of 0.83 and a false-positive rate of 0.17. Groups of 3 individuals achieved a true-positive rate of 0.91 and a false-positive rate of 0.14. These improvements increased with increasing group size. CONCLUSIONS AND RELEVANCE Collective intelligence might be a viable approach to increase diagnostic accuracy in skin cancer and reduce skin cancer-related mortality.

Original languageEnglish
Pages (from-to)1346-1353
Number of pages8
JournalJAMA Dermatology
Volume151
Issue number12
DOIs
Publication statusPublished - Dec 1 2015

Fingerprint

Skin Neoplasms
Intelligence
Skin
Dermatology
Internet
Consensus
Outcome Assessment (Health Care)
Mortality
Incidence
Datasets

ASJC Scopus subject areas

  • Dermatology

Cite this

Kurvers, R. H. J. M., Krause, J., Argenziano, G., Zalaudek, I., & Wolf, M. (2015). Detection accuracy of collective intelligence assessments for skin cancer diagnosis. JAMA Dermatology, 151(12), 1346-1353. https://doi.org/10.1001/jamadermatol.2015.3149

Detection accuracy of collective intelligence assessments for skin cancer diagnosis. / Kurvers, Ralf H J M; Krause, Jens; Argenziano, Giuseppe; Zalaudek, Iris; Wolf, Max.

In: JAMA Dermatology, Vol. 151, No. 12, 01.12.2015, p. 1346-1353.

Research output: Contribution to journalArticle

Kurvers, RHJM, Krause, J, Argenziano, G, Zalaudek, I & Wolf, M 2015, 'Detection accuracy of collective intelligence assessments for skin cancer diagnosis', JAMA Dermatology, vol. 151, no. 12, pp. 1346-1353. https://doi.org/10.1001/jamadermatol.2015.3149
Kurvers, Ralf H J M ; Krause, Jens ; Argenziano, Giuseppe ; Zalaudek, Iris ; Wolf, Max. / Detection accuracy of collective intelligence assessments for skin cancer diagnosis. In: JAMA Dermatology. 2015 ; Vol. 151, No. 12. pp. 1346-1353.
@article{9c64f919e5e24f61b269df2166d295fc,
title = "Detection accuracy of collective intelligence assessments for skin cancer diagnosis",
abstract = "IMPORTANCE Incidence rates of skin cancer are increasing globally, and the correct classification of skin lesions (SLs) into benign and malignant tissue remains a continuous challenge. A collective intelligence approach to skin cancer detection may improve accuracy. OBJECTIVE To evaluate the performance of 2 well-known collective intelligence rules (majority rule and quorum rule) that combine the independent conclusions of multiple decision makers into a single decision. DESIGN, SETTING, AND PARTICIPANTS Evaluationswere obtained from 2 large and independent data sets. The first data set consisted of 40 experienced dermoscopists, each of whom independently evaluated 108 images of SLs during the Consensus Net Meeting of 2000. The second data set consisted of 82 medical professionals with varying degrees of dermatology experience, each of whom evaluated a minimum of 110 SLs. All SLs were evaluated via the Internet. Image selection of SLs was based on high image quality and the presence of histopathologic information. Data were collected from July through October 2000 for study 1 and from February 2003 through January 2004 for study 2 and evaluated from January 5 through August 7, 2015. MAIN OUTCOMES AND MEASURES For both collective intelligence rules,we determined the true-positive rate (ie, the hit rate or specificity) and the false-positive rate (ie, the false-alarm rate or 1 - sensitivity) and compared these rates with the performance of single decision makers. Furthermore, we evaluated the effect of group size on true- and false-positive rates. RESULTS One hundred twenty-two medical professionals performed 16 029 evaluations. Use of either collective intelligence rule consistently outperformed single decision makers. The groups achieved an increased true-positive rate and a decreased false-positive rate. For example, individual decision makers in study 1, using the pattern analysis as diagnostic algorithm, achieved a true-positive rate of 0.83 and a false-positive rate of 0.17. Groups of 3 individuals achieved a true-positive rate of 0.91 and a false-positive rate of 0.14. These improvements increased with increasing group size. CONCLUSIONS AND RELEVANCE Collective intelligence might be a viable approach to increase diagnostic accuracy in skin cancer and reduce skin cancer-related mortality.",
author = "Kurvers, {Ralf H J M} and Jens Krause and Giuseppe Argenziano and Iris Zalaudek and Max Wolf",
year = "2015",
month = "12",
day = "1",
doi = "10.1001/jamadermatol.2015.3149",
language = "English",
volume = "151",
pages = "1346--1353",
journal = "JAMA Dermatology",
issn = "2168-6068",
publisher = "American Medical Association",
number = "12",

}

TY - JOUR

T1 - Detection accuracy of collective intelligence assessments for skin cancer diagnosis

AU - Kurvers, Ralf H J M

AU - Krause, Jens

AU - Argenziano, Giuseppe

AU - Zalaudek, Iris

AU - Wolf, Max

PY - 2015/12/1

Y1 - 2015/12/1

N2 - IMPORTANCE Incidence rates of skin cancer are increasing globally, and the correct classification of skin lesions (SLs) into benign and malignant tissue remains a continuous challenge. A collective intelligence approach to skin cancer detection may improve accuracy. OBJECTIVE To evaluate the performance of 2 well-known collective intelligence rules (majority rule and quorum rule) that combine the independent conclusions of multiple decision makers into a single decision. DESIGN, SETTING, AND PARTICIPANTS Evaluationswere obtained from 2 large and independent data sets. The first data set consisted of 40 experienced dermoscopists, each of whom independently evaluated 108 images of SLs during the Consensus Net Meeting of 2000. The second data set consisted of 82 medical professionals with varying degrees of dermatology experience, each of whom evaluated a minimum of 110 SLs. All SLs were evaluated via the Internet. Image selection of SLs was based on high image quality and the presence of histopathologic information. Data were collected from July through October 2000 for study 1 and from February 2003 through January 2004 for study 2 and evaluated from January 5 through August 7, 2015. MAIN OUTCOMES AND MEASURES For both collective intelligence rules,we determined the true-positive rate (ie, the hit rate or specificity) and the false-positive rate (ie, the false-alarm rate or 1 - sensitivity) and compared these rates with the performance of single decision makers. Furthermore, we evaluated the effect of group size on true- and false-positive rates. RESULTS One hundred twenty-two medical professionals performed 16 029 evaluations. Use of either collective intelligence rule consistently outperformed single decision makers. The groups achieved an increased true-positive rate and a decreased false-positive rate. For example, individual decision makers in study 1, using the pattern analysis as diagnostic algorithm, achieved a true-positive rate of 0.83 and a false-positive rate of 0.17. Groups of 3 individuals achieved a true-positive rate of 0.91 and a false-positive rate of 0.14. These improvements increased with increasing group size. CONCLUSIONS AND RELEVANCE Collective intelligence might be a viable approach to increase diagnostic accuracy in skin cancer and reduce skin cancer-related mortality.

AB - IMPORTANCE Incidence rates of skin cancer are increasing globally, and the correct classification of skin lesions (SLs) into benign and malignant tissue remains a continuous challenge. A collective intelligence approach to skin cancer detection may improve accuracy. OBJECTIVE To evaluate the performance of 2 well-known collective intelligence rules (majority rule and quorum rule) that combine the independent conclusions of multiple decision makers into a single decision. DESIGN, SETTING, AND PARTICIPANTS Evaluationswere obtained from 2 large and independent data sets. The first data set consisted of 40 experienced dermoscopists, each of whom independently evaluated 108 images of SLs during the Consensus Net Meeting of 2000. The second data set consisted of 82 medical professionals with varying degrees of dermatology experience, each of whom evaluated a minimum of 110 SLs. All SLs were evaluated via the Internet. Image selection of SLs was based on high image quality and the presence of histopathologic information. Data were collected from July through October 2000 for study 1 and from February 2003 through January 2004 for study 2 and evaluated from January 5 through August 7, 2015. MAIN OUTCOMES AND MEASURES For both collective intelligence rules,we determined the true-positive rate (ie, the hit rate or specificity) and the false-positive rate (ie, the false-alarm rate or 1 - sensitivity) and compared these rates with the performance of single decision makers. Furthermore, we evaluated the effect of group size on true- and false-positive rates. RESULTS One hundred twenty-two medical professionals performed 16 029 evaluations. Use of either collective intelligence rule consistently outperformed single decision makers. The groups achieved an increased true-positive rate and a decreased false-positive rate. For example, individual decision makers in study 1, using the pattern analysis as diagnostic algorithm, achieved a true-positive rate of 0.83 and a false-positive rate of 0.17. Groups of 3 individuals achieved a true-positive rate of 0.91 and a false-positive rate of 0.14. These improvements increased with increasing group size. CONCLUSIONS AND RELEVANCE Collective intelligence might be a viable approach to increase diagnostic accuracy in skin cancer and reduce skin cancer-related mortality.

UR - http://www.scopus.com/inward/record.url?scp=84949655573&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84949655573&partnerID=8YFLogxK

U2 - 10.1001/jamadermatol.2015.3149

DO - 10.1001/jamadermatol.2015.3149

M3 - Article

AN - SCOPUS:84949655573

VL - 151

SP - 1346

EP - 1353

JO - JAMA Dermatology

JF - JAMA Dermatology

SN - 2168-6068

IS - 12

ER -