TY - JOUR
T1 - Segmentation of cellular patterns in confocal images of melanocytic lesions in vivo via a multiscale encoder-decoder network (MED-Net)
AU - Kose, Kivanc
AU - Bozkurt, Alican
AU - Alessi-Fox, Christi
AU - Gill, Melissa
AU - Longo, Caterina
AU - Pellacani, Giovanni
AU - Dy, Jennifer G.
AU - Brooks, Dana H.
AU - Rajadhyaksha, Milind
N1 - Funding Information:
This project was supported by NIH grant R01CA199673 from NCI and in part by MSKCC’s Cancer Center core support NIH grant P30CA008748 from NCI. The authors would like to thank NVIDIA Corporation for the Titan V GPU donation through their GPU Grant Program.
Publisher Copyright:
© 2020
Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.
PY - 2021/1
Y1 - 2021/1
N2 - In-vivo optical microscopy is advancing into routine clinical practice for non-invasively guiding diagnosis and treatment of cancer and other diseases, and thus beginning to reduce the need for traditional biopsy. However, reading and analysis of the optical microscopic images are generally still qualitative, relying mainly on visual examination. Here we present an automated semantic segmentation method called “Multiscale Encoder-Decoder Network (MED-Net)” that provides pixel-wise labeling into classes of patterns in a quantitative manner. The novelty in our approach is the modeling of textural patterns at multiple scales (magnifications, resolutions). This mimics the traditional procedure for examining pathology images, which routinely starts with low magnification (low resolution, large field of view) followed by closer inspection of suspicious areas with higher magnification (higher resolution, smaller fields of view). We trained and tested our model on non-overlapping partitions of 117 reflectance confocal microscopy (RCM) mosaics of melanocytic lesions, an extensive dataset for this application, collected at four clinics in the US, and two in Italy. With patient-wise cross-validation, we achieved pixel-wise mean sensitivity and specificity of 74% and 92%, respectively, with 0.74 Dice coefficient over six classes. In the scenario, we partitioned the data clinic-wise and tested the generalizability of the model over multiple clinics. In this setting, we achieved pixel-wise mean sensitivity and specificity of 77% and 94%, respectively, with 0.77 Dice coefficient. We compared MED-Net against the state-of-the-art semantic segmentation models and achieved better quantitative segmentation performance. Our results also suggest that, due to its nested multiscale architecture, the MED-Net model annotated RCM mosaics more coherently, avoiding unrealistic-fragmented annotations.
AB - In-vivo optical microscopy is advancing into routine clinical practice for non-invasively guiding diagnosis and treatment of cancer and other diseases, and thus beginning to reduce the need for traditional biopsy. However, reading and analysis of the optical microscopic images are generally still qualitative, relying mainly on visual examination. Here we present an automated semantic segmentation method called “Multiscale Encoder-Decoder Network (MED-Net)” that provides pixel-wise labeling into classes of patterns in a quantitative manner. The novelty in our approach is the modeling of textural patterns at multiple scales (magnifications, resolutions). This mimics the traditional procedure for examining pathology images, which routinely starts with low magnification (low resolution, large field of view) followed by closer inspection of suspicious areas with higher magnification (higher resolution, smaller fields of view). We trained and tested our model on non-overlapping partitions of 117 reflectance confocal microscopy (RCM) mosaics of melanocytic lesions, an extensive dataset for this application, collected at four clinics in the US, and two in Italy. With patient-wise cross-validation, we achieved pixel-wise mean sensitivity and specificity of 74% and 92%, respectively, with 0.74 Dice coefficient over six classes. In the scenario, we partitioned the data clinic-wise and tested the generalizability of the model over multiple clinics. In this setting, we achieved pixel-wise mean sensitivity and specificity of 77% and 94%, respectively, with 0.77 Dice coefficient. We compared MED-Net against the state-of-the-art semantic segmentation models and achieved better quantitative segmentation performance. Our results also suggest that, due to its nested multiscale architecture, the MED-Net model annotated RCM mosaics more coherently, avoiding unrealistic-fragmented annotations.
KW - Dermatology
KW - In vivo segmentation
KW - Melanocytic lesion
KW - Reflectance confocal microscopy
KW - Semantic segmentation
UR - http://www.scopus.com/inward/record.url?scp=85094826884&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85094826884&partnerID=8YFLogxK
U2 - 10.1016/j.media.2020.101841
DO - 10.1016/j.media.2020.101841
M3 - Article
C2 - 33142135
AN - SCOPUS:85094826884
VL - 67
JO - Medical Image Analysis
JF - Medical Image Analysis
SN - 1361-8415
M1 - 101841
ER -