open access
A comparison between deep learning convolutional neural networks and radiologists in the differentiation of benign and malignant thyroid nodules on CT images
Affiliations- Department of Radiology, Second Affiliated Hospital of Dalian Medical University, Dalian, China
- Department of Radiology, Subei People’s Hospital of Jiangsu province, Yangzhou, China
- Beijing Yizhun-ai Technology Co. Ltd., Beijing, China
- Department of Radiology, Yangzhou University, Yangzhou, China
open access
Abstract
Introduction: We designed 5 convolutional neural network (CNN) models and ensemble models to differentiate malignant and benign thyroid nodules on CT, and compared the diagnostic performance of CNN models with that of radiologists.
Material and methods: We retrospectively included CT images of 880 patients with 986 thyroid nodules confirmed by surgical pathology between July 2017 and December 2019. Two radiologists retrospectively diagnosed benign and malignant thyroid nodules on CT images in a test set. Five CNNs (ResNet50, DenseNet121, DenseNet169, SE-ResNeXt50, and Xception) were trained-validated and tested using 788 and 198 thyroid nodule CT images, respectively. Then, we selected the 3 models with the best diagnostic performance on the test set for the model ensemble. We then compared the diagnostic performance of 2 radiologists with 5 CNN models and the integrated model.
Results: Of the 986 thyroid nodules, 541 were malignant, and 445 were benign. The area under the curves (AUCs) for diagnosing thyroid malignancy was 0.587–0.754 for 2 radiologists. The AUCs for diagnosing thyroid malignancy for the 5 CNN models and ensemble model was 0.901–0.947. There were significant differences in AUC between the radiologists’ models and the CNN models (p < 0.05). The ensemble model had the highest AUC value.
Conclusions: Five CNN models and an ensemble model performed better than radiologists in distinguishing malignant thyroid nodules from benign nodules on CT. The diagnostic performance of the ensemble model improved and showed good potential.
Abstract
Introduction: We designed 5 convolutional neural network (CNN) models and ensemble models to differentiate malignant and benign thyroid nodules on CT, and compared the diagnostic performance of CNN models with that of radiologists.
Material and methods: We retrospectively included CT images of 880 patients with 986 thyroid nodules confirmed by surgical pathology between July 2017 and December 2019. Two radiologists retrospectively diagnosed benign and malignant thyroid nodules on CT images in a test set. Five CNNs (ResNet50, DenseNet121, DenseNet169, SE-ResNeXt50, and Xception) were trained-validated and tested using 788 and 198 thyroid nodule CT images, respectively. Then, we selected the 3 models with the best diagnostic performance on the test set for the model ensemble. We then compared the diagnostic performance of 2 radiologists with 5 CNN models and the integrated model.
Results: Of the 986 thyroid nodules, 541 were malignant, and 445 were benign. The area under the curves (AUCs) for diagnosing thyroid malignancy was 0.587–0.754 for 2 radiologists. The AUCs for diagnosing thyroid malignancy for the 5 CNN models and ensemble model was 0.901–0.947. There were significant differences in AUC between the radiologists’ models and the CNN models (p < 0.05). The ensemble model had the highest AUC value.
Conclusions: Five CNN models and an ensemble model performed better than radiologists in distinguishing malignant thyroid nodules from benign nodules on CT. The diagnostic performance of the ensemble model improved and showed good potential.
Keywords
deep learning; convolutional neural network (CNN); thyroid nodule classification; computed tomography (CT)
About this articleTitle
A comparison between deep learning convolutional neural networks and radiologists in the differentiation of benign and malignant thyroid nodules on CT images
Journal
Issue
Article type
Original paper
Pages
217-225
Published online
2021-02-22
Page views
1662
Article views/downloads
908
DOI
Pubmed
Bibliographic record
Endokrynol Pol 2021;72(3):217-225.
Keywords
deep learning
convolutional neural network (CNN)
thyroid nodule classification
computed tomography (CT)
Authors
Hong-bo Zhao
Chang Liu
Jing Ye
Lu-fan Chang
Qing Xu
Bo-wen Shi
Lu-lu Liu
Yi-li Yin
Bin-bin Shi
References (33)- Russ G, Leboulleux S, Leenhardt L, et al. Thyroid incidentalomas: epidemiology, risk stratification with ultrasound and workup. Eur Thyroid J. 2014; 3(3): 154–163.
- Angell TE, Maurer R, Wang Z, et al. A Cohort Analysis of Clinical and Ultrasound Variables Predicting Cancer Risk in 20,001 Consecutive Thyroid Nodules. J Clin Endocrinol Metab. 2019; 104(11): 5665–5672.
- Hoang JK, Branstetter BF, Gafton AR, et al. Imaging of thyroid carcinoma with CT and MRI: approaches to common scenarios. Cancer Imaging. 2013; 13: 128–139.
- Shie P, Cardarelli R, Sprawls K, et al. Systematic review: prevalence of malignant incidental thyroid nodules identified on fluorine-18 fluorodeoxyglucose positron emission tomography. Nucl Med Commun. 2009; 30(9): 742–748.
- Brito JP, Gionfriddo MR, Al Nofal A, et al. The accuracy of thyroid nodule ultrasound to predict thyroid cancer: systematic review and meta-analysis. J Clin Endocrinol Metab. 2014; 99(4): 1253–1263.
- Haugen BR, Alexander EK, Bible KC, et al. 2015 American Thyroid Association ManagementGuidelines for Adult Patients with Thyroid Nodulesand Differentiated Thyroid Cancer. Thyroid. 2015; 26(1): 1–133.
- Remonti LR, Kramer CK, Leitão CB, et al. Thyroid ultrasound features and risk of carcinoma: a systematic review and meta-analysis of observational studies. Thyroid. 2015; 25(5): 538–550.
- Frates MC, Benson CB, Charboneau JW, et al. Society of Radiologists in Ultrasound. Management of thyroid nodules detected at US: Society of Radiologists in Ultrasound consensus conference statement. Radiology. 2005; 237(3): 794–800.
- Chaudhary V, Bano S. Imaging of the thyroid: Recent advances. Indian J Endocrinol Metab. 2012; 16(3): 371–376.
- Moschetta M, Ianora AA, Testini M, et al. Multidetector computed tomography in the preoperative evaluation of retrosternal goiters: a useful procedure for patients for whom magnetic resonance imaging is contraindicated. Thyroid. 2010; 20(2): 181–187.
- Ishigaki S, Shimamoto K, Satake H, et al. Multi-slice CT of thyroid nodules: comparison with ultrasonography. Radiat Med. 2004; 22(5): 346–353.
- Wu CW, Dionigi G, Lee KW, et al. Calcifications in thyroid nodules identified on preoperative computed tomography: patterns and clinical significance. Surgery. 2012; 151(3): 464–470.
- Hoang JK, Choudhury KR, Eastwood JD, et al. An exponential growth in incidence of thyroid cancer: trends and impact of CT imaging. AJNR Am J Neuroradiol. 2014; 35(4): 778–783.
- Hoang JK, Riofrio A, Bashir MR, et al. High variability in radiologists' reporting practices for incidental thyroid nodules detected on CT and MRI. AJNR Am J Neuroradiol. 2014; 35(6): 1190–1194.
- Gharib H, Papini E, Garber JR, et al. AACE/ACE/AME Task Force on Thyroid Nodules. American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules — 2016 update. Endocr Pract. 2016; 22(5): 622–639.
- Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw. 2015; 61: 85–117.
- Shen D, Wu G, Suk HI. Deep Learning in Medical Image Analysis. Annu Rev Biomed Eng. 2017; 19: 221–248.
- Alipanahi B, Delong A, Weirauch MT, et al. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat Biotechnol. 2015; 33(8): 831–838.
- Aerts HJ, Velazquez ER, Leijenaar RTH, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun. 2014; 5: 4006.
- Ko SuY, Lee JiH, Yoon JH, et al. Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. Head Neck. 2019; 41(4): 885–891.
- Russakovsky O, Deng J, Su H, et al. ImageNet Large Scale Visual Recognition Challenge. Int J Comp Vis. 2015; 115(3): 211–252.
- Bolei Z, Aditya K, Agata L et al. Learning Deep Features for Discriminative Localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Seattle, USA, 2016 June 27–30.
- Siegel RL, Fedewa SA, Miller KD, et al. Cancer statistics, 2015. CA Cancer J Clin. 2015; 65(1): 5–29.
- Thomson CS, Forman D. Cancer survival in England and the influence of early diagnosis: what can we learn from recent EUROCARE results? Br J Cancer. 2009; 101 Suppl 2: S102–S109.
- Chang CY, Chen SJ, Tsai MF. Application of support-vector-machine-based method for feature selection and classification of thyroid nodules in ultrasound images. Pattern Rec. 2010; 43(10): 3494–3506.
- LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015; 521(7553): 436–444.
- Ha EJu, Baek JH, Na DG. Risk Stratification of Thyroid Nodules on Ultrasonography: Current Status and Perspectives. Thyroid. 2017; 27(12): 1463–1468.
- Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542(7639): 115–118.
- Mazurowski MA, Buda M, Saha A, et al. Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on MRI. J Magn Reson Imaging. 2019; 49(4): 939–954.
- Zhu Y, Fu Z, Fei J. An image augmentation method using convolutional network for thyroid nodule classification by transfer learning. In Proceedings of the 3rd IEEE International Conference on Computer and Communication (pp. 1819–1823) Chengdu, China, 13–16 December 2017.
- Chi J, Walia E, Babyn P, et al. Thyroid Nodule Classification in Ultrasound Images by Fine-Tuning Deep Convolutional Neural Network. J Digit Imaging. 2017; 30(4): 477–486.
- Song W, Li S, Liu Ji, et al. Multitask Cascade Convolution Neural Networks for Automatic Thyroid Nodule Detection and Recognition. IEEE J Biomed Health Inform. 2019; 23(3): 1215–1224.
- Nguyen DT, Kang JK, Pham TD, et al. Ultrasound Image-Based Diagnosis of Malignant Thyroid Nodule Using Artificial Intelligence. Sensors (Basel). 2020; 20(7).
