open access

Ahead of print
Original paper
Published online: 2021-02-22
Submitted: 2020-12-28
Accepted: 2021-01-27
Get Citation

Comparison of Deep Learning Convolutional Neural Network with Radiologists in Differentiating Benign and Malignant Thyroid Nodules on CT Images

Hong-bo Zhao, Chang Liu, Jing Ye, Lu-fan Chang, Qing Xu, Bo-wen Shi, Lu-lu Liu, Yi-li Yin, Bin-bin Shi
DOI: 10.5603/EP.a2021.0015
·
Pubmed: 33619712

open access

Ahead of print
Original Paper
Published online: 2021-02-22
Submitted: 2020-12-28
Accepted: 2021-01-27

Abstract

Background: We designed five convolutional neural network (CNN) models and ensemble models to differentiate malignant and benign thyroid nodules on CT and compared the diagnostic performance of CNN models with that of radiologists. Methods: We retrospectively included CT images of 880 patients with 986 thyroid nodules confirmed by surgical pathology between July 2017 and December 2019. Two radiologists retrospectively diagnosed benign and malignant thyroid nodules on CT images in a test set. Five CNNs (ResNet50, DenseNet121, DenseNet169, SE-ResNeXt50, and Xception) were trained-validated and tested using 788 and 198 thyroid nodule CT images respectively. Then, we selected three models with better diagnostic performances on the test set for the model ensemble. We then compared the diagnostic performance of two radiologists with five CNN models and the integrated model. Results: Of the 986 thyroid nodules, 541 were malignant, and 445 were benign. The area under the curves (AUCs) for diagnosing thyroid malignancy was 0.587–0.754 for two radiologists. The AUCs for diagnosing thyroid malignancy for the five CNN models and ensemble model was 0.901–0.947. There were significant differences in AUC between the radiologists' models and the CNN models (P < 0.05). The ensemble model had the highest AUC value. Conclusions: Five CNN models and an ensemble model performed better than radiologists in distinguishing malignant thyroid nodules from benign nodules on CT. The diagnostic performance of the ensemble model improved and showed good potential.

Abstract

Background: We designed five convolutional neural network (CNN) models and ensemble models to differentiate malignant and benign thyroid nodules on CT and compared the diagnostic performance of CNN models with that of radiologists. Methods: We retrospectively included CT images of 880 patients with 986 thyroid nodules confirmed by surgical pathology between July 2017 and December 2019. Two radiologists retrospectively diagnosed benign and malignant thyroid nodules on CT images in a test set. Five CNNs (ResNet50, DenseNet121, DenseNet169, SE-ResNeXt50, and Xception) were trained-validated and tested using 788 and 198 thyroid nodule CT images respectively. Then, we selected three models with better diagnostic performances on the test set for the model ensemble. We then compared the diagnostic performance of two radiologists with five CNN models and the integrated model. Results: Of the 986 thyroid nodules, 541 were malignant, and 445 were benign. The area under the curves (AUCs) for diagnosing thyroid malignancy was 0.587–0.754 for two radiologists. The AUCs for diagnosing thyroid malignancy for the five CNN models and ensemble model was 0.901–0.947. There were significant differences in AUC between the radiologists' models and the CNN models (P < 0.05). The ensemble model had the highest AUC value. Conclusions: Five CNN models and an ensemble model performed better than radiologists in distinguishing malignant thyroid nodules from benign nodules on CT. The diagnostic performance of the ensemble model improved and showed good potential.

Get Citation

Keywords

Deep learning; Convolutional neural network (CNN); Thyroid nodule classification; Computed tomography (CT)

About this article
Title

Comparison of Deep Learning Convolutional Neural Network with Radiologists in Differentiating Benign and Malignant Thyroid Nodules on CT Images

Journal

Endokrynologia Polska

Issue

Ahead of print

Article type

Original paper

Published online

2021-02-22

DOI

10.5603/EP.a2021.0015

Pubmed

33619712

Keywords

Deep learning
Convolutional neural network (CNN)
Thyroid nodule classification
Computed tomography (CT)

Authors

Hong-bo Zhao
Chang Liu
Jing Ye
Lu-fan Chang
Qing Xu
Bo-wen Shi
Lu-lu Liu
Yi-li Yin
Bin-bin Shi

References (33)
  1. Russ G, Leboulleux S, Leenhardt L, et al. Thyroid incidentalomas: epidemiology, risk stratification with ultrasound and workup. Eur Thyroid J. 2014; 3(3): 154–163.
  2. Angell TE, Maurer R, Wang Z, et al. A Cohort Analysis of Clinical and Ultrasound Variables Predicting Cancer Risk in 20,001 Consecutive Thyroid Nodules. J Clin Endocrinol Metab. 2019; 104(11): 5665–5672.
  3. Hoang JK, Branstetter BF, Gafton AR, et al. Imaging of thyroid carcinoma with CT and MRI: approaches to common scenarios. Cancer Imaging. 2013; 13: 128–139.
  4. Shie P, Cardarelli R, Sprawls K, et al. Systematic review: prevalence of malignant incidental thyroid nodules identified on fluorine-18 fluorodeoxyglucose positron emission tomography. Nucl Med Commun. 2009; 30(9): 742–748.
  5. Brito JP, Gionfriddo MR, Al Nofal A, et al. The accuracy of thyroid nodule ultrasound to predict thyroid cancer: systematic review and meta-analysis. J Clin Endocrinol Metab. 2014; 99(4): 1253–1263.
  6. Haugen BR, Alexander EK, Bible KC, et al. 2015 American Thyroid Association ManagementGuidelines for Adult Patients with Thyroid Nodulesand Differentiated Thyroid Cancer. Thyroid. 2015; 26(1): 1–133.
  7. Remonti LR, Kramer CK, Leitão CB, et al. Thyroid ultrasound features and risk of carcinoma: a systematic review and meta-analysis of observational studies. Thyroid. 2015; 25(5): 538–550.
  8. Frates MC, Benson CB, Charboneau JW, et al. Society of Radiologists in Ultrasound. Management of thyroid nodules detected at US: Society of Radiologists in Ultrasound consensus conference statement. Radiology. 2005; 237(3): 794–800.
  9. Chaudhary V, Bano S. Imaging of the thyroid: Recent advances. Indian J Endocrinol Metab. 2012; 16(3): 371–376.
  10. Moschetta M, Ianora AA, Testini M, et al. Multidetector computed tomography in the preoperative evaluation of retrosternal goiters: a useful procedure for patients for whom magnetic resonance imaging is contraindicated. Thyroid. 2010; 20(2): 181–187.
  11. Ishigaki S, Shimamoto K, Satake H, et al. Multi-slice CT of thyroid nodules: comparison with ultrasonography. Radiat Med. 2004; 22(5): 346–353.
  12. Wu CW, Dionigi G, Lee KW, et al. Calcifications in thyroid nodules identified on preoperative computed tomography: patterns and clinical significance. Surgery. 2012; 151(3): 464–470.
  13. Hoang JK, Choudhury KR, Eastwood JD, et al. An exponential growth in incidence of thyroid cancer: trends and impact of CT imaging. AJNR Am J Neuroradiol. 2014; 35(4): 778–783.
  14. Hoang JK, Riofrio A, Bashir MR, et al. High variability in radiologists' reporting practices for incidental thyroid nodules detected on CT and MRI. AJNR Am J Neuroradiol. 2014; 35(6): 1190–1194.
  15. Gharib H, Papini E, Garber JR, et al. AACE/ACE/AME Task Force on Thyroid Nodules. American Association of Clinical Endocrinologists, American College of Endocrinology, and Associazione Medici Endocrinologi medical guidelines for clinical practice for the diagnosis and management of thyroid nodules — 2016 update. Endocr Pract. 2016; 22(5): 622–639.
  16. Schmidhuber J. Deep learning in neural networks: an overview. Neural Netw. 2015; 61: 85–117.
  17. Shen D, Wu G, Suk HI. Deep Learning in Medical Image Analysis. Annu Rev Biomed Eng. 2017; 19: 221–248.
  18. Alipanahi B, Delong A, Weirauch MT, et al. Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat Biotechnol. 2015; 33(8): 831–838.
  19. Aerts HJ, Velazquez ER, Leijenaar RTH, et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat Commun. 2014; 5: 4006.
  20. Ko SuY, Lee JiH, Yoon JH, et al. Deep convolutional neural network for the diagnosis of thyroid nodules on ultrasound. Head Neck. 2019; 41(4): 885–891.
  21. Russakovsky O, Deng J, Su H, et al. ImageNet Large Scale Visual Recognition Challenge. Int J Comp Vis. 2015; 115(3): 211–252.
  22. Bolei Z, Aditya K, Agata L et al. Learning Deep Features for Discriminative Localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Seattle, USA, 2016 June 27–30.
  23. Siegel RL, Fedewa SA, Miller KD, et al. Cancer statistics, 2015. CA Cancer J Clin. 2015; 65(1): 5–29.
  24. Thomson CS, Forman D. Cancer survival in England and the influence of early diagnosis: what can we learn from recent EUROCARE results? Br J Cancer. 2009; 101 Suppl 2: S102–S109.
  25. Chang CY, Chen SJ, Tsai MF. Application of support-vector-machine-based method for feature selection and classification of thyroid nodules in ultrasound images. Pattern Rec. 2010; 43(10): 3494–3506.
  26. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015; 521(7553): 436–444.
  27. Ha EJu, Baek JH, Na DG. Risk Stratification of Thyroid Nodules on Ultrasonography: Current Status and Perspectives. Thyroid. 2017; 27(12): 1463–1468.
  28. Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542(7639): 115–118.
  29. Mazurowski MA, Buda M, Saha A, et al. Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on MRI. J Magn Reson Imaging. 2019; 49(4): 939–954.
  30. Zhu Y, Fu Z, Fei J. An image augmentation method using convolutional network for thyroid nodule classification by transfer learning. In Proceedings of the 3rd IEEE International Conference on Computer and Communication (pp. 1819–1823) Chengdu, China, 13–16 December 2017.
  31. Chi J, Walia E, Babyn P, et al. Thyroid Nodule Classification in Ultrasound Images by Fine-Tuning Deep Convolutional Neural Network. J Digit Imaging. 2017; 30(4): 477–486.
  32. Song W, Li S, Liu Ji, et al. Multitask Cascade Convolution Neural Networks for Automatic Thyroid Nodule Detection and Recognition. IEEE J Biomed Health Inform. 2019; 23(3): 1215–1224.
  33. Nguyen DT, Kang JK, Pham TD, et al. Ultrasound Image-Based Diagnosis of Malignant Thyroid Nodule Using Artificial Intelligence. Sensors (Basel). 2020; 20(7).

Important: This website uses cookies. More >>

The cookies allow us to identify your computer and find out details about your last visit. They remembering whether you've visited the site before, so that you remain logged in - or to help us work out how many new website visitors we get each month. Most internet browsers accept cookies automatically, but you can change the settings of your browser to erase cookies or prevent automatic acceptance if you prefer.

Via MedicaWydawcą serwisu jest  "Via Medica sp. z o.o." sp.k., ul. Świętokrzyska 73, 80–180 Gdańsk

tel.:+48 58 320 94 94, faks:+48 58 320 94 60, e-mail:  viamedica@viamedica.pl