Evaluation of Chat GPT’s performance in Polish paediatric neurology State Specialisation Examination
, 1, 1, 1, 2, 1
Abstract
Introduction: Dynamic technological progress has contributed to significant advances in the field of Artificial Intelligence (AI). Its potential is already being used in many aspects of life, including medicine. The aim of this article was to focus on analyzing the effectiveness of AI-based language models in the context of tackling the Polish State Specialisation Examination (SSE) in paediatric neurology.
Material and methods: The study evaluated the effectiveness of two language models i.e., Chat GPT 3.5 and Chat GPT 4.0 in solving two past papers of SSE in paediatric neurology, i.e., those set in spring and autumn 2023. The point scores of both models were compared to the results of physicians taking the SSE at these two sessions. For the study, questions were divided into six thematic groups.
Results: Chat GPT 4.0 achieved a pass score (60%) in both examination sessions. Considering the total points obtained in both examination sessions, Chat GPT 4.0 achieved similar scores (72%) to physicians (74%). Significant differences were demonstrated between the results achieved by the older (48%) and newer (72%) versions of Chat GPT.
Conclusions: The results presented in our study may indicate the potential utilization of artificial intelligence in the practice of paediatric neurologists. Despite promising results, the use of AI in medicine poses serious ethical and practical challenges for physicians. Our article emphasizes the importance of further research on the use of AI in paediatric neurology and the need for continuous assessment and development of these technologies, raising issues regarding their potential applications and challenges associated with their implementation in clinical practice.
Keywords: child neurologyartificial intelligenceAIChat GPT
References
- Jiang F, Jiang Y, Zhi H, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol. 2017; 2(4): 230–243.
- Manickam P, Mariappan SA, Murugesan SM, et al. Artificial Intelligence (AI) and Internet of Medical Things (IoMT) assisted biomedical systems for intelligent healthcare. Biosensors (Basel). 2022; 12(8).
- Rao A, Kim J, Kamineni M, et al. Evaluating ChatGPT as an adjunct for radiologic decision-making. medRxiv. 2023.
- Danilov A, Aronow WS. Artificial Intelligence in cardiology: applications and obstacles. Curr Probl Cardiol. 2023; 48(9): 101750.
- Talyshinskii A, Naik N, Hameed BM, et al. Potential of AI-driven chatbots in urology: revolutionizing patient care through artificial intelligence. Curr Urol Rep. 2024; 25(1): 9–18.
- Ghorashi N, Ismail A, Ghosh P, et al. AI-powered chatbots in medical education: potential applications and implications. Cureus. 2023; 15(8): e43271.
- Hutson M. Could AI help you to write your next paper? Nature. 2022; 611(7934): 192–193.
- Centrum Egzaminów Medycznych w Łodzi. https://www.cem.edu.pl/index.php.
- Opis statystyk. Centrum Egzaminów Medycznych w Łodzi. https://www.cem.edu.pl/aktualnosci/opis_statystyk.pdf.
- https://chat.openai.com/c/3674b438-d7ab-41c0-ad95-691fbf444f58 (13.04.2024).
- Kung TH, Cheatham M, Medenilla A, et al. Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLOS Digit Health. 2023; 2(2): e0000198.
- Giannos P. Evaluating the limits of AI in medical specialisation: ChatGPT's performance on the UK Neurology Specialty Certificate Examination. BMJ Neurol Open. 2023; 5(1): e000451.
- Moshirfar M, Altaf AW, Stoakes IM, et al. Artificial Intelligence in ophthalmology: a comparative analysis of GPT-3.5, GPT-4, and human expertise in answering statpearls questions. Cureus. 2023; 15(6): e40822.
- Thirunavukarasu AJ, Ting DS, Elangovan K, et al. Large language models in medicine. Nat Med. 2023; 29(8): 1930–1940.
- Chen TC, Multala E, Kearns P, et al. Assessment of ChatGPT's performance on neurology written board examination questions. BMJ Neurol Open. 2023; 5(2): e000530.
- Lost in Translation: Large Language Models in Non-English Content Analysis. https://cdt.org/insights/lost-in-translation-large-language-models-in-non-english-content-analysis/.
- Chen L, Zaharia M, Zou J. How is ChatGPT’s behavior changing over time? [preprint]. 2023.
- Abbasi B, Goldenholz DM. Machine learning applications in epilepsy. Epilepsia. 2019; 60(10): 2037–2047.
- Lampos V, Mintz J, Qu X. An artificial intelligence approach for selecting effective teacher communication strategies in autism education. NPJ Sci Learn. 2021; 6(1): 25.
- Chaddad A, Li J, Lu Q, et al. Can autism be diagnosed with Artificial Intelligence? A narrative review. Diagnostics. 2021; 11(11): 2032.
- Al-Nafjan A, Alhakbani N, Alabdulkareem A. Measuring engagement in robot-assisted therapy for autistic children. Behav Sci (Basel). 2023; 13(8).
- Peng J, Kim DD, Patel JB, et al. Deep learning-based automatic tumor burden assessment of pediatric high-grade gliomas, medulloblastomas, and other leptomeningeal seeding tumors. Neuro Oncol. 2022; 24(2): 289–299.
- AI algorithm that detects brain abnormalities could help cure epilepsy. https://www.ucl.ac.uk/news/2022/aug/ai-algorithm-detects-brain-abnormalities-could-help-cure-epilepsy (13.04.2024).
- Morley J, Machado CCV, Burr C, et al. The ethics of AI in health care: a mapping review. Soc Sci Med. 2020; 260: 113172.
- Layard Horsfall H, Palmisciano P, Khan DZ, et al. Attitudes of the surgical team toward Artificial Intelligence in neurosurgery: international 2-stage cross-sectional survey. World Neurosurg. 2021; 146: e724–e730.