Prospective validation of a core curriculum progress assimilation instrument for radiation oncology residentship

Abstract

Objectives

To develop a tool that could assess residents’ knowledge beyond simple information gathering and evaluate its reliability.

Methods

An assessment tool of 40 objective questions and at least one essay-based question was developed to assess residents’ comprehension of general radiation oncology accordingly to validated training curricula beyond level 2 in the Bloom scale. With randomized content, questions were developed such as to be classified as at least 2 in the Bloom scale, so that reasoning and not only information gathering could be assessed. Criteria validation was made using the Classical Test Theory to describe difficulty and discrimination of each item. Reliability was tested by internal consistency using the Cronbach alpha test.

Results

Between 2016 and 2019, 24 residents were assessed. Six different versions of the test were made with a total of 240 objective questions and 18 essay-based questions. Five of the six versions were deemed valid and reliable. Comparisons between 1st (PGY-1) and 3rd (PGY-3) year residents were made. Consistently, PGY-3 residents had scores 150% higher than PGY-1 residents. Only two different PGY-3 reached the most complex level of answers in the essay-based questions. The results demonstrated that the major part of the acquired knowledge and retention occurs in the first six months of training rather than in all the following period.

Conclusion

The instrument can be considered valid. This developed instrument also raised the hypothesis that residents may not assess and analyze their acquired knowledge beyond the application level.