Artificial intelligence based on large language models has gained increasing relevance in educational and healthcare settings due to its ability to process information and generate coherent natural language responses. In the health sciences, several studies have evaluated the performance of these systems in academic and professional licensing examinations; however, evidence regarding their performance in dentistry, particularly in Latin American contexts, remains limited. This study aimed to compare the response accuracy of four artificial intelligence language models in answering the 2023 CODIRO Dental Residency admission examination in Peru, using the official answer key as the reference standard. An observational, cross-sectional, comparative study was conducted using the complete set of 100 multiple-choice questions from the examination. Claude, Copilot, Duck IA, and Qwen were each administered the same standardized prompt for every question, and responses were classified as correct or incorrect against the official key. Because all four models answered the same questions, accuracy was compared using Cochran's Q test, with pairwise exact McNemar tests and Holm correction applied when the omnibus test was significant. Copilot achieved the highest accuracy (100%), followed by Claude and Qwen (96% each) and Duck IA (86%); Cochran's Q test indicated a significant overall difference (p < 0.001). Duck IA performed significantly worse than the other three models, without significant differences among them. Agreement among models was highest in clinical disciplines and lowest in foundational disciplines such as anatomy and histology/embryology. Among items answered incorrectly, failing models frequently selected the same incorrect option, suggesting shared misconceptions rather than independent errors. These findings suggest that large language models, including platforms with limited representation in the dental education literature, can achieve high accuracy on dental residency examination questions; however, this reflects knowledge recognition rather than clinical competence, and supports their potential only as complementary tools in dental education.
Citation: Edgar S. Povis-Flores, Mariana A. Ramírez-Benavides, Fabricio Meza-Ponce, Andrea G. Ramirez-Acuña, Mónica L. Ramírez Benavides, Miguel A. Saravia-Rojas. Comparative performance of artificial intelligence models on the Peruvian CODIRO 2023 dental residency examination[J]. Journal of Dentistry and Multidisciplinary Sciences, 2026, 2(1): 52-61. doi: 10.3934/jdms.2026007
Artificial intelligence based on large language models has gained increasing relevance in educational and healthcare settings due to its ability to process information and generate coherent natural language responses. In the health sciences, several studies have evaluated the performance of these systems in academic and professional licensing examinations; however, evidence regarding their performance in dentistry, particularly in Latin American contexts, remains limited. This study aimed to compare the response accuracy of four artificial intelligence language models in answering the 2023 CODIRO Dental Residency admission examination in Peru, using the official answer key as the reference standard. An observational, cross-sectional, comparative study was conducted using the complete set of 100 multiple-choice questions from the examination. Claude, Copilot, Duck IA, and Qwen were each administered the same standardized prompt for every question, and responses were classified as correct or incorrect against the official key. Because all four models answered the same questions, accuracy was compared using Cochran's Q test, with pairwise exact McNemar tests and Holm correction applied when the omnibus test was significant. Copilot achieved the highest accuracy (100%), followed by Claude and Qwen (96% each) and Duck IA (86%); Cochran's Q test indicated a significant overall difference (p < 0.001). Duck IA performed significantly worse than the other three models, without significant differences among them. Agreement among models was highest in clinical disciplines and lowest in foundational disciplines such as anatomy and histology/embryology. Among items answered incorrectly, failing models frequently selected the same incorrect option, suggesting shared misconceptions rather than independent errors. These findings suggest that large language models, including platforms with limited representation in the dental education literature, can achieve high accuracy on dental residency examination questions; however, this reflects knowledge recognition rather than clinical competence, and supports their potential only as complementary tools in dental education.
| [1] |
H. K. Jin, H. E. Lee, E. Kim, Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: A systematic review and meta-analysis, BMC Med. Educ., 24 (2024), 1013. https://doi.org/10.1186/s12909-024-05944-8 doi: 10.1186/s12909-024-05944-8
|
| [2] |
A. Holzinger, K. Keiblinger, P. Holub, K. Zatloukal, H. Müller, AI for life: Trends in artificial intelligence for biotechnology, New Biotechnol., 74 (2023), 16-24. https://doi.org/10.1016/j.nbt.2023.02.001 doi: 10.1016/j.nbt.2023.02.001
|
| [3] |
A. Montejo-Ráez, S. M. Jiménez-Zafra, Current approaches and applications in natural language processing, Appl. Sci., 12 (2022), 4859. https://doi.org/10.3390/app12104859 doi: 10.3390/app12104859
|
| [4] |
F. F. H. Nah, R. Zheng, J. Cai, K. Siau, L. Chen, Generative AI and ChatGPT: Applications, challenges, and AI-human collaboration, J. Inf. Technol. Case Appl. Res., 25 (2023), 277-304. https://doi.org/10.1080/15228053.2023.2233814 doi: 10.1080/15228053.2023.2233814
|
| [5] |
P. Lee, S. Bubeck, J. Petro, Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine, N. Engl. J. Med., 388 (2023), 1233-1239. https://doi.org/10.1056/NEJMsr2214184 doi: 10.1056/NEJMsr2214184
|
| [6] |
L. Labadze, M. Grigolia, L. Machaidze, Role of AI chatbots in education: Systematic literature review, Int. J. Educ. Technol. High. Educ., 20 (2023), 56. https://doi.org/10.1186/s41239-023-00426-1 doi: 10.1186/s41239-023-00426-1
|
| [7] |
A. Gilson, C. W. Safranek, T. Huang, V. Socrates, L. Chi, R. A. Taylor, et al., How does ChatGPT perform on the United States Medical Licensing Examination? The implications of large language models for medical education and knowledge assessment, JMIR Med. Educ., 9 (2023), e45312. https://doi.org/10.2196/45312 doi: 10.2196/45312
|
| [8] |
A. J. López-Galisteo, O. Borrás-Gené, The creation and evaluation of an AI assistant (GPT) for educational experience design, Information, 16 (2025), 117. https://doi.org/10.3390/info16020117 doi: 10.3390/info16020117
|
| [9] |
T. Camarata, L. McCoy, R. Rosenberg, K. R. T. Grellinger, K. Brettschnieder, J. Berman, LLM-generated multiple choice practice quizzes for preclinical medical students, Adv. Physiol. Educ., 49 (2025), 758-763. https://doi.org/10.1152/advan.00106.2024 doi: 10.1152/advan.00106.2024
|
| [10] |
G. J. Hwang, H. Xie, B. W. Wah, D. Gašević, Vision, challenges, roles and research issues of artificial intelligence in education, Comput. Educ. Artif. Intell., 1 (2020), 100001. https://doi.org/10.1016/j.caeai.2020.100001 doi: 10.1016/j.caeai.2020.100001
|
| [11] | S. Y. Lin, Y. Y. Hsu, S. W. Ju, P. C. Yeh, W. H. Hsu, C. H. Kao, Assessing AI efficacy in medical knowledge tests: A study using Taiwan's internal medicine exam questions from 2020 to 2023, Digit. Health, 10 (2024), 20552076241291404. https://doi.org/10.1177/20552076241291404 |
| [12] |
T. Kawahara, Y. Sumi, GPT-4/4V's performance on the Japanese National Medical Licensing Examination, Med. Teach., 47 (2025), 450-457. https://doi.org/10.1080/0142159X.2024.2342545 doi: 10.1080/0142159X.2024.2342545
|
| [13] | M. A. Saravia-Rojas, C. Mendiola-Aquino, F. Orejuela-Ramirez, W. Tunquipa-Chacón, R. Geng-Vivanco, ¿Pueden los grandes modelos de lenguaje de inteligencia artificial aprobar el Examen Nacional de Odontología en el Perú, Rev. Estomatol. Herediana, 35 (2025), 305-311. https://doi.org/10.20453/reh.v35i4.6253 |
| [14] |
C. C. Lin, J. S. Sun, C. H. Chang, Y. H. Chang, J. Z. C. Chang, Performance of artificial intelligence chatbots in National dental licensing examination, J. Dent. Sci., 20 (2025), 2307-2314. https://doi.org/10.1016/j.jds.2025.05.012 doi: 10.1016/j.jds.2025.05.012
|
| [15] |
T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepaño, et al., Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models, PLoS Digital Health, 2 (2023), e0000198. https://doi.org/10.1371/journal.pdig.0000198 doi: 10.1371/journal.pdig.0000198
|
| [16] | R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, et al., On the opportunities and risks of foundation models, preprint, arXiv: 2108.07258. |
| [17] | N. Norton, Anatomía de Cabeza y Cuello Para Odontólogos, 4th edition, Elsevier, Barcelona, 2007. |
| [18] | T. Rakosi, Ortopedia Maxilar, SALVAT, Barcelona, 1992. |
| [19] |
N. Liu, S. Sonkar, R. G. Baraniuk, Do LLMs make mistakes like students? Exploring natural alignment between language models and human error patterns, in Artificial Intelligence in Education (AIED 2025), 15880 (2025), 364-377. https://doi.org/10.1007/978-3-031-98459-4_26 doi: 10.1007/978-3-031-98459-4_26
|
| [20] |
E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, et al., ChatGPT for good? On opportunities and challenges of large language models for education, Learn. Individ. Differ., 103 (2023), 102274. https://doi.org/10.1016/j.lindif.2023.102274 doi: 10.1016/j.lindif.2023.102274
|
| [21] |
M. Sallam, ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns, Healthcare, 11 (2023), 887. https://doi.org/10.3390/healthcare11060887 doi: 10.3390/healthcare11060887
|
| [22] |
H. A. Aluko, A. Aluko, G. A. Offiah, F. Ogunjimi, A. O. Aluko, F. M. Alalade, et al., Exploring the effectiveness of AI-generated learning materials in facilitating active learning strategies and knowledge retention in higher education, Int. J. Organ. Anal., 34 (2026), 1243-1267. https://doi.org/10.1108/IJOA-07-2024-4632 doi: 10.1108/IJOA-07-2024-4632
|
| [23] |
G. E. Miller, The assessment of clinical skills/competence/performance, Acad. Med., 65 (1990), S63-S67. https://doi.org/10.1097/00001888-199009000-00045 doi: 10.1097/00001888-199009000-00045
|
| [24] |
Z. Niu, X. Kuang, J. Chen, X. Cai, P. Zhang, The application and challenges of ChatGPT in laboratory medicine, Adv. Lab. Med., 6 (2025), 397-410. https://doi.org/10.1515/almed-2025-0080 doi: 10.1515/almed-2025-0080
|
| [25] |
A. Rao, J. Kim, M. Kamineni, M. Pang, W. Lie, K. J. Dreyer, et al., Evaluating ChatGPT as an adjunct for radiologic decision-making, J. Am. Coll. Radiol., 20 (2023), 990-997. https://doi.org/10.1016/j.jacr.2023.05.003 doi: 10.1016/j.jacr.2023.05.003
|
Edgar S. Povis-Flores, Mariana A. Ramírez-Benavides, Fabricio Meza-Ponce, Andrea G. Ramirez-Acuña, Mónica L. Ramírez Benavides, Miguel A. Saravia-Rojas. Comparative performance of artificial intelligence models on the Peruvian CODIRO 2023 dental residency examination[J]. Journal of Dentistry and Multidisciplinary Sciences, 2026, 2(1): 52-61. doi: 10.3934/jdms.2026007
DownLoad: