The next conference of the International Association for Health Professions Education (AMEE), taking place by the end of August in Basel (Switzerland), will introduce the successful use of Practicum Script in a high-stakes OSCE. The online simulation-based programme for developing students’ reasoning skills in uncertain situations, has recently shown good reliability, validity and acceptability for formative assessment at undergraduate level in an international large-scale pilot published in Medical Teacher. This presentation will focus on the application of Practicum Script as a core component of the final OSCE of five Spanish medical schools, which cemented a summative modality. 

Madrid, 12th June 2024. Dealing with clinical uncertainty is a key requirement for newly qualified doctors. However, assessment tools in licensing exams are limited in testing clinical judgment and the students’ ability to manage ambiguity in clinical practice, which are core competences of future practitioners. Moreover, traditional OSCEs are not able to measure the reasoning processes beyond observed results. Acknowledging the distance between the requirements for an optimal clinical performance of medical graduates and the assessment methods, in 2020 five Spanish medical schools integrated Practicum Script into their final OSCE to measure cognitive skills in decision-making in uncertain situations. The conclusion? It is effective and has good discriminative strength.

The presentation “Assessment of clinical reasoning in uncertain situations in a high-stakes OSCE: results of a multicentre experience” will describe the feasibility of using Practicum Script to assess medical students’ reasoning and problem-solving skills in authentic clinical situations as part of the OSCE for final year undergraduates. The exam, today known as the PS-OSCE test, consisted of a 3-hour set of 10 internal medicine clinical cases, concerning diagnoses, investigations, and treatments. Prior to this, students got familiar with the Practicum Script methodology thanks to a 5-week formative cycle consisting of another 10 cases. During the training, they had feedback based on experts’ answers and scientific evidence.

All the cases used for this experience had been validated by 20 experienced internists from 16 medical faculties across the United States, Latin America and Europe. The cases for the exam were selected through the agreement of an interfaculty committee, composed by senior educators and internal medicine professors from the participating medical schools: Francisco de Vitoria University, European University of Madrid, Cardenal Herrera of Castellón University, University of Zaragoza, and International University of Catalonia (UIC). The group reviewed the pool designed for a previous international large scale pilot study and chose 10 relevant cases, placing special attention to achieve a good balance between clinical domains and types of decisions required by junior doctors.
During the examination, the clinical cases were presented to the students consecutively, but in a random order to minimise the risks of copying and fraud. In total, 654 students took the PS-OSCE test. With an average time of 18 minutes per case, students were asked to generate three hypotheses in ‘free text’ format and justify them. Subsequently, they had to report, in 5 different scenarios, how new data might affect the likelihood of their original hypotheses. 

The final mark and grading of the students was based on hypothesis generation (HG), hypothesis argumentation (HA), and knowledge application (KA). Reliability estimates for the full test were good (Cronbach's alpha = 0.784, 95% CI = 0.765-0.802; McDonald’s omega = 0.828, 95% CI = 0.801-0.849). Knowledge application items were analysed using cognitive diagnostic modelling, which according to the psychometrician specialist of the European Board of Medical Assessors, and presenter of the communication at AMEE 2024, Dr Carlos Collares, “constitutes a revolutionary approach to psychometrics that enables diagnostic classification of examinees on the demonstration of a multidimensional set of cognitive attributes.” 

Also, the test level accuracy reached 0.67 and cut-off marks were determined based on the mean score and standard deviations. The mean score was 7.83 (min = 4.74, max = 9.86). Students’ performances were classified in 3 standards: good = 31.8% (+0.5 SD), satisfactory = 53.6% (+0.5 SD / -1 SD) and fail = 14.6% (-1 SD). 

It makes the difference

The findings suggest that Practicum Script may contribute to a better sampling and evaluation of the clinical competencies of final year students, complementing the flaws of the current OSCE formats for assessing reasoning processes. Beyond the solid psychometric results, this methodology may offer several advantages to medical schools, such as saving time and resources in preparing at least some stations of the OSCE or being used extensively for national licensing exams in which all participants can be tested simultaneously, using a standardised methodology and the same clinical cases.

While the OSCEs have proven to be a good assessment method in evaluating some clinical skills and problem-solving abilities through the students’ interactions with simulated patients (i.e. physical exam, communication, history-taking, etc.), traditional formats also have important constraints to test reasoning skills. Put simply, candidates are only assessed in routine and well-defined situations. They are not expected to show their thinking processes nor the way they integrate principles of Bayesian reasoning for decision-making, and the students’ abilities to make inferences under conditions of uncertainty are not tested. Practicum Script remediates this gap.