This website uses cookies to enhance your experience in navigating the site, to personalize your academic tutoring, and to contribute to our research in education. By continuing, we understand that you accept the use of cookies. I Accept Cookies policy

Methodology of Psychometric Analysis
of the Practicum Script Concordance Test

The psychometric analysis of the results of the Practicum Script Concordance Test is carried out in various stages.

In the first stage, the analysis is done based on the results of the reference experts when the test is being done and then the study is complemented with the results of the professionals who take part in the medical training activities.

We use the following methods for the psychometric analysis of the results of both the validating experts’ results and the participants’ results:

Descriptive statistics to analyze the items and identify the different factors that make it possible to establish the structure of the tests

This type of analysis enables a first evaluation of the quality of the questions. It is based on the total score on the test, on the results on each of the clinical cases, and on the answers to each of the items (questions) in each clinical case.

At this stage, the aim is to establish the normality of the distributions of the scores for each of these three levels (total test group, clinical case, and items) to verify that they correspond to acceptable profiles for a good assessment test.

The statistical indices calculated include:

  • Average measurement
  • Standard deviation
  • Analysis of skewness and kurtosis

Exploratory and confirmatory factor analyses to verify the stability of the factors or dimensions that make up the Practicum Script Concordance Test, taking into account that each training activity (complete test) is composed of clinical cases in different subject areas within the specialty and includes different types of medical decisions (about diagnoses, complementary studies, and treatment).

Comparative analyses among different groups of validating experts, in the cases where a particular test group had been validated by reference panels from different countries, with the aim of stabilizing the reference group or subgroup for the participants in each training activity. The methods used for this type of study are:

  • Analysis of variance (ANOVA), when there are more than two groups of validating experts (reference panels)
  • ANOVA is followed by Levene’s test and Bartlett’s test to verify the homogeneity of variances. If the variances are unequal, Welch’s test for calculations of F with unequal standard deviations is used.
  • The T-Test whenever dealing with only 2 groups of experts.

Analyses of items, through reliability studies to measure the overall homogeneity of the questions (items) within the test group, also analyzing each question’s capacity for discriminating among participants in function of their responses.

Generalizability studies, first with a crossed-facets design and then with facets with nested questions, which make it possible to calculate the proportion of each element in the test to the overall score observed. These analyses also make it possible to determine the impact of the subgroups analyzed on the overall scores and in particular the variances among different groups of participants, as well as the impact of the order of the cases on the results observed.

N.B. Except when calculating the overall scores, for the study of the coefficients, the grouped results of the items corresponding to each case are used to avoid signal amplification bias in the intercorrelations owing to the self-correlation among items within the same case.

Prof. Dr. Carlos A. Brailovsky Kurlat
International Consultant for the Practicum Script proyect. Professor emeritus at Laval University.

Try Practicum Script free of charge

Script Connect
Personalized help center

Our personalized help center enables you to obtain technical support and help for navigating through the site and using the program.

Frequently asked questions

I accept the data privacy policy