The use of animation to improve the assessment of complex competencies

Anastasios Karakolidis

There are increasing calls for assessments that measure complex competencies such as critical thinking and problem-solving skills. Such assessments often require the use of tests with longer and more complex pieces of written text requiring verbal, reading comprehension and interpretation skills, that, in some cases, are not immediately relevant to the skills that the test purports to measure (the construct-irrelevant variance problem). A number of prominent testing companies, including Prometric – funders of the Centre for Assessment Research, Policy and Practice in Education (CARPE), have identified this issue as one of the key validity concerns facing the credentialing industry. The concern is that candidates taking certification or licencing test may have the knowledge and skills required, but struggle with the assessment because of low levels of reading comprehension or poor proficiency in the language of the test.

Anastasios Karakolidis, the winner of DCU President’s Award for Innovation 2020, developed an innovative solution to this problem using animation technology. Specifically, an animated version of a situational judgement test (SJT) measuring pre-service teachers’ problem-solving skills was developed.  In this case, a text-based test using vignette descriptions was replaced by one that used 15 short animated videos developed in cooperation with a UK-based animation company. In addition, a bespoke platform to administer the animated and text-based versions of the test was developed in cooperation with a software based in Athens.

The initial prototype of the SJT was shared with several groups of subject matter experts and volunteer test-takers in a series of pilot studies. This yielded important data that informed future developments in the content, fidelity and delivery of the animated assessment. The main trial was conducted using a true experimental design with 129 native and non-native English speakers, each of whom took either the animated or the text-based version of the test in English.  Test-takers’ reading comprehension and proficiency in English were measured, in addition to their perceptions of and reactions to the test.

The results provided empirical evidence in favour of the animated version of the test. Specifically, the use of animated videos reduced the dependency of test-takers’ performance on factors that were irrelevant to the skills that the test aimed to measure, such as their reading comprehension and level of proficiency in English.  Moreover, test-takers found the animated test to be more enjoyable than its text-based counterpart, less difficult, and above all, a significantly fairer and more valid indicator of one’s interpersonal skills in the teaching profession. 

The implications of this innovation are directly relevant beyond the context of where the study took place. Examinations across the educational sector as well as licensure, certification, and personnel selection assessments often attract candidates with profiles that can significantly affect their ability to process written text. Equity and fairness demand that testing organisations ensure that decisions based on test performance are not being undermined by construct-irrelevant variance. Additionally, the use of animated videos could also be valuable in non-cognitive measurement contexts. For  instance, the use of animated videos instead or on in addition to verbal messages could accommodate young children who do not have fully-functional verbal language, allowing them to comprehend items in psychological assessments.

This innovation has already been shared with educational and industry colleagues at peer reviewed conferences. A number of papers are currently being prepared for submission to top quartile peer reviewed journals. Many individuals and companies have expressed substantial interest and enthusiasm for the concept of animated computer-based assessments, especially in the context of SJTs for the measurement of complex competences. The animation approach is also being considered by a number of Prometric’s clients and by the company in planning for how to mitigate low literacy levels of candidates taking tests the company oversees.