Annual CARPE Lecture 2019: What you always wanted to know about Process Data

On Monday, 21st October 2019, CARPE hosted its fourth annual public lecture. This year's topic was ‘Process Data: What you wanted to know but were too afraid to ask’, and we were pleased to host the highly accomplished Dr. Mathias von Davier to deliver the lecture. This video available to view under the 'Lecture Resources' section of our website or here.

Mathias holds the Distinguished Research Scientist position at the National Board of Medical Examiners (NBME), in Philadelphia since January 2017. Prior to this appointment, he was a Senior Research Director in the Research & Development Division at Educational Testing Service (ETS), and co-director of the center for Global Assessment at ETS. He helped to lead the psychometric research and operational analysis of the center in relation to a range of large-scale assessments such as PISA, PIAAC, TIMSS and PIRLS. He is currently the editor of Psychometrika and has written extensively in the fields of education, research methodology and psychometrics.

In his lecture, Mathias provided the audience with an overview of process data, explaining it as those actions (such as time spent on task or number of mouse-clicks) that are automatically recorded by a computer in their system logs as coded and time-stamped strings of data. These strings can be used for instant feedback to students, or for diagnostic and scoring purposes at a later time. Mathias noted that process data has the potential to enrich psychometric methods and can allow us to gain a greater understanding of test-takers’ knowledge, skills and abilities. Mathias provided some compelling case studies that highlighted the value of process data for teachers and learners. For example, he discussed how educationalists working with vocational students in third-level institutes in the Netherlands used process data to better understand what strategies car mechanic apprentices use when trying to diagnose car problems. This then informed the design of their future lectures and courses.

The audience offered some thoughtful comments and questions. Prof. Gerry Shiel from the Educational Research Centre (ERC) began the responses to Mathias’ lecture, noting the relevance of this topic to research currently being conducted within the Irish context. In particular, Gerry highlighted how current research by the ERC aims to use process data to better understand Irish performance in PISA. However, he cautioned that there was a significant need for investment in the field in order to ensure that researchers have the capacity to work with such data. Other members of the audience noted that it may take several years before we make any substantial moves forward in integrating process data into standard analysis practices. The team at CARPE are extremely grateful to all those who attended the event. We would also like to thank Mathias for all his work in preparing this engaging lecture, Gerry Shiel for his insightful response to some of the issues raised by Mathias, and Paul Murphy, who captured the photographs and video on the day