Spotlight on Research: Supporting Societal challenges through Evaluation

This week’s Spotlight on Research is with Dr Martin Brown, Assistant Professor at the School of Policy and Practice and Researcher at the Centre for Evaluation, Quality and Inspection (EQI) DCU Institute of Education

You work in the area of evaluation in education, can you explain?

Evaluation can cover a broad spectrum of areas. With colleagues at EQI, I would, for example, carry out a variety of research projects such as comparative education studies that examine how school or tertiary evaluations operate in other jurisdictions.

Within this, we would also investigate the role of various stakeholders such as parents and students in evaluation, community-based evaluation using, for example, social network theory.

We also research other areas such as network governance and what is now gaining a lot of momentum in Europe given the refugee crisis, a subset of evaluation referred to as Culturally Responsive Evaluation and Assessment.

A significant body of research that I’d carry out also relates to that of school evaluation, examining the fault lines between evaluation for accountability and evaluation for improvement. What are the benefits, issues, unintended consequences of such systems? What are the capacity deficits, what professional learning is provided and required for inspectorates and schools in various jurisdictions? Are the models of continuing professional development that are provided having the desired impact? Is there a need for capacity building in for example, quantitative, mixed methods, that will assist schools with, for example, nuanced terms such as Data Informed Decision Making?

What are the issues facing evaluation in education today?

First and foremost, time will always be an issue. However, without in any way being cold about it, in the modern era, this is the case with all aspects of society.

This aside, regarding educational evaluation, school self-evaluation for example, I’ve quite frequently heard it stated that teachers require capacity building on how to carry out the six-step process for school self-evaluation that we have in Ireland.

That is true to an extent, but sometimes I think that the message and value of school evaluation is lost in translation. Teachers in Ireland are massively competent, highly trained and dedicated professionals who I have no doubt could very easily learn how to carry out what is required of school self-evaluation.

So we hear about the need for upskilling in areas such as data-informed decision making, etc. which is undoubtedly true. However, to be honest, with time and high-quality training provided, online or otherwise, I can’t imagine School Self-Evaluation being particularly taxing for teachers. However, issues remain.

What’s the issue there?

The main issue for me is to show that, yes, for example, school self-evaluation or indeed, the outcomes of an inspection can, in fact, lead to improvement and here is the empirical evidence to suggest why it leads to improvement. In other words, how does one make high-quality empirical studies more explicitly known to the profession?

Embedding evaluation or any new initiative into a system requires a lot of convincing and if teachers are in some way convinced or shown how evaluation will improve the quality of teaching and learning - and it’s not this superfluous thing that’s being bolted onto an already initiative heavy system - then I have no doubt that the vast majority of teachers will respond if they see the process being of benefit to teaching and learning.

How can you help?

As evaluators, we need to look at some of the genuinely new challenges facing society and, in my case, education and evaluation and to see how we as evaluators can in some way help.

For example, to take one issue of which there are many. There are approximately 250 million migrants in the world and of this cohort, around 15% are under the age of 20. Yet, in most education systems, despite all of the high-quality resources and supports that exist, we still have evaluation and assessment systems that, and one can understand why are specifically designed for the majority culture.

So, for example, in Ireland, we have the Leaving Certificate, of which I’m a fan of and has served our nation very well. However, in today’s multicultural society, how does this system of assessment equitably caters for migrant background children despite all of the evidence to suggest that academic proficiency in the test language can take up to seven years to master. In other words, should the concept of what might be referred to as culturally responsive evaluation and assessment be brought more to the fore at all levels of education.

As evaluators, we can help by with ethics to the fore, asking if various macro and micro level policies and practices do in fact, in some way, improve an aspect of education. If not, why not? What are the proposed and hopefully, in a democratic, consensus-based society, debatable solutions? A tall order, given the limitations of time.

What other aspects of evaluation are you looking at?

Apart from coordinating the two Erasmus+ projects (called Aiding culturally responsive Assessment and Distributed Evaluation and Planning in Schools), I’m also working with colleagues in Northern Ireland on a project evaluating the use of digital technologies in initial teacher education as well as two projects with Arizona State University on intrapreneurship and the perceived quality and efficacy of professional doctorates in Europe and the States.

Fingers crossed, there are also two more Erasmus+ bids that were recently submitted on accountability of school networks and network governance that have great potential. One never knows.

Is Ireland a good place to work on bettering evaluation?

So, I had this most inspirational English teacher when I was at school by the name of Tommy Halferty, an excellent jazz Musician as well I might add. He would sometimes tell us about different education systems in areas of Europe such as Northern Ireland and I suppose, from this early age, I've always been fascinated with the fact that you can drive a few miles up the road and can quite easily carry out research, comparative or otherwise on a different health, judicial or education system to what we have down here. So, yes, the Island of Ireland really is a fascinating place to work on bettering evaluation.

I think that is primarily down to the fact that various actors in the system such as inspectorates, schools, teachers, parents, policymakers, academics, etc. have a strong conviction on the central importance of education in society.

So, even with the limited time that they have, from my own experience anyway, they're very generous with the time that they give to researchers like myself, always 99% of the time, with a disclaimer though that the research will in some way, impact in a positive sense, those we as educational researchers are privileged to serve; namely parents, students and teachers.

In the case of Northern Ireland for example, I’d do a fair bit of research and evaluation capacity building with colleagues, schools and other stakeholders in Northern Ireland and you can really get excellent insights into the educational challenges, hopes, dreams and aspirations for education in these communities.

Undoubtedly Brexit will be a challenge though. However, regardless of what will happen, without fair of contradiction, I've no doubt that both systems at all levels of education will continue to support and learn from each other.

What is the best thing about being a researcher?

Well I’m very fortunate that I do research with a cadre of highly skilled and knowledgeable colleagues, not only in EQI and DCU but in other institutions and countries, all with their own unique skills and perspectives on education that, in a positive way, challenge my own research and ultimately, hopefully, improve the quality of outputs. On a personal level though, the best thing about being a researcher is that your research has or might in some way potentially improve the quality of life chances for students and society.

And the challenges?

More generally, the most significant challenge is to always ask yourself, regardless of the research that you do, if your research has had, if even in a small way, any impact at all. One can’t assume that it always does. There’s also a challenge to relaying the results of an evaluation, especially when the outcome of the evaluation might not be that positive. However, once you’re polite about it and have a framework of indicators, it’s not so bad really.

One other significant challenge, especially when managing international projects, is to find a balance between the time devoted to the enjoyable part of the project, the research, and that of the phenomenal amount of time devoted to the administration and management of the project. A lot of the time, the latter wins I fear. Also, to politely ensure that the various international researchers working in distant locations are meeting targets, etc.

How do you spend your time away from work?

Well, I’d like to say that I do exciting activities away from work like rock climbing but no, what’s left of my time away from work apart from time spent with family, would be spent in the garden or going for a walk, supporting Dublin in the Hurling and of course, lest not we forget the glorious Liverpool, despite the minor setbacks of late.