At AES 24, Katrina Barnes will present on work she conducted at Oxfam, and Oxfam’s journey to decolonise its knowledge and evaluation processes. Read about it in this article!
AES 2024: This pre-conference workshop intends to empower emerging evaluators with a foundational understanding of Measurement, Evaluation, and Learning (MEL). Learn more.
By now I’ve had a few weeks to reflect on my first Australian Evaluation Society conference, where I was exposed to an amazing variety of evolving and inspiring ideas from within the Australasian evaluation community.
On the first day I found myself inspired by Dr Tracey Westerman, who raised the dubiousness of using or adapting Western-focused data collection tools in a First Nations context. For me, this highlighted the necessity of remaining reflective and adaptive in our approach, as well as tailoring every evaluation’s approach and methods to the context, preferably in partnership with community. This in turn made me reflect on the work of one of my Clear Horizon colleagues, Skye Trudgett, a proud Aboriginal woman who is passionately pursuing Indigenous Data Sovereignty and is leading a program to build First Nations young women’s evaluation capacity in remote Australia.
Following Dr Tracey Westerman’s plenary, I attended a Systems Evaluation Theory Application presentation by Lewis Atkinson, Brian Keogh and Ralph Renger, which helped frame my thinking regarding complexity. I found myself identifying with the concept of cascading success or failure as a neat way to consider the upstream challenges which produce a downstream issue. I could see the concept’s similarity to framing challenges through approaches such as the Problem Tree and found it a resonating concept in which to couch my thinking on casual pathways.
My third and final reflection was on the space for emerging evaluators. The conference provided a valuable sounding board for ideas and challenges facing those new to the profession and was highlighted on the final day by Eunice Sotelo and Francesca Demetriou, who are conducting research on these experiences. I found myself identifying closely with the key findings, and introspectively, the session highlighted a need to establish a community of support and practice for emerging evaluators. Personally, I will be seeking to develop an informal network, but I believe that an AES-associated group would be invaluable in attracting, retaining and developing those new to the space. Young people are not only the future (inevitably), but have the potential to bring new ideas, perspectives and approaches to evaluation. I feel that this potential influx of creativity and vigour should be encouraged and enhanced through more formal arrangements.
As Kailash Satyarthi is credited as saying, ‘No segment in the society can match the power, idealism, enthusiasm and courage of the young people’, or in this context, young (not necessarily age-specific) evaluators.
It’s been an amazing AES conference so far – lots of interesting topics and great conversations. Like Jess, the highlight for me so far has been the key note speaker from day one, Dr Tracey Westerman – an Aboriginal woman from the Pilbara in WA. She has been a trail blazer in Aboriginal mental health. The key take away message for me was that measurement matters – but even more importantly, the right measures matter. She described that in many cases of Aboriginal youth suicide, there was been no prior mental health assessment. But when assessment tools are used, they are western based and not culturally appropriate. This can lead to misdiagnosis. For over 20 years, Tracey has argued that it is not appropriate to ‘modify’ existing measures because of their inherent racism. The only way is to develop new tools from the ground up. Tracey has developed seven tools specifically for Aboriginal youth mental health with not a lot of funding – no easy feat. It was a truly inspiring presentation from an amazingly passionate and optimistic woman who really cares about her people.
A highlights from day 2 was a panel of designers and evaluators from Australia and New Zealand: Jess Dart, Kate McKegg, Adian Field, Jenny Riley and Jacqueline (Jax) Wechsler, who explored how we might move out of the traditional box of program evaluation, to make a bigger difference. They discussed the role of evaluators in supporting people to move beyond measuring to think though whether we are doing the right things and whether we are really making a difference across complex systems. Questions were covered such as where can evaluators add value in a co-design process, does evaluation get in the way and slow things down, do evaluators need new skills to help analyse and make sense of big data? Jenny reminded us that evaluators are learners and we are curious, and that we need to get on board with the digital revolution.
One really interesting concurrent session I attended was on the use of Rubrics by Julian King, Kate McKegg, Judy Oakden and Adrian Field. They presented the basics of rubric and then described how rubrics can be a tool for democratising evaluative reasoning, stakeholder engagement and communicating of results. They presented two very different examples– one in a developmental evaluation and the other was using rubrics to evaluate the value for money of an agricultural funding program. I found the second example particularly interesting having experienced the challenges of answering the value for money question. Using a rubric in this way is great for balancing the multiple dimensions of value from different perspectives.
Another memorable moment was at an ignite session (which is a really short presentation). Damien Sweeny and Dave Green from Clear Horizon did a great job at presenting a rather convincing argument for placing more emphasis on monitoring over evaluation – Big M vs small e as they call it. And they cheekily suggested changing the name of the AES to AMeS. An interesting thought.
The day finished with a second keynote speaker, Gary VanLandingham, from the Askey School of Public Administration and Policy. He reminded us of the vast amount of evaluative information available through ‘What Works’ warehouses. They are a good place to start when starting an evaluation, but there are warnings. The main caution for me is that they privilege certain types of data over others, and they don’t include what doesn’t work or things not measured using experimental approach (such as randomised control trials, and quasi-experimental methods).
The day was topped off by a short ferry ride to Luna Park where we had a lovely conference dinner overlooking the opera house. Sydney is a very beautiful city and a great setting for a wonderful conference.
Now for day three….
Have you visited our booth at the conference?
Dr Tracey Westerman, proud Aboriginal woman, had me totally gripped throughout her keynote presentation on Day 1 at the AES International Evaluation Conference. She began with the statistic that Australia has the highest rate of child suicide in the word. But she cautioned us to be optimistic and focus also on the positive outcomes that are occurring, such as the six young Aboriginal people graduating from medicine in WA this year alone. She stressed that education is the most powerful solution in the world and described how in one generation her family ‘closed the gap’ – she gained a doctorate despite living very remotely, with a family background of very limited formal education.
She walked us through the importance of developing assessment tools that are culturally sensitive and to avoid confusing causes with risk factors. It seems many tools in existence today are culture-blind and can lead to stereotyping and actual discrimination. She has developed a whole range of specific assessment tools that are culturally sensitive, including an assessment tool for work-based culture. She made the case that there hasn’t been the right sort of research with Aboriginal people, that the causes are different, and need to be assessed in culturally sensitive ways.
She’s working on ‘building an army’ of Indigenous psychologists across the country to address child suicide and influence child protection. She ended with the note that there is nothing that can’t be achieved by Aboriginal people if they believe in themselves.
After this I had the privilege of moderating a panel about the client-consultant relationship, a topic dear to my heart and my business! The panel was from the Victorian Department of Education and training (DEET), as well as consultants from Melbourne Uni and Deloitte Access Economics. DEET have set up a ‘state of the art’ supplier panel with over 30 suppliers on it, and are working to deepen the partnership between evaluation suppliers and commissioners, as well as embedding a culture of evaluation. They were generous in sharing their challenges, including lots of tricky moments around data sharing and IP.
Just before lunch I had the pleasure of indulging in a session of evaluation theory led by Brad Astbury and Andrew Hawkins from (ARTD). The explored the seven great thinkers of evaluation, who laid the foundations of our seven-decade-long journey of building our theoretical foundations. So lovely to wallow in theory, I remember savouring that learning when studying for my PhD. Their conversation was framed around Foundations of Program Evaluation by Shadish, Cook and Leviton (1991) – it was my evaluation textbook back then and good to see it’s still valued!
It was a busy day for me, and I also convened a fun panel on digital disruption. We had a great spread of panellists, with a designer from Paper Giant, Ruben Stanton; a data scientist, Kristi Mansfield; a social procurement expert, Chris Newman; as well as our very own Chief Innovation Officer, Jenny Riley. We explored the trends, the opportunities and the scary stuff that might come with the fourth industrial revolution – yes, the robots are here! I saw a few people jump in their seats when Jen described how Survey Monkey stores data overseas and is not bound by the same data security regulations. We also looked into the amazing opportunities for evaluators to be the sensemakers of Big Data. When the firehose of data hits you in the face, maybe the evaluators will be there to bring you calmly back to the most important questions. We also explored the opportunities for evaluators to get involved in evaluating new technology and innovation, and to help consider how ready the system is to receive these innovations. I didn’t want it to end!
The day was topped off with a keynote from David Fetterman on empowerment evaluation. Empowerment evaluation is now 26 years old! David explained how empowerment evaluation is a self-evaluation approach designed to help people help themselves. It felt familiar!
Designing Rubrics for Evaluation
Last week, I attended an Australasian Evaluation Society (AES) workshop on “Foundations of Rubric Design”. It was a thought-provoking workshop. Kystin Martens, the presenter of the workshop, explained and challenged our understanding about rubric design as well as presented some practical tips to develop and use rubrics properly in evaluation. Here are my key takeaway points from the workshop:
Why do we use rubrics in evaluation?
A rubric is a tool or matrix or guide that outlines specific criteria and standards for judging different levels of performance. These days, more and more evaluators are using rubrics to guide their judgement on program performance. Rubrics enable evaluators to transform data from one form to another form, for example from qualitative evidence into quantitative data. Rubrics also provide an opportunity to analyse and synthesis evidence into a general evaluative judgement transparently throughout the evaluation process.
Three systematic steps to create a rubric for evaluation
There are three logical steps to develop a rubric:
Ensuring reliability of judgement and creating gold standards in evaluation
A calibration process is required so all evaluators will assess the program performance consistently and in alignment with the scoring rubric. This process will ensure that all evaluators will produce a similar evaluation score when assessing same program performance. This is a critical process to create gold standards for assessment and increase reliability of the assessment data. For example when we evaluate a multi-country development program and deploy more than one evaluator to assess the program, we need to make sure that all evaluators agree upon the rubric and understand the performance expectations expressed in the rubric thus they are able to interpret and apply the rubric consistently.
If you have any experience using rubrics in evaluation, please share and tweet your experience and thoughts with us @ClearHorizonAU.
References
Tools for Assessment. Retrieved from: https://www.cmu.edu/teaching/assessment/examples/courselevel-bycollege/hss/tools/jeria.pdf
Rubrics: Tools for Making Learning Goals and Evaluation Criteria Explicit for Both Teachers and Learners. Retrieved from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1618692/
Roger, Patricia. Rubrics. Retrieved from: https://www.betterevaluation.org/en/evaluation-options/rubrics
132B Gwynne St, Cremorne 3121 VIC
P (+61) 3 9425 7777 | info@clearhorizon.com.au
Clear Horizon acknowledges Aboriginal and Torres Strait Islander people as the Traditional Custodians of the land and acknowledges and pays respect to their Elders, past and present.
