School evaluation indicators: How they help us understand the education system

A growing desire for systems evaluation

The recent Tomorrow’s Schools Independent Taskforce report emphasised the need for increased evaluation of education system performance in New Zealand. The Government’s response to the review confirmed that ERO will increasingly undertake evaluation and assessment to provide system level insights:

ERO… will ensure their… research and evaluation functions provide a strong basis for generating effective system level information and evaluation that informs prioritisation, action, and improvement.

This article explores how aggregated information from ERO’s school reviews can inform our understanding of education system performance.

How does ERO make a judgment about a school?1

In recent times, ERO has provided the school and its community with a report after completing a review, which includes an overall judgment of how well the school is achieving and supporting students to succeed. A school could receive a judgment of “needs development”, “developing”, “well-placed” or “strong”.

The high-level overall judgments are informed by ERO’s school evaluation indicators. These indicators represent what we know from the research evidence about great schools, and reviewers use their judgment to evaluate how close schools are to achieving this goal for each indicator.

The school evaluation indicators are split broadly into outcomes and processes. Outcome indicators are focused on learners and include student outcomes such as progress and achievement. They assume a holistic approach to learners’ wellbeing, development and success. Process indicators describe the conditions and practices that contribute to school effectiveness and improvement, such as stewardship, quality teaching or internal evaluation.

How can the school indicators support systems evaluation?

Traditionally, school evaluation indicator judgments have only contributed towards a school’s overall review judgment. However, these indicator judgments contain a wealth of information about specific areas in which schools are performing well or may need further development. When indicator judgments are compared across many schools, we can identify strengths and areas in need of development across the education system.

In this article we look at 105 school reviews to see how ERO review teams assessed performance across three student outcomes and eight school processes. This sample is made up of reviews completed during term 4 2019 and the beginning of term 1 2020.

What we found

Schools received an overall judgment of well-placed or strong (ERO’s two highest judgments) in 64 percent of reviews. However, the judgment on specific student outcomes and school processes varied substantially. These results are summarised in the figure below, which shows the percentage of schools which were judged as well-placed or strong for each of the student outcome and school process domains, across the 105 reviews analysed.

Figure 1: Percentage of schools that received an overall judgment of well-placed or strong in student outcome and school process domains

Overall judgement graphic

Schools in this study performed the best on the student outcome “Confident, connected, lifelong learners”, with 77 percent of schools judged as well-placed or strong. Other well-performing indicators were “Effective teaching and opportunities to learn” and “Responsive curriculum”, with 70 percent of schools being judged as well-placed or strong for each of these processes.

At the other end of the spectrum, two indicators stand out as areas for development. Only 49 percent of the schools in this study received a well-placed or strong judgment for the student outcome indicator “Acceleration for students/groups of students”. Similarly, only 50 percent of the schools received the rating of well-placed or strong for the school process indicator “Evaluation for improvement and innovation”.

ERO has undertaken work to build school capability for effective internal evaluation and accelerating student achievement in recent years, largely through the development of resources intended to guide schools’ improvement.

In 2015, ERO released the publication Effective school evaluation: How to do and use internal evaluation for improvement, alongside a companion good practice report. These resources describe what effective internal evaluation is, what it involves, and how to conduct it in ways that will enhance educational outcomes for students.

Schools and their communities need to be continuously evaluating the impact of their endeavours on learner outcomes. To do this, they need strong leadership and evaluation expertise. Their systems, processes and resources should support purposeful data gathering, collaborative inquiry and decision making, and align closely with the school’s vision, values, strategic direction, goals, and equity and excellence priorities.

In 2014, ERO provided guidance on student acceleration in the publication Accelerating student achievement: A resource for schools. This report details how schools can:

  • create a plan to support students to improve their results
  • carry out additional assessments with students who need to accelerate their progress, to better understand their strengths and needs
  • build “educationally powerful connections” with students, their parents and whānau.

Effective internal evaluation and accelerating student achievement are also linked. Internal evaluation is needed to identify and support students to accelerate their achievement by monitoring student progression, as well as to identify teaching strategies that have successfully accelerated student achievement.

Additional insights on the school system

As part of a holistic systems evaluation, the school evaluation indicators can guide and inform complimentary evaluations and research to generate meaningful change and improvement.

The above findings indicate that knowing ‘what to do’ may not be enough for improving internal evaluation or student acceleration. Rather, other factors may limit schools’ ability to undertake these processes effectively. For example, schools may not have the necessary capacity or capability required to improve these processes.

As changes to the education system are made, analyses of ERO’s school evaluation indicators judgments can also inform evaluations of new initiatives or approaches. A large increase in the number of schools judged to be doing well on a given indicator lends support to the effectiveness of an initiative, while no change or a decrease could reflect the introduction of ineffective initiatives.

Conclusion

ERO’s school evaluation indicators contain a wealth of information about education system performance in New Zealand schools. Internal evaluation and accelerating student achievement were both identified as areas in need of improvement, across the schools examined in this article. School leaders can utilise ERO’s existing publications to guide their improvement in these areas.

Looking forward, this previously untapped source of system-level information can also be used by ERO to inform prioritisation, action, and improvement of the education system. Over time these indicators can be used to monitor changes in system performance and to evaluate new initiatives.

1 ERO is currently changing its operating model for school reviews (see Working with the sector to refine the new model for school reviews), however the indicator framework described here remains a common core of any evaluation approach.