Measuring performance

Measuring performance

Quality of education reports and services

ERO posts its reports on system-wide performance and issues on its website. It also provides copies of selected reports directly to schools and services. For reports of particular interest, ERO issues media releases and conducts media interviews. It also shares and discusses report findings with education sector groups at regular liaison meetings, conferences and workshops.

ERO discusses draft national evaluation findings on system-level issues with the Ministry and other agencies. It discusses its recommendations to ensure they are well understood and contribute to improvements.

ERO uses surveys to measure the effectiveness and utility of national evaluations.

How effective were ERO’s national education evaluations?

ERO’s national evaluations are intended to promote debate and influence change in the education system. The national evaluation programme is designed to provide topical, timely, and practical recommendations. The national evaluations of good practice are used by the following groups:

  • the Ministry and other government agencies to establish priorities, target resources and plan interventions
  • schools and early learning services to inform and improve their own practice
  • parents and whānau as a source of information, knowledge and understanding
  • initial teacher education providers and in professional learning and development programmes.

Table 1: ERO’s national evaluation programme performance 2016/17

 Actual
2015/16
Education Review Office Performance Measures
Quality of Education Reports and Services
 Actual
2016/17
Budget 2016/17 
  System-wide evaluations and related services    
14 Number of national education evaluations 15 up to 20
100% Education evaluations are consistent with approved plans and procedures 100% 100%
New measure Key audiences report that ERO's evaluations are informative and useful for identifying or planning improvements within the system or its component parts 90% 80%-100%

ERO’s national evaluation programme currently delivers up to 20 reports each financial year. In 2016/17, 15 reports published were as follows:

To evolve ERO’s evaluation methodologies, approaches and programme

  • Partners in Learning (August 2016)
  • An Evaluation of Stand Children's Services: Children's Villages (September 2016)
  • Early Learning Curriculum (October 2016)
  • Child Youth and Family Residential Schools (November 2016)
  • Extending Their Language - Expanding Their World: Children’s Oral Language (Birth-8 years) (February 2017)
  • School Trustees Booklet: Helping You Ask the Right Questions (February 2017)
  • Food, Nutrition and Physical Activity in NZ Schools and Early Learning Services (April 2017)
  • He Pou Tātaki: How ERO Reviews Hospital-Based Education and Care Services (June 2017)

To support collaborative effort across the system

  • Communities of Learning | Kāhui Ako: Collaboration to Improve Learner Outcomes (January 2017)
  • Communities of Learning | Kāhui Ako in Action (January 2017)
  • Communities of Learning | Kāhui Ako: Working Towards Collaborative Practice (January 2017)

To embed the evaluation indicators

  • School Evaluation Indicators (July 2016)
  • School Leadership That Works (November 2016)
  • Effective Internal Evaluation for Improvement (December 2016)

To build internal evaluation practice across the system

  • Appraisal as a Catalyst for Improved Learner Outcomes (December 2016)

All 15 reports were produced in accordance with approved plans and procedures.

Every year, ERO surveys one or more audiences to help assess how useful its national education evaluation reports have been for identifying or planning improvements. For 2016/17, staff and managers of early childhood education services were asked about five different ERO reports. A 90 percent positive response was received from those who had read one or more of these reports.

National education evaluations provide information on significant education issues and education sector performance. The information for these evaluations is gathered from the institutional evaluations carried out by ERO or by specialist ERO teams for particular topics.

 

Accountability reviews - early childhood education services, schools and other education service providers

ERO provides assurance on the quality of education provided by early childhood education services, schools and other education service providers.

An ERO evaluation looks at how an early learning service or school reaches positive learning outcomes as expressed in knowledge, skills, attitude and habits. ERO is interested in what is working well and where improvements can be made.

ERO evaluations provide information for individual early learning services, schools, communities and the Government. The funding appropriation for this includes the following Departmental Output Expense categories.

Early childhood education services

This category is limited to evaluations of the performance of early childhood education services.

Schools and other education service providers

This category is limited to evaluations of the performance of schools and other education service providers.

During 2016/17, ERO’s teams evaluated the performance of 1,217 early learning services / Kōhanga reo and 700 schools /kura. For each evaluation, ERO’s aim was to engage in a meaningful review process and make sound judgements about performance. ERO used a range of evaluation techniques to collect feedback and effectiveness data about the capability of early learning services and schools.

In its reports, ERO aims to highlight performance issues and to communicate them accurately, clearly and concisely for all stakeholders. In the vast majority of cases, ERO was able to report a positive evaluation finding or review outcome.

Table 2: Evaluations of education service providers 2016/17

 Actual
2015/16
Education Review Office Performance Measures
Accountability Reviews
 Actual
2016/17
Budget 2016/17 
  Evaluations of education service providers    
 New measure  Total number of evaluations of education service providers (includes evaluations of Early Child Education Services, Schools and other providers)            1,950 >1,850
 New measure  Percentage of education service providers that indicate ERO's evaluations are making a contribution to their decisions about how to improve learner outcomes 85% 80%-100%
                73 Public satisfaction with ERO's services (Kiwis Count score)                 74                 70

* The State Service Commission changed its survey reporting period from a financial year to a calendar year in 2016/17. The reported number for 2015/16 has been restated for the survey period change.

ERO’s review teams completed 1,950 accountability reviews in 2016/17. The total number of evaluations includes those for early childhood education services, state schools, private schools and home schools. Evaluations are accounted for in the year in which the ERO report is published.

ERO’s external evaluation process works with an early learning service or school’s own internal evaluation processes. It also supports a culture of ongoing improvement.

A final report is published on the ERO website once the external evaluation process is complete and the early childhood service or school has had a chance to take in and discuss the results. This enables parents, whānau, families and the wider community to know about the teaching and learning at each individual early childhood service and school.

During the course of the year, school principals, trustee board chairs and early childhood education (ECE) managers have used our post-evaluation survey questionnaires to provide valuable feedback about where the ERO process works well and where it can be improved. ERO uses this feedback for its own learning and the professional development of its teams.

In the 2016/17 year, 85 percent of respondents to our surveys indicated that ERO’s evaluations had made, or were likely to make, a contribution to their decisions about how to improve learner outcomes. The responses received often included comments which show a strong appreciation of ERO’s work and an acknowledgement of its positive impact. For schools, the positive response was 86 percent. For early learning services, it was 83 percent. Adjusting for the respective sample sizes, the overall positive response rate was 85 percent.

The public satisfaction measure is from the Kiwis Count survey carried out by the State Services Commission (SSC). It assesses the level of satisfaction with ERO’s school and ECE evaluation reports. ERO’s results for the 2015 and 2016 calendar years were 73 and 74 (out of 100) respectively. This is a sound result but one which it will need to work hard to maintain.

Evaluations of early childhood education services

ERO used the following performance measures for the category Accountability Reviews – Early Childhood Education Services for the year ended 30 June 2017. This category is limited to evaluations of the performance of early childhood education services.

Table 3: Evaluations of early childhood education services

 Actual
2015/16
Education Review Office Performance Measures
Accountability Reviews
 Actual
2016/17
Budget 2016/17 
  Early Childhood Education Services    
           1,259 Number of early childhood education services evaluations            1,217 1,200-1,460
98% ERO uses a moderation panel to assess levels of compliance with
approved standard procedures for a sample of education review reports
of early childhood education services
96% 90%-100%
85%, 96%
and 100%
respectively
Percentages of draft (near final) findings sent to early childhood
education services for confirmation of accuracy and comment will meet target for reporting to the Minister within 20, 25 and 35 working days of the end of the last week on site
74%, 90%
and 98%
respectively
80%, 90%
and 98%
respectively
 New measure  Percentage of early childhood services evaluated that indicate that ERO's
evaluations are making a contribution to their decisions about how
to improve learner outcomes
83% 80%

During 2016/17, ERO’s teams evaluated the performance of 1,217 early childhood education services and Kōhanga reo. ERO’s review teams make judgements by reference to established criteria which are detailed in its guidance, methodologies and effective practice indicators. In recent years, the finding of an evaluation has determined the length of time before ERO returns for its next evaluation.

ERO uses a national moderation panel to determine the levels of compliance against internal standard procedures. Moderation findings are followed up in quality assurance guidelines and processes. At 96 percent compliance, ERO’s moderation result was within the target range for 2016/17.

ERO also monitors the timeliness and consistency of its reporting to early learning services / Kōhanga reo and schools/kura. Unconfirmed (near-final) reports are sent to service providers for comment on any issues of fact or matters relating to findings. The timeliness criteria require 80 percent of near-final reports to be sent within 20 working days of the end of the last week on site, 90 percent of near-final reports within 25 working days, and 98 percent of near-final reports within 35 working days.

In 2016/17, ERO did not meet all its timeliness criteria for early learning services reports. In some cases staff non-availability, largely because of ill health, affected ERO’s ability to meet the criteria. In other cases, time was a lesser consideration relative to other aspects of quality. ERO’s work was also impacted by the November 2016 earthquake, which required us to permanently vacate one of our damaged premises.

As already reported, 83 percent of early learning services staff who completed our post evaluation survey indicated that ERO’s external evaluation had made, or were likely to make, a contribution to their decisions about how to improve learner outcomes.

ERO’s early childhood methodology described in the publication, He Pou Tātaki – How ERO reviews early learning services, focuses attention on the quality and effectiveness of services’ internal evaluation and on the capacity of the service to promote positive learning outcomes for all children.

Since 2013, ERO has used the following differentiated judgements and timing:

  • Very well placed - the next ERO review will be in four years
  • Well placed - the next ERO review will be in three years
  • Requires further development - the next ERO review will be within two years
  • Not well placed - the next ERO review in consultation with the Ministry.

ERO monitors changes in the status of individual services. In 2016/17, 89 percent of ECEs that had ‘required further development’ in their previous review had improved sufficiently to move to “well-placed”. This means a relatively high number of these services will now have their next review after three years.

At the end of 2016/17, there were about 4,600 early childhood education services in New Zealand’s early learning sector.

Evaluations of schools and other education service providers

ERO used the following performance measures for the category Accountability Reviews – Schools and Other Education Service Providers for the year ended 30 June 2017. This category is limited to evaluations of the performance of schools and other education service providers.

ERO adopts an approach to external evaluation of schools which is designed to build each school’s internal evaluation capability and encourage them to establish a cycle of ongoing improvement.

Evaluation reports for state schools and kura inform the Crown, the governing bodies of schools, their staff and parents about the quality of education and management practices.

Table 4: Evaluations of schools and other education service providers

 Actual
2015/16
Education Review Office Performance Measures
Accountability Reviews
 Actual
2016/17
Budget 2016/17 
  School and other education service providers    
678 Total number of state schools education reviews 700 650-840
4 Number of homeschooling education reviews  9 Up to 35
20 Number of private school education reviews 24 Up to 25
93% ERO uses a moderation panel to assess levels of compliance with
approved standard procedures for a sample of education review
reports of schools and other education service providers
99% 90%-100%
75%, 88%
and 96%
respectively
Percentages of draft (near final) findings sent to schools and
other education service providers for confirmation of accuracy and
comment will meet target for reporting to the Minister within 20, 25
and 35 working days of the end of the last week on site
56%, 77%
and 92%
respectively
80%, 90%
and 98%
respectively
 New measure  Percentage of schools that indicate that ERO's evaluations are making
a contribution to their decisions about how to improve learner outcomes
86% 80%-100%
84 Number of Communities of Learning reports 73 Up to 100
100% Communities of Learning reports are consistent with approved
presentational standards and agreed terms of reference
100% 100%

During 2016/17, ERO published 700 evaluation reports for state schools and kura and completed nine homeschooling reviews and 24 private school reviews. These outputs were within target ranges.

ERO’s reviews of privately-owned schools are different from those undertaken for state and state-integrated schools, and for kura. For private schools, ERO’s primary focus is on how well the school meets the criteria for registration as a private school as required under section 35I and Part 28 of the Education Act 1989.

Homeschooling reviews are reviews of programmes for students granted exemption from attendance at school, and are carried out at the request of the Ministry. ERO met these demands and supplied the requested reviews to the Ministry.

For evaluations of state schools, unconfirmed (near-final) reports are sent to school boards for comment on any issues of fact or matters relating to findings. The timeliness criteria require 80 percent of near-final reports to be sent within 20 working days of the end of the last week on site, 90 percent of near-final reports within 25 working days, and 98 percent of near-final reports within 35 working days.

In 2016/17, ERO did not meet the timeliness criteria for schools for all three categories. This is due to the introduction of the new approach to accelerating student achievement in primary schools (2016) and intermediate schools and area schools (from January 2017). ERO has carefully embedded its new approach and this has meant additional time for moderation. Also ERO’s work was also impacted by the November 2016 earthquake.

ERO has continued to undertake its own surveys to determine, among other things, the extent to which schools have used the external evaluation process to make improvements.

ERO developed a new post-evaluation questionnaire for use from July 2016 which includes feedback from school leaders about the effectiveness of our external evaluations. As already reported, 86 percent of responses from school principals, staff and boards indicate that ERO’s external evaluation in 2016/17 had made, or was likely to make, a positive contribution to their decisions about how to improve learner outcomes.

In 2016/17, ERO completed 73 reports for CoL | Kāhui Ako to support the Government’s IES initiative. These include tailored reports for each community and bespoke reports for schools receiving the Principal Recruitment Allowance.

ERO evaluates how each school’s programmes, processes and teaching promote positive outcomes for learners, and how effectively the school focuses on improvement and accelerated student achievement. The differentiated review categories for schools are as follows:

  • the one to two year return category describes those schools where ERO’s longitudinal review processes are supporting them to develop their self-review capacity so that they can develop strategies to focus on and improve student achievement
  • the three year return category describes those schools that have established effective processes for student engagement, progress and achievement
  • the four to five year return category describes those schools who can consistently demonstrate sustained student engagement, progress and achievement.

In recent years, the finding of an evaluation has determined the length of time before ERO returns for its next evaluation. A relatively positive finding of sustainable improvement has generally resulted in a longer period (four or five years) before the next ERO evaluation. If material issues reducing school performance exist and these are having a negative impact on learning outcomes a longitudinal review is scheduled over the next one to two years.

In 2016/17, the overall picture for the evaluations of schools was that more schools improved following a one to two year return process than has previously been the case. However, fewer schools improved to a four to five year return (from a three year return process) than has been the norm over recent years.

In 2016/17, the New Zealand Council for Educational Research (NZCER) completed its survey of primary school principals. The survey, expected to be published later this year, revealed the following:

  • most principals were positive about their interactions with ERO and ERO reviews
  • many principals had used their last ERO review report in their strategic planning, and made changes in their school as a result of ERO’s national reports
  • principals of schools currently on a four to five year return were more likely to strongly agree that their last ERO review had focused on goals or progress towards those goals
  • higher proportions of principals whose schools were on the four to five year review cycle strongly agreed that ERO reports were reliable in their view of teaching/learning quality and of school improvement capacity.