Measuring our performance

Use and influence of ERO's national evaluations

ERO publishes its National Evaluation reports on its website and provides copies of selected reports to schools and services. For national reports of particular interest we issue news releases and conduct media interviews.

We also share and discuss report findings with education sector groups at regular liaison meetings, as well as at conferences and workshops.

ERO discusses draft national evaluation findings with MoE and other agencies. We discuss each recommendation to ensure that they are well understood and can lead to a work programme intended to bring about necessary improvements.

We use surveys to measure the effectiveness and utility of our national evaluations.

How effective were our national evaluations?

ERO's national evaluations are intended to promote debate and influence change in the education system.

The national evaluation programme is designed to provide topical, timely, and practical recommendations. The national evaluations of good practice are used:

  • by MoE and other government agencies to establish priorities, target resources and plan interventions
  • by schools and early learning services to inform and improve their own practice
  • by parents and whānau as a source of information, knowledge and understanding
  • by initial teacher education providers and in professional learning and development programmes.

Table 1: ERO's National Evaluation Programme Performance 2015-2016

Actual

 

2014/15

ERO National Evaluation Services                                        Note

Actual

 

2015/16

 

Budget Standard

2015/16

12

Number of education evaluation reports

14

Up to 20

100%

Education evaluation reports are consistent with approved presentational standards and agreed terms of reference

100%

100%

87%

Sample studies of key government audiences confirm                1  evaluation reports are used to inform policy and/or to 
establish priorities for the sector

100%

80%-100%

90%

Sample studies of schools and early childhood services that report use of education evaluation reports to inform and improve their own practice

89%

80%-100%

68

Public satisfaction with ERO’s reports (Kiwis Count score out of 100)

75

70

Note 1: In 2016, at ERO's request, MoE undertook a review of the actions that had been taken in response to ERO's national evaluations. In each case ERO's recommendations and findings were acknowledged as being useful in establishment of priorities for the sector and influencing policy decisions by the Ministry.

Early childhood service and school evaluations

During 2015/16, ERO's teams evaluated the performance of 1,259 early learning services /kōhanga reo and 702 schools /kura. For each evaluation our aim was to engage in a meaningful review process and make sound judgements about performance. We use a range of evaluation techniques to collect feedback and effectiveness data about the capability of early learning services and schools.

In our reports we aim to highlight performance issues and to communicate them accurately, clearly and concisely for all stakeholders. In the vast majority of cases we were able to report a positive evaluation finding or review outcome.

It is recognised that participation in high quality early childhood education is fundamental to achievement in later years. However, ERO's work in the early learning sector has highlighted considerable variability in the quality of curriculum implementation, assessment and evaluation.

The availability of ever-improving data from MoE provides the opportunity to more clearly identify where the greatest need for intervention lies. This information enables ERO to be more flexible and efficient in its approach.

ERO's review teams make judgements by reference to established criteria which are detailed in our guidance, methodologies and effective practice indicators. In recent years the finding of an evaluation has determined the length of time before ERO returns for its next evaluation. A relatively positive finding of sustainable improvement has generally resulted in a longer period (4 or 5 years) before the next ERO evaluation. If material issues reducing school performance exist and these are having a negative impact on learning outcomes a longitudinal review is scheduled over the next 1-2 years. ERO uses a national moderation panel to determine the levels of compliance against internal standard procedures. Moderation findings are followed up in quality assurance guidelines and processes. Quality assurance processes were updated in 2014/15 and show the emphasis that is being placed on learners at risk of inequitable educational outcomes. Information on engagement with families and whānau is also carefully considered as part of the review process.

We have also monitored the timeliness and consistency of our reporting to early learning services/kōhanga reo and schools/kura.

School principals, board chairs and ECE managers have used our post-survey questionnaires to provide valuable feedback about where the ERO review process works well and where it can be improved. We use this feedback for professional development of our review teams. The responses received often included comments which show a strong appreciation of ERO's work and an acknowledgement of its positive impact.

ERO has participated in the State Services Commission's Kiwis Count survey to assess the level of public satisfaction with its reports. We are provided with benchmarked satisfaction ratings and in 2015/16, the annual result score for ERO was 75. The increase of 7 points over the 2014/15 year was statistically significant and, we like to think, a testament to the improving quality of our reports.

Results are summarised in Tables 2 and 3, and in each case ERO has completed sufficient evaluations to meet its performance targets for 2015/16.

students mingling together with teacher

Evaluations of early childhood education services

ERO's early childhood methodology, He Pou Tātaki-How ERO reviews early learning services, focuses attention on the quality and effectiveness of services' internal evaluation and on the capacity of the service to promote positive learning outcomes for all children.

Since 2013, ERO has used the following differentiated judgements and timing:

  • Very well placed - the next ERO review will be in four years
  • Well placed - the next ERO review will be in three years
  • Requires further development - the next ERO review will be within two years
  • Not well placed - the next ERO review in consultation with the MoE.

Where services are deemed to require further development, ERO works with the Ministry of Education and the service to develop action plans for improving practice. These services and the Ministry provide ERO with progress reports after six to nine months. At this time, ERO determines the level of risk to children's learning and wellbeing and sets a date for the next evaluation.

The table below summarises the measures used by ERO to monitor its performance in respect of its external evaluations of Early Childhood Education Services.

Table 2: ERO Early Childhood education service reviews 2015-2016

 

Actual

 

2014/15

Review of Early Childhood Education Services                                  

Actual

 

2015/16

 

Budget Standard

2015/16

1,350

Number of early childhood services reviews

1,259

1,200-1,460

95%

ERO uses a moderation panel to assess levels of compliance with approved standard procedures for a sample of education review reports of early childhood education services

98%

90%-100%

85%, 95%

and 99% respectively

Percentages of unconfirmed (near final) reports sent to early childhood education services for confirmation of accuracy and comment will meet target for reporting to the Minister within 20, 25 and 35 working days of the end of the last week on site

85%, 96% and 100% respectively

80%, 90% and 98% respectively

92%

Proportion of early childhood services evaluated that will have used ERO's review and evaluation process to make improvements (Survey)

93%

80%

Establish baseline

% of early childhood services evaluated previously within the 2 year review cycle moving to the 3 year review cycle.

96%

Establish baseline

At the end of 2015/16 there were about 4,500 fully licensed early childhood education services in New Zealand's early learning sector (Source: Ministry of Education).

Evaluations of Schools/Kura

ERO adopts an approach to external evaluation of schools which is designed to build schools' internal evaluation capability and encourage them to establish a cycle of ongoing improvement.

ERO evaluates how each school's programmes, processes and teaching promote positive outcomes for learners, and how effectively the school focuses on improvement and accelerated student achievement. The differentiated review categories for schools are:

  • The 1-2 year return category describes those schools where ERO's longitudinal review processes are supporting them to develop their self-review capacity so that they can develop strategies to focus on and improve student achievement
  • The 3 year return category describes those schools that have established effective processes for student engagement, progress and achievement
  • The 4-5 year return category describes those schools who can consistently demonstrate sustained student engagement, progress and achievement.

The following table sets out the measures used by ERO to monitor its performance in respect of its external evaluations of state schools, private schools and home schools.

Table 3: ERO School reviews 2015-2016

Actual

 

2014/15

Review of Schools and other providers                                 Note                                  

Actual

 

2015/16

 

Budget Standard

2015/16

103

Reviews of Primary and Secondary Schools: 1-2 year review

76

 

552

Reviews of Primary and Secondary Schools: 3 year return

492

 

131

Reviews of Primary and Secondary Schools: 4-5 year return

90

 

9

Reviews of Primary and Secondary Schools: Other

20

 

795

Total number of state schools education reviews

678

650-840

4

Number of homeschooling education reviews                              2

4

Up to 35

21

Number of private school education reviews

20

Up to 25

100%

ERO uses a moderation panel to assess levels of compliance with approved standard procedures for a sample of education review reports of schools and other education service providers

93%

90%-100%

85%, 96% and 99% respectively

Percentages of unconfirmed (near final) reports sent to schools and other education service providers for confirmation of accuracy and comment will meet target for reporting to the Minister within 20, 25 and 35 working days of the end of the last week on site

75%, 88% and 96% respectively

80%, 90% and 98% respectively

93%

Proportion of schools evaluated that will have used its review and evaluation process to make improvements

91%

80%

67%

% of schools evaluated previously on the 1-2 year review cycle moving to the 3 year review cycle

62%

60%-65%

15%

% of schools evaluated previously on the 3 year review cycle moving to the 4-5 year review cycle

14%

12%-15%

11

Number of Communities of Learning reports

84

Up to 100

New measure

Communities of Learning reports are consistent with approved presentational standards and agreed terms of reference

100%

100%

Note 2: Homeschooling reviews are carried out at the request of the Ministry of Education. All reviews requested in 2015/16 were completed.

In June 2016, there were about 2,430 state and state-integrated schools in New Zealand - over 1,960 primary or intermediate schools, and about 470 secondary or composite schools.

In 2015/16 the New Zealand Council for Educational Research (NZCER) released the results of its survey of secondary school principals. They found that:

  • Most principals were positive about their interactions with ERO and ERO reviews
    • Many principals had used their last ERO review report in their strategic planning, and made changes in their school as a result of ERO's national reports
    • Principals of schools currently on a 4-5-year return were more likely to strongly agree that their last ERO review had focused on goals or progress towards those goals
    • Higher proportions of principals whose schools were on the 4-5-year review cycle strongly agreed that ERO reports were reliable in their view of teaching/learning quality and of school improvement capacity.

ERO has continued to undertake its own surveys to determine, among other things, the extent to which schools have used the external evaluation process to make improvements.

In 2015/16, we have found that over 90% of the principals and Boards responding believed that:

  • the evaluation process was either “very useful" or “quite useful"
  • the electronic notification for the review contained useful information
  • the pre-review discussions were useful and the explanation of how the review process would work reflected the actual experience
  • they were able to participate meaningfully in the review process
  • the school's context and self-review information was taken into account
  • the discussion of findings with the ERO review team was helpful.

We have developed a new post-evaluation questionnaire for use from July 2016 which will include additional feedback from school leaders and stewards about what works well in our external evaluations.

Movements between review categories

In the four years from 1 July 2012 to 30 June 2016, ERO has conducted over 3,000 primary and secondary school evaluations. A breakdown of these reviews by category and movement is provided below.

graph showing review movement

1-2 year reviews

About 67 percent of the reviews under the 1-2 year review category resulted in the schools being evaluated as improving and able to move to a 3 year return. However, about a third of these 1-2 year reviews lead to a continuation of the 1-2 year review process for the schools concerned.

3 year reviews

About 77 percent of the 3 year reviews resulted in no change in status for the schools and a next review was scheduled to take place in three years. However, for about 13 percent of these reviews the school showed sufficient progress to be moved to the 4-5 year return cycle. Conversely, about (10 percent) were evaluated as needing to move down to the 1-2 year return cycle.

4-5 year reviews

In the 4-5 year category, 54 percent (169 out of 314 schools) had been evaluated as meeting the criteria to remain in the 4-5 year return. However, 44 percent (137 schools) did not sustain their performance to a level that warranted maintenance of the 4-5 year return and were moved on to the 3 year return cycle. Unfortunately 8 schools (less than 3 percent) had struggled with material issues and were moved from the 4-5 year cycle down to the 1-2 year cycle.

Summary

Overall the outcomes of the reviews undertaken over the four year period have seen 17 percent of schools progress to a higher category; about 70 percent stay in the same return category; and about 13 percent move backwards because the review has highlighted performance issues. At the end of the 2015/16 year, 179 schools (under 8 percent) were on ERO's 1-2 year return review cycle - 134 primary/intermediate schools and 45 secondary/composite schools. There was an average of about 110 pupils enrolled in the primary/intermediate schools and about 370 in secondary/composite schools compared to the respective national averages of 220 and 680.