Appendix One: Methodology

The Ministry of Pacific Island Affairs (MPIA) is working with 25 secondary schools to raise overall achievement of Pacific learners. MPIA initially targeted 20 schools with a high number of Pacific learners and five more schools asked to be included in this initiative. All but one of these 25 schools are in Auckland. Between them, these 25 schools have, on their rolls, approximately half of all the Pacific secondary-age students in New Zealand.

This evaluation was carried out by analysing the 25 schools’ most recent ERO review reports and associated file notes (dated from 2009 to 2013), charter documents, ERO’s National reports 1 and their file notes, the Ministry’s reporting on PEP progress, New Zealand Qualifications Authority’s (NZQA) data about schools, students’ achievement (for years 2009-2012) and reports written by other providers relating to two initiatives in schools. 2 ERO answered the following evaluative questions:

  • To what extent is the presence of the five key factors related to positive achievement for Pacific learners?
  • What other features are characteristic of those schools that demonstrate success in achieving positive academic outcomes for Pacific learners?

ERO relied on current research regarding good practice in education, as well as good practice in improving achievement outcomes specifically for Pacific learners.

ERO determined, for this context, that a school was successful if its Pacific learners had high levels of achievement, as measured against the targets set by the Ministry, and received a positive ERO report. These are the five schools cited as examples of good practice in this report. The less successful schools had either low levels of achievement and/or an ERO review report which identified significant areas for improvement.

Challenges in the methodology

The information in the schools’ ERO review reports pertinent to this analysis varies in content depending on each school’s context. This variation is influenced by the specific design for each review when investigating ERO’s major evaluative question,3  changes over time in the methodology of reviews, and the particular National Education Topic (NET) being investigated at the time of each school’s review.

ERO used achievement data for NCEA Level 1 and 2 for 2009-2012 to determine how successful a school was. ERO did not have data for achievement levels for students in Years 9 and 10 and so the notion of value added (how much difference a school makes overall in raising its students’ achievement levels) was not able to be taken into account. For example, a school with relatively low achievement for its Pacific learners may still, in fact, be very successful in its work with these students if, on entry to the school, their literacy and numeracy levels were lower than national expectations for their age. Good self-review processes, using longitudinal data, helps schools to identify the extent of value added and this was done well in successful schools.

ERO’s analysis relied on what was included in the ERO review reports relating to Pacific learners. Pacific learners were not always mentioned specifically as a sub-group, therefore the analysis may be under-reporting good practice. To mitigate under‑reporting of practices ERO determined that, where schools had a roll in excess of 55 percent Pacific learners, generalised comments made about targets and achievement included Pacific learners.4 Where there were fewer than 55 percent Pacific learners, ERO accepted that generalised statements did not necessarily include Pacific learners.

Different agencies and government departments talk about and compare Pacific learners to other groups of learners using different terms. This complicates the comparative analysis of achievement and its discussion. In this report, ERO uses the terms as they are used by the agency/department whose data is being referred to. This means that ERO refers to comparisons between Pasifika or Pacific learners and non-Pasifika, Pākeha, European, total students (in a school context) and sometimes to all students nationally.