A. Whānau self review informing external evaluation

An example

Te Tino Uaratanga provides a centre of focus for the education for students in Te Aho Matua kura. This is also the main focus for an ERO review.

In the example below, the kura whānau has chosen aspects of Te Tino Uaratanga as its focus for self review: celebrating success, student success and professional development for staff.

This diagram shows the self review and evaluaiton focus. On the left are three boxes for evaluation focus they are from top to bottom stategic focus, regular self review and emergent self review.  The right side is evaluation plan and there are six boxes feeding up into the plan they are from the bottom upward, engaged, explore and investigate, strong relationships, communication, responsibility and students are high achievers who exemplify the hopes and aspirations of their people.

Figure 3: Self review and external evaluation focusing on Te Tino Uaratanga - an example

The kura whānau will then develop an evaluation plan. This will be based on an evaluative question to guide the external evaluation process. The evaluative question is informed by self review. When constructing an evaluative question whānau could begin with the following:

  • To what extent….. (could be used when the whānau wishes to analyse progress with an improvement focus)
  • How well do students….. (could be used when things are going well and the whānau wishes to find out how well)
  • How effectively do students ….. (could be used when things are effective and the whānau wishes to discover how effective)

The kura documents its own investigative pathway with its whānau so that they are well informed and can all identify what happens, when, by who, how and the impact on students. Some whānau may select one or more of the wāhanga while others may choose all wāhanga with only certain parts for focus.

Timing of next review

Depending on its findings, ERO makes a decision at the end of each review on the type and timing of the next ERO review. This could be a development evaluation carried out over the next one‑to‑two years (Te Pupuketanga). It could be an expansion evaluation after three years (Te Rākeitanga). Or, in the case of an exceptionally high performing kura, ERO could decide to return in four‑to‑five years for an enrichment evaluation (Te Manakotanga).

Kura that are performing well will continue to be reviewed under the Te Rākeitanga methodology.

Factors in the timing decisions are the amount of support needed, the improvements that can be made and the capability of each kura for self review and future action.

When making a decision on the next review, ERO considers the whānau evaluation focus, the information provided in the Whānau Assurance Statement (WHAS), and ERO’s Te Aho Matua criteria for review timing.

ERO’s formal decision on the timing of the next review is set out in the confirmed review report, which is printed and published on ERO’s website, www.ero.govt.nz.

The nature of development from ERO’s Te Pupuketanga (over the course of one-to-two years) review, to Te Rākeitanga (three year) review, through to Te Manakotanga (four-to-five year) review is designed to be progressive and aspirational.

The criteria for the timing of reviews are outlined in this Framework under each type of review.

Kura whānau evaluation plan

This is the kura whanau evaluation plan looking at how well do students achieve and exemplify the hopes and aspirations of their people and what are the aspirations of the whanau.

Figure 4: Evaluation plan *See Appendix 1 for a bigger version to print off and use

The kura whānau gives its evaluation plan to ERO as a part of the first self review presentation hui at the onsite stage of the review. In some instances whānau can provide a draft for ERO to consider before ERO comes to the kura. This plan provides relevant detail for ERO so that review officers can develop their onsite evaluation design, to guide investigations at the kura.

ERO’s evaluation design

As the external evaluator, ERO will develop the evaluation design.

The evaluation design will include key areas of focus, investigative pathways and questions to prompt the review process and cover the information provided by whānau. In conjunction with whānau, review officers will also create a timetable that includes additional hui, classroom observations and time for team analysis and synthesis. This also allows for ongoing consideration to the review design, ensuring flexibility where necessary.

This is the kura whanau evaluation plan looking at how well do students achieve and exemplify the hopes and aspirations of their people and what are the aspirations of the whanau.

Figure 5: ERO Evaluation design

Reviewers discuss the design with whānau to ensure that the process of evaluation is transparent and reflects the intentions of whānau self review. Where possible ERO will provide additional guidance with the evaluative question to ensure the review is purposeful.

The kura whānau makes evidence available to ERO during the review and there will be additional opportunities for hui and discussions as required.

ERO uses the evaluation design to focus its investigations. This allows time to be specifically allocated to the ‘thinking’ role of an evaluator. The evaluation framework focuses on high quality analysis to inform clear and concise evaluative judgments.

Throughout Te Rākeitanga there are three distinct review process phases - Whanaungatanga, Whakaaetanga and Whakataunga (see Figure 1). These are the formal stages of the process and involve interactions, communication, investigation, collection and collation, analysis, synthesis, decision‑making, recommendations and reporting.

Evaluation Indicators

A balanced approach to evaluation takes into account evidence of both outcomes for students and the processes and philosophies implemented in the kura to achieve these outcomes. ERO looks for this balance of evidence across both outcomes and processes.

Outcome indicators describe the knowledge, skills, attitudes and higher-level competence directly associated with what Te Aho Matua kura kaupapa Māori are trying to achieve for their students. They include evidence of the academic, physical, emotional and spiritual outcomes that are the direct indicators of success.

Process indicators describe the practices, processes and policies of the kura that, when implemented, are intended to achieve the desired results. The presence of process indicators does not, however, necessarily guarantee positive outcomes for students.

Evidence of both types of indicators needs to be considered if the reviewers are to reach genuinely well-informed conclusions. Considering outcomes alone can distort the evaluation by emphasising results without considering whether the results are what could reasonably be expected. A sole focus on process and philosophy can give an impression of success without actual evidence that students are progressing and achieving as expected and desired.

The Evaluation Focus and Student Outcomes

Consider how the potential evaluation focus contributes to student outcomes. What is the magnitude of the effect in terms of:

  • the numbers of students affected (low, medium or high)
  • the likely size of the impact (low, medium or high).

In general, priority areas for review will be those where there is likely to be a high impact on the achievement of some students, or where high numbers of students are affected.This diagram is the matrix to determine evaluation focus on the x axis it is called likely effect on student outcomes and the three boxes are labelled low, medium and high the y axis is called numbers of students affected and is also labelled low medium and high.

Figure 6: Matrix to determine evaluation focus

Analytical Framework

The framework diagram is a series of boxes that feed into the top set of boxes. the top portion is four boxes which sit around synthesis they are evaluative discussions, ERO confirmed report, student outcomes and inputs. Off to the right side of this sits team view.  The boxes underneath which feed into this are from the bottom up level 1 recorded evidence, natural analysis evidence documentation, observation and discussions - proceed or refocus. Level 2 collated analysis against areas of focus - Analysis focus evaluation design and evidential summation -early notification stage. Level 3 collated anaylsis aligned to the evaluation design, Analysis alignment - check coverage and Level 4 final individual analysis - Analysis overview individual overview - support direction making.

Figure 7: Analytical framework for the evaluation

The analytical framework used by ERO is included to help whānau expand their self‑review practice.

The evaluation design contributes to the content of ERO’s investigations and the content section of its report.

Guide to Reaching Informed Conclusions

At each stage, ERO considers the answers to the evaluative questions. What is the evidence of effective process? What is the evidence of positive outcomes?

Considering each of the evaluative questions being addressed, how do they rate?

Is this practice excellent/very good, good, adequate, fair, or poor?

Are these outcomes excellent/very good, good, adequate, or poor?

The following key  provides a guide to these terms.

This diagram is titled Terms of Evaluation judgements. It has three columns which are Descriptor, meaning in process evaluation and meaning in outcome evaluation. The range is then from excellent/very good, good, adequate, fair and poor and describes the minimums acceptable.

Figure 8: Terms of evaluation judgements

The key decision for teams to reach is a judgement about whether or not the quality of the programme, and the outcomes for students, are acceptable as a minimum.