What did ERO do?

This evaluation included all 40 RTLB clusters. The data gathering took place in two phases. The first phase was a trial of the approach in two RTLB clusters in early May 2017. The trial provided an opportunity to test the evaluation framework, approach to scoping, data gathering, exit discussion and analysis of data using a synthesis rubric.[1] The second phase took place from mid‑June to late‑September 2017 and involved the project team evaluating the remaining 38 RTLB clusters.

In this evaluation, ERO asked the following questions: [2]

  • To what extent has the quality and consistency of RTLB cluster governance and management improved to address the issues identified in ERO’s 2009 evaluation?
  • To what extent has the transformation of the RTLB service contributed to increased capability and capacity within clusters, to monitor and evaluate practice and service provision to identify what is working well and what needs to improve?
  • How are RTLB clusters involved with Kāhui Ako, and how is the relationship developing? What is working well and what are the challenges?
  • What contribution is the RTLB service making to the wider provision of learning support?
  • What evidence is there of the impact of the RTLB service on improving learner outcomes?

The Ministry of Education and an External Reference Group made up of four RTLB cluster managers, four lead school principals and a representative from the RTLB Association were involved in developing these questions.

ERO’s Approach to data gathering

ERO considered information from a variety of sources. Meetings were held with key stakeholders including representatives from cluster schools and kura (including principals/tumuaki) and Special Education Needs Coordinators (SENCO) or their equivalent, and cluster managers, lead school board of trustees, lead school principals, practice leaders, groups of RTLB. [3] At the end of each onsite phase ERO evaluators met with the cluster manager, lead school principal and a lead school board member to discuss the tentative findings for their cluster. This ‘exit meeting’ focused on:

  • ERO’s tentative judgements for the cluster in relation to the synthesis rubric
  • clarifying and testing ERO’s findings, sharing strengths and areas for improvement in the cluster
  • checking the nature and accuracy of information gathered about the cluster’s involvement with Communities of Learning|Kāhui Ako.

ERO also analysed cluster documentation, including internal evaluation information, monitoring and reporting and operational policies and procedures.

A rubric was used as a tool to synthesise the findings in each cluster review. The rubric was based on the one used in the 2009 ERO evaluation, with refinements made to reflect the current RTLB cluster model and broad expectations in the Funding and Service Agreement. The synthesis rubric was the basis of reviewers’ judgments about the performance of each cluster. These judgements were moderated as part of ERO’s quality assurance process. Moderation of the judgments took place mid way through the reviews, and at the end when ERO had visited all 40 clusters. The evidence gathered in each cluster review and the criteria in the rubric were used for the moderation process. This national evaluation report pulls together the overall findings.

[1] See Appendix 5 for the RTLB Synthesis Rubric.

[2] See Appendix 4 for the Evaluation Framework and Investigative Prompts.

[3] See Appendix 3 for information about meetings with key stakeholders.