Findings

The findings reported in this section are structured around the five evaluation questions.

Cluster governance and management

To what extent has the quality and consistency of RTLB cluster governance and management improved to address the issues identified in ERO’s 2009 evaluation?

ERO’s findings show a substantial improvement in the quality and consistency of the governance and management of the RTLB service. The 2012 transformation has successfully addressed most of the issues identified in the 2009 evaluation. ERO attributes this improvement to the following factors:

  • effective leadership of the transformation through changes to the leadership of RTLB clusters, including the appointment of cluster managers and lead schools (boards of trustees and principals), and opportunities for RTLB to step up into leadership roles (practice leaders and RTLB areas of responsibility)
  • a reduction in the number of clusters resulting in clusters with larger ‘pools of RTLB’ expertise to draw on in undertaking the much widened scope of the RTLB role
  • a high level of adherence to the guidance and expectations for how RTLB clusters are governed and managed, and for the professional practice of RTLB.

As shown in Figure 1, ERO found that 36 of the 40 RTLB clusters were well governed and managed. The focus of the transformation of the service on structures, systems and processes has led to these improvements. In most clusters, ERO found well-documented policies and procedures which guided day-to-day operation of the service; improved processes to support cluster schools and kura to access the service; and better people management (RTLB) in terms of appraisal, professional development, induction, mentoring and supervision.

Figure 1: Cluster governance and management

This image is a bar graph. The top statement is 36 of the 40 RTLB clusteres were well governed and managed.  The graph's y axis is four points from the top they are Very good, Sound, Limited and Minimal. The x axis is the Numbers of clusters. Very good 22, Sound 14, Limited 4 and Minimal 0.

The following section highlights ERO’s findings for each of the following dimensions:

  • operating according to requirements
  • internal evaluation (self review), planning and reporting
  • access to service
  • personnel management and professional support
  • professional relationships
  • communication
  • leadership.

Two dimensions of the rubric ERO used (seamless collaboration with the Ministry of Education and outcomes for learners) are not reported on here, as the findings are included in subsequent sections of the report.

Operates according to requirements

ERO found considerable improvement in the extent to which RTLB clusters were operating according to the requirements of the Funding and Service Agreement and guidance documents (see Figure 2 and Table 1). Most clusters were working within the guidelines and expectations set out in Governing and Managing RTLB Clusters and the Professional Practice Toolkit. Annual and quarterly reporting to the Ministry provides an accountability mechanism for clusters. ERO noted some variability in the quality of reporting and in understanding the requirements. Understanding and awareness of specific expectations in the Funding and Service Agreement was limited in a few clusters. Also the interpretation of the Agreement varied in some clusters, particularly in relation to the level of detail required when reporting to the lead school board of trustees and wider community. This is an area for the Ministry to work on with clusters in light of the Funding Agreement (renewed to 31 December 2019) and the responsibilities it sets out.

Figure 2: Operates according to requirements

This image is of two bar graphs. The top statement reads More RTLB clusters than in 2009 now operate according to requirements. The first graph is 2009 the Y axis is four points which are from the top Very good, Sound, Limited and minimal the x axis is number of clusters.  The figures are Very good 6, Sound 12, Limited 16 and Minimal 6.  The second graph is 2017 the Y axis is four points which are from the top Very good, Sound, Limited and minimal the x axis is number of clusters.  The figures are Very good 21, Sound 15, Limited 3 and Minimal 1.

The following table shows the key shifts in relation to RTLB clusters operating according to requirements.

Table 1: Shifts from 2009 to 2017 – operating according to requirements

From (2009) To (2017)

Poor governance and management practices impacting on RTLB service quality, particularly management of RTLB.

Lack of awareness of, and adherence, to the Ministry of Education’s Policy Toolkit (2007).

Limited systems (Ministry of Education) to monitor how well clusters were meeting policy requirements.

Well-implemented, comprehensive policies and processes support cluster governance and management.

Regularly-reviewed policies and procedures leading to improved personnel management.

Most clusters operating within Ministry of Education agreement, guidelines and expectations.

Strengthened accountability through annual and quarterly reporting to Ministry of Education.

Regular National Forums provided opportunities for cluster managers and lead school principals to meet and discuss matters of interest. The Ministry also provided some support for clusters, particularly just before the ERO evaluation.

Over the course of this evaluation ERO noted that some clusters had or were experiencing changes in cluster manager and/or lead school/kura. While such changes were mostly well managed within the cluster, the Ministry does not have processes in place to make sure new cluster managers and lead schools are well supported when such changes occur.

ERO found tight accountabilities and reporting for Learning Support Funding (LSF) and Year 11 to 13 funding for schools and kura. However clusters were not using the reports to evaluate the difference this funding was making for learners. In many clusters the LSF funding was seen as a source of funding for teacher aides. Clusters need to be assured this funding is bringing about the outcomes expected, for whom it is intended.

Next steps for improvement

The Ministry considers developing and implementing a formal induction process for new cluster managers, lead school principals and boards of trustees when personnel change.

Internal evaluation, planning and reporting

ERO found some positive improvements in this aspect of performance, particularly for cluster planning and reporting. Most clusters aligned their strategic and annual planning to reflect cluster needs and national priorities. Although the quality, nature and extent of reporting varied, most clusters were meeting Ministry requirements.

Some clusters were beginning to develop the capability and capacity to evaluate, however, internal evaluation was not sufficiently robust to provide evidence of the impact of the work of RTLB on learner outcomes.

As shown in Figure 3, there is still work to be done to strengthen this aspect of cluster governance and management.

Figure 3: Internal evaluation, planning and reporting

This image is of two bar graphs. The top statement reads Overall clusters are now better at internal evaluation, planning and reporting than in 2009. The first graph is 2009. The Y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2009 this is Very good 3, Sound 4, Limited 17 and Minimal 16. The second graph is 2017. The Y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2017 this is Very good 6, Sound 26, Limited 7 and Minimal 1.

The following table shows the key shifts.

Table 2: Shifts from 2009 to 2017 internal evaluation, planning and reporting

From (2009) To (2017)

Strategic and annual planning not informed by systematic self-review processes.

Issues with the quality and accuracy of data reported.

Most clusters did not have a good understanding of self review, planning and reporting as it pertains to the RTLB service.

Self review was not seen as relevant or useful in many clusters.

In most clusters strategic and annual planning reflected cluster needs and national priorities.

Improved and clearer expectations for reporting. Still some concerns about the quality of data reported.

Increased understanding of monitoring and review, and requirements for planning and reporting.

Clusters are beginning to develop the capability and capacity to engage in robust internal evaluation.

Clusters were reporting quarterly and annually to the Ministry, however this information was not being analysed and used by the Ministry to provide system-level information about the RTLB service in terms of service provision and improved outcomes.

See the section Capability and capacity to monitor and evaluate for further findings.

Access to RTLB service

In most RTLB clusters, school and kura personnel knew how to access the service and found the online request for support system easy to use. As shown in Figure 4, 38 of the 40 clusters had systems and processes to provide very good or sound access to the service. The RTLB liaison role was key for supporting access to the service. The liaison RTLB worked with SENCO to make sure requests for support fitted with the scope of the RTLB role.

Figure 4: Access to RTLB service

This image is of two bar graphs. The top statement reads RTLB systems have improved access to the service since 2009. The first graph is 2009. The Y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2009 this is Very good 6, Sound 16, Limited 14 and Minimal 4. The second graph is 2017 The Y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2017 this is Very good 26, Sound 12, Limited 1 and Minimal 1.

ERO found increasing use of the RTLB service by kura and wharekura in some clusters. This use was very much dependent on the capability of RTLB to work in Māori immersion settings and the ways cluster managers were building relationships with tumuaki and kaiako in kura and wharekura. Cluster managers were acutely aware of the need to build RTLB capability and cluster capacity to work in Māori immersion settings. ERO identified some barriers for Māori-immersion kura and wharekura regarding equitable access to the service. This was largely because the referral system and associated databases have been set up for English-medium schools. These systems are not relevant for Māori immersion kura and wharekura as they are not aligned to curriculum frameworks or assessment tools used in Māori immersion settings. ERO is publishing a companion report that discusses these issues further and provides examples of effective practice.

In many clusters, managers and practice leaders effectively monitored caseloads. Some were using a points or rating system to allocate cases and projects to individual RTLB. ERO found variability across clusters in their analysis of case work and referral data for patterns and trends, and in the use and reporting of this information. Some clusters were also seeking feedback on the quality of RTLB service delivery from school and kura personnel (teachers/kaiako and SENCO) at the time of case closure. This feedback is a rich source of information that could potentially contribute to evaluation of service provision and RTLB practice.

The following table shows the key shifts in access to the RTLB service.

Table 3: Shifts from 2009 to 2017 – Access to service

From (2009) To (2017)

Management of access to the service was highly variable.

Processes for accessing the service were not well known or transparent in all clusters.

Misunderstandings or misconceptions of the RTLB role were having a negative impact on the use of the service by some schools.

Lack of monitoring of RTLB practice.

Access to the service is well managed through cluster systems and processes that were well known and used.

The RTLB liaison role was key to schools and kura accessing the service and building a clear understanding of the scope of the RTLB role.

Cluster managers and practice leaders were highly focused on monitoring RTLB case work and caseloads. This had a positive impact on ongoing access to the service by schools and kura.

ERO identified a trend across the clusters that showed about 75 percent of requests for support were for boys. Discussions in some clusters indicated a level of acceptance that this was the case. Little was being done to explore why this was so or what was working or not working to improve outcomes for boys. It is interesting to note that the Funding Agreement does not include boys as a specific priority group in its Service Priorities.

An analysis of the nature of requests for support in a sample of RTLB clusters showed that ‘learning’ or ‘learning and behaviour’ made up the bulk of requests. Referrals for ‘behaviour issues’ tended to be much fewer than for learning related categories. This is an area that clusters might want to explore further.

ERO found mixed views about how well the RTLB service was working in the secondary school context. Several clusters were trialling different approaches to RTLB in secondary schools. Principals and SENCO highlighted that RTLB credibility to work in secondary schools and a teaching background in this context was desirable. Some principals wanted a different approach for secondary schools, including a view that giving the funding to individual schools would be the best solution.

ERO identified variability across the 40 clusters in the following areas:

  • case length (15 to 40 plus weeks)
  • frequency of referral and intake meetings (weekly to six weekly)
  • the involvement of Regional Ministry of Education Learning Support managers in referral and intake meetings
  • the extent to which clusters were monitoring, analysing and reporting on referral data and case work.

Next steps for improvement

RTLB clusters ensure equitable access to the service for all schools and kura, by identifying enablers and barriers to access as part of their internal evaluation.

RTLB clusters evaluate the impact of their interventions, programmes and initiatives for improving learning and wellbeing outcomes for boys. 

RTLB cluster managers discuss the variability in practice (identified above) to determine if there is a need for clearer expectations and parameters. 

Personnel management

ERO found a very positive shift in personnel management through the transformation process. As shown in Figure 5, most (34) clusters were managing people well. Some had recently reviewed their appraisal processes, resulting in a more robust approach aligned to recent changes to requirements from the Education Council. ERO also found many clusters had adopted a strategic approach to appointing new RTLB based on cluster needs. They would not appoint unless they had the right person with the right fit of capabilities they were looking for.

Figure 5: Personnel management

This image is of two bar graphs. The top statement reads Personnel management has improved since 2009. The first graph is 2009. The y axis is four points which from the top are Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2009 this is Very good 3, Sound 14, Limited 15 and Minimal 8. The second graph is 2017. The y axis is four points which from the top are Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2009 this is Very good 28, Sound 6, Limited 6 and Minimal 0.

In most clusters induction and mentoring processes were well embedded and personalised for individual RTLB. RTLB had multiple pathways through which they accessed supervision including one to one with a practice leader or cluster manager, peer supervision and external supervision. Practice leaders played a key role in supporting and growing RTLB capability.

In some clusters, RTLB had opportunities to share practice and reflect on case work through Communities of Practice or other similar professional learning groups. Professional development was clearly linked to cluster priorities and RTLB needs. Such development was valued and responsive. Learning opportunities were collegial, drawing on RTLB expertise, along with access to external expertise when needed.

Where ERO found personnel management was ‘limited’ the issues identified included one or more of the following:

  • appraisal not formalised or sufficiently robust, particularly where there were also issues with RTLB performance and practice
  • supervision was not linked to the Professional Practice Toolkit
  • teething issues with the implementation of new personnel management systems, for example appraisal processes
  • ongoing relationship issues within the RTLB cluster impacting on service provision.

The following table shows the key shifts for personnel management.

Table 4: Shifts from 2009 to 2017 – Personnel management

From (2009 To (2017)

Personnel practices not in accordance with RTLB policy.

Not all RTLB getting their entitlements in relation to appraisal, professional development and professional supervision.

Issues with RTLB not undertaking their work in a professional manner.

Robust policies and procedures, well implemented and clearly aligned to Ministry expectations and requirements.

Ongoing improvement to RTLB practice and capability through up-to-date processes for appraisal, professional development and supervision.

Highly professional RTLB workforce in most clusters.

Next steps for improvement

Next steps in a few clusters include one or more of the following:

  • increasing access to supervision
  • ensuring access to cultural supervision for Māori and Pacific RTLB
  • providing the opportunity for Māori RTLB to be appraised by Māori
  • strengthening appraisal and fully implementing new/revised processes
  • including a focus on the role and responsibilities of practice leaders in their appraisal
  • building RTLB capability to mentor their peers
  • addressing the resistance of a few RTLB to change.

Professional relationships

ERO found evidence of strong professional relationships in most clusters. Where these were found to be ‘very good’ they were strong at all levels of the cluster. Cluster managers were proactive in developing and maintaining relationships. This has been a strong focus as part of the transformation process. The RTLB liaison role was pivotal in promoting and maintaining positive working relationships with SENCO and principals in individual schools/kura. Cluster Advisory Groups (CAG), where these existed, were an important forum for fostering professional relationships across the cluster.

As shown in Figure 6, 36 of the 40 clusters have sound to very good professional relationships within the RTLB team and with schools, kura and other stakeholders in their cluster.

Figure 6: Professional relationships

This image is of two bar graphs. The top statement reads Professional relationships in the RTLB clusters are greatly improved. The first graph is 2009. The y axis is four points which are from the top down Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2009 this is Very good 12, Sound 13, Limited 12 and Minimal 3. The second graph is 2017. The y axis is four points which are from the top down Very good, Sound, Limited and Minimal. The x axis is number of clusters. For 2009 this is Very good 29, Sound 7, Limited 4 and Minimal 0.

The following table shows the key shifts in professional relationships.

Table 5: Shifts from 2009 to 2017 –Professional relationships

From (2009) To (2017)

Professional relationships were a positive feature of many clusters and this was largely because of the personal qualities of individuals.

Where relationship issues existed they were often deeply embedded in the culture of the cluster and were negatively impacting on the quality of the RTLB service.

Professional relationships continue to be a strength of most clusters and these have been strengthened through the transformation process.

Cluster managers have been a key influence in strengthening and sustaining professional relationships at all levels of the cluster.

Professional relationships need strengthening in a few clusters.

In the few clusters where ERO found professional relationship issues, these were impacting negatively on the cluster. However, the extent of these issues varied and new leadership or mediation interventions in place in these clusters were helping to bring about improvement.

Next steps for improvement

The Ministry monitors the situation in the RTLB clusters where relationship issues have the potential to impact on service provision and RTLB practice and provides support and guidance as needed.

Communication

Communication was a positive feature of the way most clusters operated. Communication was multi-faceted, acknowledging face-to-face interactions were really important. The visibility of the cluster manager through regular visits to schools and kura, and at meetings of various groups within the cluster was also important. Regular newsletters, emails and feedback via CAGs (where these existed) helped to keep schools and kura informed about what was happening in the cluster. Figure 7 shows the positive shifts from 2009 to 2017, with 37 of the 40 clusters having sound to very good communication.

Figure 7: Communication

This image is of two bar graphs. The top statement reads Communication has much improved since 2009. The first graph is 2009. The y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is Number of clusters. For 2009 this is Very good 9, Sound 15, Limited 13 and Minimal 3. The second graph is 2017. The y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is Number of clusters. For 2017 this is Very good 23, Sound 15, Limited 2 and Minimal 0.

The following table shows the key shifts in relation to communication.

Table 6: Shifts from 2009 to 2017 – Communication

From (2009) To (2017)

Effective communication activities kept all involved in the cluster well informed.

Where communication was poor it led to lack of involvement and participation of schools in the cluster, and unrealistic expectations of RTLB and under-utilisation of the service.

Clusters taking a multi-faceted approach to communication using differentiated approaches to suit different contexts.

Communication clearly promoting high levels of participation in and use of the RTLB service.

A challenge for many clusters was to broaden communication activities to include iwi and early learning services as required in the Funding and Service Agreement. In a few clusters, improved communication could help address misunderstandings about the role of the RTLB and what the service provides.

Next steps for improvement

RTLB clusters broaden and improve communication activities to include iwi and early learning services involved in transition support projects and to address misunderstandings about the role of RTLB and what the service provides.

Leadership

Leadership, particularly by cluster managers and lead school principals, was a key factor in the successful transformation of the RTLB service. In all but one cluster, ERO’s findings highlighted the critical role of the cluster manager and lead school principal in bringing about the improved systems and processes at the core of the transformation. When working well, the layers of leadership within RTLB clusters (cluster manager, lead school board of trustees, lead school principal, practice leaders and RTLB) strengthened the quality and responsiveness of service provision.

Leadership changes in some clusters since 2012 have generally been well managed and resulted in improvements to the service.

As shown in Figure 8, the leadership of clusters has improved with 39 of the 40 clusters being well led.

Figure 8: Leadership

This image is of two bar graphs. The top statement reads Leadership has strengthened considerably over time. The first graph is 2009. The y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is Number of clusters. For 2009 this is Very good 4, Sound 13, Limited 17 and Minimal 6. The second graph is 2017. The y axis is four points which are from the top Very good, Sound, Limited and Minimal. The x axis is Number of clusters. For 2017 this is Very good 23, Sound 16, Limited 1 and Minimal 0.

The following table shows the key shifts for leadership.

Table 7: Shifts from 2009 to 2017 –Leadership

From (2009) To (2017)

Leadership was a key factor in the governance and management of RTLB clusters.

Weak professional leadership and poor governance and management practices meant RTLB took the initiative and largely managed themselves.

Limited leadership opportunities.

Lack of opportunities for ongoing training and support to enhance leadership capability in clusters.

Leadership, particularly by the cluster manager, was a key factor in the successful transformation in most clusters.

Cluster managers and lead school principals working together with a clear understanding of their respective roles and responsibilities.

Many leadership opportunities in clusters.

National forums and formal and informal networking between clusters providing opportunities for sharing and capability building.

Next steps for improvement

The Ministry provides induction support for new cluster managers, lead school/kura boards of trustees and principals when there are personnel changes.

Capability and capacity to monitor and evaluate

To what extent has the transformation of the RTLB service contributed to increased capability and capacity, within clusters, to monitor and evaluate RTLB practice and service in order to identify what is working well and what needs to improve?

The capacity and capability to monitor and evaluate RTLB practice and service provision has improved since the transformation of the service. Self review was the weakest area of cluster performance in ERO’s 2009 evaluation. Since then the Ministry has provided further guidance in Governing and Managing RTLB Clusters, including a process of peer cluster review in which just over a quarter of clusters have engaged.

Figure 10: Capacity and capability to monitor and evaluate

This image is a bar graph. The top statement reads Evaluation is an area for ongoing development. The Y axis is four points they are from the top High, Developing, Limited and Minimal. The x axis is number of clusters. This graph reads High 9, Developing 24, Limited 6 and Minimal 1.

Common features of the nine clusters rated ‘high’ included:

  • robust and aligned strategic and annual planning based on robust needs analysis
  • regular reporting to key stakeholders
  • ongoing monitoring and review of progress against cluster goals and priorities, and of RTLB practice in case work
  • an emerging evaluative approach to internal evaluation.

Examples of developing evaluation practice included:

Example 1
This cluster undertook an internal evaluation of data from the feedback in their case closure survey. They graphed the responses and although the responses were mostly positive they looked more closely at the not so positive comments and practice leaders responded to these. A framework of questions was used to reflect on their actions going forward:

  • what have we learnt?
  • what decisions did we make and why?
  • what impact will these changes have on people, processes, resources or the environment?
  • how will these changes benefit children and their learning?
  • who is responsible for implementing the changes?
  • what are the timeframes for implementation?
  • how can we monitor the effects of these changes?

For this cluster, these reflective questions would be useful in a subsequent follow-up evaluation to frame both the evaluative inquiry and associated thinking. Such questions would deepen the focus on learners and RTLB practice. For example, asking at the beginning of the evaluation “to what extent have the changes we made benefitted children and their learning? Which children benefitted and why, and which have not and why? What has been the impact of RTLB practice in building teacher capability to respond to the learning and behaviour needs of their learners?

Example 2
This cluster engaged in a peer-review process with another cluster that focused on case work review meetings. The leadership team wanted to know whether a new system introduced in 2014 was achieving what they intended - “to support RTLB to fully understand how to link intervention with data gathering and analysis, as well as to discuss with practice leaders legitimate concerns about case work.” Data was gathered from an online questionnaire, along with observations and face-to-face interviews with individual practice leaders and RTLB. The findings of the peer review concluded that the case work review meetings were an effective and useful vehicle to achieve the intended purpose. The review found there were shared understandings about the purpose of the meetings. Some suggestions for improvement were noted by the peer review team.

This peer review could have been more evaluative had the questions used been more evaluative in their nature. For example:

Instead of asking:
“Do cluster XX RTLB understand the purpose of both meetings?”
Ask instead
“How well do cluster XX RTLB understand the purpose of both meetings?”

Instead of asking:
“Is there consistency across the teams?”
Ask instead:
“To what extent is there consistency across the teams?”

Some of the 33 clusters rated ‘high’ or ‘developing’ were beginning to use ERO’s Effective Internal Evaluation for Improvement to build their capability to evaluate provision and practice. A few were using ERO’s 2009 RTLB synthesis rubric as a framework for their review. ERO also found increasing use of data from the database to monitor case work and review aspects of practice. Planning and reporting requirements were being met, with some variability in robustness of the needs-analysis process. A reflective culture of inquiry was fostered, particularly for case work and appraisal processes. Next steps for most of these clusters included taking a more evaluative approach to their reviews, digging deeper into patterns and trends, identifying outcomes and what works for learners. A few also needed to strengthen their analysis of data, and planning and reporting processes.

Six of the clusters were found to have ‘limited’ capability and capacity to monitor and evaluate service provision and RTLB practice. These clusters had a variety of issues with planning, reporting and internal evaluation. The one cluster found to have ‘minimal’ capability and capacity had other issues that affected its capacity to monitor and evaluate its performance.

Priority needs to be given to ongoing evaluative capacity and capability building so that cluster practice can move beyond monitoring and review. Clusters need to adopt a more evaluative approach to current self-review processes. This includes analysing and evaluating patterns and trends in data, and reporting the findings. Strengthening the robustness of needs analysis, and the extent to which this information is used to identify cluster needs and priorities, would enable clusters to target resources where best needed.

Next steps for improvement

RTLB clusters build their capability and capacity to undertake robust internal evaluation.

Involvement with Kāhui Ako

How are RTLB clusters involved with Kāhui Ako and how is the relationship developing? What’s working well and what are the challenges?

In many RTLB clusters, relationships with Kāhui Ako were at an early stage of development. Kāhui Ako were also at different stages of forming, with some having had their leaders appointed and their achievement challenges endorsed, and others not yet at that stage.

A key feature of the RTLB clusters where relationships with Kāhui Ako were developing was the pro-active approach of cluster managers. Cluster managers were:

  • in contact with Kāhui Ako leaders via email, phone calls and visits
  • emailing schools in a Kāhui Ako outlining how they could help and support the work of the community
  • meeting with Kāhui Ako lead principals on a one to one basis
  • discussing options for engaging with Kāhui Ako with their CAG
  • helping Kāhui Ako with the development of achievement challenges
  • working with a neighbouring RTLB cluster where there was crossover in Kāhui Ako boundaries.

The following are a few examples of how RTLB clusters are beginning to work with Kāhui Ako:

  • Allocating a liaison RTLB to each of the four Kāhui Ako in the cluster.
  • Giving short-term funding (1.0 FTE) to a Kāhui Ako because the cluster was yet to fill an RTLB vacancy.
  • Establishing a protocol for RTLB working in the Kāhui Ako.
  • Extracting data from the cluster database for use by the Kāhui Ako.
  • Aligning the RTLB cluster’s focus on boys’ learning with one of the achievement challenges in the Kāhui Ako.
  • Allocating $1,000 of RTLB Cluster LSF to each school in the Kāhui Ako.
  • Trialling a triage approach to allocating referrals for support in a Kāhui Ako.

RTLB clusters were facing many and varied challenges as the relationships and ways of working with Kāhui Ako were developing. In some clusters, concerns were raised about the expectations of RTLB to work with Kāhui Ako without any national engagement protocols. ERO found a degree of uncertainty amongst RTLB cluster personnel about how to proceed in a very dynamic and changing environment as Kāhui Ako were becoming established in increasing numbers. ERO also found that if there was some resistance to using the RTLB service by one or two schools in a Kāhui Ako, this negatively impacted on the RTLB cluster working with the Kāhui Ako as a whole. In one case the resistance was being driven by a view that RTLB clusters should be disbanded and the funding given directly to the Kāhui Ako. Concern was also expressed that RTLB may be captured and working only in one Kāhui Ako, as was the case in some RTLB clusters before the transformation of the service.

Since ERO undertook the data gathering for this evaluation, the Ministry has revised its Funding Agreement with RTLB clusters. This new agreement, which is for the period 1 September 2017 to 31 December 2019, sets out expectations that RTLB clusters will work with Kāhui Ako. The Ministry is also piloting a new service delivery approach of Learning Support through Kāhui Ako that will provide the opportunity to strengthen relationships with RTLB clusters.

Next steps for improvement

The Ministry monitors how RTLB clusters and Kāhui Ako are working together and provides timely support and guidance to clusters, to make sure the learning and wellbeing of students is central to decision making and collaborative efforts.

Contribution to wider provision of learning support

What contribution is the RTLB service making to the wider provision of learning support?

ERO found that the RTLB service is making an important and valued contribution to the wider provision of learning support in the schooling sector, including, as mentioned above, the developing work with Kāhui Ako. Since 2009, the accountabilities and responsibilities of the RTLB service have become more diverse, resulting in a much wider scope, while maintaining the traditional casework of requests for support for individuals, groups and at a school level. RTLB were involved in, and contributed to, a wider provision of support through various projects, programmes and initiatives and interventions. This support includes each school and kura having a liaison RTLB with a focus on establishing and maintaining professional relationships, building SENCO capability to provide support to teachers before requests for support are made, and to prioritise requests when they are made. The impact of much of this liaison work was not captured in the database or evidenced in any way other than through anecdotal comments.

RTLB were also contributing through involvement in, and leadership of, a variety of projects and initiatives (as per the Funding and Service Agreement). These included:

Some of this work was well monitored and reported on in the database and some of it was not captured in any way. Sometimes this was because of the joint nature of the work, with no clear expectations about which agency was responsible for monitoring and evaluating the impact on learner outcomes.

Considerable change in personnel in Ministry regional Learning Support staff has negatively impacted on the extent to which many RTLB and Ministry Learning Support staff have been able to work in collaborative and seamless ways.

ERO found instances where RTLB were ‘gap filling’ because of a delay in response from other agencies, often due to capacity issues in these agencies. ERO was told about the lack of clarity for schools about which service was providing which support. Principals and SENCO also expressed concern about the RTLB service being stretched, particularly with increasing demands for support beyond the RTLB role. This included the need for support for students in relation to trauma, mental health issues and extreme behaviour. ERO also identified issues with students ‘falling through the cracks’ when transitioning between services or between RTLB clusters.

Despite these barriers, ERO found examples of productive collaboration in RTLB clusters. In some clusters the Ministry regional Learning Support Manager was involved in regular referral and intake meetings. This involvement helped to identify where co-working on a case was necessary or when another agency or service needed to be involved. As noted earlier, waiting lists or capacity issues in other agencies meant requests for support were not always responded to in a timely manner.

Other examples of collaboration included:

  • RTLB and Ministry regional Learning Support staff engaging in shared professional learning opportunities
  • neighbouring RTLB clusters working together
  • the RTLB cluster manager, RTLB and Ministry regional Learning Support all working on the same site
  • regular meetings between RTLB cluster personnel and Regional Ministry Learning Support staff
  • developing protocols for joint work by RTLB and Ministry regional Learning Support staff
  • RTLB helping schools to engage with Māori and Pacific families (and parents more widely).

Transitions

One of the expectations in the Funding and Service Agreement is for RTLB to support transitions. However ERO found variability in the extent to which RTLB clusters focused on supporting transitions. It was clearly a strength for some, with cluster personnel and principals and SENCO commenting on the value of RTLB involvement in transitions, whether from an early learning service to primary school, or from primary to secondary school. Some RTLB clusters, working with Early Intervention teams and early learning services, have enabled smooth transitions into primary school for children who were receiving learning support. Generally this work did not include supporting children in Kōhanga Reo to transition to primary school or kura. ERO also heard of transition support being needed for students moving from kura to English-medium schools, and particularly when moving to an English-medium secondary school.

Although many RTLB clusters were supporting transitions in a variety of ways and at different stages in the learner pathway, there was little robust evidence to show what difference this work was making. Evidence was mostly anecdotal; for example, ERO was told that because of this transition work, schools were better prepared for students and stronger relationships were developing between early learning services and schools.

Next steps for improvement

Clusters evaluate the effectiveness of transition support and its impact on student wellbeing and learning.

The Ministry and RTLB clusters work together to establish clear expectations and responsibilities for monitoring and evaluating the impact of joint collaborative work on learner outcomes.

The Ministry works with RTLB clusters to make sure there is clear and consistent communication to schools and kura about the changing landscape in which RTLB are working and contributing to the wider provision of Learning Support.

Impact of RTLB service

What evidence is there of the impact of the RTLB service on improving learner outcomes?

Evidence of the impact of the RTLB service on improving learner outcomes was somewhat limited. Systems to capture such evidence were not yet able to generate information in meaningful and useful ways.

ERO found that while most clusters were data rich for individual case data and casework monitoring, its use beyond the individual case was often limited. A few clusters were evaluating specific interventions or programmes using data from a variety of sources, however this was not common practice across the service. Cluster managers and lead school principals need support with gathering, analysing, reporting and using data to evaluate the impact of the RTLB service on improving outcomes for learners.

Most of the RTLB clusters are using the Outcomes Framework as set out in the Professional Practice Toolkit  and reporting some data to the Ministry and to the lead school board of trustees. However a few are still coming to grips with the database and this has limited their reporting to the Ministry, the lead school board of trustees and wider community.

ERO identified several issues with the Outcomes Framework that limited clusters’ ability to report meaningfully on the impact of the service on learner outcomes. Issues with the framework included its broad nature, which left it open to interpretation and subjectivity. A lack of processes for moderation in many clusters compromised the reliability of judgements made. As a result, averaging the outcomes data from individual or group case work to a cluster level resulted in meaningless data unsuitable for reporting and decision-making purposes.

Another issue with the Outcomes Framework was lack of a clear connection with achievement and wellbeing data in schools and kura, and to the curriculum frameworks and assessment tools used by schools and kura. Coupled with this is the question of how to monitor the sustainability of progress for learners who have been the focus of an RTLB intervention and how to capture and evaluate the impact of RTLB practice on improving teacher capability.

Generally, at case closure, RTLB report against specific goals and outcomes. Some clusters have identified the need to improve the robustness and relevance of the goals set for individuals or groups of students. Often case closure reports included whānau and student voices, and some clusters were seeking teacher and SENCO feedback at case closure and collating it as part of their review processes. This information is valuable but yet to be fully used to contribute to a wider evaluation of provision and practice in RTLB clusters.

ERO found that although many clusters had sound systems and processes for monitoring and review, they had not yet shifted to using a more evaluative lens that focused on what was working and for whom and why (and what was not working, for whom and why). This is an important next step for clusters to be able make evidence-based decisions about the allocation of resource and expertise to maximise effectiveness of their service.

ERO also found variable practice regarding analysis of outcomes data for Māori students. Some clusters had data at the cluster level, but the averaging of this data meant it was not useful. For example, in one cluster the averaged outcomes data showed that Māori students in kura were making greater gains (learning achievement) that non-Māori. However there was no explanation as to why this was so and what was contributing to the gain (or if the gains were good enough or great).

At the time of the ERO evaluation, RTLB clusters were using one of two different databases and in some clusters different versions of the same database. This situation impacted on the extent to which the Ministry could use the data and information it received from clusters on a quarterly and annual basis. The data management systems being used in the RTLB service were also not compatible with systems in other agencies thus limiting the extent to which data could be shared and used to make sure learners are at the heart of the system.

Next steps for improvement

The Ministry works with RTLB clusters to review the Outcomes Framework, and address issues raised in this report about its purpose and usefulness.

The Ministry supports cluster managers and lead school principals to evaluate the impact of the RTLB service on improving outcomes for learners using a more evaluative lens that focuses on what is working and for whom and why.

The Ministry works with RTLB clusters to explore how to best monitor and evaluate the sustainability of progress for learners who have been the focus of an RTLB intervention.

The Ministry works with the RTLB clusters to make more use of the database for monitoring, evaluation and reporting.