Appendix 4: Evaluation framework and investigative prompts

Overarching question for the national evaluation report:
What evidence is there of the impact of the RTLB service on improving learner outcomes?

Additional questions to be answered in the national evaluation report:

  1. To what extent has the quality and consistency of RTLB cluster governance and management improved to address the issues identified in ERO’s 2009 evaluation?
  2. To what extent has the transformation of the RTLB service contributed to increased capability and capacity within clusters to monitor and evaluate RTLB practice and service provision in order to identify what is working well and what needs to improve?
  3. What contribution is the RTLB service making to the wider provision of learning support?
  4. How are RTLB clusters involved in Communities of Learning and how is the relationship developing? What’s working well and what are the challenges?

In order to answer these questions (above) we need to investigate and evaluate the following in each RTLB Cluster:

What evidence is there of the impact of the RTLB service on improving learner outcomes?

  • What difference are you making as a cluster for the learners you serve? How do you know?
  • What evidence do you have of improved outcomes?
  • Where are you at in terms of working with outcomes framework in RTLB Professional Practice Toolkit? Usefulness of this framework? Issues?
  • What do you know about outcomes for Māori learners – in Māori immersion kura? In rumaki/immersion classes? Mainstream?
  • What do you know about outcomes for Pacific learners?
  • Where are you having the most success in terms of outcomes and why? Where are your challenges? How are responding to these?

How well is this cluster governed and managed to improve outcomes for learners?

(An overall judgement as per synthesis rubric - Very good, Sound, Limited, Minimal)

Use the RTLB synthesis rubric 2017 to make a judgement in relation to:

  • operating according to requirements (Funding and Service Agreement Governing and Managing RTLB Clusters and the RTLB Professional Practice Toolkit)
  • funding- including LSF, Y11-13
  • internal evaluation (self review), planning and reporting
  • access to service
  • professional relationships
  • collaboration with Ministry of Education –“seamless provision of service”
  • communication
  • leadership
  • outcomes for learners

This is a snap shot of the RTLB synthesis rubic 2017

Investigative prompts

  • What do you know about the extent to which you operate according to requirements (Funding and Service Agreement, Governing and Managing RTLB Clusters and the RTLB Professional Practice Toolkit)? How do know this? (what evidence do you have?)
  • How have you kept up-to-date with changes in requirements, for example to the RTLB Professional Practice Toolkit?
  • How do you ensure funding is allocated, tracked and accounted for to achieve cluster priorities?
  • How does your cluster’s needs analysis process take account of the voices and perspectives of key stakeholders including iwi and whānau? How do these voices and perspectives contribute to the review and development of your strategic and annual planning?
  • How useful is your cluster needs analysis process? Strengths? Barriers?
  • What are the strengths of your strategic and annual planning? What challenges do you face when planning?
  • How does cluster strategic and annual planning include specific objectives for improving outcomes for Māori and Pacific learners?
  • Your cluster undertakes quite a lot of reporting - how useful is this reporting? What feedback do you get? About what? Who from? How useful is it?
  • How would you describe the capacity and capability you have in your cluster to undertake effective internal evaluation? Have cluster personnel been involved in any professional development to build capability?
  • What are the ‘conditions’ in your cluster that support your internal evaluation? For example a high level of relational trust, tools and methods, embedding evaluative thinking.
  • How do you ensure equitable access to the RTLB service? How do you prioritise? Make decisions?
  • How do you ensure you have robust systems for personal management including the recruitment, induction, appraisal, supervision and development of cluster personnel (particularly Practice leaders and RTLB)?
  • How do you identify areas for professional support for Practice Leaders and RTLB? What works and what are your challenges?
  • What does collaboration with Ministry of Education (national and regional) and ‘seamless provision’ look like in your cluster? What successes have you had and what challenges or issues are you facing?
  • How do you manage the wide range of service expectations (in the Funding and Service Agreement)? How do you manage workloads around these expectations?
  • What does communication look like in your cluster? How does it work (or not) and how do you know?
  • What are the leadership opportunities in your cluster? Who leads? How are you building leadership capability? What are your leadership successes and challenges?

What is the capacity and capability in this RTLB cluster to monitor and evaluate service provision and RTLB practice to identify what is working well and what needs to improve?

(An overall judgement – High, Developing, Limited, Minimal)

Investigative prompts

  • Do you have Māori immersion kura in your cluster? What do you know about the access these kura have to the RTLB service? How well aligned are referral and intervention processes to the kura immersion context?
  • In what ways has your cluster increased the knowledge and capability of RTLB to effectively respond to Māori students (mainstream, rumaki and immersion) referred to the service?
  • How do you monitor and evaluate service provision and practice?
  • What does improvement look like in your cluster?
  • How has capacity and capability to monitor and evaluate been developed or strengthened? For whom?
  • Who leads? Who is involved? How and in what aspects/processes?
  • What opportunities are there for collaborative inquiry and evaluation?
  • What do you know about what is working well and what needs to improve?
  • What are your priorities? How have you determined these? How are they changing over time?
  • How do these priorities align to ‘annual service priorities’ (Māori student achievement, Pasifika student achievement, inclusion)?
  • How do you monitor and evaluate practice and outcomes against these ‘annual service priorities’?

What contribution is the RTLB service making to the wider provision of learning support?

Investigative prompts

  • How has the role of the RTLB service changed since 2012?
  • Where does the RTLB service sit in relation to the wider provision of learning support?
  • What does this RTLB service contribute to this wider provision?
  • What’s working well? What are the challenges?

How are RTLB clusters involved in Communities of Learning|Kāhui Ako and how is the relationship developing? What’s working well and what are the challenges?

Investigative prompts

  • What is the involvement of your RTLB cluster in Communities of Learning|Kāhui Ako?
  • If some involvement, what does this look like?
  • Who is involved? For how long?
  • How is it working?
  • Successes? Challenges