Appendix 2: Evaluation Framework and synthesis tools

The overarching evaluative question was:  How effective are activity centres in promoting positive outcomes for students? 

Exploring:

  • What is working well?
  • Who does it work for?
  • How does it work i.e. what is good practice?
  • What, if anything, impedes the achievement and promotion of positive student outcomes?

ERO domain

Evaluation questions

Possible investigative questions

Indicators of what ‘good’ could look like

 1.  Stewardship

  • To what extent is the Board of Trustees of the managing school involved in the operation of the activity centres?
  • How well does the managing school ensure quality outcomes for students?

 

  • What info does the Board of Trustees require/receive?
  • What strategic direction does the board give?
  • How does it know how if the ‘right’ students are being supported by attending the activity centre?
  • How does it know how well students do?
  • How is information used?
  • How does the managing school’s board govern the activity centre?
  • What reporting is undertaken by the activity centre? To whom does this reporting go?
  • How frequently does reporting to the board, or to the principal, occur?
  • The board is proactive in meeting its statutory obligations:
    • awareness of MoU and acts on its responsibilities regarding the activity centre
    • written agreements with enrolling schools, as per MoU, are explicit about expectations and responsibilities for students (including selection and transition processes and access to ongoing extra‑curricular activities at their enrolling school)
    • the board checks that these agreements are upheld and follows up if there are issues
    • collective responsibility for students is evident.
  • Board scrutinises the effectiveness of the activity centre in achieving valued student outcomes
    • Host principal is held accountable – activity centre in the principal’s performance agreement
    • Strategic planning includes activity centre (alignment to appraisal, PLD, improvement, resources)
    • Internal evaluation re effectiveness of activity centre is of high quality – including seeking feedback from enrolling schools, teachers, students and whānau
    • Reports to board include sufficient information to make informed decisions (incl Accountability, effectiveness and challenges)

 

2.  Leadership

  • How effectively does the leader of the managing school (or management committee of activity centre) support equity and excellence for students in the activity centre?
  • How effectively do the managing principal and director of the activity centre ensure:
    • the environment is orderly and supportive, conducive to student learning and wellbeing?
    • effective planning and evaluation of the curriculum and teaching?
  • How does managing school leader support activity centre director and staff?
  • Are there clear guidelines for staff at the activity centre?

How is teacher capacity improved? 

  • Leadership establishes clear and consistent expectations for student behaviour and respectful behaviour management strategies.

  • Leaders have high expectation for student success.High quality leadership promotes teacher learning and development

      • managing school principal is professionally engaged with the director of the activity centre
      • managing principal supports the activity centre with appropriate resources,  PLD, access to curriculum
    • Leadership provides an accurate and defensible evaluation of the activity centre’s performance.

    • Leadership engages constructively with external evaluation.

    3.  Educationally powerful connections and relationships

    Selection

    • How effectively are the connections with parents/whānau, enrolling schools and external agencies used to inform suitability of student for the activity centre?  
    • Are the students that will benefit the most the ones being identified to attend the activity centre?
    • How are decisions made with regard to which students to take?
    • Do all schools follow criteria for selection?
    • Activity centre leadership has clear understanding of reasons for each student being there.
    • Students, parents/whānau and teachers know the different pathways, programmes and options and supports available and participate in decision making at critical transition points.
    • Selection processes are consistently followed across enrolling schools:
      • students have fair and equitable access to the activity centre.
    Transition
    • How effective are the transition processes into the activity centre?
    • What steps do they take to connect with students, parents/whānau to introduce them to the activity centre and its purpose?
    • How are students settled into the centre?
    • How is te ao Māori reflected in the activity centre?  
    • Students, parents/whānau and teachers know the different pathways, programmes and options and supports available and participate in decision making at critical transition points.
    • Transitions are well managed, enabling students to settle quickly into the centre and to their learning:
      • transition processes actively involve parents/whānau
      • ILPs are developed in a timely manner.
    • Students are well settled.

    • Activity centres are culturally welcoming environments.
    Knowing the students
    • To what extent does the activity centre identify and address each student’s specific needs, interests and strengths, background and experiences?
    • Who do they get information from?
    • What is the quality/reliability of the information?
    • How do they develop learning partnerships with student and parents/whānau and other agencies involved?
    • What assessments, if any, do they do themselves?
    • How do they build relationships and trust with their students?
    • Activity centre receives a variety of good information to learn about their students (social, emotional and academic):
      • students are asked about themselves: interests, strengths, areas to work on and what supports their learning
      • parents/whānau provide information about their child and aspirations for them
      • enrolling schools provide detailed information
      • teachers from enrolling schools visit their students at the activity centre
      • agencies involved contribute to understanding the student.
    • Teachers and leaders use valid and reliable approaches to identify the educational strengths and areas for development of students:
      • including identifying the needs of students with any physical, sensory, psychological, neurological, behavioural or intellectual impairments
      • processes to identify and support the needs and aspirations of Māori and Pacific students and their parents whānau are culturally responsive and embedded in the tikanga of the centre.
    4.  Responsive curriculum, effective teaching and opportunity to learn
    ILPs
    • To what extent do ILPs support students to achieve positive outcomes?
    • What is the development process and who is involved?
    • How are focus areas chosen?
    • Do ILPs reflect how Māori students are encouraged to gain success as Māori?
    • How often are these reviewed?
    • Parents/whānau and teachers work together with students to identify their strengths and learning needs, set goals, and plan responsive learning strategies and activities:
      • minimum ILPs criteria met as in Memorandum of Understanding
      • goals are specific and measurable
      • goals include a focus on building self‑regulation and self esteem
      • where appropriate there is a te ao Māori focus for the student.
    • ILPs contain set challenging and appropriate expectations for learning and social and emotional  development:
      • they identify the support (strategies & agencies) to help students achieve their goals
      • student agency and responsibility for learning are featured in the ILPs
      • ILPs are regularly and formally reviewed in partnership with students, parents/whānau, and revised in line with student progress and needs
      • there is a plan for transition to future education/training/employment. 
    Curriculum design
    • What is the relevance and quality of the curriculum in relation to the ILPs?

     

    • To what extent is the activity centre focused on improving educational and social outcomes for each student?
    • How does the curriculum link to ILPs, pathways and transitions out of the activity centre?

     

    • What cross curricular access is there (e.g. to managing school curriculum)?
    • Teachers are making good use of the ILPs to plan, implement and review the curriculum for them:
      • educational activities include contexts that are authentic and relevant to individual students
      • teachers promote achievement by deliberately aligning tasks design, teaching activities and resources
      • appropriate use is made of external programmes to support students – e.g. te Kura courses.
    • Management of the curriculum ensures it is coherent and students have sufficient opportunity to learn.
    • Curriculum offered to students engages them in learning.
    • Students are aware of the progress they are making.
    • There are processes for accessing and maintaining coherent interagency support for students while they are in activity centres.
    Culture
    • To what extent is the culture of the activity centre focused on improving educational and social outcomes for each student?
    • How well settled are the students?
    • How safe, supportive and affirming is the environment?
    • How are students’ wellbeing needs addressed?
    • Do staff use the wellbeing indicators?
    • How is the Health and PE curriculum delivered for these young people (including sexuality education)?
    • The learning environment is:
      • safe emotionally and physically
      • managed in ways that support participation, engagement and agency in learning. 
    • Relationships are respectful and productive; difference and diversity are valued:
      • staff are compassionate, actively listening to students and reflecting their points of view.
      • students express a sense of security and comfort with the environment and relationships.
    • Educational environment reflects manaakitanga, whanaungatanga, ako, mahi tahi.
    • The learning community is characterised by respect, empathy, relational trust, cooperation and teamwork.
    • Students participate and learn in caring, collaborative, inclusive learning communities.
    Curriculum
    • To what extent is the pedagogical approach in the centre supporting positive outcomes for students?
    • How effective are external programmes in improving student outcomes?
    • What is the quality of the curriculum?
    • How is the curriculum impacted by the capacity and quality of teachers in the activity centre?
    • What cultural emphasis is integrated into the curriculum (te ao Māori) to support Māori succeeding as Māori?
    • Is the use of external programmes appropriately linked to ILPs?
    • How are students supported in managing successful completion? 
    • Student identities, whānau and community knowledge, language and culture are represented in curriculum materials and enacted curriculum.
    • The curriculum makes connections to learner’s lives, prior understandings, out of school experiences and real‑world contexts:
      • students are engaged in their learning.
    • Curriculum supports the development of the key competencies (NZC):
      • students are able to work together in some situations, discussing ideas, reaching conclusions and teaching each other
      • students develop dialogue and group work skills.
    • Assessment activities are inclusive, authentic and fit for purpose:
      • providing meaningful evidence of achievement
      • students receive and give timely, specific and descriptive feedback to determine their goals, progress, next steps and success criteria
      • students are taught to evaluate their own learning and are aware of their achievements and next steps.
    • Students identify their own learning needs and take responsibility for their own learning.
    • Any significant social or health issues are addressed through appropriate agencies.
    • External programmes are well linked to ILPs.
    • Students manage the completion of external programmes well.
    5. Professional capability and capacity
    • How effective is the curriculum delivery at the activity centre?

    • How effective is the pedagogical leadership in the activity centre?
    • What is the capacity and quality of teachers in the activity centre?
    • What are the processes for managing teacher performance?
    • What pedagogically focussed PLD is available for staff?
    • How are teachers supported to manage behaviour and motivate students to learn?
    • Where is this PLD sourced/ delivered from?
    • What pedagogic resources are available to staff in the centre? 
    • Teachers are well qualified and have relevant curriculum, assessment and pedagogical knowledge:
      • staff have positive expectations and enthusiastic about making a difference for students
      • staff demonstrate the importance of social and pastoral care as a pathway to support achievement of students
      • assessment information (including narratives) demonstrate progress made by students – progress measured again social adjustment, behaviour and academic achievement.
    • Teachers are skilled at employing the appropriate strategies to engage, manage and extend these learners:
      • teachers employ the appropriate strategies to engage, manage and extend these learners
      • teachers form positive professional relationships with students, focused on learning.
      • Coherent performance management processes:
        • identify teachers’ learning and development needs
        • provide professional learning opportunities that are responsive to identified needs and align with the school/activity centre strategic plan.
        • Teachers have professional learning opportunities:
          • teachers inquire into their practice
          • teachers receive appropriate PLD to support them in their work
          • teachers have the resources they need and make effective use of them
    teachers are culturally sensitive, enabling Māori to succeed as Māori.
    6. Evaluation
    • How effectively does the activity centre work with outcome and transition information to continually improve processes?
    • How do staff evaluate the effectiveness of the curriculum?
    • How do they know how well their students succeed beyond the activity centre?
    • What evaluation for improvement do they do?
    • Evaluation, inquiry and knowledge building are embedded in processes and practices in the activity centre and managing school:
      • processes are systematic and coherent at every level.
    • Appropriate tools and methods are used to gather valid data:
      • information to inform evaluation is sought from students, parent/whānau, teachers, enrolling school and external agencies.
    • The activity centre tracks students beyond their time in the centre, using indicators about behaviour, attendance and academic progress:
      • uses this information in their internal evaluation to improve practice.
    Student outcomes
    • How effective is the activity centre in achieving and promoting positive outcomes for the students?
    • How do the activity centres define ‘success’ and valued outcomes for students?
    • What evidence is there of improved outcomes for students?
    • What data do they have on successes?
    • Students are:
      • confident in their identity, language and culture
      • socially and emotionally competent
      • resilient
      • optimistic about their future.
    • Students improve:
      • attendance – attending regularly
      • behaviour – self regulate.
    • They have confidence in themselves as learners:
      • students manage their learning
      • key competencies
      • literacy and numeracy gains
    • NCEA credits – where applicable
    Transitions
    • How effective are transition processes?
    • How do they define an effective transition?
    • How are successes and outcomes communicated to the managing and enrolling schools, to students and parents/whānau?
    • What processes are in place for students to transition out of the activity centre?
    • Transition processes are set up to ensure ongoing interagency support for students when they leave the activity centre.
    • High quality career education and guidance is given with an emphasis on transition to the workplace or further education/training.
    • Students know about their pathways to further education, training or employment.
    • Students are confident about their next steps after the activity centre.