Appendix 2: Evaluation framework and questions

Appendix 2: Evaluation framework and questions

Overall evaluation questions

  • How confident and prepared do newly graduated teachers in early learning services feel/think they were to use the early childhood curriculum Te Whāriki to:
    • design, implement and evaluate a curriculum that is responsive to the language, culture and identity of all of the children they are responsible for in their service, particularly those children at risk of poor educational outcomes
    • assess children’s progress and learning and use this information to develop teaching strategies to promote positive learning outcomes
    • work collaboratively with the parents and whānau of children to develop and maintain learner-centred relationships.
  • How confident and prepared do newly graduated teachers in schools feel/think they were in relation to:
    • having content and pedagogical knowledge appropriate to the learners and learning areas of their programme
    • having an understanding of the complex influences that personal, social and cultural factors may have on teachers and learners
    • using assessment tools to assess students’ progress and achievement
    • using assessment data to plan teaching strategies that respond to the identity, language and culture of all learners and accelerate the progress of students at risk of underachievement in their class
    • systematically and critically engaging with evidence to reflect on and refine their practice
    • working collaboratively with the parents and whānau of identified students to develop and maintain learner-centred relationships.

Approach to data gathering and synthesis

ERO gathered data from interviews with leaders and NGTs in early learning services and schools and from two online surveys of NGTs. The interviews were undertaken in early learning services and schools having an education review between 23 May and 23 September 2016. ERO reviewers provided NGTs with links to the online survey during the onsite phase of the review. The surveys could only be accessed by NGTs in the services or schools being reviewed.

NGTs’ responses to the surveys were synthesised alongside interview data from leaders and NGTs to develop higher‑level findings.

Questions for semi-structured interviews with leaders and NGTs in services and schools

Questions for semi-structured interviews with leaders in early learning services

  1. What is your experience of employing newly graduated teachers (provisionally registered teachers) in the last three years?
  2. How confident and prepared do you think they were to:

a. design, implement and evaluate a curriculum that is responsive to the language, culture and identity of all of the children they are responsible for in their service

b. assess children’s progress and learning and use this information to develop teaching strategies to promote positive learning outcomes

c. work collaboratively with the parents and whānau of children to develop and maintain learner-centred relationships

      3. What do you see as their strengths? What areas have been a focus for building capability as PCTs? What gaps have you noticed? What support are you providing?

      4. What is the employment status of your current PCTs/FCTs (in first year of being fully certificated?) –eg permanent teaching positions, long or short term relieving?

      5. From your experience of employing new graduates do you have a preference for particular ITE provider(s)? which providers? why?

      6. Are there any ITE providers you wouldn’t employ graduates from? which providers? why?

      7. Do you have any other comments regarding the preparation and confidence of newly graduated teachers?

Questions for semi-structured interviews with PCTs/FCTs in early learning services

  1. Teachers to introduce themselves to reviewers sharing information about when they graduated and what year of PCT/FCT they are. 
  2. How confident and prepared were you when you graduated to:
    • design, implement and evaluate a curriculum that is responsive to the language, culture and identity of all of the children they are responsible for in your service
    • assess children’s progress and learning and use this information to develop teaching strategies to promote positive learning outcomes
    • work collaboratively with the parents and whānau of children to develop and maintain learner-centred relationships.

     3. What did your ITE provider do well to prepare you in relation to these aspects of practice?

     4. What could your ITE provider have done to better prepare you in relation to these aspects of practice?

     5. What support are you receiving to address aspects of practice you need to improve?

     6. How do you know about your strengths and areas for improvement? Self assessment? Peer assessment?

     7. To what extent have you been able to use what you learnt through your initial teacher education programme in this service? What has enabled you to use what you learnt? What have been the barriers or challenges?

     8. What are your views about assessment and assessing children’s learning? To what extent have you been able to use what you learnt about assessment in this service? Is what you learnt reflected in your current assessment practice in this service? If yes, in what ways? If not, what is     different? Why?

     9. What is your employment tenure?

Questions for semi-structured interviews with leaders in schools

  1. What is your experience of employing newly graduated teachers (provisionally registered teachers) in the last three years?
  2. How confident and prepared do you think they were in relation to:

a. having content and pedagogical knowledge appropriate to the learners and learning areas of their programme

    b. having an understanding of the complex influences that personal, social and cultural factors may have on teachers and learners

      c. using assessment tools to assess students’ progress and achievement

        d. using assessment data to plan teaching strategies that are responsive to the identity, language and culture and accelerate the progress of students at risk of underachievement in their class

          e. systematically and critically engaging with evidence to reflect on and refine their practice

            f. working collaboratively with the parents and whānau of identified students to develop and maintain learner-centred relationships.

                 3. What do you see as their strengths? What areas have been a focus for building capability as PCTs? What gaps have you noticed? What support are you providing?

                 4. What is the employment status of your current PCTs/FCTs (in first year of being fully certificated?) –eg permanent teaching positions, long or short term relieving?

                 5. From your experience of employing new graduates do you have a preference for particular ITE provider(s)? which providers? why?

                 6. Are there any ITE providers you wouldn’t employ graduates from? which providers? why?

                 7. Do you have any other comments regarding the preparation and confidence of newly graduated teachers?

            Questions for semi-structured interviews with PCTs/FCTs in schools

            1.  Teachers to introduce themselves to reviewers sharing information about when they graduated and what year of PCT/FCT they are. 
            2.  How well prepared do you think you were in relation to:

            a. having content and pedagogical knowledge appropriate to the learners and learning areas of their programme

              b. having an understanding of the complex influences that personal, social and cultural factors may have on teachers and learners

                c. using assessment tools to assess students’ progress and achievement

                d. using assessment data to plan teaching strategies that respond to the identity, language and culture of all learners and accelerate the progress of students at risk of underachievement in your class

                e. systematically and critically engaging with evidence to reflect on and refine their practice

                  f. working collaboratively with the parents and whānau of identified students to develop and maintain learner-centred relationships.

                          3.  What did your ITE provider do well to prepare you in relation to these aspects of practice?

                          4.  What could your ITE provider have done to better prepare you in relation to these aspects of practice?

                          5.  What support are you receiving to address aspects of practice you need to improve?

                          6.  How do you know about your strengths and areas for improvement? Self assessment? Peer assessment?

                          7.  To what extent have you been able to use what you learnt through your initial teacher education programme in this school? What has enabled you to use what you learnt? What have been the barriers or challenges?

                          8.  What are your views about the analysis and use of data in teaching and learning? To what extent have you been able to use what you learnt about analysing and using data in this school? Is what you learnt reflected in your current practice in this school? If yes, in what ways? If not what is different? Why?

                          9.  What is your employment tenure?

                          10. Any other comments

                    Online survey questions

                    Questions for PCTs/FCTs in early learning services

                    1. Confidence and preparedness to teach

                    Use the seven Graduating Teacher Standards (with indicators alongside each standard) and a Likert scale to rate their confidence and preparedness against the standards (4 point scale)

                    Provide a box for overall comment

                          2. How confident and prepared do you think you were you to (use 4 point scale and comment):

                        • design, implement and evaluate a curriculum that is responsive to the language, culture and identity of all of the children you are responsible for in your service
                        • assess children’s progress and learning and use this information to develop teaching strategies to promote positive learning outcomes
                        • work collaboratively with the parents and whānau of children to develop and maintain learner-centred relationships.

                    3. What did your ITE provider do well to prepare you in relation to these aspects of practice?
                    4. What could your ITE provider have done to better prepare you in relation to these aspects of practice?
                    5. What aspects of practice have you had support with since you graduated?
                    6. What aspects of practice do you need further support with?
                    7. To what extent have you been able to use what you learnt through your initial teacher education programme in this service? (4 point Likert scale)
                    8. Are you in a permanent teaching position?

                    If yes, how long have you been in this position?

                    If no, what is your tenure?

                    9. Did you participate in a semi-structured interview with ERO reviewers? Y/N

                    10. Any other comments

                    Questions for PCTs/FCTs in schools

                    Preparedness to teach in relation to the Graduating Teacher Standards (GTS)

                    1. Use the seven GTS (with indicators alongside each standards) and a Likert scale to rate their preparedness against the standards (4 point scale)

                        Provide a box for overall comment.

                    2. How well prepared do you think you were in relation to (use 4 point scale and comment):

                    a. having content and pedagogical knowledge appropriate to the learners and learning areas of their programme
                    b. having an understanding of the complex influences that personal, social and cultural factors may have on teachers and learners
                    c. using assessment tools to assess students’ progress and achievement
                    d. using assessment data to plan teaching strategies that respond to the identity, language and culture of all learners and accelerate the progress of students at risk of underachievement in your class
                    e. systematically and critically engaging with evidence to reflect on and refine their practice
                    f. working collaboratively with the parents and whānau of identified students to develop and maintain learner-centred relationships.

                    3. What did your ITE provider do well to prepare you in relation to these aspects of practice?

                    4. What could your ITE provider have done to better prepare you in relation to these aspects of practice? 

                    5. What aspects of practice have you had support with since you graduated?

                    6. What aspects of practice do you need further support with?

                    7. To what extent have you been able to use what you learnt through your initial teacher education programme in this school? (4 point Likert scale)

                    8. Are you in a permanent teaching position?

                    If yes, how long have you been in this position?

                    If no,  what is your tenure?

                    9. Did you participate in a semi-structured interview with ERO reviewers? Y/N

                    10. Any other comments.