Appendix One: Methodology

ERO gathered information for phase two of this evaluation from 44 schools and five wharekura across New Zealand during Term 2, 2013. ERO review officers talked with school personnel, including school leaders, board chairpersons, guidance counsellors, staff involved in guidance and counselling, and adults working in schools but employed by external agencies.

Evaluation framework

The framework for phase two of this evaluation was developed by ERO’s internal reference group of specialist review officers, in conjunction with the Ministry of Education personnel and an External Reference Group.[53] A specialist team of review officers collected information related to the following questions:

  1. What are the important aspects of the school/wharekura and its community that have an impact on the need for, and provision of, guidance and counselling?
  2. What is the school’s/wharekura’s overall approach to providing guidance and counselling?
  3. How does the school/wharekura implement its approach?
  4. What does the school/wharekura do to promote and maintain the ongoing professional practice and development of those providing guidance and counselling?
  5. How does the school/wharekura manage their relationships both internally and externally for the benefit of the students?
  6. What does the school/wharekura know about the effectiveness and/or impact of its guidance and counselling?
  7. How well does the school/wharekura provide guidance and counselling to students? (overall judgement)

Review officers made a final overall judgement about how well each school provided guidance and counselling to students. They were supported in their judgements by a set of indicators developed by ERO and the External Reference Group. These indicators are included in Appendix 2.

Sample of schools

The 49 schools and wharekura in this evaluation were selected from 438 schools receiving the Guidance Staffing Entitlement in 2012.[54] Schools that were scheduled to have an ERO review in 2013 were, for the most part, excluded, as were those who had recently participated in ERO national evaluations about Careers Education and Appointments and Appraisals, or were participating in other school-based YMH Project initiatives.

Schools were selected to best represent decile, school type, state or state integrated authority, and the geographical location of these 438 schools.

The sample was also selected to include schools that had a variety of government school-based initiatives.[55] A sample of 44 schools and six wharekura were originally chosen to best represent these factors.[56] Some schools originally selected subsequently declined to participate, and one wharekura was unable to be replaced. Appendix 3 provides more detail about the schools in the sample.

Student survey

Students at each of the 49 schools/wharekura were invited to complete an online student survey in either English or Te Reo Māori. The questions in the survey were the same as those used in the phase one student survey. The survey for students was developed in conjunction with the Ministry of Youth Development and focused on:

  • characteristics of good guidance and counselling
  • models of practice – what works well and what does not work as well in schools
  • access to, and approachability of, guidance and counselling staff
  • suggested changes to improve guidance and counselling in schools.

School leaders were asked to invite students to complete the survey in the lead up to ERO’s visit. Students from 18 schools completed the online survey, with a final sample size of 671 students.[57] There were no responses from students at wharekura. The responses from students included those from schools in all four groups discussed in the findings from the school/wharekura visits (judged as doing ‘very well’, ‘well’, ‘somewhat well’ and ‘not at all well’).[58]

Limitations of the student survey

Because the invitations to complete the survey were made by school leaders and not directly by ERO, there was variation across schools in terms of how many students completed the survey and ERO was not able to directly influence the make-up of the sample.

Both Year 12 students and male students were significantly over‑represented in the survey sample. Percentages of Māori and Pacific students were in line with the population of schools receiving guidance and counselling funding. The results below should be considered in light of these demographics.

Appendix 4 includes the quantitative results of the survey, as well as the phase one student survey findings. It also includes a comparison of student demographic data with the rolls of schools that receive funding for guidance and counselling provision.[59]