Appendix 1: Process followed

An evaluation matrix guided ERO’s work

ERO developed a matrix to focus desk-based analysis and onsite observations, organise data collected and inform analysis. The matrix was focused on the impact of NCEA on five aspects determined by the Ministry. These five aspects were:

  • Programme design refers to the courses and opportunities offered by schools and TEOs to provide coherent and flexible pathways for students, taking them through the senior school and beyond to further education, training or employment.
  • Pedagogy refers to how teachers make use of strategies and approaches that develop students’ knowledge and their expertise as learners. It focuses on learning rather than credit acquisition.
  • Assessment strategies refer to how the school or TEO selects and administers assessments (Unit Standards or Achievement Standards, internal or external assessment); how they track student progress towards the NCEA qualification; and manage the number and timing of assessments for each student.
  • Resourcing considers provision of time for teachers to plan, mark and oversee assessments; provision of professional learning for teachers; funding of programmes and assessment opportunities for students inside and outside the school or TEO; and the physical resources. Resourcing is allocated to reflect the needs of all students not just those following the university pathway.
  • Pastoral care and wellbeing refer to the systems in place to maintain a focus on student learning and progress towards a meaningful pathway, and to mitigate stress and pressure that comes from a number of sources.

ERO identified indicators of good practice against each of these aspects, based on international research and ERO’s School Evaluation Indicators.

For each aspects ERO asked the school or TEO:

  • What are the challenges/frustrations for a school/TEO of your type in achieving in this area?
  • What are the triumphs in this area? Things that have worked really well?
  • Any specific innovations you have done or heard other similar schools/TEOs doing?

Matrix for NCEA observational studies

 Aspects Principles9                              Features to consider – what is good practice                          

Curriculum design
Coherence

  • The curriculum is responsive, relevant and meets the individual’s needs and future career pathway
    • good design
    • choice
    • knowledge of learner
    • student centred
    • good future focused advice
    • flexible timetable enables choice
  • Culturally and contextually located, responding to tikanga, promoting Māori success as Māori, authentically integrated

Pedagogy
Inclusion and equity

  • Students and teachers have high expectations for achievement
  • Strong relational pedagogy student:teacher:whānau
  • Teachers use a wide range of teaching strategies to meet differing learning needs
  • Teachers develop students’ expertise as learners – Key Competencies
  • Pedagogy which is culturally responsive to Māori – however few
    • Deliberate design and action

Assessment strategies
Coherence Pathways
Wellbeing
Credibility

  • Student progress is closely tracked and monitored, and additional learning support is provided as necessary
    • Attendance
    • Literacy and numeracy
    • Multiple opportunities for success
    • Student and whānau involvement
    • Students understand the timeframe required for success
  • A rigorous and fair approach to assessment results in students, parents and whānau and the community seeing outcomes as valid and reliable
    • Students are not over assessed
    • Cross-curricular work contributes to a variety of standards
    • Feedback after assessments is both timely and useful
  • Assessment schedules in the school enables distribution of pressure points for learners

Resourcing

  • Learning areas are appropriately resourced and staffed
  • Staff are provided with appropriate PLD:
    • effective teaching strategies
    • individual tracking and monitoring
    • developing individual student learning plans
    • Culturally responsive pedagogy (CRP)
  • Teachers have sufficient time to:
    • monitor and mentor
    • design curriculum, plan, administer, mark and moderate NCEA
    • reflect on the effectiveness of teaching and learning
  • Resources shared with other schools/TEOs
  • Strategic resourcing ensures equitable access for Māori students (especially those coming from Kura)

Pastoral care wellbeing
Pathways
Wellbeing

  • Student wellbeing is paramount
  • Students have coherent learning pathways
  • Senior and middle leaders in the school/TEO manage teachers’ welfare in terms of workload
  • Teachers draw on the specialist expertise of the guidance counsellor and careers advisor in supporting student achievement
  • Parents and whānau participate in strong learning partnerships with the students and teachers
  • Māori and Pacific students are not automatically channeled to practical classes

Individual profiles developed

A profile was constructed for each of the nine school types (Appendix 3). This, to a large extent, was based on the school of that type visited, with consideration given to the findings from the desk-based analysis of other schools of similar type. It proved challenging to provide an accurate composite picture because every school is different with limited similarities between schools within a type.

In the case of the TEOs, each profile represented the findings for the specific TEO visited. The schools and TEOs were given fictitious names to maintain confidentiality. ERO is aware the TEOs may be readily identified and each organisation assured us that would not be a concern to them.

Sampling to development of profiles

Sampling

ERO selected 28 schools reviewed during 2017 who received a three-year return date. ERO also reviewed the files of schools visited as part of ERO’s 2018 evaluation ‘What drives learning in the senior secondary school’. From both groups of schools, ERO selected 40 schools broadly representative of New Zealand secondary schools, providing a range of types, deciles, sizes and locations.

ERO chose the following categories as representative of different school types and the school profile which exemplifies the category is appended:

  • Roll 800, single-sex, low decile, main urban state school – Ngutukākā College
  • Roll 1000, private co-educational school, high decile, main urban, offers dual pathways – Pōhutukawa College
  • Roll 2400, high decile, main urban, state school – Kōwhai College
  • Roll 1400, medium decile, secondary urban, state school – Harekeke College
  • Roll 500, medium decile, minor urban, state school – Horoeka College
  • Roll <200, medium decile, rural, state school – Haumata College
  • Roll 500, low decile, main urban, state, school – Nikau College
  • Integrated (Catholic), high decile, main urban – Rātā College
  • Roll 1500, high decile, minor urban, state school – Manuka College

Two schools, one with a proportionately higher Māori roll (Ngutukākā College) and one with a proportionately higher Pacific roll (Nikau College) were purposefully included when sampling.

Following the desk-based analysis of information about these 40 schools, ERO selected nine schools for onsite visits. Selection was based on the school characteristics identified above and obtaining a geographical spread across the country.

The TEOs were selected from those offering NCEA and following consultation with the Ministry Regional Principal Advisers: Secondary-Tertiary. Consideration was given to location and type. ERO selected three different private tertiary establishments (PTEs) working with Youth Guarantee and Dual Pathway students as follows:

  • major urban, hospitality and building
  • rural, primary industry focus
  • urban, foundation skills.

ERO notes every school and PTE operates within its own unique context and therefore no school will fit our findings in a category perfectly. ERO acknowledges, for example, not every single sex girls’ school will exhibit the same features listed in Ngutukākā College.

Desk-based analysis

ERO carried out a desk-based analysis of the 2017 ERO reports for the original 40 schools against the matrix and good practice indicators. After synthesising the findings against the school types, ERO developed some tentative general assumptions relating to schools, to be tested during the onsite visits of the sample schools. ERO also developed assumptions for the three Tertiary Education Organisations (TEOs) visited.

The synthesis of this desk-based work provided an early indication that the differences within the groups of schools tended to be as great as the differences between the groups themselves despite similarities of size, location and decile.

ERO used these findings together with the indicators of good practice to develop questions for discussion with school leaders and teachers, and to facilitate student voice. ERO also consulted with the New Zealand Qualifications Authority (NZQA) who provided a breakdown of schools’ use of Achievement and Unit Standards.

Onsite visits

While in the schools, ERO spoke with school leaders, teachers with specific relevant responsibilities such as the principal’s nominee, senior school deans, careers advisers, guidance counsellors, and groups of students from Years 11 to 13.

In the TEOs, ERO spoke to staff with similar roles to those in schools, including curriculum leaders, tutors, administrators with oversight of assessment practices, and staff responsible for student wellbeing. ERO also spoke to diverse groups of students, including those from Dual Pathway and Youth Guarantee streams.

Synthesis and profile development

Following the onsite phase, ERO synthesised the findings for each school or TEO against the matrix and tested, where possible, the assumptions made prior to the visit. ERO then identified issues common to all schools or TEOs.

Profiles constructed were based on the school of that type visited, with consideration given to the findings from the desk-based analysis of other schools of similar type. It proved challenging to provide an accurate composite picture because every school is different with limited similarities between schools within a type. Nevertheless, ERO captured the overall key challenges, triumphs and innovations shared by schools.

In the case of the TEOs, each profile represented the findings for the specific TEO visited. The schools and TEOs were given fictitious names to maintain confidentiality. ERO is aware the TEOs may be readily identified and each organisation assured us that would not be a concern to them.

The Ministry developed a profile for Māori-medium schools. This is not a part of this report.

ERO engaged in ongoing consultation and discussion with the Ministry during all the above processes.

Assumptions about schools

These assumptions were tested in onsite visits with each school. The findings itemised in the profiles clearly show, where possible, if any assumptions held true.

Programme Design

  • Curriculum is credit driven, not focused on pathways
  • Alignment of The New Zealand Curriculum and NCEA is not strong
  • Learning areas are siloed and few develop cross-curricular programmes
  • University has imposed a hierarchy of subjects
  • UE puts boundaries on student choice
  • Validity of expectations and limits on gaining UE
  • Accountability pressures influence NCEA choices for students (NZCER)
  • If academic potential not known and students not clear of implications of NCEA – pathways can close off too early

Pedagogy

  • Strong learning partnerships support learners to find appropriate pathways and to succeed in those
  • Teachers have gaps in epistemic thinking
  • Teachers teach to the assessment — students are encouraged to rote learn specific phrases required
  • Teachers tend to avoid the Unfamiliar Text standard
  • Effective implementation of cross-disciplinary planning and project-based learning is not a teacher strength and Professional learning is required
  • NCEA is about the assessment not the learning
  • Purpose of learning unclear (NZCER)
  • Learning How to learn is compromised by content coverage (NZCER)

Assessment strategies

  • Relationships and support from NZQA are positive
  • Assessment is relentless
  • External pressures influence assessment strategies
  • The flexibility of the curriculum and NCEA is used to address students’ potential for success
  • Validity and integrity of assessment across schools remains an issue
  • Perceived unfairnesses: ‘worth’ of credits; Unit Standards vs Achievement Standards, internal vs external
  • Students, and teachers, work the system to get credits required – credit grabbing, credits not relevant to a clear pathway
  • Easier to get excellence if attempting only one paper in external three-hour exam
  • Some difficulty tracking students’ progress to UE as student management systems do not include UE
  • Assessing individuals in collaborative work is a challenge

Resourcing

  • Competition for students or for funding plays a significant part in school practices to show themselves in the best possible light
  • Limited resourcing for careers personnel limits their focus to Year 12 and 13 students

Pastoral care and wellbeing

  • Wellbeing is paramount. Some schools report to their board of trustees about wellbeing outcomes
  • Students pace themselves, reducing stress by limiting the number of standards they attempt
  • Some students driven to gain as many credits or endorsements as possible — pressures from parents, teachers and themselves
  • Parents don’t understand NCEA so find it hard to accept students’ choices
  • Students find it hard to locate information to support their learning from multiple websites when these are not cross linked

Assumptions about TEOs

Programme Design

  • courses are practical programmes
  • very linear
  • no choice for students
  • students do not necessarily move onto further training or employment

Pedagogy

  • many students have been disengaged learners at school
  • less restrictive learning environment
  • students are difficult to motivate
  • students’ motivation impacted significantly by influences outside the TEO
  • students experience success for the first time
  • most students do not have literacy and numeracy skills

Assessment strategies

  • multiple sites make consistent moderation and practice difficult
  • assessments paced throughout the course
  • assessments less challenging

Resourcing

  • well-resourced for staffing and equipment

Pastoral care and wellbeing

  • large and more impersonal environments
  • limited support services for students.

9 The principles are those underlying the NCEA Review.