ERO's progress in 2016/17

ERO purpose, activities and role

ERO’s purpose

ERO’s evaluation insights are a catalyst for change so that every child achieves success as a lifelong learner.

ERO’s core activities

ERO is a government department established in October 1989 under the State Sector Act 1988. The Chief Executive of ERO is the Chief Review Officer.

Under Part 28 of the Education Act 1989, ERO is required to review the performance of pre-tertiary education providers in relation to the educational services they provide. Review Officers are statutory officers designated under the Act. They exercise powers of entry, investigation and reporting.

At an education system level, ERO carries out evaluations of education sector performance and policy implementation, and reports to the Minister of Education about practice in the pre-tertiary sector.

ERO’s core activities in 2016/17 included:

  • National evaluations – on education system-level issues including sector performance, policy implementation and pre-tertiary educational practice
  • Education evaluations – scheduled external evaluation reviews carried out with schools and early learning services to complement and strengthen their own internal evaluation processes
  • Special reviews – carried out where a matter needed to be reviewed and reported outside regular reviews
  • New school assurance reviews – carried out to provide assurance to new school boards and their communities that the school has undertaken suitable administration processes and curriculum preparation
  • Private school reviews – carried out under section 35I and Part 28 of the Education Act 1989
  • Partnership school reviews – readiness reviews and subsequent assurance reviews under individual partnership school agreements
  • Homeschooling reviews – reviews of programmes for students exempt from enrolment at a registered school, undertaken in the context of section 21 and Part 28 of the Education Act 1989 and usually at the request of the Ministry of Education (the Ministry)
  • CoL | Kāhui Ako reports – as part of the Government’s Investing in Educational Success (IES) initiative. These included tailored reports for each community, bespoke reports for schools receiving the Principal Recruitment Allowance, and national reports on trends and issues
  • Teacher practising certificates audit – From 1 July 2015, ERO has been contracted by the Education Council of Aotearoa New Zealand (the Education Council), to undertake an audit of appraisals for the endorsement of teacher practising certificates.

ERO’s role in the education system

ERO is the New Zealand Government’s external education evaluation agency. Its evaluations are integrated with and strengthen the internal evaluation activities of schools, early learning services and CoL | Kāhui Ako. At an education system level, ERO undertakes national evaluations of education sector performance and policy.

ERO works alongside the Ministry and other agencies represented on the Education System Stewardship Forum, to achieve equitable levels of participation, engagement and achievement. There is a collective determination that the system should deliver equity and excellence – improving outcomes for all learners. ERO understands that to achieve increased levels of social, cultural and economic wellbeing across the country, the education system needs to be relevant and reach all children and young people.

ERO is developing the resources and capability to ensure that its extensive knowledge and expertise has an increasingly positive impact, particularly on the quality of early learning services, schools and CoL | Kāhui Ako to raise achievement for all.

ERO contributes to sector priorities through its influence on how schools and early childhood services perform. ERO identifies what works, establishes indicators for success, and uses this evidence to influence change in the education system.

ERO uses its knowledge about what works for children and young people (0-18 years) to influence decision making within the education system. ERO continues to model and support greater collaboration to raise the aspirations of children, parents and whānau. Features of ERO’s current evaluation approach include:

  • a specific focus on the outcomes for students not well served by the system
  • an integrated approach to external evaluation and internal evaluation
  • a participatory/collaborative approach to the evaluation process
  • a context specific approach to evaluation design, the gathering, analysis and synthesis of data, and the communication of findings
  • an emphasis on evaluation as a learning process, building knowledge at local, community and education system level.

ERO’s evaluations inform parents and families about the quality of educational provision in their communities. ERO’s engagement with Boards of Trustees, leaders and teachers is a vital aspect of its work. Through its evaluations, ERO strives to generate the conditions that enable ownership and improvement.

ERO’s impact on education outcomes is determined by its ability to work within and across the education system as an effective catalyst for change.

ERO’s progress in 2016/17

What ERO aimed to achieve

ERO’s Strategic Intentions 2016-2020 identified the following strategic goals and objectives for ERO:

Achieving equity and excellence

Goal 1: focusing on accelerating student achievement in evaluations

Goal 2: evolving our evaluation methodologies, approaches and programme

Generating improvement and innovation

Goal 3: supporting collaborative effort across the system

Goal 4: embedding the effective practice indicators

Goal 5: building internal evaluation practice across the system

Enhancing ERO’s capability and capacity

Goal 6: implementing its professional practice strategy

Goal 7: applying its operating model principles to all its work programmes

Goal 8: developing its corporate capability and supporting infrastructure

____________________________________________________________________________________

Reflecting its Strategic Intentions 2016-2020, ERO’s strategic goals and objectives for 2016/17 were as follows:

Goal 1: Focus on accelerating student achievement in evaluations

  • continue to evaluate and evolve its approach to accelerating student achievement in primary schools and extend its approach to intermediate and area schools
  • ensure its accelerating student achievement approach and external evaluations increasingly reinforce the importance of collaboration, student voice and whānau empowerment

Goal 2: Evolve ERO’s evaluation methodologies, approaches and programme

  • improve the accessibility of ERO’s findings, sharing examples of effective practice with parents and whānau
  • inform the profession about the support needed by beginning teachers to ensure their data literacy and readiness to accelerate student achievement
  • shed light on in-school variations and disparity in learner achievement and enable parents to identify what they need to become more active partners in their children’s learning
  • assess the progress to maturity of CoL |Kāhui Ako and other aspects of the Investing in Educational Success (IES) policies' impact on school leadership, teaching and student achievement
  • continue to focus on how Māori and English medium schools and early learning services promote success for those at risk of under achievement
  • better understand the impact of various interventions, and their effectiveness in improving performance and learning outcomes

Goal 3: Support collaborative effort across the system

  • publish effective practice case studies to reveal the contribution of enhanced learner pathways to improved outcomes
  • develop and trial a methodology to evaluate CoL | Kāhui Ako
  • develop and trial effective practice indicators to assist CoL | Kāhui Ako to build evaluation capability and capacity
  • continue to support developing CoL | Kāhui Ako with workshops to identify and agree achievement challenges

Goal 4: Embed the evaluation indicators

  • begin to revise and embed a comprehensive set of evaluation indicators for 0-18 years
  • begin to develop digital and online resources that illustrate aspects of the indicators and effective pedagogical (the science of teaching) practice
  • begin to position the indicators so as to influence the work of other agencies
  • continue to develop effective teaching and learning practice resources to make them more interactive and accessible
  • develop Māori medium versions of key resources and revise ERO’s approach to Māori medium in mainstream settings

Goal 5: Build internal evaluation practice across the system

  • develop resources that enable early learning services to undertake effective internal evaluation
  • initiate Ngā Kōhanga Reo Effective Practice Evaluation Indicators Project

Goal 6: Implement ERO’s professional practice strategy

  • clearly articulate capabilities for high quality education evaluation in New Zealand
  • further develop professional pathways, including access to post graduate qualifications in evaluation
  • provide opportunities for evaluators to engage in professional inquiry and research to build knowledge about effective evaluation practice

Goal 7: Apply ERO’s operating model principles to its work programme

  • develop the programme co-ordination role to strengthen programme and project management
  • deliver on the operating model principles for data, information and design
  • develop an information management strategy
  • update ERO’s Code of Conduct and related policies to ensure clarity, fairness and transparency
  • continue to evolve ERO’s performance measurement framework and ensure alignment with developments through the Education System Stewardship Work Programme

Goal 8: Develop ERO’s corporate capability and supporting infrastructure

  • complete a review and redevelopment of ERO’s approach to induction at national and regional levels
  • start to develop new plans and/or processes to exploit technology for greatest impact and grow the capability of the workforce
  • continued to develop ERO’s risk, assurance, security and privacy processes.

Progress against each of these goals and the related objectives in 2016/17 is detailed in the following sections.

ERO’s national evaluation programme

ERO’s national evaluation programme spotlights those areas of curriculum, pedagogy and school improvement that are most likely to influence equity and excellence in the New Zealand education system. ERO works with other education agencies to align its national evaluation programme with system priorities where necessary.

ERO’s national evaluations look at models of collaboration and coherence. Through its work, ERO develops insights and delivers evidence that improves government’s knowledge of the effects of its interventions and their impact in education and across the social sector.

ERO publishes exemplars of effective practice, so that parents and whānau are better informed and empowered to seek improvement and innovation. These exemplars support accelerated student achievement, promote effective teaching practice and focus on educationally powerful relationships.

The following reports were published during 2016/17.

Partners in Learning (August 2016)

Strong connections between schools and parents and whānau are essential to accelerating the achievement of students, particularly those at risk of underachieving. This booklet helps parents, families and whānau to form effective relationships and educationally powerful connections.

Partners in Learning sets out what parents can expect from their child’s school and, more importantly, how they can help their child do well at school. It describes what parents can do if they are concerned about their child’s learning and progress and what they can expect the school to do to help.

The booklet contains links to parent information on ERO, New Zealand Qualifications Authority (NZQA), the Ministry of Education and Careers New Zealand websites.

An evaluation of Stand Children's Services: Children's Villages (September 2016)

Since 2008, the Ministries of Education and Social Development have changed provision for children who have experienced trauma. Health camp schools were closed and the task of helping children that have experienced trauma was passed to Stand Children’s Services (Stand).

Stand is a charity funded by the Ministries of Education and Social Development to provide services for children aged 5 to 12 years who have experienced trauma. Children usually stay for about five weeks in one of seven children’s villages.

This review is the first since the service was reformed. It focused on how effectively the service responds to the wellbeing and learning of children that have experienced trauma. ERO visited each of the villages in Term 1, 2016 and reviewed the approaches and processes from the time a child is referred until they transition back to school.

Early Learning Curriculum (October 2016)

This retrospective study brought together findings from 17 national reports about curriculum implementation in early learning services, published over the past 10 years.

While the national evaluations over this time have not covered every aspect of early childhood curriculum, they provide insight into how effectively early learning services are designing, implementing and evaluating their curriculum, based on the early childhood curriculum Te Whāriki. The evaluations highlight pedagogical leadership, teacher knowledge and capacity to develop powerful learning partnerships as key factors impacting on quality.

School Leadership That Works (November 2016)

This evaluation report illustrated how the leadership domain within ERO’s School Evaluation Indicators works in practice by:

  • drawing together ERO’s findings from recent evaluative work with school leaders across the country
  • combining these findings with international evidence of what works in school leadership and why
  • showing the practices used by effective leaders to achieve their goals of equity and excellence in their schools.

The report was designed as a resource to guide the practice of school leadership. It is aimed at all school leaders, but particularly less experienced school leaders seeking to learn from the good practices of others.

Child Youth and Family Residential Schools (November 2016)

In 2016, ERO evaluated the quality of education at the Child Youth and Family (CYF) residential schools. CYF residential schools provide education for young people in Youth Justice and Care and Protection services.

Students in CYF residential schools are among the most vulnerable in New Zealand, and present with multiple high and complex needs. The specialised knowledge, skills and practice of teachers and leaders in residential schools needs to be of a very high standard to support student engagement and achievement.

ERO found respectful relationships between staff and students in all of the schools. However, as noted in previous ERO evaluations, exit transitions remained a weakness.

Effective Internal Evaluation for Improvement (December 2016)

This revised booklet is for organisations interested in internal evaluation. This overview of the processes and reasoning involved in effective internal evaluation for improvement draws on a recently published resource Effective Internal Evaluation – How to do and use internal evaluation for improvement (2015).

Extending their language - expanding their world: Children’s oral language (birth-8 years) (February 2017)

Research shows a child’s early life is critical in terms of the rapid language development that takes place, particularly in the first two to three years.

This evaluation investigated how effectively young children’s oral language learning and development were supported in their early years of education.

ERO asked early learning services and schools what they are doing in response to children’s oral language learning and development, including concerns about and needs of particular children. The findings were based on analysis of data gathered from 167 early learning services and 104 schools.

School trustees booklet: helping you ask the right questions (February 2017)

This booklet was written for boards of trustees to help board members. It focuses on student achievement and wellbeing, and how boards contribute to this. The booklet provides information to guide trustees and help them with their discussions with school leaders.

Food, Nutrition and Physical Activity in NZ Schools and Early Learning Services (April 2017)

In late 2015, the Cabinet Social Policy Committee asked ERO to assess ‘the current status of food, nutrition and physical activity in schools and early childhood services’ and to report on schools that were included in the sample. ERO visited 202 early learning services, 46 primary schools and 29 secondary schools in the first half of 2016.

ERO asked: How well does the service/school promote positive attitudes to physical activity, and food and nutrition to benefit children?

ERO found that most schools and services were doing a good job of equipping young people with the knowledge, skills and attitudes to make healthy choices around food, nutrition and physical activity.

In addition to the main report, a separate summary report provided an overview of ERO's findings and three case studies.

He Pou Tātaki: How ERO reviews hospital-based education and care services (June 2017)

This document was the result of a collaborative effort between ERO and representatives from hospital‑based education and care services. It supported improved service performance and accountability, and focussed on how well placed hospital‑based services contribute to children’s learning and promote their wellbeing.

Te Whāriki

At the end of the review period, ERO began planning for an iterative series of national evaluations about early learning services awareness and implementation of the updated Early Childhood Education Curriculum Te Whāriki.

Support for collaborative effort across the system

During 2016/17, ERO supported CoL | Kāhui Ako by developing integrated evaluative approaches that gather iterative information about how communities are developing.

ERO initiated the design of case study evaluation to generate insights into how communities are evolving and progressing in different contexts, and to explore the assumptions underpinning the CoL | Kāhui Ako policy design.

ERO published the following reports in January 2017:

Communities of Learning | Kāhui Ako: Collaboration to Improve Learner Outcomes

This publication supported CoL | Kāhui Ako by bringing together research findings about effective collaboration in education communities. It was supported by the publication, Communities of Learning | Kāhui Ako: Working Towards Collaborative Practice. These resources can be used in conjunction with the publications School Evaluation Indicators: Effective Practice for Improvement and Learner Success (2016) and Effective School Evaluation: How to do and use Internal Evaluation for Improvement (2016).

Communities of Learning | Kāhui Ako in Action

This was the first of a series of iterative reports which draw together what ERO knows about CoL | Kāhui Ako, as communities move from establishment to implementation. This report was based on information gathered from the following sources:

  • regular ERO evaluations of schools that are already members of a CoL | Kāhui Ako
  • workshops ERO has conducted with CoL | Kāhui Ako
  • in-depth work ERO is doing alongside one CoL | Kāhui Ako.

Information was collected in Terms 2 and 3 of 2016 from 82 schools representing 40 percent of the current 148 CoL | Kāhui Ako and from 20 workshops conducted nationally during 2015 and 2016.

Communities of Learning | Kāhui Ako: Working Towards Collaborative Practice

This resource supports CoL | Kāhui Ako as communities work towards effective collaborative practice. It is framed around key questions in each of the seven effective practice areas and can be used both as evidence-based progressions and as a useful internal evaluation tool.

In June 2017, ERO and the Ministry agreed to an overarching CoL | Kāhui Ako evaluation strategy. The strategy reflects a ‘theory of change’ that captures how policy settings can be expected to contribute to desired government outcomes, and identifies a high-level evaluation framework to guide the evaluation activities of each agency.

ERO’s contribution to the evaluation strategy work programme will include integrating investigations of schools’ and early learning services’ participation in CoL | Kāhui Ako into all reviews. It is beginning to cluster reviews of schools and services in a community, so that the progress of each community can be monitored more effectively by ERO. Information gathered from ERO evaluations and case studies will continue to contribute in the following ways:

  • by developing the knowledge base about what works collaboratively in the New Zealand education setting
  • by tracking progress towards the desired outcomes to inform learning and decision-making
  • by ensuring the government receives timely and evidence-based advice about the implementation of the IES policy.

During the year, ERO also agreed on a joint strategy with the Ministry for working with persistently poorly performing schools. ERO has set up a national team to undertake diagnostic reviews and to evaluate schools’ action plans.

ERO approach to school turnaround

ERO and the Ministry are developing new protocols, systems and processes for working together to ensure that targeted interventions and support are provided when schools are having difficulty.

In 2011, ERO developed a differentiated review methodology according to each school’s context, performance and evaluation capacity. This included a longitudinal review methodology for schools experiencing difficulties which were considered unlikely to be able to resolve their difficulties and raise student achievement without external intervention, support and ERO monitoring.

Since then, it has become apparent that, despite ERO’s best efforts, a small number of schools continue to perform poorly over long periods. Using the 2016 ERO School Evaluation Indicators, these schools demonstrate persistent poor performance in five key areas:

  • student achievement
  • other valued student outcomes - student engagement and wellbeing
  • the quality of teaching and curriculum design
  • professional leadership
  • stewardship - notably the cycle of planning, reporting and resourcing, and health and safety.

ERO is piloting a more intensive “School Turnaround” evaluation process to help ensure these schools improve their systems and processes to improve student achievement and school performance. ERO has worked closely with the Ministry to design this process which includes a diagnostic review focusing on school conditions most likely to support improvement in teaching and learning. With an update to the Education Act 1989 and the range of new interventions, the aim is to provide a collaborative education system response to support improvement and accelerate school turnaround.

Embedding the evaluation indicators

In July 2016, ERO published its final School Evaluation Indicators report. The content had been refined based on feedback from the education sector and ERO’s evaluators.

The new School Evaluation Indicators document is designed to focus schools and ERO evaluators on the things that matter most in improving student outcomes. The framework identifies two types of indicator: outcome and process.

The outcome indicators are drawn from The New Zealand Curriculum and Te Marautanga o Aotearoa. They can be used to assess the impact of school policies and actions. Indicators of student achievement and progress are a direct measure of what it is that schools are expected to achieve. The measures related to students’ confidence in their identity, language and culture, and to wellbeing, participation and contribution. All these factors are important in their own right, as well as being essential for achievement and progress.

The process indicators describe practices and processes that contribute to school effectiveness and improvement. They are organised in six key domains that work together to promote equity and excellence in student outcomes. They assist schools to identify areas in which changes are needed.

Where evaluation against the outcome indicators indicates poor performance, the process indicators can be used as a tool for investigating what’s contributing to this poor performance.

During 2016/17, ERO developed resources to assist schools, kura, early learning services and Kōhanga reo to explore opportunities to deliver curriculum that enables children to realise their full potential with the skills they need for study, work and lifelong learning.

In 2016, ERO initiated a film project in selected schools to produce videos that illustrate the School Evaluation Indicators. This resource, called Improvement in Action / Te Ahu Whakamua, was published on ERO’s website to support the development of understanding and capability in relation to improving student outcomes.

ERO also started to develop Māori medium versions of key resources and took initial steps to revise its approach to Māori medium education in mainstream settings.   

The evaluation indicators and supporting materials will evolve and change over time in the light of new research and evaluation findings.

Appraisal as a catalyst for improved learner outcomes (December 2016)

The Education Council of Aotearoa New Zealand (the Council) was established on 1 July 2015 to replace the New Zealand Teachers Council. One of 16 functions outlined in the enabling legislation was the requirement to audit and moderate the appraisals of at least 10 percent of practising certificates issued or renewed each year. This focus was to ensure that appraisals used for teacher certification achieve a “reasonable and consistent standard”. On the first day of its operation, ERO signed an agreement with the Council to undertake this work on behalf of the new Council.

In 2016/17, ERO successfully completed the contracted number of audits and reported back to the Council.

Build internal evaluation practice across the system

In November 2015, ERO and the Ministry released a trial internal evaluation resource for schools. The resource is designed to assist schools make the most of internal evaluation. ERO has integrated the use of this internal evaluation resource into the follow up workshops that are provided as part of schools’ evaluations.

By embedding effective internal evaluation practice across the system, ERO hopes to influence the development of teaching practice and curriculum design for improved learner outcomes.

During 2016/17, ERO, in partnership with the National Te Kōhanga Reo Trust, completed a review and report about effective practice in Ngā Kōhanga Reo. ERO also established a negotiated project plan for the development of Ngā Kōhanga Reo Effective Practice evaluation indicators.

ERO’s Pacific strategy

ERO’s Pacific Strategy 2012/2017 was evaluated in 2015. As a result, some Pacific communities have shared their experiences and provided feedback about ERO’s review process and reports. This feedback has contributed to a wider collation of information from communities about new and innovative ways to engage with Pacific families.

ERO’s Pacific Strategy review work continued in 2016/17 and will help to inform ERO’s future work with and for Pacific communities. ERO is committed to increasing the involvement and understanding of Pacific parents and whānau in their children’s learning. This will improve outcomes for all Pacific learners and, in particular, those most at risk of not achieving their potential.

ERO continues to strive for educational success for all Pacific learners through high quality evaluation practice. Ongoing capacity and capability building for staff in issues of greatest relevance to Pacific people is an important part of ERO’s professional learning and development efforts.

Implement ERO’s professional practice strategy

Quality evaluation is ERO’s lifeblood. ERO is developing a profession of highly skilled evaluators, fully equipped for the challenges of the role in the 21st century. This means ERO’s review officers will have the right mix of knowledge and skills, work in a highly supportive environment, and have access to modern tools and resources.

ERO’s professional practice strategy provides a framework for a multi-faceted approach to developing leadership and evaluation practice in the field.

The development of professional pathways has continued, with an increase in the number of evaluators participating in post graduate qualifications in evaluation at the University of Melbourne.

The National Professional Practice Forum provided an opportunity for reviewers and evaluators to engage with key international and national leaders in the fields of education and evaluation.

ERO reviewers also attended workshops focused on key domains of the School Evaluation Indicators at ERO’s Professional Forum. These workshops were led by the academics involved in the development process.

In 2016/17, ERO published the report Capabilities for High Quality Education Evaluation in Aotearoa New Zealand. The capabilities framework defines the capabilities that are needed to undertake and deliver high quality education evaluations.

ERO also completed the draft report, Principles of Practice for High Quality External Education Evaluation in Aotearoa New Zealand. The principles of practice complete the suite of publications designed to promote and support improved professional practice in ERO.

During the year, ERO’s professional practice leaders participated in international and national evaluation institutes focused on evaluation methodology and practice.

ERO’s evaluation expertise continues to be recognised through an increased demand for presentations in international and national contexts.

ERO’s corporate capability and supporting infrastructure

ERO continues to develop its corporate capability and infrastructure. ERO aims to apply its operating model principles in managing change projects. ERO’s infrastructure needs to support high quality professional evaluation practice but, as a relatively small government department, it looks to leverage off the assets of other education agencies where possible.

During the review period, ERO:

  • developed its programme co-ordination role
  • commenced development of a model to improve the use of available data for scheduling
  • reframed ERO’s ICT management approach focusing on collaboration with other agencies
  • continued to evolve its performance measurement framework with input from the Treasury
  • reviewed the ERO Code of Conduct
  • conducted another staff survey, using the Gallup Q12 methodology to measure employee engagement.

ERO’s Executive Leadership Team is responsible for identifying, assessing and managing key strategic and operational risks by reference to an organisational risk management framework.

In 2016/17, ERO’s risk and assurance framework continued to evolve to reflect changes in ERO’s organisational strategy. The integration of national quality assurance findings into the national professional learning and development programme is contributing to the quality and consistency of evaluation practice. ERO intends to continue this programme of regular internal reviews to identify and guide further improvements.

ERO remains committed to integrating equality and diversity into all aspects of its business. The Te Uepū ā-Motu hui (for all staff of Māori descent) and Fono Pacifika forum (for staff of Pacific descent) are examples of ERO’s commitment to equal employment opportunities.

In line with the Public Service Equal Employment Opportunities Policy, ERO strives to:

  • treat people fairly and with respect, ensuring equality of access to opportunities (equality)
  • understand, appreciate and realise the benefits of individual differences (diversity).

ERO remains committed to equality and diversity in all aspects of its business. Its gender pay gap in 2017 was 9.7 percent. Just under 78 percent of our staff are female. More than 20 percent of ERO’s workforce identify as Māori.

ERO is subject to Cabinet Circular (16) 3 issued on 25 May 2016, which sets out requirements for the implementation of the New Zealand Business Number (NZBN) by the end of December 2018. ERO is well advanced in implementing these requirements. ERO has completed a review of its business-facing systems, processes and forms to incorporate NZBN. As a result of this review, finance and other systems have been updated to recognise NZBN. Also ERO’s website and forms such as letterheads have been amended to show ERO’s NZBN.

Relationship with the Public Service Association

In 2016/17, ERO completed negotiations for a new collective employment agreement, expiring in June 2019.

ERO and the PSA continue to meet regularly at national and local levels as part of the working relationship agreement. Both parties find these meetings useful.