ERO’s evaluation insights are a catalyst for change so that every child achieves success as a lifelong learner.
ERO is a government department established in October 1989 under the State Sector Act 1988. The Chief Executive of ERO is the Chief Review Officer.
Under Part 28 of the Education Act 1989, ERO is required to review the performance of pre-tertiary education providers in relation to the educational services they provide. Review Officers are statutory officers designated under the Act. They exercise powers of entry, investigation and reporting.
At an education system level, ERO carries out evaluations of education sector performance and policy implementation, and reports to the Minister of Education about practice in the pre-tertiary sector.
ERO’s core activities in 2016/17 included:
ERO is the New Zealand Government’s external education evaluation agency. Its evaluations are integrated with and strengthen the internal evaluation activities of schools, early learning services and CoL | Kāhui Ako. At an education system level, ERO undertakes national evaluations of education sector performance and policy.
ERO works alongside the Ministry and other agencies represented on the Education System Stewardship Forum, to achieve equitable levels of participation, engagement and achievement. There is a collective determination that the system should deliver equity and excellence – improving outcomes for all learners. ERO understands that to achieve increased levels of social, cultural and economic wellbeing across the country, the education system needs to be relevant and reach all children and young people.
ERO is developing the resources and capability to ensure that its extensive knowledge and expertise has an increasingly positive impact, particularly on the quality of early learning services, schools and CoL | Kāhui Ako to raise achievement for all.
ERO contributes to sector priorities through its influence on how schools and early childhood services perform. ERO identifies what works, establishes indicators for success, and uses this evidence to influence change in the education system.
ERO uses its knowledge about what works for children and young people (0-18 years) to influence decision making within the education system. ERO continues to model and support greater collaboration to raise the aspirations of children, parents and whānau. Features of ERO’s current evaluation approach include:
ERO’s evaluations inform parents and families about the quality of educational provision in their communities. ERO’s engagement with Boards of Trustees, leaders and teachers is a vital aspect of its work. Through its evaluations, ERO strives to generate the conditions that enable ownership and improvement.
ERO’s impact on education outcomes is determined by its ability to work within and across the education system as an effective catalyst for change.
ERO’s Strategic Intentions 2016-2020 identified the following strategic goals and objectives for ERO:
Goal 1: focusing on accelerating student achievement in evaluations
Goal 2: evolving our evaluation methodologies, approaches and programme
Goal 3: supporting collaborative effort across the system
Goal 4: embedding the effective practice indicators
Goal 5: building internal evaluation practice across the system
Goal 6: implementing its professional practice strategy
Goal 7: applying its operating model principles to all its work programmes
Goal 8: developing its corporate capability and supporting infrastructure
Reflecting its Strategic Intentions 2016-2020, ERO’s strategic goals and objectives for 2016/17 were as follows:
Progress against each of these goals and the related objectives in 2016/17 is detailed in the following sections.
ERO’s national evaluation programme spotlights those areas of curriculum, pedagogy and school improvement that are most likely to influence equity and excellence in the New Zealand education system. ERO works with other education agencies to align its national evaluation programme with system priorities where necessary.
ERO’s national evaluations look at models of collaboration and coherence. Through its work, ERO develops insights and delivers evidence that improves government’s knowledge of the effects of its interventions and their impact in education and across the social sector.
ERO publishes exemplars of effective practice, so that parents and whānau are better informed and empowered to seek improvement and innovation. These exemplars support accelerated student achievement, promote effective teaching practice and focus on educationally powerful relationships.
The following reports were published during 2016/17.
Partners in Learning (August 2016)
Strong connections between schools and parents and whānau are essential to accelerating the achievement of students, particularly those at risk of underachieving. This booklet helps parents, families and whānau to form effective relationships and educationally powerful connections.
Partners in Learning sets out what parents can expect from their child’s school and, more importantly, how they can help their child do well at school. It describes what parents can do if they are concerned about their child’s learning and progress and what they can expect the school to do to help.
The booklet contains links to parent information on ERO, New Zealand Qualifications Authority (NZQA), the Ministry of Education and Careers New Zealand websites.
An evaluation of Stand Children's Services: Children's Villages (September 2016)
Since 2008, the Ministries of Education and Social Development have changed provision for children who have experienced trauma. Health camp schools were closed and the task of helping children that have experienced trauma was passed to Stand Children’s Services (Stand).
Stand is a charity funded by the Ministries of Education and Social Development to provide services for children aged 5 to 12 years who have experienced trauma. Children usually stay for about five weeks in one of seven children’s villages.
This review is the first since the service was reformed. It focused on how effectively the service responds to the wellbeing and learning of children that have experienced trauma. ERO visited each of the villages in Term 1, 2016 and reviewed the approaches and processes from the time a child is referred until they transition back to school.
Early Learning Curriculum (October 2016)
This retrospective study brought together findings from 17 national reports about curriculum implementation in early learning services, published over the past 10 years.
While the national evaluations over this time have not covered every aspect of early childhood curriculum, they provide insight into how effectively early learning services are designing, implementing and evaluating their curriculum, based on the early childhood curriculum Te Whāriki. The evaluations highlight pedagogical leadership, teacher knowledge and capacity to develop powerful learning partnerships as key factors impacting on quality.
School Leadership That Works (November 2016)
This evaluation report illustrated how the leadership domain within ERO’s School Evaluation Indicators works in practice by:
The report was designed as a resource to guide the practice of school leadership. It is aimed at all school leaders, but particularly less experienced school leaders seeking to learn from the good practices of others.
Child Youth and Family Residential Schools (November 2016)
In 2016, ERO evaluated the quality of education at the Child Youth and Family (CYF) residential schools. CYF residential schools provide education for young people in Youth Justice and Care and Protection services.
Students in CYF residential schools are among the most vulnerable in New Zealand, and present with multiple high and complex needs. The specialised knowledge, skills and practice of teachers and leaders in residential schools needs to be of a very high standard to support student engagement and achievement.
ERO found respectful relationships between staff and students in all of the schools. However, as noted in previous ERO evaluations, exit transitions remained a weakness.
Effective Internal Evaluation for Improvement (December 2016)
This revised booklet is for organisations interested in internal evaluation. This overview of the processes and reasoning involved in effective internal evaluation for improvement draws on a recently published resource Effective Internal Evaluation – How to do and use internal evaluation for improvement (2015).
Research shows a child’s early life is critical in terms of the rapid language development that takes place, particularly in the first two to three years.
This evaluation investigated how effectively young children’s oral language learning and development were supported in their early years of education.
ERO asked early learning services and schools what they are doing in response to children’s oral language learning and development, including concerns about and needs of particular children. The findings were based on analysis of data gathered from 167 early learning services and 104 schools.
School trustees booklet: helping you ask the right questions (February 2017)
This booklet was written for boards of trustees to help board members. It focuses on student achievement and wellbeing, and how boards contribute to this. The booklet provides information to guide trustees and help them with their discussions with school leaders.
In late 2015, the Cabinet Social Policy Committee asked ERO to assess ‘the current status of food, nutrition and physical activity in schools and early childhood services’ and to report on schools that were included in the sample. ERO visited 202 early learning services, 46 primary schools and 29 secondary schools in the first half of 2016.
ERO asked: How well does the service/school promote positive attitudes to physical activity, and food and nutrition to benefit children?
ERO found that most schools and services were doing a good job of equipping young people with the knowledge, skills and attitudes to make healthy choices around food, nutrition and physical activity.
In addition to the main report, a separate summary report provided an overview of ERO's findings and three case studies.
This document was the result of a collaborative effort between ERO and representatives from hospital‑based education and care services. It supported improved service performance and accountability, and focussed on how well placed hospital‑based services contribute to children’s learning and promote their wellbeing.
At the end of the review period, ERO began planning for an iterative series of national evaluations about early learning services awareness and implementation of the updated Early Childhood Education Curriculum Te Whāriki.
During 2016/17, ERO supported CoL | Kāhui Ako by developing integrated evaluative approaches that gather iterative information about how communities are developing.
ERO initiated the design of case study evaluation to generate insights into how communities are evolving and progressing in different contexts, and to explore the assumptions underpinning the CoL | Kāhui Ako policy design.
ERO published the following reports in January 2017:
This publication supported CoL | Kāhui Ako by bringing together research findings about effective collaboration in education communities. It was supported by the publication, Communities of Learning | Kāhui Ako: Working Towards Collaborative Practice. These resources can be used in conjunction with the publications School Evaluation Indicators: Effective Practice for Improvement and Learner Success (2016) and Effective School Evaluation: How to do and use Internal Evaluation for Improvement (2016).
This was the first of a series of iterative reports which draw together what ERO knows about CoL | Kāhui Ako, as communities move from establishment to implementation. This report was based on information gathered from the following sources:
Information was collected in Terms 2 and 3 of 2016 from 82 schools representing 40 percent of the current 148 CoL | Kāhui Ako and from 20 workshops conducted nationally during 2015 and 2016.
This resource supports CoL | Kāhui Ako as communities work towards effective collaborative practice. It is framed around key questions in each of the seven effective practice areas and can be used both as evidence-based progressions and as a useful internal evaluation tool.
In June 2017, ERO and the Ministry agreed to an overarching CoL | Kāhui Ako evaluation strategy. The strategy reflects a ‘theory of change’ that captures how policy settings can be expected to contribute to desired government outcomes, and identifies a high-level evaluation framework to guide the evaluation activities of each agency.
ERO’s contribution to the evaluation strategy work programme will include integrating investigations of schools’ and early learning services’ participation in CoL | Kāhui Ako into all reviews. It is beginning to cluster reviews of schools and services in a community, so that the progress of each community can be monitored more effectively by ERO. Information gathered from ERO evaluations and case studies will continue to contribute in the following ways:
During the year, ERO also agreed on a joint strategy with the Ministry for working with persistently poorly performing schools. ERO has set up a national team to undertake diagnostic reviews and to evaluate schools’ action plans.
ERO and the Ministry are developing new protocols, systems and processes for working together to ensure that targeted interventions and support are provided when schools are having difficulty.
In 2011, ERO developed a differentiated review methodology according to each school’s context, performance and evaluation capacity. This included a longitudinal review methodology for schools experiencing difficulties which were considered unlikely to be able to resolve their difficulties and raise student achievement without external intervention, support and ERO monitoring.
Since then, it has become apparent that, despite ERO’s best efforts, a small number of schools continue to perform poorly over long periods. Using the 2016 ERO School Evaluation Indicators, these schools demonstrate persistent poor performance in five key areas:
ERO is piloting a more intensive “School Turnaround” evaluation process to help ensure these schools improve their systems and processes to improve student achievement and school performance. ERO has worked closely with the Ministry to design this process which includes a diagnostic review focusing on school conditions most likely to support improvement in teaching and learning. With an update to the Education Act 1989 and the range of new interventions, the aim is to provide a collaborative education system response to support improvement and accelerate school turnaround.
In July 2016, ERO published its final School Evaluation Indicators report. The content had been refined based on feedback from the education sector and ERO’s evaluators.
The new School Evaluation Indicators document is designed to focus schools and ERO evaluators on the things that matter most in improving student outcomes. The framework identifies two types of indicator: outcome and process.
The outcome indicators are drawn from The New Zealand Curriculum and Te Marautanga o Aotearoa. They can be used to assess the impact of school policies and actions. Indicators of student achievement and progress are a direct measure of what it is that schools are expected to achieve. The measures related to students’ confidence in their identity, language and culture, and to wellbeing, participation and contribution. All these factors are important in their own right, as well as being essential for achievement and progress.
The process indicators describe practices and processes that contribute to school effectiveness and improvement. They are organised in six key domains that work together to promote equity and excellence in student outcomes. They assist schools to identify areas in which changes are needed.
Where evaluation against the outcome indicators indicates poor performance, the process indicators can be used as a tool for investigating what’s contributing to this poor performance.
During 2016/17, ERO developed resources to assist schools, kura, early learning services and Kōhanga reo to explore opportunities to deliver curriculum that enables children to realise their full potential with the skills they need for study, work and lifelong learning.
In 2016, ERO initiated a film project in selected schools to produce videos that illustrate the School Evaluation Indicators. This resource, called Improvement in Action / Te Ahu Whakamua, was published on ERO’s website to support the development of understanding and capability in relation to improving student outcomes.
ERO also started to develop Māori medium versions of key resources and took initial steps to revise its approach to Māori medium education in mainstream settings.
The evaluation indicators and supporting materials will evolve and change over time in the light of new research and evaluation findings.
Appraisal as a catalyst for improved learner outcomes (December 2016)
The Education Council of Aotearoa New Zealand (the Council) was established on 1 July 2015 to replace the New Zealand Teachers Council. One of 16 functions outlined in the enabling legislation was the requirement to audit and moderate the appraisals of at least 10 percent of practising certificates issued or renewed each year. This focus was to ensure that appraisals used for teacher certification achieve a “reasonable and consistent standard”. On the first day of its operation, ERO signed an agreement with the Council to undertake this work on behalf of the new Council.
In 2016/17, ERO successfully completed the contracted number of audits and reported back to the Council.
In November 2015, ERO and the Ministry released a trial internal evaluation resource for schools. The resource is designed to assist schools make the most of internal evaluation. ERO has integrated the use of this internal evaluation resource into the follow up workshops that are provided as part of schools’ evaluations.
By embedding effective internal evaluation practice across the system, ERO hopes to influence the development of teaching practice and curriculum design for improved learner outcomes.
During 2016/17, ERO, in partnership with the National Te Kōhanga Reo Trust, completed a review and report about effective practice in Ngā Kōhanga Reo. ERO also established a negotiated project plan for the development of Ngā Kōhanga Reo Effective Practice evaluation indicators.
ERO’s Pacific Strategy 2012/2017 was evaluated in 2015. As a result, some Pacific communities have shared their experiences and provided feedback about ERO’s review process and reports. This feedback has contributed to a wider collation of information from communities about new and innovative ways to engage with Pacific families.
ERO’s Pacific Strategy review work continued in 2016/17 and will help to inform ERO’s future work with and for Pacific communities. ERO is committed to increasing the involvement and understanding of Pacific parents and whānau in their children’s learning. This will improve outcomes for all Pacific learners and, in particular, those most at risk of not achieving their potential.
ERO continues to strive for educational success for all Pacific learners through high quality evaluation practice. Ongoing capacity and capability building for staff in issues of greatest relevance to Pacific people is an important part of ERO’s professional learning and development efforts.
Quality evaluation is ERO’s lifeblood. ERO is developing a profession of highly skilled evaluators, fully equipped for the challenges of the role in the 21st century. This means ERO’s review officers will have the right mix of knowledge and skills, work in a highly supportive environment, and have access to modern tools and resources.
ERO’s professional practice strategy provides a framework for a multi-faceted approach to developing leadership and evaluation practice in the field.
The development of professional pathways has continued, with an increase in the number of evaluators participating in post graduate qualifications in evaluation at the University of Melbourne.
The National Professional Practice Forum provided an opportunity for reviewers and evaluators to engage with key international and national leaders in the fields of education and evaluation.
ERO reviewers also attended workshops focused on key domains of the School Evaluation Indicators at ERO’s Professional Forum. These workshops were led by the academics involved in the development process.
In 2016/17, ERO published the report Capabilities for High Quality Education Evaluation in Aotearoa New Zealand. The capabilities framework defines the capabilities that are needed to undertake and deliver high quality education evaluations.
ERO also completed the draft report, Principles of Practice for High Quality External Education Evaluation in Aotearoa New Zealand. The principles of practice complete the suite of publications designed to promote and support improved professional practice in ERO.
During the year, ERO’s professional practice leaders participated in international and national evaluation institutes focused on evaluation methodology and practice.
ERO’s evaluation expertise continues to be recognised through an increased demand for presentations in international and national contexts.
ERO continues to develop its corporate capability and infrastructure. ERO aims to apply its operating model principles in managing change projects. ERO’s infrastructure needs to support high quality professional evaluation practice but, as a relatively small government department, it looks to leverage off the assets of other education agencies where possible.
During the review period, ERO:
ERO’s Executive Leadership Team is responsible for identifying, assessing and managing key strategic and operational risks by reference to an organisational risk management framework.
In 2016/17, ERO’s risk and assurance framework continued to evolve to reflect changes in ERO’s organisational strategy. The integration of national quality assurance findings into the national professional learning and development programme is contributing to the quality and consistency of evaluation practice. ERO intends to continue this programme of regular internal reviews to identify and guide further improvements.
ERO remains committed to integrating equality and diversity into all aspects of its business. The Te Uepū ā-Motu hui (for all staff of Māori descent) and Fono Pacifika forum (for staff of Pacific descent) are examples of ERO’s commitment to equal employment opportunities.
In line with the Public Service Equal Employment Opportunities Policy, ERO strives to:
ERO remains committed to equality and diversity in all aspects of its business. Its gender pay gap in 2017 was 9.7 percent. Just under 78 percent of our staff are female. More than 20 percent of ERO’s workforce identify as Māori.
ERO is subject to Cabinet Circular (16) 3 issued on 25 May 2016, which sets out requirements for the implementation of the New Zealand Business Number (NZBN) by the end of December 2018. ERO is well advanced in implementing these requirements. ERO has completed a review of its business-facing systems, processes and forms to incorporate NZBN. As a result of this review, finance and other systems have been updated to recognise NZBN. Also ERO’s website and forms such as letterheads have been amended to show ERO’s NZBN.
In 2016/17, ERO completed negotiations for a new collective employment agreement, expiring in June 2019.
ERO and the PSA continue to meet regularly at national and local levels as part of the working relationship agreement. Both parties find these meetings useful.