2014 briefing to the incoming Minister

Part one – making a difference to student achievement

The challenge

For the last 25 years the Education Review Office (ERO) has reviewed schools and early childhood services. ERO’s first National Report was published in 1994. Our review methodologies are world renowned and ERO’s people are highly regarded by schools and services. ERO’s reviews and evaluations are a catalyst for change. We can be proud of the work that this small and highly capable organisation does and the impact that it has.

ERO knows that schools and services with high levels of achievement, regardless of any other variable, have leaders who are committed to continuous improvement; are focused on the needs of the learner; and engage effectively with family and whānau.

ERO has been reporting this for more than two decades. However, it remains that too many children and young people do not achieve well enough to have a range of future options. This is the challenge facing the education system. While there is no simple response, if we continue to do the same things, we will deliver the same results.

The Ministry of Education’s system briefing (“Aspiration and Achievement: the Education System Briefing to the Incoming Ministers”) sets out a new approach for the sector to work together as stewards of the system to bring about positive change. ERO endorses the Ministry’s approach and will work alongside them to achieve the ambition it sets out. In particular the system needs to perform differently to close the achievement gap and improve the outcomes for all children and young people. There is also an opportunity for ERO to do things differently.

ERO’s role

Under Part 28 of the Education Act 1989, ERO is required to administer reviews of general or particular matters about the performance of pre-tertiary education providers in relation to the educational services they provide, under its own motion, or at the direction of the Minister of Education.

The Act does not prescribe how these obligations may be met. Currently, each year, we undertake approximately 1,300-1,400 reviews of early childhood services, 710-840 reviews of schools, and up to 20 national evaluation reports including reports of good practice. ERO also undertakes reviews of home-based services and reviews of private schools. These volumes, assuming current review processes, are fiscally unsustainable after 2015/16 1. Current and future demands require more efficient and effective use of our resources in order to influence the system’s response to those children and young people who remain at risk of underachieving.

ERO’s role is to gain deep insight into what works to support achievement and to use that evidence to influence change in individual parts of the system, as well as the system as a whole. We need to know how we can best inform the improvements needed to better respond to children and young people for whom the system has consistently failed. We know that the cost to the wider economic, social and justice system of this group of young people is huge. We can be confident that investing more of our effort in this group will be a net gain for New Zealand.We have a number of strategically important initiatives underway that have the potential to strengthen system capability to raise overall performance and to focus on those who need the most support. We are also considering some key opportunities that we would like to discuss with you over the next few months.

Current strategic initiatives to improve the performance of the system

  • Reviewing our evaluation indicators for school reviews
  • Improving the way that self review is undertaken across the system
  • Initiatives to improve the quality of engagement with parents, family and whānau in their children’s learning
  • Implementing a new methodology for home-based education and care services in the early childhood sector
  • Supporting improvements to quality in early childhood services
  • Developing a new methodology for kura ā iwi (and indicators for kura and wharekura)
  • Using evaluation to support the Investing in Educational Success initiative

Strategic opportunities

We are reviewing our strategic direction, taking into account the current and future environment. For example, the availability of ever improving data from the Ministry of Education provides the opportunity to more clearly identify where the greatest need for intervention lies. This information enables ERO to be more strategic in its approach. In this context we will consider how we review schools and services, which ones we review, and how often. We will need to respond to the growth of early childhood education and care providers. We will also consider the scope and scale of our national reports to ensure we focus on those areas that need the greatest attention and in a way that provides the most robust information to effect change.

Current Initiatives

Review of ERO’s Evaluation Indicators for School Reviews

To promote improvement, ERO and the sector must be kept informed about what influences learner success. ERO uses Evaluation Indicators to inform the judgements that our review officers make when reviewing schools and early childhood services. To inform the judgements we make, the indicators identify what matters most in improving student achievement, wellbeing and progress. By clearly outlining the key domains of influence on school effectiveness and improvement, ERO can make the review process transparent and provide a tool to assist schools and services to build their own capability to improve outcomes. The Evaluation Indicators review responds to the recent recommendations made by the Organisation for Economic Co-operation and Development (OECD)2 that ERO should build on the indicator framework to consolidate a common understanding of quality.

Because the indicators are intended to drive action and behaviour, it is essential that they are selected on the basis of strong evidence of what schools need to do to perform well for children. The project is therefore being supported by a panel of leading academics3 who have prepared papers on different aspects of the framework and participated in a think tank. The academics recommended that:

  • students, especially those for whom the system has not performed well, need to be at the centre of the organising framework
  • the organising framework needs to be strongly aligned to the Best Evidence Synthesis
  • prominence should be given to the importance of effective relationships with parents focused on their children’s learning
  • the number of indicators should be reduced, signalling the key elements of school effectiveness and improvement and showing how these work together in a connected way

Aside from being an incredibly important tool for ERO and schools to review their own performance, the indicators should inform those responsible for recruitment, initial teacher education, performance appraisal, and professional development for teachers. We will work with the Ministry of Education to ensure that the evaluation indicators inform policy and implementation.

The review will also represent an opportunity to simplify the number of indicators, align quality descriptors across the school sector, and develop a more coherent and comprehensive understanding of the behaviours that schools must exhibit so that all children experience success.

A revised draft set of indicators will be available for sector consultation early in 2015. A new set of indicators should be agreed by mid 2015.

Self review

Research shows that education improvement is best sustained through effective self-review, inquiry and knowledge-building processes. Effective self review enables schools and services to identify the strengths and weaknesses of current performance and what they need to do better to improve the achievement of their learners.

The OECD review identified that further work is now needed to promote the effectiveness in the way that self review is used by schools for improvement. In particular, the review recommended that ERO and the Ministry of Education identify how self review can capture the breadth of the curriculum and enhance the support structure around schools to support their self review.

ERO is leading a joint project which has two key parts. The first is the development of a new resource focused on effective school self review in conjunction with the Ministry of Education. The second is the development of school case studies that exemplify effective practice and its relationship to improved outcomes. The second aspect of this work is as important as the first. The greater the confidence we have in the quality of self review the more flexible ERO can be in its distribution of resources.

The work also represents an excellent opportunity to identify how school trustees, leaders and teachers, and communities of schools could benefit from the use of self review to improve the progress of our learners.

ERO will report progress to you in 2015.

Increasing the engagement of parents, family and whānau

The Evaluation Indicators project highlights the critical involvement of parents, family and whānau in student achievement. Through our review processes ERO will focus on how effectively schools are developing relationships with families, in particular those families who have been traditionally less engaged in education. We will identify and share successful strategies and good practice. To support this process, project work will aim to improve our engagement with parents, family and whānau, and better integrate their voice into our reports and feedback to schools. So that parents, family and whānau can better support the learning process, we will also provide useful and clear information that helps them navigate the education system and voice their aspirations.

We also recognise that parents, families and whānau are not homogenous. Our review processes will need to respond to Māori, and take into account the differences and diversity in our society.

Quality improvements in Early Childhood Services

The early childhood sector has experienced significant growth in the demand for early childhood services (total enrolments have increased by 13% between 2008 and 2013), reflecting recent priorities for increased participation. This has led to an increase in the number of licensed services, and equally, a demand for higher quality education and care (there were 4,255 licensed services in 2013 compared with 3,881 in 2008, an increase of 9.5%).

ERO’s work in the early childhood sector has highlighted considerable variability in the quality of curriculum implementation, assessment and evaluation in early childhood services. Variable quality is particularly notable in those areas with high numbers of Māori and Pacific children enrolling. Many services are yet to focus their curriculum on meaningful and relevant outcomes for children’s learning.

As far back as 2007 ERO reported that good assessment practice was underpinned by two factors: teachers’ understanding of the early childhood curriculum, Te Whāriki, and the alignment between the focus of their assessment of children’s learning and their service’s philosophy. ERO is concerned that assessment and a focus on valued outcomes remains an area for improvement in many services, as evidenced by our recent national evaluations. ERO has discussed these concerns with the Ministry of Education and is involved in meetings with sector groups about the way forward. We are also working with the OECD Early Childhood and Care network contributing to international work on monitoring quality, outcomes and transitions in early childhood education.

ERO has recently investigated how well services were supporting children as they approach transition to school. This report will be available to you before the end of 2014. This evaluation highlights the need for greater understanding in the early childhood sector of curriculum (both Te Whāriki and The New Zealand Curriculum) and the need to bridge children’s learning as they transition between these two settings. We also confirm the importance of schools and services working together to support young children and their families when they are transitioning to school.

New methodology for home-based education and care services in the early childhood sector

An area of rapid sector growth is that of home-based early childhood services (there were 348 home based services in 2013 compared with 244 in 2008, an increase of 42%) To support improved performance and accountability in these services, ERO has recently developed and published a draft revised methodology for reviewing home-based early childhood education and care services. The new methodology reflects a collaborative effort between ERO, representatives from the home-based early childhood sector, and researchers with an interest in home-based education and care.

The revised methodology is being trialled in some home-based services scheduled for review in September and October 2014. Once the trial is over, ERO will finalise the document and publish it for use in all home-based education and care reviews from early 2015.

Developing a new methodology for kura ā iwi

ERO is focused on providing high quality evaluation for Māori immersion education settings. ERO has begun initial discussions with Kura ā Iwi, about the development of a culturally responsive evaluation methodology and indicators for these kura. In developing methodologies for the Māori medium sector we acknowledge that both partners bring different and equally comprehensive knowledge, alternative perspectives and share a common goal that tamariki are the focus of the work.

ERO also acknowledges key immersion evaluation features should include:

  • evaluation principles that demonstrate acknowledgement of te reo Māori, tikanga Māori and the developmental nature of the immersion education setting
  • a review process explicitly including the marae protocol of encounter, ongoing dialogue, value of korero - kanohi ki te kanohi
  • the inclusion of a representative of the governing/parent body as a part of the review process

This methodology will be developed by Māori, for Māori and with Māori and will focus on successful outcomes for Māori.

Evaluation to support Investing in Educational Success

While the detailed policy for Investing in Educational Success is still being worked through, ERO anticipates that it will have implications for how we review schools that are part of the initiative. We are considering the likely implications ahead of its implementation in order to be ready to support this initiative.

Areas of likely attention include the building of self-review capability to enable communities of schools to develop evidence-based achievement challenges – as well as evaluating the impact of the policy on student achievement in individual schools, communities of schools, and nationally.

Part two – Matters for the Incoming Minister

Our immediate engagement with the incoming Minister

As part of our initial engagement, we will seek your input into our strategic direction, our key strategic projects and methodologies underway to advance the system, and our national evaluations due for impending release. We will discuss with you where our effort and resources will have the most effective impact, and where there are opportunities to divest from some activities and re-invest in others.

ERO people are out working in the field every day providing evaluation services that influence change and build self improvement capability. ERO is therefore in a unique position to provide evidence-based information and insights to you. We also consolidate data and information to provide a national view about the performance of the education system.

ERO will assist you to intervene early in those schools and services at risk of underperforming. We will immediately alert you to any schools/early childhood services (or clusters) of concern, as well as those areas of good practice that we have identified through our national reporting.

Our regular reporting will keep you informed about the progress of our work and provide a means to capture your input into the work that we do. In addition we will provide advice as required/requested to ensure you have an ERO perspective on policy and practice.

Key decisions for Cabinet in the next six months

There are no urgent matters or pressing decisions from ERO that Cabinet needs to consider in the next six months.

Part three – Building our capability

In the last six years, ERO has continued to manage within its baseline. However, increasing workforce costs and other financial pressures mean that this is unsustainable. We will therefore be reviewing our operating model and seeking to make changes to the business processes that support our reviews.

The organisation is re-defining its purpose statement to emphasise our role as a catalyst for change in the system. The purpose statement will then lead to a revised operating model that focuses our resources towards having a greater impact, especially for those children for whom the system has consistently failed. The model will describe the people, processes and systems, including governance and decision rights that we will need now and in the future.

We are reviewing our engagement strategy to establish different partnerships and relationships with the wider evaluative community. This includes national and international evaluation associations, government agencies (such as SupERU 4), and non- government agencies (such as Cognition, CORE and COMET)5. This will help us to share and leverage expertise. We are also improving our communication and social media capabilities. This includes making improvements to our website, and building our presence though Facebook and Twitter, so that we can communicate and interact with our audience better.

Our current information systems, including information and communications technology, will not support the changes we wish to make. For example, we will need to digitise our data, and be able to share information across the system. We will therefore be developing a new information strategy and will discuss this with you.

Our People

Our people are our greatest asset. We recruit senior staff from the ranks of early childhood and school teachers, and from management. The role of the review officer is a demanding one, requiring deep knowledge of education and evaluation, and having the ability to adapt practice to the context and situation of each school and early childhood service. However, our demography is such that half of this workforce could potentially retire in the next ten years (54% of ERO’s review officers are above the age of 60). This represents a significant risk to ERO, creating a potential knowledge gap, if a high number of trained staff (with a high degree of knowledge and expertise) leave the system within a similar timeframe. ERO will need to carefully consider its future workforce requirements, the placement of resources, and establish a profile of skills and experience required for the future.

We are also looking to increase the movement of staff between the sector and the organisation, and will be scoping a secondment approach to allow practitioners to work as review staff for periods as part of their development. We will also look to broaden the base of experience in our workforce.

We will remain committed to integrating equality and diversity into all aspects of the business. ERO’s reviewers, where appropriate, reflect the make-up of the community and ERO will continue to recruit specialist Māori and Pacific staff with competence in te reo Māori and Pacific languages. ERO also promotes cultural diversity through its Te Uepū and Moana Pasefika forums6.

System Stewardship

In the last twelve months, the Secretary of Education has chaired an Education Sector Stewardship Group (ESSG), which aims to increase collaboration, and discuss strategic priorities and systemic issues. ERO develops joint work programmes with the Ministry of Education and other education agencies. Our internal strategies align closely to the work of the sector. For example, He Toa Takitini (ERO’s strategy for Māori success) articulates how we can support the priorities identified in Ka Hikitia: Accelerating Success (the sector’s strategy for Māori). ERO’s Pacific strategy also contributes to the wider sector’s commitment to accelerating Pacific success.

The collaboration provides an opportunity for ERO to be part of a concerted effort by the sector to improve the system as well as reduce compliance costs. For example, currently there is an over-abundance of information and guidance material, and programmes and initiatives, aimed at schools and early childhood services. The sector will need to work together to reduce all unnecessary clutter.

ERO is refreshing its stakeholder engagement to ensure that our programme of liaison (at national and regional levels) provides an effective and consistent two way communication channel with a range of evaluation agencies, as well as a range of professional and representative bodies.

Our Organisation Chart

This is ERO's organisation chart showing the Chief Review Officer and their direct reports

Regional Structure

Review services, other than for kōhanga reo and kura kaupapa Māori, are grouped into three regions and managed by three National Managers. Site offices are located in Auckland, Hamilton, Napier, Whanganui, Wellington, Christchurch and Dunedin.

Reporting services for kōhanga reo and kura kaupapa Māori are managed by the National Manager Māori Review Services and carried out by Te Uepū ā-Motu, a team of review officers with specific knowledge and skills in tīkanga and te reo Māori.This diagram is of New Zealand and where the officers of ERO are located in the Northern region there are Auckalnd and Hamilton offices with staff numbers of 74 FTE, Central region has Napier, Whanganui and Wellington offices with 44 FTE staff, Southern region has Christchurch and Dunedin with 32 FTE staff. Te Upeu a Motu has 12 FTE staff and National Office has 44 FTE staff giving a total staff number of 206 FTE

Evaluation Services / Corporate Functions

Evaluation Services is responsible for our national evaluation programme, our evaluation methodologies development programme and third party evaluation projects. Evaluation Services has a small team of evaluators and analysts based in National Office in Wellington. They work closely with the wider Review Services team to undertake the Evaluation Services work programme.

Our corporate functions also operate from National Office in Wellington.

Appendix 1 – Summary of Outputs and Expenses

Expenses and Revenue

 

2014/15

2013/14

 

Budget$000

Budgeted $000

Estimated Actual$000

Total Appropriation

     

Early Childhood Education Services

9,919

9,927

9,926

Schools and Other Education Service Providers

14,835

15,110

14,872

Quality of Education Reports and Services

3,136

3,962

3,857

Revenue from the Crown

     

Early Childhood Education Services

9,904

9,908

9,908

Schools and Other Education Service Providers

14,801

15,069

15,069

Quality of Education Reports and Services

2,594

2,438

2,438

Revenue from Others

     

Early Childhood Education Services (services)

15

19

18

Schools and Other Education Service Providers

34

41

38

Quality of Education Reports and Services

542

1,524

1419

         

In addition to Crown revenue, ERO provides contract-based services to third parties on a fee-for-service basis and generates a small amount of revenue from rent recoveries.

Summary of Outputs

 

2014/15 Estimated Quantity

2013/14 Estimated Quantity

Early Childhood Education Services (services) output class

   

Number of early childhood services education reviews

1,300-1,460

1,300-1,460

Number of post-review assistance workshops to early childhood services

-

80-120

Schools and Other Education Service Providers output class

   

Number of state schools education reviews

710-840

710-840

Number of homeschooling education reviews

Up to 35

25-35

Number of private school education reviews

15-25

15-25

National Evaluation Reports and Policy Services

   

Number of national evaluation reports

Up to 20

Up to 20

Number of policy services

-

25-30

Policy services, ministerial and contractual services are subject to external demand factors.