New assurance and quality reviews

ERO’s approach to review and evaluation in centre-based early childhood services

ERO is updating its approach to review and evaluation, initially in centre-based early childhood services. Ngā Ara Whai Hua: Quality Framework for Evaluation and Improvement in Centre-based Early Childhood Services outlines our approach to review and evaluation for accountability and improvement. This framework is very strongly focused on improvement whilst maintaining a focus on compliance with regulatory standards.

Akanuku | Assurance Reviews assess whether services are meeting regulatory standards. We have been implementing Akanuku since July 2019 following a trial in the first half of 2019.

Akarangi | Quality Evaluations evaluate the extent to which services have the learning and organisational conditions to support children’s learning.

Akamatua | Evaluating Excellence and Innovation identify, investigate and evaluate cases of excellence and innovation. We are developing this case study approach and we will be undertaking some initial case studies in selected services later in 2020 to inform the design of this approach prior to implementation in 2021.

Akanuku | Assurance Reviews

ERO is responsible for reporting publicly on the quality of education provided in New Zealand schools and early childhood services. Its role encompasses accountability (including compliance with legal requirements), educational improvement, and knowledge generation.

ERO undertakes an Akanuku | Assurance Review process in any centre-based service:

  • having its first ERO review – including if it is part of a governing organisation
  • previously identified as ‘not well placed’ or ‘requiring further development’
  • that has moved from a provisional to a full licence
  • that has been re-licensed due to a change of ownership
  • where an Akanuku | Assurance Review process is determined to be appropriate.

An ERO Akanuku | Assurance Review assesses whether a service meets regulatory standards/licensing requirements. As part of an Akanuku | Assurance Review process, ERO makes one of the following judgements:

  • At the time of the review ERO found the service was taking reasonable steps to meet regulatory standards; or
  • Since the onsite visit the service has provided ERO with evidence that shows it has addressed non-compliances and is now taking reasonable steps to meet regulatory standards; or
  • At the time of the review ERO identified non-compliance with regulatory standards that must be addressed; or
  • At the time of the review ERO identified significant areas of non-compliance with regulatory standards that are an unacceptable risk to children.

ERO will recommend that the next review of the service is likely to be an Akarangi | Quality Evaluation; or that the next ERO review of the service will be in consultation with the Ministry of Education.

Key documents if your service is having an Akanuku | Assurance Review

Akarangi | Quality Evaluations

Akarangi | Quality Evaluations evaluate the extent to which an early childhood service has the learning and organisational conditions to support equitable and excellent outcomes for all learners. Te Ara Poutama- indicators of quality for early childhood education: what matters most are the basis for making judgements about the effectiveness of the service in achieving equity and excellence for all learners. The Akarangi Quality Evaluation Judgement Rubric derived from the indicators, is used to inform the ERO team’s judgement about this service’s performance in promoting equity and excellence.

In an Akarangi | Quality Evaluation, judgements are made in relation to outcomes for learners for te Akatoro | each domain in Te Ara Poutama- indicators of quality for early childhood education: what matters most.  The Akarangi Quality Evaluation Judgement Rubric is used to evaluate where this service is on a quality improvement continuum. ERO is encouraging early childhood services to use the same judgements in their internal evaluation supported by evidence about where they are on the continuum and what they need to do to improve.

Judgement

Description

Whakatō

Emerging

The seed is sown, a place of potential

Emerging means that the service is at an early stage of developing the learning and organisational conditions (systems, processes and practices) to improve quality and realise equitable outcomes for children.

Whāngai

Establishing

Nourishment is provided

Establishing means that the service is establishing the learning and organisational conditions (systems, processes and practices) and is building shared understandings to improve quality towards realising equitable outcomes for children.

Whakaū

Embedding

Consolidation of all elements has occurred

Embedding means that the service has established and is embedding coherent learning and organisational conditions (systems, processes and practices) and is consistently implementing practices and processes to enable improvement in quality towards realising equitable outcomes for children.

Whakawhanake

Sustaining

Appropriate conditions provide strong and ongoing support

Sustaining means that the service has evidence of its capability and capacity to sustain ongoing improvement to the learning and organisational conditions (systems, processes and practices) and the impact of this improvement in realising equitable outcomes for children.

Kia rangatira ai te tipu

Excelling

A combination of favourable conditions and a nourishing environment produces ongoing quality of results

Excelling means that the service is sector leading and demonstrates exemplary practice resulting in equitable outcomes for children.

Documents used in Akarangi | Quality Evaluations

Te Ara Poutama- indicators of quality for early childhood education: what matters most

Akarangi Quality Evaluation Judgement Rubric 

Ngā Rāpupuku Indicators – poster 

Akarangi | Quality Evaluation process 

Notifying

ERO begins the process with a notification email that details:

  • the timing of the evaluation
  • an outline of the documentation the service is asked to provide for ERO
  • links to the appropriate resources on ERO’s website including

the indicators

  • any other information required before the onsite phase of the evaluation.

Exchange of information

Before the on-site phase information is shared between ERO and the service. This communication gives you an opportunity to:

  • share information
  • become familiar with the evaluation approach and resources
  • ask questions to clarify the evaluation process
  • clarify any administrative details including the ERO team’s date and time of arrival, any protocols the team should observe, dates and times for specific meetings and location of an on-site base for the team.

The exchange of information is essential in preparing your service and the ERO team for the evaluation.

Compliance

In a Quality Evaluation ERO will begin with checking some compliance items working with a nominated person from the service. The following items are checked because they have a potentially high impact on children’s learning, wellbeing and health and safety:

  • emotional safety including positive guidance, child protection, including safety checking
  • physical safety including supervision, sleeping practices, accidents, medication, hygiene and excursion policies and procedures
  • suitable staffing including qualification levels, police vetting, teacher registration and ratios
  • evacuation procedures and practices for fire and earthquake.

If this checking process identifies non-compliance that is an ‘unacceptable risk’ to children (related to any of the items above), ERO will shift the focus of the evaluation to that of an Akanuku | Assurance Review and do a full assessment of the regulatory standards and licensing criteria.

ERO also checks to see that areas of non-compliance in the previous ERO report (if applicable) have been addressed.

Internal evaluation

ERO considers your service’s internal evaluation information about what you know about your effectiveness in achieving equitable and excellent outcomes for learners and the extent to which your service’s learning and organisational conditions support or hinder improvement.

Evaluation Design

The ERO team uses all available information, including discussions with you, to design the evaluation.

Onsite information gathering, analysis and synthesis

The onsite phase of the evaluation begins with an initial discussion between your service (key personnel) and the ERO team.

The discussion is an opportunity for the ERO team to:

  • share and discuss the proposed evaluation design
  • clarify the proposed involvement of people in your service in the evaluation process and compliance checking
  • provide information about the evaluation process.

The ERO team gathers information in a range of ways.  Review officers will read documents, meet with key people and observe practice including some aspects of your curriculum in action.  At the end of the onsite phase the ERO team will meet with nominated service personnel to share and clarify emerging findings of the evaluation.

Reporting

Akarangi | Quality Evaluation reports are concise. They make clear ERO’s evaluative judgements in relation to the outcome and process indicators.

The Akarangi | Quality Evaluation report will include:

  • ERO’s judgements in relation to te Akatoro | each domain of the indicators and about outcomes for children
  • a short context section
  • a summary of findings
  • improvement actions
  • any actions for compliance.

The unconfirmed report is emailed to your service 20 working days from the last day the ERO team was at your service. You will be invited to respond noting any errors of fact, significant omissions or comment on any other matters related to the findings, including providing additional evidence to support this. Your service has 15 working days to respond.

The ERO coordinator considers the response from your service including the supporting documentation supplied. The ERO coordinator may want to discuss the response with others in ERO. The review coordinator then decides whether to amend the report and, if appropriate, makes appropriate amendments.

ERO will publish the Akarangi | Quality Evaluation report on its website two weeks after it is sent to your service.  

Preparation for your Akarangi | Quality Evaluation

ERO’s intent is to ensure that all learners in our system are successful. We are interested in supporting your service’s improvement journey. Before our Quality Evaluation starts on site, we encourage you to reflect on the questions below. These questions provide the broad context for discussion likely to occur during your external evaluation.  In reflecting on these questions, you are likely to refer to a range of existing information you already have in your service.

We are particularly interested in discussing with you:

  • what you know about children’s learning in relation to the learning outcomes in
    Te Whāriki?
  • what you know about the learning and development of Māori children; children of Pacific heritage, children with additional learning needs and children up to the age of 3 in your service?
  • what your priorities are for children’s learning and why these priorities?
  • what you know about how your learning and organisational conditions (see Te Ara Poutama- indicators of quality for early childhood education: what matters most and Akarangi Quality Evaluation Judgement Rubric are enabling (or hindering) equitable and excellent outcomes for all children?
  • your current improvement focus/foci?

Pre-evaluation information