Part 7: Process guidelines for reviews

Notification

ERO begins the process with an email notification that gives the home-based early childhood service time to prepare for the external review.

In the email, ERO includes links to electronic versions of this document, the Assurance Statement and Self-Audit Checklists, a hard copy of the Self Report and links to electronic versions of other relevant documents.

ERO gives the service information to distribute to coordinators/visiting teachers, parents, families, whānau and educators about the review.

Self reports

The self report plays a significant part in the ERO review. ERO’s evaluation builds on the information that the service provides in the self report about what is going well and where improvement may be needed. It helps ERO to design a review that is responsive to each context.

The self report is largely structured on Ngā Pou Here. Completing the self report is an opportunity for the service to reflect on its practice in relation to promoting positive learning outcomes for children.

The self reports are available to download from ERO’s website and complete electronically.

Information exchange

Before the on-site stage of the review, information is shared between ERO and the service.

Home-based services with upcoming ERO reviews are invited to participate in a group briefing meeting, coordinated by local ERO personnel. Individual schemes/networks involved in cluster reviews will participate in a briefing meeting within their association or umbrella organisation. The briefing meeting will explain ERO’s review process and general focus.

The review coordinator gives the service an opportunity to clarify information about the review process, either at the briefing meeting or through telephone or email contact, and explores with the service who will take responsibility for being involved in the review.

The review coordinator specifies the date for the early childhood service to provide ERO with the completed Self Report and Assurance Statement and other requested documentation.

Designing the review

Review design involves ERO deciding, often in collaboration with the service, where it will place its evaluation emphasis during the ERO review. A significant determinant of review design is the quality of internal evaluation within a service.

The organisation and nature of the ERO review is determined in response to context. Context includes the service’s philosophy, community and its capacity, as indicated through documentation and other sources of information.

ERO uses information provided by the service, including the completed relevantAssurance Statement and Self Report, to scope the review. ERO also draws on internal information it has about the service including the last ERO report and reporting history. The scoping process helps ERO to determine what needs to be further explored during the review.

ERO uses Ngā Pou Here as a framework for organising the information it has and the investigative questions it wants to answer on the review. Consideration is given to the interrelationship between the Pou and where the review emphasis will be placed.

ERO’s emphasis, in reviews of home-based education and care, will be the service provider and coordinator/visiting teacher as the people who have primary responsibility for service operation and those who oversee the education and care requirements.

The review team plans the review process taking into account the resources available for the review. The review design is shared with the service.

The focus of the review

Decisions about the balance between the Pou and where to focus reviews depend on information in the completed Self Report and Assurance Statement.

The time allocated to gathering information within each Pou, the processes used and the reporting of findings varies between reviews. ERO uses a ‘one-size-fits-one’ approach so that the process is tailored to be as responsive and contextual as possible. Each review will look different.

The Ngā Pou Here framework shows relationships between each Pou and outcomes for children. Areas of strength and areas for review and development that are identified by ERO and the service should lead to (or help create the conditions for) improved learning outcomes for children.

The national evaluation topics (NETs) provide a way for ERO to investigate key aspects of early childhood services’ performance in relation to the Government’s education priorities. Each topic is explored through the review framework, Ngā Pou Here, and is evaluated in this context.

In some services, reviews need to focus on compliance, because of risks to the safety and wellbeing of children. This is likely to be the case where, despite attestation made by the service in the Assurance Statement, there do not appear to be adequate systems for the internal checking of compliance.

Investigation and synthesis

During its time on-site the review team:

  • uses evaluative questions, investigative questions and evaluation indicators as a basis to gather and document evidence
  • reads the service’s documentation, talks to managers and others as appropriate, determines the purpose and focus of any visits to homes if required
  • synthesises the review findings.

The service delegates appropriate personnel to be involved in the review and negotiates the level of their involvement with the review team. Ongoing interaction between service personnel and the review team will be a feature of the process.

During the scoping process the review teams plan with the service who ERO should talk to during the review. Participants may include:

  • coordinators/visiting teachers
  • the owner of the service
  • the contact person/service provider
  • management
  • personnel from the umbrella organisation
  • children
  • parents
  • educators
  • the Māori community
  • other groups.

From the service’s internal evaluation information, ERO identifies if the early childhood service is already performing well in a specific area or if there is a need for improvement.

For areas where the service appears to be performing well, the priority is on validating the results of internal evaluation. For areas where review or development is needed ERO uses its processes to build the service’s capability to evaluate and improve its own performance.

Consultation

ERO encourages the home-based education and care service to share information about the consultation it has undertaken. ERO is particularly interested in any consultation that shows:

  • how the service works in partnership with parents, whanau and educators and how they and all staff contribute to the service’s internal evaluation.

Discussion of findings

Towards the end of its time at the service the review team will discuss the review findings with the personnel nominated by the service. The discussion will highlight areas of good performance and areas for review and development. In situations where significant development is needed ERO will indicate the likelihood of an early return review.

This discussion of findings should include the service provider and coordinator/visiting teacher and any others who will have the responsibility for taking action as a result of the external review.

It may be that findings are shared throughout the review process, in which case a discussion of findings may not be necessary at the end of the onsite stage of the review.

Reporting

The audience for ERO reports includes the public and the Government, as well as those in the early childhood education sector.

Early childhood service reports will start with an overall judgement. The report will include the material findings that answer the overarching evaluation question. The report is sent to the early childhood service as an unconfirmed report within 20 working days of the completion of the on-site part of the review. A service that is identified as Not Well Placed will receive the unconfirmed report within 10 days of the last day on site.

A service that is identified as Not Well Placed has 10 days to respond. The management of the early childhood service has 15 working days from the date ERO sends the report in which to query the evidential basis for reported judgements; and/or advise ERO in writing of any errors of fact and provide supporting documentation.

ERO considers any response from the service and, where justified, makes amendments to the report. The report is confirmed and a copy is sent to the service provider.

The confirmed report is released publicly on ERO’s website two weeks after it is sent to the service provider.

Cluster reviews

Cluster reviews: reviews of individual services managed by umbrella organisations

ERO has adapted its approach to reviews of individual early childhood services managed by umbrella organisations. Cluster reviews apply to kindergartens, playcentres and other individual early childhood services that operate under an umbrella organisation.

The cluster refers to the grouping of individual scheme/network reviews together. It is not a review of an association or umbrella organisation.

Where there is an umbrella organisation, ERO refers to each home-based network within that umbrella as the service, and refers to the umbrella as the organisation.

Cluster reviews:

  • make better use of the umbrella organisation and individual services’ internal evaluationcontribute to internal evaluation capacity at the management/professional practice level of the umbrella organisation as well as at the service level
  • ensure the efficient use of ERO’s resource.

Determining a cluster for review

A cluster review can only be undertaken if:

  • there is an umbrella organisation that receives funding, provides services and monitors the quality of its individual services
  • there is a degree of commonality across policies, structures and quality assurance for individual services within the cluster.

The size and make-up of a cluster takes into account the distance between services, overall size of the umbrella organisation, and previous ERO return times for each service.

Home-based services that are part of national umbrella organisations automatically meet the criteria for cluster reviews. This does not mean that all of the services within an organisation will be reviewed at one time. There may be situations where it is more appropriate for the review of a service to be undertaken as a stand-alone review, rather than as part of a cluster of reviews.

Notification

ERO consults with the umbrella organisation to identify the cluster size and specific services to be reviewed.

Each service licence holder receives a notification electronically with links to this document, the Assurance Statement and Self-Audit Checklists, a hard copy of the self report and electronic versions of relevant documents. ERO provides the service with information to distribute to coordinators/visiting teachers, educators, parents and whānau about the review.

ERO sends an information letter and a self report for the umbrella organisation having a cluster review. In the self report the umbrella organisation describes what it knows about the performance of each scheme/network within the cluster and how internal evaluation in the organisation contributes to quality improvements for children.

Review process

The umbrella organisation nominates representatives to be involved in the ERO process.

The review team meets with nominated representatives to explore internal evaluation within the organisation as it relates to individual services and to develop review planning for the cluster.

The review team arranges a briefing meeting for nominated representatives and service leaders. The nature of this meeting will depend on ERO’s previous involvement with and knowledge of the umbrella organisation.

The service and the umbrella organisation provide ERO with the required review documentation.

The review teams work with the nominated representative and service leader as appropriate to design each review to reflect the context of each service.

The nominated representatives negotiate their role with the ERO cluster coordinator and/or each review team.

Each review proceeds according to ERO’s review process guidelines. Refer to previous information in this section.

ERO writes an individual education review report for each licensed service. Evaluation findings about governance, management or organisation leadership, to the extent that they impact on outcomes for children, will be specified in these reports.

ERO uses its evaluation criteria to make a decision about the return time for each service’s scheme/network. In making this decision the review team will also take into account the capacity of the umbrella organisation to generate and sustain improved performance in that service.

ERO does not undertake a separate review of the umbrella association, or provide a written report based on the collation of trends or patterns evident through the individual reports. ERO does however have the scope to conduct a special review of the umbrella organisation if major issues arise in the course of a cluster review.

Differentiated return times

The timing of the next ERO review will depend on how well placed the service is to promote positive learning outcomes for children. There are four options:

Very well placed - the next ERO review in four years

ERO will next review the service in four years when it finds that the service is consistently effective in promoting children’s wellbeing and learning. High quality performance in relation to ERO’s evaluation indicators for Ngā Pou Here (home-based) will be evident.

Well placed - the next ERO review in three years

ERO will next review the service in three years when it finds that the service is effective in promoting children’s wellbeing and largely effective in promoting children’s learning. Good performance in relation to Ngā Pou Here (home-based) will be evident.

Requires further development - the next ERO review within two years

This option is used when many of the factors that contribute to positive learning outcomes for children are not evident or require significant development. ERO will have some confidence that the service can improve with support.

After receiving the confirmed report, the service will be expected to participate in a meeting with ERO and the Ministry of Education to begin developing a plan for improvement.

The Ministry of Education will oversee the support that the service needs to become well placed to promote positive learning outcomes for children.

Approximately six to nine months after the ERO review, the service will provide ERO with an update about its progress. The Ministry of Education will also update ERO on the progress made. This information will be used to determine the appropriate ERO return time within the two year period.

Should ERO find that there has not been sufficient improvement the service will be identified as Not well placed. ERO will recommend that the service’s licence is reassessed by the Ministry of Education.

The next ERO review in consultation with the Ministry of Education

This option will be used when a service is not performing adequately, is not meeting legal requirements and does not have the capacity to make improvements without support or Ministry intervention.

The service will be expected to address concerns and prevent a continuation of poor performance. There will be licensing consequences for continued poor performance.

ERO will not review the service again until the Ministry of Education is satisfied that the service meets licensing requirements.

The four different review return times provide ERO with greater flexibility in how it responds to an early childhood service’s performance. ERO and the Ministry of Education will work closely with services that need to improve and help them build capability. ERO will make less frequent visits to those services that are performing very well.

Most early childhood services are likely to be considered Well placed and will continue to be reviewed every three years.

ERO has developed criteria to support the overall judgements and their corresponding return times for home-based services. These are on ERO’s website (www.ero.govt.nz) under Review Process/Early Childhood. Ngā Pou Here and the evaluation indicators support the criteria and provide a deeper insight into what ERO considers to be high quality early childhood education and care in home-based services.

Nga Pou Here criteria return times. The first part of the diagram is the Nga Pou Here evaluation indications and the four Pou. The second is the part is three boxes which have the return times. The first is 2 years, the second is 3 years and the third is 4 years.