PART 7: Process Guidelines for Reviews

Notification

ERO begins the process with a notification letter that gives the early childhood service time to prepare for the external review.

With the notification letter ERO sends He Pou Tātaki (this document) the Guidelines, Assurance Statement and Self-Audit Checklists, a hard copy of the Self Report and links to electronic versions of relevant documents.

ERO gives the service information to distribute to teachers, parents and whānau about the review.

Self reports

The self report plays a significant part in the ERO review. ERO’s evaluation builds on the information that the service provides in the self report about what is going well and where improvement may be needed. It helps ERO to design a review that is responsive to each context.

The self report is largely structured on Ngā Pou Here. Completing the self report is an opportunity for the service to reflect on its practice in relation to promoting positive learning outcomes for children.

The self reports are available to download from ERO’s website and complete electronically.

Information exchange

Before the on-site stage of the review, information is shared between ERO and the service.

Early childhood services with upcoming ERO reviews are invited to participate in a group briefing meeting, coordinated by local ERO personnel. Individual centres involved in cluster reviews will participate in a briefing meeting within their association or umbrella organisation. The briefing meeting will explain ERO’s review process and general focus.

The review coordinator gives the service an opportunity to clarify information about the review process, either at the briefing meeting or through telephone or email contact, and explores with the service who will take responsibility for being involved in the review.

The review coordinator specifies the date for the early childhood service to provide ERO with the completed Self Report and Assurance Statement and other requested documentation.

Designing the review

Review design involves ERO deciding, often in collaboration with the service, where it will place its evaluation emphasis during the ERO review. A significant determinant of review design is the quality of self review within a service.

The organisation and nature of the ERO review is determined in response to context. Context includes the service’s philosophy, community and its capacity, as indicated through documentation and other sources of information.

ERO uses information provided by the service, including the completed relevantAssurance Statement and Self Report, to scope the review. ERO also draws on internal information it has about the service including the last ERO report and reporting history. The scoping process helps ERO to determine what needs to be further explored during the review.

ERO uses Ngā Pou Here as a framework for organising the information it has and the investigative questions it wants to answer on the review. Consideration is given to the interrelationship between the Pou and where the emphasis will be placed.

The review team plans the review process taking into account the resources available for the review. The review design is shared with the service.

The focus of the review

Decisions about the balance between the Pou and where to focus reviews depend on information in the completed Self Report and Assurance Statement.

The time allocated to gathering information within each Pou, the processes used and the reporting of findings varies between reviews. ERO uses a ‘one-size-fits-one’ approach so that the process is tailored to be as responsive and contextual as possible. Each review will look different.

The Ngā Pou Here framework shows relationships between each Pou and outcomes for children. Areas of strength and areas for review and development that are identified by ERO and the service should lead to (or help create the conditions for) improved learning outcomes for children.

The national evaluation topics (NETs) provide a way for ERO to investigate key aspects of early childhood services’ performance in relation to the Government’s education priorities. Each topic is explored through the review framework, Ngā Pou Here, and is evaluated in this context.

In some services, reviews need to focus on compliance, because of risks to the safety and wellbeing of children. This is likely to be the case where, despite attestation made by the service in the Assurance Statement, there do not appear to be adequate systems for the internal checking of compliance.

Investigation and synthesis

During its time on-site the review team:

  • uses evaluative questions, investigative questions and evaluation indicators as a basis to gather and document evidence
  • reads the service’s documentation, talks to managers and others as appropriate, and observes the programme in action
  • synthesises the review findings.

The service delegates appropriate personnel to be involved in the review and negotiates the level of their involvement with the review team. Ongoing interaction between service personnel and the review team will be a feature of the process.

The scoping process helps review teams to plan with the service who else ERO should talk to during the review. Participants may include:

  • teachers
  • the owner of the service
  • the contact person/service provider
  • management
  • personnel from the umbrella organisation
  • children
  • parents
  • the Māori community
  • other groups.

From the service’s self-review information ERO identifies if the early childhood service is already performing well in a specific area or if there is a need for improvement.

For areas where the service is performing well, the priority is on validating the results of self review. For areas where review or development is needed ERO uses its processes to build the service’s capability to evaluate and improve its own practice.

Consultation

ERO encourages the early childhood service to share information about the consultation it has undertaken. ERO is particularly interested in any consultation that shows:

  • how the service works in partnership with parents and whānau
  • how staff contribute to review and development in the service.

Discussion of findings

Towards the end of its time at the service the review team will discuss the review findings with the personnel nominated by the service. The discussion will highlight areas of good performance and areas for review and development. In situations where significant development is needed ERO will indicate the likelihood of an early return review.

This discussion of findings should include management, staff and any others who will have the responsibility for taking action as a result of the external review.

It may be that findings are shared throughout the review process, in which case a discussion of findings may not be necessary at the end of the onsite stage of the review.

Reporting

The audience for ERO reports includes the Government and the public, as well as those in the early childhood education sector.

Early childhood service reports will start with an overall judgement. The report will include the material findings that answer the overarching evaluation question. The report is sent to the early childhood service as an unconfirmed report within 20 working days of the completion of the on-site part of the review. A service that is identified as Not Well Placed will receive the unconfirmed report within 10 days of the last day on site.

The management of the early childhood service has 15 working days from the date ERO sends the report in which to query the evidential basis for reported judgements; and/or advise ERO in writing of any errors of fact and provide supporting documentation. A service that is identified as Not Well Placed has 10 days to respond.

ERO considers any response from the service and, where justified, makes amendments to the report. The report is confirmed and a copy is sent to the service provider. The confirmed report is released publicly on ERO’s website two weeks after it is sent to the service provider.

Cluster Reviews: Reviews of Individual Services Managed by Umbrella Organisations

ERO has adapted its approach to reviews of individual early childhood services managed by umbrella organisations. Cluster reviews apply to kindergartens, playcentres and other individual early childhood services that operate under an umbrella organisation. The cluster refers to the grouping of individual centre reviews together. It is not a review of an association or umbrella organisation.

Where there is an umbrella organisation, ERO refers to each early childhood service within that umbrella as the centre, and refers to the umbrella as the service.

Cluster reviews:

  • make better use of the umbrella organisation and individual centre self review
  • contribute to self-review capacity at the management/professional practice level of the umbrella organisation as well as at the centre level
  • ensure the efficient use of ERO’s resource.

Determining a cluster for review

A cluster review can only be undertaken if:

  • there is an umbrella organisation that receives funding, provides services and monitors the quality of its individual centres
  • there is a degree of commonality across policies, structures and quality assurance for individual centres within the cluster.

The size and make-up of a cluster takes into account the distance between centres, overall size of the umbrella organisation, and previous ERO return times for each centre.

Kindergartens and playcentres operate under associations and automatically meet the criteria for cluster reviews. This does not mean that all of the centres within an association will be reviewed at one time. There may be situations where it is more appropriate for the review of a centre to be undertaken as a stand-alone review, rather than as part of a cluster of reviews.

Notification

ERO consults with the umbrella organisation to identify the cluster size and specific centres to be reviewed.

Each centre receives a notification pack with He Pou Tātaki (this document), theGuidelines, Assurance Statement and Self-Audit Checklists, a hard copy of the Self Report and links to electronic versions of relevant documents. ERO provides the service with information to distribute to teachers, parents and whānau about the review.

ERO sends an information letter and a self report for the umbrella organisation having a cluster review. In the self report the umbrella organisation describes what it knows about the performance of each centre within the cluster and how self review in the organisation contributes to quality improvements for children.

Review process

The umbrella organisation nominates representatives to be involved in the ERO process.

The review team meets with nominated representatives to explore self review within the organisation as it relates to individual centres and to develop review planning for the cluster.

The review team arranges a briefing meeting for nominated representatives and centre leaders. The nature of this meeting will depend on ERO’s previous involvement with and knowledge of the umbrella organisation.

The centre and the umbrella organisation provide ERO with the required review documentation.

The review teams work with the nominated representative and centre leader as appropriate to design each review to reflect the context of each centre.

The nominated representatives negotiate their role with the ERO cluster coordinator and/or each review team.

Each review proceeds according to ERO’s review process guidelines. Refer to previous information in this section.

ERO writes an individual education review report for each centre. Evaluation findings about governance, management or organisation leadership, to the extent that they impact on outcomes for children, will be specified in centre reports.

ERO uses its evaluation criteria to make a decision about the return time for each centre. In making this decision the review team will also take into account the capacity of the umbrella organisation to generate and sustain improved performance in that centre.

ERO does not undertake a separate review of the umbrella association, or provide a written report based on the collation of trends or patterns evident through the individual reports. ERO does however have the scope to conduct a special review of the umbrella organisation if major issues arise in the course of a cluster review.

ERO's overall judgement and criteria

Very well placed to promote positive learning outcomes for children

ERO finds that the service is consistently effective in promoting positive learning outcomes for children. High quality performance 1 in relation to ERO’s evaluation indicators will be evident.

The service will have:

  • systems that promote stability in staffing, leadership and management
  • self review that demonstrates the service is likely to sustain high quality performance.

It is likely that the service will have a positive ERO reporting history (of at least two consecutive ERO reports.)

In addition there will be evidence of all of the following:

  • the curriculum responds very effectively to the interests, strengths and abilities of all children
  • the curriculum is highly responsive to children’s cultures, languages and identities
  • a bicultural curriculum is in place and bicultural practice is highly evident
  • teaching is highly effective in developing children’s confidence and competence as lifelong learners
  • the wellbeing and learning of priority groups (including Māori, Pacific, children with special needs, and children up to the age of two) are actively promoted
  • Māori children are well supported to achieve success as Māori
  • there are well established partnerships with parents/whānau that contribute to positive outcomes for children
  • leadership is highly effective in building the quality of learning and teaching
  • management implements highly effective planning, systems, policies and procedures
  • a shared vision, focused on high quality early childhood education, is evident in practice
  • self review results in continuous improvement

Well placed to promote positive learning outcomes for children

ERO finds that the service is largely effective in promoting positive learning outcomes for children. Good performance in relation to ERO’s evaluation indicators will be evident.

There will be sound evidence of the following:

  • the curriculum responds to the interests, strengths and abilities of all children
  • the curriculum is responsive to children’s cultures, languages and identities
  • bicultural practice is evident
  • teaching develops children’s confidence and competence as lifelong learners
  • the wellbeing and learning of priority groups (including Māori, Pacific, children with special needs, and children up to the age of two) are promoted
  • Māori children are supported to achieve success as Māori
  • partnerships with parents/whānau are established
  • leadership helps build the quality of learning and teaching
  • management implements planning, systems, policies and procedures
  • a shared vision for the service is in place
  • self review is becoming established and results in improvement

ERO will have confidence that the service has the capacity to address the areas where limited evidence or effectiveness is identified.

Requires further development to promote positive learning outcomes for children

ERO finds many of the factors that contribute to positive learning outcomes for children are not evident or require significant development. ERO will have some confidence that the service can improve with support.

ERO will maintain an ongoing relationship with the service when it has material concerns about aspects of the following:

  • provision for children’s learning
  • responsiveness to children’s cultures, languages and identities
  • the quality of teaching
  • the effectiveness of leadership
  • the effectiveness of management
  • the effectiveness of governance
  • the quality of self review

ERO will identify areas where the service does not comply with regulations.

If ERO continues to have concerns about children’s health and safety it will determine the next review in consultation with the Ministry of Education.

Not well placed to promote positive learning outcomes for children

ERO finds the service is not performing adequately, is not meeting legal requirements and does not have the capacity to make improvements without support or Ministry involvement. 

There will be significant non-compliance with regulations. ERO will have material concerns about most of the following:

  • the provision of a safe environment
  • provision for children’s wellbeing
  • provision for children’s learning
  • the appropriateness of the curriculum
  • the quality of teaching
  • the effectiveness of leadership
  • the effectiveness of management
  • the effectiveness of governance
  • the quality of self review

ERO will not review the service again until the Ministry of Education is satisfied that the service meets licensing requirements. The service’s ERO reporting history will be considered in guiding this decision.

This approach will be used where there is a high level of concern about children’s health and safety.

List of references for the evaluation indicators

This attachment includes a full list of the references used to develop the evaluation indicators for education reviews in early childhood services.

ECE Indicators References by Pou