PART 3: Complementary evaluation in early childhood service reviews

Complementary evaluation is the purposeful interaction between internal and external evaluation. It implies a mutually beneficial relationship that recognises the distinct purposes of each and where the overlap lies.

ERO’s external evaluation process is both proportional and responsive to the service’s self review. It responds to the early childhood service’s overall capacity and capability to evaluate its own performance. ERO’s external evaluation also has a role to play in building the evaluation capacity of the service by strengthening self review according to each service’s context.

Diagram 1: Complementary evaluation

Thsi diagram is a long rectangle box. This box is divdied in two three parts the left and right side boxes being smaller than the centre box. There is a dashed line running on an angle from the bottom left to top right corner across all three boxes dividing them in half. The top centre box half is called ERO review, the bottom half is called EC service self review. Under the three boxes it reads the following from left to right. EC services with very limited self review - EC services operating well with established processes for self review - EC services sustaining high quality provision and continuous improvement through self review.

An evaluation approach that balances external ERO review with service self review according to each service's circumstances.

External evaluation can:

  • stimulate internal evaluation
  • expand the scope of internal evaluation
  • validate the results of internal evaluation
  • provide an additional perspective
  • include a capacity building role as part of the evaluation.

Internal evaluation can:

  • deepen the scope of external evaluation
  • give a context to the external evaluation
  • provide important insights
  • improve the interpretation of external evaluation findings
  • increase the use of external evaluation findings.

Building evaluation capability

ERO uses its external evaluation process to increase the capability within early childhood services to undertake internal evaluation (self review) as a routine activity for both accountability and improvement purposes. The intention is for evaluation to become embedded in the day-to-day practice of managers and educators.

ERO builds the evaluation capability of the early childhood service through:

  • making its own external review processes transparent
  • modelling evaluation practice
  • encouraging participation in ERO’s evaluation process
  • having discussions about the service’s self-review processes
  • involving nominated service personnel in evaluation design, analysis and synthesis processes
  • providing tools (e.g. examples of evaluation questions and indicators) that services can use in their self review
  • discussing resources that services can use to help them with self review.

Self review in early childhood services

The term self review is often used synonymously with other terms such as self evaluation, internal evaluation, evaluation, assessment, monitoring and appraisal. In early childhood services in Aotearoa New Zealand self review has its roots in terms such as programme review, internal review and quality review.

ERO defines self review as the use of robust processes to systematically inquire into and evaluate the effectiveness of policies, programmes and practices. Self-review findings are used to inform decision-making, improve the quality of practice and promote positive outcomes for all children.

Review implies a looking back or taking stock of some activity, practices or processes. However, self review is also evaluative in nature.

Definitions of evaluation vary according to the underpinning theory or theories that they are based on. All evaluation involves reaching judgements or producing knowledge about what is being evaluated. The purpose of each evaluation differs: it can be for accountability purposes, to help with decision making, to contribute to improved effectiveness, to increase understanding or to advance a principle such as equity. This is referred to as evaluation use.

Regulatory requirements for self review

Early childhood services are required to undertake self review as part of their licensing requirements. The criteria to assess the governance management and administration standard specifies that an ongoing process of self-review helps the service maintain and improve the quality of its education and care. 1

The licensing criteria require early childhood services to document:

  • a process for reviewing and evaluating their operation
  • a schedule showing timelines for planned review of different areas of operation
  • recorded outcomes from the review process.

Self review is an integral part of professional practice in the governance, management and administration of an early childhood service.

Guidance and support to help services with self review

The Ministry of Education resources such as Quality in Action: Te Mahi Whai Hua 2  and The Quality Journey: He Haerenga Whai Hua: Improving quality in early childhood services  3  help early childhood services understand the concept of self review. 

The Ministry has also published guidelines for self review in early childhood services: Ngā Arohaehae Whai Hua  4. These guidelines set out a process for services to use to undertake self review. This process is one that requires a systematic process of preparation, data gathering, analysis and decision-making. It implies an evaluative approach.

Highly effective self review

In 2008 ERO undertook a national evaluation 5  that focused on how well self review was understood, supported and implemented in early childhood services. ERO found that where self review was highly effective:

  • improvement and accountability were understood to be the main purposes of self review
  • management and educators shared the same understandings about self review
  • it was embedded in practice and integral to the service’s operation
  • reviews had a clear focus
  • it was strongly focused on improvement and with well-established procedures to guide practice
  • reviews were both planned and spontaneous
  • planned reviews included scheduled policy reviews and more in-depth reviews of targeted areas of practice
  • spontaneous reviews were responsive to emerging issues.

The factors that emerged as common to all services where self review was well understood and implemented included:

  • strong leadership to promote self review
  • professional development to support self review
  • stable and collaborative staff
  • sound, sustainable systems for self review
  • the use of relevant resources and support systems.