Part 3: Complementary approach to evaluation in early childhood service reviews

As New Zealand’s external education evaluation agency, ERO complements the internal evaluation activities of early childhood services.

ERO’s external evaluation process is both proportional and responsive to the service’s internal evaluation. It responds to the early childhood service’s overall capacity and capability to evaluate its own performance. ERO’s external evaluation also has a role to play in building the evaluation capacity of the service by strengthening internal evaluation according to each service’s context.

Diagram 1: Complementary evaluation

This diagram shows three boxes with an upward trajectory arrow from bottom left to top right across all three boxes. The first and third box are smaller sizes with the middle box being the largest. Box 1 is Services with very limited internal evaluation. The second box is Services operating well with established processes for internal evaluation and the third box is Services sustatining high quality provision and continous improvement through effective internal evaluation.

The complementary nature of external and internal evaluation

External evaluation can:

  • stimulate internal evaluation
  • expand the scope of internal evaluation
  • validate the results of internal evaluation
  • provide an additional perspective
  • include a capacity building role as part of the evaluation.

Internal evaluation can:

  • deepen the scope of external evaluation
  • give a context to the external evaluation
  • provide important insights
  • improve the interpretation of external evaluation findings
  • increase the use of external evaluation findings.

Building evaluation capability

ERO uses its external evaluation process to increase the capability within home-based education and care services to undertake internal evaluation as a routine activity for both accountability and improvement purposes. The intention is for evaluation to become embedded in the day-to-day practice of managers and educators.

ERO builds the evaluation capability of the early childhood service through:

  • making its own external review processes transparent
  • modelling evaluation practice
  • encouraging participation in ERO’s evaluation process
  • having discussions about the service’s internal evaluation processesinvolving nominated service personnel in evaluation design, analysis and synthesis processes
  • providing tools (e.g. examples of evaluation questions and indicators) that services can use in their internal evaluation
  • discussing resources that services can use to help them with internal evaluation.

Internal evaluation in early childhood services

The term internal evaluation is often used synonymously with other terms such as self evaluation, self review, evaluation, assessment, monitoring and appraisal. In early childhood services in Aotearoa New Zealand internal evaluation has its roots in terms such as programme review, internal review and quality review.

ERO defines internal evaluation as the use of robust processes to systematically inquire into and evaluate the effectiveness of policies, programmes and practices. Internal evaluation findings are used to inform decision-making, improve the quality of practice and promote positive outcomes for all children.

Definitions of evaluation vary according to the underpinning theory or theories that they are based on. All evaluation involves reaching judgements or producing knowledge about what is being evaluated. The purpose of each evaluation differs: it can be for accountability purposes, to help with decision making, to contribute to improved effectiveness, to increase understanding or to advance a principle such as equity. This is referred to as evaluation use.

Regulatory requirements for internal evaluation

Early childhood services are required to undertake internal evaluation as part of their licensing requirements. The criteria to assess the governance, management and administration standard specifies that an ongoing process of internal evaluation helps the service maintain and improve the quality of its education and care. 1

The licensing criteria require early childhood services to document:

  • a process for reviewing and evaluating their operation
  • a schedule showing timelines for planned review of different areas of operation
  • recorded outcomes from the review process.

Internal evaluation is an integral part of professional practice in the governance, management and administration of a home-based service.

Guidance and support to help services with internal evaluation

The Ministry of Education resources such as Quality in Action: Te Mahi Whai Hua 2 and The Quality Journey: He Haerenga Whai Hua: Improving quality in early childhood services 3 help early childhood services understand the concept of internal evaluation.

The Ministry has also published guidelines for internal evaluation in early childhood services: Nga Arohaehae Whai Hua. 4 These guidelines set out a process for services to use to undertake internal evaluation. This process is one that requires a systematic process of preparation, data gathering, analysis and decision-making. It implies an evaluative approach.

Highly effective internal evaluation

In 2008 ERO undertook a national evaluation 5 that focused on how well internal evaluation was understood, supported and implemented in early childhood services. ERO found that where internal evaluation was highly effective:

  • improvement and accountability were understood to be the main purposes of internal evaluation
  • management and educators shared the same understandings about internal evaluation
  • it was embedded in practice and integral to the service’s operation
  • reviews had a clear focus
  • it was strongly focused on improvement and with well-established procedures to guide practice
  • reviews were both planned and spontaneous
  • planned reviews included scheduled policy reviews and more in-depth reviews of targeted areas of practice
  • spontaneous reviews were responsive to emerging issues

The factors that emerged as common to all services where internal evaluation was well understood and implemented included:

  • strong leadership to promote internal evaluation
  • professional development to support internal evaluation
  • stable and collaborative staff
  • sound, sustainable systems for internal evaluation
  • the use of relevant resources and support systems.

Types of internal evaluation

Internal evaluation processes let an early childhood service know how well it is promoting positive learning outcomes for children. As a result of information from internal evaluation, the service is able to identify the contributing factors and priorities to enhance children’s wellbeing and learning.

There are different ways of describing the various types of internal evaluation. The Ministry of Education’s guidelines refer to a framework of planned and spontaneous review. The following diagram expands on that framework for internal evaluation by including strategic internal evaluation. Strategic internal evaluation is the process by which the home-based service evaluates how well it is achieving its vision, goals or philosophy.

Diagram 2: Types of internal evaluation

This diagram shows the three types of evaluation, Strategic, Regular and Emergent. They sit within Koru shaped cirlces with Strategic being the outside the largest and regular and emergent in decreasingly smaller koru circles.

Strategic internal evaluation is long term, and focused on key goals related to the early childhood service’s vision, goals or philosophy.

Regular (or planned) internal evaluations are about ‘business as usual’. They are smaller, focused and ongoing, feeding regular information into the strategic internal evaluation.

Emergent (or spontaneous) internal evaluations are in response to unplanned events or issues as they arise. They are one-off spontaneous reviews but should fit with overall goals and link to other reviews.

All internal evaluation involves gathering information which is used as evidence to support judgements and make decisions about service direction and priorities.

Diagram 3: Internal evaluation continuum

This diagram shows the three stages of internal evaluation. The first box is Early stage internal evaluation - box two is Developing internal evaluation and box three is Highly effective well developed evalution. Internal evaluation that meets regulartory requirementts sits between these two stages. ERO's evaluation indicators provide more detailed information about highly effective internal evaluaiton.