Capabilities for High Quality Education Evaluation

Introduction

Purposes and uses of the ERO evaluation capabilities framework

The framework defines the capabilities that are needed to undertake and deliver high quality education evaluations in the diverse contexts of Aotearoa New Zealand.

The capabilities are intended to:

• guide professionalisation of our evaluation thinking and practice
• guide provision of our evaluation training and professional development
• assist individual evaluators to self-evaluate their own knowledge, skills and practice; make decisions about and monitor their own development
• underpin quality assurance, both internally and externally
• underpin the credibility and integrity of ERO’s evaluation work
• support high standards and consistency of our professional evaluation practice across the organisation
• be an explicit basis for Individual Development and Performance Management (IDPM) processes.

At the centre of the diagram is a circle marked "The Evaluator". Outside that is another larger circle marked "Personal qualities and dispositions", then another marked "Education and evaluation knowledge and theory". Finally outside that is a circle marked "The Public Servant" that also says "AOTEAROA NEW ZEALAND" and "TE TIRITI O WAITANGI" around the edges. 

Inside the diagram are four transparent circles that each overlap between the outer three circles. They are labeled "Professional leadership
and teamwork", "Interpersonal and communication skills", "Cultural/contextual understanding and responsiveness" and "Evaluation practice".

Outside the outer rim is a decorative koru shape. 

Defining the space that the Education Review Office evaluator occupies

• We are professional, public service, education evaluators.
• We are committed to honouring Te Tiriti o Waitangi and value bicultural partnerships.
• We are guided in our work by our whakataukī: Ko te Tamaiti te Pūtake o te Kaupapa.
• Our evaluation insights are a catalyst for change so that every child achieves success as a lifelong learner.
• We meet the requirements of State Services Commission Standards of Integrity and Conduct, to be: Fair, Impartial, Responsible and Trustworthy.
• We advance ERO’s commitment to achieving equity and excellence for all learners.
• We evaluate the performance of educational services, for purposes of improvement and accountability.
• When we evaluate, we systematically determine the quality, value and significance for their learners of what educational services provide.
• We focus our evaluations firstly on outcomes for learners; then on organisational conditions (processes) for accelerated and sustainable improvement in learner success.
• Our work is informed by knowledge and understanding of education and evaluation, their theory and practice; and of the social, political, cultural and linguistic contexts in which they occur.
• We know about, understand and make adaptive and responsive use of high quality evaluation approaches, methods and techniques.
• We use utilisation-focused and participatory evaluation approaches, and acknowledge the influence of other, related evaluation theories.
• We make well-considered and rigorous use of quantitative and qualitative methods and techniques to ensure the integrity of our evaluations so that our judgements are credible and valid.
• We ensure purposeful interaction between our external evaluation of educational services and their internal evaluation.
• Our work is transparent and well understood.
• We produce high quality, useful and relevant evaluation information which we communicate well for impact.

Context

Education Review Office evaluators professionally exercising their role in the context of:

Domains and Capabilities

1. Evaluation role     

1.1 The public servant

Acts, communicates and practises professionally as a public servant

• Knows and acts according to the legislative, regulatory and policy contexts of education and education evaluation
• Understands and advances Government priorities in relation to ERO’s statutory evaluation role
• Demonstrates political insight
• Advances ERO’s purpose
• Demonstrates the requirements of the State Services Commission Standards of Integrity and Conduct

1.2 Education and evaluation knowledge and theory

Demonstrates education and evaluation knowledge and understanding of theory in relation to:

• the education system and the role of education evaluation in Aotearoa New Zealand
• the system context for education for Māori and Māori medium education
• the diverse social, cultural and linguistic contexts of education and education evaluation
• equity and excellence of education opportunities and outcomes for diverse learners
• effective education practice for improvement and learner success, and its research base
• curriculum, assessment and pedagogy for learner progress and accelerated progress and achievement
• culturally responsive pedagogy for Māori learners: manaakitanga; whanaungatanga; mahi tahi; kotahitanga; whakapapa; ako; wānanga; kaupapa
• evaluation theories and approaches
• evaluation purposes, programme theory, theories of change and logic
• evaluation and research methods and techniques

Te Tiriti o Waitangi

Aotearoa New Zealand’s bicultural heritage and diverse society

Domains and Capabilities

2. Professional practice

2.1 Leadership and teamwork

Leads teams effectively and participates productively as a team member

• Articulates a clear view of self as a professional ERO evaluator
• Develops the professional knowledge, skills and practice of self and others
• Mentors and coaches others to good effect, maximises team and individual capability
• Provides timely and useful feedback and feed forward that supports development
• Conducts constructive team and working relationships, achieves purposeful collaboration
• Solves problems, provides solutions and resolves conflicts
• Makes valuable contributions to the discipline of evaluation and the professional evaluation community

2.2 Evaluation

Uses accepted approaches, methods and techniques* skilfully to ensure evaluation credibility and validity

• Analyses situations, contexts and available data to inform decision making for evaluations
• Designs evaluations that have a clear purpose, are adaptive, prioritised and strategic
• Focuses evaluations firstly on outcomes for learners, and then on organisational conditions for learner success
• Organises and manages evaluation processes that are transparent and well understood
• Ensures purposeful interaction between external and internal evaluation
• Undertakes systematic inquiry to establish a sufficient evidential basis for judgements
• Gathers robust and rich data for system evaluation
• Withholds judgement until the required evidence is gathered and carefully considered
• Makes good use of indicators to assist evaluation thinking, discussion and sense making
• Analyses, synthesises and applies evaluation reasoning to evidence, for meaning making
• Makes clear and fair judgements about educational quality, value and significance for learners
• Constructs a coherent evaluation picture of the performance of an educational service for its learners
• Produces useful and actionable evaluation information
• Communicates evaluation findings effectively, for impact

The public sector and machinery of government

ERO’s role and purpose

Domains and Capabilities

2.3 Contextual and cultural understanding and responsiveness       

Undertakes high quality evaluations that are responsive to culture and context, in terms of:

• cultures, languages and identities
• te ao Māori and ngā tikanga Māori
• bicultural and multicultural contexts

Uses te reo Māori with increasing expertise and confidence

2.4 Interpersonal and communication skills  

Uses interpersonal and communication skills effectively

• Manages evaluation relationships astutely and effectively
• Recognises and manages evaluation anxiety in others and self
• Listens actively to understand, ensures others have the time and space they need to think and speak
• Employs negotiation and conflict resolution skills to good effect when needed
• Facilitates and co-constructs evaluation discussions and activities for authentic and meaningful participation, and evaluation capacity building
• Communicates evaluation information and reasoning clearly and convincingly to promote understanding, ownership and use for improvement
• Communicates well in a range of modes: written, oral, visual, digital; and according to culture and context
• Adapts communication of evaluation information skilfully to meet the needs of diverse audiences and situations

3. Personal qualities and dispositions

• Respects people, their values and ways of being, and upholds their mana
• Responds to people and situations with insight and sensitivity
• Is a reflective practitioner and shows humility
• Is open minded and maintains objectivity
• Thinks critically and acts ethically
• Is open to professional critique, receives and uses feedback for own development
• Is aware of and examines own biases and assumptions, and recognises own limits
• Is self-managing and professionally disciplined
• Takes a considered approach and works well under pressure
• Remains calm and deals productively with ambiguity, uncertainty and complexity
• Shows resilience and resoluteness
• Works in ways that are adaptive, flexible and agile
• Demonstrates self efficacy and agency
• Seeks and accepts help from others when it is needed

Appendix:

* Evaluation theories and approaches, methods and techniques – knowledge and skills of the ERO evaluator

Theories and approaches

utilisation-focused, participatory and collaborative, accountability-focused, democratic, culturally-responsive, kaupapa Māori and indigenous, social justice, empowerment, success case, developmental

Methods

• quantitative, qualitative and mixed
• outcome and impact evaluation
• process evaluation
• formative evaluation
• summative evaluation

Techniques and skills

• programme logic/logic modelling
• theories of change, theories of improvement
• situation analysis and systems thinking
• risk identification and management, problem solving, conflict resolution
• organisation and management, time management, project management, relationship management
• evaluation design and focusing – deciding evaluation questions, evaluation tools and activities, and investigative pathways
• facilitation, co-construction and negotiation
• interviewing, questioning and group discussion facilitation
• listening
• direct observation
• triangulation of information, determining sufficiency of evidence, systematic and focused evidence gathering
• analysis and interpretation of data/information – identification of trends and patterns
• auditing for compliance and accountability
• use of evaluation indicators and criteria
• evaluation reasoning
• synthesis and evaluation meaning making
• formulation of valid evaluation judgements
• oral and written communication, reporting, digital fluency

Our evaluation insights are a catalyst for change so that every child achieves success as a lifelong learner

Ko te Tamaiti te Pūtake o te Kaupapa

The Child – the Heart of the Matter