Engaging in effective internal evaluation

The whole point of internal evaluation is to assess what is and is not working, and for whom, and then to determine what changes are needed, particularly to advance equity and excellence goals. Internal evaluation involves asking good questions, gathering fit-for-purpose data and information, and then making sense of that information. Much more than a technical process, evaluation is deeply influenced by the school's values and how it sees its role in the community. Effective internal evaluation is always driven by the motivation to improve, to do better for the students.

When internal evaluation is done well, processes are coherent and align with schools' visions and strategic goals. Leaders and teachers work collaboratively across teams, syndicates, departments, faculties and in some cases communities of learning, to ensure that the efforts that go into evaluation lead to improvement. The urgency to improve is shared by all, and can be articulated by all.

Evidence from research and the case study schools shows that there are organisational conditions that support development of the capacity to do and use evaluation for improvement and innovation. These include:

  • evaluation leadership
  • a learning-oriented community of professionals that demonstrates agency in using evaluation for improvement in practice and outcomes
  • opportunity to develop technical evaluation expertise (including access to external expertise)
  • access to, and use of, appropriate tools and methods
  • systems, processes and resources that support purposeful data gathering, knowledge building and decision making.

See the Domain 6 indicators for practices that are characteristic of effective evaluation, inquiry and knowledge building for improvement and innovation and Section 3 of this guide for examples of how the conditions, capability and capacity to undertake effective internal evaluation in a school can be developed. Section 6 of this guide emphasises the importance of using the same set of indicators for both internal and external evaluation.

Scope, depth and focus

Internal evaluations vary greatly in scope, depth and focus depending on the purpose and the context. An evaluation may be strategic, linked to vision, values, goals and targets; or it may be a business-as-usual review of, for example, the curriculum or a learning area; or it may be a response to an unforeseen (emergent) event or issue. Figure 2 shows how these different purposes can all be viewed as part of a common improvement agenda.

FIGURE 2: TYPES OF INTERNAL EVALUATION

This diagram is circluar and at the centre is Learners, surrounding this are three layers of differing koru shapes. Encompassing all of this is a closed circle which is divided into three types of evaluation they are from top clockwise, Strategic, regular and emergent.

Strategic evaluation

Strategic evaluations focus on activities related to the vision, values, goals and targets of the school community. They aim to find out to what extent the vision is being realised, goals and targets achieved, and progress made. For boards of trustees and leaders strategic evaluations are a means of answering such key questions as: To what extent are all our learners experiencing success? To what extent are improvement initiatives making a difference for all learners? How can we do better? Because strategic evaluations delve into matters that affect the school as a whole, and the wider community, they need to be in-depth and they take time.

A strategic evaluation

An intermediate school and a secondary school decided to adopt a joint approach to engaging with their community. In 2007 and 2011 they carried out extensive community consultation to determine what parents and community members considered important outcomes for their children. The results led to the development of seven strategic goals that were reaffirmed in 2011 and 2013, included in the schools' strategic plans, and incorporated into annual goals and targets.

In the intermediate school, a 'story board' approach is used to report on the annual action plan. The principal's reports to the board use a 'traffic light' system to indicate how the school is tracking in relation to its different strategic goals. This reporting is to a schedule, with one strategic goal as the focus each month. This approach keeps board members informed and supports their decision making.

To monitor progress towards its goals the board seeks regular reports from school leaders, asking How well are we doing? What evidence of progress do we have? What is working well? What do we need to adjust and why?

Regular monitoring and reporting enables the board to make resourcing decisions that align with its strategic priorities.

Regular evaluation

Regular (planned) evaluations are business-as-usual evaluations or inquiries, where boards of trustees, leaders and teachers (and, where appropriate, students) gather data, monitor progress towards goals, and assess the effectiveness of programmes or interventions. They ask: To what extent do our policies and practices promote the learning and wellbeing of all students? How fully have we implemented the policies we have put in place to improve outcomes for all learners? How effective are our strategies for accelerating the progress of target learners? Business-as-usual evaluations vary in scope and depth and feed back into the school's strategic and annual plans.

A regular evaluation

The social sciences faculty came to evaluate its junior programme as part of its regular review cycle. Feedback from some senior students had highlighted an apparent lack of alignment between the junior and senior curriculums.

The first step was to find out what the students thought about the social sciences curriculum, so faculty staff developed and carried out a survey. But the resulting data was compromised (some students misunderstood questions and some completed the survey too quickly) and did not give staff the necessary depth of information. They then realised that they lacked the technical expertise to design a survey that would tell them what they needed to know.

Deciding that a different approach was needed, staff set up think tank groups that included students from each class. Teachers presented the survey data to the students and, by discussing it, were able to get in-depth answers to the survey questions.

Following the think tanks there was a lot of discussion within the faculty about the philosophy underpinning the current curriculum, the types of assessments used, and what was and was not working. Staff visited other schools to see their programmes and assess the possible relevance for them of the different approaches. A Google Doc was set up to facilitate collaborative redevelopment of the faculty's programme. A framework for planning units was agreed, and responsibilities for specific topics allocated. Much of this work took place at the faculty's regular professional learning meetings.

Because the level 5 social studies achievement objectives (AOs) are very broad, staff developed explicit curriculum links to senior social sciences subjects - links that the students could understand. A benefit of this process was that it required staff to focus closely on the AOs and ask themselves whether they were really meeting those objectives and providing a platform for student achievement in the senior years. Sometime later, when faculty staff conducted a follow-up evaluation, again using a survey and think tank, the response to the changes was overwhelming positive.

Emergent evaluation

Emergent (or spontaneous) evaluations are a response to an unforeseen event or an issue picked up by routine scanning or monitoring. Possible focus questions include: What is happening? Who for? Is this okay? Should we be concerned? Why? Do we need to take a closer look? Emergent evaluations arise out of high levels of awareness about what is happening for learners.

An emergent evaluation

Leaders and teachers in a primary school found that samples of student writing gathered at the start of the school year revealed that the writing of 42 percent of year 4-6 students deteriorated over the summer holiday break. This decline in performance was observed across ethnicity, year level and teacher. An analysis of November-to-November data showed the students not making the expected progress. Leaders recognised that they needed to do something differently, but that the school had limited influence over what happened in the holidays.

Thinking about what action they might take to improve the situation the staff did some reading on the 'summer effect' but found little of use. So they decided to experiment with an entirely new strategy. This involved collecting a sample of each student's writing at the end of the year, pasting it into the front of their exercise book at the start of the new year, and then making their final learning intention for the old year their first learning intention for the new year.

Expectations are now set on day one. Teachers refer back to the previous year's writing sample and discuss with their students the quality of work that is expected of them, emphasising that this is what they are capable of. Leaders ensure that teachers have the student data from the previous year and that they have the time to determine learning strategies for the start of the year. In this way each teacher can start where the previous teacher left off, without the need to reassess students.

In the first year of implementing this strategy the leadership team rigorously monitored writing data and what teachers were doing in their classrooms, ensuring that the students were writing every day. These strategies have worked well. Now, less than 10 percent of students go backwards in their writing over summer. Explicit teaching strategies target this group with the aim of accelerating their progress.

To think about and discuss

  • What is the focus of evaluation in our school community?
  • Do we use all three types of evaluation (strategic, regular and emergent)?
  • In what areas could we improve our understanding and practice of evaluation?
  • How do we ensure that all evaluation supports our school's vision, values and strategic goals?
  • How do we ensure that all evaluation promotes equitable outcomes for learners?

Processes and reasoning

Internal evaluation requires boards, leaders and teachers to engage in deliberate, systematic processes and reasoning, with improved outcomes for all learners as the ultimate aim.

Those involved collaborate to:

  • investigate and scrutinise practice
  • analyse data and use it to identify priorities for improvement
  • monitor implementation of improvement actions and evaluate their impact
  • generate timely information about progress towards goals and the impact of actions taken.

Figure 3 identifies five interconnected, learner-focused processes that are integral to effective evaluation for improvement.

Figures 4 to 8 unpack each of these processes in terms of the conditions that support their effectiveness, the reasoning involved, and the activities or actions involved.

FIGURE 3: LEARNER-FOCUSED EVALUATION PROCESSES AND REASONINGThis diagram has at the centre a circle which reads Learner focused evaluation processes - We can do better. Surround this are 5 smaller circles which link back to the centre circle with double ended arrows. From top clockwise they are, Monitoring and evaluating impact, Noticing, Investigating, Collaborative sense making and Prioritising to take action.This diagram is the noticing strand and has four boxes with a circle called noticing joining the boxes together. These boxes are from left clockwise- Box 1 -When noticing - we maintain a learner focus, we are aware of what is happening for all learners, we have a deep understanding of the New Zealand curriculum, we have an inquiry habit of mind, we are open to scrutinising our data and we see dissonance and discrepancy as opportunities for deeper inquiry. Box 2- We ask ourselves - whats going on here, for which learners, is this what we expected, is this good, should we be concerned - why, what is the problem or issue and do we need to take a closer look.  Box 3 - If the results dont look good we need to be honest about them, Knowing what the problem is, is critical. Box 4- Noticing involves - scanning, being aware of hunches, gut reactions and anecdotes, knowing whe nto be deliberate and intenional and recognising the context and focus for evaluation - strategic, regular or emergent.This diagram is the investigating strand and has four boxes with a circle called investigating joining the boxes together. These boxes are from left clockwise- Box 1 -When investigating - we focus on what is happening for all our learners, we ensure we have sufficent data to help us respond to our questions, we check that our data is fit for purpose, we actively seek students and parents perspectives and we check we know about all learners in different situations. Box 2- We ask ourselves - what do we already know about this , how do we know this, what do we need to find out, how might we do this and what good questions should we ask and how will we gather relevant and useful data. Box 3 - We try and use data from whatever source we can get it, we had to know how teachers were teaching writing. Box 4- Investigating involves - taking stock, bringing together what we already know data/information, using existing tools or developing new ones to gather data, identifying relevant sources of data/information and seeking different perspectives.This diagram is the collaborative sense making strand and has four boxes with a circle called Collaborative sense making joining the boxes together. These boxes are from left clockwise- Box 1 -When making sense of our data and information- we ensure we have the necesary capability (data literacy) and capacity (people time), we are open to new learning, we know what is so and have determined so what, we know what good looks like so that we can recognise our strengths and areas for improvement and we have a robust evidence base to inform our decision making and prioritising. Box 2- We ask ourselves - what is our data telling us /what insights does it provide, is this good enough, how do we feel aobut what we have found, do we have different interpreations of the data - if so why, what might we need to explore further, what can we learn from research evidence about what good looks like and how close are we to that. Box 3 - We want to know whats good and whats not good enough - to effect change teachers needed to be on board it was not going to be a two meeting process and what are we doing well, what can we improve on and how can we enrich and accelerate the learning of our students. Box 4- Collaborative sense making involves - scrutinsing our data with an open mind, working with different kinds of data both quantiative and qualiative and drawing on research evidence and using suitable frameworks or indicators when analysisng and making sense of our data.This diagram is the prioritising to take action strand and has four boxes with a circle called Prioritising to take action joining the boxes together. These boxes are from left clockwise- Box 1 -When prioritising to take action- we are clear what problem or issue we are trying to solve, we understand what we need to improve, we are determined to achieve equitable progress and outcomes for all students, we are clear about the actions we need to take and why, we have the resources necessary to take action and we have a plan that sets out clear expectations for change.  2- We ask ourselves - what do we need to do and why, what are our opinions, have we faced this situation before, what can we do to ensure better progress and outcomes for more of our learners, how big is the change we have in mind, can we get the outcomes we want within the timeframe we have specified, what strengths do we have to draw on/build on and what support/resources might we need. Box 3 - Prioritising is base on having capacity - you cant stretch yourself too far, Everything we have done has been based on the evidence, we are about wise owls not bandwagons, everything we have done has been decided with data, both qualitative and quantitative and if we keep doing the same things we will keep getting the same results. Box 4- Prioritising to take action involves - considering possible options in light of the what works evidence, being clear about what needs to change and what doesn't, identifying where we have capability and capacity to improve, identifying what external expertise we might need and prioritising our resources to achieve equitable outcomes.This diagram is the Monitoring and evaluating strand and has four boxes with a circle called Monitoring and evaluating impact joining the boxes together. These boxes are from left clockwise- Box 1 -When Monitoring and evaluating- we know what we are aiming to achieve, we are clear about how we will monitor progress, we know how we will recognise progress, we are focused on ensuring all learners have equitable opportunities to learn and we have the capability and capacity to evaluate the impact of our actions.  2- We ask ourselves - what is happening as a result of our improvement actions, what evidence we to have, which of our students are/are not benefiting, how do we know,is this good enough, doe we need to adjust what we are doing, what are we learning and can we use this learning in other areas. Box 3 - Success is stil fragile - if you have a group taht is not achieving to expectations you focus on them and keep focusing on them, results after one year showed a 14% upward shift in students acheiveing at or above the standard. Box 4- Monitoring and evaluating involves - Keeping an eye on the data for evidence of what is/is not working for all learners, having systems, processesand tools in place to track progress and impact, developing progress markers that will help us to know whether we are on the right track, checking in wiht students and their parents and whanau and knowing when to adjust or change actions or strategies

Why not start a discussion about what each of the five evaluation processes might mean in your own school community? This will clarify your thinking about evaluation and evaluation practices and help identify areas where you need to develop greater capability or capacity.

To think about and discuss

  • What processes do we currently use for the purposes of evaluation and review?
  • How do our processes reflect those described above?
  • Which parts of these processes do we do well? How do we know?
  • How might we use the processes described above to improve the quality and effectiveness of our evaluation practice?

Effective evaluation requires us to think deeply about the data and information we gather and what it means in terms of priorities for action. By asking the right questions of ourselves, we will keep the focus on our learners, particularly those for whom current practice is not working. The twin imperatives of excellence and equitable outcomes should always be front and centre whatever it is that we are evaluating.

Organisational conditions and collective capacity

Internal evaluation is most effective when the organisational conditions are supportive and staff members are encouraged to develop the capabilities to do it well. There is no one way of developing these conditions and capabilities, but there are some actions and decisions that are likely to help.

The diagrams that follow (Figures 9 and 10) are based on the experiences of some of the case study schools as they went about developing the conditions and capabilities they needed to engage in effective internal evaluation. Consider using them in your school community to initiate discussion about the extent to which you have practices in place that promote effective evaluation and identify areas for improvement.

To think about and discuss

  • To what extent do the organisational conditions in our school community promote evaluation?
  • How well do our policies, systems and practices embed evaluation that contributes to ongoing improvement for all students?
  • To what extent do we have the relational trust needed to support collaboration and risk taking at all levels of the school community?
  • How deliberate are we in using our internal evaluation processes as opportunities for individual and collective professional learning?

FIGURE 9: ORGANISATIONAL CONDITIONS FOR INTERNAL EVALUATION

How can we develop the organisational conditions to support evaluation and inquiry in our school community?

 

FROM

 

TO

We have not yet developed coherent organisational conditions that support evaluation and inquiry

Our vision, values, goals and priorities are developed, planned and implemented as isolated and disconnected activities rather than underpinned by systematic evaluation and inquiry processes.

We have a limited range of tools and methods to gather, store and retrieve a range of valid and fit-for-purpose data.

We gather feedback from students and our community on a regular basis but are not using this well to identify priorities for inquiry and improvement.

Our efforts to change and improve outcomes for all learners need to be better aligned across the school, building on the 'pockets' of collaboration that already exist.

We are not yet allocating sufficient resources (for example, time, expertise, staffing) to support change and improvement.

Our processes for evaluation and inquiry tend to be ad hoc and disconnected.

We have limited ways of sharing and disseminating new knowledge and this is a barrier to improvement and innovation.

Examples of development/improvement actions taken by case study schools

A collaborative approach to evaluation and reporting is encouraged by involving trustees, leaders and teachers in reviewing student achievement outcomes and identifying areas for strategic focus for the next year.

We draw on an increasing range of information as a starting point for our regular curriculum/syndicate reviews: emerging issues; teacher reflection and feedback; student achievement data; student feedback through surveys and classroom observation; and research.

Student feedback is actively sought to ascertain the success of initiatives and issues for further investigation. Several methods for eliciting student feedback are being trialled including focus groups or think tanks and in-class discussions.

Leaders take time to introduce new processes to build trust necessary for engagement in meaningful inquiry. They keep workloads and change processes manageable.

Leaders have aligned our inquiry and evaluation activities, for example linking teacher appraisal and inquiry projects to maximise the impact on what is happening for identified learners.

Professional learning groups provide a forum to discuss and inquire into particular issues. Teachers explore, test and monitor their responses to their identified issues. New learnings are shared through whole staff feedback sessions.

We have coherent organisational conditions that support evaluation and inquiry

We embed evaluation into the way we plan and take action to realise our vision, values, goals and priorities.

We use appropriate tools and methods to gather, store and retrieve a range of valid and fit-for-purpose data.

We recognise the importance of student and community voice and draw on these perspectives to identify priorities for inquiry and improvement.

We have high levels of relational trust that support collaboration, risk taking, and openness to change and improvement.

We allocate sufficient resources (for example, time, expertise, staffing) to support change and improvement.

We use systematic, coherent and 'smart' evaluation and reasoning processes at all levels of our school.

We share and disseminate new knowledge in ways that promote improvement and innovation.

Figure 10: CAPABILITY AND COLLECTIVE CAPACITY

How can we build the capability and collective capacity to do and use evaluation for improvement?

 

FROM

 

TO

We are yet to build the capability and collective capacity to do and use evaluation for improvement

Our professional learning is focused on teachers' interests and/or the expertise of available professional development providers. We are yet to engage in professional learning that develops the knowledge, skills and confidence needed for evaluation and inquiry.

We have limited or no access to expertise (internal/external) to build our capability and collective capacity in evaluation and inquiry.

Our leaders and teachers are starting to develop data literacy. We do rely on one or two people to manage/analyse and interpret data.

We are not aware of/well informed about current research evidence about what makes the biggest difference for learners.

We are not yet confident in using evaluation and inquiry processes for improvement.

Our use of evidence in decision making is limited.

Examples of development/improvement actions taken by the case study school

Staff meetings are professional forums. The focus is on building professional capability by teachers sharing readings, research and practice.

Leaders are clear about the steps in the change process, framing up questions for teachers to ask of their practice and developing tools to support the inquiry process.

Leaders adopt a more collaborative approach to working with data by increasing the use of data as a focus for discussions in syndicate meetings; conversations between teachers; and within the senior leadership team.

Leaders engage with research evidence and share relevant research with teachers to make better use of evidence in improving teaching practice.

Leaders have a clear focus on embedding evaluation and inquiry processes by supporting team leaders to work with teachers to unpack their data; deepening conversations about data and practice; de-privatising classroom practice through use of video; and extending teacher engagement in professional reading through regular discussions and debate.

We have the capability and collective capacity to do and use evaluation for improvement

We engage in professional learning, mentoring and coaching to develop the knowledge, skills and confidence needed for evaluation and inquiry.

We carefully select and use relevant expertise (internal and external) to build our capability and collective capacity in evaluation and inquiry.

Our leaders and teachers are data literate.

They ask good questions, use relevant data, clarify purposes, recognise sound and unsound evidence, understand statistical concepts, focus on interpretation, and engage in evidence-informed conversations.

When generating solutions and making decisions, we draw on current research evidence about what makes a bigger difference for learners.

Engaging in evaluation contributes to changes in our thinking and behaviour and builds inquiry 'habits of mind'.

Participation in evidence-based decision making builds our efficacy and agency.