Using pilots to measure impact

Using pilots to measure impact: a think-tank event hosted by ERO - 12 July 2018

Download using pilots to measure impact agenda booklet (PDF, file size 441KB)

Event Presentations

Video

Watch Nandita Verma's presentation: Building evidence to inform practice and policy below.

PDF presentations

Click the presentation title below to download as a PDF file.

Girol Karacaoglu Inauguration (PDF, file size 1MB)

IR Case Study Journey With Behavioural Pilots (PDF, file size 1MB)

MSD Case study Intensive Client Support (PDF, file size 1.2MB) 

Nicholas Mays Policy pilots and their evaluation (PDF, file size 1MB)

Philippa Howden-Chapman Using pilots to measure impacts (PDF, file size 1.6MB)

Nandita Verma Building evidence to inform practice and policy (PDF, file size 1MB)

Overview

Piloting an idea or programme is part of a wider evidence and learning system. Pilots are core to innovation and creativity in public policy and also save time and money in continuing with ineffective programmes. Using pilots to measure impact is a robust way of establishing what works, for whom, why and at what cost. While there is enthusiasm for pilots in New Zealand, getting them up and running can be challenging. There may be confusion about if and when to use pilots to measure impact, how to transition from a pre-pilot to a pilot phase and a lack of guidelines or good practice on how to design and run impact pilots.

We think it is timely to bring public sector agencies together for a strategic conversation on using pilots to measure impact. This event will help policy and delivery agencies better understand, manage and use pilots and their results and embed such approaches within the broader policy development cycle. This think-tank is therefore aimed at agencies’ policy and service delivery staff as well as research and evaluation practitioners.

The key takeaways for participants from this valuable good practice exercise would include understanding:

  • the purposes of pilots as policy and evaluation activities
  • the wider system including incentive structures for enabling pilots
  • the considerations that precede pilot design and implementation
  • when an initiative is ready to be evaluated as a pilot; and
  • identifying what evidence is required for establishing impact and scaling up.

Download using pilots to measure impact agenda booklet (PDF, file size 441KB) for more information on the speakers and the event.

ERO sees the workshop as the start of a partnership across the public sector and with decision makers about how to improve our collective understanding and use of pilots and to raise the bar on what counts as robust evidence.

Nicholas Pole, Chief Review Officer, ERO