Using pilots to measure impact

Using pilots to measure impact: a think-tank event hosted by ERO

Event Details

Date: Thursday 12 July, 2018

Time: 9am-4.20pm

Venue: Kakapo Room, Level 1, Oranga Tamariki, The Aurora Centre, 56 – 66 The Terrace, Wellington

Registrations: Open to government agencies only


Agency - $1000 (plus GST). Price inludes up to five attendees. $200 for each participant over

Individuals - $250 (plus GST)

Contact: For further information or to register, please contact Sankar Ramasamy


Piloting an idea or programme is part of a wider evidence and learning system. Pilots are core to innovation and creativity in public policy and also save time and money in continuing with ineffective programmes. Using pilots to measure impact is a robust way of establishing what works, for whom, why and at what cost. While there is enthusiasm for pilots in New Zealand, getting them up and running can be challenging. There may be confusion about if and when to use pilots to measure impact, how to transition from a pre-pilot to a pilot phase and a lack of guidelines or good practice on how to design and run impact pilots.

We think it is timely to bring public sector agencies together for a strategic conversation on using pilots to measure impact. This event will help policy and delivery agencies better understand, manage and use pilots and their results and embed such approaches within the broader policy development cycle. This think-tank is therefore aimed at agencies’ policy and service delivery staff as well as research and evaluation practitioners.

The key takeaways for participants from this valuable good practice exercise would include understanding:

  • the purposes of pilots as policy and evaluation activities
  • the wider system including incentive structures for enabling pilots
  • the considerations that precede pilot design and implementation
  • when an initiative is ready to be evaluated as a pilot; and
  • identifying what evidence is required for establishing impact and scaling up.

International Keynote Speakers

The workshop will feature two international keynote speakers with extensive experience of understanding and managing pilots.

A photo of keynote presenter Nicholas Mays

Prof Nicholas Mays

Professor of Health Policy and Director, Policy Innovation Research Unit, London School of Hygiene and Tropical Medicine.

Prof Mays has a wealth of experience in the development and evaluation of pilots and demonstration programmes as well as studying the way in which evaluations of pilots are commissioned, designed and used in health and social care in England.

Read nmore about keynote presenter Dr Nandita Verma

Dr Nandita Verma

Senior Associate, MDRC, New York.

Dr Verma is an expert in the design, analysis, and management of evaluation and demonstration projects including complex initiatives such as the Chicago New Communities Program, Opportunity NYC and the Jobs-Plus Demonstration.


The event will have interactive sessions where agencies showcase examples of their practice and share their experiences of the challenges they face and the strategies and innovative approaches they use to overcome them.




Mihi whakatau


Nicholas Pole
Chief Review Officer 


Prof Girol Karacaoglu
School of Government
Victoria University of Wellington


Prof Nicholas Mays
London School of Hygiene and Tropical Medicine


Coffee break


Case studies
Piloting experiences: MSD, Otago University and xx




Dr Nandita Verma


Interactive session on challenges and solutions: break-out into groups




Panel discussion
(Mays, Verma, Chapman)


Rapporteur report


Closing remarks
Nicholas Pole
Chief Review Office, ERO

*Please note, the agenda is subject to change without notice.

ERO sees the workshop as the start of a partnership across the public sector and with decision makers about how to improve our collective understanding and use of pilots and to raise the bar on what counts as robust evidence.

Nicholas Pole, Chief Review Officer, ERO