Impact and Innovation Unit Experimentation direction for Deputy Heads - December 2016

Context

This document reinforces the Government’s commitment to devote a fixed percentage of program funds to experimenting with new approaches and measuring impact to instill a culture of measurement, evaluation and innovation in program and policy design and delivery. It provides context and directions for Deputy Heads on how to implement this commitment.

Definition

For the purpose of implementing the President of the Treasury Board’s mandate letter commitment to “work with [his] colleagues to ensure that they are devoting a fixed percentage of program funds to experimenting with new approaches to existing problems and measuring the impact of their programs”, experimentation is defined as:

Testing new approaches to learn what works and what does not work using a rigorous method that could feature:

• deliberate, thoughtful, and ethical experimental design;
• comparisons between interventions and base cases to capture evidence (e.g., randomized controlled trials, A/B testing, counterfactual experiments, baseline performance data, pre- and post-tests);
• randomized assignment to test and control groups, whenever possible;
• rigorous impact measurement and causality assessment; and
• transparent publication of positive, negative and neutral results.

Features such as randomization, analysis to demonstrate statistical significance of results, and strong counterfactuals may not always be possible; smaller-scale or other experiments that do not meet such requirements should not necessarily be ruled out, but rather every effort should be made to establish as rigorous as possible baseline information and impact assessment strategies.

In this context, new and innovative approaches to be tested include but are not limited to tools and methods such as:

• user-centred design;
• behavioural insights;
• open policy-making, including co-creation approaches to develop policy, programs, and services with stakeholders, civil society, citizens, and other governments;
• staged funding approaches to enable scaling;
• outcomes-based and pay-for-performance funding, including social and development finance and Social Impact Bond-like instruments;
• gamification;
• data analytics and modelling;
• investing in pooled funds that use these tools and methods.

For the purpose of this commitment, experimenting with new approaches may also mean testing an existing intervention in a new context or application. In addition, ongoing or recently launched experimentation initiatives should be included in reporting on departmental experimentation plans and results.

Why experiment?

There is a recognized need for the federal government to innovate and find new ways to address persistent problems that traditional approaches have failed to solve.

Ethical and rigorous experimentation is central to the Government’s focus on evidence-based policy-making, results and delivery. By testing program options, departments can generate evidence to learn what works and inform decision-making.

Experimentation supports sound fiscal management, providing the rigorous evidence the Government needs to stop programs that don’t work and invest public resources where they are likely to have the greatest impact.

Requirements

Deputy Heads are expected to identify a percentage of program funds that their organization is currently devoting or plans to devote to experimenting with new approaches and to report on their experimentation efforts in the 2017-18 Departmental Plan.

Departmental managers are expected to foster work environments that are conducive to experimentation, innovation and intelligent risk-taking so that public servants try new approaches and are not reprimanded for well-managed risks that fail to produce improvements, so long as lessons are captured and reflected in subsequent plans.

Departments are expected to share the results of their experiments, positive, negative or neutral/null, as broadly as possible, with a strong default to public release.

Within the context of their portfolios, and leveraging the Treasury Board Policy on Results, Deputy Heads are expected to establish evaluation and impact measurement strategies to strengthen the experimentation evidence base. Progressively, the TBS Centre of Excellence on Evaluation and evaluators across government will invest in the capacity required to support departments in establishing the necessary strategies and tools to evaluate and measure the impact of experiments and innovation. Likewise, PCO’s Innovation Hub will work with other hubs and labs across federal organizations to support innovation and experimentation.

Central agencies and enabling departments are responsible for establishing or providing access to experimentation training and resources for public servants. Where they have the appropriate expertise and specialized skills, federal hubs and labs are expected to support department-led experiments and disseminate lessons learned. Departments are also encouraged to consult and partner with external experts and organizations, as appropriate, to support high quality, ethical experimentation.

Departmental managers should consider establishing clear processes that will allow them to systematically integrate the lessons from experiments into their programs and course-correct on an ongoing basis. Departments are also encouraged to put in place processes that identify successful experiments for the purposes of scaling-up successful innovations that are able to demonstrate positive outcomes, or replicating successful innovations in other program areas.

Performance information from experiments, regardless of whether they result in demonstrable improved results, will be shared openly by Deputy Heads and results will be reported publicly through Departmental Plans and Departmental Results Reports.

Support from TBS and PCO

TBS and PCO will work to help create the conditions for implementing rigorous experimentation approaches into the core business of departments and agencies. This includes working with Departments to ensure the enabling authorities are in place to support experimentation; helping to build capacity; providing practical tools and resources; and leveraging existing platforms and reporting structures so that departments can track and share experiences and showcase success. To support this work, TBS and PCO will establish an ADM-led interdepartmental mechanism on experimentation. Further details will be made available in January 2017.