Measuring the Impact of Challenges
Impact Canada seeks to understand the impact of challenges at two levels:
- Do the solutions produced by challenges improve socioeconomic and/or environmental outcomes in attributable ways?
- How does the challenge approach itself add value as a public sector instrument, as compared to traditional ways of doing business?
Measuring the Impact of Individual Challenges
The solutions produced by Impact Canada challenges are intended to create public value, in the form of improved socioeconomic or environmental outcomes. Given that Impact Canada is a whole-of-government initiative, these outcomes vary widely from challenge to challenge. As winning solutions are deployed, Impact Canada works with federal partner organizations to evaluate their impact in a rigorous way. At an initiative level, federal organizations report and collect information on the impact challenges have had on achieving their broader objectives.
The Impact Canada guideline, Measuring Impact by Design provides an accessible introduction to the main set of methods that can be used to understand impact, drawing on current best practices in the social sciences.
Measuring Challenges as a Public Sector Instrument
To address the first, Impact Canada has established a partnership with Statistics Canada to conduct a horizontal impact assessment of challenges, using quantitative and qualitative data. All Impact Canada challenges participate in this impact assessment.
The aim of the impact assessment to determine whether and to what extent Impact Canada challenges are successful in achieving a set of process outcomes that they are designed to achieve. In particular, Impact Canada challenges aim to:
- Enhance public awareness of a problem or issue;
- Mobilize new talent;
- Build collaborative networks and partnerships;
- Incent the development of innovative products/ services;
- Enhance skill and capacity (of participants);
- Increase investment (in participants and their solutions); and
- Create greater public value in the form of improved socioeconomic and/or environmental outcomes.
Drawing on the expertise of Statistics Canada, Impact Canada will build evidence over a six-year period, using linked administrative data. These data will tell us to what extent Impact Canada challenge participants collectively achieve these process outcomes, as compared to a similar group of non-participants. This type of analysis will help us to quantify the benefits of the challenge mechanism. Qualitative surveys of participants administered at regular intervals will complement these data. To learn more about the impact assessment, please refer to the Impact Canada Challenge Logic Model.
Approach to Scale
In order to maximize the impact of a challenge, federal partners should think about how to scale innovations beyond the final prize(s). Ultimately, the goal of any challenge is to ensure that solutions are delivered to the people that need them most. Consideration must be given to the opportunities available to scaling support post-challenge in order to ensure that solutions get beyond research and development.
Approaches to scaling will differ depending on the nature of a challenge. Challenge practitioners should think about whether it is it likely that the solutions generated will be from private enterprises, with an ultimate goal to commercialize a new technology or business model, or if the solutions are likely to require a public program or procurement response to scale.