Summary ReportHow effective are Challenges? Results from Impact Canada Challenge Surveys

1. Summary

Since 2017, Impact Canada has been partnering with federal partners to leverage Challenges as an innovative financing tool to help solve complex problems for Canadians. Impact Canada Challenges differ from other traditional government funding programs as they are designed as an "outcomes-based" approach using financial and non-financial incentives to draw innovators to direct their talents towards finding solutions to specific problems.

Since the introduction of Challenges as a tool in the Government of Canada toolbox, Impact Canada has been working on measuring their impact, as well as improving how they are designed and implemented. Guided by a clear Logic Model, Impact Canada has operationalized both quantitative research with Statistics Canada, and qualitative research, including case studies, surveys and social network analyses.

This summary report focuses on one line of inquiry that Impact Canada is using for impact measurement of Challenges: Stakeholder surveys. In March and April 2022, Impact Canada launched a series of surveys targeting three stakeholder groups: (1) Applicants, (2) Jury panel members, and (3) Departmental partners.

2. About Impact Canada

Impact Canada is part of a comprehensive set of initiatives launched by the Government of Canada in 2017 aimed at solving complex policy challenges using challenge prizes, pay-for-success, and behaviourally based models. Impact Canada enables the use of challenge prizes through a set of flexible Terms and Conditions (Ts&Cs) for Grants and Contributions that allow funding to be linked to outcomes achieved as well as funding rigorous, research based evaluations.

The Impact and Innovation Unit at the Privy Council Office helps departments develop innovative funding opportunities that bring together the best ideas and brightest talent, and reward those organizations and individuals that deliver high impact results for Canadians.

3. Methodology: Surveys

Since the launch of the first set of Challenges, Impact Canada has been developing a framework for measuring and evaluating the impact of Challenges. Guided by a clear Logic Model, which presents the outcomes we seek to achieve through Challenges, Impact Canada has been using multiple lines of inquiry to understand short, medium and long-term outcomes of Challenges. For an overview of the different research methods and approaches we have been leveraging for impact measurement, read a recent blogpost by Kiran Toor. In short, Impact Canada has operationalized both quantitative research with Statistics Canada, and qualitative research, including case studies, surveys and social network analyses.

This mini report is focused on one line of inquiry that Impact Canada is using for impact measurement of Challenges: Stakeholder surveys. In March and April 2022, Impact Canada launched a series of surveys targeting three types of audiences:

1. Departmental survey: Over 140 Government of Canada staff from departments implementing Challenges with Impact Canada.
2. Jury survey: Over 300 experts that reviewed and evaluated Challenge applications and recommended solutions for funding. Most Impact Canada Challenges use an external jury composed of experts bringing together a diversity of backgrounds and perspectives to the assessment process. To learn more about Challenge juries, read this blogpost.
3. Challenge participant survey: Over 2500 applicants and participants in Challenges, including unsuccessful applicants, and those who progressed through challenges.

For operational purposes, the surveys were administered with relatively short timelines for response (data collection took place between March 24 and April 1, 2022), potentially explaining the lower than expected response rate in some categories (between 17–33%).

4. Results: What we’ve learned so far

Overall, survey results demonstrate that the Impact Canada Challenge model is an effective policy approach for delivering on government priorities, with most respondents acknowledging that challenges are delivering on their intended outcomes.

Are Challenges viewed as appropriate to address the problem area?

66% of jury respondents and 64% of department respondents indicated that a Challenge was the appropriate mechanism to address the problem area to a very large or large extent. Further, only 2% of jury respondents and 8% of department respondents indicated that the Challenge was a minimally or not at all effective mechanism to address the problem area. This large proportion of respondents confirming that the use of a Challenge was appropriate to address the problem area, is a good indication of the usefulness of Challenges as a tool in Government of Canada toolkit to address complex problems.

Are Challenges meeting the needs of Canadian innovators?

While departmental respondents held positive views that Impact Canada Challenges are meeting stated objectives, Challenge applicants and jurors, had even stronger positive views. This is an indication that Challenges may be more closely meeting the needs of Canadians than anticipated by our departmental partners.

For example, applicants and jurors placed a higher value for the role of Challenges in raising awareness, creating innovative solutions, enhancing solvers’ skills, increasing investments, and achieving long term outcomes.

Jurors also placed higher value on improving the assessment of applications, better aligning funding decision making to industry needs, increasing transparency, and democratizing funding decision making.

Who is participating in Challenges? And why?

The survey results show that Challenges are successful in drawing new talent and engaging non-traditional actors, with above-average proportions of youth, women, and minorities applying to Challenges. For example, youth (35 years old or under) represented nearly one fifth of Challenge applicants that responded to the survey; women were nearly one fourth, entrepreneurs from visible minorities or indigenous groups were nearly 40%, and entrepreneurs living with a disability (e.g., physical, mental, or other long-term condition) were 7% of applicants.

There is also evidence that Impact Canada Challenges have contributed to mobilizing new talent to solving complex problems; talent that might not have been mobilized otherwise. Nearly one-fourth (23%) of applicant respondents indicated that the Challenge was the sole motivation for them to work on the specific problem area and nearly half (48%) indicated that the Challenge motivated them to work on the problem to a large extent.

Among the applicants, 16% had minimal or no knowledge of the problem area prior to the launch of the Challenge.

With respect to the sector/domain, 19% of applicant respondents indicated that they were previously working in a different sector than the problem area.

In addition, 18% of applicant respondents indicated that they developed their solutions after launch of the Challenge.

For applicants who were working on their solutions prior to the Challenge, 72% admitted having made improvements and changes to their solution to apply to the Challenge.

Who applies to Impact Canada Challenges?

There is a wide range of applicants to Impact Canada Challenges. At the time of application, only about half of respondents (52%) were businesses and over one-fifth (22%) were not-for-profits. Under one-fifth (18%) were individuals or group of individuals, many of whome ended up incorporating into companies or not-for-profits. Applicants also included academic institutions (4%), indigenous groups (3%), and government organizations (1%, mostly municipalities that applied to the Smart Cities Challenge).

More than two-thirds (69%) had never applied to government funding before, resulting in Impact Canada Challenges being their first opportunity to seek government financial support for their work.

The majority (89%) of applicant respondents were considered small businesses (defined by Industry Canada as having less than 100 employees) and half (50%) had less than 10 employees. Only 4% of applicants were large enterprises (over 500 employees), and most were recently established as organizations.

More than half (52%) were established less than 5 years prior to Challenge launch and one-tenth (11%) were established after Challenge launch (i.e., organizations incorporated to participate in the Challenge).

What motivates applicants?

Applicants to Challenges are more motivated by working on the Challenge problem than by the financial reward. Successful applicants in particular, were more likely to value the non-financial support associated with Challenges (e.g., partnerships, access to experts, mentoring and skills development).

5. What's next

Conducting a survey was an effective way of not only collecting information where data gaps exist for impact measurement purposes, but it was also beneficial to obtain feedback on ways to enhance Impact Canada Challenges, as well as collect information on our partners’ experience participating . Having the ability of curating questionnaires to particular policy needs makes surveys a method that complements other modes of inquiry well.

Stay tuned to this site for more on how Impact Canada is exploring different methods to measure the impact of our work and evaluate Challenges as a public sector instrument.