Impact and Innovation Unit Logic model and narrative - Impact assessment of challenges under Impact Canada

1.0 Executive summary

Impact Canada supports a number of initiatives under the umbrella of ‘challenges’. These can include competitive accelerators, grand challenges, challenge prizes, and indigenous innovation initiatives. While these vary in their design features, collectively they represent a set of open innovation initiatives that provide incentives (financial or non-financial) to encourage a broad set of innovators to tackle problems where solutions are not apparent.

This document presents (in a general way) a basic theory of what Impact Canada expects to achieve with its suite of challenges. It emphasizes important processrelated outcomes that distinguish challenge approaches from traditional modes of government business and program delivery (like grant making), and that anticipate the longer-term socio-economic and environmental outcomes that challenges are designed to achieve.

With room to build upon the existing evidence of challenge effectiveness, this document serves as a foundation for an impact assessment of Impact Canada’s suite of challenges, which will help fill gaps in our understanding of whether and how challenges can be effective instruments of policy and program delivery.

2.0 Introduction

Launched in 2017, Impact Canada is a whole-of-government effort designed to help departments and agencies accelerate the adoption of innovative approaches to deliver meaningful results for Canadians. Specifically, Impact Canada champions the use of novel outcomes-based approaches within government. In the area of grants and contributions, Impact Canada’s horizontal terms and conditions enable departments to implement pay-for-success projects within their existing portfolios of grants and contributions. A key means of doing this is through the use of challenges also known as inducement prizes.

Challenges are open innovation approaches, designed to provide incentives (both financial and non-financial) to encourage a broad set of innovators to tackle problems where solutions are not apparent. Instead of prescribing a particular set of specifications, those that issue challenges are most often looking for the most promising solutions to meet a set of defined criteria. There are a number of methods and innovation tools that can be applied to challenges themselves. For example, Impact Canada supports a number of more specific innovation methods under the umbrella of challenges¹,including:

• Competitive Accelerators: Accelerators provide intensive and time limited business support for cohorts of startups, aiming to get them ready for investment more quickly than traditional incubators. Prizes can be layered in to accelerators to reward the most promising entrepreneurs as part of a competitive cohort.


• Grand Challenges: Grand challenges use open and thematic competitions to fund a broad range of potential innovations on a prospective basis, often at an early phase. Those models that prove to be successful sometimes based on rigorous evaluations of effectiveness can be candidates for scaling funding (sometimes referred to as ‘stage-gated challenges’).


• Challenge Prizes: Challenge prizes offer a reward to whoever can first or most effectively meet a defined challenge or solve a specific problem according to a set of objectively verifiable criteria (sometimes referred to as ‘pure prizes’ – however, challenge prizes also usually incorporate phases to move participants through the challenge prize).

Indigenous innovation initiatives

The Government of Canada seeks new ways to build partnerships with Indigenous communities to improve outcomes in relevant ways. The Government of Canada’s approach to designing and delivering programs for Indigenous communities could be improved by working in a co-creative manner with Indigenous partners.

Impact Canada creates space for doing this, by operating on the understanding that those who live with a problem every day are often best placed to solve it. In many ways, Impact Canada’s approach inverts the traditional ‘top-down’ model of government service provision, recognizing that Indigenous communities are themselves valuable sources of innovative ideas that have the potential to develop into impactful solutions to longstanding problems faced by their communities. In other words, a lack of innovative ideas is not the problem, but government can be a better partner to Indigenous communities in ways that reduce barriers to innovation and create the conditions within which they can thrive.

Borrowing elements from challenge-based approaches, Indigenous innovation initiatives under Impact Canada rely less on the competitive aspects of the challenge model, and emphasize instead the importance of Indigenous leadership and collaboration with and among communities. They aim to bring forward ideas inspired by Indigenous knowledge, regardless of their stage of development, while finding pathways to develop these into sustainable solutions that can have meaningful, positive impacts on Indigenous communities.

Building on the past success of the challenge model

While challenges may be a new approach to programming for the Government, they enjoy a lengthy history. Nesta, a global innovation foundation which promotes the use of challenges, traces their use as far back as the Napoleonic wars in France (Nesta, 2014). Their uses in Victorian England to incentivize innovation in the agricultural industry (Brunt, Lerner, & Nicholas, 2012) as well as throughout the 20th century in areas like motorized flight and refrigeration (Davis & Davis, 2004) have also been well documented by researchers. More recently, challenges have enjoyed a renaissance since about 2010, when the Office of Social Innovation and Civic Participation within the Obama White House began promoting their use to solve difficult public policy problems with the launch of Challenge.gov (The White House, 2010).

Despite this rich history, there is room to improve upon the existing evidence of the effectiveness of challenges (Williams, 2012). As their use in public policy continues to proliferate (including in Canada), the need for better evidence of their effectiveness is increasing. Policy makers need a better understanding of challenge effectiveness so that they can be used more strategically. The purpose of this brief document is to clearly articulate in a general way a challenge ‘theory of change’ to support an impact assessment of the current suite of challenges that are planned or in the field under Impact Canada. It synthesizes some of the key literature discussing the primary aims of challenges to arrive at a general theory of what they are meant to achieve, recognizing that no two are identical.

3.0 Challenge Logic Model

4.0 Process Outcomes

4.1 Enhanced public awareness of a problem or issue

Challenges are designed to have a ‘look and feel’ that is very different from the types of funding programs that governments and their usual stakeholders are accustomed to running and participating in. This strategy of openness and transparency has a dual purpose. In the early term, it can have the effect of attracting new talent, and with new talent, new ideas (see below) which may bear fruit as viable solutions to the problems that challenges aim to solve. In this way, enhanced awareness among communities of potential solvers is a critical early success factor for challenges.

Over time, challenges can also have the effect of better engaging the general public on the broader policy issue(s) at hand. By working in the open through a challenge, its host creates opportunities to communicate in a strategic manner with the public on matters that are relevant to its central mandate, in a style that may be more attractive to a broader audience than traditional communications approaches may be. It is possible to think of challenges in this manner as ‘flagship’ initiatives that might themselves have a narrow scope, but which have the potential to shed light on the broader social or environmental issues at hand.

This can help generate greater public demand or social licence for the host to more effectively deliver on its core mandate in the future. In this way, it is possible to think of Natural Resources Canada’s suite of clean technology challenges as vehicles to enhance the communication of the Government’s environmental agenda, Health Canada’s Drug Checking Technology Challenge as a clear commitment to harm reduction, or Indigenous Services Canada’s Indigenous Homes Innovation Initiative as an opportunity to advance reconciliation.

4.2 New talent mobilized

Challenges are generally run in an open and transparent manner as a strategy to encourage broad participation among diverse actors, in order to produce a wide range of potential solutions. They feature low barriers to entry to help entice entrants with lower capacity or those who may otherwise be uninterested in the burden of traditional application and reporting processes associated with many government grant and contribution programs. This mobilization of new talent is a frequently cited desired outcome of challenges, and probably one of the features of a challenge that most clearly distinguishes it from traditional ways of doing government business, like calls for proposals or traditional procurement.

Part of the fundamental rationale for running a challenge is that there is perceived value in uncovering potential solutions to longstanding problems that come from outside the usual pool of actors in a policy area. The key assumption here is that ‘outsiders’ bring new perspectives, skills, ideas, resources etc…, that those well established in an area of policy may be incapable of producing themselves. Some evidence exists to suggest that these types of non-conventional participants are generally motivated to put forward their potential solutions for largely non-financial reasons (Gok, 2013). This suggests that a challenge combining financial and nonfinancial incentives may be an effective way to engage new talent for the purpose of driving innovation for better outcomes.

Most of Impact Canada’s current challenges are concerned with engaging new talent in some way. Health Canada’s Drug Checking Technology Challenge has attracted a range of innovators that its longstanding Substance Use and Addictions Program has not previously funded. Natural Resources Canada’s Women in Cleantech Challenge is expressly designed to not only identify new talent in clean technology industries, but to develop it over time to improve the representation of women with new ideas in a traditionally male dominated field.

4.3 Build collaborative networks and partnerships

Although they are fundamentally competitive processes, challenges often create opportunities to build networks and partnerships among their communities of solvers (and often, end users) as a design feature. Where this can be achieved, it can extend the impact of the challenge further into the future, as new partnerships can yield new opportunities for further impact far beyond the scope of a particular challenge process itself (McKinsey & Company, 2009). Well-designed, challenges create opportunities for collaboration throughout the process in a variety of ways. This can include having a vocal spokesperson who draws the public’s attention to the challenge, which can uncover and draw in other investors/business partners who might otherwise be unaware of the opportunity to partner with new innovators. It can also include the creation of opportunities to have end users or other intermediaries test solutions in real time, creating valuable connections that might yield insights to improve products and services.

Indigenous Services Canada’s Indigenous Homes Innovation Initiative treats collaboration and partnership building as a core feature of its design. Semi-finalist solvers benefit from a structured ‘accelerator’ phase, which will bring the community together over a period of eighteen months to develop their proposals, receive mentorship and benefit from established expertise where relevant, and build new partnerships to help bring their ideas from concept to construction. Infrastructure Canada’s Smart Cities Challenge is explicit in its objective of encouraging communities to engage citizens and forge non-traditional partnerships to improve outcomes for citizens using data and connected technology.

5.0 Sustainable Business Models

5.1 Increased investment

Challenges can drive investment in neglected areas in a few different ways. One way is to design a prize as an incentive to invest in areas where no natural incentive for innovation yet exists (a market failure) (Williams, 2012). Where there is a perception among the community of potential sponsors that the risk of investing in potential solutions is offset by the value of the prize, this can have the effect of generating activity in previously neglected areas where the status quo prevails. Natural Resources Canada’s Crush It! and The Sky’s the Limit! challenges, and the Atlantic Canada Opportunities Agency’s Hull Design Efficiency Challenge are good examples of challenges that aim to do this. In essence, they have created ‘artificial’ market incentives in the form of a prize to incent potential solvers to address long standing inefficiencies in specific industries, in order to drive better environmental and economic outcomes in the longer term.

A second way that challenges can drive innovation is by mobilizing a broad community of solvers around a well-defined issue. Where a prize appeals to a range of possible solvers that includes non-traditional actors (see above), it can appeal to a set of motivations that extend beyond financial interests. Some evidence exists that the appeal of challenges is not limited to the potential of financial gain, but associated with non-financial attributes of participation in the process (Brunt, Lerner, & Nicholas, 2012). Those who might not normally invest their time and effort in a solution to a defined problem might be persuaded to do so because they perceive the challenge process as an opportunity to network or achieve profile, for example (McKinsey & Company, 2009).

This can also mean that a challenge with broad appeal can incent a lot of activity in pursuit of a solution to a problem that perhaps traditional modes of doing government business (e.g., grant making) would not. The nature of a challenge is such that the majority will not win (i.e. the grand prize winner), however many might generate new opportunities for the solvers (i.e. the semi-finalists) that would not have otherwise transpired. Non-winners may develop fruitful partnerships, promising prototypes, new business processes, or attract other investments in their activities that all go on to improve socioeconomic or environmental outcomes in their own ways, regardless of their winning or non-winning status. This is all part of the impact of a well-designed challenge.

5.2 Enhanced skill and capacity

Enhancing the skill and capacities of the community of ‘solvers’ is an example of an important process outcome associated with challenges. Challenges are very often ‘staged’ such that solvers need to graduate through a series of successive ‘gates’ leading to the award of the final prize. While the literature generally sees enhancements to skills and capacities within organizations as ‘by-products’, and things that are generally overlooked by challenge sponsors (McKinsey & Company, 2009), for Impact Canada, this is a critical outcome. Typically, Impact Canada challenges are intentionally designed in a staged manner, seizing opportunities to build the capacity of the entire community of solvers in meaningful ways. A clear example of this is Natural Resources Canada’s Women in Cleantech Challenge, which seeks to address the un>der-representation of women in the clean technology industry. Semi-finalists are provided with a range of supports (both financial and non-financial) to develop their clean technology ideas and business plans over a period of two and a half years before a winner is announced. This intensive process is designed to build capacity among women-led firms. Although one ‘winner’ will emerge, this challenge creates broader impact via the investments it makes in women-led firms involved throughout the process.

5.3 Innovative products, services and/or technologies

Incentivizing the creation of innovative products, especially technologies, is probably the most widely cited desired outcome of challenges, and the primary aim of the majority of past challenges (Williams, 2012). There are several discussions of challenges documenting their lengthy history, and incenting the development of new technologies is a clear theme (McKinsey & Company, 2009). Well known challenges have been used throughout history to develop technologies that ultimately improved motorized flight, better preserved food, and expanded space exploration.

This is a key outcome for many Impact Canada challenges as well. Health Canada’s Drug Checking Technology Challenge seeks to incent the development of technology that will allow those who use drugs to make informed decisions about their consumption, based on the composition of the drug itself. Most of Natural Resources Canada’s challenges are primarily focussed on the development of clean technologies – these include everything from smart grids to improved batteries. These types of challenges are largely seen as a way for government to intervene where there is little or no market incentive to develop a technology that would (if it existed) lead to improved socioeconomic or environmental outcomes that have public value.

6.0 Social, environmental and economic benefits

In the long run, challenges are developed by their hosts because they are perceived as a useful means to achieve strategic socioeconomic or environmental benefits that have public value. They are ultimately meant to serve as a means to help achieve the broader policy outcomes of existing programs. The shorter term outcomes described above that are concerned with process and organizational capacity are seen as useful stepping stones toward the achievement of their more strategic aims, which vary from case to case. Where those early outcomes are seen to be achieved, there is greater potential that the community of solvers developed by the challenge leaves the process better equipped to create public value as they pursue their business in the future.

7.0 Effective and efficient government

The key questions for governments with respect to challenges are whether challenges are effective and efficient. Of these two questions, the effectiveness question is (in theory) more easily addressed. A challenge is effective when and where it achieves the outcomes (as relevant) that are described above in the body of this document, leading to the creation of public value in the form of improved socioeconomic or environmental outcomes.

The answer to the efficiency question is more elusive, but key to government departments which are looking to drive innovation in their policy domains using challenges. In the context of public sector innovation, the onus is on those driving innovation to prove both that the innovation (in this case, a challenge) worked, but also that it worked better than the known alternatives. Challenges take time to develop and implement, are complex, and can draw heavily on internal and external resources. They often run into conflict with established processes and procedures, and raise important questions from corporate services like legal, communications, and finance. There is a need to generate better evidence that the perceived risks of running a challenge, and the level of investment that is required to run them, is justified by the achievement of better outcomes.

There is room to build evidence of this nature (Gok, 2013; Williams, 2012). In theory, the design features of challenges mean that they are well-suited to engaging a broad range of (possibly unconventional) solvers, building sectoral capacity and generating a range of solutions that (whether they win or not) are likely to evolve over time and generate public value in the form of better socioeconomic or environmental outcomes. There is some preliminary empirical evidence that supports this (Brunt, Lerner, & Nicholas, 2012), however the need to build upon this is growing as challenges continue to proliferate in diverse areas of public policy.