Estimated time – three to six months
This phase focuses on setting the design parameters that will support the most effective delivery of a challenge. In general, this will consist of making design decisions about the elements described on the following page.
Selecting a Challenge Type
During the Design phase, it is important to identify the type of challenge that is most suitable to tackle your problem (e.g. challenge prize, grand challenge or competitive accelerator).
Challenge Prize
- The Understand phase led to a specific problem definition where incentivizing new innovators to participate would help solve the issue
- Clear criteria can be established to know when the problem has been solved and/or when measurable progress has been made
Grand Challenge
- Understand phase identified that it is not possible, or not preferable, to identify a prescribed problem statement that would be common for all applications
- Broad evaluation criteria are established to test a variety of innovations
- Once a set of ideas are presented, there would be potential for future iterations of the challenge that could become more focused and specific as a clearer idea emerges of where innovation can make the greatest impact, based on lessons learned from early challenge phases
Competitive Accelerator
- The Understand phase identified that a target group of start-up or early phase entrepreneurs would benefit from business supports to make progress in a given area (e.g. to redress inequities in capital investment among certain groups, to reward emerging firms based on milestones of success)
- Prize funding, layered into traditional accelerator business supports, would accelerate progress towards a goal
Crush It! Challenge
The goal of this challenge was to develop new clean technologies or processes that produce a transformational reduction in how much energy is needed for crushing and grinding mined material at a mill into a usable product for downstream mineral liberation. At the conclusion of the Understand phase, challenge practitioners were able to identify a specific part of the value chain in mining that would benefit from greater innovation and the application of new technologies to improve rock crushing processes. Therefore, a challenge prize was deemed most effective in spurring innovation to meet a specific set of criteria.
Important design features
- A tailored problem statement designed to organize participation around a specific issue – to transform how energy is used for crushing and grinding rocks in the mining industry
- Focus on three specific policy objectives for the mining sector: reduce energy consumption and pollution, increase economic competitiveness, and transform the mining cycle
The Smart Cities Challenge
The aim of the Smart Cities Challenge is to achieve meaningful outcomes for residents by leveraging the fundamental benefits of data and connected technology. At the conclusion of the Understand phase, it was determined that a variety of technologies and approaches would be best applied in a smart cities context. Therefore, a grand challenge approach was chosen to meet these needs.
Important design features
- Open innovation approach, where communities define challenge statements explaining how they would use smart cities approaches to address the most pressing needs identified by residents
- Stage-gated delivery and inclusion of capacity supports
- Outcomes based contribution agreements for implementing winning ideas, as selected by a diverse jury based on a general set of criteria
Women in CleanTech Challenge
This challenge supported the representation of women in the cleantech sector. At the conclusion of the Understand phase, a competitive accelerator format was chosen because it targeted a specific group of innovators that required support to build early stage innovations. Following a national call and expert selection process, six women were identified to participate in an intensive three-year program, during which they will be provided with business advice, and the technical and financial support they need to grow and succeed as cleantech entrepreneurs, including unprecedented access to federal labs and researchers.
Important design features
- Open innovation approach, where communities define challenge statements explaining how they would use smart cities approaches to address the most pressing needs identified by residents
- Stage-gated delivery and inclusion of capacity supports
Drafting a Challenge Statement
Before launching a challenge, it is important to establish a challenge statement (where applicable), which can be refined and made more precise through engagement with key stakeholders.
The challenge statement is the challenge’s call to action in response to the problem that it is trying to address. It provides a clear, concise goal for applicants, and helps them to understand the objectives and parameters of the challenge.
In certain cases, such as a Grand Challenge, specific themes are identified for the challenge overall and applicants submit their own challenge statements defining more specifically how they would meet a broader set of objectives.
In general, a challenge statement includes the objective of the challenge and outlines some of the selection criteria that will be used to assess whether or not the goal has been met. The challenge statement may also include the prize incentives and/or the population or group who will be the expected beneficiaries or end users of the proposed solution. The Impact Canada blog titled “A Strong Challenge Statement Brings Innovators to Your Prize/Challenge” provides more guidance on developing a challenge statement.
Given the interconnectedness between the call to action and other challenge design elements, challenge practitioners should expect to make several iterations of the challenge statement throughout the Design phase. In fact, it is likely that the challenge statement will not be finalized until after final decisions have been made regarding the other challenge elements.
Take a look at this Impact Canada video which demonstrates how the many different iterations of the challenge statement reflect the changing scope of the Drug Checking Technology Challenge.
Example of Challenge Statements
Drug Checking Technology Challenge
Create a rapid, accurate, easy to use, and low-cost testing device or instrument that can be used with minimal training and preparation work.
Hull Design Efficiency Challenge
Develop an innovative hull design that meets the needs of the Atlantic inshore fishery, maximizes energy efficiency, lowers operational costs and reduces greenhouse gas emissions.
Smart Cities Challenge
In a “grand challenge” context, communities were invited to define and set their own challenge statements. For example, the City of Guelph and Wellington County, Ontario, set a challenge statement to “become Canada’s first technology-enabled Circular Food Economy, reimagining an inclusive food‐secure ecosystem that increases access to affordable, nutritious food by 50%, where “waste” becomes a resource, 50 new circular businesses and collaborations are created, and circular economic revenues are increased by 50% by 2025.”
For more information on what Impact Canada expects to achieve with its suite of challenges, please consult “Logic model and narrative - Impact assessment of challenges under Impact Canada” (March 2020).
Structuring a Challenge
The design should also identify how the challenge will be structured. This phase requires sophisticated analysis to determine which outcomes might be achieved at each stage of a given challenge.
Impact Canada challenges typically employ a stage-gated approach where challenge participants receive incentives (both financial and non-financial) at different stages of the challenge. Generally, Impact Canada stage-gated challenges have three stages (semi finalist, finalist, winner). In certain cases, challenge practitioners may choose a pure prize approach in specific cases; however, this is typically not a recommended approach in most contexts.
Pure Prize Structure
A pure prize structure consists of a single round (i.e. from challenge launch to issuing the final prize). This is often the model that comes to mind when thinking about how challenges work, in part because it been used in some high-profile challenges. For example, the pure prize model was used in the first trans-Atlantic flight challenge, which rewarded the $25,000 Orteig Prize to Charles Lindbergh for flying the Spirit of St. Louis from New York to Paris in 1927, and some argue precipitated larger scale investments into long-distance aviation.
However, in practice, these models are likely only applicable in cases where innovators can “self-finance” and absorb the full risks of developing solutions to meet the challenge objectives.
This approach tends to increase the level of risk for participants, may prevent them from participating at all (e.g. risk of zero return on investment of time and resources if they do not win).
This model can also increase the level of risk for challenge practitioners (e.g. no opportunity to refine or re-direct the challenge once it is launched) and limits the ability to understand whether any solutions might be forthcoming, resulting in complex administration and a potential for stranded financial assets, where prize funds could be locked up for an indefinite period of time.
Stage-Gated Structure
A stage-gated structure consists of multiple rounds (e.g. challenge launch, semi-finalists, finalists and winner). Participants are usually provided with support as they move through the process to enable their participation and project development, such as interim funding and/or non-financial support.
Advantages
- Provides additional incentives to participate beyond the financial prize; • Attracts a more inclusive pool of innovators, especially new or emerging innovators; • Builds public awareness and sustains momentum; • Results in a pipeline of more refined solutions and/or innovators that are better positioned to attract funding and support from other sources; and • Allows challenge practitioners to select and filter the best submissions, and refine or re-direct the challenge as necessary
Disadvantages
- Requires time between stages to demonstrate and assess progress; and • Requires a higher level of coordination, administration, and resources.
Setting a Challenge Timeline
It is important to break down a challenge into key segments to support effective delivery, particularly for stagegated challenges. It is important that these segments are clearly and publicly communicated at challenge launch to set appropriate expectations.
Challenges typically have application periods (e.g. initial application, stage 2 application, final application) and assessment periods before prizes or funding are awarded. For challenges with stages, the application and assessment periods will repeat throughout the challenge duration.
Determining the timeline of the application periods requires challenge practitioners to determine the appropriate time likely needed to reach a particular outcome for each stage (e.g. time required from initial call out to submission of applications, time required between stages for the development and testing of solutions).
Time required for assessment periods is typically shorter and depends on a few factors such as the length and complexity of the application, jury time commitment, and requirements for additional experts outside the jury (often technical experts).
Practitioners often engage with subject-matter experts, potential innovators, and jury members to develop a realistic challenge timeline. It is important to build in time for flexibility, in order to make any necessary changes or improvements as the challenge unfolds. Timelines may also need to be adjusted due to unforseen circumstances.
Example of a Challenge Timeline
-
Consultation Phase Opened
July 17th, 2018
-
Consultation Phase Closed
August 17th, 2018
-
Challenge Launched
October 2nd, 2018
-
Application Deadline
February 1st, 2019
-
First-Stage Screening: Up to 10 applicants chosen
April 2019
-
Incubation Period
April to September 2019
Selected applicants further develop prototypes
-
Second-Stage Screening: Up to 5 finalists chosen
December 2019
-
Pilot-Phase
January 2020 to December 2020
Prototypes piloted
-
Grand Prize Winner Announced
Spring 2021
Determining Eligibility Criteria
Eligibility criteria provides guidance on who can apply for a challenge. In general, eligibility should be kept broad in order to open up the problem solving to new players and create the conditions for solutions to be generated from non-traditional actors.
Therefore, it is recommended that barriers to participation are as limited as possible; however, a given organization may have to set certain limitations based on organizational requirements (e.g. Indigenous initiatives or regional specificity).
Under Impact Canada Terms and Conditions, when offering the challenge to international applicants, the outcomes/results of the contribution or grant funding must be of benefit to Canadians (i.e. advance a clear Government of Canada priority or objective).
Example of Eligibility Criteria
Drug Checking Technology Challenge
The Challenge is open to any for-profit and not-for-profit organizations such as companies, industry associations, Indigenous organizations and research associations, as well as post-secondary institutions.
Hull Design Efficiency Challenge
The Challenge is pan-Atlantic in scope and open to small and medium-sized businesses, other for-profit and not-for-profit organizations, Indigenous organizations and groups, and post-secondary/academic institutions registered to do business in Atlantic Canada
Establishing Assessment Criteria
The assessment criteria are the metrics that are used to judge the performance of the participants throughout the challenge to determine the semi-finalists, finalists and, ultimately, the grand prize winner(s). This step is a critical design parameter to ensure the success of a challenge.
The criteria should reflect the objective of the challenge, as described in the challenge statement (if applicable), and clearly describe the conditions that need to be met in order to receive the prize (or pass to the next stage). For example, assessment criteria may include: technical specifications, cost-effectiveness, business sustainability or potential for scale/replication.
Assessment criteria should be clear, concise, unambiguous, and easy-to-understand. Criteria that can be objectively measured (e.g. by using a quantitative benchmark) are preferred, but subjective measures can be used as long as they are well-defined and the standards for adjudication are clear (e.g. assessment of ease of use of a given technology or approach using a ranking framework as determined by qualified judges).
Challenges designers should aim to strike a balance between too many criteria (which may be too difficult for participants to meet and for the jury to assess) and too few (which may be too easy to accomplish and result in limited impact). It is also recommended that assessment criteria be kept the same (or close to) throughout the life of the challenge.
How the assessment criteria will be measured and evaluated is an important consideration for determining which criteria to include in your challenge. As you work through the criteria, consider:
- Do you think new or emerging innovators would be able to meet these criteria?
- Are the criteria likely to prompt new ideas to address the problem?
- How can each criterion be measured? Does baseline data exist?
- How long will it take to measure each criterion? Does this align with the challenge timing?
- What resources (e.g. expertise, data, equipment, testing facilities) would be needed to measure each criterion?
- Do you have access to these resources? Will they be available when you need them?
- How will criterion be scored? Do some criteria need to be weighted over others?
- What type of expertise is needed to assess these criteria (i.e., jury members)?
Example of Assessment Criteria
Hull Design Efficiency Challenge
The challenge statement of the Hull Design Efficiency Challenge was to develop an innovative hull design that meets the needs of the Atlantic inshore fishery, maximizes energy efficiency, lowers operational costs and reduces greenhouse gas emissions. In order to assess whether challenge participants achieved this goal, the following assessment criteria was used:
- Increase in hull efficiency as measured by the power required to move a hull at a given speed with a given engine
- Operational suitability for the Atlantic Canadian inshore fishing industry, considering:
- Safety Standards (Transport Canada Small Fishing Vessel Safety Regulations)
- Appropriateness for operational environment (fishing harbours, Atlantic Canadian sea state, etc.)
- Affordability
- Feasibility of Commercialization Plan, considering:
- Sustainability of commercial model
- Achievable timeline
Setting Prize Incentives
Prize incentives (financial or non-financial) are vitally important for motivating innovators to tackle a problem identified by a challenge.
The federal government can issue financial rewards either as grants (limited restrictions on funding to give innovators maximum flexibility) or contributions (restrictions on use of funds to help guide and focus innovators’ activities). There is no precise science to setting the value of a financial prize, but it should be proportionate to the investment you are asking innovators to make in developing a solution as well as the potential market value of the new product or service. The prize does not need to cover the entire cost of the development but it does need to cover at least some of the risks challenge participants assume.
Financial rewards are often the first thing that captures the interest of would-be innovators. However, financial incentives are not the only thing that motivates people to participate in a challenge. A prize structure that offers non-financial support such as expert support and advice, access to testing facilities, networking opportunities, and exposure to investors can be particularly useful for challenges that require innovators to develop new knowledge and capacities.
Example of Financial and Non-Financial Prize Incentives
Hull Design Efficiency Challenge
Up to 10 Semi-Finalists: Atlantic Canada Opportunities Agency (ACOA) will cover the cost of computer simulation evaluations for up to 10 semi-finalists selected after the first stage screening valued at approximately $6,000 each, to be carried out by the National Research Council’s Ocean, Coastal and River Engineering Research Centre in St. John’s, NL.
Up to 3 Finalists: ACOA will cover the cost of construction and testing of scale models for up to three finalists selected after the computer simulation screening, valued at approximately $60,000 each, to be carried out by the National Research Council’s Ocean, Coastal and River Engineering Research Centre in St. John’s, NL.
Grand Prize Winner: The Grand Prize winner will receive $500,000 to support further basic research and development of their design through pre-commercialization activities, such as prototype testing, etc.
Designing Evaluation and Scaling Plans
While evaluation and scaling of solutions would typically take place after a challenge concludes, it is important to put in place data collection and management plans at the outset of a challenge to support evaluation and impact assessment at later phases. Similarly, understanding how an innovation that is surfaced through a challenge might scale (either through commercialization or through public funding routes) can be helpful in organizing communications and partnership activities, setting prize incentives, and challenge areas of focus, among other elements.
Evaluation Plan
It is important to develop an evaluation approach during the Design phase in order to ensure that the desired challenge outcomes are measurable, and to collect the appropriate baseline data at the outset. The evaluation approach should consider different metrics of relevance, effectiveness and efficiency at the macro and micro level, and not lose sight of the fact that prizes may have different impacts during the competition and in the longer-term. For example, evaluation metrics may include: technological achievements, investment leverage and in-kind contributions, prize participation, entrepreneurship, public perception, and program continuation. Project management tools such as logic models and theories of change may be useful to frame this work.
Measuring Impact by Design is Impact Canada’s guidance for impact measurement, and should be used as a reference tool to ensure evaluation plans meet a basic standard for impact evaluations. Typically evaluation includes key quantitative data to measure impacts, but is also supplemented by literature reviews, interviews and case studies
Scaling Plan
During the Design phase, challenge practitioners should give consideration to what role they play in scaling the impact of the challenge after the final winner(s) is chosen. This may include developing a plan for post challenge support, business investments or partnerships for commercialization, or scaling through mainstream government programs.
A critical design element is considering what opportunities or off-ramps or connections might exist to support innovators who do not win prizes in a challenge but have a solution that could be useful elsewhere. Where opportunities present themselves, Impact Canada’s Centre of Expertise can assist its federal partners to identify external and interdepartmental opportunities to collaborate and scale solutions emerging from challenges.
Intended to be both an accessible introduction to the topic, as well as a reference for those involved in the design, delivery, procurement or appraisal of impact measurement strategies for Impact Canada projects, "Measuring Impact by Design" (2019) was written to guide its readers to think differently about measuring impact than we have traditionally done within the federal public service.
This publication is one of a number of supports that the IIU provides to deliver on its commitment to improve measurement practices for Impact Canada.