Impact Canada, Impact & Innovation Unit, Privy Council Office
This month, the Impact and Innovation Unit is proud to launch our first ever impact measurement guidance, which we call Measuring Impact by Design. This document is the product of a year’s worth of collaborative effort, in which we worked with experts and potential users of this guide to compile a plain-language explanation of some key concepts related to impact measurement. It introduces some of the main ways in which we can measure impact for the projects we will be delivering under Impact Canada.
At its core, Impact Canada is about doing things differently to improve results for Canadians. Pay-for-success is a key means by which we do this, and as you will have seen, we’ve already launched a series of challenge prizes in diverse areas to this end. Pay-for-success can take many forms, and we have some exciting work planned to develop social impact bonds as well.
Measuring Impact by Design was written to equip ourselves and our partners with a general guideline for rigorous impact measurement. The basic premise of pay-for-success is that government only pays when a desired result is achieved. In this context, we need to be proactive and systematic in our measurement efforts, and use designs that can attribute results to our programs, ruling out other possible causes as much as possible. To this end, Measuring Impact by Design places a strong emphasis on experimental and quasi-experimental methods for measuring impact.
Fundamentally, this is a guide for the non-specialist. It is written for program and policy experts who are developing Impact Canada pay-for-success projects, who might not know what methods of impact measurement are available to them, what the advantages and disadvantages of various methods are, or which are best suited to which contexts. The goal is not to turn its readers into specialists, but rather to equip them with a deeper appreciation of the science of impact measurement, so that they can be critical consumers of it in their own pay-for-success work.
This is just the beginning. Over the coming months, watch for our impact measurement blog post series, which will dive a little deeper into the key messages of Measuring Impact by Design. We’ll be writing a little more on topics like randomisation and impact measurement, the relative advantages of experiments and quasi-experiments, why we don’t use a traditional evidence hierarchy, what the relationship between impact measurement and pay-for-success is, and how we are working with Statistics Canada to measure the impact of our own work on prizes and challenges.
So stay tuned! There is much more to come. In the meantime, we hope Measuring Impact by Design is useful to you. We love feedback, so please do not hesitate to reach out to us.
How Can Challenge Prizes Help Advance Broader Policy Objectives?
Julie Greene, Lead, Capacity and Partnerships
Impact Canada, Impact & Innovation Unit, Privy Council Office
In 2017, the Canadian Government took bold and concerted strides to accelerate the use of innovative approaches to improve outcomes on issues that citizens care about, when it established the Impact and Innovation Unit (IIU). Situated within a central agency — the Privy Council Office — we were asked to experiment with new program delivery models and assess whether they help close the gap between policy development and implementation.
Under the auspices of Impact Canada (a part of the IIU), the principal approaches we use are challenge prizes, pay-for-success funding models, and behavioural insights. Since launching in 2017, Impact Canada has experienced rapid growth, with over $720 million of funding under its program authorities and a significant portfolio of behavioral insights projects. To read more about what we do and how we do it, please have a look at our annual report.
My name is Julie Greene, and I have the privilege of being a member of the Centre of Expertise for Impact Canada. Nesta Challenges has been an integral partner for our team, guiding and supporting our efforts in co-designing challenge prizes with some of Impact Canada’s first clients.
In this blog, I want to reflect on something important we have learned while scoping, designing, launching and assessing challenge prizes over the last two years. When thinking about why a program or department might want to run a Challenge, we usually consider some of the following factors:
Because there is an unknown solution to a known problem;
To support and accelerate change in an issue area;
To attract new innovators to the issue and;
To build new markets or innovations developed as a result of challenges.
Those are often our opening considerations when investigating a problem area that we might address or advance through a challenge prize.
As we have worked with multiple departments to launch Challenges that address issues of key importance to Canadians — creating smarter cities and clean technologies, improving Indigenous housing and responding to the opioid crisis — we are learning that when each partner decides to spend the extensive time and energy it can require to define and launch a well-researched and designed challenge, they have considered what might be accomplished beyond the solutions the Challenge may surface.
At the outset of planning, and until first stage assessment, the possible solutions that a Challenge may offer are unknown. We have defined, to the best of our ability, or (as we have heard Nesta’s Olivier Usher say many times), (insert Scottish brogue here) “we are reasonably certain”, that we understand the problem, current approaches, barriers to success and that a gap or opportunity exists that can be helped using a challenge prize. But we have no idea at that time what will actually happen when the Challenge is launched! So, as they consider the rationale for undertaking a Challenge, there are broader policy objectives that our partners want to express or explore through launching a prize. Here are a two examples.
Health Canada is the lead for Opioid Response, and Canada is in the midst of a crisis. The number of overdoses and deaths caused by opioids, including fentanyl, has risen sharply and continues to rise. We launched the Drug Checking Technology Challenge to improve on drug checking technology that will allow the community of people who use drugs and those who support them to make more informed decisions based on the composition of a drug. This opening statement on the Challenge platform sends a very clear and public signal that harm reduction (measures that reduce the negative effects of drugs and substances on individuals and communities, without requiring abstinence) is a key pillar of the Government’s approach. While critics of harm reduction believe that this approach normalizes risky behaviour, or delays movement toward treatment and abstinence, the Canadian Drugs and Substances Strategy “supports measures that reduce the harmful health, social and economic effects of substance use”.
Launching a Challenge to “create a rapid, accurate, easy to use, and low-cost testing device or instrument that can be used with minimal training and preparation work” sends a clear and direct message that our policy priority is to reduce death by opioid overdose, using all available levers. What more public way to communicate this priority than by inviting innovators of all kinds to help stop overdoses? Because as Tris Dyson says, “We want to use these prizes to solve these problems faster and better — making the most of talent, ideas and motivations out there, wherever they may be”.
Another example of how Challenges can help to communicate and advance broader priorities comes from Canada’s East coast, where commercial fishing and boatbuilding are important economic drivers for the region. Atlantic Canada is home to nearly 13,000 inshore fishing boats that use diesel fuel to power them to lucrative fishing grounds. In support of both better economic and environmental outcomes, the Atlantic Canada Opportunities Agency launched a Challenge to boat builders to develop an innovative hull that maximizes energy efficiency, lowers operational costs and reduces greenhouse gas emissions.
And yet, the Hull Design Efficiency Challenge has greater aspirations — to again send a clear and transparent signal that there is room to incentivize the acceleration and uptake of clean technologies in an industry which has traditionally relied on “tried and true” methods, often handed down over generations. Adoption of new, novel and more efficient approaches may face barriers to uptake including those are both financial and attitudinal. In this case, by designing a Challenge with specific supports and incentives that will move ideas toward commercialization, the Government is defining a clear space for innovation in clean technologies, and respectfully incenting more rapid progress in an industry almost as old as time itself.
When considering whether a challenge prize might be the right mechanism to help advance progress on your policy issue, it is also worth thinking about the ways this approach can help clearly articulate and communicate your policy positions to your target innovators, stakeholders and larger public.
So ask yourself, what ELSE can challenge prizes do for you?
This blog was originally published on November 07, 2019 by Nesta Challenges.
Published: 2018-11-19
Last modified: 2025-05-25
Blog
The Impacts of Innovation
Measuring the Effects of Challenges
Noah Conrad, Junior Analyst
Impact Canada, Impact & Innovation Unit, Privy Council Office
What are Challenges? What are their desired outcomes
Impact Canada Challenges are outcomes-based competitions run by the Impact and Innovation Unit (IIU), in partnership with other government departments to find solutions to some of Canada’s most pressing issues. Using an open innovation approach, challenges leverage financial and non-financial incentives to entice innovators to direct their talents towards helping to solve major societal problems. Challenge applicants typically apply to a staged competition where the ultimate winners are awarded large financial prizes.
My name is Noah Conrad, and for the last four months, I have been supporting the IIU in the measurement of challenges. What follows is a brief overview of the challenge impact measurement approach developed by the IIU, and some options being considered for future use.
Why is the IIU Measuring Impact?
As challenges are completed, formally measuring their direct and indirect effects will help the IIU to assess whether challenges are having their desired impacts and whether they are being designed and delivered in the most effective manner. Indeed, to continually enhance its programs, the IIU needs to understand the aspects of its methods that are functioning as intended and the ones that require further refinement. Moreover, given the monetary prizes associated with each challenge, formally measuring the effects of challenges will help maximize their effectiveness and efficiency within a public policy context.
Barriers to overcome
To date, there have been relatively few recent studies that have rigorously studied the impacts of challenges. By their very nature, challenges work on multifaceted problems that may not have one single cause or solution. Consequently, it can be difficult to attribute positive changes within an issue area to challenges themselves when many other variables and factors could have influenced the observed outcomes. Moreover, since each challenge works on a distinct problem and can employ one of three basic structures (challenge prizes, grand challenges, and competitive accelerators), all challenges are likely to have somewhat divergent pathways to impact, making it hard to develop a standardized approach to impact measurement. For example, one challenge may impact an issue by making high quality information more readily available (see Housing Supply Round One) while others may change an area by incentivizing a technological breakthrough (see Sky’s the Limit Challenge). Clearly then, determining the causal impacts of challenges will necessitate the use of multiple methods and lines of inquiry. The following sections can be read as a limited subset of the approaches to impact measurement that the IIU has employed to date and is considering in the future.
Approaches to Impact Measurement
In 2020, the IIU published a document called Logic Model and Narrative — Impact Assessment of Challenges under Impact Canada, which presents a basic theory of what we expect to achieve with our suite of challenges. Each of the methods of inquiry below is discussed in relation to the outcomes from the logic model that it is designed to measure.
Social Network Analysis: Measuring increased awareness, talent mobilization, and the formation of collaborative networks and partnerships
A desired process outcome from the logic model is the formation of partnerships both internal and external to the challenges themselves. Indeed, studies have shown that the formation of knowledge communities and networks through which actors exchange information are an important driver of innovation. As part of their design, challenges have developed various opportunities for participants to form connections with end users, potential financiers, local communities, and other competitors. Ideally, connections with financiers should help attract initial investments into start-ups while networking with fellow participants may assist in the exchange of ideas. Additionally, challenges should raise attention and generate increased awareness about important issues. Open communication from Impact Canada with strong engagement from innovators, end users, and the media contribute to raising the public profile of policy priorities and problems that a challenge has been designed to address or solve.
Another primary motivation behind the creation of challenges is to mobilize actors who do not typically apply to government programs as well as those who lie outside the normal constituency that may develop solutions to the issue in question. Challenges feature a streamlined application process designed to incentivize non-traditional actors to apply to them. It is expected that new actors who apply to challenges will bring new ideas, capacities, and experiences. These unconventional ways of thinking may help uncover new solutions to the difficult problems that challenges are designed to solve.
A line of inquiry which could help the IIU to measure all three of these outcomes is social network analysis. For those unfamiliar, social network analysis is a research method which outlines the exchanges and connections between actorsFootnote 1. Undertaking social network analysis would help paint a picture of the kinds of actors that are engaging in Impact Canada challenges as well as the interconnections between them. Beyond illustrating the connections between actors, social network analysis could reveal the relative centrality and importance of different actors within a network.
Surveys: Measuring the Mobilization of Talent
In addition to social network analysis, the IIU is making use of surveys to determine whether challenges are succeeding in attracting a wide variety of innovators. Surveys can help gather important information about applicants to understand who applies to challenges and what some of their motivations are. Some of the variables that can be analyzed from a survey include the age and gender distribution, location, background, and prior activity of applicants. Analysis of these variables will allow the IIU to measure the effectiveness of challenges in attracting new talent.
Quasi-Experimental Approaches: Measuring Economic Value
By de-risking investments into nascent technologies, challenges should help participants to raise new capital and financing. Evidence from a Nesta case study has shown that challenges are capable of helping participants generate investment into their products and solutionsFootnote 2. Moreover, the staged approach that Impact Canada has taken to challenges in which non-winners are provided with various forms of support should help improve economic outcomes even amongst non-grand-prize-winners. Determining whether challenges have succeeded in generating greater public value will require the use of quasi-experimental approaches in which counter-factual outcomes from a control group are compared to the outcomes seen from applicants. By determining whether participants have experienced increases in economic indicators such as R&D spending, productivity, payroll spending, exports, patents, intangible assets, revenue, capital raised, and job creation, the IIU will be able to determine whether challenges have supported sustainable, innovative businesses that have experienced positive economic outcomes. Some initial work in this direction has begun in partnership with Statistics Canada (StatsCan). Moving forwards, the IIU will continue to collaborate with StatsCan to use data available in its Linkable File Environment to determine the most effective ways to measure the economic impacts of challenges while accounting for relevant covariates that could influence outcomes.Footnote 3
Short-Term Case Studies: Examining the Innovation Process
Many innovation oriented organizations have used case studies to evaluate the impacts of their programs. One reason for launching challenges is to change the incentives and conditions that currently exist around innovation in Canada. For example, the Women in Clean Tech challenge seeks to strengthen the innovation ecosystem in clean tech by creating a space for talented female entrepreneurs to develop and cultivate their ideas. In light of these goals, the IIU will consider using case studies that examine the impact of challenges on the innovation process. A pioneering approach here has been the short-term case study undertaken by Sitra of their challenge Ratkaisu 100Footnote 4 which documents the impact of a challenge on participants as they are going through it. To date, the IIU has produced two case studies and will be publishing more in the future.
Long-Term Impact Studies: Measuring Public Value
As solutions developed throughout challenges come to be deployed in the real world, it will become imperative to examine the concrete impacts that these solutions are having on their respective issue areas. Through a longer term impact study, the IIU could examine the ways in which technologies, products, and services developed during challenges are being utilized. For example, if data eventually becomes available on the number of daily users the winning solution from the Drug Checking Technology Challenge has, the IIU could conduct statistical analysis about how the technology has been deployed to prevent overdoses within specific geographic areas. Each long-term impact study is likely to be very different given divergent pathways to impact that every challenge is likely to have. Nonetheless, long-term impact studies will ideally show how challenges have had an influence on some of Canada’s must pressing issues, responding to the longer-term outcomes of the challenges logic model, focused on societal, environmental and economic outcomes.
Conclusion
As the IIU accumulates evidence on the effects of challenges, this knowledge will be used to understand the process and outcomes of challenges and to improve future challenges and initiatives. However, the importance of impact measurement extends beyond the need for institutional learning. Over the coming years, the insights obtained during impact measurement will help tell the story about how challenges have played a role in improving the lives of Canadians.
Published: 2021-12-20
Last modified: 2025-05-25
Blog
Health Canada’s Drug Checking Technology Challenge
Julie Greene, Lead, Capacity and Partnerships
Impact Canada, Impact & Innovation Unit, Privy Council Office
On October 2, 2018, in partnership with Health Canada, we launched the Impact Canada Drug Checking Technology Challenge to “create a rapid, accurate, easy to use, and low-cost testing device or instrument that can be used with minimal training and preparation work”. Our goal is to serve people who use drugs and those who support them, to allow them to make decisions that may reduce their risk of overdose and death.
Over the last year, a small group of public servants has been working to scope, define and launch this Challenge. Dr. Michael Haber, a Health Canada scientist and Government of Canada Entrepreneur, is one of the key Challenge team members. Through his blog on Medium, Mike has very thoughtfully reflected on the how and why of planning and launching this Challenge.
Prize-Challenges and other forms of outcomes-based approaches are a new way of working for Government, and unlike many of our processes, no one has done this a hundred times already. It’s important that we capture and share our experiences in this brave new world, and build communities of the curious and the willing.
If you count yourself in these communities, I encourage you to read Mike’s blog. We are also working on a visual illustration of Challenge design and planning, called, Evolution of a Challenge: From Idea to Launch, so keep following the Impact and Innovation Unit website and Twitter for fresh content, and as the Drug Checking Technology Challenge unfolds over the next year.
A Thankful Reflection on a Positive Onboarding Experience
Jacob Graham, Behavioural Science Fellow
Impact Canada, Impact & Innovation Unit, Privy Council Office
Hi. I’m Jacob Graham. February 15th marked my third full month as an Impact Canada Behavioural Science Fellow. The experience of being employed as a federal public servant working within a cutting-edge field of research at the forefront of policy development has been nothing short of fantastic, spectacular, and — at times — overwhelming. The last few months have been a blur, but I would love to take the opportunity to digest and articulate exactly what it is that has made the first few months of the Fellowship so special.
The Process
I applied to the Fellowship program in April of 2021 — in the midst of final exams for my Master’s degree, on an impulse that I really ought to have something lined up for after I graduate from the academic world. I shared the concern that many students have; that they will end up circling job boards and interview postings after graduation with the hopes of landing any job reasonably within their field of study. What I did not share with many other students, however, were the same struggles of locating a position. By some shred of luck for which I am eternally grateful, the first job posting that I managed to come across was that of the Impact Canada Fellowship program. It advertised a position in behavioural science — a broad field of study under which my graduate behavioural economics research fell.
By August of 2021, after passing a series of interviews and technical skills tests, I was notified that I had successfully pre-qualified for to Impact Canada’s Behavioural Science stream. By October 2021, I was notified of a potential placement, and by November 15th I was official onboarding.
The onboarding for Behavioural Science Fellows should be considered the industry gold-standard. The rigorous two-week process saw me engage with members of several teams and projects across various areas of expertise, and included opportunities for me to ask questions, meet other Fellows both formally and informally, and made sure to leave time for reading and training. Project spotlights offered in-depth dives at some of the work ongoing under the Impact Canada mandate, while icebreaker activities and fireside chats gave me the chance to meet the people behind those engaging projects. The onboarding team made conscious efforts to check-in with my progress and well-being through the 14 days, and were excellent at addressing any concerns I had.
The entire process — from application to evaluation to pre-qualification to placement to employment — defied everything that I thought I knew about how government works. It was efficient, it was human, and it was enjoyable.
The People
Academia is wrought with intelligent people feeling as though they are outsiders. The constant feeling of being “the dumbest in the room” was no stranger to my experience at graduate school. Knowing the feelings that Imposter Syndrome brings, I was wary of how my experience in the Fellowship program may echo my experience in academia.
I found, however, that this was not the case. Although the Impact Canada team is full of experts in the field of Behavioural Science and some of the brightest minds in the country in this realm, there is no latent feeling of being unwelcome due to a lack of knowledge or understanding. The team is wrought with expertise on several subjects, but just as prevalent is a willingness to educate and take the time to explain these subjects. Where academia closes doors to those with lay-understanding, the Impact Canada team is inclusive and welcoming to those who are hungry to learn, encouraging dialogue in a safe and investigative environment. I have found support for not only my projects but also my personal wellbeing and growth from my managers at Impact Canada and at my host organization. Recently on-boarded Fellows have been candid and honest about their experiences and have been available for questions at all stages of my early employment. New Fellows joining the program should feel at peace knowing that such a welcoming environment awaits them.
The Practice
The first few months of my placement have included important steps meant to alleviate any natural uncertainty that comes with a new position. These activities include meeting with the Innovation and Youth Engagement Division (IYED) within Environment and Climate Change Canada (ECCC) where my placement is. The team at IYED are extremely warm and welcoming, and have done a fantastic job making me feel at home.
The work I will support on is undertaken by the Product Policy Section (PPS) of the Environmental Protection Branch (EPB) within ECCC. My research will investigate behavioural challenges surrounding the consumption of products with chemicals such as plastics, with a focus on determining how to provide better or more comprehensive product information via paper and digital labelling. The objectives of these research activities is to better understand Canadians’ explicit attitudes and knowledge as well as implicit biases surrounding the consumption of products with toxics such as plastics, and seeks to answer the question of how Canadian consumers can be encouraged to purchase products that are safer for themselves and the environment. These insights from these research activities will support PPS with their work, as well as contribute to the Impact Canada’s Program of Applied Research on Climate Action in Canada (PARCA) as deep-dive investigative studies.
While much of this work is still under development, what I have been able to contribute on so far has been thoroughly enjoyable and stimulating. As my Fellowship continues to develop and my work begins to mature, I look forward not only to engaging in exciting research but also sharing the results.
The one of Bambi, with long bony legs splayed in four different directions, struggling to stand on a frozen pond? It has come to mind a few times over the past year: During the first skate on the canal in Ottawa, trying to stand-up paddle board for the first time, and this fall, helping to set up the new Office of Behavioural Science (BeSciO) at the Public Health Agency of Canada (PHAC).
BeSciO Vision Statement: Ensuring that behavioural science evidence is fully translated and integrated across policy, communication, and programming interventions supporting PHAC’s revitalization as a world leading, data-driven public health organization.
We are Tyler Good and Sarah Wall, two Impact Canada Fellows working in close collaboration with BeSciO. In April 2021, the President of PHAC approved the establishment of the Office of Behavioural Science (BeSciO) to act as a centralized hub that provides expertise and helps build capacity and community for behavioural science (BeSci) throughout the Agency. BeSci has been used within teams across PHAC for years, but the valuable ways that it contributed to the COVID-19 pandemic response demonstrated the need for a central office. You can read more about how the Fellowship program supported the pandemic response in this blog post and by checking out our project timeline.
Tyler Good: I joined BeSciO in June 2021, the fourth employee in the door. In these early days, our favourite way to describe how things were going was to say that we were building the plane while flying it. I have never worked at a start-up, but I think BeSciO probably resembled one. We were a small team trying to make progress on lots of different fronts, so this was a fantastic opportunity for me to see how decisions are made and teams are built in the federal public service. Some highlights included responding to rapid review requests about COVID-19, supporting longer-term COVID-19 related research projects lead by my peers embedded within other teams at PHAC, and participating in our first project scoping exercise by identifying and engaging teams across PHAC to develop relationships and prioritize research opportunities.
Sarah Wall: I joined PHAC in March 2021 as one of the Fellows who was embedded into program areas on the frontlines of the pandemic response. In October 2021, we were brought into BeSciO to continue to formalize our presence and processes within PHAC, broaden our partnerships to other areas of public health, all while still supporting our original host program areas as clients. It has been a unique and rewarding experience to be a part of the initial BeSciO team. I have been able to directly apply my lessons learned, maintain and expand my networks of colleagues, and further develop as a BeSci Practitioner. Some highlights include leading a study related to public health measures, writing a report on how BeSci can inform communications during COVID-19, developing a BeSciO playbook, advising on several communication campaigns and public health education products, and providing support to our CPHO.
We have learned a lot in these early days at BeSciO. We’ve watched our Manager and Director make strategic decisions when working with partners and establishing team culture. We’ve participated in meetings where we introduced BeSciO to other teams within PHAC, which required us to first identify how we might be able to contribute and then communicate that effectively. It was sometimes awkward (think Bambi) and not every conversation led to a collaboration, but it was an invaluable experience to learn more about the excellent work done by teams throughout PHAC. We benefited from staying connected with our colleagues at Impact Canada during this time and our office benefitted from our mutual partnership, and leveraging their existing BeSci infrastructure and expertise. The experience bucked (Bambi pun intended) a lot of our expectations about the operation of the federal public service: We worked quickly and flexibly and we were consistently pleased to find others at PHAC keen to hear about how they might adapt their work to apply principles of BeSci.
Today, BeSciO consists of 19 team members (five fellows) engaged in several different activities. Each of us is responsible for a substantive, longer-term research project. These are done in collaboration with teams at PHAC with subject-matter expertise and focus on topics like mental health literacy, adherence to public health measures, and promotion of antimicrobial stewardship. We also get to help with several side-of-desk tasks like helping with rapid review requests on urgent projects led by other folks at PHAC, developing onboarding materials for incoming Fellows, and creating frameworks and knowledge products like white papers and playbooks for the office.
We’re still maturing our practice at BeSciO. Much of our work is still driven by the pandemic and we are still building some program fundamentals. But as we look ahead, it’s exciting to see the signs of a fully formed BeSci program emerging. It has been a wild ride and we feel privileged to be a small part of it.
Impact Canada, Impact & Innovation Unit, Privy Council Office
For many, Impact Canada’s Fellowship program has served as a unique entry point into the Public Service. Not only are Fellows often joining the Public Service for the first time, but they are also positioned between and across both the program’s central team at the Privy Council Office and their host organization’s project team. To facilitate the integration of Fellows into their placements, each new Fellow receives an onboarding module that provides them with the necessary tools and connections for early and ongoing success, tailored to their individual situation.
Since it was initiated in 2018, all elements of the Fellowship program have been constantly evolving and adapting. Our onboarding modules are continuously undergoing experimentation to address the needs of Fellows, host organizations, and home managers in the most effective way possible. Collecting, analysing, and implementing feedback from newly onboarded Fellows has been most valuable in contributing to the program’s progress from what was once a 1-day module to our current 1–2-week format with a detailed agenda and reading package. This familiarity with adaptation is also what allowed for a seamless transition from in-person to virtual onboardings as a result of COVID-19, with no negative impact on knowledge transfer, integration, or networking.
The Fellowship program’s onboarding modules define and achieve goals from the perspectives of Fellows, host managers, and home managers. Ultimately, we strive to provide Fellows with the tools they need for early and ongoing success, as well as the development of appropriate knowledge, connections, and expectations. From the perspective of home managers, it is important that newly onboarded Fellows have an understanding of the Impact and Innovation Unit and the support that it offers, in addition to a knowledge of the application of Behavioural Science or Challenge Prize approaches in a government context. These understandings are built through reviews of specific Challenges or Behavioural Science (BeSci) projects and their contextualization within the machinery of government. Host managers require that Fellows feel integrated into their host organization’s team and prepared for project work. From our perspective, a successful onboarding gives Fellows a warm welcome, connects them with their managers and teams, gives them access to information and resources, exposes them to projects, and provides them with an understanding of their expectations.
A typical Impact Canada Fellowship Onboarding agenda.
Text version
November 15–23, 2021.
Orientation Week 1. Objective: To acquire an understanding of government processes; the organizational structure; and the role and expected goals of an Impact Canada Fellow.
9:00–9:30: Welcome to the Public Service — this session will serve as a first official welcome to the public service and the Impact and Innovation Unit.
9:30–10:30: Orientation to the Fellowship program — This session will shed light on the machinery of government, cover government processes, including all things HR (compensation, benefits, etc.) and social media awareness.
10:30–10:45: Health Break. 10:45–11:45: Orientation to the Fellowship program — continued.
11:45–1:00: Lunch.
1:00–2:30: Welcome to the BeSci Team — This session will cover the Behavioural Science (BeSci) team structure and address the BeSci role within the Impact and Innovation Unit.
2:30–2:45: Health Break.
2:45–3:15: IT 101 — This session will serve as a quick introduction to accessing Outlook email and calendar, MS Teams, and computer drives at the Privy Council Office.
3:15–5:00: Reading time.
The Fellowship program onboards both individuals and interdisciplinary cohorts of several Fellows at once who are placed across various departments and projects. Regardless of the resulting complexities of scheduling, each onboarded Fellow receives the information and guidance they need to succeed in the first 90 days of their position. This may include bilateral mentorship meetings between new and experienced Fellows, an in-depth reading package with dedicated reading time, contextual presentations related to the Impact and Innovation Unit and the machinery of government, team introductions and project spotlights related to placement area, a review of the host organization’s responsibilities, mandate, and priorities, networking opportunities, integration into the Fellowship and interdisciplinary cohorts, required training for new public servants, and team-building activities. Not every session is included in every onboarding module, but these more occasional sessions are always open to those who have not yet had a chance to attend. Many of these sessions are also useful to Impact and Innovation Unit staff beyond the Fellowship program, serving as both a learning opportunity and chance to network beyond one’s immediate team.
As we prepare to welcome new BeSci and Challenge Fellows in Fall/Winter 2022–23, we are delighted to share the positive sentiments of past Fellows regarding their onboarding experiences:
“I was really impressed by the organization and content of the two-week onboarding. I was not expecting to learn about most projects, interact with members at all levels of IIU, and already be included in meetings about current projects within a short period of time” — Behavioural Science Fellow
“Overall, the onboarding was a really good experience for introducing us to the public service and the organizational structure of government” — Behavioural Science Fellow
“The IIU really cares about Fellows above all else, and that we will have the support to succeed in our projects” — Behavioural Science Fellow
“The close-knit ‘culture’ of the workplace is immediately apparent, despite not having any in-person meetings. The organization and planning that has gone into my onboarding schedule is second-to-none” — Challenge Prize Fellow
“It’s no small feat to coordinate and accommodate multiple different schedules and agendas” — Challenge Prize Fellow
“The onboarding module made me that much more aware of how special this Fellowship really is. Thank you” — Behavioural Science Fellow
We look forward to immersing new Fellows into the Fellowship cohort, and to sharing their reflections in 2023.
From Personality Psychology to Environmental Conservation
Blog
From Personality Psychology to Environmental Conservation
My reflections on the transition from academia to the public service
Dr. Elliot MacDonell, Behavioural Science Fellow
Impact Canada, Impact & Innovation Unit, Privy Council Office
It has been approximately one year since I had an initial meeting with the Impact Canada team to discuss potential opportunities for conducting behavioural science work in the public sector, and in that time, there are a few things I have learned that I’d like to share in this blog. It is my hope that these reflections might help others in similar situations who are considering this type of career switch and are perhaps daunted by this process. In particular, I hope to demonstrate the value in applying your research expertise to any subject, instill some confidence in your abilities and their generalizability, and highlight the benefits of welcoming new career challenges. Throughout, I will share some examples from my experience to provide context.
Your Research Skills and Expertise are Widely Applicable
At the time of my initial meeting with Impact Canada, my applied research experience was rather limited and the opportunities proposed by the team seemed almost too good to be true — they were innovative, multidisciplinary, and timely and would allow me to achieve a long-term goal of combining research and practice. On top of that, I was in the process of finishing my PhD and felt incredibly fortunate to transition straight from university into a position as a Behavioural Science Fellow within Environment and Climate Change Canada (ECCC). Needless to say, my mom could not have been prouder, and I was excited about this new challenge. However, as I onboarded with these teams, I learned that I would be working with ECCC to help protect and conserve Canada’s natural landscape to mitigate the detrimental effects of climate change — while this task was exciting, I immediately felt unprepared, as it appeared to call for a strong understanding of current conservation goals and policy, and an extensive background in environmental science. My training and knowledge on the other hand, was noticeably lacking in those areas and had largely focused on Personality and Forensic Psychology and understanding aggressive behaviour in undergraduate samples.
Luckily, I was (and continue to be) well supported by my colleagues and collaborators, as both the central team at Impact Canada and those within ECCC understood the challenges of adjusting to a new work environment and were more than helpful in getting me acclimated. As I got to know my various managers and team members better and got caught up to speed on the subject matter, my confidence returned and I started to snap out of the hazy, onboarding fog. Most importantly, I recognized that despite the foreign subject matter, I had something to offer and that I had core knowledge and abilities that I could bring to any situation. In other words, I had to engage in a bit of positive self-talk and remind myself that I am an experienced scientist and that regardless of the context, I have a strong understanding of how to apply critical thought and rigor. This helped me to recognize that I know how to overcome barriers in the research process, generate new ideas, and take a project from start to finish.
That being said, I have also learned that it is okay to leave some things behind when moving into a new research space. It is normal to want to form a good impression and a great way to do that is to demonstrate the knowledge that you do have. I mean, who wouldn’t be impressed by insights into the differential predictors of proactive and reactive aggression, or the incremental validity of the six-factor model of personality? However, I had to holster some of my unique knowledge that was not applicable to my current context. Instead, I decided to treat this transition as an opportunity to learn new areas of knowledge. For many of us who engage in scientific investigation, we are not only motivated by our interest in a particular topic, but because we have a natural curiosity and love of learning. Remembering this interest in investigation has given me a new perspective on my work and has allowed me to embrace diverse ways of understanding and conducting research with different collaborators who I may not have interacted with otherwise.
Welcome New Challenges
As I reflect on my experience, I am also reminded of the importance of accepting new challenges. My personal growth across various skill sets has been pleasantly surprising, and mostly a byproduct of the applied sector’s environment. In contrast to my experiences in academic communication where others know the value of science and research, I had to communicate the purpose of research and convey ideas effectively to audiences that sometimes knew nothing about me or the work I hoped to conduct. The balancing act between my own lack of knowledge in the land conservation space and trying to appear as a knowledgeable practitioner further complicated these efforts. Even so, these experiences have given me the opportunity to work with passionate people, who are excited to pair their knowledge and expertise with my own. Working hard to build and strengthen these relationships has helped me to improve my people and communication skills.
Importantly, while navigating my new workspace I began to recognize and accept that I was going to make mistakes. Indeed, I have had many experiences over the past year where I felt I could have explained things more clearly or struggled to demonstrate the value of a plan or idea. While some times I felt embarrassed, or foolish, I tried my best to prevent self-doubt from creeping in, and recognize that this was all part of the process — with each conversation, presentation, and email sent, I was becoming more efficient and effective as a real-world scientist.
One day I may find myself returning to my research roots and assessing deviant personality once again; however, this experience has taught me to value the process of conducting research, whatever subject it may focus on, and to continue to accept new research challenges. I hope that my reflections have been validating and encouraging for readers; though this transition from academia to the public service is an ongoing process, it’s been incredibly rewarding and has taught me a lot about myself.
Land Trusts and Conservancies are constrained by a lack of staff and financial capacity, limiting their ability to expand their conservation networks and effectively manage their lands.
Canadians have an interest in supporting these organizations financially.
The Ecological Gifts Program (EGP) operates efficiently and effectively, but Land Trusts and Conservancies struggle to find donors.
Analysis of past donors has revealed 3 primary types of Canadians who are most likely to donate to the EGP, which LTCs can use to tailor outreach.
In addition to Protected Areas, Other Effective Area-Based Conservation Measures (OECMs) need to be better promoted and understood across Canada to meet conservation targets.
Clear and open communication about OECMs will overcome confusion surrounding them and provide stakeholders with the foundational knowledge needed to identify OECMs across Canada.
Background
Canada is committed to protecting and conserving 30% of its lands and oceans by 2030Footnote 1. One way the Government of Canada is helping to achieve this goal is through the Ecological Gifts Program (EGP). The EGP provides people and organizations who own ecologically sensitive land (i.e., contribute significantly to the conservation of Canada’s biodiversity) an avenue to donate their lands to eligible recipients, or restrict what can be done on their land (e.g., no destruction or development), and to receive tax benefits for doing so. Most often, the recipients of EGP donations are environmental charities called Land Trusts and Conservancies, who manage and protect the lands within their networks.
Importantly, there are two primary types of land conservation that count towards the 2030 goal: (1) Protected areas (PAs), and (2) Other Effective area-based Conservation Measures (OECMs). The primary objective of a PA (e.g., a National Park) is conservation, while an OECM is primarily managed for another purpose (e.g., woodlot, pasture) but still achieves conservation and biodiversity outcomes. For example, low-impact forestry operations, fisheries, and grazing land for cattle all achieve environmental goals, despite ongoing human activities. OECMs are a more novel conservation tool than PAs, and their recognition has become a global priority in the fight against climate change because they help to capture carbon and maintain biodiversity. Canada has not been able to protect as many OECMs as desired because of low awareness and issues with ensuring that the lands that could qualify are protected long-term.
Together, the EGP and OECMs will play a critical role in helping Canada reach its conservation targets. Through a partnership with Impact Canada’s Program of Applied Research on Climate Action (PARCA) and Environment and Climate Change Canada’s (ECCC) Canadian Wildlife Service (CWS), we are leveraging behavioural science to understand ways of increasing the amount of land donated though the EGP and increasing the amount of land that is registered as an OECM.
Methodology
To better understand the challenges with OECM adoption in Canada, the research team used a mixed-methods approach which consisted of two main research activities that were completed simultaneously. First, they conducted 14 semi-structured interviews with teams of stakeholders from provincial and federal levels of government in the Summer of 2022, such as the Land Use Regional Planning branch of the Alberta Government and the Prairie Division of CWS. In addition, interviews were conducted with several non-government organizations (NGOs; e.g., Forest Stewardship Council, Nature Conservancy of Canada). Each interview had 12 questions and took approximately one hour to complete. The questions related to participants’ familiarity with Canada’s conservation goals, their knowledge and experience with OECMs (including the barriers faced), the usefulness of current supports, confusion about what lands should qualify, and incentives for participation. Two independent notetakers summarized responses to each question, and these summaries were then analyzed to determine which barriers were the most notable and persistent across stakeholders.
Second, the research team developed a survey for Land Trusts and Conservancies (LTCs) to gain a better understanding of organizational barriers, experiences with the EGP, and general knowledge about OECMs. These organizations are the primary recipients of Ecological Gifts and thus have valuable insight into the barriers and drivers of land donations in Canada. The online survey took respondents about 15 minutes to complete and was divided into three sections: Organizational information, OECM questions, and EGP questions. The survey fielded during Fall 2022 and was sent to 157 LTCs across Canada. Using a staggered approach to distribution, organizations in the Prairies received the survey first, followed by Atlantic Canada, British Columbia, Ontario, and Quebec. Each region had two weeks to complete the survey and recipients were sent a reminder email 48 hours before the deadline. In total, 97 LTCs completed the survey, with representation from across Canada.
Given the complementary nature of these two methodological approaches, findings from the stakeholder interviews and survey are presented here together. Descriptive statistics were used to summarize survey results.
Findings
A lack of capacity among LTCs hinders their ability to expand and manage their lands effectively
LTCs manage a diverse set of landscapes (e.g., forests, grasslands, marshes), and their conservation networks differ greatly in size (i.e., a few hectares to thousands). Despite these variabilities, 95% of surveyed LTCs report having good relationships with each other and with other stakeholders (e.g., 79% report an excellent or good relationship with the federal government). Unfortunately, however, most LTCs are constrained by a lack of staff and financial capacity, limiting their ability to expand their conservation networks and effectively manage their lands. For instance, more than half of LTCs (53%) report having twenty or fewer volunteers, and over three quarters (76%) report having five or fewer paid staff. Only a quarter believe they have sufficient funding (24%) and staff (26%) to manage their current workloads. Importantly, these staff and funding challenges persist despite the location and size of LTCs (see Figure 1).
Figure 1. The percentage of Land Trusts and Conservancies who selected major, average, or minor barrier across different considerations and actions when deciding to participate in the Ecological Gifts Program.
Figure - Text description
Table 1. The percentage of Land Trusts and Conservancies who selected major, average, or minor barrier across different considerations and actions when deciding to participate in the Ecological Gifts Program.
The EGP is an effective government program, but many LTCs struggle to find donors
Overall, LTCs report positive experiences with the EGP. Many LTCs (72%) feel confident in managing the EGP process from start to finish, and they report that most (75%) of their submitted EGP applications are successful. In addition, many LTCs (76%) feel confident in communicating the benefits of the EGP to potential donors, and they find that the current EGP resources (e.g., handbook, website, workshops, etc.) are helpful. However, some barriers noted by these LTCs were awareness and knowledge of the program (e.g., 40% of LTCs report relevant landowners are unaware of the EGP), the length of time it takes to complete an application, and disagreements with land appraisers on the fair market valuation of the land being donated (see Figure 1). Further, while they believe donors can see the benefits of the EGP, LTCs indicated that finding donors is more of a challenge than is the application process. Finally, many Ecological Gifts could count towards Canada’s 2030 conservation goal, yet they are not included in Canada’s Protected and Conserved Areas Database. The good news is that more than half of LTCs would be interested in having their Ecological Gifts included in this database, but would need government help with the land screening process.
OECMs are a conservation tool that is poorly understood by LTCs
The survey revealed critical issues with OECM awareness and knowledge across Canada. OECMs are not well understood by LTCs, with only about a third (36%) of them saying that they have a clear understanding of the distinction between a PA and an OECM. A similar share (34%) report that they have received no information about OECMs from any source (see Figure 2). Notably, the Federal Government is the most common source of information, with 29% of LTCs reporting that they have received information about OECMs from this source. More than half (54%) of the LTCs were not aware of, or using, the supporting guidelines that were developed to make the land screening process easier. These findings are consistent with our initial stakeholder interviews with provincial governments and NGOs in Canada, which found a lack of participation, awareness, and understanding of OECMs among stakeholders. Barriers to adoption that were frequently identified by participants include a lack of experience with the process of registering OECMs, confusion and misinformation about the inclusion criteria for OECMs (e.g., allowable activities on the land after registration), and a lack of incentives for participation. While some provincial governments and NGOs have a better understanding of OECMs than others, most stakeholders report operating independently from one another. Overall, these results suggest that OECM work in Canada is fragmented and advancing slowly.
Figure 2. The percentage of Land Trusts and Conservancies (LTCs) who have received information about OECMs across different information sources.
Figure - Text description
Table 2. The percentage of Land Trusts and Conservancies (LTCs) who have received information about OECMs across different information sources.
Discussion
At time of publication, approximately 14% of Canada’s lands and oceans are currently protected, and the EGP is an important contributor to the success of the government’s “30% by 2030” goal. The LTCs that participated in this research generally report positive experiences with the program, and feel confident in managing the land donation process from start to finish. Nevertheless, there are several opportunities to improve components of the EGP to help increase the amount of protected land two-fold by 2030. First, LTCs believe the process can take too long and there are sometimes disagreements over the value of land. These timing and communication barriers can be improved by developing behavioural science-informed strategies to help LTCs plan more effectively, and by simplifying messaging while proactively addressing concerns about the land appraisal process. For example, the EGP and Impact Canada co-developed a communication product for land appraisers that was shared with them at EGP workshops in Ontario. This document increased operational transparencyFootnote 2 by explaining the purpose of the land appraisal process, normalized receiving feedback on appraisal applications with supporting statistics, and helped “chunk” the process into clear steps to improve planning for organizations. The product was very well received by appraisers who attended these workshops and will hopefully result in a more efficient appraisal process in the future.
Second, another challenge is helping EGP recipients find donors to expand their conservation networks. Although the number of donors is limited to those who own qualifying land, there are opportunities to improve awareness of the program, and to identify key attributes of past donors to better reach and more effectively engage potential new donors. For instance, the research team partnered with an analytics firm to develop donor profiles of Canadians who have, historically, been most likely to participate in the EGP. This resulted in the identification of 4 primary segments of EGP donors, differentiated by demographics, behaviours, and values. This information has been organized into a communication guide for the EGP which can be used to tailor outreach and communication efforts in the future.
Third, and perhaps most critically, financial resources are an important constraint for LTCs. These organizations need the financial capacity to acquire lands through the EGP once potential donors have been identified and recruited, and to hire necessary staff as their conservation networks expand. Otherwise, it will become increasingly difficult for them to manage their lands effectively and ensure that their lands are protected in the long term. One way to improve capacity is through greater public support and recognition of LTCs in Canada. In a recent survey of Canadians, PARCAasked individuals if they were aware of LTCs, and about their interest in donating to charities whose primary purpose is conservation. Interestingly, 71% of Canadians report having no or limited knowledge of LTCs, but 53% somewhat or strongly agree that a donation to one would be very appealing when these organizations were defined. Thus, there is an opportunity to promote these organizations and the work that they do, and to provide the public with actionable steps they can take to improve the capacity of LTCs (i.e., donating or volunteering).
Across Canada, there is low awareness of OECMs, and key stakeholders are missing the foundational knowledge needed for them to contribute to the success of OECM recognition in Canada. This is evident from the above survey of LTCs and stakeholder interviews, which asked about experiences with OECMs among a variety of federal, provincial, and private sector actors. This lack of knowledge makes it difficult for LTCs to promote the concept to others, who may lack sufficient familiarity with the concept and related processes. While conservation organizations and provincial government officials express interest in learning more about OECMs, some view the major barriers to identifying and formally recognizing OECMs as structural (e.g., political, legislative challenges). Nonetheless, behavioural science can help accelerate the recognition of OECMs by conducting research to identify key incentives for participation; bridging knowledge, awareness, and other gaps among stakeholders; and improving communication and clarity across partners and other stakeholders about why OECMs are important, how they contribute to Canada’s conservation goals, how they are recognized, and what the implications are of this recognition for the future management of those lands.
For Further Information
To learn more or connect about Impact Canada activities, please contact us at iiu-uii@pco-bcp.gc.ca
Library and Archives Canada Cataloguing in Publication
Using behavioural science to understand barriers to land conservation in Canada
Information contained in this publication or product may be reproduced, in part or in whole, and by any means, for personal or public non-commercial purposes without charge or further permission, unless otherwise specified. Commercial reproduction and distribution are prohibited except with written permission from the Privy Council Office.
For more information, contact:
Impact Canada, Privy Council Office
85 Sparks Street, Room 1000
Ottawa ON Canada K1A 0A3
info@pco-bcp.gc.ca
Cette publication est également disponible en français :
Utilisation de la science du comportement pour comprendre les obstacles à la conservation des terres au Canada
Impact Canada supports a number of initiatives under the umbrella of ‘challenges’. These can include competitive accelerators, grand challenges, challenge prizes, and indigenous innovation initiatives. While these vary in their design features, collectively they represent a set of open innovation initiatives that provide incentives (financial or non-financial) to encourage a broad set of innovators to tackle problems where solutions are not apparent.
This document presents (in a general way) a basic theory of what Impact Canada expects to achieve with its suite of challenges. It emphasizes important processrelated outcomes that distinguish challenge approaches from traditional modes of government business and program delivery (like grant making), and that anticipate the longer-term socio-economic and environmental outcomes that challenges are designed to achieve.
With room to build upon the existing evidence of challenge effectiveness, this document serves as a foundation for an impact assessment of Impact Canada’s suite of challenges, which will help fill gaps in our understanding of whether and how challenges can be effective instruments of policy and program delivery.
2.0 Introduction
Launched in 2017, Impact Canada is a whole-of-government effort designed to help departments and agencies accelerate the adoption of innovative approaches to deliver meaningful results for Canadians. Specifically, Impact Canada champions the use of novel outcomes-based approaches within government. In the area of grants and contributions, Impact Canada’s horizontal terms and conditions enable departments to implement pay-for-success projects within their existing portfolios of grants and contributions. A key means of doing this is through the use of challenges also known as inducement prizes.
Challenges are open innovation approaches, designed to provide incentives (both financial and non-financial) to encourage a broad set of innovators to tackle problems where solutions are not apparent. Instead of prescribing a particular set of specifications, those that issue challenges are most often looking for the most promising solutions to meet a set of defined criteria. There are a number of methods and innovation tools that can be applied to challenges themselves. For example, Impact Canada supports a number of more specific innovation methods under the umbrella of challenges1,including:
Competitive Accelerators: Accelerators provide intensive and time limited business support for cohorts of startups, aiming to get them ready for investment more quickly than traditional incubators. Prizes can be layered in to accelerators to reward the most promising entrepreneurs as part of a competitive cohort.
Grand Challenges: Grand challenges use open and thematic competitions to fund a broad range of potential innovations on a prospective basis, often at an early phase. Those models that prove to be successful sometimes based on rigorous evaluations of effectiveness can be candidates for scaling funding (sometimes referred to as ‘stage-gated challenges’).
Challenge Prizes: Challenge prizes offer a reward to whoever can first or most effectively meet a defined challenge or solve a specific problem according to a set of objectively verifiable criteria (sometimes referred to as ‘pure prizes’ – however, challenge prizes also usually incorporate phases to move participants through the challenge prize).
Indigenous innovation initiatives
The Government of Canada seeks new ways to build partnerships with Indigenous communities to improve outcomes in relevant ways. The Government of Canada’s approach to designing and delivering programs for Indigenous communities could be improved by working in a co-creative manner with Indigenous partners.
Impact Canada creates space for doing this, by operating on the understanding that those who live with a problem every day are often best placed to solve it. In many ways, Impact Canada’s approach inverts the traditional ‘top-down’ model of government service provision, recognizing that Indigenous communities are themselves valuable sources of innovative ideas that have the potential to develop into impactful solutions to longstanding problems faced by their communities. In other words, a lack of innovative ideas is not the problem, but government can be a better partner to Indigenous communities in ways that reduce barriers to innovation and create the conditions within which they can thrive.
Borrowing elements from challenge-based approaches, Indigenous innovation initiatives under Impact Canada rely less on the competitive aspects of the challenge model, and emphasize instead the importance of Indigenous leadership and collaboration with and among communities. They aim to bring forward ideas inspired by Indigenous knowledge, regardless of their stage of development, while finding pathways to develop these into sustainable solutions that can have meaningful, positive impacts on Indigenous communities.
Building on the past success of the challenge model
While challenges may be a new approach to programming for the Government, they enjoy a lengthy history. Nesta, a global innovation foundation which promotes the use of challenges, traces their use as far back as the Napoleonic wars in France (Nesta, 2014). Their uses in Victorian England to incentivize innovation in the agricultural industry (Brunt, Lerner, & Nicholas, 2012) as well as throughout the
20th
century in areas like motorized flight and refrigeration (Davis & Davis, 2004) have also been well documented by researchers. More recently, challenges have enjoyed a renaissance since about 2010, when the Office of Social Innovation and Civic Participation within the Obama White House began promoting their use to solve difficult public policy problems with the launch of Challenge.gov (The White House, 2010).
Despite this rich history, there is room to improve upon the existing evidence of the effectiveness of challenges (Williams, 2012). As their use in public policy continues to proliferate (including in Canada), the need for better evidence of their effectiveness is increasing. Policy makers need a better understanding of challenge effectiveness so that they can be used more strategically. The purpose of this brief document is to clearly articulate in a general way a challenge ‘theory of change’ to support an impact assessment of the current suite of challenges that are planned or in the field under Impact Canada. It synthesizes some of the key literature discussing the primary aims of challenges to arrive at a general theory of what they are meant to achieve, recognizing that no two are identical.
This document is presented in the spirit of arriving at a general understanding of what challenges are designed to achieve, but does not presume that the components described are present or equally relevant in all cases. Given this, this narrative should not be read as prescriptive, or in a strictly linear fashion.
3.0 Challenge Logic Model
Text version
Process Outcomes
Challenge or Initiative Launched
Enhanced Public Awareness of a Problem or Issue
New Talent Mobilized
Collaborative Networks and Partnerships Built
Collaborative Networks and Partnerships Built
Sustainable Business Models
Innovative products and services
Enhanced Skill and Capacity
Increased Investment
Greater Public Value
Social Benefits
Environmental Impact
Economic Improvements
Effective and Efficient Government
4.0 Process Outcomes
4.1 Enhanced public awareness of a problem or issue
Challenges are designed to have a ‘look and feel’ that is very different from the types of funding programs that governments and their usual stakeholders are accustomed to running and participating in. This strategy of openness and transparency has a dual purpose. In the early term, it can have the effect of attracting new talent, and with new talent, new ideas (see below) which may bear fruit as viable solutions to the problems that challenges aim to solve. In this way, enhanced awareness among communities of potential solvers is a critical early success factor for challenges.
Over time, challenges can also have the effect of better engaging the general public on the broader policy issue(s) at hand. By working in the open through a challenge, its host creates opportunities to communicate in a strategic manner with the public on matters that are relevant to its central mandate, in a style that may be more attractive to a broader audience than traditional communications approaches may be. It is possible to think of challenges in this manner as ‘flagship’ initiatives that might themselves have a narrow scope, but which have the potential to shed light on the broader social or environmental issues at hand.
This can help generate greater public demand or social licence for the host to more effectively deliver on its core mandate in the future. In this way, it is possible to think of Natural Resources Canada’s suite of clean technology challenges as vehicles to enhance the communication of the Government’s environmental agenda, Health Canada’s Drug Checking Technology Challenge as a clear commitment to harm reduction, or Indigenous Services Canada’s Indigenous Homes Innovation Initiative as an opportunity to advance reconciliation.
4.2 New talent mobilized
Challenges are generally run in an open and transparent manner as a strategy to encourage broad participation among diverse actors, in order to produce a wide range of potential solutions. They feature low barriers to entry to help entice entrants with lower capacity or those who may otherwise be uninterested in the burden of traditional application and reporting processes associated with many government grant and contribution programs. This mobilization of new talent is a frequently cited desired outcome of challenges, and probably one of the features of a challenge that most clearly distinguishes it from traditional ways of doing government business, like calls for proposals or traditional procurement.
Part of the fundamental rationale for running a challenge is that there is perceived value in uncovering potential solutions to longstanding problems that come from outside the usual pool of actors in a policy area. The key assumption here is that ‘outsiders’ bring new perspectives, skills, ideas, resources etc…, that those well established in an area of policy may be incapable of producing themselves. Some evidence exists to suggest that these types of non-conventional participants are generally motivated to put forward their potential solutions for largely non-financial reasons (Gok, 2013). This suggests that a challenge combining financial and nonfinancial incentives may be an effective way to engage new talent for the purpose of driving innovation for better outcomes.
Most of Impact Canada’s current challenges are concerned with engaging new talent in some way. Health Canada’s Drug Checking Technology Challenge has attracted a range of innovators that its longstanding Substance Use and Addictions Program has not previously funded. Natural Resources Canada’s Women in Cleantech Challenge is expressly designed to not only identify new talent in clean technology industries, but to develop it over time to improve the representation of women with new ideas in a traditionally male dominated field.
4.3 Build collaborative networks and partnerships
Although they are fundamentally competitive processes, challenges often create opportunities to build networks and partnerships among their communities of solvers (and often, end users) as a design feature. Where this can be achieved, it can extend the impact of the challenge further into the future, as new partnerships can yield new opportunities for further impact far beyond the scope of a particular challenge process itself (McKinsey & Company, 2009). Well-designed, challenges create opportunities for collaboration throughout the process in a variety of ways. This can include having a vocal spokesperson who draws the public’s attention to the challenge, which can uncover and draw in other investors/business partners who might otherwise be unaware of the opportunity to partner with new innovators. It can also include the creation of opportunities to have end users or other intermediaries test solutions in real time, creating valuable connections that might yield insights to improve products and services.
Indigenous Services Canada’s Indigenous Homes Innovation Initiative treats collaboration and partnership building as a core feature of its design. Semi-finalist solvers benefit from a structured ‘accelerator’ phase, which will bring the community together over a period of eighteen months to develop their proposals, receive mentorship and benefit from established expertise where relevant, and build new partnerships to help bring their ideas from concept to construction. Infrastructure Canada’s Smart Cities Challenge is explicit in its objective of encouraging communities to engage citizens and forge non-traditional partnerships to improve outcomes for citizens using data and connected technology.
5.0 Sustainable Business Models
5.1 Increased investment
Challenges can drive investment in neglected areas in a few different ways. One way is to design a prize as an incentive to invest in areas where no natural incentive for innovation yet exists (a market failure) (Williams, 2012). Where there is a perception among the community of potential sponsors that the risk of investing in potential solutions is offset by the value of the prize, this can have the effect of generating activity in previously neglected areas where the status quo prevails. Natural Resources Canada’s Crush It! and The Sky’s the Limit! challenges, and the Atlantic Canada Opportunities Agency’s Hull Design Efficiency Challenge are good examples of challenges that aim to do this. In essence, they have created ‘artificial’ market incentives in the form of a prize to incent potential solvers to address long standing inefficiencies in specific industries, in order to drive better environmental and economic outcomes in the longer term.
A second way that challenges can drive innovation is by mobilizing a broad community of solvers around a well-defined issue. Where a prize appeals to a range of possible solvers that includes non-traditional actors (see above), it can appeal to a set of motivations that extend beyond financial interests. Some evidence exists that the appeal of challenges is not limited to the potential of financial gain, but associated with non-financial attributes of participation in the process (Brunt, Lerner, & Nicholas, 2012). Those who might not normally invest their time and effort in a solution to a defined problem might be persuaded to do so because they perceive the challenge process as an opportunity to network or achieve profile, for example (McKinsey & Company, 2009).
This can also mean that a challenge with broad appeal can incent a lot of activity in pursuit of a solution to a problem that perhaps traditional modes of doing government business (e.g., grant making) would not. The nature of a challenge is such that the majority will not win (i.e. the grand prize winner), however many might generate new opportunities for the solvers (i.e. the semi-finalists) that would not have otherwise transpired. Non-winners may develop fruitful partnerships, promising prototypes, new business processes, or attract other investments in their activities that all go on to improve socioeconomic or environmental outcomes in their own ways, regardless of their winning or non-winning status. This is all part of the impact of a well-designed challenge.
5.2 Enhanced skill and capacity
Enhancing the skill and capacities of the community of ‘solvers’ is an example of an important process outcome associated with challenges. Challenges are very often ‘staged’ such that solvers need to graduate through a series of successive ‘gates’ leading to the award of the final prize. While the literature generally sees enhancements to skills and capacities within organizations as ‘by-products’, and things that are generally overlooked by challenge sponsors (McKinsey & Company, 2009), for Impact Canada, this is a critical outcome. Typically, Impact Canada challenges are intentionally designed in a staged manner, seizing opportunities to build the capacity of the entire community of solvers in meaningful ways. A clear example of this is Natural Resources Canada’s Women in Cleantech Challenge, which seeks to address the un>der-representation of women in the clean technology industry. Semi-finalists are provided with a range of supports (both financial and non-financial) to develop their clean technology ideas and business plans over a period of two and a half years before a winner is announced. This intensive process is designed to build capacity among women-led firms. Although one ‘winner’ will emerge, this challenge creates broader impact via the investments it makes in women-led firms involved throughout the process.
Incentivizing the creation of innovative products, especially technologies, is probably the most widely cited desired outcome of challenges, and the primary aim of the majority of past challenges (Williams, 2012). There are several discussions of challenges documenting their lengthy history, and incenting the development of new technologies is a clear theme (McKinsey & Company, 2009). Well known challenges have been used throughout history to develop technologies that ultimately improved motorized flight, better preserved food, and expanded space exploration.
This is a key outcome for many Impact Canada challenges as well. Health Canada’s Drug Checking Technology Challenge seeks to incent the development of technology that will allow those who use drugs to make informed decisions about their consumption, based on the composition of the drug itself. Most of Natural Resources Canada’s challenges are primarily focussed on the development of clean technologies – these include everything from smart grids to improved batteries. These types of challenges are largely seen as a way for government to intervene where there is little or no market incentive to develop a technology that would (if it existed) lead to improved socioeconomic or environmental outcomes that have public value.
6.0 Social, environmental and economic benefits
In the long run, challenges are developed by their hosts because they are perceived as a useful means to achieve strategic socioeconomic or environmental benefits that have public value. They are ultimately meant to serve as a means to help achieve the broader policy outcomes of existing programs. The shorter term outcomes described above that are concerned with process and organizational capacity are seen as useful stepping stones toward the achievement of their more strategic aims, which vary from case to case. Where those early outcomes are seen to be achieved, there is greater potential that the community of solvers developed by the challenge leaves the process better equipped to create public value as they pursue their business in the future.
7.0 Effective and efficient government
The key questions for governments with respect to challenges are whether challenges are effective and efficient. Of these two questions, the effectiveness question is (in theory) more easily addressed. A challenge is effective when and where it achieves the outcomes (as relevant) that are described above in the body of this document, leading to the creation of public value in the form of improved socioeconomic or environmental outcomes.
"There is a need to generate better evidence that the perceived risks of running a challenge, and the level of investment that is required to run them, is justified by the achievement of better outcomes"
The answer to the efficiency question is more elusive, but key to government departments which are looking to drive innovation in their policy domains using challenges. In the context of public sector innovation, the onus is on those driving innovation to prove both that the innovation (in this case, a challenge) worked, but also that it worked better than the known alternatives. Challenges take time to develop and implement, are complex, and can draw heavily on internal and external resources. They often run into conflict with established processes and procedures, and raise important questions from corporate services like legal, communications, and finance. There is a need to generate better evidence that the perceived risks of running a challenge, and the level of investment that is required to run them, is justified by the achievement of better outcomes.
There is room to build evidence of this nature (Gok, 2013; Williams, 2012). In theory, the design features of challenges mean that they are well-suited to engaging a broad range of (possibly unconventional) solvers, building sectoral capacity and generating a range of solutions that (whether they win or not) are likely to evolve over time and generate public value in the form of better socioeconomic or environmental outcomes. There is some preliminary empirical evidence that supports this (Brunt, Lerner, & Nicholas, 2012), however the need to build upon this is growing as challenges continue to proliferate in diverse areas of public policy.
Works cited
Brunt, L., Lerner, J., & Nicholas, T. (2012). Inducement Prizes and Innovation. The Journal of Industrial Economics, 60(4).
Davis, L., & Davis, J. (2004). How effective are prizes as incentives to innovation? Evidence from three
20th
century contests. Druid Summer Conference. Elsinore.
Gok, A. (2013). The Impact of Innovation Inducement Prizes. Manchester.
Nesta. (2014). Nesta. Retrieved Septe
Challenges: A practice guide.
The White House. (2010). Prizes and Challenges, Retrieved September 23, 2019, from Office of Social Innovation and Civic Participation.
Williams, H. (2012). Innovation Inducement Prizes: Connecting Research to Policy. Journal of Policy Analysis andManagement, 1-25.
Library and Archives Canada Cataloguing in Publication
Logic model and narrative
Information contained in this publication or product may be reproduced, in part or in whole, and by any means, for personal or public non-commercial purposes without charge or further permission, unless otherwise specified. Commercial reproduction and distribution are prohibited except with written permission from the Privy Council Office.
For more information, contact:
Impact Canada, Privy Council Office
85 Sparks Street, Room 1000
Ottawa ON Canada K1A 0A3
info@pco-bcp.gc.ca
