Augmenting vaccination data among priority populations
Impact Canada, Impact & Innovation Unit, Privy Council Office
Summary
Survey data were collected in two phases: (1) respondents were sampled from the general population in Canada with proportional quotas for age/sex/region; and (2) respondents were oversampled from according to ethnic, racial or Indigenous identity-based quotas. Ethnic, racial or Indigenous identity information was collected (based on census variable categories from Statistics Canada) and used to over-sample respondents from among the following groups: Indigenous, Black, Arab/West Asian, South Asian, Southeast Asian and Latin American. Respondents answered a number of questions to assess several different attitudinal, cognitive and behavioural factors related to COVID-19 vaccination, as well as a number of different demographic factors.
Key Insights
Results indicated that there exists gaps in some racialized groups among key attitudinal and cognitive factors that are strongly associated with vaccination status and intentions. These include vaccine confidence, knowledge of COVID-19 vaccines, positive perceptions of COVID-19 vaccines and the ability to identify misinformation. Despite these differences, attitudinal and cognitive factors such as vaccine confidence and knowledge of COVID-19 vaccines are more strongly associated with vaccine intentions among unvaccinated respondents than most other demographic factors. Top reasons for vaccine hesitancy (among unvaccinated and partially vaccinated respondents) include concerns that not enough testing or research has been done and concerns about the safety and effectiveness of COVID-19 vaccines. Among unvaccinated respondents, initiatives that promote the accessibility of vaccination services were cited as most likely to increase the likelihood of vaccination.
Understanding Parent/Guardian intentions to have their child(ren) receive a COVID-19 vaccine
Impact Canada, Impact & Innovation Unit, Privy Council Office
Summary
Data were collected between October 30 and November 15, 2021. This was just prior to Health Canada’s approval of COVID-19 vaccines for 5-11 year olds.
Key Insights
The study revealed several trends in vaccination intentions among Canadian parents and guardians. A variety of factors were uniquely associated with lower parental intentions to get their child(ren) vaccinated, including parents’/guardians’ own vaccination status (i.e., not vaccinated with at least one dose), younger child age, lower household income, indifference about COVID-19 infection, lower COVID-19 vaccine confidence, concerns of side-effects from routine childhood vaccinations, and children not receiving a seasonal flu vaccination. Top stated reasons for uncertainty or refusal of a COVID-19 vaccine for their child(ren) fell into two overarching categories (1) mistrust of vaccines & institutions and (2) accessibility concerns (e.g., transportation to appointments is difficult, no time off work, no childcare for other children).
Insights and recommendations generated will help inform outreach, communications and program development for continued vaccine roll out, aiming to further improve accessibility of childhood vaccination services and empower parents to perform quality risk assessments when making vaccination-related decisions for their children.
Analyzing 125 sources of information to gain insights on what is working in COVID-19 communication campaigns and policy responses internationally
Impact Canada, Impact & Innovation Unit, Privy Council Office
1. Summary
In support of federal response efforts, Impact Canada has analysed over 125 sources of information to gain insights on what is working in COVID-19 communication campaigns and policy responses internationally, as well as what has been effective during past epidemics. By applying a behavioural science lens to messaging and design components of COVID-19 communications materials, we can help maximize their accuracy and effectiveness in promoting the desired health and response behaviours (e.g., hand washing and physical distancing).
In a time where the individual and collective actions of Canadians is critically important to reducing the spread of the virus, communications materials need to reflect our best understanding of human behaviour.
2. About Impact Canada
Impact Canada is part of a comprehensive set of initiatives launched by the Government of Canada in 2017 aimed at solving complex policy challenges using challenge prizes, pay-for-success, and behaviourally based models. Impact Canada enables the use of challenge prizes through a set of flexible Terms and Conditions (Ts&Cs) for Grants and Contributions that allow funding to be linked to outcomes achieved as well as funding rigorous, research based evaluations.
The Impact and Innovation Unit at the Privy Council Office helps departments develop innovative funding opportunities that bring together the best ideas and brightest talent, and reward those organizations and individuals that deliver high impact results for Canadians.
3. About Behavioural Science
The Behavioural Science (BeSci) team was established in 2015, and has been growing the practive of BeSci across the Government of Canada. Today, the team is composed of a a central group of researchers housed in the Privy Council Office, and multiple BeSci Impact Canada Fellows who are embedded within various Government of Canada departments, supporting the integration of BeSci insights and methods into priority program and policy areas.
While our team members have diverse academic backgrounds and an array of methodological specialities, we all share a passion for understanding human behaviour and using that understanding to drive change. Combining our knowledge of the literature with cutting-edge research and experimentation practice is what we do best.
3.1 Our approach: The behavioural design process
We are data-driven and evidence-based. Testing is integral to how we work. We conduct high-quality research to understand behaviour, and use a variety of rigorous experimental and quasi-experimental approaches to test - and scale - effective solutions.
Continue reading
Our projects often follow a multi-stage process, which we tailor to fit the needs and contexts of each project and partner.
1 Identify
Consult with departmental leadership to identify priority target outcomes of interest and key behaviours to target
Discuss current approaches and additional opportunities for innovation
2 Understand
Conduct initial research (e.g. literature scans, stakeholder interviews) to inform hypotheses of drivers and barriers to behaviours
Conduct additional mixed-methods research (e.g. interviews, observations, surveys, and analysis of administrative data) to test hypotheses
3 Design
Ideate solutions for confirmed hypotheses, referencing the behavioural science literature
Present prototypes of solutions to partners for feedback
Iterate to refine solutions and prepare priority solutions for testing
4 Test
Prepare plans for experimental or quasi-experimental tests of solutions and analysis of results
Implement solutions in line with testing plan
Collect data
Analyse and present results
5 Scale
Prepare plans for scaling (e.g. for broad dissemination of results, preparing policy memos, and supporting other key stakeholders to adapt and adopt provides solutions)
Support plan execution
4. Key insights
To help maximize the accuracy and effectiveness of communications materials in Canada, Impact Canada reviewed over 125 sources to rapidly garner insights on what is working in COVID-19 communications campaigns and policy responses internationally, as well as what has been effective during past epidemics more broadly.
The following key themes and additional considerations were identified:
4.1 Demonstrating Efficacy
Contextualized experimental evidence (i.e., tested as COVID response)
In crisis situations, people are more likely to engage in the desired behaviour if:
they think the infection would cause severe illness for them,
the behaviour is perceived as effective,
the behaviour is perceived as having few costs/barriers,
they believe they could perform the behaviour if they wanted to.
4.2 Evoking Emotional Responses
Contextualized experimental evidence (i.e., tested as COVID response)
Emotions are strong drivers of behaviour. Used with care and coupled with advice about protective actions, evoking an emotional response can amplify messaging efforts.
4.3 Emphasizing Collective Action, Altruism/Moral Responsibility and Civic Duty
Behavioural science literature on crisis communication
Emphasizing that self- isolating and physical distancing are altruistic choices in that they help keep each other safe - especially vulnerable populations - can drive compliance in times of public health pandemic, as can communicating gratitude for supporting greater social good.
4.4 Making Social Norms Salient
Behavioural science literature on crisis communication
We conform to normative information about other people’s current attitudes and behaviors. Ads should build strong norms around protective behaviours, making them seem normal and expected by making unseen behaviours more visible and leveraging dynamic norms (i.e., information about how other people’s behavior is changing over time).
4.5 Emphasizing Adaptiveness and Instilling a Sense of Control to Support Mental Health
Public health literature on strategic communication during pandemics/epidemics
Uncertainty and loss of control while in self-isolation have been suggested as possible drivers of negative mental health outcomes. Ads should provide support to the general public to effectively cope with quarantine and isolation, including practical advice on seeking help, maintaining a routine, exercising at home, combating boredom, and engaging with loved ones virtually.
4.6 Addressing pandemic fatigue
Public health literature on strategic communication during pandemics/epidemics
However unwarranted, perceived severity of the pandemic may decrease over time, leading willingness to perform protective behaviours or comply with other public health advice to also decrease among the general population. It is important for ads to acknowledge and reinforce long-term effort and temporal framing, as appropriate, to combat potential “pandemic fatigue”.
5. Sources
Behavioural science literature on crisis communication, as well as public health literature on strategic communication during epidemics;
Rigorous experimental data from international messaging trials (e.g., UK, Ireland) that tested public perceptions of COVID-19 communication materials, as well as the psychological mechanisms likely driving their efficacy; and,
PCO’s internal behavioural science expertise, as well as insights from a global community of academics, practitioners, and experts actively applying a behavioural lens to the COVID-19 pandemic.
Canadians hold some misperceptions about zero-emission vehicles (ZEVs): many underestimate ZEV driving range and performance, and overestimate costs.
Shifting relevant beliefs and perceptions concerning ZEV affordability, norms, performance, and public charging availability can shift Canadians’ interest in adopting a ZEV.
Overall, knowledge and attitudes about ZEVs and perceived social norms had the largest unique impact on Canadians’ interest in adopting a ZEV.
Background
In 2022, the Government of Canada released its Emissions Reductions Plan which outlined the Government’s effort to mitigate climate change through significant emissions reductions across multiple sectors. A key component of this plan concerns reducing emissions from the transportation sector, including from private vehicles.
Scientists have demonstrated that switching from a gas- or diesel-powered car to a ZEV is one of the most impactful emissions reductions actions an individual can take. However, while rates of ZEV adoption are on the rise in Canada, in Q1 2023, just 10.5% of new vehicles registered were zero-emission. Further, research from the Program of Applied Research on Climate Action (PARCA) has demonstrated that more than 62% of Canadians support greater climate action, with 25% saying they are willing to make substantial changes in their own lives to limit further climate change. With ZEVs being a pro-climate choice, what might be standing in the way of greater adoption?
This is the question we set out to answer in an online survey experiment. In partnership with Natural Resources Canada, this survey sought information about Canadians interest in ZEVs and some barriers to uptake, in support of the Government of Canada’s target of 100% of new light-duty cars and passenger truck sales being zero-emission by 2035 . Our goal was to better understand how to empower Canadians interested in ZEVs to take action.
Methodology
We recruited 2,731 Canadians – representative of the Canadian population by region, age, and gender – to complete an online study. This study was administered through Qualtrics , an online survey platform that recruits from existing pools of research participants who have agreed to contacted for research studies. Respondents were only eligible for the current study if they intended to purchase or lease a new vehicle within the next five years and did not already own a ZEV.
All respondents first completed questions assessing their familiarity with and attitudes about different aspects of both ZEVs and gas vehicles such as their average cost, driving range, and performance in extreme weather conditions.
Next, using a randomized controlled trial design (also known as an RCT), respondents were assigned to view one of six communications about ZEVs (see Figure 1). Five of the communications addressed different potential barriers to adopting ZEVs, which had been identified in previous research (i.e., cost, convenience, range, norms, lifestyle). The sixth communication was a standard message about the environmental benefits of ZEVs (i.e., the control condition). Respondents then asked whether they intended to purchase a ZEV as their next vehicle.
Respondents were also asked to rate their agreement with a series of belief statements that corresponded to each communication. We hypothesized that more effective messages would increase respondents’ agreement with relevant statements, which could in turn impact respondents’ interest in ZEVs.
Control and Treatment messages shown to survey respondents
Cost (Treatment condition 1)
ZEVs dismissed out-of-hand as categorically too expensive; consumers are more aware of luxury brands and pricing. This message was designed to reduce sticker shock and establish a lower anchor when it comes to the price of ZEVs.
Convenience (Treatment condition 2)
People feel ZEVs are restrictive, without realizing inconveniences that come with ICE driving. This message makes habituated aspects (of ICE driving) more salient.
Range (Treatment condition 3)
People feel there aren’t enough chargers to meet their needs but overestimate the extent to which they need public charging. This message increase salience of the closest available charger – at home.
Norms (Treatment condition 4)
People tend to behave in line with their social group(s) and don’t see ZEVs as a typical, normalized purchase. This message invokes identity and presents a social norm.
Lifestyle (Treatment condition 5)
People do not feel ZEVs have enough offerings in the desired car class to meet lifestyle needs. This message visually represents choices.
Environmental benefits (Control condition)
Standard message about the environmental benefits of ZEVs.
Table 1. Zero-emission vehicle belief statements
After seeing one of six messages, all respondents rated their agreement with the following 8 ZEV belief statements:
Electric vehicles can be affordable
New fully electric vehicles start at $32,000
Canada has a rapidly growing network of public charging stations for electric vehicles
Most people who have an electric vehicle charge at home
Many Canadians are now switching from gas vehicles to electric vehicles
Operating and maintaining an electric vehicle has conveniences over a gas vehicle
Electric vehicles are better for the environment than gas vehicles
There are now electric vehicles sold in every vehicle class (e.g. sedans, SUVs, trucks) by major automakers
Findings
Overall, we found that a majority (58%) of our respondents were at least somewhat interested in purchasing or leasing a ZEV as their next vehicle. However, respondents held a variety of misperceptions about ZEVs, from their costs and range to their performance.
Tendency to overestimate costs, and underestimate range and performance
Respondents overestimated ZEV costs
We found that respondents believed ZEVs to be, on average, $20,000 more expensive than comparable gas vehicles, which is an overestimation of approximately $10,000 when comparing to the average retail price of the bestselling ZEV models in 2021 (see Figure 1). Overall, half of respondents (47%) overestimated the average purchase price of a new ZEV. Respondents, when making these estimates, did not take into consideration federal level rebates, of which 51% of respondents were unaware.
Respondents underestimated a ZEV’s typical driving range
More than half of respondents (55%) estimated typical battery electric vehicles to have 200km to 400km of driving range, which is less than the range from a typical mid-sized electric vehicle (~466km; See Figure 2). Moreover, 86% of respondents reported driving fewer than 300km weekly in comparison to the average milage per year driven by Canadians which is 15,200 km. These results suggest that typical ZEV driving ranges should be sufficient for most Canadians on a single charge per week but that Canadians may be making inaccurate assessments of how current ZEV capabilities align with their weekly driving needs.
Respondents rated ZEVs as inferior to gas- and diesel-powered cars on performance.
When asked to rate ZEVs versus a comparable gas-powered car on performance, safety, reliability, and ease and cost of maintenance, we found that respondents rated ZEVs as inferior to gas- and diesel-powered cars on each aspect. This difference was more prominent when respondents were asked to rate these ZEV performance factors under extreme weather conditions such as extreme heat or cold. Two thirds of respondents (68%) rated ZEVs as slightly worse or much worse than a comparable gas car in extreme weather. Studies show that ZEV safety and reliability are on par with – and have lower maintenance costs than – gas vehicles, suggesting that respondents misperceive the performance of ZEVs overall.
None of the Communications had a significant impact on respondents’ interest in ZEVs; however, perceived norms and knowledge of ZEVs positively predicted intent to adopt a ZEV.
Respondents who saw the Treatment messages were no more likely than those in the Control group to report an interest in adopting a ZEV. However, as planned in our pre-registration, a mediation analysis was performed, finding that while both the Treatment and Control messages successfully shifted respondents’ perceptions of ZEVs, only half of the ZEV belief statements (4 of 8; see Table1, bolded text) were positively related to whether respondents would consider purchasing a ZEV as their next vehicle.
We explored the data using a multivariate regression model to help identify which factors had the largest unique impacts on interest in ZEV adoption. This model revealed 12 key features accounting for 42% of the differences observed in reported ZEV adoption. The largest positive predictors in our model suggested that the more accurate a respondents’ knowledge of ZEVs was, and the more respondents perceived ZEVs as a mainstream choice (a socially normative choice), the more likely they were to express an interest in adopting a ZEV themselves.
Discussion
Results from this work suggest that despite rapidly growing interest, Canadians face numerous barriers to ZEV adoption that behaviourally-informed approaches to communication could address. This research demonstrated that many Canadians perceive ZEVs to be unaffordable, with a plurality (47%) of respondents overestimating the average upfront sticker price of ZEVs by more than $10,000. Previous behavioural science research has demonstrated the impact of present bias – the tendency for people to place greater weight on immediate rewards over future payoffs. In this context, people may be overestimating a ZEV’s upfront cost and placing undue focus on increased upfront spending while underweighting the future operational and maintenance cost-savings a ZEV provides compared to a gas car. Thus, interventions that reduce people’s present bias of the upfront cost and increase the salience of available incentives, or focus on long-term cost savings, may help to frame ZEVs as a more cost-effective choice.
In addition, this research showed that Canadians typically underestimate the driving range offered by ZEVs relative to their typical driving needs. Interventions that increase the salience of a drivers’ actual range needs – for instance, by framing how many commutes or grocery trips could be done on a single charge – or increase the visibility and salience of home and public charging stations could alleviate prospective buyers’ range anxiety.
This study demonstrated that even small digital communications can shift beliefs about ZEV affordability, the growing availability of public chargers, the convenience of choosing an electric vehicle, and the growing popularity of ZEVs. In addition, this research found that accurate knowledge about ZEVs, and believing ZEVs are becoming more and more common, is highly associated with and perhaps even predictive of interest in them. Along with other strategies, messaging that addresses misperceptions about ZEV performance and costs, and/or highlights the growing rates of ZEV adoption, may be worth advancing to overcome key behavioural barriers to ZEV adoption identified in the literature and in this study.
Acknowledgements
This research was conducted by IIU staff, with additional support from Dr. Jennifer Lynes Murray at the University of Waterloo. The study received ethical clearance from the University of Waterloo Research Ethics Board (REB #44620).
Library and Archives Canada Cataloguing in Publication
Accelerating Zero-Emission Vehicle Adoption using Behavioural Science
Information contained in this publication or product may be reproduced, in part or in whole, and by any means, for personal or public non-commercial purposes without charge or further permission, unless otherwise specified. Commercial reproduction and distribution are prohibited except with written permission from the Privy Council Office.
For more information, contact:
Impact Canada, Privy Council Office
85 Sparks Street, Room 1000
Ottawa ON Canada K1A 0A3
info@pco-bcp.gc.ca
Luke Babad is a Senior Lead for Behavioural Science at the Impact and Innovation Unit within the Privy Council Office where he focuses on emerging programs in borders, travel and immigration. In this role he works with a team of behavioural scientists to identify and understand people-centred public policy challenges and to test and scale solutions, with the end goal of helping Government of Canada services resonate more meaningfully in the lives of Canadians.
Dominique is a Senior Analyst on the Data and Measurement Strategies team. She is working with Agriculture and Agri-Food Canada to explore new ways of analyzing and using data to advance measurement and reporting of GHG emissions-related policies, programs, and investments.
Since 2017, Impact Canada has been partnering with federal partners to leverage Challenges as an
innovative financing tool to help solve complex problems for Canadians.
Impact Canada Challenges
differ from other traditional government funding programs as they are designed as an outcomes-based
approach using financial and non-financial incentives to draw innovators to direct their talents towards
finding solutions to specific problems.
Since the introduction of Challenges, Impact Canada has been working on measuring their impact, as well
as improving how they are designed and implemented. Guided by a clear
Logic Model,
Impact Canada has operationalized both quantitative research with Statistics Canada, and qualitative
research, including case studies, surveys and social network analyses. To read more about Impact
Canada’s impact measurement approach, please see this
blog.
Since 2019, Impact Canada has partnered with Statistics Canada’s Centre for Special Business Projects
(CSBP) to explore the use of tax data to inform on the performance of Challenge applicants and to
evaluate the economic impact of Challenges using Statistics Canada’s Business - Linkable File
Environment (B-LFE). The B-LFE is an environment in which different administrative and survey data
sources are linked to Statistics Canada's business microdata.
Statistics Canada’s Business - Linkable File Environment (B-LFE) Data Sources
Administrative data sources
Business Register (BR)
Longitudinal Employment Analysis Program (LEAP)
General Index of Financial Information unincorporated businesses
General Index of Financial Information incorporated businesses
T4 (Statement of Remuneration Paid) Supplemental File
PMF (Personal Master File) of primary owner/manager of enterprise and the
Longitudinal Immigration Database
TED (T1 Enhanced Database) of primary owner/manager of enterprise (including
some variables from Census and the Longitudinal Immigration Database
Payroll Deductions Account
Exporter Register
Importer Register
Patents (Canadian Intellectual Property office)
United States Patent OfficeCanadian enterprises only
Business Innovation and Growth Support
Agriculture Tax Data Program
Charities
Canadian Patent Research Database
Diversity and Skills Database
Survey data sources
Annual Survey of Research and Development in Canadian Industry
Balance of Payments: Canadian Direct Investment Abroad
Balance of Payments: Foreign Direct Investment in Canada
Trade in Commercial Services
Survey of Innovation and Business Strategy
Surveys of Innovation
Survey of Electronic Commerce Technology
Survey of Advanced Technology
Survey of Commercialization of Innovation
Survey on Financing (and Growth) of Small and Medium Enterprises
Survey of Intellectual Property Management
Survey of Digital Technology and Internet Use
Survey of Regulatory Compliance Cost
Schedule 32 - Claim for Scientific Research and Experimental Development
(SR&ED) Carried out in Canada (Form T661)
Quasi-experimental analyses were explored to attempt to replicate the rigour of randomized controlled
trials, which is not possible to implement in a Challenge context as competition winners are not
randomly assigned. To demonstrate causal effects of Challenges on outcomes of interest, data on
Challenge applicants, collected by Impact Canada, were linked to the Business Register (BR). The BR is a
repository of registered businesses operating in Canada maintained by Statistics Canada. This dataset
was subsequently linked to tax data, from the B-LFE, to understand what types of businesses
Footnote 2
participate in Challenges by assessing their business characteristics and economic performance. A
non-applicant group was created by exploiting business similarities across a series of variables to
conduct a comparative analysis.
This approach was two-fold. First, to understand if Challenges mobilize new talent, we profiled business
applicants by geographic and business characteristics. Second, to understand if Challenges increase
investments and enhance skills and capacity, we compared the business performance between challenge
applicants and similar non-applicants on the following indicators:
Revenue
Assets
Liabilities
Employment
Salaries and wages
Labour productivity
Research and development (R&D) expenses
Two rounds of quasi-experimental analyses have been conducted. The first was conducted to assess the
possibility of evaluating the success of Challenges achieving their goals. As Impact Canada launched new
Challenges and the number of Challenge applicants grew since the first study, a second analysis was
conducted. The second analysis aimed to expand depth and scope of the first study by including the
additional years of business data and assessing the feasibility of performing a program impact analysis.
Why was this done?
Unlike surveys and other methods of gathering data, using administrative data, such as tax data, for
purposes other than it’s intended use enables access to high quality data to profile business economic
performance. Importantly, the use of administrative data reduces respondent burden of applicants
involved in Challenges from having to answer questionnaires.
The Statistic Act allows Statistics Canada to link tax data with other survey and administrative
data for research purposes. This allows Statistics Canada to collaborate with government departments “in
the collection, compilation and publication of statistical information, including statistics derived
from the activities of those departments”. The linked data offers the ability to increase the scope and
depth of possible research to be conducted. For example, using tax data can open possibilities for proxy
variables to measure outcomes.
Exploring a quasi-experimental analysis using the B-LFE was compelling to assess the impact of Challenges
because it generates evidence on how challenge applicants perform post-Challenge from an investment
perspective. It also enables the comparison of applicants to similar businesses that did not apply to
Impact Canada Challenges – a feature that allows us to assess how effectively the Challenge
program outcomes are being achieved.
To systematically summarize the results and learnings from this work, a rose analogy is used. The report
below first summarizes what went well and the findings from the analyses (our rose), then outlines the
obstacles we’ve faced (our thorn), and lastly discusses what we’ve learned and future opportunities to
further develop this work (our bud).
Our rose: What we’ve learned so far
The first quasi-experimental analysis was conducted in 2021, which analysed 7 Challenges from 2017 to
2019. As more data became available, a second quasi-experimental analysis was conducted in 2023 on 12
Challenges from 2016 to 2021. In conjunction with the second analysis, a complementary analysis was
conducted to assess the feasibility of performing an impact analysis of the Challenge program.
The results from the second analysis, as shown in Figure 1, illustrate that 90% of Challenge business
applicants were for-profit organizations. As shown in Figure 2, applicants were spread out geographically, with three out of
four business applicants operating in Ontario (41%), the Prairies (18%) or British Columbia (16%).
The majority (95%) of applicants were small businesses, with less than 100 employees, and over half (59%)
were young businesses who had been operating for less than 5 years, as seen in Figures 3 and 4.
Figure 5 illustrates that while businesses that apply to Impact Canada Challenges are diverse and operate
in 20 different industries, over half (59%) of them are in the Manufacturing (22%) and Professional,
scientific and technical services (37%) industries.
To inform on whether Challenge applicants were indeed able to increase investments as a result of the
supports provided throughout Challenges, revenues, assets, liabilities of Challenge applicants were
analysed and then compared against non-applicants.
The comparative analysis illustrated that Challenge applicants are in fact increasing investments more
than non-applicants, as growth rates of revenue (104% and 50%), assets (125% and 111%) and liabilities
(226% and 122%) between 2016 and 2020 were higher for applicants than non-applicants, as seen in Figures
6 to 9.
To inform on whether Challenges were improving skills and capacity of applicants, employment and payroll
data was analysed. Figure 5 demonstrates that Challenges are indeed enhancing skills and capacity as
growth rates from 2016 to 2021 of employment (58% and 36%), wages (100% and 62%), wage per employee (23%
and 15%) and labour productivityFootnote 3 (124% and 62%) were higher for Challenge
applicants than non-applicants.
R&D expendituresFootnote
4 were assessed as a proxy to understand if Challenges were contributing to the
development of innovative products and services. The results illustrate that between 2016 and 2020
non-applicants (28%) increased their spending on R&D expenditures more than applicants (14%), as
shown in Figure 6.
Part of the second round of analysis included assessing the feasibility of performing a program impact
analysis. The work intended to explore the types of statistical methods that could be employed for an
impact analysis and the feasibility of conducting an impact analysis in the future once additional data
becomes available. The feasibility study would be used to document strength and weaknesses of
approaches, and suggestions on what should be done in subsequent years to facilitate a successful impact
analysis.
What did the feasibility analysis inform?
An assessment of the feasibility of performing an impact analysis was conducted based on the data
available to determine the impact of participating in Impact Canada Challenges. That is, to answer the
question: is there statistically significant evidence that participating in Impact Canada Challenges
measurably improves applicants’ investment and employment outcomes?
To ensure sufficient data was available pre and post Challenge, the impact analysis was conducted on 40
businesses who won at least one prize (semi-finalists, finalists, and grand-prize winners), coming from
6 Challenges between 2016 and 2021. A difference-in-differencesFootnote 5 approach was used to test the
effects of participating in Challenges on a set of outcome variables including, revenue, expenses,
assets, liabilities, employment, wages, and labour productivity.
Results from the feasibility study showed no statistically significant evidence that participating in a
Challenge had an impact on any of the dependent variables tested (revenue, expenses, assets,
liabilities, employment, wages, and labour productivity). However, this work has provided the groundwork
towards estimating the impact of Impact Canada Challenges in the future.
Our thorn: The limitations
The limitations when measuring the impact of Challenges
Impact Canada Challenges vary by subject-matter, intended outcomes, timelines, number of stages, prize
purse, and number of applicants. This results in a complex setting for impact evaluation, especially
given the variability across Challenges and applicants. This variability implies a degree of
heterogeneity.
This heterogeneity among applicants is further increased as Challenges draw from diverse types of
applicants such as businesses, academics, individuals, government and indigenous organization, and
international applicants. Coupled with this, as the Challenge stages progress, fewer applicants advance
to become semi-finalists, finalists, and grand-prize winners, which thereby reduces the sample size of
applicants that are in a better position to demonstrate if Challenge outcomes are being achieved. This
poses a challenge, as it makes sense to focus on applicants that have successfully progressed further as
they have incrementally received more government funding than unsuccessful applicants; and have
presumably had more time to commercialize and scale. To justify this notion and to generate evidence on
the impact of Challenges on business performance of successful applicants, further empirically testing
based on theory and past studies would need to be done.
The small sample size of semi-finalists, finalist and grand-prize winners also prevents the ability to
estimate the impact on a per-Challenge basis. As a result, to assess the impact of Challenges on
successful applicants, Challenges that may have different impacts at different scales are aggregated.
The limitations of using tax data to measure the impact of
Challenges
Despite the advantages of using corporate tax data to measure the outcomes and impact of Challenges on
business performance, it also poses limitations as only business applicants whose information can be
linked to tax data can be used for analysis. This approach does not apply to academics, individuals, and
non-business applicants, limiting program impact to only a business perspective. Hence, this further
decreases the sample size and has implications on the level of precisions of statistics presented to
profile business performance using tax data given data confidentiality requirements. Furthermore, some
tax information is not provided by all businesses, such as R&D expenditures, which limits proxy
variables available to assess if investment are made in developing innovative products and services with
respect to support provided by Challenges.
The heterogeneity both at the program and applicant levels, small sample size, and inevitable data
outliers – which cannot always be removed due to the already small sample size – make it difficult for a
statistical model to provide evidence of impact in the context of this research and have highlighted the
trade-offs between increasing precision and robustness.
These limitations coupled with varying data availability also restricts the ability to compare the impact
of a Challenge against another in a given year. For the time being, point-in-time measures are obtained
for Challenges both completed and in-progress, to get a sense of the impacts of Challenges.
Our bud: Lessons learned and future work
What have we learned?
Exploring quasi-experimental analyses using tax data has demonstrated to be a rigorous method of
generating evidence that is adaptable to measure outcomes and impact as they unfold. Furthermore, this
work has demonstrated that using administrative data to assess economic impacts of programs can also
inform on larger program or policy implications.
Given that corporate tax datasets are both comprehensive and timely, this opens up opportunities for
future longitudinal and cross-sectional analyses and offers the ability to increase the scope and
capabilities of research if needed. However, to do this, and to overcome many of the limitations posed
by the Challenges design, methodology, and current data availability, we have learned that we’ll need
the fullness of time. As more recent Challenges are completed and sufficient pre and post data becomes
available, the number of successful applicants also increases, which can contribute to improving the
robustness of statistical analyses and allows to inform on longer-term outcomes, such as economic
improvements.
The partnership between Impact Canada and Statistics Canada has enabled an iterative research and
analysis initiative, where we are able to build on our learnings. The collaborative work done to assess
the impact of Challenges thus far has set a solid foundation and provides many opportunities to explore
other quasi-experimental approaches, and to increase the robustness of analyses conducted in upcoming
years. As more data becomes available, in 2024 and beyond, we are hopeful that additional analytical
opportunities will enable us to explore more research questions to further inform on outcome-based
funding.
What could we explore in the future?
To improve how Impact Canada Challenge outcomes are measured, the quasi-experimental analyses could be
enhanced in the future work by exploring potential linkages between available tax data, patent data, and
investment open data. Additional methodologies, such as dynamic difference-in-differences models which
could provide impact estimates for each year after the beginning of a Challenge, could also be explored
to enhance the impact analyses.
Acknowledgments
Thank you to our partners at Statistics Canada, Xavier Bouchard-Vachon and Simon Dessureault, for the
ongoing partnership and collaboration on this research. Working with a statistical agency has been
helpful for Impact Canada to test and refine our approaches.
Library and Archives Canada Cataloguing in Publication
Our rose, thorn and bud
Information contained in this publication or product may be reproduced, in part or in whole, and by any means, for personal or public non-commercial purposes without charge or further permission, unless otherwise specified. Commercial reproduction and distribution are prohibited except with written permission from the Privy Council Office.
For more information, contact:
Impact Canada, Privy Council Office
85 Sparks Street, Room 1000
Ottawa ON Canada K1A 0A3
info@pco-bcp.gc.ca
Emily is a Behavioural Science Fellow at the Impact and Innovation Unit (IIU) within the Privy Council Office. She is working with partners at Natural Resources Canada applying behavioural science to support home energy and adaptation labelling.
Prior to joining the IIU, Emily worked at a charitable research institute conducting applied research to improve mental health and addictions treatment and services.
Impact Canada, Impact & Innovation Unit, Privy Council Office
What’s love got to do with it?
We’ve all been there — you’re ready for the world to see the breakthrough idea you’ve obsessed over and that can create positive changes for the masses. A fire burns in your stomach and your blood pumps with excitement because the idea you’ve fallen in love with is about to become real. However, once you share it, other people might find the idea less than perfect — they might even tell you that it will never actually work. …Ouch.
As an Impact Canada Challenge Prize fellow, I worked with the Public Health Agency of Canada to design and launch the Type 2 Diabetes Prevention Challenge. When I shared the first draft of the Challenge prototype with subject matter experts in chronic disease prevention, their comments highlighted where my design (that I had spent countless hours creating), would falter and fail, enlightening me on “unknown unknowns” I had not yet uncovered. Soliciting these criticisms from problem holders and authorities in diabetes prevention was an extremely important part of using innovation as a problem-solving process. It allowed the Challenge design to be adjusted so that it could enable innovators to have the highest impact on preventing type 2 diabetes in Canada. Throughout this process, I routinely checked my biases to ensure I was not dismissing the shortcomings identified by others in an effort to see my own idea thrive. Some people refer to those that succumb to this mindset as being struck with “ugly baby syndrome” — the tendency to fall in love with your idea or innovation, believe it is perfect in every way, and think that it will continue along a charted course without requiring any changes.
The truth is that whether you are creating a Challenge prize, applying to one as an innovator, or think you found a breakthrough, every single idea starts off imperfect and needs to be nurtured to grow past its shortcomings to turn into something beautiful. To minimize the risk of creating something that nobody asked for or wants, trailblazers spend the time nurturing their idea to smooth out it’s edges or even drastically change it, rather than give it unearned praise. The process of testing your idea to see where even more time and energy need to be invested on top of the countless amount already put in is a growing pain that transforms the idea into something meaningful for a greater number of people.
I have found, through painful experience, that the most important step a [person] can take is always the next one.
— Brandon Sanderson
Gotta have faith
Challenge prizes run in partnership with Impact Canada use a stage-gated process to create the environment and timeline for innovators to continuously test and refine the idea. This process serves as a scaffolding for ideas to achieve partial outcomes related to a Challenge goal, and for ones that have developed the most potential for impact to proceed to subsequent Challenge stages. At each Challenge stage, innovators have access to financial prizes and non-financial supports that enable them to transform their innovation into more impactful versions of their initial idea. Even the Challenge’s assessment criteria focuses the innovators’ efforts (commensurate to the progress at each subsequent stage) to solicit, respond, and pivot their work, where necessary, in a way that yields the most value for the problems being solved. Participating in a Challenge prize helps innovators nurture their idea.
Creativity is thinking up new things. Innovation is doing new things
— Theodore Levitt
Don’t stop believin’
I propose that loving your idea means ensuring that it is has every opportunity to become the best version of itself. This mindset does not mean letting everyone else direct the idea’s growth but means having the confidence to synthesize all the feedback received with your own expertise to make the idea better. Constantly re-examining what value the idea creates and who benefits from it are important parts of this reflection process used in an Impact Canada Challenge. When framing the problem, during an Understand phase, whether a Challenge is the right method to address the problem is a mantra that ensures the power of innovation to unlock persisting problems is truly leveraged. For example, we need to ensure that solutions for the problem area benefit from opening up to wider pool of innovators, that a Challenge does accelerate progress, that outcomes can be measured, and the solutions can be adopted or scaled once the Challenge is complete. During the Design and Test phases where the Challenge structure takes shape and is refined, it is important to collect the voices, opinions, and feedback of many people to protect against blind spots or group think, balancing delicate tensions related to what the outcomes will be achieved. A small sample of tensions I’ve encountered include the following musings:
Does the design and outcome benefit a small group significantly or have a broader but less significant impact for many?
Will it advance a new tool, technology, or process that has a benefit for some but create a gap for others to access it?
Will it play off path dependency or facilitate breaking out of the mould?
Does it create social good or economic development?
The constant examination of questions like these will nurture any idea (or Challenge aimed at surfacing ideas) to continue to develop and is the basis of giving it the requirements for growth. Thus, loving your idea means not being blind to its imperfections. It means trusting in a reflection process where even the most imperfect idea has tremendous potential to be unlocked — it is this potential that can lead to a breakthrough that could create profound change.
Jon is an Impact Canada Challenges Fellow who will be supporting the Department of Fisheries and Oceans Canada in developing a MarineSpatial Planning Challenge Prize.
