From Theory to Practice

Since its inception five years ago, Impact Canada has developed and refined its design for impact measurement of Challenges. In this blogpost, I will highlight how theoretical concepts and research on impact measurement have been put into practice over the last year at Impact Canada.

My name is Kiran Toor, and I joined Impact Canada in June 2022 as an Advisor to support the Challenge team with impact measurement. Prior to joining Impact Canada, I was working at Statistics Canada as an Economist leading research projects related to program impact evaluation, business performance and ownership, and conducting data analysis of economic trends, including work for Impact Canada. I joined the team to explore how data is used to inform policy and program decisions, particularly given my experience having been a data user and analyst.

What does impact mean for Impact Canada?

Impact is defined as “change attributed to our program and policy investments. It represents the ultimate aim or objective of a policy — a specific response to a societal challenge” ^{Footnote 1}. We are interested, in particular, in the change in socioeconomic and environmental outcomes that are directly attributable to an intervention. Impact measurement is supported within Impact Canada as a means of assessing the effects of outcomes-based policy and program approaches, in order for us and our partners to use multiple lines of evidence to inform our decision-making.

Impact Canada’s theoretical design for impact measurement

Nearly three years ago, Craig Joyce introduced a guide to measuring impact which highlighted pay-for-success funding approaches, and the use of experimental and quasi-experimental methods of measuring impact to enable policy and program readers to be critical consumers of their own pay-for-success work.

Last year, Noah Conrad provided an overview of different methods of measuring the effects of challenges. Each method of inquiry was tied to outcomes of Impact Canada’s logic model, which outlines the theory behind what Impact Canada expects to achieve through its Challenges, the corresponding process outcomes and the anticipated longer-term socio-economic and environmental outcomes that the Challenges are designed to achieve.

Impact Canada now has 30 Challenges in its portfolio, in domains ranging from public health, housing and infrastructure, food waste reduction to clean technology, and deep space food and healthcare. Challenges are run concurrently, with the stages of ongoing Challenges varying and covering multiple years. As a result, a strategic approach was needed to be established to systematically measure impact across Challenges and leverage available information on Challenges — at whichever stage they are at — in an efficient manner. However, as a Centre of Expertise designated to explore the relevance, effectiveness, and efficiency of outcomes-based funding models, we also needed to understand the impacts of the overall initiative itself, the aggregate of the challenge design process, implementation, and outcomes. In order to do so, we needed to look beyond individual project results and think more systemically about what challenges could accomplish within the Government.

Before impact measurement could be conducted, the logic model needed to be considered. The logic model is broken down into three segments:

1. Process Outcomes,which includes indicators of public awareness, talent mobilization and building collaborative networks.
2. Sustainable Business Models,which includes indicators that innovative products and services are being created, investment is increasing, and skills and capacity are being enhanced.
3. Greater Public Value,which includes measuring the social, environmental, and economic benefit of Challenges — the reason why Challenges are perceived to be useful in providing public value.

Putting it into practice — the approaches we adopted

In the past year, three methods of inquiry were explored to measure the impact of Challenges: surveys, a social network analysis and a quasi-experimental economic impact analysis.

1. Surveys were launched by Impact Canada to obtain feedback from Government of Canada departments, jury members, and Challenge participants on the effectiveness of Challenges, spanning from new talent mobilization, raising awareness of issues, and innovative products and services being created. The survey was also used to obtain perspectives on the use of Challenges to foster innovation capacity within departments, open up government decision-making processes and stimulate innovation.
2. A Social Network Analysis (SNA) was conducted by UBC Master of Data Science students to measure if a Challenge helped raise awareness of a topic, who was engaging with the Impact Canada Challenges, if networks and partnerships were formed from the Challenges, and what the connections were between Challenge participants. This analysis was conducted for the Deep Space Food Challenge, given the social media attention it garnered through collaboration with NASA and the Canada Space Agency (CSA).
3. Quasi-Experimental Analysis was done by Statistics Canada using their Linkable File Environment. In the first round of analysis, Challenge participants were profiled by their type of organization, industry, number of employees, years in operation and age and gender of primary decision maker. This analysis also evaluating the success of Challenges in achieving their economic goals by assessing business performance indicators of participants against a control group. Assessed indicators included revenue, R&D expenditures, salaries and wages, employment, and labour productivity.

What we’ve learned: the importance of data and collaboration

Exploring these three methods has highlighted the importance of data — both qualitative and quantitative — and aspects to consider, such as data collection, analysis, dissemination, and maintenance of databases. Having the adequate and appropriate data has enabled impact measurement to be conducted on Impact Canada Challenges.

We have learned through the social network analysis, that by scraping tweets from Twitter we were able to obtain relevant information to conduct impact measurement, while also reducing response burden. Through our collaboration with the UBC students, this is now a method we are able to replicate based on the methodology they created. We are continuing to conduct social network analyses on other Challenges, while also exploring other tools to enable this work.

Through our partnership with Statistics Canada, we have been able to measure the economic impacts of our Challenges by utilizing their rich business and tax data. Economic improvements are longer-term outcomes that require a number of Challenges to be completed and a sufficient amount of time and data pre and post Challenge to evaluate the impact of a given Challenge. However, with timely tax data, Statistics Canada is able to provide financial performance indicators for participants of Challenges that are both completed and in-progress, to give us a sense of the impacts of Challenges. We are continuing to work with Statistics Canada on a second round of analysis that builds on the first and digs deeper into comparing Challenge participants vs. non-participants in order to measure the impact of the Challenges, while also exploring other quasi-experimental approaches such as difference-in-difference, and propensity score matching.

Given that the Challenge process is collaborative in nature, as we engage with departments, jurors, and participants, conducting a survey was an effective way of not only collecting information where data gaps exist for impact measurement purposes, but it was also beneficial to obtain feedback on ways to enhance the Challenges, as well as collect information on our partners’ experience participating in a Challenge. Having the ability of curating questionnaires to particular policy needs makes surveys a method that complements other modes of inquiry well.

Exploring these different methods of inquiry helps Impact Canada to measure the impact of individual Challenges and evaluate Challenges as a public sector instrument. This work puts Impact Canada in a position to continue to deliver evidence-based products and services to clients and support their outcome-based funding approaches.

What’s coming soon?

In the upcoming months, we will be showcasing our findings from the surveys and social network analysis we conducted. Stay tuned for those blogs!