Meet the IIU’s Behavioural Science Fellow: Daniel Almeida
Impact Canada
Impact Canada, Impact & Innovation Unit, Privy Council Office
What unique perspectives do you bring to this work, as someone entering the public service for the first time?
I believe that my track record of collaboration allows me to provide a unique perspective on behavioural science. Having worked with a variety of professionals, I have developed an ability to analyze, understand and solve problems using tools drawn from different frameworks. One of the aspects of my doctoral training that I enjoyed the most was all of the problem solving that came with my research. I am, therefore, excited about using the skills I developed during my doctoral studies to solve real issues affecting the lives of Canadians.
What attracted you to apply to the Fellowship program?
I was attracted to the fellowship program because of the approach that Impact Canada uses. The final scaling step, in particular, was what I felt was lacking in my former research. I am excited about the opportunity to disseminate knowledge and create meaningful change through my work.
In what way do you believe the application of behavioural science can effectively support Canada’s public health response?
I firmly believe that an empirically guided response is needed to address matters of public health. Grounding policy in science ensures that solutions take advantage of all of the existing scientific literature and are supported by a strong rationale and rich body of evidence.
"I am excited about the opportunity to disseminate knowledge and create meaningful change through my work."
Published: 2022-12-21
Last modified: 2025-02-03
Blog
Meet the IIU’s Behavioural Science Fellow: Klajdi Puka
Impact Canada
Impact Canada, Impact & Innovation Unit, Privy Council Office
Meet the IIU’s Behavioural Science Fellow: Arvin Jagayat
Blog
Meet the IIU’s Behavioural Science Fellow: Arvin Jagayat
Impact Canada
Impact Canada, Impact & Innovation Unit, Privy Council Office
What unique perspectives do you bring to this work, as someone entering the public service for the first time?
In both my personal life and academic career, I am fascinated by the intersection between technology and behaviour. Digital social spaces have evolved far faster than our physical ones, and our physical experiences seemingly integrate with our digital lives more and more by the day. From studying the ever-expanding ways that women and girls are targeted online to evaluating ways that social media platforms can be designed differently to reduce engagement with misinformation, I have become attuned to the affordances and pitfalls of digital social spaces. So much so that I led an interdisciplinary team to develop the Mock Social Media Website Tool: a method for researchers to simulate existing, modified, or propose completely new social media platforms and measure how participants engage with them. I believe that my experiences developing research software to be highly modular, accessible, and to meet the expressed needs of researchers all over the world has provided me with diverse perspectives on people’s digital lives that I hope I can bring to my work with Impact Canada.
What attracted you to apply to the Fellowship program?
Above all, the Fellowship program seemed like a rare opportunity to do conduct research that is simultaneously rigorous, meaningful, and has a clear pathway to policy through being a part of the public service. My research was always more applied, so the prospect of joining a team of researchers that are conducting actionable research was very attractive to me. The cherry on top was learning that these teams were interdisciplinary, and that I could have the opportunity to work alongside and learn from experts with completely different research backgrounds. Many of the biggest societal issues are cross-sector and cross-discipline, so applying was a no brainer when I discovered that Impact Canada was fundamentally committed to fostering these diverse perspectives.
In what way do you believe the application of behavioural science can effectively support the response to some of the challenges we face as a society?
All of us often wonder why people do the things they do, and it is easy to speculate what the underlying reasons might be. However, human behaviour is incredibly variable, messy, and often influenced by many dispositional and circumstantial factors simultaneously. However, speculation alone cannot tell us for certain which of these factors matter most. To be effective in combatting our biggest societal issues, a high degree of certainty is needed. Without that certainty, potential solutions to these societal issues can backfire and potentially make them worse. Behavioural science equips us with the tools we need to replace speculation with certainty to ultimately make the lives of people better.
"Behavioural science equips us with tools we need to replace speculation with certainty to ultimately make the lives of people better."
Meet the IIU’s Challenge Prize Fellow: Deena Newaz
Blog
Meet the IIU’s Challenge Prize Fellow: Deena Newaz
Impact Canada
Impact Canada, Impact & Innovation Unit, Privy Council Office
Published: 2022-12-06
Last modified: 2025-05-25
Blog
From Theory to Practice
Measuring the impact of Challenges
Kiran Toor, Advisor
Impact Canada, Impact & Innovation Unit, Privy Council Office
Since its inception five years ago, Impact Canada has developed and refined its design for impact measurement of Challenges. In this blogpost, I will highlight how theoretical concepts and research on impact measurement have been put into practice over the last year at Impact Canada.
My name is Kiran Toor, and I joined Impact Canada in June 2022 as an Advisor to support the Challenge team with impact measurement. Prior to joining Impact Canada, I was working at Statistics Canada as an Economist leading research projects related to program impact evaluation, business performance and ownership, and conducting data analysis of economic trends, including work for Impact Canada. I joined the team to explore how data is used to inform policy and program decisions, particularly given my experience having been a data user and analyst.
What does impact mean for Impact Canada?
Impact is defined as “change attributed to our program and policy investments. It represents the ultimate aim or objective of a policy — a specific response to a societal challenge” Footnote 1. We are interested, in particular, in the change in socioeconomic and environmental outcomes that are directly attributable to an intervention. Impact measurement is supported within Impact Canada as a means of assessing the effects of outcomes-based policy and program approaches, in order for us and our partners to use multiple lines of evidence to inform our decision-making.
Impact Canada’s theoretical design for impact measurement
Nearly three years ago, Craig Joyce introduced a guide to measuring impact which highlighted pay-for-success funding approaches, and the use of experimental and quasi-experimental methods of measuring impact to enable policy and program readers to be critical consumers of their own pay-for-success work.
Last year, Noah Conrad provided an overview of different methods of measuring the effects of challenges. Each method of inquiry was tied to outcomes of Impact Canada’s logic model, which outlines the theory behind what Impact Canada expects to achieve through its Challenges, the corresponding process outcomes and the anticipated longer-term socio-economic and environmental outcomes that the Challenges are designed to achieve.
Impact Canada now has 30 Challenges in its portfolio, in domains ranging from public health, housing and infrastructure, food waste reduction to clean technology, and deep space food and healthcare. Challenges are run concurrently, with the stages of ongoing Challenges varying and covering multiple years. As a result, a strategic approach was needed to be established to systematically measure impact across Challenges and leverage available information on Challenges — at whichever stage they are at — in an efficient manner. However, as a Centre of Expertise designated to explore the relevance, effectiveness, and efficiency of outcomes-based funding models, we also needed to understand the impacts of the overall initiative itself, the aggregate of the challenge design process, implementation, and outcomes. In order to do so, we needed to look beyond individual project results and think more systemically about what challenges could accomplish within the Government.
Before impact measurement could be conducted, the logic model needed to be considered. The logic model is broken down into three segments:
Process Outcomes,which includes indicators of public awareness, talent mobilization and building collaborative networks.
Sustainable Business Models,which includes indicators that innovative products and services are being created, investment is increasing, and skills and capacity are being enhanced.
Greater Public Value,which includes measuring the social, environmental, and economic benefit of Challenges — the reason why Challenges are perceived to be useful in providing public value.
Text version
Process Outcomes
Challenge or Initiative Launched
Enhanced Public Awareness of a Problem or Issue
New Talent Mobilized
Collaborative Networks and Partnerships Built
Collaborative Networks and Partnerships Built
Sustainable Business Models
Innovative products and services
Enhanced Skill and Capacity
Increased Investment
Greater Public Value
Social Benefits
Environmental Impact
Economic Improvements
Effective and Efficient Government
Putting it into practice — the approaches we adopted
In the past year, three methods of inquiry were explored to measure the impact of Challenges: surveys, a social network analysis and a quasi-experimental economic impact analysis.
Surveys were launched by Impact Canada to obtain feedback from Government of Canada departments, jury members, and Challenge participants on the effectiveness of Challenges, spanning from new talent mobilization, raising awareness of issues, and innovative products and services being created. The survey was also used to obtain perspectives on the use of Challenges to foster innovation capacity within departments, open up government decision-making processes and stimulate innovation.
A Social Network Analysis (SNA) was conducted by UBC Master of Data Science students to measure if a Challenge helped raise awareness of a topic, who was engaging with the Impact Canada Challenges, if networks and partnerships were formed from the Challenges, and what the connections were between Challenge participants. This analysis was conducted for the Deep Space Food Challenge, given the social media attention it garnered through collaboration with NASA and the Canada Space Agency (CSA).
Quasi-Experimental Analysis was done by Statistics Canada using their Linkable File Environment. In the first round of analysis, Challenge participants were profiled by their type of organization, industry, number of employees, years in operation and age and gender of primary decision maker. This analysis also evaluating the success of Challenges in achieving their economic goals by assessing business performance indicators of participants against a control group. Assessed indicators included revenue, R&D expenditures, salaries and wages, employment, and labour productivity.
What we’ve learned: the importance of data and collaboration
Exploring these three methods has highlighted the importance of data — both qualitative and quantitative — and aspects to consider, such as data collection, analysis, dissemination, and maintenance of databases. Having the adequate and appropriate data has enabled impact measurement to be conducted on Impact Canada Challenges.
We have learned through the social network analysis, that by scraping tweets from Twitter we were able to obtain relevant information to conduct impact measurement, while also reducing response burden. Through our collaboration with the UBC students, this is now a method we are able to replicate based on the methodology they created. We are continuing to conduct social network analyses on other Challenges, while also exploring other tools to enable this work.
Through our partnership with Statistics Canada, we have been able to measure the economic impacts of our Challenges by utilizing their rich business and tax data. Economic improvements are longer-term outcomes that require a number of Challenges to be completed and a sufficient amount of time and data pre and post Challenge to evaluate the impact of a given Challenge. However, with timely tax data, Statistics Canada is able to provide financial performance indicators for participants of Challenges that are both completed and in-progress, to give us a sense of the impacts of Challenges. We are continuing to work with Statistics Canada on a second round of analysis that builds on the first and digs deeper into comparing Challenge participants vs. non-participants in order to measure the impact of the Challenges, while also exploring other quasi-experimental approaches such as difference-in-difference, and propensity score matching.
Given that the Challenge process is collaborative in nature, as we engage with departments, jurors, and participants, conducting a survey was an effective way of not only collecting information where data gaps exist for impact measurement purposes, but it was also beneficial to obtain feedback on ways to enhance the Challenges, as well as collect information on our partners’ experience participating in a Challenge. Having the ability of curating questionnaires to particular policy needs makes surveys a method that complements other modes of inquiry well.
Exploring these different methods of inquiry helps Impact Canada to measure the impact of individual Challenges and evaluate Challenges as a public sector instrument. This work puts Impact Canada in a position to continue to deliver evidence-based products and services to clients and support their outcome-based funding approaches.
What’s coming soon?
In the upcoming months, we will be showcasing our findings from the surveys and social network analysis we conducted. Stay tuned for those blogs!
Published: 2022-12-02
Last modified: 2025-05-25
Blog
Meet the IIU’s Behavioural Science Fellow: Dr. Annabel Sibalis
Impact Canada
Impact Canada, Impact & Innovation Unit, Privy Council Office
Published: 2022-11-24
Last modified: 2022-11-28
Blog
Meet the IIU’s Behavioural Science Fellow: Kaitlyn Fallow
Impact Canada
Impact Canada, Impact & Innovation Unit, Privy Council Office