Summary reportHow effective are Challenges? Using Social Network Analysis to Measure Impact

1. Summary

Since 2017, Impact Canada has been partnering with federal partners to leverage Challenges as an innovative financing tool to help solve complex problems for Canadians. Impact Canada Challenges differ from other traditional government funding programs as they are designed as an "outcomes-based" approach using financial and non-financial incentives to draw innovators to direct their talents towards finding solutions to specific problems.

Since the introduction of Challenges as a tool in the Government of Canada toolbox, Impact Canada has been working on measuring their impact, as well as improving how they are designed and implemented. Guided by a clear Logic Model, Impact Canada has operationalized both quantitative research with Statistics Canada, and qualitative research, including case studies, surveys and social network analyses. To read more about Impact Canada’s impact measurement approach, read our Logic Model, and this blog.

In a previous summary report, we presented highlights from Stakeholder surveys. Here, we present results from another line of inquiry that we are exploring for measuring the impact of Impact Canada Challenges: Social network analysis. In collaboration with students from the Master of Data Science at the University of British Columbia (UBC), and Statistics Canada, our team at Impact Canada embarked on a pilot project to leverage social network analysis in measuring the impact of our Deep Space Food Challenge. The result of this exercise was the development of a methodology that can be replicated for impact measurement of all challenges.

2. About Impact Canada

Impact Canada is part of a comprehensive set of initiatives launched by the Government of Canada in 2017 aimed at solving complex policy challenges using challenge prizes, pay-for-success, and behaviourally based models. Impact Canada enables the use of challenge prizes through a set of flexible Terms and Conditions (Ts&Cs) for Grants and Contributions that allow funding to be linked to outcomes achieved as well as funding rigorous, research based evaluations.

The Impact and Innovation Unit at the Privy Council Office helps departments develop innovative funding opportunities that bring together the best ideas and brightest talent, and reward those organizations and individuals that deliver high impact results for Canadians.

3. Methodology: Social Network Analysis

Social network analysis is a process of studying relationships among people and organizations. Using a graph theoretical approach, social networks can be conceptualized as a graph with nodes (units or points) representing the relationships and interactions between social actors.

With this approach, our team tried to answer the following research questions around the impact of Challenges:

• Do challenges help raise awareness about a particular topic? Is there an increase in social media mentions about a particular topic?
• Who are challenges reaching and what are the connections between them?
• Which groups and actors are engaging with Impact Canada Challenges?
• What types of networks and connections have formed between challenge participants?

While a number of social media platforms were explored, we ultimately chose to use Twitter, as it is a popular platform used by our audience and has wildly available public data (e.g. user followers, followings, tweets, and profile information) to develop a proof-of-concept. Further, given its overall popularity and open structure, there are a significant number existing and available research tools designed for collecting and analyzing its data.

In order to answer our questions on the Impact of Challenges, we developed a step-wise approach to developing the proof-of-concept. Briefly, we used the steps outlined below:

1. Data Scraping: The Python package snscrape was used to collect tweets using a variety of keyword search queries related to the Deep Space Food Challenge. Ultimately, the term "Deep Space Food" was used to collect 9,323 tweets.
2. Data Processing: After removing duplicate tweets, the dataset was reduced to 5,562 tweets.
3. Data Visualization: Using the tool Gephi, the links between the posts within the dataset were graphed. Twitter users engaged in discussions around the Deep Space Food Challenge were visualized as individual nodes sized on the basis of their total number of connections within the dataset. Further, different communities of users were represented by different colours.

4. Results

Do challenges help raise awareness about a particular topic?

The preliminary results of our social network analysis suggests that the Impact Canada Challenge model is effective in drawing attention to the problem and expanding networks of people and organizations engaged with the problem.

Using 'deep space food' to query and visualize social network activity, we observed a significant expansion in the network of actors and the number of interactions around the topic after the launch of the Deep Space Food Challenge.

Figure 1. Deep Space Food Social Networks Before and After Challenge Launch

• Figure Description - Engagement before Challenge launch.

  There were only a limited number of posts and engagement on the topic of 'deep space food' in the period between August 1st, 2019 and January 11th, 2021. Unsurprisingly, the largest nodes, representing the highest ranking social media actors in terms of the number of posts, are NASA and CSA_ASC.

• Figure Description - Engagement after Challenge launch. 

  A great expansion in the number of posts and individuals engaged on the topic of 'deep space food' were observed in the period after the launch of the Challenge between January 12th and May 31st, 2021.  

• Figure Description - Animation demonstrating the expansion of engagement from before to after the Challenge launched.

  Before the Challenge launched, there were only a small number of nodes. After launch, there was a sudden and sustained increase in the number of individuals and independent communities of connections discussing 'deep space food'. By the end of the observed time period, on May 31st, 2021, there are 1,709 tweets with over fourty thousand connections between them.

Another way to measure the impact of Challenges in raising awareness about the problem is to look at the number of interactions and engagements on social media. Table 1 below shows that there was a large increase in the number of discussions (e.g. tweets) about the issue on social media (nearly five times more tweets after launch of Challenge), as well as a significant increase in interactions and engagements (e.g. retweets, mentions, comments).

Table 1. Twitter posts and interactions mentioning 'deep space food' before and after the launch of the Challenge.

Sentiment analysis was conducted on the tweets to better understand the content of the conversations about the issue. Interestingly, the overall sentiment of the social media messages was positive and had a more positive score after the Challenge was launched.

Who are Challenges reaching and who is engaging with Impact Canada Challenges?

Using social network analysis, the team investigated who are the actors engaging with Impact Canada Challenges. As the Deep Space Food Challenge in particular was an international endeavour launched jointly in Canada by the Canadian Space Agency and Impact Canada and in the United States by NASA, it was no surprise that the Challenge reach was international in nature, with thousands of connections between the United States, Canada, and the rest of the world.

During the first 18 months since Challenge launch, there were 1,309 actors engaged in the network (represented as nodes) and 1,402 distinct interactions and relationships between those actors (represented as edges). The network is shown in Figure 2 below, using a geography-based layout, where the American network is illustrated in pink and the Canadian one in blue.

Figure 2. Deep Space Food Challenge Network using a Geography-Based Layout

• Figure Description - Mapping actors and interactions on a map

  1,309 actors engaged in the network (represented as nodes) and 1,402 distinct interactions and relationships between those actors (represented as lines connecting nodes) shown on a map. The American network is illustrated in pink and the Canadian one in blue. The Canadian network is centered around the Canadian Space Agency (CSA) and Impact Canada (Impact and Innovation Unit) in Montreal and Ottawa. Lines all over the map reveal interactions from across the globe.

Zooming in on the Canadian network, one can clearly see that the network is centered around the Canadian Space Agency (CSA) and Impact Canada (Impact and Innovation Unit). The size of the nodes in Figure 3 below is scaled based on eigenvector centrality calculations, which measure the degree of influence in the network. One interesting finding is that Impact Canada has nearly an equal level of influence as the CSA who is the problem-owner and is implementing the Challenge. This is a validation of the importance of Impact Canada’s reach and the benefit for Government of Canada departments to leverage Impact Canada’s resources to expand the network of solvers and reach a large talent pool. The graph also allows for the identification of different groups/communities engaged with the Challenge. Using modularity measures and color to determine community association, subnetworks can be identified, such as the orange subnetwork below, which is an academic network centered around the University of Guelph.

Figure 3. Canadian Network for Deep Space Food Challenge - Node Size by Influence

• Figure Description - Degree of influence within the network

  Shown are nodes representing actors engaged in the network. The size of the nodes are scaled based on eigenvector centrality calculations, which measure the degree of influence in the network. The node for Impact Canada has nearly an equal level of influence as the one representing CSA who is the problem-owner and is implementing the Challenge. Using modularity measures and color to determine community association, subnetworks can be identified. Shown are a number of subnetworks including one academic network centered around the University of Guelph.

What types of networks and connections have formed between challenge participants?

The team also used social network analysis to investigate the networks formed by applicants and participants in the Deep Space Food Challenge. The objective was to understand whether Challenges support the expansion of collaborative networks and partnerships. Interestingly, the team found that participants who were successful in the first stage of the Challenge (selected as a semi-finalist) were more likely than unsuccessful applicants to have a bigger network online and more likely to have a higher follower to following ratio (i.e. they had more followers for every person they followed on social media).

However, there are limitations to this data. First, the team was only able to identify the public social media account information for approximately 20% of the challenge applicants. Second, given the time and resources constraints, it was not possible to extract time-series data to understand when different connections were made. This meant that there was no meaningful way to study the evolution of those networks over time, including the establishment of a baseline for the size and structure of the participants’ network prior to Challenge Launch.

The participants’ network based on data from May/June 2022 is shown in Figure 4 below.

Figure 4: Deep Space Food Challenge Participants Network

• Figure Description - Challenge participants' network

  Shown is a graph of participant networks, displayed as spheres of nodes representing each participants reach and lines to represent connections between them (using approximately 20% of challenge participants and data from May and June 2022). In this representation, Noblegeninc and JustVertical have the largest networks.

5. Conclusions: What have we learned?

Social network analysis is a powerful tool that can be leveraged to measure the impact of Challenge competitions. For the Deep Space Food Challenge, used as a pilot for this study, the analysis seems to confirm that Challenges can greatly help in raising awareness about a topic. It can also study the diversity of actors engaged with Challenges, as well as the different networks formed between those actors, particularly Challenge applicants and participants. Identifying influential actors in a network can be an important tool for Challenges to use as a medium for reaching a larger number of solvers, as well as communicating results of the Challenge and promoting solutions.

6. What’s Next?

Moving forward, it would be beneficial to integrate the collection of social network data early in the Challenge process as many types of data cannot be collected retroactively (for example number of followers/following prior to a certain date). It would also be helpful to have closer coordination with Challenge applicants and participants in the collection of this data (for example, ask applicants to share their Twitter handles or other public social media pages).

In addition to Twitter, LinkedIn could be a very useful tool in impact measurement of Challenges. It can be used to better study participants growth via job postings and number of employees, as well as the type of talent hired and the expansion of professional connections over time. However, accessing LinkedIn data will most likely require a formal partnership with LinkedIn since public access to their API was shut off in 2015^{Footnote 1} and web scraping of LinkedIn data was deemed difficult due to the security features of the site^{Footnote 2}.

7. Acknowledgements

Thank you to co-authors Andrew Nguyen, Kristy Phipps, Nelson Tang, and Carl Zou from the Master of Data Science at the University of British Columbia (UBC), and to Statistics Canada.

8. Footnotes