Impact and Innovation Unit Increasing response rates to a Statistics Canada Survey

1. Executive Summary

During the past several years, there has been a steady decline in Canadians’ rate of response to the Government of Canada’s mandatory and voluntary surveys. To address this problem, in Spring 2016, Statistics Canada approached the Privy Council Office’s Innovation Hub (the Hub) for consultation on how to increase response rates to these surveys. The Hub designed an experiment using the principles of behavioural insights (BI), a multidisciplinary approach that applies knowledge and tools from the behavioural sciences, including behavioural economics, psychology, and social marketing, to encourage positive changes in behaviour.

Collaborating with a behavioural scientist from the Behavioural Economics in Action Research Centre (BEAR), Rotman School of Management, University of Toronto, the Hub developed two simple, low-cost BI-interventions. The Farm Financial survey, a national mandatory survey, collects critical financial data from about 10,000 farms in total. This data is used to inform Government of Canada policies and programs. Currently, the survey is conducted by phone, after farmers receive an initial notification letter by mail. The experiment involved testing two new notices sent to farmers after the initial notification: a reminder letter and a reminder letter that included a significant process change—a five day deadline to proactively call Statistics Canada to schedule an appointment to complete the survey.

In July 2016, a total of 4,542 farms who had not yet completed the survey were randomly assigned to one of three conditions as part of a randomized controlled trial (RCT): a control group (no reminder letter); a second group (reminder letter); and a third group (reminder letter with the process change). The results of the RCT revealed that the reminder letter with the process change significantly increased the overall national response rate by 6.8 % (% change relative to control), and the provincial response rate by as much as 15.6% (% change relative to control).

This research project demonstrates that it is possible to significantly increase the response rate to a Government of Canada survey leveraging the principles of BI, experimentally validated through a RCT. Statistics Canada is one of the first departments to leverage the power of experimentation to produce better data and evidence-based results to improve service delivery and program outcomes.

This research project demonstrates that it is possible to significantly increase the response rate to a Government of Canada survey leveraging the principles of BI, experimen- tally validated through a RCT. Statistics Canada is one of the first departments to leverage the power of experimenta- tion to produce better data and evidence-based results to improve service delivery and program outcomes.

2. Introduction

For several years, Statistics Canada has observed a steady decline in the number of individuals and businesses completing Government of Canada mandatory and voluntary surveys. For example, in the last five years, there has been a four percent point decrease in the overall response rate to Statistic Canada’s household surveys. Recognizing a need to explore alternative approaches to encouraging survey responses, Statistics Canada partnered with the Privy Council Office’s Innovation Hub (the Hub) to apply behavioural insights (BI) principles to increase participation in a mandatory, agricultural survey.

After a careful review of a number of Statistics Canada surveys, the decision was made to focus on the Farm Financial Survey for a research project. The Farm Financial Survey, an initiative by Agriculture and Agri-Food Canada and Statistics Canada, provides data on farm assets, liabilities, revenues, expenses, capital investments and capital sales. This infor- mation is critical to examining the effects of federal and provincial agriculture programs and policies on different farm operations. The survey uses the Census of Agriculture to create a list stratified by province, farm type, and farm size. The target population for the survey consists of all active Canadian agriculture operations. Specific farms excluded from this target population include farms with less than $25,000 in sales from agricultural activities, institutional farms, community pastures, farms on First Nations reserves in select provinces, and farms that are part of multi-holding companies. This survey selected a sample size of approximately 10,000 units from the target population.

"Recognizing a need to explore alternative approaches to encouraging survey responses, Statistic Canada partnered with the Privy Council Office’s Innovation Hub (the Hub) to apply behavioural insights (BI) principles to increase participation in a mandatory, agricultural survey."

Participation in the survey by selected farms is mandatory. Currently, selected participants are informed by mail via a notification letter that includes a copy of the paper questionnaire for reference and preparation. Several weeks after the notification letter is sent, Statistics Canada employees call participants to collect data via a 25 minute telephone interview. If a participant declines to answer questions during the first phone contact, Statistics Canada continues to attempt to collect responses until the end of the data collection period. In an effort to reduce the burden on farmers, Statistics Canada offers participants the option of using the respondent’s tax data, acquired from the Canadian Revenue Agency, to complete questions on revenues and expenses.

3. The Experiment

The Hub worked with a behavioural scientist from the Behavioural Economics in Action Research Centre (BEAR), Rotman School of Management, University of Toronto, to develop an experiment to test the effect of two simple, low-cost behavioural interventions on Farm Financial Survey response rates: a reminder letter; and a reminder letter that includes a process change. The reminder letters incorporated principles of behavioural science, including framing and imposing a deadline, to encourage participants to respond to the survey. An initial notification letter, letting participants know about the survey and asking them to complete it, was sent out in June 2016.

Intervention 1—Soft reminder letter

This version of the reminder letter (see Figure 1) involved a relatively simple “soft” reminder letter, incorporating BI principles such as framing, grouping, and simplification. Extensive research in the field of behavioural science has shown that simplifying processes, incorporating plain language, and grouping concepts together can increase cognitive ease, and result in improved outcomes.

Intervention 2—reminder letter with process change

This version of the reminder letter (see Figure 2) also incorporated framing, grouping, and simplification principles, but also included the addition of a process change. Currently, Statistics Canada phones participants at various times of the day in the hope that they will have time available to complete the survey. With this intervention, participants were given a five-day deadline to proactively call Statistics Canada to schedule an appointment to complete the survey. The idea of encouraging participants to proactively call Statistics Canada to schedule an appointment builds on existing research that shows that individuals are more likely to follow through with an action, such as a medical appointment, if a date and time are scheduled for them. Although pre-scheduling appointments was not an option with the current appointment system, we wanted to test whether encouraging farmers to proactively call to set an appointment time that was convenient for them would have any impact on outcomes. Additionally, existing research in the behavioural sciences shows that imposing a deadline can create a sense of urgency and encourage people to take action. The letter also included a tear-away feature to help prevent missed appointments by encouraging participants to write down the date and time of their appointment.

The randomized controlled trial

A randomized controlled trial (RCT) was designed to test and compare the effectiveness of the two reminder letters against the control group (no reminder letter). On July 14, 2016, a total of 4,542 participants that had not yet completed the Farm Financial Survey were randomly assigned to one of three conditions:

Control Group: No Reminder Letter
Intervention 1: Soft Reminder Letter
Intervention 2: Reminder Letter with Process Change

Figure 1 - Intervention 1

Dear Sir or Madam:

You have been selected to participate in the 2016 Farm Financial Survey. Information gathered from survey participants like you is needed to understand the challenges that farmers face today. Results from the survey will help inform important policy decisions that may affect you and your business.

You cannot be replaced

Farms are carefully selected across Canada using known statistical methods and we cannot replace your farm with another. We are relying on your participation to ensure that the survey results are as accurate as possible.

This survey benefits you because it provides the required evidence to:

determine subsidy levels and the prices of agricultural products
examine the effects of agriculture programs and policies on different types of farm operations by province
study the competitiveness of the agricultural industry.

Participating is easy and secure:

An interviewer will contact you to arrange an interview time that is convenient for you - days, evenings or weekends.
Under the Statistics Act, all information you provide to Statistics Canada is confidential, and used only for statistical purposes.

For more information, please call 1-XXX-XXX-XXXX (TTY: 1-XXX-XXX-XXXX) or visit www.statcan.gc.ca/survey.

Thank you for your participation in this important survey,

Name
Title
XXXXX

If you have already completed the 2016 Farm Financial Survey, please accept our thanks and disregard this letter.

Figure 2 - Intervention 2

Dear Sir or Madam:

You need to make an arrangement to complete the survey within 5 days of receiving this letter. To complete it now, or to schedule an appointment at a time that is convenient for you (days, evenings or weekends), please call 1-XXX-XXX-XXXX (TTY: 1-XXX-XXX-XXXX).

This survey benefits you because it provides the required evidence to:

determine subsidy levels and the prices of agricultural products
examine the effects of agriculture programs and policies on different types of farm operations by province
study the competitiveness of the agricultural industry.

Participating is easy and secure:

Call 1-XXX-XXX-XXXX to complete the survey now, or to schedule an appointment at a time that is convenient for you – days, evenings or weekends.
Under the Statistics Act, all information you provide to Statistics Canada is confidential, and used only for statistical purposes.

For more information, please visit www.statcan.gc.ca/survey.

Thank you for your participation in this important survey,

Name

Title
XXXXX

If you have already completed the 2016 Farm Financial Survey, please accept our thanks and disregard this letter.

-----------------------------------------------------------------------------------------------✂----------------------------------------------------------------------------------------------------------
2016 Farm Financial Survey, Statistics Canada
Appointment scheduled for:

Date: Time:

Number: 1-XXX-XXX-XXXX (TTY: 1-XXX-XXX-XXXX).

4. Results

National results

At an aggregate national level, the randomized controlled trial results show that the reminder letter with process change (intervention 2), in which participants were given a five-day deadline to call Statistics Canada to schedule an appointment, significantly increased response rates to the Farm Financial Survey. This intervention increased the overall response rate from 59.1 % (the control) to 63.1 %, a statistically significant (p < .05) increase of 6.8 % relative to control (% change). The soft reminder letter (intervention 1) also had a greater response rate than the control group (61.4 % vs. 59.1 %), but this difference was not statistically significant.

Figure 3. Overall results of the randomized controlled trial

Text version

Group	Response Rate	Increase relative to control group
Control Group	59.1%	N/A
Intervention 1	64.4%	3.9%
Intervention 2	63.1%	6.8%

Provincial results

Broken down by province (see Appendix A), the group receiving the reminder letter with process change (interven- tion 2) performed significantly better than the group receiving no reminder letter (control group) in several provinces: Alberta a 15.6 % increase, British Columbia a 13.9 % increase, and the Atlantic Provinces¹ a 13.1 % increase (all % change relative to control). Although not statistically significant, it is important to note that in the province of Quebec, the control group out-performed both interventions 1 and 2, with intervention 2 resulting in a 6.5 % decrease in response rate (% change relative to control). Consideration should be given to exploring the underlying factors that might have led to this difference.

Figure 4. Select Provincial Results of the Randomized Controlled Trial

¹Atlantic provinces were grouped together in order to have a large enough sample size for analysis

Text version

Province	Group	Response Rate
Atlantic Provinces	Control Group	57.8%
	Intervention 1	63.0%
	Intervention 2	65.4%
Quebec	Control Group	61.7%
	Intervention 1	64.9%
	Intervention 2	57.9%
Alberta	Control Group	52.4%
	Intervention 1	54.2%
	Intervention 2	60.6%
British Columbia	Control Group	57.2%
	Intervention 1	59.1%
	Intervention 2	65.2%

Conclusion

This pilot project with Statistics Canada demonstrates that it is possible to significantly improve Government of Canada survey response rates through experimentation and the use of behavioural insights. In the case of the Farm Financial Survey, on an aggregate national level, the addition of a reminder letter including a process change with a five-day deadline (intervention 2) increased survey completion by 6.8 % relative to not sending a reminder letter. Broken down by province, this same intervention resulted in increases of as much as of 15.6 % relative to the no-reminder condition.

Statistics Canada is one of the first departments to lever- age the power of experimentation to produce better data and evidence-based results to improve service delivery and program outcomes. This study demonstrates that significant positive changes can be achieved using simple, low-cost innovative approaches and interventions.

Appendix A: Table 1 - Provincial test results

Province	Group	Response Rate	RR Confidence Interval
Atlantic Provinces	Control Group	57.8%	(49.2% - 66.3%)
	Intervention 1	63.0%	(54.6% - 71.4%)
	Intervention 2	65.4%	(57.8% - 72.9%)
Quebec	Control Group	61.7%	(55.0% - 68.2%)
	Intervention 1	64.9%	(58.2% - 71.4%)
	Intervention 2	57.9%	(50.9% - 64.7%)
Ontario	Control Group	69.5%	(64.3% - 74.7%)
	Intervention 1	71.2%	(66.1% - 76.4%)
	Intervention 2	72.4%	(67.5% - 77.4%)
Manitoba	Control Group	54.1%	(46.0% - 62.2%)
	Intervention 1	52.4%	(44.8% - 60.0%)
	Intervention 2	54.9%	(46.7% - 63.1%)
Saskatchewan	Control Group	55.2%	(49.0% - 61.3%)
	Intervention 1	59.7%	(53.4% - 66.0%)
	Intervention 2	59.6%	(53.5% - 65.7%)
Alberta	Control Group	52.4%	(45.6% - 59.1%)
	Intervention 1	54.2%	(47.5% - 61.0%)
	Intervention 2	60.6%	(54.3% - 66.9%)
British Columbia	Control Group	57.2%	(49.5% - 64.9%)
	Intervention 1	59.1%	(51.6% - 66.7%)
	Intervention 2	65.2%	(57.7% - 72.7%)

*Note: At the significance level α = 5%, the significant difference between the Control Group and others are seen in bold.

Appendix B: Glossary of terms

Term	Definition
Behavioural Insights (BI)	A multidisciplinary approach that uses the principles and tools of behavioural economics, psychology, and social marketing to influence behavioural change in a positive way.
Grouping	The easier information is to process, the more likely people are to act on it.
Cognitive Ease	The easier something is to do, the more likely people are to do it.
Deadlines	Imposing a deadline can create a sense of urgency and encourage people to take action.
Framing	The particular way in which a request is worded can substantially influence people’s responses.
Implementation Intentions	Encouraging people to form a plan to carry out their intentions has been shown to increase the attainment of desired goals in a variety of domains.
Randomized Controlled Trial (RCT)	A research method that tests the effectiveness of implementing different types of interventions. By randomly assigning different groups of people to receive different interventions (or no intervention) and comparing these groups, this method provides clear evidence indicating which intervention(s) result(s) in superior outcomes.
Timing	Changing the timing of an activity can affect the way in which individuals respond.