Do you have time to answer the same questions again?
Data Values Digest #37
“Aap log kal hi to aye the na sawal jawab ke liye?” Laxmi* said when I asked whether she would participate in a survey. “Didn’t you just come yesterday for the interview?”
“No, that wasn’t us,” I replied. Laxmi was one of our sampled respondents for a quantitative baseline survey in 2016 in the Madhya Pradesh state of India for an academic research project on women’s political and social empowerment.
The day before I arrived, Laxmi had been interviewed by a different organization asking similar questions. This is a common occurrence in rural villages in India where household members—and especially women—are interview subjects of academic and private research groups, government, and non-governmental organizations.
According to governmental policy think tank NITI Aayog, there are 81 All India Surveys (nation-wide surveys on all topics), 70 percent percent of which were or continue to be conducted by government agencies. Ten surveys are collected annually. Of these, four cover topics related to demography and health. This list, which was published in mid-2021, does not account for surveys conducted by academic or private research organizations.
An employee of PRADAN, a non-governmental organization based in India, told me in a phone conversation last month that he estimated at least ten surveys are conducted in any rural village in India every year.
Most of these surveys share the same questionnaire or similar parts of a questionnaire such as the frequently-used Demographic and Health Survey’s section on women’s empowerment. PRADAN conducts a women’s empowerment survey in the villages where they work at least once a year. As a research associate with J-PAL South Asia between 2016 and 2018, I conducted surveys with empowerment questions twice in 2016 and once in 2017. These surveys are on average 45-60 mins long.
The burden of answering survey questions doesn’t fall equally on members of a household. In about 80 percent of household surveys, the respondents are women. For example, in the National Family and Health Survey conducted between 2020 and 2021, 724,115 (88 percent) women were surveyed compared to 101,839 men (12 percent).
Surveys are extremely important. They provide representative facts, views, and opinions and help shape policy based on the needs of individuals and communities. However, there are certain problems with conducting repeated large-scale surveys in the same communities with the same individuals, especially for women.
The problems with repeatedly surveying the same people
Firstly, this practice shows that surveying organizations place extraordinarily little value on the time of women who are already burdened with household chores, agricultural and farm work, and childcare responsibilities. Of course, consent is obtained before each survey. However, research shows that women—especially those in patriarchal and rural areas—have a tough time saying “no,” especially to an outsider who represents a figure of authority. Time spent responding to surveys is not spent on fulfilling her duties as a mother, daughter-in-law, and wife and can lead to backlash in the form of verbal and physical abuse.
Secondly, direct benefits of these surveys to the respondents are often either non-existent or hard to quantify. Sometimes individuals are offered payment for their time, but this doesn’t necessarily address the problems that the survey has identified. Even when there is a long-term benefit to a community, research takes years to publish and policies require time to take effect. Meanwhile, a woman who is asked the same “time-use” questions again and again still spends the same amount of time doing the same chores that are considered to be a woman’s responsibility.
The third issue with repeated surveys is the risk of response bias. Response bias refers to the ways respondents may be unduly influenced when providing answers on a survey. This can lead to survey errors causing serious threats to the internal validity of the data and any research published using these data. “Oh, I know the answer to that question,” Rani* grinned when I asked how often she went to the panchayat (village council) meeting. “People like you asked me the same questions a few months ago.”
The dilemma at the heart of this problem
There are two important reasons why the same questions are asked repeatedly by different organizations. First, even though most journals and funders require manuscript authors to share their data, most do not do so for various reasons. The lack of open data sharing is a major cause of repeated information gathering.
Secondly, organizations that share data are held to extremely high standards of maintaining confidentiality by Institutional Review Boards (commonly known as IRBs). To maintain privacy of individuals, these organizations publish data removing any sort of information that would reveal the personal identity of the interviewee. If another organization needs data with some personal identifiers for any reason such as merging two datasets or identifying areas where policies were enacted, the only option is conduct their own survey research and own the new dataset. Hence, they must embark on another data gathering expedition.
This creates a huge dilemma between valuing time or valuing privacy! Is there a solution that can find a balance between valuing both?
What can be done?
Better collaboration is key to addressing this problem. Surveying organizations are usually hired to conduct surveys and may not have capacity or authority to coordinate with other organizations. While there are opportunities to collaborate among agencies which hire these organizations (and indeed a UN agency exists to address this), one of the most efficient and feasible solutions (and one which is already in place albeit more in paper than in action) is data sharing or open data.
Where personally identifiable information is not required, manuscript authors, researchers, government, and non-governmental organizations are frequently required to publicly share their data. In most cases, they are required to make the data available on request or share it on repositories. If the dataset is not publicly available or authors do not share the data on request, as happens in many instances, serious actions should be taken against them. The process of data compiling, sharing, and accessibility should be streamlined. This will go a long way in reducing repeated information gathering.
Another possible solution could be to create independent entities at the local level to keep an inventory of any survey done in any village in the area. Let us, for example, assume that villages A, B, C, D are under the purview of local entity X. If an organization wants to conduct a survey in villages A and B, it would have to register the survey with all the details about the type of survey with X. With this system in place, if another survey organization wanted to conduct a similar survey in A, then X can ask them to contact the first organization to see if they could come to a data sharing agreement. If data sharing is not possible due to any reason, X can ask them to conduct surveys in areas C or D to avoid repeatedly surveying the same areas.
While maintaining privacy has been an important goal in the field of data collection up to this point, valuing respondent’s time has not. If we want to empower individual respondents and make policies to benefit them through collecting data from them, it is our responsibility as data collectors to respect respondents’ time. Otherwise, can we really claim we are doing the right, fair, and equitable thing?
Jasleen Kaur is a PhD Candidate at the LBJ School of Public Affairs and a Population Research trainee at the University of Texas at Austin. Her research aims to improve the social, economic, and political status of women. She focuses on the "implementation science" of policies and programs meant to empower women. Kaur has previously worked with J-PAL South Asia and Sanford School of Public Policy at Duke University, where she was involved with hands-on monitoring and evaluation of gender livelihood and maternal health projects respectively. Follow her on Twitter and Linkedin or send her an email.
* Names have been changed to maintain the privacy of respondents.
 | A guest post by
|