Tomorrow I will be moving to New Delhi, where I will spend the next nine months studying the sampling methods of Indian pollsters through the sponsorship of a Fulbright-Nehru grant. This blog is here to collect my research findings, share my experiences, and record post-worthy anecdotes. I intend for it to be a mix of the professional and the personal.
For this first post, I have tried to lay out in question-and-answer form what my research is, how I’m doing it, and why it is important. I hope you will read it and continue to follow updates on my research.
What are you researching?
This project will explore how Indian public opinion researchers accurately measure public opinion in the world’s most populous and diverse democracy. I will do this by working closely with three different organizations–CSDS, CVoter, and Impetus Research–to examine instances of systematic sampling error that are unique to the Indian context, and study how Indian researchers address these challenges.
Sounds technical. Why does this research matter to anyone not interested in public opinion polling?
Public opinion influences your opinion. No matter what type of society you live in, or how much or little you care about the politics of your country, what everyone else around you thinks has enormous influence on you. How many times have you seen a news article or headline saying that “__% of Americans think so-and-so about immigration” or “__% of Americans disapprove of the job Obama is doing,” and paused to think about your own opinion? We are social creatures, and naturally want to know how our opinions stand with the rest of the group. Measuring the opinions of the public accurately is important to the construction of our own identities.
Public opinion also has tremendous influence on politics–local, regional, national, and global. It influences the policies that your government prioritizes, writes into law, and implements. It influences the way politicians try to market those policies, and the terms they will use to describe those policies. Politicians want to know that they aren’t out-of-step with their constituencies, so that they represent them properly and continue to be elected. They care so much about this that they pay large sums of money to pollsters to keep a close eye on public opinion and their standing with the public. Even dictators try to keep a a finger on the pulse of public opinion, as they want to know that their actions will not provoke mass protests and their overthrow.
So the accurate measurement of public opinion is important in the formation of our individual opinions, and for the functioning of our bodies politic. However, the methods we have to measure public opinion are inevitably flawed. My hope is that by looking into one of the potential pitfalls of polling–instances of systematic sampling error–in the Indian context, lessons can be learned about how to more accurately measure public opinion in India and elsewhere.
What is systematic sampling error?
Surveys, by their definition, do not record the opinions of every member of a population. They only record the opinions of a sample, which is supposed to be representative of the population they are measuring. However, there are many reasons why a survey’s sample may not be representative of the population it is designed to study. Pollsters love thinking about all the types of error that may explain why their survey results do not accurately measure public opinion. For simplicity’s sake, we will think here only about sampling error, which can be divided into two categories: 1) random sampling error and 2) systematic sampling error.
Random sampling error is easily quantifiable and almost always reported with survey results. It is conveyed by the margin of error that is reported with a survey. You might see that a poll says “50% of Brazilians think X”, and has a margin of error of 3%. This means that that if you were to do the same survey of Brazilians many times, you would expect results showing that between 47% and 53% of Brazilians think X about 95% of the time. In other words, you have about 95% confidence that the true value of how many Brazilians’ thinking X is somewhere between 47% and 53%. You can’t report with 100% certainty, because you are only talking to a sample of Brazilians, and randomness in how the sample is drawn might slightly bias the sample in one way or another (towards richer Brazilians, or poorer Brazilians, or Brazilians in a certain region, etc.).
Systematic sampling error is a hairier beast. Also referred to as sampling bias, it stems from a systematic bias in how the survey sample is drawn. If the survey instrument being used is imperfect in some systematic way, so that it over-samples certain types of a population and under-samples others, then you have systematic sampling error. Some examples will help illustrate this concept.
The classic example of systematic sampling error comes from the 1936 presidential election in the United States, at the dawn of public opinion polling. President Franklin Roosevelt was running for re-election against the Republican governor of Kansas, Alf Landon, and two famous polls were taken that year predicting the election result. The first was conducted by the Literary Digest, which sent out 10 million(!) ballots to a list of addresses it acquired from telephone books and automobile registries, as well as the Digest‘s own subscription list. Two and a half million ballots were sent back in. The second was conducted by the American Institute of Public Opinion, led by George Gallup. Gallup interviewed only 60,000 respondents, but used quotas to ensure that his sample matched the demographic profile of the United States. The Digest predicted that Landon would win by 55% to 41%, Gallup predicted that Roosevelt would win by 56% to 44%. Roosevelt won in a landslide, defeating Landon by 61% to 37%.
Why was the Digest so wrong? Gallup conducted surveys the following year that showed how systematic sampling error was to blame. Remember that the Digest drew their sample from telephone and automobile owners, as well as Digest readers; these groups were more likely to vote Republican than the rest of the population. By not sampling at all from the considerable share of Americans with no telephone, automobile, or Digest, the Digest was missing the population that was most likely to vote for Roosevelt, an error that Gallup did not make. Systematic bias in the Digest‘s method of sampling badly damaged their credibility, and they were shuttered only a few years later (Anyone more interested in the 1936 poll result can read a more comprehensive analysis here).
Recent examples from the American example include the 2010 Senate race in Nevada. Pollsters predicted that Democratic senator Harry Reid would lose to his Republican opponent Sharon Angle, but Reid was re-elected by a comfortable margin. Analysis after the election suggested that pollsters without Spanish-language interviewers were systematically under-sampling households where Spanish was the only or preferred language, a sizable Democrat-leaning share of the population in Nevada. Another recent example is the 2012 US presidential election. Pollsters that did not dial cell phones showed more favorable results for Romney. This was because the households that used only cell phones were more likely to belong to demographic groups–younger, lower income, minorities–that leaned towards Democrats.
Of course, systematic sampling error is not unique to American polling, or Indian polling. Any time a sample is drawn and a survey is taken in a certain way, there is a risk of systematic sampling error. It is simply a feature of the landscape of public opinion research with which pollsters must grapple.
Why is India the right country to study how pollsters address the challenges of systematic sampling error?
For three reasons: 1) history, 2) diversity, and 3) democracy.
First, India has a rich history of high-quality survey research. The first national polls were done in the 1950s, only two decades after survey research began gathering steam as a field in the United States in the 1930s, by the Indian Institute of Public Opinion, which was modeled after Gallup’s American Institute of Public Opinion. Since its first national election survey in 1967, the Center for the Study of Developing Societies has regularly conducted national and state election polling. This history predates the advent of widespread survey research in other developing countries. During the 1990s, there was a proliferation of opinion surveys and exit polls as media coverage of polling increased. Indian researchers therefore have decades of experience to draw upon in addressing the challenges of systematic sampling error.
Second, India is an incredibly diverse nation. At 1.2 billion people, India is the second world’s largest country. Borrowing from Ramachandra Guha’s India After Gandhi, there are five main demographic axes along which Indians can be sorted. First, there is the unique caste structure that sorts Indians into five strata of social status. Second, there is language; India’s constitution recognizes twenty-two official languages, and which languages are recognized as “official” has been a source of contention in the past. Third, there is religion; India is an 80% Hindu nation, but has the second largest population of Muslims in the world. There are a number of other religious minority communities, including Christians, Sikhs, Buddhists, Jains, and Parsees. Fourth, there is socioeconomic class; large disparities exist between the rich and poor in both urban and rural areas. Fifth is gender; although India has had both a female prime minister and president, women face discrimination in a number of realms as patriarchal elements of Indian society persist. Based on my preliminary research, systematic sampling error has historically occurred along the first, third, and fifth of these axes. Lower castes, religious minorities, and women tend to be underrepresented in polls.
Third, with the exception of a two-year period during the 1970s, India has been a democratic nation since its independence in 1947. Quinquennial national elections and frequent state elections are ripe for the natural experiments of social science researchers. India’s democratic politics means that survey respondents are more willing to express their political views, and that survey researchers need not fear the government shutting down their work.
With whom will you be working to do this research?
I have already reached out to and received approval to work with three different organizations: the Centre for the Study of Democratic Societies (CSDS), the Center for Voting Opinion & Trends in Electoral Research (CVoter), and Impetus Research. In order to examine the issues related to systematic sampling error that each organization faces, I will be interviewing staff, observing fieldwork, and working directly on projects with all three organizations.
Presently, I intend to structure my research around the electoral calendar of the upcoming year. The state of Bihar has elections this November, West Bengal and Kerala have elections in May of the upcoming year, and Puducherry has elections in June, so I anticipate working on polls in each of these states. CSDS, CVoter, and Impetus will also be conducting surveys outside these provinces that are not related to elections, so I anticipate looking at sampling issues in more than just these states.
Public opinion plays an enormous role in shaping your own opinions and the policies of your government. India’s long history of high-quality public opinion polls, its incredibly diverse population, and its democratic politics make it the ideal country to study how pollsters grapple with the challenges of measuring public opinion correctly. By working closely with three Indian research organizations over the next year, I hope to shed light on how pollsters address the challenges of measuring public opinion in the world’s largest and most diverse democracy.