Dhananjai Joshi on the challenges of polling in India

Dhananjai Joshi, Managing Director of Cicero Associates and Consultants, has over eleven years of experience conducting large-scale sample surveys. He received his training in Research Methods at the Survey Research Center, Institute of Social Research; University of Michigan, USA and the Afrobarometer School at the University of Cape Town, South Africa. From 2003-2009, he worked at the Centre for Study of Developing Societies (CSDS) with Professor Yogendra Yadav and accredits most of his unique learning to this association. Dhananjai has been published in the Economic and Political Weekly, The Hindu, Hindustan Times, The Indian Express and India Today besides his chapters in books published by SAGE and Oxford University Press. He has been a political commentator on TV channels such as Lok Sabha TV, Sahara Samay, Focus TV, News X and Headlines Today.

On Wednesday afternoon, I met with Dhananjai at the Cicero office in Malviya Nagar to talk about his experience doing public opinion research in India and the challenges he has encountered. The transcript of the conversation below has been lightly edited for length and clarity.

On polling in India

Sam Solomon: Please tell me about your background conducting public opinion research in India.

Dhananjai Joshi: I was a student of history. When I was towards the end of my college, I met Professor Yogendra Yadav. Way back in 2003. Then he was the most prominent pollster in india. Besides being a media pollster, he was also a very serious academic. So I met him and I got very bad grades at my college. So he said that’s the best opportunity to work with him. So he got me in as an intern during the 2003 Delhi elections.

SS: At Lokniti?

DJ: At Lokniti. At that time he was the co-director of Lokniti. He used to head it. So he got me in at Lokniti-CSDS. I started with coordinating fieldwork and towards the end of the election I was already doing analysis using SPSS. So he made an offer, and I stayed there until I realized that I needed more education. So CSDS sponsored me to the University of Michigan – Ann Arbor summer school, at the Survey Research Center, and to an Afrobarometer summer school. I went there, learned my analysis skills, and of course coordinating surveys doing fieldwork, looking at the challenges of actually executing polls is something that I was doing at CSDS. Then I also did a Master’s in political science — switched from history — and went back to CSDS, worked there for a few more years, until 2009, after which I started off on my own.

In 2011, we formed Cicero and in 2013, Professor Yadav was our first big client. The AAP [Aam Adami Party] was our client in 2013. And thereafter we started, and have been polling ever since then. Lately, we’ve been working with the India Today Group.

SS: You’ve been doing survey research in India since 2003. During that time, how have surveys in India and the polling industry changed, and what aspects of it have not changed?

DJ: The polling industry has had its own highs and lows. 2004 was a big low. Everybody predicted that the Vajpayee government, the NDA government, would come back to power. They didn’t. So the whole industry went through a credibility crisis. This is something that kept happening on and off after that because there are times when the whole industry gets polls wrong. All of us get up and say, “Let’s do something about this.” And, “What can we do about this?”

The one thing that we want to do is to get some basic standards of polling, something like a basic disclosure about your methodology so that we and the consumers of polls can actually differentiate between a credible poll and a poll which is done without a very rigorous methodology. Unfortunately, that hasn’t happened. I remember way back in 2004 there was a WAPOR [World Association of Public Opinion Research] meeting in India. Some time in 2004, 2005. Nothing happened after that. So unfortunately the industry hasn’t really grown from that.

Media consumption has grown. At that time, we had about two or three channels. Now we have God knows how many channels, fifteen hundred channels. Local channels, national channels, regional channels. Everyone wants to do a poll.

People are still not very clear about the methods of polling. We as an industry haven’t done much to educate the consumers. Or our clients, for that matter. So people also don’t pay a lot. So ultimately what happens is that there is no organized fieldwork protocols. It’s a highly unorganized sector at the ground level because there are small vendors who are practically man-powered suppliers of unskilled labor. So what we do is we try to train them to follow our protocols and this is something that’s been happening for years. In the last few years, we at Cicero have tried to organize it. But still it’s an ongoing challenge. And as an industry, we haven’t really moved much yet.

SS: When you talk about basic standards of disclosure, what specific things are you talking about?

DJ: Your sampling methods, your sample profile. Your basic sampling method.

SS: Margin of error?

DJ: Margin of error.

SS: I didn’t see margin of error in the Cicero poll in India Today.

DJ: Margin of error has been something that we consciously don’t put in, because what people tend to see is that margin of error is one big figure that gives you the whole error in the survey. Margin of error is something that we can only compute on one variable. Right?

SS: One variable, that’s correct, yes.

DJ: Right. Now unfortunately, about ten years ago, the fashion was to write, “2 percentage point margin of error at 95 percent confidence interval.” Now this is something that nobody will understand. A normal viewer will not understand. So margin of error is something that we stick to this, that it’s about 2 percentage points.

SS: Depending on the sample size.

DJ: Depending on the sample size, and depending on the kind of confidence interval at which you are computing. But again these are things about educating the consumer, and our client needs to give us that kind of space to educate the consumer. It’s been quite a task to get our clients to publish or give us time on air to give our sample profile. That’s something that we’ve succeeded in doing. At least I’ve succeeded in doing with my clients. And I know for sure that CSDS has also succeeded.

SS: But margin of error, no. Because I’ve seen this in other polls — it’s not just Cicero — where there is no reporting of margin of error.

DJ: Margin of error is again something that we must put. And we need to put in a disclosure that this is the margin of error at X confidence interval, and this means that this margin of error has been computed on this variable. Now in most election surveys, voting is the crucial question. So you need to compute it on the voting question. We need to do that, yeah.

On the challenges of polling in India

SS: What are some of the challenges that you’ve encountered during your career 1) in terms of accurately measuring public opinion, but 2) — and we kind of discussed this already — communicating your results to clients, to the media, or to the public?

DJ: The first question is about accuracy. There, fieldwork quality has always been a big concern. If your fieldwork quality is right, and we have our parameters of checking it — back-checking is one — then we look at the profile of the same. And the profile of the sample gives us a good idea of how good or bad the fieldwork was. Then, it’s statistical techniques that one needs to get your forecasts right. That’s not been that big a challenge.

Fieldwork quality is a constant challenge — investing into workshops, training people, back-checking, using technology. For instance, we used a mobile tracker to track the location of our field workers. Are they actually at the polling booth or not? What time? For how many hours are they in the field at the location? What is their movement? So we started using technology. Technology will further help us.

The second thing is about communicating results. And that’s been a big problem because if you look at any pollsters’ Twitter handle, you are going to see there are trolls all over the place.

SS: [laughs] I have seen that actually, yes.

DJ: As one of my clients and a very senior editor once said, “When you do a lot of polls and every political party has been angry with you, that means you are on the right track.” So we’ve had a track record where everyone’s been angry with us.

It’s difficult to communicate it. Because these are after all estimates. And these are pre-election estimates. Even an exit poll is an estimate. And it’s thrilling to get results 72 hours before the actual result. But in 72 to 96 hours, you have the actual results. So it’s an estimate. One needs to explain to people that it’s after all just an estimate. We are not biased. We have showed each and every political party winning or losing. That’s the way it happens. But it’s still difficult to convince people that we are not biased. That polls don’t influence results. That’s been a very, very big problem.

SS: You’re saying polls don’t influence results?

DJ: They don’t influence results to a large extent. You are working on it. You’ll probably come across a lot of research where there is a bandwagon effect and there’s also an underdog effect. That way we assume the errors cancel each other. I, in a normal election, would assume that the bandwagon effect and the underdog effect cancel out each other.

In some cases, like the AAP in 2013 was using our poll and publishing it, to show that they are viable. My own sense is–and we did some work on it–that on the one hand, there are people who are sympathizing for the AAP and they thought that it’s viable and that’s why they went and voted for it. But at the same time, there were a lot of people–because 70%, 70 odd percent, of Delhiites didn’t vote for it–these are the people who thought well, the AAP is viable, so they went out in larger numbers and voted against it.

So I don’t think polls really influence results, although I agree with the Election Commission of India that it’s not ethical to show poll results while voting is ongoing. So that is why unlike ten years ago, I think until about ten years ago, you could see poll results while voting was ongoing. You would get trends from 3 o’clock, trends from 4 o’clock. Now until voting doesn’t end, we can’t air a poll, which is fair.

But by and large if polls could influence election results then I think it would be a different world altogether. Most democracies would be very different, and pollsters would be very powerful.

SS: You worked with CSDS for some time, which is a publicly funded research institute, and you started Cicero, which is private sector. You’ve done work for political parties and political candidates. How has your work for Cicero been different from your work for CSDS?

DJ: Well, getting in the money has been more difficult. To fund your work has been more difficult. To get clients has been more difficult.

SS: With Cicero?

DJ: With Cicero. Otherwise, as far as the work is concerned, it’s the same. Because basically all of us follow what’s known as the Michigan model of behavioral research. That’s what’s been followed by most pollsters in India. In fact, I think all pollsters. Which is to say, you may get a cross-sectional sample and do a face-to-face interview.

CVoter has started doing a lot of telephone interviews. I am still skeptical about the sample profile of telephone interviews, though in a few years of course, the way mobiles are penetrated in India, you will get a cross-sectional sample there as well. A good cross-sectional sample.

SS: A number of people that I’ve talked to have critiqued the lack of transparency from pollsters in India regarding methodology. You were saying this a little earlier. What do you think can be done to make pollsters more transparent about how their surveys are done? Do you think that responsibility lies with pollsters?

DJ: I don’t believe in external regulation. Regulation has to be internal. Some of us have got together and informally founded something called the Indian Polling Council. We might stick with this name, we might change the name, but eventually we plan to get it registered as an association and have a certain protocol for mandatory disclosure. So sample profile, margin of error on whatever is the key variable, defining that key variable, basic sampling methods. These are certain things that we need to do. Mode of interview. These are certain things that we need to do. And what we plan to do is to actually make it mandatory, and those pollsters who will not follow this will not be part of the council. So that’s one of the things that we are thinking of doing.

The other thing that we plan to do is to actually have a provision for peer review. Where we are going to be six or seven founding members, including non-profit organizations like CSDS, private sector players like Cicero and CVoter, multinationals like Ipsos and Nielsen, and all of us together are going to have a panel. And in case there’s a controversy with a particular survey, the raw data should be able to go for peer review. And there will be some basics that the peer group is going to check into the data and give it a clean chit or say there is a problem with the data. So those who are going to sign up to this are going to be in the council, and I think that’s going to get some credibility. So that’s what we’re planning.

SS: The narrow objective of my research is systematic sampling error, whether certain populations–you know, religious groups, caste groups, gender, etc.–because of cultural factors or socio-economic factors are more or less likely to be selected and participate in a survey. I’m curious to know which populations you’ve found to be most likely to be undersampled in a survey.

DJ: Traditional sampling error in a field setting like India has been [that] women get undersampled, urban areas are oversampled, upper castes are oversampled, and upper classes are undersampled, are underrepresented.

SS: They’re underrepresented?

DJ: The upper classes in urban areas. Because field investigators don’t get access to these people. Now they are anyway a very small proportion of the population. And then field investigators who come from more humble backgrounds don’t get access to houses in Delhi’s…

SS: South Delhi.

DJ: Yes, South Delhi elite localities. Or even for that matter in the outskirts of Delhi, group housing societies. It’s quite a task to get permission. So what we do is we send a supervisor well in advance, get permission from the society, generally to interview only on a Sunday or something like that, and then go ahead with it. These are very common sampling errors that even with the most rigorous sampling method, polls tend to make these errors.

We’ve tried to control for it. We’ve been fairly successful at controlling these problems. Women have always been a problem because you have less female investigators.

SS: And female respondents here are not comfortable speaking with [male investigators]?

DJ: Not always. Not always. So we, in our polls–even in the recent Bihar polls–women are underrepresented. But we’ve been fortunate with tribals, dalits, etc. These groups are getting underrepresented.

SS: This is in Bihar or just generally?

DJ: Generally. Tribals and dalits are underrepresented. Because they are in rural pockets where generally people don’t go. We have been fortunate because our sampling method is quite rigorous. So we try to do new things all the time and try to get that extra effort to get an edge over our competition.

SS: What about religious groups?

DJ: Religious groups, minorities haven’t been a problem.

SS: No? Not at all?

DJ: Not with us.

SS: Not Muslims?

DJ: Not with us. I haven’t seen minorities being a problem. Muslim women have been a problem.

SS: What about with Sikhs or Christians?

DJ: Not a problem. Not a problem. If there is a group which is more than 2.5–3% in the population, and your sample is somewhere bigger than 2,000, you can get it. So you don’t get exactly 50 cases. You get 30 cases. You get 35 cases. Something like that.

On the challenges of making seat projections

SS: In light of the challenges of converting vote share into seat share, do you think that Indian pollsters are held to an unreasonably high standard in terms of predicting number of seats?

A lot of people I’ve been talking to have mentioned the 2004 election. You mentioned it earlier. But that was just one election cycle. It’s naturally going to happen that some cycles the polls will be a little off. Are the standards just too high?

DJ: That is because our election system is such. In a first-past-the-post system, your proportion of votes and proportion of seats are not equal. So we get the multiplier effect. It’s something that we need to look at.

SS: The multiplier effect. Can you explain that?

DJ: The multiplier effect is the ratio of vote shares to seats, in crude terms. In India, if a political party gets 30% of votes, it’s not important that they’ll get 30% of seats. They might end up with 30% of votes, but they might end up getting 70% of seats, or 60% of seats, where the multiplier effect is positive. It’s over two.

On the other hand, there are elections where the party might get 30% of votes and only get 15% or 20% of seats. So now this is something that is quite to difficult to predict. This is one of the reasons.

And looking at past results, looking at patterns, if the sample is very well spread, one can control these errors. All I will say is that if we are put to higher standards, that’s what we’ve signed up for, right? We’ve signed up to give an accurate number. I think it’s important to be humble and acknowledge that it’s a challenge, but at the same time saying it can’t be done is something that… If it can’t be done, then why are we in the trade? We are in the trade to give a number.

SS: The multiplier effect is really interesting. You have a similar phenomenon in the United States for the House of Representatives. So, for example, in 2012, Democratic House candidates won more votes but Republicans won more seats. But it seems like the multiplier effect here in India is so much greater because there are so many more political parties.

DJ: One is that. The second is the demography of votes. I’ll give you a small example. You’ve heard of the Bahujan Samaj Party, the BSP? Mayawati is its supreme leader.

SS: Yes.

DJ: The BSP, for years, used to have a negative multiplier effect. The reason was the BSP had a primarily dalit vote base. Dalits are spread across the state. So in every constituency of an average 1 lakh, 1 lakh twenty thousand voters, BSP would have anywhere between twenty to thirty thousand votes. In every constituency. So if you take an average, it would be about 20 odd percentage. If let’s say 120,000 voters is the average, 20,000, 24,000 would be what? About 20 odd percent. So the BSP had a social base of 20 percent in every constituency, or 18-20% in every constituency. That gets it 18-20% of votes average in the state. But with 20% of votes, it can’t win a single constituency. To get that, they needed a plus. So let’s say, in a state like Uttar Pradesh where there are 403 seats, in 100 odd seats it managed to get more than 40,000 votes. It would win 100 seats. But its vote share spread across the state. So how do you predict that? BSP adds up its 25% of votes [to] say, 18% of seats, because the demography is such that it’s spread across the state.

On the other hand, a political party like the Samajwadi Party has a core of Muslim-Yadav votes. So in pockets where Muslims and Yadavs live together in higher numbers, they’ll have higher votes there. So seats are going to convert in higher proportion. So the multiplier is positive. So these are the kind of challenges that we go through.

SS: But in order for Indian pollsters to make the calculation from vote share to seat share, then don’t you have to do individual constituency-level polling?

DJ: No, we can’t do that. Nobody sponsors that.

SS: That would be so expensive to do that.

DJ: That would be so expensive. And our per-cost interview is much cheaper than what it is in the United States. In India, I don’t think there is a single pollster who will be getting — at least from the media — something like even $8-10 per interview, which is very cheap. I remember going to Michigan in 2006 and there we were talking to the Director of the American National Election Study. I told him that we had a sample of 27,000 in our National Election Study. He said, “That’s a lot of money!” I said, “No, it wasn’t a lot of money.” I gave him an account, and he said, “What the hell? This is cheaper than our 2,500 interviews.” [laughs]

So that wouldn’t happen. Constituency-level polling will not happen. So we have to fine-tune our swing models. So someone like me would use different base files, different swing models, to get to one number. And then not actually average out different models, but you know, pick something from some model and something from some model and…

SS: Make some individual judgments.

DJ: Make some individual judgments.

SS: And is there such a thing in Indian elections as uniform swing? For example, in the 2014 midterms [in the United States], Republicans generally overperformed expectations, overperformed the polls, not just in one state or district but across the country. We knew from the fact that Mitch McConnell did this well in Kentucky that Republicans were also going to win in North Carolina and maybe in Virginia and other states. Is there such a thing where you can look at a certain state or constituency in India and make projections about other constituencies?

DJ: I don’t know [for] others, but I can’t think of anything like this.

SS: Every district is really its own universe?

DJ: Yeah. Although in the last Lok Sabha election, everyone underestimated the National Democratic Alliance, the BJP-led alliance, in every state. In UP [Uttar Pradesh], we were very, very close. We were one of the most accurate ones. We said 41.5% votes and 62 seats for the BJP. 60 or 62 was our higher end. Still, we underestimated them. They got 42.5% or 42.6% and 73 seats. This is something that happened in every state.

Lately, many people have been arguing that we tend to underestimate the winner. I don’t see much merit in it because Maharashtra was an election where we got it bang-on.

SS: This was when?

DJ: The 2014 [Vidhan Sabha elections in] Maharashtra where the BJP came to power. They got, I think, 124 seats, we said 122 was our mean. We were the most accurate poll. But we were the most accurate poll because we were more conservative than others. Everyone else had the BJP end up at about 140, 145. Some of our friends said 160 or something.

What I feel is that every poll, every state, every election is a new set of challenges. Standardizing it is very difficult.

Bihar is turning out to be such a close election. The first wave of the pre-poll that we did, we had a two point lead for the NDA. Now a two point lead is quite a risky lead, because it’s well within the margin of error. So we said 40 and 42 for the two main alliances. And I told my sponsors, this could be actually 38 to 44. And they were of course…

SS: They didn’t like that.

DJ: [laughs] They didn’t like that. They said, “We paid you to give us a number.” I said, “Okay. Fair enough.”

Now this time around, I said 41 for the alliance which got 40 last time [the Grand Alliance] and 39 for the alliance which got 42 last time [the NDA].

SS: Statistically within the margin of error. It’s almost the same result.

DJ: It’s the same result. Now what the trolls would say is, “You got paid. You’re corrupt. You manipulated it. You got paid.” One can’t respond to each and every troll.

I know that on the eighth of next month, the results will be out. We’ll know how accurate or how wrong we were. If we got it bang-on, I’ll still say, “Well, this is fortunate, but next election will be another challenge.” And if we got it wrong, we’ll say, “We’ll introspect, and try and get the next one right.” So each election, I think, has its own set of challenges.

SS: Thank you very much.