Praveen Rai on the challenges of polling in India and why he thinks seat predictions should be banned

Praveen Rai

Praveen Rai is the Academic Secretary at the Centre for the Study of Developing Societies (CSDS) in Delhi. Prior to working as Academic Secretary, he was a Project Manager at Lokniti, Programme for Comparative Democracy, a research programme of CSDS that specializes in election studies (full disclosure: CSDS is my primary research affiliate, and I am working with Lokniti on my research). While at Lokniti, he handled more than fifty election studies and opinion polls between 2005 and 2009.

His research interests include electoral politics, monitoring election opinion/exit polls and political participation of women in India. His seminal work “Electoral Participation of Women in India: Key Determinants and Barriers” was published as a special article in Economic and Political Weekly in 2011. He also co-authored a book, Measuring Voting Behaviour in India, with CSDS director Sanjay Kumar that was published by Sage in 2013. His articles have been published in Indian academic journals, edited books, newspapers and blogs. In much of his writing, he has been critical of the use of opinion polls as tools of political communication.

Last Wednesday, I sat down with Praveen in the CSDS lounge to talk about his experiences managing polls with Lokniti and his criticisms of media reporting on opinion polls. The transcript of the conversation below has been lightly edited for length and clarity.

On his involvement in public opinion research

Sam Solomon: Please tell me about about your background conducting public opinion research in India and how you got into this field.

Praveen Rai: I am basically an LLB. I have done my master’s in history from Delhi University and my law [degree] from Delhi University. I have worked in different private companies, small and big, looking after legal affairs. But somehow that job didn’t interest me, being a very routine thing. So I ha[d] been trying to make a career switch.

I just got a chance to come to Lokniti while I was looking for a job. I was just working here as a research assistant. But while working on that slowly I developed interest in opinion polling.

SS: You didn’t have the interest before though.

PR: No. In fact, I had been interested in politics and looking into election surveys on television and newspapers. That used to interest me. But I had never thought of getting into this area.

So I came here as a research assistant on a very temporary contract looking for a job, making a job change. And slowly I got into this. So I started right from the ground by doing the first time exit poll, which went wrong for me, as a field investigator. And then I did a few more fieldworks, and then slowly I got into questionnaire designing. So with a period of time I got into this field more by default in the sense that that was not my job. But I started learning , right from questionnaire designing to conducting fieldwork to managing large projects, and then data analysis and all that. So slowly I got into this. I started writing about it. I got an opportunity at Lokniti. And over a period of time I developed an interest in this.

So I think I worked almost four years where I was the project manager to Lokniti, and I think I conducted more than fifty or sixty opinion polls. That included election studies and studies in democracy. And different kinds of opinion polls. The Prime Minister’s survey, to find out what are the opinions that people had of Prime Minister Manmohan Singh after his first term. In fact, during his term in three years. So slowly I got into that but then I learned the whole thing and it became an interest area for me.

SS: Over what time period is that? When did you start getting into Lokniti and CSDS?

PR: I have a very long association with Lokniti. As I told you, the first one when I came here in 2002 as a research assistant. So I worked there for six months, and as I told you I did field investigation, I did field supervision, about how interviews are done, how student field investigators do. And after that I again went into the private sector where I was handling corporate communications. But there also I conducted some election studies for some political parties also. That was a learning stage for me.

In 2005, when Lokniti got into an informal contract with CNN-IBN for doing all the election studies and different opinion polls, that was the time I got the invitation from Lokniti. In October 2005. Then I came here. But by that time I was managing projects, I knew how to do field investigation, supervision, and all those things. And from 2005 to December 2009 when I was here, I got a chance to get into questionnaire designing, data analysis, and then I started writing. Now I’m all about opinion polling right from field investigation to putting the reports out.

SS: You have an office at CSDS. Are you still affiliated with Lokniti?

PR: No. In 2009, I got into my current assignment, that is like work as an academic secretary. So my role is a little different. To look about academically and liasoning with funders. The reason why I got here was this is a permanent job. In Lokniti, that was a contractual thing. Every year it used to be renewed. So at the moment I am not actively involved in opinion polls in this sense.

But still I write. If you see over the period of the last four or five years, it has changed in a sense. Now I don’t do much quantitative writings. I think everything is finished. All my writings and book chapters. And then there was a book also which I wrote with Sanjay [Kumar, Director of CSDS]. it’s called Measuring Voting Behavior in India.

Since last year I started tracking opinion polls. The way opinion polls and findings– in fact, now ten or fifteen agencies do [opinion polls]. So five or seven agencies say this is winning, and five or seven agencies are saying the other parties is winning. That confused me a lot. So then I got into writing about opinion polls. I tried to find out, I tried calling them up to find out what is their methodology, where was the fieldwork done, do you have the questionnaires. But the response from the different polling agencies has not been good.

SS: Were some better than others at responding?

PR: No, except for a few agencies like AC-Nielsen and all, which are market leaders in a sense. They are market research agencies who do it in a very, very proper methodological way. In fact, the quality of their surveys are, I would say, as good as Lokniti-CSDS. But apart from that, most of the agencies won’t even spell out who are the people who did the field investigation, what exactly was their sample, was it in residences or on streets. So these are things which are not coming out into the public domain.

Opinions polls are mostly on elections. Apart from elections, you get very few opinion polls. Eight to ten, you’ll get, to see in country which are on other issues. That is mostly done by Lokniti. The agricultural studies, the youth study. Apart from that, most of them are doing [surveys] during elections. Round after round — pre-polls, post-polls, exit polls.

SS: You’ve been involved with Lokniti-CSDS for almost fifteen years, since 2002. I’d like to know how measurement of public opinion has changed over that time, for CSDS and more generally in India.

PR: I told you that since things are not in the public domain, we don’t have a forum where the oldest people who were involved in public opinion polling get together and share those things. Nothing of that sort has developed in the last ten years. Though there have been a lot of people who have been trying to take initiative, I think it is not going. So at the moment I can just speak about CSDS surveys, Lokniti surveys. So I will just speak about the time when I joined and how we changed. And before that we have the history which you can find out from our book where me and Sanjay, we have written a very long chapter on how the surveys in Lokniti have changed.

During my period, in fact, the changes which happened in Lokniti surveys [were] at the stage of getting all the questionnaires here, cleaning it up, data punching, and then doing the analysis. From 2005 onwards, slowly we started decentralizing data entry and data cleaning. We developed it in different states. So now what’s happened is that it’s not centralized in a sense. All the questionnaires from India don’t come here and then we don’t have to put in huge teams with deadlines to do all the data cleaning and data punching for the analysis. So we decentralized it.

Apart from that, also since the number of sample increased — if you see our National Election Study, the National Election Study’s sample increased — so what we started doing was that we had four or five questionnaires. Questions on elections and politics, we had ten or fifteen questions which were in all the questionnaires. But in different questionnaires we started asking people on different issues, like security issues, economy, and all that. That way, in fact, we had more spread of getting opinion about politics, economics, price rise, and all these things. So I think these are the two major changes that have happened in the last ten years in the Lokniti surveys.

But the common thread which runs right from the first survey until now, and that we haven’t changed, that is we still do face-to-face interview of sampled voters in their residence. That is one common thing that is there in the last forty years of CSDS surveys.

SS: Going back to the 60s.

PR: Yes. Starting with the 60s.

SS: My research is looking at sampling error and whether certain populations are more or less likely to be sampled and participate in a survey. I’d like to know whether, in your experience working on Lokniti-CSDS polls, certain populations were more or less likely to be sampled in surveys, because of cultural factors or socioeconomic factors.

PR: In fact, the best part of the opinion polling which you do on elections is that in the the last ten years, all the voter lists are available online. So your universe of the study is– what happens is that you get the list of the complete universe. And we have been using the random stratified sampling system. So what has happened is that we have tested it in the 90s. We have been using those and once we get our data then there are certain indicators. We have matched the profile of the voters which we have done in our survey with the profile that exists in the state or at the all-India level. We have government data on the gender breakup, on religion, on caste and community. By using our random stratified sample, by the end of the 90s we could see that in fact we had been including a very, very representative sample. I think the sampling method in fact does not exclude anyone and is quite representative of the population.

SS: Even with the sampling being properly inclusive, did you find that certain populations had a higher likelihood of non-response and not participating in the survey?

PR: Well, I think not. We don’t have data for that. No, in fact, we have data for that and there is no difference in the non-response based on caste, community, or socioeconomic [level]. Non-response has been for many reasons, like people not living in that area, or migrating somewhere, or the day you reach for the interview they’re out of their village, they’re not in their residences. But there is no difference in the sense that the non-response of rich people is more or the non-response of poor people is less. It’s more or less uniform.

SS: What about for men or women? For religious minorities?

PR: For that we also don’t see any differentiation because what we do for the Lokniti surveys — you know how the network operates — we have a network of different universities and they’re mostly students who are doing their master’s or their MPhil. They are trained in how to do interviews, how to handle questionnaires. There we keep a gender equilibrium, in the sense that 50% of the investigators are men and 50% are women. Female investigators mostly go and do interviews with females because in villages or even if you live in religious communities, like Muslims, for a male to go and interview is a problem. So we have been using female investigators for the interview so we don’t have any problem of non-response from a particular section like Muslim women. Even in north India where you have the purdah system, where a male cannot go and do a face-to-face interview, but definitely the females goes to do [the] interview. So we have taken care of this. I don’t think there is any high non-response.

SS: In your article for Economic and Political Weekly, “Fallibility of Opinion Polls in India,” you write about four types of errors: coverage error, sampling error, measurement error, non-response error. Which of these four types of error do you think presents the most significant challenge for Indian researchers? If you have specific examples you can refer to in your time working for Lokniti or CSDS, that would be most helpful.

PR: When I am talking about all these errors, then definitely it is there in most of the surveys. But you have to see how high it is. Even if you see a non-response error — if it goes to three or four or five percent — it doesn’t make much difference in the sample, because the non-response of people… It’s not like the non-response of male respondents are more or of female respondents are less. I think non-response error does not create much problem. I am talking specifically for the Lokniti surveys. After that, I’ll talk about the marketing or opinion polls. In fact, all these errors are within the limits and they cancel each other out and they don’t have [an] effect on the total sample.

But now as far as the market research opinion polls which are coming to the public domain in the last four or five years, because of which I wrote that article, they are not coming out with the proper methodology. They are not telling us exactly who are the people who did [the survey]. I think the main reason they are getting it wrong could be, one, is the sampling method. Because a face-to-face interview on a proper random sample is a very, very costly proposition in India with the kind of length and breadth of the country, and very interior areas. So I think one of the major areas where most of the polls are going wrong is on sampling.

Even if they get their samples correct and their field investigation correct, the second error could be the measurement error because getting the right vote share… [between] the time the vote share of a survey which you do and the election results are declared, the vote shares are within the limit of one or two percent. It doesn’t make much difference when you translate it into seats.

But the kind of results which we are getting — as I told you, five saying Aam Adami is going to win and five saying BJP is going to win in [the 2015] Delhi assembly elections. And it was a clean sweep. Just three seats for BJP. So there in fact it will be very, very hard to tell that because of these errors the surveys were wrong because with huge margins of the vote share, no survey can get it wrong. There comes in what I talk about, the vested interest of people in the media, their manipulating seats to create a bandwagon effect before the elections.

On the use of polls as tools of political communication

SS: In the article, you used pretty strong language, talking about how election surveys have been reduced to a “media gimmick.” If that wasn’t the situation in the past, why do you think that has happened and what can be done to change that?

PR: As far as the bandwagon effect is concerned, I think it was way back, around twelve or fourteen years back, when a series of surveys were done by CSDS and they used to come out with different report cards for MLAs [members of legislative assembly] and that was there in all the Hindi and English newspapers. A lot of people read it — we cannot say exactly what percent — but most of the people were aware of this survey. In fact, it came out daily for one month. So in the last round of surveys when they did all these MLA report cards, we had put a question, how many people have read about it? We were very surprised that just seven or eight percent of people had read about all those MLA report cards. And it was coming out in all the Hindi newspapers and English newspapers. But only six or seven percent were aware of it, the MLA report cards in the survey. And those who had read about it or heard about those polls, we asked them another question, what effect did those readings have on voting for a particular party? Did it change your original decision? They said no. Whatever decision they had taken was taken by us and the poll had no effect. The findings of our poll had no effect.

This is way back, I think, 2002. In the last eight to ten years — in fact, I am talking about pre-poll surveys just before the election — when you have a series of elections, say, OK, BJP is going to win, Aam Adami Party is going to win. Even Aam Adami Party, they also did a survey for the first [2013 Delhi assembly] election where they had spoken about getting something more than fifty seats. I think they got something like twenty-three or twenty-four seats. There also they had used survey findings done and Yogendra Yadav had joined the party at that time. They were saying that they were going to get more than fifty seats. And on the other hand, Congress was going into the elections. Because they were the incumbent party, they were saying they were going to get a majority. [The] BJP did their own polls and also said, “We are going to get a majority.” Now these are the three main parties and all three are doing that. But when the election results came out, the BJP had thirty-three seats, [AAP had] twenty-four seats, and Congress was completely wiped out [Editor’s note: In the 2013 Delhi legislative assembly elections, the BJP won 31 seats, AAP won 28 seats, and Congress won 8 seats, leading to an AAP-led government with the support of Congress].

So that means all the surveys that they did and all the findings which [were] put out, those were just to solicit votes. That was to create a bandwagon effect, that, “OK, now we are winning.” In fact, when all these reports were out, they were putting out banners and all those things. They were doing a lot of publicity.

SS: But based on that and what you just said, it seems like there isn’t a bandwagon effect then.

PR: No, this is something conjectural until the time we do a study to find out that people are aware of those findings and based on what they’re finding, “OK, Aam Adami Party is going to win and then all the traditional BJP supporters are voting for that.” So this is just conjectural because a study has to be done and definitely none of the polling agencies are going to do that. So I think the only one who can get into this and do a proper study to find out whether we have a bandwagon effect or not is the Election Commission of India. So that is what I have been telling them. Just ban seat predictions. Seat predictions. I’m not talking about banning surveys. Surveys are very, very important. Election studies are important.

The reason why I wrote in EPW is the fact that the purpose of doing election surveys from a instituit . Political parties have also been doing that. But the purpose is now to find out what is the vote share they are going to get. Will this candidate do better? So they are using polls to find out how they are placed, how they can do that, what kind of candidate should do that. So the purpose was that.

So I think until that stage, until the stage where the Vajpayee government was there, before that, in fact, they had a minister called Pramod Mahajan, the first to come out openly and admit, “OK, we also do opinion polls but the purpose of doing our polls is to find out how we are placed in a particular contest, what kind of candidates we have to put in.” He was the first politician to admit that they do their opinion polls.

Until that stage, that’s not a problem. But doing an opinion poll, fudging data, putting out reports to say, “We are going to win,” that is not a purpose of opinion polls. That is not a purpose of election studies. And that is basically telling lies. The purpose of opinion polls in India were to find out the mood of the voters, like what kind of assessment we have of the incumbent government, what are the issues they think are important, and whether they have formed voting decisions until now or whether they are going to make voting decisions once the campaign ends. That was how opinion polls developed since the 1990s.

But I think over the last five years now, whatever they are doing, the most important thing is to just tell the people who is going to win. So now they are using it as a political communication. That is giving an extremely bad name to the polling agencies. And though we [CSDS] do a different kind of work — our purpose is to do election studies, do evidence-based reporting of how a party fared, do post-mortem analysis, and [make] data available for researchers who come in — at the moment everybody is clubbed in. Lokniti did seat projections for a couple of years and we dropped them because it was not scientific and it was not working well. But still people feel that, “OK, CSDS is also saying that.” So people have now started looking at opinion polls, they think it’s all biased and being done by interests to further that.

On why seat forecasting should be banned

SS: You make an argument that seat forecasting should be banned. You think the reason is not just because there’s a lack of transparency with how they come up with the seat forecasts, it’s [also] that the parties are using these seat forecasts to generate some kind of bandwagon effect that will help the party. And it’s not being done using scientific examples.

I spoke to Rajeeva Karandikar, who used to do the seat forecasts for CSDS. He’s pretty open about how he does the seat forecasts. He doesn’t give you all the specifics, but he kind of walks you through the general principles. He wrote a big article for The Hindu Centre [on this].

If more seat forecasts were done in a way that a very detailed methodology statement was included with the forecast, do you think that would be okay? Or do you think it will still be a form of political communication that political parties would use to manipulate public opinion?

PR: What has happened with opinion polls is that seat forecasting has taken the center stage. If we get a report out in a newspaper on an opinion poll on elections, the only figure that people want to see is who is winning, how many seats. As I told you, whether it has a bandwagon effect or not, most political parties think that it is a part of a communication campaign. So they have started using this for doing it.

I have [said] that the Election Commission has taken a very, very harsh view in the last five years. Nothing much has been done. A time had come when they were talking about banning election surveys, in fact, before the elections. Now they have a time period. Once the polling process starts until it ends, that is a time you can do your survey but you not come out with any reports. But just before the election process starts, you can do as many polls as you want to do, as many seat projections. Instead of banning election surveys, what I have argued is that seat prediction should be banned or else the other alternative can be what happens in Japan.

In Japan, seat projections have been banned. What had happened is that earlier they used to do seat predictions but when it went wrong, people went to the courts, the political parties went to the courts and filed a case, and people who did wrong forecasting were penalized heavily. The courts there said that if you get your predictions wrong, you have to pay for that. So seat projection doesn’t happen. At the best, they do a lot of election surveys and opinion polls, but they talk like, “BJP is leading and Aam Adami Party is second or third.” They talk like this, but they don’t assign any specific figures about, “This is the number of seats they’re going to win.”

So I think that would also be one way to deter media. In fact, if a political party does something and then does it through media, you know, an India Today poll… What happens is you’re masking your poll. Ultimately [the] media has to be made accountable, because television is  a very, very important way of reaching the homes. And elections, politics in India, it creates more passion as compared with other countries. (laughs) People are political in that sense that they are interested. And when a media house puts out something it’s like putting a stamp on that survey. So either you put a huge penalty that if your prediction goes wrong — you cannot just say that BJP is going to win a two-thirds majority and when the results come out you find that the opponent side won [a] two-thirds majority. You can’t be so wrong. Either you impose a huge penalty on the polling agency and the media house which is showing it so that they stop doing all these gimmicks and all these bandwagon effects. Or the second and easier method could be that we just tell them which party’s vote share they can talk about, which party is leading, which is winning, but don’t give out those numbers.

SS: But don’t you think that the vote share would then become… It seems to me that even if you banned seat projections, the political parties still might try to gin up bandwagon effects by saying vote share: “According to this CSDS poll, we have 45 and these guys have 41.” I feel like vote share could also be a form of political communication, even if the seat share were banned.

PR: No, I agree with that. Once you ban the seat projection, people can come out with the vote share and they can talk about it, “This party is going to win the election and get a huge majority.” All those kinds of things will happen. So I think there has to be some body, the Electoral Commission of India has to set up one unit whose work would be to see what kind of media reports are coming out. And if somebody says that, “NDA is going to get a majority, and BJP will  get a majority but they will be the largest party,” but after the election you find the BJP gets a majority on their own. That does not amount to much tinkering of data or misuse of election surveys.

But you can’t do a projection that NDA is going to get a two-thirds majority, and when the results are out, you find that the opposite party has got that. So when they do that, I think the Election Commission has the right to get all the data and find out whether the survey or fieldwork has gone wrong, the vote share they have computed has gone wrong but the survey is correct, the vote shares are correct and they have manipulated the data, and based on that some punishment needs to be enforced if they want to restore the faith in election studies back. Because it has taken a very, very bad beating since the last 2004 national elections, they are getting it wrong.

And even if you see this election [2014 Lok Sabha elections], because I did not do a follow-up article on this, here also in fact nobody spoke about a BJP majority. They all said NDA will get a majority. But nobody could in fact that BJP is getting a majority. So that means the vote share which you got in your survey which you are showing, there’s some problem. And the problem can be of fieldwork, of data collection, wrong sampling. At the moment, we are just grappling in thin air. Whether the fieldwork is wrong or the fieldwork is right and people are manipulating at this stage, the media are manipulating. So I think the responsibility needs to be fixed, either with the polling agencies or the media houses.

SS: And do you think the responsibility for changing this, the culture around polling in India, that falls on the Election Commission of India?

PR: That’s the body which puts a stop to election studies because once elections are declared, most of the powers are with the Election Commission of India. The last one which we had, whatever orders they had were [carried] out. As I told you, the day the election notification takes place, from that day until the day the last vote is cast, that is a period they have said you cannot come out with a report of opinion poll. Even that is ambiguous in a sense. You can do your survey but you cannot come out with those reports. And I think that is being followed by most of the polling agencies and media houses.

So I think the Election Commission of India should get a study done to find out the bandwagon effect, so that it can tell the political parties, media houses, and polling agencies, “Your reports [don’t] make a difference.” Though I am just being conjectural. Maybe it makes an impact on the voters and some voters are changing because of the election surveys. And based on that the responsibility needs to be fixed so we have fairness and transparency in opinion polling to restore its image. Because at the moment it has taken a very, very bad beating.

SS: Have you heard about the Indian Polling Council initiative that’s supposed to be launched in the next month or so? There was an article in The Hindu about it.

PR: I am not aware of it.

SS: In the article, some pollsters — in the article, Yashwant Deshmukh and CVoter are the ones that are quoted — CVoter and five other polling agencies are thinking of coming together and starting their own group to come up with some standard guidelines for disclosure of methodology and transparency in terms of reporting on their polls. The idea is that if you belong to the Indian Polling Council, you have to report things like sample size, sampling methodology, etc.

PR: If this is happening, I think it is a very good initiative in a sense because what I feel is that instead of the Election Commission of India drawing guidelines for do’s and don’t’s for polling agencies, if they all come together and form their own thing where they share their things, and they can come out with reports, I think it would go a long way of restoring the credibility of opinion polls in India. So it’s a good initiative and if it happens, it will definitely help.

On the challenges of polling in India

SS: In your own experience — you’ve worked on fieldwork, you’ve worked on questionnaire management, you’ve worked on all the stages of opinion polls — what are the biggest challenges that you found in accurately measuring public opinion in India?

PR: As far as challenges are concerned, there are a lot of challenges in the sense that this whole face-to-face interview is a very, very hard task. There are villages where you have to walk for kilometers on the hills to reach them. Lokniti does it, but the cost involved of doing a face-to-face interview is very, very huge in this country. That is one of the biggest challenges, and I feel for most polling agencies they don’t get enough funds to do this.

Secondly, doing a telephonic interview in India is still not possible because the mobile phones’ reach has increased but still most people don’t have the telephones. So making a shift from face-to-face, compared with US and other European countries where face-to-face interviews are very, very less and most people do telephonic interviews and now using different platforms [the] Internet. So I think this presents a huge challenge.

Apart from that, as you know we have a very, very diverse country. Different languages and all that. So if you do a survey at an all-India level, like Lokniti, we translate a lot of questionnaires into state languages. For most of the states, we are doing that. But for other polling agencies, we are not sure whether they are employing, like if you want to do a survey in Karnataka, you need to have a researcher from Karnataka and you need to have the survey in the local languages. That is one of the big challenges and this is also [a] huge cost. So whether these polling agencies are doing it or not, that I am not very sure about it. At least Lokniti does it, but still it’s a major challenge because even if you see a language, even in a state, there are so many dialects, that translating the questionnaire into dialects and getting the investigators with those dialects to do the interview, that is a huge challenge.

In fact, these are the most important challenges of doing an opinion poll in India.

SS: Anything else you’d like to add?

PR: I’ll add an example of the kind of the challenges you face. As I told you, from 2005 until 2008, for three years Lokniti-CSDS, we did seat predictions. And that was mostly for assembly elections. So I think we had nine assembly elections during that time. We got seven right, the [election results] were exactly within the range. But we got one completely wrong and one partially wrong.

The election had taken place in Punjab. There also the fieldwork and everything went off well. We were using exit poll data. With the exit poll, what happens is that the moment the polling ends, say the polling ends at 5 o’clock, the television program has to go live exactly at 5 o’clock and by 5:30 you have to tell exactly which party is winning. So that means that you have to close your fieldwork by say, around 3 PM, because then you have to transmit data and all those things, and analysis has to be done, seat prediction has to be done. You get just two hours. So with Punjab we did that. It stopped at 3 o’clock and we got our data. We did the seat prediction and all.

But what happened is that the election was extended from 5 until 9 PM because there were heavy rainfalls that took place. Here all the predictions and television programs were on since 5 o’clock. We collected our data until 3 o’clock. And then the rain stopped at around 5 or 6.

Congress and the Akali Dal, they were in a very, very strong contest, just a margin of two or three percent. In fact, the Akalis were winning. So what happened is the rain stopped. A lot of Akali voters, they put their voters in their buses and they went and voted at 9 o’clock. Now for the exit poll we closed our field investigation at 3 o’clock. So our data was loaded in favor of the Congress. So we mentioned that Congress was going to win. But actually it was Akali Dal which won.

So we predicted something, it went wrong. We did a fair investigation to find out how come we got it wrong. Even for exit polls we do purposive sampling to make it representative. So we missed the whole, the Akali Dal voters go when the rain stopped at 4 o’clock. They went to the voting. And the voting was extended until 8 o’clock in the evening. That also, people didn’t inform us. So a huge section of people in a lot of places voted at that time. It’s a very strange thing. With all the Akali voters, what they do is that they get buses and tractors and all those kinds of things, where they go together and vote. So this is one challenge which nobody foresaw. And we got it wrong.

SS: What year was this?

PR: You have to check. This was Punjab elections. I think, Punjab elections 2007 or 2008. You will get that if you do a search [editor’s note: Punjab Vidhan Sabha elections were held on 13 February 2007].

SS: That was the one seat projection that you got way wrong, and there was another one which you said you got partially wrong.

PR: The other one where we couldn’t predict, I need to check. Was it Tamil Nadu? No, Tamil Nadu, we got it right. The other one I don’t remember. But two we got wrong. In the third, also, in fact with Tamil Nadu, what happened, as I told you, is that you have some error, the coverage error. There was some errors. But what happened is that the errors for non-response and the errors for non-coverage, both cancelled each other out. So we got the vote share correct. But if you see the survey, and if you see the initial data that was coming in, there was some problem in fieldwork. We couldn’t get the fieldwork done properly. It was a problem with investigators, with fieldwork, with supervision. There were logistical problems. We did not get the vote share. But one mistake, cancelled by the other mistake, and we got the vote share correct and we made the right prediction (laughs).

SS: And what was this for?

PR: This happened in Tamil Nadu, this was also between 2005 and 2009 [editor’s note: Tamil Nadu Vidhan Sabha elections were held on 8 May 2006].

SS: For Vidhan Sabha?

PR: Yeah, for Vidhan Sabha. Our prediction was perfect. But if we see our data, and what had come, and what were the errors, as we do a review of all our surveys once we do [them], and mostly during seat projections.

So two wrong, and third one we got it right by default. So we decided that doing seat predictions and vote shares, it’s a very, very tough situation in India. And so we stopped. I think 2008 we stopped. 2009 Lok Sabha elections, we did not make any prediction.

And then there are some interesting cases. In one of the Karnataka assembly elections — not the last one, the one before that — I very well remember that Congress had got 32% of the vote and [the] BJP got 30% of [the] vote, but [the] BJP got the majority of seats and they formed the government [editor’s note: In Karnataka’s Vidhan Sabha elections held over two phases in April 2004, Congress won 35% of the vote share and 65 seats, while the BJP won 28% of vote share and 79 seats].

SS: Because their vote share were spread out in a way that was more advantageous.

PR: Yeah. The Congress vote share was completely spread out and they were concentrated in different assembly constituencies.

I think that was also one phenomenon that we first saw in Karnataka, because before they we hadn’t seen in any elections, having more votes but getting less seats. Karnataka gave us that lesson.

And apart from that, if you see the case of Bihar, so [in] Bihar they were completely new political coalition formations. If you see Bihar, first of all, there were so many parties, new coalitions. When you have new coalition partners, you don’t have any record of the previous vote shares. You have nothing. And how that coalition works on the ground, the vote transfers and all that. That is very, very difficult to ascertain through an opinion poll. Until the time that you don’t have some quantitative study done to substantiate your quantitative thing, you can never get a seat prediction.

Where there is just two main parties in contest like it happens in Tamil Nadu, a couple of states, it happens in Rajasthan where [the] BJP and Congress is there, it happens in Uttarakhand, there once you get a vote share [which is] correct, like 45% for Congress and 38% for [the] BJP, you can easily do a projection based on that. And mostly you get it right.

SS: But it’s more difficult when there’s more parties, like Bihar or UP [Uttar Pradesh].

PR: These are the challenges, and how the model is going to take into account these things, for that, we say that until the time polling agencies don’t come on a single platform, they don’t share their prediction models, and if they’re not going to improve upon that, they are going to be more and more wrong.

SS: And for the Karnataka poll where Congress got a larger vote share, but BJP got a larger seat share, did CSDS do a seat projection for that election?

PR: I think we did that and maybe that was one survey we got wrong.

SS: You had the vote share right but the seat projection didn’t predict that [the] BJP would end up winning more seats with less votes. OK, I’ll check that.

PR: That you can check. I think we have it on our record though I don’t have it because we did a complete check of our seat predictions.

It started in 2005, exactly with the Bihar elections. We got it right.

SS: The seat projections.

PR: The seat projections. For all the political formations, we got it correct. We got the vote shares correct. That was the first seat prediction we had done that was really celebrated for getting it right. And then it went on for a couple of years. It was 2008 or something, I think, when we got two or three wrong.

SS: This was the year of Punjab and Karnataka.

PR: Yeah. Punjab and Karnataka.

SS: Wouldn’t you say, if you got seven out of nine right, that’s a pretty good record, right? Maybe you think statistically because of some of the errors, because some of the challenges of polling, you’re going to get a few wrong, but the overall track record of seven out of nine, or whatever it is, that seems pretty good. I guess it’s a matter of focus on the fact that we got seven out of nine right, or we got two out of nine wrong.

PR: No, but see, if you see the pre-poll surveys which we have done — at the moment we are just doing post-poll surveys — you can see the data that we have got in all the pre-poll surveys. You will not find a single poll where you find the actual vote share in that election — if Congress got 40%, then we have got 30% — it was never so wide. Our vote share would come to around 38%, or maybe 1% more. It has always been in the range of 2%. Plus or minus 2%. In fact, the vote share has never swung more than that.

SS: Is this for post[-polls] or pre[-polls]?

PR: Pre-polls. Just before the elections. In fact, we have got it within the range of 1-2%. So you can just imagine that we have got the vote share 95% of times right. So that means that the sampling method which we are doing, the field investigation we are doing, our systems are perfect in the sense that we get it right.

But even with those right vote shares, when you do a seat prediction — as I told you, we got six or seven right out of nine. One or two we got it right by default. By default, one error cancelling the other error, as I told you. That means that that system is still not completely developed. Either you do a lot of research and development, when you can put in all these variables, like concentration of votes and all these kinds of things and different vagaries different coalitions. Either statistically over a period of time, and if somebody does that, but at the moment I don’t see anyone.

Because even if you see a new polling agency which just came up, its name is Chanakya, Chanakya got it right in 2014. You will not believe. They got the seats right for BJP, okay, and all the polling agencies missed it. Everybody missed it. And I think everybody laughed also again. BJP will get a majority. Everybody was laughing. “Who is this Chanakya?” and all that. And then Chanakya got such a big name. And then again have started floundering. Delhi, they have got it completely– Bihar, they have got it completely opposite. Now people are laughing. They are saying that Chanakya by mistake, instead of putting UPA, they are putting NDA with some grammatical mistakes [editor’s note: Today’s Chanakya exit polls showed an NDA landslide. Following the announcement of election results, they said their prediction was off because a computer error had mixed up the names of the alliances]. (laughs)

SS: Yeah. I saw that. Interesting excuse, yeah.

PR: Prannoy Roy is one of the leading psephologists in this country. He was the one who started doing opinion polls through his channel, and all the swing votes, and all that, he made it so popular. So I think Yogendra Yadav and Prannoy Roy, not only are they the best political analysts but they are the leading psephologists. And the way Prannoy Roy got it wrong this time, he had to render an apology. So now this really calls for a question. In fact, people should ask. The media should go and ask him, “What kind of a survey [did] you [do]? Your survey was wrong.”

Because there are a lot of stories going around, and they were trying to show this, push this, that should [the] BJP comes in power, they have all these taxes left and these debts will be paid. They are getting pressures from above.

SS: It sounds like CSDS has a pretty solid track record. You’ve gotten a few seat projections wrong. And maybe that’s because of the methodology, because of the focus on post-polls rather than pre-polls. It seems like when you do pre-polls, you try to do them as close as possible to the election as you can.

PR: The whole purpose of doing an opinion poll for us is it’s a study. In fact, we call it election studies. Our purpose is that. The reason why we started going to the media is that there is a huge cost involved in doing this. Until the 80s, the government agencies provided funds, but after [the] 80s, there was no funds available for conducting election studies. It is still not a discipline in India. It is still a sub-discipline.

So when we had the tie-ups with the media, that helped us in doing our surveys and giving them the required data like they are just interested in the vote share, the popularity ratings of the leaders, those kinds of questions. But apart from that, using those funds, we have been doing a huge, a very, very in-depth study of that election. So the data is with us. So our purpose was that. And when we got into seat prediction, at that time [the] media said, “Since you all are experts in election studies, why don’t you get into this?” And we also went into that to see how can we refine, and the model which Rajeeva Karandikar, I think, is still using is the one that was developed at that time. Then we started doing the seat predictions. But once they started going wrong, we said we don’t want to do it. Because our purpose is not that. Our purpose is to have a set of data on each and every election that happens in the country.

SS: And to understand why the results are happening and not necessarily just predict them beforehand.

PR: Yes.

SS: Thank you very much.


3 thoughts on “Praveen Rai on the challenges of polling in India and why he thinks seat predictions should be banned

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s