The pollsters' margin of error

They told us it was as good as done. They told us the elections had already been won. They said there would only be slight shifts within the political blocs. They said that the right-wing bloc would win big. They said that there was no doubt about the projected results. For three months, the media, the politicians and our best commentators engaged with the public on the assumption that this election would be the most boring election in years because everything, or "almost everything", was already known in advance. A few days before the election, Professor Ytzhak Katz who heads the Maagar Mochot research institute, went as far as to say, "This election may not even be necessary."

Katz, it turns out, was wrong, but not entirely. He was off on everything having to do with Likud, which he predicted would win 37 seats, and even more off on Lapid, who he forecast to win only 8 seats. But Katz was also the only one who accurately predicted the result for Labor — 15 seats — and was nearly spot on in regard to Hatnuah, predicting 5 seats for Tzipi Livni's party.

Avi Degani of the Geocartography Knowledge Group also issued a number of polls over the last few months that were utterly refuted by reality this last Tuesday. Among other predictions, at one point Degani projected that Strong Israel, headed by Aryeh Eldad, would win 6 Knesset seats, that Green Leaf would win 3 and that Koach Lehashpia, headed by Rabbi Amnon Yitzhak, would win 4. None of the three even met the minimum threshold of votes. Most of the polls also gave too much credit to Naftali Bennett and his Habayit Hayehudi, predicting that he would win 14 seats or more.

But no one foresaw the biggest surprise of the 2013 election: Yair Lapid. Regardless, veteran and experienced pollster Mina Tzemach was closer than anyone else. Like Professor Camil Fuchs, Avi Degani and Mano Geva, Tzemach predicted a Likud result that was very close to the ultimate reality: a mere 32 seats (Likud ended up winning 31 seats). She predicted that Lapid would win 13 seats, the highest number of seats Lapid was forecast to win by any poll (he ended up winning 19).

On the Friday before the election, Tzemach told Channel 2 viewers that Lapid's popularity was steadily growing, but she was not permitted to predict a number because of the law prohibiting polls in the days leading up to the election. In her final poll before the election, Tzemach also accurately predicted the number of seats that the ultra-Orthodox party Shas would win — 11 seats — as well as Habayit Hayehudi, which she predicted would win 12 seats (and after the final votes were tallied it emerged that Habayit Hayehudi did in fact win 12 seats). She accurately predicted that Kadima, which hovered on the cusp of the minimum threshold, would indeed meet the threshold and enter the Knesset with 2 seats, but she also predicted that Strong Israel would meet the minimum threshold, and she was wrong. She was right about the growth of United Torah Judaism and Meretz, but she gave Livni's Hatnuah and Shelly Yachimovich's Labor each 2 seats more than the voters did.

The respondent is not available

Tzemach's relative success in predicting the election results with her final poll doesn't begin to mask the general sense created by most of the pollsters and polls for many months that the results had been decided long before the election ever took place, in terms of political blocs and in terms of the individual parties.

University of Haifa's Professor Gabriel Weimann, who has been following polls and election results since the 1993 election, is very critical of today's pollsters as well as the consumers of the polls: The media, the public and the politicians. Weimann warned of the polls' questionable credibility on the pages of this very newspaper back in November. Now he is reiterating: "The polls failed again. The pollsters tripped up the politicians. The politicians were once again sucked in by the toxic polls — they believed the polls and followed them almost blindly. The public discourse also believed and relied on the polls, and the misguided commentary was entirely based on misleading polls."

Professor Weimann rejects arguments, like the one made by pollster Rafi Smith on the day after the election, that the big change occurred right before the election, after polling was no longer permitted. "A professional pollster should have identified this change, if only as a trend, long before," Weimann believes. "The pollsters are just trying to cover up their mistake. They were wrong, big time."

"Before the election," Weimann argues, "the pollsters broadcast the results of their polls to whomever would listen, but what we should have heard from them, even before the election, is that there are polls that are unreliable, that there are polls that are commissioned by the various parties, that there are leaks on behalf of the various campaigns. But what they really should have told us, in tandem with the release of every poll, was their big secret, which they have consistently tried to conceal: Only 40 percent of the people they approach are even willing to take part in this game they call a poll to begin with. That means that 60% of the people refuse to participate, and a large number of those who do participate are undecided."

"Some two weeks ago," Weimann recalls, "Channel 10 did an investigative report on the issue of the polls, and their findings were incredible: Only one of every 15 people actually answers the pollsters' questions. The Channel 10 reporters put a hidden camera inside a room where telephone interviews were being conducted, and they found out a lot more. Among other things, they found that some of the interviewers, when respondents fail to answer a question and hang up, simply invent the answers themselves."

"There are laws, but they are very seldom followed," Weimann adds. "According to the law, the pollster and the outlet that publicizes the poll must notify the public of who commissioned the poll, how many people refused to answer questions and how the original poll questions were worded, but everyone ignores the law, even the media. They behave like criminals and we all get misled."

Weimann mentions four well-known pollsters — Tzemach, Fuchs, Katz and Geva — as pollsters whose "integrity and desperate efforts to accurately predict the election results" are not in question. But he adds that there are pollsters, whom he declines to mention by name, "who wear two hats — one on behalf of the media and one on behalf of the parties."

"There are others who are motivated by money," he continues. "All they care about is that the media covers them and their work and mentions the name of their institutes to boost their image."

"There are also pollsters who are simply unprofessional," he adds.

He remarks with grave disappointment that "the research institutes that prepare political polls refuse to adopt a shared ethical code, as is customary abroad."

There are professionals and there are hacks

Tzemach rejects most of the Weimann's arguments. "Polls are the best tool we have to determine voting intentions," she says. "With every poll we stress that it is true only for the time in which it was conducted. We have no control over the political commentators, the media and the politicians who take the data further."

Tzemach maintains that the research institutes that conduct the polls actually do have a shared ethical code. "Factually speaking, Weimann is wrong on this point. The Israeli Union of Research Institutes decided to adopt an ethical code for market analysis instituted by the European Society for Opinion and Marketing Research. The ethical code has been translated into Hebrew and it has been signed by Geocartography, New Wave Research, our own Dahaf institute, Rafi Smith and many others."

So is Weimann wrong? Weimann claims that the code Tzemach is referring to is a "general, voluntary, non-binding code that doesn't apply specifically to political polls but rather to general market research." Our investigation revealed that no less than 17 research institutes, all members of the Israeli Union of Research Institutes, have in fact adopted this code. But it does not specifically apply to political polls. It applies to social and market analyses, as Weimann maintains. It also turns out that not all the research institutes in Israel, just the ones belonging to the union, have adopted this ethical code and view it as a binding guideline.

Tzemach only partially accepts Weimann's criticism. "It is true that there is a high percentage of people who refuse to participate. It may be wise to include the number of refusals in every poll," she admits. "However, you have to take into account that making these numbers public could encourage people to refuse to answer polls and we will all lose out. It is important to keep in mind that for every person that refuses to respond, and there are many as aforementioned, I find an alternate from the same group. Therefore, I don't think that the polls' reliability is compromised."

Tzemach tends to agree with Weimann's criticism toward the level of professionalism at some of the institutes: "Just like in any other field — there are professionals and there are hacks. There are pollsters with integrity and there are pollsters with less integrity."

And still, the leaders of the political parties in Israel conduct themselves in accordance with the polls. The prime minister decided to merge his Likud party with Yisrael Beytenu on the basis of a poll. But the polls, hindsight teaches us, led him astray.

"The prime minister shouldn't make decisions based on polls. A poll is a good indication of where the public stands, its leanings, but as a predictor it is good only on the day it is conducted," says Tzemach.

Katz, whose last poll predicted 37 seats for Likud and only 8 seats for Lapid, reiterates that the polls are only a reflection of the reality on the day the poll is conducted. Katz, among the heads of Maagar Mochot, says that in the final days, and even hours, before the election, major and anomalous changes occurred, which managed to surprise the pollsters. The number of undecided voters, those people who didn't know who they would vote for until the very last minute, was very high this time. Likud's free fall and Lapid's meteoric rise occurred in the final days and hours.

But Katz, like Tzemach, admits that only a small number of those asked to respond to the polls actually agree to take part. "One out of every five," he reveals. "We make sure to include this data in our polls. Some of the media outlets provide this information, but others ignore it.

Katz, too, would be pleased if the ethical code were more up-to-date and if it obligated more of the pollsters, but he doesn't want any binding legislation on the matter. "Laws already exist, we just need to implement them."

"Further governmental or legal intervention would not be appropriate," he insists. He remarks that every pollster includes a margin of error in the results, and this margin can be as high as five Knesset seats. "But the public judges you differently — only by the level of accuracy on the number of seats. If you are accurate, you succeeded. If you miss, you failed. The public doesn't care that you are still within the margin of error."

The pensioners effect

Professor Avi Diskin, a political science expert from the Interdisciplinary Center in Herzliya and Hebrew University in Jerusalem, was one of the founders of The Third Way party that ran in the 1996 and 1999 elections. In 1984, he was the internal pollster for Labor.

During this week's election, Diskin was in charge of compiling voter turnout data on behalf of the Central Elections Committee. Diskin didn't conduct any polls on behalf of any party or newspaper this time around, but he did write in an article that ran three months ago that "the right-wing majority is consistent, but not a certainty." As early as three months ago, he already assessed that Prime Minister Benjamin Netanyahu's merger with Yisrael Beytenu had hurt his image among Middle Eastern and Russian Jews, the nationalist Right and even among centrists.

This week, once the results began to emerge, Diskin speculated that three main events dictated the results of the election: "The merger with [Avigdor] Lieberman that hurt Netanyahu on all fronts, Kadima's crash and burn which made way for the rise of new centrist parties, and Yachimovich's misguided electoral maneuver (giving up on the Center, succumbing to pressure and focusing on the Left, and declaring that she won't enter a Netanyahu-led coalition, thus alienating voters)."

"It is easy to say now that the polls failed," says Diskin. "But the public needs to know that pollsters work extremely hard when they prepare a poll. The poll gives the pollster an accurate picture of the moment in which it is conducted, and sometimes, it can hint or point to potential from this or that direction. In such a case, the pollster can share with the public the results that are true for that moment and describe the potential for change, but most of the pollsters don't do that."

"Yair Lapid's success," Diskin believes, "could not have been foreseen. There are things that are not given to early prediction. We had an enormous number of undecided voters in the political Center, up until the absolute last minute. Most of the voters today are in the Center, even if they don't always vote for the Center. Most of the public supports the establishment of a Palestinian state, but most people also don't believe that peace can be achieved. These undecided voters from the Center can't stand Netanyahu or the ultra-Orthodox parties or the far Right, but on the other hand, Livni, Shelly and [Meretz Chairwoman Zahava] Gal-On really get on their nerves. So they go to the polls and make a 'no other choice' vote."

Diskin recalls a similar phenomenon in 2006, when the Pensioners' Party won a surprising number of seats. "[The Pensioners' Party] was less attractive than Yair Lapid, and they didn't work as hard as Lapid did, and still, in the final hours of the election, a huge constituency felt that it was the right choice and the party won seven Knesset seats," says Diskin. "On Tuesday, the same phenomenon occurred, but on a much bigger scale."

Diskin, who was interviewed before the soldiers' ballots had been counted, said that if the final votes were to give Netanyahu an additional seat, and the right-wing bloc were to gain a majority of 61 seats over the Left's 59, Netanyahu will be the clear winner because even a majority of one seat is a majority. In such a case, Netanyahu's maneuverability will be lower than in the past, but higher than people tend to think. And in any case, victory will be his. "If, however, the equality between the blocs is preserved even after the soldiers' votes are counted, Lapid can demand whatever he wants. It all comes down to one seat."

טעינו? נתקן! אם מצאתם טעות בכתבה, נשמח שתשתפו אותנו