Photo credit: Staatliche Schlösser und Gärten Baden-Württemberg

Data Obtained by Opinion Research Institutes Often Unreliable

In a recent study, an international research team with participation of Mannheim data scientist Prof. Annelies Blom has come to the conclusion that many polls taken by private electoral groups, opinion research institutes and market research institutes yield inaccurate results. It is a problem that not only applies to newcomers, but also to some of the most renowned research institutes in Germany.

Public opinion research institutes are tasked with gathering reliable data to make predictions about election results and to reflect political moods and opinions. For decades, election forecasts and public opinion polls have been based on data from major German institutes such as Allensbach, Infratest, and Forsa. However, for several years now, the quality of these data has been suffering as many opinion research institutes have been employing methods that are cheaper on the whole, but lack scientific basis.

The problem underlying these methods applies to the whole industry, including the established market leaders: They use online panels for which participants register themselves, or place online banner advertisements on the websites of major media outlets. By doing so, they receive responses from people who voluntarily and actively decide to participate in such online surveys. However, this group of participants is unsuited to represent the entire population since it mostly consists of middle-aged, educated individuals who are internet-savvy and politically interested.

International study shows that many data are not representative for the population
An international research team with participation of Annelies Blom, professor for Political Science and Data Science at the University of Mannheim, has now dedicated a study to this issue. Their research results were recently published in a special edition of the Journal for Survey Statistics and Methodology (see link below). Three other data scientists from Mannheim also contributed to the article:  Dr. Carina Cornesse, Dr. Alexander Wenz, and Prof. Joseph Sakshaug.

In their article, the researchers summarize the results of 25 comparative studies which examined to what level these nonprobability sample surveys are actually representative. The result: “Although these surveys encompass several thousands of participants, the results are distorted because they do not adequately represent the population. That way the institutes cannot keep their promise that the inferences based on one sample will apply for the general public,” Prof. Blom says.

One reason for the poor data quality of many institutes is, according to Blom, the enormous financial pressure the industry finds itself under, having to deliver data to customers faster and faster. “The insufficiency of data is highly problematic. There are even some researchers who use such panels for their work,” the data scientist explains. Poor data quality might even lead to wrong economic or political decisions.

Link to the article:
Carina Cornesse, Annelies G Blom et al.A Review of Conceptual Approaches and Empirical Evidence on Probability and Nonprobability Sample Survey Research. Journal of Survey Statistics and Methodology. January 2020.