Testing the credibility of a survey

A great deal has been written about the credibility of research. If you genuinely want to verify how credible a survey is, some detective work and a lot of effort await you. A historical memory going back at least two election cycles will come in handy, as will familiarity with the seven questions about public opinion research and a knowledge of research methodologies. The guide below may help you a little.

How to recognise a credible source, in brief:

• The client commissioning the survey and its purpose are transparent.
• Information about how the survey was carried out is published – the research passport; the passport and the research report are publicly available and contain as much detail as possible.
• The author of the survey uses proven methods with a scientific basis and is willing to discuss the methodology.
• A representative of the agency is easy to reach and open to a debate about the methodology.
• The agency genuinely meets the conditions of the ICC/ESOMAR Code.
• The quality of data collection is checked, including independently.

A more detailed guide:

a) A transparent client and a transparent purpose

Check whether the survey is being carried out by a reputable, independent research organisation. You can look up the owners in the commercial register and find out whether the company really is based at its registered address. Another indicator is whether the company has employees and whether it has had a profile in the research field for some time. A credible agency has other clients as well, takes part in self-regulatory mechanisms and joins the industry dialogue through its membership of the international research association ESOMAR and of the local partner association SIMAR.

If you are still unsure about an agency’s profile, check the information the agency itself publishes about the project on its own website, or on the website of the client who commissioned it. That information should at the very least cover the scope of a public opinion research passport. Established organisations usually publish a more detailed research report and will transparently give you the information you need in order to assess the survey’s credibility. Another option is to test how willing the agency is to answer questions about how the survey was conducted, or to defend its results before the professional community – which is why they provide a contact for the team that carried the survey out.

b) Who funded this survey, and why?

It is good practice for an agency to declare the funds from which the cost of the survey was covered. If the survey was carried out for a respected media outlet, or by independent researchers drawing on public funding, there is a good chance that it is an impartial study. Where a survey has been carried out by a group of supporters, a pressure group or a political party, it may still be well conducted – but readers, viewers and listeners should be told who carried it out and who commissioned it.

Whether a survey is sound depends on whether it was carried out by a research organisation using scientific methods of sampling, whether the questionnaire is properly constructed, whether the questions are asked impartially, and whether information about the questions asked and about the results is fully available. Where that information is not available, such a survey should be approached with caution. In any case, watch out for leading questions and for selective findings designed to reinforce an opinion close to the client’s own rather than to report objectively on public opinion. SIMAR member agencies have a self-regulatory mechanism for this area too.

c) How many people were interviewed?

The more people, the better – although a small sample in a scientifically designed survey is always better than a large sample made up of people who chose to take part themselves. It is important to realise that sample size is by no means the only relevant figure. In an election model, for example, the data reported often exclude undecided voters, those who refused to reveal their voting preference, and respondents who stated that they would not vote at all. Excluding those groups does capture the views of the most relevant group – voters who have made up their minds – but the final sample may end up much smaller, and the sampling error correspondingly larger.

Caution is also needed when the sample is broken down into individual groups – women and men, say: the sampling error for such a subgroup is larger than for the sample as a whole. If the overall sample size is 500 and it consists of equal numbers of men and women, the confidence interval for each gender is roughly 6 percentage points.

d) How were the respondents selected?

Is it clearly stated who was included in the sample – who is part of it and who was left out? Did the research organisation use one of the selection methods described in the Seven questions and answers section? If the survey was carried out only in certain places – in cities but not in rural areas, for instance – then that information has to be stated in every report, and the findings have to be read in the context of that methodological limitation.

If it was a poll in which respondents took part on their own initiative – readers of a magazine or television viewers sending in a letter, an e‑mail or a text message – the information obtained in that way cannot be regarded as research representing the views of the population.

e) When was the survey conducted?

Various events can have a dramatic impact on the results of public opinion research. Results therefore have to be interpreted in relation to key events. Even the freshest results can go out of date quickly. Surveys that are several weeks or months old may be perfectly sound if they relate, for example, to underlying cultural attitudes or behaviour.

In pre-election research the exact date of fieldwork matters especially, because voting preferences can change right up to the moment the voter casts their ballot. That is why it is always important, when results are published, to state and to take into account the period in which the survey was conducted – not merely when it was published. Surveys genuinely predict nothing. Further interpretation of the data, the significance of changes and an estimate of future developments may of course be the subject of sociological or political-science estimation by individuals, or of public debate on the basis of the data presented.

f) How were the data collected?

There are four main methods: face-to-face, telephone and online interviewing, and questionnaires sent by post (the last of these is essentially no longer used in the Czech Republic). Each method has its strengths and weaknesses, and it is always necessary to make sure there is no barrier to data collection within the population concerned (access to the internet among older people, for example).

All of the methods depend on being able to reach people and on their willingness to co-operate; response rates can differ considerably, and the subject of a survey always motivates somewhat different respondents to take part. Hypotheses can also be advanced about the effect each interviewing method has on the quality of answers. In any case, established companies have developed statistical techniques that address many of these problems and process the raw data so as to arrive at representative results and, to a degree, to remove non-sampling error as well.

g) What questions were the respondents asked?

Try to obtain a copy of the complete questionnaire together with a description of how the interviewing was done – not just the questions that were published. A credible research organisation publishes the full wording of the questionnaire on its website, or provides it on request. Judge for yourself whether the questions are put in a balanced way and whether they are structured so as to lead the respondent towards a particular conclusion.

h) Are the results consistent with other research?

Where possible, compare both the differences and the similarities with other available research. Surveys on the same topic ought to arrive at similar conclusions. If they differ markedly, the explanation should be apparent from the construction of the questionnaire and from the way the sample was selected. If a research organisation publishes surveys regularly, it is worth looking at the trends a given survey shows. Are the trends confirmed over a longer time series? And where there is a marked deviation, is the team that carried out the survey willing to explain it?

i) Was it a push poll?

The purpose of a push poll is to spread rumours, or outright lies, about rivals or opponents. Such activities cannot be regarded as research, but rather as political manipulation attempting to hide behind the smokescreen of an opinion poll. In a push poll of this kind, large numbers of people are called and asked to take part in a supposed survey, for instance. The questions are in essence thinly veiled accusations against a rival, or a repetition of slurs about an opposing candidate’s professional or personal conduct. Established research organisations have nothing to do with push polls of this sort, even though it is a phenomenon that has been growing in some countries of late. The research industry also has corrective mechanisms within its self-regulatory rules.

j) Is data collection independently checked?

Honest data collection is demanding and expensive. Face-to-face and telephone interviewing require trained interviewers who conduct the interview correctly. Checks are made on whether the interview with the respondent took place as specified in the interviewing brief. Parts of interviews are recorded and supervisors then listen back to them, checking whether interviewers are influencing respondents in any way and whether they are recording the answers correctly. With online interviewing, the respondent’s identity has to be verified, control questions are inserted, completion time is measured and the data obtained are checked for logical consistency.

Besides the internal checks carried out by an agency’s own experienced fieldwork team, there are also random checks of completed projects by the SIMAR audit team. These checks are carried out regularly at SIMAR member agencies, and the SIMAR team also checks data collection at the request of a client, at both member and non-member agencies. More about the checks here.