Seven questions and answers about public opinion research
Everything essential about conducting, interpreting and publishing public opinion research is summed up here in seven questions and answers. They will give you a basic orientation in the field and equip you with the criteria by which you can assess the informative value and quality of individual surveys. The text draws on material prepared by the SIMAR working group on public opinion research and on documents of the international research organisations ESOMAR and WAPOR, in particular this publication.
1. What is public opinion research?
2. Why is public opinion research carried out, and what is it for?
3. How do I check whether a survey is credible?
4. Why can survey results differ?
5. How is public opinion research regulated?
6. What is the methodology: representativeness, collection methods and data quality?
7. How should public opinion research be interpreted?
1. What is public opinion research?
Public opinion research is the best and most widely used way of measuring the attitudes and views of the public. It is a scientific and representative study, designed and carried out to measure the opinions of a particular specified population – voters in a given country, for instance.
Research conceived along scientific lines meets certain conditions:
a) respondents are selected by the research organisation using clearly formulated criteria, so as to ensure representativeness within the population concerned; respondents therefore do not select themselves for participation (self-selection);
b) the questions are worded in a balanced way and do not lead respondents towards a particular kind of answer.
Carrying out a research study calls for long experience and expertise. For more on checking credibility, see question 3.
2. Why is public opinion research carried out, and what is it for?
The views of the public matter, and many organisations – government, political parties, businesses, non-profits and charities, and the news media – need them in order to make decisions.
Public opinion is an important element in the making and the conduct of policy. Surveys give government information about whether voters support its actions, how they see political figures, what people are struggling with in their lives and what they expect. In a democracy, knowing what the public thinks helps parties choose and deliver a programme for an election campaign.
Journalists take an interest in public opinion too, and reporting on it is part of the news. Think tanks, non-profits and charities may also conduct surveys on social problems in order to uncover or draw attention to a specific issue or to citizens’ needs. Publishing attitudes in this way helps public debate and the search for a solution to the social problem in question.
Objective information about public opinion is available to the public at large. Well-conducted and transparently published opinion research gives people access to the same kind of information as those who take the decisions – government, politicians and other organisations. Research thus democratises information and decision-making. You can read about what else research can be used for here.
3. How do I check whether a survey is credible?
The credibility of a survey can be checked. For a basic orientation it is enough to interpret the results in the context of the information contained in the public opinion research passport. To examine the quality of a survey in more detail you need to have the research report available, to find your way around the basics of research methodology, and to find answers to a number of key questions.
For more detail, see the following questions in the Survey credibility test:
- Who carried the survey out, and who paid for it?
- In what period were the data collected?
- By what method was the survey conducted?
- How were the respondents selected? How representative is the sample?
- What questions were the respondents asked?
- How does the agency demonstrate the quality of its data collection, and to what extent is its model transparently set out and justified before the public?
- Are the methods and the overall model corrected and adjusted, and is the agency team ready to explain the details of the methodology and the differences compared with voters’ actual behaviour at the ballot box, for example?
4. Why can survey results differ?
Surveys are not infallible, but they remain the best known way of measuring what the public thinks. In most countries the results of well-designed surveys fall within a tolerance of 3 percentage points, even if they occasionally stray beyond the margin of statistical error. Good-quality surveys are very accurate in the Czech Republic, as comparisons of pre-election surveys with the elections to the Chamber of Deputies show. In most cases research is a good guide to how opinion is developing, even if it is sometimes wrong by a wider margin. If a well-designed representative survey finds that the public is divided 70 to 30 per cent on a given issue, then even a ten-per-cent error cannot change the fact that one view prevails.
It is also true, however, that in an election where the gap between candidates is very narrow, a survey result giving one candidate an advantage of less than 5 per cent cannot be taken as a clear indicator of victory. It also has to be borne in mind that a survey always relates to the time at which it was conducted – in the days, weeks or months that follow, the situation continues to develop, just as the will of the population changes.
Even research conducted professionally and scientifically may, despite all the effort put in and all the rules observed, show differences compared with other similar surveys and with election results. It is always important to think of research findings as an indicative snapshot valid at the time the data were collected and constructed on certain assumptions – not as a prediction of future voting behaviour. Election campaigns, and with them voters’ decisions, go on changing after a survey has been carried out. It is a film that keeps running.
The differences may be due to the following factors:
a) The period in which the survey was conducted. Surveys may have been carried out in different periods even though they were published at the same time. Many people’s views are changeable and shift in response to various events. It may therefore be that both surveys reflected reality in their respective periods and that the public mood changed in the time between the two. With a survey carried out a long way ahead of an election it is normal for voting preferences to change.
b) Sampling. Surveys may have defined the target population on which the study was carried out in different ways (different age groups, regions, ethnic groups and so on). Representativeness, and the way it is achieved, is crucial to the quality of a survey. See the basics of research methodology.
c) The data collection method. A survey may have been carried out using a different method of data collection. Results can be affected by systematic error arising from the methodology. Each method of data collection has its own procedures for achieving a representative sample; again, see the basics of research methodology for details.
On some sensitive questions, respondents are more open in self-completion surveys. Across different methods, people consciously or unconsciously answer differently when questioned by an interviewer face to face than they do when answering telephone questions on their own, or a self-completion questionnaire sent by post or electronically. Researchers therefore have to deal with the shy-voter syndrome appropriately in each method.
d) Questionnaire design. The wording of questions matters a great deal, especially on topics on which people do not hold firm views. Where survey results differ, it is always important to compare the exact wording of the question. The effect of question order may also be apparent. One survey may put a given question at the start of the interview, another towards the end, after a series of further questions on the topic. A different order of questions can produce two sets of differing results, particularly where people have no firm opinion. Sometimes they answer differently depending on whether the question was put without context, or whether they were first asked to think about aspects of the issue. Besides differences in questionnaire design, question order and the interviewing environment, a further source of difference can be how researchers handle the shy-voter phenomenon, where it is advisable to use data collection methods that offer respondents greater anonymity, or to adapt the wording of the questions.
e) Stated and actual turnout. Some people make up their minds shortly before an election, and a survey carried out earlier cannot capture their intentions. Dramatic events in the closing stretch of a campaign can affect both turnout and preferences. Even the weather has an influence on turnout. Expected turnout then changes, or groups of voters come to the polls in different proportions from those they stated earlier in surveys. An election model takes these assumptions into account in the way it is constructed. Each author of an election model may work slightly differently, include a different range of factors or weight them differently. The transparency of such a model is therefore another important consideration in assessing the quality and credibility of a research finding.
f) It is not research at all, or the research was done badly. Just as a stopped clock shows the right time twice a day, a badly conducted survey will occasionally produce the right figures. It can therefore sometimes be difficult for the public to tell the difference between a good-quality survey and, say, a self-selected poll or an attempt by some group to manipulate public opinion. Behind a sound survey there is usually an identifiable research team and an agency with the resources to match, able to defend its results before both the professional community and the wider public (see the Credibility test for details).
5. How is public opinion research regulated?
Conducting and publishing public opinion research is governed by the profession’s own rules. Credible public opinion research follows the ESOMAR and WAPOR guidelines for conducting research, available here. A shorter summary of the requirements and obligations for ESOMAR members is offered by the document Key requirements for opinion polls and published surveys.
Czech professionals also build on these documents and other materials. In this area they take part in the work of SIMAR’s joint self-regulatory platform, see here. The group regularly addresses the assessment of differences and the transparency rules for publicly published research. Those who take part in the industry dialogue have to stand behind their research, explain differences or transparently provide more detailed information about their procedure.
6. What is the methodology: representativeness, collection methods and data quality?
The crucial question for a survey is how the respondents were selected. Under certain conditions, selection matters more to the reliability of the findings than sample size does. Sampling and non-sampling error are explained on the CVVM website. Data collection is usually carried out using random or quota sampling. The method of selection and how it is adapted to the data collection methodology is explained in more detail in the Basics of research methodology section.
Quantitative studies measuring the attitudes and views of the population are usually carried out using one of the standard, long-proven methods of data collection: face-to-face interviewing, telephone interviewing or online interviewing. Each of these methods has its strengths and certain drawbacks, which a research team’s long experience in the field enables it to handle. Again, see the Basics of research methodology for a fuller explanation.
7. How should public opinion research be interpreted?
Interpreting research findings is a shared responsibility of the researchers who provide them and the media or other parties who publish them. When research is published, the public must always be given the information contained in the public opinion research passport, together with a reference to more detailed information about the methodology.
To interpret a survey properly it is advisable first to read the whole research report, to become familiar with the general basics of the methodology and, if need be, to check your interpretation with the authors of the survey before publication. It is always important to bear in mind that research is not a prediction but a snapshot of the population’s attitudes at a given time. It is always better to look at the longer-term trend in a given survey (that is, at a complete data series) and then, where possible, to compare the findings of similar surveys. In commenting on the differences between individual political groupings it is also worth remembering that the output of a survey is not precise figures but rather ranges with a certain statistical significance – which is why shifts, or an ordering of parties, that fall within the margin of statistical error are not the stuff of a headline.
More detailed rules of interpretation can be found here.
