The basics of research methodology
Representativeness and sampling
1. How is a genuinely representative sample selected?
2. Do research agencies do anything else to make a sample representative?
4. How can you know what a million people think when you ask only 1,000 or 2,000 respondents?
7. So is the statistical error in fact much larger?
Data collection methodology
8. How are research data collected?
9. How is the quality of the data collected checked?
10. Can I trust online research?
11. Can the way a question is asked influence the answer?
12. What are party preferences, and how is an election model constructed?
Representativeness and sampling
1. How is a genuinely representative sample selected?
There are two main methods of sampling. The first is random sampling, the second is quota sampling.
Random sampling ensures that everyone in a given population has some chance of being selected for the survey. In telephone research, random sampling can be done from a list of randomly generated telephone numbers (see how random sampling works in CATI), or in face-to-face interviewing, where addresses or locations are selected at random and the member of the household to be interviewed is determined. Respondents selected in this way are then contacted by interviewers and asked to take part in the survey. For various reasons, though, genuinely random selection is harder to achieve in practice. Ideal conditions are made more difficult for researchers by the declining willingness of respondents to answer, for instance, or by restricted access to a good sampling frame (a suitable set of addresses, say).
To improve the quality of random sampling, researchers check whether the population is represented in such a way that the resulting sample corresponds as closely as possible to the demographic composition of the population. A comparison is made of whether the sample collected contains the basic demographic categories in the proportions in which they occur in the population – as reported by the complete Czech Statistical Office census, for example. Researchers may also monitor other characteristics of the population that are known to play a role in the subject of a given study, and adapt the data collection accordingly.
Quota sampling assumes that certain quota characteristics are set – age, gender and region, for instance – and the interviewer is then tasked with finding respondents whose profile fits those characteristics. As explained above, quota sampling can also be applied to a sample originally created by random selection. For example, a person meeting the required quota criteria is selected from a randomly chosen household. This speeds up data collection.
Online methods usually use quota sampling to build a sample from a database of respondents who have already provided demographic information about themselves; the question of representativeness is therefore more complicated here and is dealt with in more detail in question 10.
An example of achieving representativeness through quota sampling: if, according to Czech Statistical Office data, 52% of the population are women and 30% of those are aged 55+, a scientific survey will ensure that these groups are represented in the study in the corresponding proportions.
2. Do research agencies do anything else to make a sample representative?
Yes, they do. While well-executed random or quota sampling will in general terms secure the right representation of the population, for several reasons it may still contain too many members of one group and too few of another. In the course of a study an agency usually finds out a respondent’s views, but also asks for a few pieces of information about the respondent themselves. That information is then used to compare the selected sample with (for example) the results of the most recent census.
The raw survey data can then be adjusted slightly so as to match the distribution of groups in the population being studied. If, for instance, data have been collected from 100 members of a given demographic group where the population would imply 110 (in a total sample of 1,000 or 2,000 respondents), the answers will be weighted so that those 100 respondents count as 1.1 persons each. The percentages published in this way then reflect the population as a whole.
3. But if a million people vote in a telephone or online poll, surely the results are more accurate than a survey with a sample of 1,000 respondents?
That is not the case. A biased sample is a biased sample, however large it may be. One famous example illustrating this well was the 1936 presidential election in the United States. A magazine, a literary digest, sent out 10 million postcards asking about voting preferences and received almost 2.3 million replies, which gave victory to Alfred Landon over Franklin Roosevelt by 57% to 43%. The magazine collected no information about the composition of its sample that would have allowed its quality to be assessed, nor did it weight the groups that were over- or under-represented. Since the magazine sent its postcards mainly to people who owned a telephone and a car, its sample contained very few working-class people. The young researcher George Gallup used a much smaller sample (although at 50,000 respondents it was still far larger than those used today). Because he made sure that the sample was representative of the underlying population, however, he correctly showed that Roosevelt would win hands down.
Self-selected polls – among the readers of a particular magazine, say, or the views of the viewers of a television programme – will always contain what are known as irremovable sampling errors. Respondents decide for themselves whether to take part; they will therefore tend to be people with strong views on the issue being studied. A further error lies in the fact that the demographic profile of the respondents obtained in this way does not match the overall demographic composition of the population in which the study needs to be carried out. The error caused by self-selection and by a non-representative population cannot be removed by further work with the data, which is why a self-selected poll is not a reliable basis for decision-making.
4. How can you know what a million people think when you ask only 1,000 or 2,000 respondents?
Essentially in the same way a cook judges a big pot of soup by tasting a single spoonful. Provided the soup is well stirred and the spoonful is „representative“, one spoonful is amply enough. Election research works on the same principle: obtaining a representative sample of the population is in essence rather like stirring the soup.
An unscientific poll is like a pot of soup that has not been stirred. The cook may eat a great deal from the top and still get a misleading picture if some of the ingredients have sunk to the bottom. As with the trick of the well-stirred soup, the essence of a scientific survey is securing a sample that is representative of the target population: what is needed is not a very large sample, but one that captures the whole population correctly.
5. And is there not a risk of sampling error with a survey of 1,000 or 2,000 respondents?
Yes, the risk exists. But statistical theory can estimate how big it is. Imagine a country with some issue on which 50% of people hold one view while the other 50% think the opposite. Statistical theory tells us that in a random survey of 1,000 people with a 100% response rate, the error will be within 3 percentage points in 19 cases out of 20. In other words: the result recorded may be as low as 47% and as high as 53% for each view. But there is a one-in-20 chance that the result will fall outside that range. With a sample of 2,000 respondents the statistical error will be within 2 percentage points, again in 19 cases out of 20.
6. You say these calculations apply to „a random survey with a 100% response rate“. Is that not an unattainable goal?
There is something in that. Many surveys are not random, and response rates are often much lower – under 50%. Research professionals do, however, compare their results with reality and work to ensure that the data obtained are a good guide for decision-making even where willingness to respond is lower. Samples obtained at a lower response rate can therefore still provide reliable information.
7. So is the statistical error in fact much larger?
Perhaps it is, and perhaps it is not. Let us look at two examples.
Let us go back to our example of a country split down the middle. Suppose everyone holding view B lives in the south. If the study is conducted half in the north and half in the south, the result will be exactly right. Adjusting the sampling proportions to the distribution of the population is called stratification. If stratification is done correctly, the measurement will be reasonably accurate.
Let us make another assumption about our mythical, evenly divided country. Suppose that people who hold view A are, more likely than group B, also members of a religious or ethnic minority that is reluctant to answer questions. If the research agency is unaware of this anomaly, it may publish results suggesting that view B is far more popular than view A.
To establish the true deviation we would have to take into account the error arising from random sampling, the effect of stratification and also systematic error. The problem is that it is difficult – and probably impossible – to know the real influence of stratification and of systematic error. Of course, if the influence of systematic error could be quantified, the research agency would adjust the results accordingly.
Data collection methodology
8. How are research data collected?
Most pre-election research in developed markets is done by telephone and online. In the Czech Republic some of this research is also carried out by face-to-face interviewing, or by a combination of several methods of data collection. Each method has its advantages and disadvantages; an experienced research team is aware of the strengths and limits of each one and proceeds so as to draw a genuinely representative sample from the population – that is, from the universe of all voters.
With face-to-face and telephone research, random sampling from the whole population is usually used. For face-to-face interviewing there is a protocol for interviewers that secures appropriate geographical coverage of households and respondents across the country. In telephone interviewing, a sample of randomly generated telephone numbers is interviewed. In the Czech Republic, 98.8% of the population aged 16+ uses a mobile telephone (Czech Statistical Office, 2020).
On the question of online interviewing, see question 10.
9. How is the quality of the data collected checked?
The agency team always checks that data collection was carried out as specified, with real respondents and without undue influence on the results by the interviewer. Questionnaires that show errors are discarded. Data collection at SIMAR member agencies is checked using proven procedures and independently audited against the Quality Standards; more here.
10. Can I trust online research?
The answer depends on how the sample in the target population was selected. That holds for any research method – for face-to-face or telephone interviews just as much. If a given method can produce a representative sample of the target population, then the answer is yes, such research can be trusted. There are areas and groups of the population that are very hard for interviewers to reach in person, just as there are people who have no access to the internet. If the bias arising from the method of data collection cannot be corrected by stratification or weighting, then that method does not provide a representative sample and cannot be regarded as credible.
In some cases internet penetration may be high enough that the internet is in fact the best option for collecting data. Given the low internet penetration in some segments of the population, and hence the low representativeness, online research on the general population in particular should be approached with caution, because people without internet access are excluded from such a survey from the outset. Internet penetration among people aged 16+ in the Czech Republic is 81.3%, but among people aged 65+ it is 40.3% (Czech Statistical Office, 2020). If it turned out that the views of the 65+ population were important for a given issue and that people who do not use the internet hold different views, data collected in this way would show differences.
It should also be borne in mind that online surveys use online panels made up of people who have given prior consent to being contacted for research purposes. With online surveys, then, suitable steps have to be taken to produce a sample that genuinely represents the target population. A good-quality online panel must also verify respondents’ identity thoroughly and apply measures ensuring an appropriate quality of answers – see the Quality Standard for online interviewing.
11. Can the way a question is asked influence the answer?
Without any doubt. We know a great deal, however, about how questions should be worded, and considerable emphasis is placed on this. There is an extensive body of knowledge about how people process information, on the basis of which questions can be worded properly. It is important to look specifically at how a question was put and, where possible, to check the questions that preceded it in the questionnaire. Common faults include questions containing terms that lead the respondent in a particular direction – „There seem to be fewer police officers on the streets, and a lot of people here are worried about rising crime; do you think the police here are understaffed?“ – or questions containing more than one idea while the answer options allow a reply to only one of them: „How well is the city council contributing to solving traffic congestion and shortcomings in public transport?“, in a study about public transport. Questions like these will not yield clear and useful answers about what people really think of the police or of public transport. Designing a questionnaire takes experience and sound judgement.
The context in which questions are asked can of course influence the way people respond. If we ask a question about crime after a battery of questions about whether people have ever felt threatened on public transport, or whether they have relatives or friends who have been mugged, and so on, it is likely that more people will express concern than if we had asked the crime question without that context. In interpreting answers to questions put in this way, it is important to be aware of the bias and the ambiguity: the results are not an accurate reflection of what people really think. This type of questioning is especially popular with pressure groups, which often use it to gain media coverage for their point of view. A responsible journalist should not publish such results, or should point out the bias caused by badly framed questions if they do decide to publish them.
12. What are party preferences, and how is an election model constructed?
Every sociological study examining how voters will decide in the forthcoming elections to the Chamber of Deputies of the Czech Parliament has to deal with two important questions:
1. What to do, in the survey results, with the votes of those who have not yet firmly decided which party they will vote for.
2. What to do with the votes of those who state that they will not exercise their right to vote, or who are not sure whether they will actually go to the polls.
How the current mood of voters will change further is not easy to estimate. Actual turnout depends even on the weather. It also has to be borne in mind that the results of interviewing are valid at the moment of the study, but that the very next day they may be affected by further developments in the individual parties’ campaigns. Research is a snapshot at a given time, but the film keeps running. Recent elections have also shown a trend of voters deciding in the last week before the election, or even in the polling station itself. The research community is therefore discussing various forms of output that include voters who have definitely made up their minds and are planning to go and vote, and is thinking about the design of questions and of the questionnaire as a whole so as to capture the natural stages of voter decision-making better.
The output of a public opinion survey may therefore take several forms that differ fundamentally from one another. Each of these outputs has its advantages and disadvantages. Party preferences can be regarded as the more inclusive model, taking in a broader spectrum of voters’ evolving preferences, while an election model seeks to come as close as possible to the actual election results.
