Determining whether SDB (Social desirability bias) exists in the questionnaire survey

It can be difficult to determine whether or not the responses to a question have been influenced by SDB.

1. Matched cells

One approach to determining whether or not there is a problem is to use one of the techniques described above and to have part of the sample as a control cell that is asked the same question but in a direct form.

The control cell must be matched on all relevant criteria to the rest of the sample and must be sufficiently large to enable reasonably sized differences to be statistically significant. If the responses from the control cell differ significantly from the rest of the sample, then this may confirm that SDB exists and that the questionnaire writer was correct to take the appropriate precautions.

This approach is likely to mean sacrificing a significant part of the sample on the appropriate questions, and the uncertainty resulting if no difference in responses is found. It is unlikely in most commercial studies that this technique can be justified. It is a better use of resources to assume that SDB does exist and to use an appropriate question technique that will minimize it.

2. Matching known facts

Where it is possible to cross-check responses against known data from other sources, then this can highlight differences that may be due to SDB. The cross-checkable facts will tend to be factual or behavioural data, such as volume of product sold. Attitudinal questions cannot be checked in this way. Even with factual data it is frequently difficult to match external data sources with survey data because of differences in definitions, time periods and so on. Survey data can sometimes provide their own internal cross-checking. Pantry checks, to see what is actually in a respondent’s store cupboard, can be used as a check against what the respondent has previously claimed to be there.

It has been suggested that, to check the level of SDB in attitudinal data, friends of the respondents might be interviewed and asked to evaluate their perceptions of the respondents’ attitudes. This seems fraught with difficulties regarding both the accuracy of the friends’ evaluations and their motivations. The scale and complexity of such a study is, anyway, likely to make it impracticable for commercial market research projects (Sudman and Bradburn, 1982).

3. Checking against measures with known SDB

For attitudinal questions it is possible to design a battery of scales that measure a sample’s tendency to SDB. Such a battery would include: behaviours that are common (majority of the population) and socially undesirable; and behaviours that are not common (minority of the popu­lation) but are socially desirable.

Consistently low scores on the first group (indicating low levels of undesirable behaviour) and a high score on the second (indicating high levels of desirable behaviour) would suggest that the respondent either falls into a small and angelic minority of the population or that SDB exists in the responses. Individual respondents with these response patterns can be identified, and if on another topic the sample has a higher-than- expected level of claimed desirable behaviour or a lower level of claimed undesirable behaviour, then the researcher knows that there is an SDB problem with the sample as a whole.

There are several published batteries of scales to help the questionnaire writer, including the Edwards (1957), Crowne and Marlowe (1960) and Paulhus (Paulhus and Reid, 1991) batteries of scales. In addition, short­ened versions of the Crowne-Marlowe scale have been tested by Strahan and Gerbasi (1972) and by Greenwald and Satow (1970) that may be more suited to market research interviews.

4. Rating the question for social desirability

Questions can be included that directly ask the respondents to assess the attitude or behaviour for social desirability (Phillips and Clancy, 1972). This can indicate the relative problem between different scales or ques­tions. However, there must be doubt about whether such questions do not suffer from SDB themselves.

5. Noting physiological manifestations of unease

It is likely that there will be physiological signs that a respondent is trying to mislead an interviewer, such as facial muscle movement, galvanic skin response and pupil dilation. However, interpreting these even in labora­tory conditions is problematic and outside laboratory conditions is likely to be impossible and outside the skill set of most market research inter­viewers.

It will be seen that there are few ways of eliminating SDB with certainty. However, if researchers recognize the possibility or even probability of its existence, this may help them to design questionnaires that minimize its occurrence and to avoid misinterpretation of the data.

Source: Brace Ian (2018), Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research, Kogan Page; 4th edition.