Questionnaire experiments: Response method and scale effects

Finding ways to reduce social desirability bias has long been one of the biggest challenges for researchers when designing questionnaires. The theory goes that while most people want to be ‘good respondents’ and provide the information that is asked for, they also want to appear to be ‘good people’ and so will try to represent themselves to the interviewer in a way that reflects well on themselves. Therefore, when asking about socially desirable (or undesirable) attitudes or behaviour it is important to design questions in a way that reduces the likely level of social desirability bias.

‘Green’ behaviour is one subject that might be affected in this way. With an emerging focus on green lifestyle choices, many people want to be seen to be doing their bit for the environment. This has led to a concern among researchers that people will overestimate how green they are in order to represent themselves as good people who act in the wider interests of society.

We presented respondents with the following four attitude statements in relation to the environment and asked whether they agreed or disagreed with them:

1. Acting in an environmentally friendly way is a high priority in the way I live my life
2. People have a responsibility to use more environmentally friendly forms of transport wherever possible
3. People should be prepared to make personal sacrifices to help the environment
4. People have a responsibility to recycle all of the household waste that they can

Our previous experiment (Aspects 23, November 2007) tested the impact of different question wording. This time we decided to look at the impact of different response options. Firstly, we tested a self-completion mode against an interviewer-administered mode; secondly, we tested a reverse order scale against a standard order scale. We split our sample into three groups:

• Group One – the control. Presented with a standard 5 point scale, going from ‘agree strongly’ to ‘disagree strongly’.
• Group Two – presented with the same 5 point scale but was asked the questions in a self-completion module (the other two * groups were asked the questions by the interviewer).
• Group Three – presented with a reverse 5 point scale (going from ‘disagree strongly’ to ‘agree strongly’).

The impact of self-completion on response

It is generally believed that when we ask respondents about attitudes or behaviours that are very sensitive, we should ask them in a self-completion module. Instead of the interviewer asking the question and subsequently coding the response, the respondent reads the question and enters a response into the computer themselves. This is designed to make the respondent feel more comfortable and encourage an honest response. With this in mind we wanted to test whether this would have an impact on this set of ‘green’ questions. If some respondents were trying to show the interviewer they were ‘good people’ by giving a socially desirable response, would we get a more honest response if we took the interviewer out of the equation?

Figure 1, below, compares the proportions agreeing with each statement between the interviewer administered and self-completion approaches. There were no significant differences to be seen, suggesting that shifting these questions into a self-completion module had no impact on response distribution.

Of course this does not mean that using self-completion methods never works. Other tests of this type have found that reports of socially undesirable behaviour have increased when a self-completion mode is used.

Rather, it might tell us that there is no bias problem with the standard versions of these questions. It is worth considering why we think that questions about the environment are likely to be affected by social desirability bias. If, for example, three quarters of people agree that ‘acting in an environmentally friendly way is a high priority in the way I live my life’ this may accurately reflect their opinion even if we would not regard their behaviour as sufficiently green to warrant it! We are dealing with highly subjective concepts: a ‘high priority’ for one person might be leaving the recycling out every week, for another it might be giving up flying. Perhaps people are answering honestly (or at least responding honestly to their interpretation of the question) and the problem is actually the way we interpret the question and the responses to it?

However, we should also acknowledge that simply passing the laptop to the respondent does not completely cut the interviewer out of the equation in the eyes of the respondent and so there may be some degree of bias in both versions. So while it is not possible to conclusively state that there is no social desirability bias in the responses to our questions, our test does suggest that there is not bias to the extent that a self-completion approach makes a difference.

Reversing the scale

Something else that is often discussed when designing questionnaires is the extent to which less engaged respondents tend to pay more attention to answers at the top of the list in face-to-face interviews. We therefore reversed the scale (so it went from ‘disagree strongly’ to ‘agree strongly’) to see what impact this would have on response. Our hypothesis was that respondents are more likely to take an answer from the top of a list than from the bottom and therefore changing the order of the response list would have an effect on the distribution of responses.

Figure 2, below, shows a comparison between the proportions agreeing with each statement when presented with both a forward and reverse 5 point scale.

For all of the statements a reverse 5 point scale significantly reduced the proportion agreeing (and in particular the proportion agreeing strongly) and therefore supported our hypothesis.

However, just because a reverse scale might reduce the level of agreement (and therefore perhaps reduce social desirability bias in some contexts) it may not always be the best way to ask about such attitudes. Most questionnaire scales go from ‘agree’ at the top to ‘disagree’ at the bottom. If we simply reverse the scale for a small set of statements this may confuse respondents (and interviewers too!) and lead to coding a response that is the opposite of their true opinion. Consequently, we have to make sure the response context is clear and that interviewers are clearly briefed about any variation in scales.

In the future it might be interesting to present another group of respondents with a reverse scale, but in a self-completion module. If this test produces the same results as in the self completion test with the forward scale (see figure 1) then we can conclude that the reason a reverse scale produces a different set of responses is due to interviewer error and nothing to do with disengaged respondents. However, at this stage, we cannot conclude this.

What we can firmly conclude is that a reverse scale produces a different distribution of responses from a standard scale. This tells us that attitudinal response distributions are very sensitive to fairly moderate changes in format, let alone different question wording to cover the same concept. Therefore regardless of how we present attitude statements to respondents we should always be wary of the subsequent conclusions we draw, and avoid falling into the trap of reporting attitudinal data as fact in the way that some behavioural data is reported.

We are currently setting up our next experiments, which will look at the impact of examples in questions on response and at whether a new technique known as ‘item count’ might increase levels of reporting socially undesirable behaviour. See future issues of Aspects for a discussion of the results from these tests.

For more information about the experiments discussed in this issue please contact Tim Hanson on 020 8433 4233, email tim.hanson@bmrb.co.uk

Commenting is not available in this weblog/section entry.