Self-completion surveys

Self-completion methods, whether paper based or electronic, can benefit from the complete absence of an interviewer from the process. This removes a major source of potential bias in the responses, and makes it easier for respondents to be honest about sensitive subjects.

However, self-completion studies can also suffer from there being no interviewer to identify when a respondent has misunderstood, or to ask for clarification where there are inconsistencies, or to probe for fuller answers.

From the aspect of the survey design, self-completion questionnaires are often considerably cheaper per interview to administer than inter­viewer-administered ones. Against that must be balanced the difficulties of achieving a representative sample when there is such a high degree of self-selection as is typical with self-completion studies, and particularly when there is a low response rate.

1. Paper questionnaires

Paper self-completion questionnaires are typically sent by mail to people who qualify or are thought to qualify as eligible for the study. They may be members of a panel who have agreed to take part in surveys, or they may be taken from a database such as customers of a company or members of an organization.

1.1. Advantages of paper questionnaires

With a paper self-completion questionnaire, respondents have time to consider their answers. They can leave the questionnaire whilst they think about an issue, or whilst they go away to check something or look up some information. With little time pressure on them, they can write lengthy and full answers to open questions if they wish to do so.

Descriptive material can be included for evaluation. Written descrip­tions and pictures of new concepts, products or ideas can be included, and again the respondents have the time to read and digest these before giving their responses. For photographs and drawings, as well as written material, a level of production quality can be achieved that is appropriate to the study.

1.2. Disadvantages of paper questionnaires

With a paper self-completion questionnaire, it is impossible to stop respondents from reading through all of the questions before respond­ing. In other modes the question sequence is often carefully chosen by the questionnaire writer in order to reveal certain pieces of information at a specific point in the interview. That is impossible with this type of questionnaire.

Certain measures cannot therefore be taken. It is not possible to ask a spontaneous brand awareness question if the questionnaire includes brand names in any of the other questions. Respondents may have read through the questionnaire and will have been prompted by mentions of a brand before completing the spontaneous awareness question.

Having time to consider answers, whilst often an advantage, is not always what the questionnaire writer wants. With attitudinal and image questions, it is often the first reaction that is sought, rather than a consid­ered response. An instruction in the question for respondents to give their first reaction cannot be enforced, nor encouraged in the way that an inter­viewer can, either face to face or by telephone.

Where prompt material has been sent to the respondents for their reac­tion, it is difficult to retrieve all of it. This can present a security concern if the material is commercially sensitive.

2. Web-based self-completion

There are several different ways of carrying out surveys using the inter­net. The questionnaire can either be delivered by e-mail or accessed via a web page. The main approaches are summarized by Bradley (1999) as follows:

• Open web – a website open to anyone who visits it.
• Closed web – respondents are invited to visit a website to complete a questionnaire.
• Hidden web – the questionnaire appears to a visitor only when trig­gered by some mechanism (eg date, visitor number, interest in a specific page). This includes pop-up surveys.
• E-mail URL embedded – a respondent is invited by e-mail to the survey site, and the e-mail contains a URL or web address on which respon­dents click.
• Simple e-mail – an e-mail with questions contained in it.
• E-mail attachment – the questionnaire is sent as an attachment to an e­mail.

The last two of these, the simple e-mail and e-mail attachment, are rarely used in commercial research for a variety of practical reasons. Attachments require respondents to download the questionnaire, complete it and then return it. This requires a lot of cooperation and has been shown to lead to low response rates. Questionnaires embedded within e-mails can have their layout distorted, depending on the e-mail software with which it is opened. This can lead to the questionnaire being incomprehensible to the recipient. Both of these routes also suffer from the inability to include complex routeing.

Most practitioners now use questionnaires hosted on a website to which respondents are invited or routed in some way. This book will therefore concentrate on the web-based questionnaire.

As noted above, the invitation to the website or questionnaire can be delivered in a number of ways:

• It can be delivered by e-mail to people on a panel or to a mailing list of customers or people who might qualify for the survey.
• Pop-ups can be used to direct respondents to the questionnaire whilst they are visiting another site. (These are particularly useful where the objectives of the survey relate to the site being visited, such as evaluat­ing the site.)
• Invitations can be posted as banner ads on other sites (eg ISP home pages).
• Respondents can be directed to the site following a recruitment inter­view by telephone or face to face.

There are many different ways of capturing a sample online. There are also many issues regarding how representative such samples are of a population that contains people other than those with internet access. These issues are outside the scope of this book and are well covered else­where.

2.1. Advantages of web-based self-completion

Web-based questionnaires have the same strength as paper self-completion questionnaires in that, in theory at least, respondents can complete the questionnaire in their own time, going away from it if they are interrupted, and returning to it later. In practice, there is little evidence that respondents leave a questionnaire whilst they think about it and return later.

In terms of data collection, the major differences between online surveys and other forms of data collection are the same as between postal self­completion and interviewer-administered surveys. Any advantages are those that come from being technology driven (Ilieva, Baron and Healey, 2002).

Some of the differences between online and other forms of data collec­tion are given by Taylor (2000) as:

• It is a visual medium, allowing images, messages and longer lists of response options. (One survey of motorists has a list of more than 90 different car makes and models for respondents to code their vehicle against. This level of detail would be difficult in any other medium.)
• It captures the unedited voice of the consumer, so that open-ended responses can be richer, longer and more revealing.
• It may be more effective in addressing sensitive issues (medical issues, in particular, may be more easily discussed).
• Scales may elicit different response patterns – it has been the experi­ence both of Taylor and of other researchers that the extremes of scales are used less often.
• More ‘Don’t knows’ may be generated, which is likely to be a function of the ‘Don’t know’ code appearing as a response option.

In addition to online surveys being more effective with sensitive issues, evidence from Kellner (2004) and Basi (1999) supports the view that because there is no interviewer there is less social desirability bias and the respondents answer more honestly (see Chapter 12). This means that data on ‘threatening’ questions, where respondents feel a need to appear to be socially acceptable, are likely to represent better how the survey popula­tion really feels.

The distribution of usage of the points on rating scales has been shown to be different, with less use of the extreme points than is found with face- to-face or telephone interviewing. However, Cobanoglu, Warde and Moreo (2001) have shown that mean scores for data collected via a web- based questionnaire are the same as for other self-completion methods, postal and fax surveys. This supports the view that using a web-based questionnaire should be seen as an alternative method of administering a self-completion survey.

Most studies of how people respond to web-based questionnaires have found that they are completed more quickly than their equivalent tele­phone or face-to-face administered versions. Being quicker can help to make it a more pleasurable experience for respondents.

The presentation of the questionnaire can also help to make its comple­tion pleasurable. With a little flair and imagination, web questionnaires can be designed to have visual appeal, an equivalent level of which is often too costly to achieve with paper questionnaires. In addition to the page design, techniques such as showing icons to represent each brand can be used for respondents to move around the screen and drop into the appropriate response box. By involving the respondents more, the inter­view is more likely to keep their attention and continue to provide good- quality data through to the end of the questionnaire.

Demonstration of material can also be achieved with a web-based survey in many of the same ways as with CAPI surveys. Television advertisements can be shown, although the quality with which they are seen will depend on the specification of the equipment that the respon­dent is using to view it. High-quality representation of still images can be achieved, so that pack designs can be shown either for new or for existing products. There is software available that allows the respondent to rotate the pack representation in three dimensions and even to change elements of it such as colour or text. This kind of technique allows much more interaction in the interview, again involving the respondents and main­taining their interest.

A disadvantage of paper self-completion questionnaires is that the respondents can look ahead. With web-based questionnaires the ques­tions are presented in the sequence that the researcher wants them to be. Generally, web-based questionnaires will allow respondents to go back over questions already answered in order either to check or to change previous answers. However, it is unlikely that respondents will go completely through the interview and then go back to the beginning and change all of their answers.

As with other electronic questionnaires, CATI and CAPI, the web- based questionnaire can change the order of questions between respon­dents; rotate or randomize response lists; customize response lists against previous answers; cope with complex routeing; and carry out calcula­tions within the interview.

2.2. Disadvantages of web-based self-completion

As with all self-completion media, a major disadvantage is not having an interviewer on hand to clarify questions or to repair misunderstandings.

It might be thought that an issue with web-based questionnaires would be the difficulty of recording open-ended verbatim responses. Most respondents are not accomplished typists, and it might be expected that questions that require responses to be typed in verbatim would be poorly completed, and be at best completed perfunctorily and in abbreviated fashion. However, experience has shown that, whilst this is undoubtedly an issue with some respondents, the overall level of detail to which this type of question is completed is high. The ability of respondents to take their time and think about their answer appears to more than cancel out any typing difficulties, and responses are generally as complete as for inter­viewer-administered questionnaires.

Web-based surveys have other disadvantages compared to face-to-face surveys, such as the inability to touch or smell stimuli, but these tend to be issues of survey design rather than questionnaire design.

Source: Brace Ian (2018), Questionnaire Design: How to Plan, Structure and Write Survey Material for Effective Market Research, Kogan Page; 4th edition.