Methods of determining the reliability of an instrument in quantitative research

There are a number of ways of determining the reliability of an instrument and these can be classified as either external or internal consistency procedures.

1. External consistency procedures

External consistency procedures compare findings from two independent processes of data collection with each other as a means of verifying the reliability of the measure. The two methods of doing this are as follows:

• Test/retest – This is a commonly used method for establishing the reliability of a research tool. In the test/retest (repeatability test) an instrument is administered once, and then again, under the same or similar conditions. The ratio between the test and retest scores (or any other finding, for example the prevalence of domestic violence, a disease or incidence of an illness) is an indication of the reliability of the instrument – the greater the value of the ratio, the higher the reliability of the instrument. As an equation,

(test score)/(retest) = 1

or

(test score) – (retest) = 0

A ratio of 1 shows 100 per cent reliability (no difference between test and retest) and any deviation from it indicates less reliability – the less the value of this ratio, the less the reliabil­ity of the instrument. Expressed in another way, zero difference between the test and retest scores is an indication of 100 per cent reliability. The greater the difference between scores or findings obtained from the two tests, the greater the unreliability of the instrument.

The main advantage of the test/retest procedure is that it permits the instrument to be com­pared with itself, thus avoiding the sort of problems that could arise with the use of another instrument.

The main disadvantage of this method is that a respondent may recall the responses that s/he gave in the first round, which in turn may affect the reliability of the instrument. Where an instrument is reactive in nature (when an instrument educates the respondent with respect to what the researcher is trying to find out) this method will not provide an accurate assess­ment of its reliability. One of the ways of overcoming this problem is to increase the time span between the two tests, but this may affect reliability for other reasons, such as the maturation of respondents and the impossibility of achieving conditions similar to those under which the questionnaire was first administered.

• Parallel forms of the same test – In this procedure you construct two instruments that are intended to measure the same phenomenon. The two instruments are then administered to two similar populations. The results obtained from one test are compared with those obtained from the other. If they are similar, it is assumed that the instrument is reliable.

The main advantage of this procedure is that it does not suffer from the problem of recall found in the test/retest procedure. Also, a time lapse between the two tests is not required. A disadvantage is that you need to construct two instruments instead of one. Moreover, it is extremely difficult to construct two instruments that are comparable in their measurement of a phenomenon. It is equally difficult to achieve comparability in the two population groups and in the two conditions under which the tests are administered.

2. Internal consistency procedures

The idea behind internal consistency procedures is that items or questions measuring the same phenomenon, if they are reliable indicators, should produce similar results irrespective of their number in an instrument. Even if you randomly select a few items or questions out of the total pool to test the reliability of an instrument, each segment of questions thus constructed should reflect reliability more or less to the same extent. It is based upon the logic that if each item or question is an indicator of some aspect of a phenomenon, each segment constructed will still reflect different aspects of the phenomenon even though it is based upon fewer items/questions. Hence, even if we reduce the number of items or questions, as long as they reflect some aspect of a phenomenon, a lesser number of items can provide an indication of the reliability of an instrument. The internal consistency procedure is based upon this logic. The following method is commonly used for measuring the reliability of an instrument in this way:

• The split-half technique – This technique is designed to correlate half of the items with the other half and is appropriate for instruments that are designed to measure attitudes towards an issue or phenomenon. The questions or statements are divided in half in such a way that any two questions or statements intended to measure the same aspect fall into different halves. The scores obtained by administering the two halves are correlated. Reliability is calculated by using the product moment correlation (a statistical procedure) between scores obtained from the two halves. Because the product moment correlation is calculated on the basis of only half the instrument, it needs to be corrected to assess reliability for the whole. This is known as stepped-up reliability. The stepped-up reliability for the whole instrument is calculated by a formula called the Spearman-Brown formula (a statistical procedure).

Source: Kumar Ranjit (2012), Research methodology: a step-by-step guide for beginners, SAGE Publications Ltd; Third edition.