Reliability of the Research

Doing research takes time and involves a community of researchers. However, it would be very prejudicial if the soundness or precision of results produced in this way were dependent on the individual method of each researcher in conducting a project, or again, on conditions peculiar to that study. The relia­bility of research results, over time and across a community of researchers, is an important consideration.

1. Definitions and Overview

Evaluating the reliability of research (that is, the reliability of its results) con­sists in establishing and verifying that the various processes involved will be able to be repeated with the same results being obtained by different researchers and/or at different periods. Researchers who are integrated into a scientific team must be able to convey as faithfully as possible their method of carrying out a project. This is the concept of diachronic reliability (Kirk and Miller, 1986), which examines the stability of an observation over time. Researchers must be also able to duplicate exactly a study they have previously conducted, for example, when they are conducting multi-site research over several months (synchronic reliability, Kirk and Miller, 1986, which examines the similarity of observations over the same period of time).

Kirk and Miller (1986) and Silverman (1993) both mention the quixotic reli­ability dimension of a research project’s reliability, which evaluates the circum­stances in which the same method of observation will lead to the same results. This dimension of reliability is strongly linked to the reliability of the measur­ing instrument (refer to Section 2 of this chapter).

The question of reliability concerns all operational stages of quantitative or qualitative research. These include data collection, coding, and all other pro­cesses of preparing and analyzing data, including the presentation of the results when the vocabulary or the tables used are specific to the research. It is impor­tant for researchers to precisely describe their research design, so as to aim for a higher degree of reliability. Research is a complex process (whose evolution is never linear) and often takes place over a long period of time. Researchers may forget what they have done, and why and how they did it, by the time they attempt to repeat their research or initiate a research team within a different observational field.

A social science that is situated in time, research is also a personal exercise that relies on the intuition of the researcher. It is an imaginative practice: ‘the process of theory construction in organizational studies is portrayed as imagi­nation disciplined by evolutionary processes analogous to artificial selection’ (Weick, 1989: 516). An aptitude for imagination, for perception when doing fieldwork, is not transmissible, but the process of questioning is. The degree to which a researcher will be able to duplicate a study will depend also on the accuracy of his or her description of the research process employed.

The principle techniques for attaining sound reliability in research are presented below. In general these relate to the organization and the quality of the research protocol.

2. Assessing Research Reliability

2.1. Methods common to both quantitative and qualitative research

Most importantly, researchers should always pay great attention to the com­munication of methodological information (the research process) from one researcher to another, or from one observational field to another. The different stages in the research should be clearly described, including discussion of the choice of observational field, the methods used in the collection and analysis of data, and the steps taken to control the effect the researcher may have on the observational field.

There must be a concern at all times to control the effect of the researcher on the observational field, and not only in the case of the solitary qualitative researcher. In fact, in quantitative research, the administration of a questionnaire can be disrupted by the attitude of a researcher who appears, for example, to be hurried, or who might be judgemental about the responses of the people being questioned. This kind of attitude cannot fail to disturb and to influence respondents.

At the same time, particular attention should also be given to certain other aspects of the research, according to its type. In the case of quantitative research, research reliability seems to depend more on the reliability of the mea­suring instrument. In the case of qualitative research, it seems to depend more on the ability of the researcher to understand and reconstruct the observational field.

2.2. Qualitative research

The reliability of qualitative research depends partly on the reliability of the measuring instrument. However, the interaction between the researcher and the observational field and the role of the researcher in administering the mea­suring instrument have a greater impact on research reliability in the case of qualitative research than quantitative, by reason of the very nature of the mea­suring instruments used (the qualitative instrument). Researchers must pay particular attention to writing concise instructions if qualitative measuring instruments are to be used by several people or at different times. They should explain how to use the instrument, how to understand questions that may be posed if respondents want further explanation before replying, how to select people to be questioned and, finally, how to take notes (extensive or pre-coded, for example) on the interviewee’s replies. These instructions can take different forms; such as a manual for the observer in the studied observational field, or as notes accompanying a guide to the interviewing technique, explaining the contents of the questionnaire and how it is to be administered. Carrying out a pre-test can be an appropriate occasion for developing these guidelines, which can also be used in the post-test. Finally, particular importance must be given to training those who will administer this measuring instrument.

The reliability of qualitative research depends mainly on the ability and honesty of the researcher in describing the entire research process employed, particularly in the phases which relate to condensing and analyzing the col­lected data (Miles and Huberman, 1984a). The operation of condensing data consists of a group of processes of selection, grouping, simplifying and trans­forming the raw data collected (Miles and Huberman, 1984a). The researcher arrives at a set of data that has been simplified, transformed and reduced in number (condensed), and the task of data analysis is made easier.

The dependence of reliability on the ability and honesty of the researcher concerns both the qualitative and the quantitative researcher. However, quan­titative research makes use of numerous techniques, statistical tools and tests which can be explained very precisely. This emphasis on the analysis process appeared for a long time to be less important to the qualitative researcher, par­ticularly as no specific analysis techniques exist for such research.

The following discussion is based on an article by Gioia and Thomas (1996), who give their readers a precise description of their research procedure in such a way that it can be reproduced. This description is interesting in that it gives a direct account of the methodical progression of the research, from the phase of data collection to data analysis. If research relies on a logical progression of analysis phases (Glaser and Strauss, 1967), an explanation of the process must account for this.

Similarly, Miles and Huberman (1984a), as well as Silverman (1993), recom­mend drawing up and using identical note-taking formats so as to compare different sites and to quickly develop a methodology for gathering raw data that is easy to repeat. Miles and Huberman (1984a) propose different techniques (in the form of a matrix presenting and analyzing data) to improve research reliability.

The use of matrices to present, reduce, and analyze qualitative data collec­ted in the field allows the researcher to increase the level of reliability of the research. But particular care should be taken to describe the reasons behind (the why) and the methods used (the how) in the construction of these matrices. The process of compiling and analyzing data then seems more precise, or ‘objective’, as it is no longer based on only a few personal and inaccessible methods used by a particular researcher, but instead is based on clearly explained methods.

All the same, it would not be extreme to recommend the inclusion in research projects of a certain amount of data relating to the researcher himself or herself (professional background, academic training, etc.).

Source: Thietart Raymond-Alain et al. (2001), Doing Management Research: A Comprehensive Guide, SAGE Publications Ltd; 1 edition.