External Validity of the Research

1. Definition and Overview

To assess the external validity of a research project we examine the possibilities and conditions for generalizing and appropriating the model to other sites. There are two facets to external validity, which we present in the following dis­cussion; these correspond to a logical progression in two stages in assessing research validity.

The researcher must first examine the degree to which results found from a sample can be generalized to the whole parent population (supposedly repre­sented by the sample, which has been established for the research at hand). It is only in a second stage that we can evaluate to what extent these results can be transferred or appropriated to the study and understanding of other obser­vational fields.

The potential to generalize from research results is a concern that is more familiar to researchers who apply a quantitative methodology than to those who use qualitative methods. Quantitative researchers are generally used to working with samples, as accessing the whole population is either too difficult or too costly, or may not even be necessary (a statistically representative num­ber of individuals of a population may be sufficient). Quantitative researchers are familiar with having to define the characteristics of the population under study. These characteristics provide them with criteria to use when selecting the units that will make up their study samples. Research results may then be extra­polated, taking certain statistical precautions, to the entire target population – and the status of these results can be established. However, this concern with the need to be able to generalize from their results should not be neglected by researchers who carry out qualitative work.

The status of the case studies conducted during a qualitative study can sometimes be poorly defined. Say, for example, that an in-depth case study has been carried out on company ‘X’ to study a process of industrial restructuring. Are we than to consider ‘X’ as a representative sample of a population of com­panies possessing similar characteristics, or confronted with identical strategic issues or, on the contrary, does it constitute a specific population? The status of the results of qualitative research is dependent on the answer to this question.

In both quantitative and qualitative research, the sample studied and the population targeted must be defined before we can determine the generaliza­tion perimeter of the results. To do this, qualitative research draws on a process of statistical generalization, whereas qualitative research draws on a process of analytical generalization (Yin, 1989).

The second facet of external validity – that of the transferability of results – concerns both work that assesses the potential to extrapolate research into other observational fields, and researchers who incorporate into their own research approach results imported from a different domain to that in which they are studying. In both of these situations researchers should always consider the possible contextual dependence of the research results they are working with. Contextual dependence is a measure of whether a result demonstrated in one observational field is dependent solely on one or more of the research variables, or whether it depends also on other characteristics particular to the studied field – in which case the research is culturally, his­torically or socially anchored to some extent to the field (contextualization). Although this problem is no impediment to research in itself, it should be taken into account when determining the possibilities or conditions for extra­polating results to other observational fields which do not necessarily present the same contextual characteristics.

This problem is very often raised in assessing the external validity of quali­tative research, when the results have been drawn from analysis of a single case or a small number of cases. Whereas qualitative research is often criticized for being too contextualized, Guba and Lincoln (1994) consider that work based on quantitative data too often favors precision to the detriment of contextualiza- tion. Transferability can be limited if the aggregate data has no particular appli­cation to practical cases in management. Quantitative researchers working with

large amounts of data should not be under the illusion that they understand the studied observational field in all its principal contextual characteristics (Silverman, 1993). Qualitative research can give invaluable information about the context from which the results are derived, and consequently about the con­texts in which these results can be used again. In more general terms, a detailed, rich and intimate knowledge of the context of their research enables researchers to estimate the possibilities of, and the conditions for, generalizing or transfer­ring their results to other settings.

Although they are often linked, the two facets of external validity discussed above (that is, generalization and transferability of results) should be distinct in each research project. Researchers may not necessarily aim to produce results that can be generalized to the entire population, nor to assess the possibilities for transferring their results to other observational fields. Or researchers may consider the question of generalization without having to consider that of the transferability of the results (and vice versa). It is important, though, for researchers to define their research objectives and, consequently, to marshal appropriate techniques and methods to ensure their results meet one or the other of these conditions of external validity.

2. Assessing External Validity

We set out, below, a number of techniques, tests or procedures (the list is not exhaustive) that can be used to improve the external validity of research results, according to whether the research is quantitative or qualitative. For both types of research we outline, where necessary, any problem areas with regard to the generalization and transferability of research results.

The external validity of a study depends essentially on the external validity of the measuring instrument used in the case of quantitative research, and of the research procedure itself in the case of qualitative research. For this reason, external validity techniques or tests differ greatly according to the nature of the research.

Before examining these techniques of improving external validity, let us point out that researchers will be far better able to ensure the external validity of their research if they take a hard look at the particularities of their observa­tional field from the outset.

In particular, researchers can include certain control variables in their mea­suring instrument, from its conception, to delimit and accurately characterize the population they are studying. By doing so they will improve the level of external validity of the results they obtained on completion of the study.

Researchers should also examine very carefully the variables they use in their study. Generalizing from research, or moving from one context to another, often implies modifying how these variables are operationalized. For example, the relationship between capacity to change and organizational size presupposes that the variable ‘size of the organization’ is to be measured. In the industrial sector, size may be measured by the turnover of the businesses under study. In the non-profit sector, a different measure will have to be devised (for example, the number of volunteers working for these organizations).

2.1. Quantitative research

The researcher must first determine the degree to which the results drawn from a sample can be taken as applying to the whole population, and to what degree these results can be compared to the norms or standards generally accepted about this population (as a result of previous studies for example). These two questions relate to the practice of statistical inference in quantitative research, and a number of statistical tests are available to researchers to answer them. These tests are discussed in Chapter 14.

When research has been carried out on a sample, researchers often hope to generalize their results to the population from which the sample has been drawn. The results of quantitative research are often presented in statistical form, to reduce the large amount of numerical data involved (percentages, means, standard deviations, etc.). To generalize from these results researchers must apply statistical generalization. Correspondingly, statistical formulae are used to evaluate results that have been generalized from a sample to a popula­tion. These formulae differ according to whether the results are in the form of a mean or a proportion. In both cases, we speak of the error margin within which the result generalized to the whole population lies. To determine this error margin, the researcher has to know the size of the sample and its confi­dence level (generally 95 per cent).

As we have shown in this chapter, the question of the transferability of results from one study to other related observational fields depends essentially on the following two factors:

• the external validity of the measuring instrument used (this is discussed fur­ther in Section 2 of this chapter)
• the validity of inferring from results from one population to another (this is discussed further in Chapter 14).

We emphasize that researchers should make use of appropriate statistical tests, which are called non-parametric tests, when working on small samples. Chapter 14 explains how these tests can be used during the research process to assure its external validity.

2.2. Qualitative research

When qualitative research produces figures in the form of proportions or means, the techniques we have presented above for quantitative research can equally be applied to generalize a result within a set margin of error, or to transfer results using statistical tests. The sample should in this case, however, comprise at least 30 units (companies observed, managers questioned, etc.), a number which is not abnormal in qualitative research.

However, the results of a qualitative study are generally presented in the form of a proposition or a written statement derived from qualitative data, in which case the use of statistical tests is not possible.

The passage from qualitative data, collected in a large quantity and often diverse in nature, to a conclusion in the form of a proposition of results, depends above all on a certain number of techniques of data collection, reduction (condensing) and analysis (Altheide and Johnson, 1994; Miles and Huberman, 1984a; Silverman, 1993). It depends, too, on the expertise and the experience of the researcher in collating this mass of information. For this reason, tech­niques aimed at assuring the external validity of qualitative research apply principally to the research process (Silverman, 1993). Only the researcher is really in a position to say how much the observational field has been taken into account and how he or she intends to allow for specific local factors in each case, in order to be able to generalize the results to a much greater arena. Researchers should always question their methods of working. They should examine the relationship between their research question and the broader historical and social context of their work, and give consideration to the rela­tionship between the observer, the observed and the place of observation, and to the observer’s point of view and his or her interpretation of the obser­vational field.

Two aspects of the qualitative research procedure need to be examined in more detail, however, as they have a direct bearing on the external validity of the research: the method used to select the observational field, and the method used to analyze the collected data.

A number of different techniques may be used when a researcher wishes to generalize from case study results that might be considered as idiosyncratic situations. Numerous authors recommend using several case studies (Eisenhardt, 1989; Guba and Lincoln, 1994) to vary the contextual characteristics of qualitative research and to limit or control as much as possible particularities of individual cases. A poorly thought-out choice of several cases does not always provide any real improvement to the external validity of the results. The following methods can be used to avoid this trap.

First, repeating a case study (Yin, 1989) will normally help to reach a theo­retical and literal generalization. In repeating a study, the researcher may either select a case for which the same results are predicted (literal replication) or select a case for which different results are produced, but for anticipated rea­sons (theoretical replication). The results of the qualitative study may then be compared or contrasted according to the characteristics – identical or different – of the cases available to the researcher.

For such case comparison to be effective, certain criteria (carefully chosen in line with the research question) must be included in each case, and the cases should vary in relation to these criteria. A good knowledge of the observational field is then vital when formulating these criteria – a knowledge based at the least on a detailed description of the field’s general framework and the activi­ties or actors being studied. These criteria may be determined at the outset of the study or they may be formulated as the research progresses, in relation to early results obtained.

The process of choosing different sites for case studies has to be carried out with care, as researchers can be influenced by a ‘representativeness bias’ (Miles and Huberman, 1994). Researchers sometimes have a tendency to select ‘simi­lar’ cases, so that their results converge and they avoid being temporarily thrown off the track by contradictory or unforeseen results. Researchers should always set down clearly the criteria they have used for selecting cases to study, and consider them with a critical eye; it can be useful to seek an external opin­ion on this selection.

There are no fixed rules setting a maximum number of repeated case studies below which a research study can maintain its qualitative nature, or above which the researcher must use statistical tools to deal with too great an amount of information. Although it is generally accepted that an understanding of local causality becomes crucial when there are more than 15 sites, this depends to a great extent on the researcher’s expertise in carrying out such qualitative studies.

The external validity of qualitative research depends also on the way in which the collected data is condensed and analyzed. Different techniques have been proposed (see Miles and Huberman, 1984a) with which researchers can move from a local explanation (causality) to an inter-site explanation, and reach a higher level of external validity. These techniques are based essentially on the use of data analysis matrices.

Although researchers do not always have the opportunity or the time to carry out a multi-site study, they can try to assure the external validity of their results by using paradox, or apparent contradiction (Quinn and Cameron, 1988) – by comparing their results with the work of other researchers (Eisenhardt, 1989) so as to interpret their single case study in a different way.

Source: Thietart Raymond-Alain et al. (2001), Doing Management Research: A Comprehensive Guide, SAGE Publications Ltd; 1 edition.