The question of validity in observational research

Quantitative researchers can demonstrate both the validity and the reliability of their data through statistical means. ‘Reliability’ is a measure of the degree to which any given observation is consistent with a general pattern and not the result of random chance. ‘Validity’ is a measure of the degree to which an observation actually demonstrates what it appears to demonstrate. Qualitative ethnographic researchers are not usually concerned with reliability, since they recognize the fact that much of what they do is, in the last analysis, not truly replicable. There is, in other words, no expectation that one researcher observing a community at one time will exactly duplicate the findings of a different researcher observing that same community at a different time. By contrast, a biologist observing cellular processes under a microscope should come up with standard results no matter who he or she is, when the observation was made, and so forth.

Nevertheless, there are some ways in which observation-based researchers can achieve something approaching criteria of scientific reliability. For example, observations that are conducted in a systematic fashion (i.e. using some sort of standardized technique for the recording and analysis of the data) and that are repeated regularly over a course of time can be considered credible if they yield roughly comparable results. The desire to approximate scientific reliability in observational research, however, represents a reliance on a view of social research as a species of science in which human behavior is ‘lawful’ and regular and can be objectively described and analyzed. Such a position would, of course, be considered irrelevant by postmodernists of various types, as discussed in an earlier chapter.

On the other hand, even postmodernists must be very concerned with validity; if there is no basis for trusting the observation, then the research is meaningless. The question of validity haunts qualitative research in general, but it poses par­ticular problems for observation-based research. Observations are susceptible to bias from subjective interpretations. Unlike interview-based research, which can feature direct quotes from people in the community, observational findings are rarely ‘confirmable’. Nevertheless, there are some ways that observational researchers can legitimize their work for their scientific peers. (Note that they may not have to do so for general or popular audiences, for whom the fact that the observer was ‘there’ and speaks with a voice of authority about what he or she has found out, is often good enough.) Some of the most commonly deployed means of achieving validity include the following:

• It is often advisable to work with multiple observers or teams (see also Flick, 2007b), particularly if they represent various viewpoints (e.g. gender, age, ethnic background); the members of such teams can cross-check each others’ findings in order to discover and eliminate inaccuracies. Of course, an observer whose findings are not in agreement with those of his or her colleagues is not necessarily ‘wrong’; he or she may, in fact, be the only one to have gotten it right. However, unless there is compelling reason to suspect that the loner/maverick is on to something important, the consensus of the group usually prevails.
• It may be possible to follow the methodology of analytic induction (see also Flick, 2007b), which in this case means that emergent propositions (findings that describe patterns in the observations) are tested in a search for negative cases. The goal is to achieve assertions that can be taken as universal (or ‘grounded’, in the language of some schools of theory).
• When writing up results, the observation-based researcher may be encouraged to use techniques of verisimilitude (or vraisemblance, a term that has come into English via French scholars). This is a style of writing that draws the reader into the world that has been studied so as to evoke a mood of recog­nition; it uses rich descriptive language (rather than abstract ‘facts and figures’). Verisimilitude is also achieved when the description seems to be internally coherent, plausible, and recognizable by readers from their own experiences or from other things they have read or heard about. A work that achieves these goals is said to be authentic in the eyes of those who read it. In other words, more than other types of scientific ‘data’, ethnographic observations only become ‘valid’ when they have been rendered into some sort of coherent, consistent narrative.

The whole matter of standards for assuring the quality of research findings generated in non-quantified contexts has been studied extensively and summa­rized by Seale (1999). Guba and Lincoln (2005, pp. 205-9) provide both a brief review of the literature and a complex philosophical reflection on the question of validity in qualitative research. After considerable examination of the ways in which qualitative researchers collect data, including those who use observational and other ethnographic means to collect information, Miles and Huberman (1994, pp. 278-80) have come up with some practical ‘pointers’ (not ‘rules’, they care­fully explain) to help us judge the quality of research conclusions. They divide their pointers into five basic categories:

• Objectivity/confirmability (or ‘external reliability’): the degree to which con­clusions flow from the information that has been collected, and not from any biases on the part of the researcher.
• Reliability/dependability/auditability: the degree to which the process of research has been consistent and reasonably stable over time and across var­ious researchers and methods.
• Internal validity/credibility/authenticity (or ‘truth value’): the degree to which the conclusions of a study make sense, if they are credible to the people studied as well as to readers of the report, and if the final product is an authentic record of whatever it was that was observed.
• External validity/transferability/fittingness: the degree to which the conclu­sions of a study have relevance to matters beyond the study itself (i.e. can the findings be generalized to other contexts?).
• Utilization/application/action orientation (the ‘pragmatic validity’ of a study): the degree to which programs or actions result from a study’s findings and/or the degree to which ethical issues are forthrightly dealt with (for criteria in qualitative research more generally, see Flick, 2007b).

Source: Angrosino Michael (2008), Doing Ethnographic and Observational Research, SAGE Publications Ltd; 1st edition.