Three Principles of Data Collection in Case Study Method

The benefits from these six sources of evidence can be maximized if you fol­low three principles. These principles are relevant to all six sources and, when used properly, can help to deal with the problems of establishing the construct validity and reliability of the case study evidence. The three are as follows.

1. Principle 1: Use Multiple Sources of Evidence

Any of the preceding sources of evidence can and have been the sole basis for entire studies. For example, some studies have relied only on participant- observation but have not examined a single document; similarly, numerous studies have relied on archival records but have not involved a single interview.

This isolated use of sources may be a function of the independent way that sources have typically been conceived—as if an investigator should choose the single most appropriate source or the one with which she or he is most famil­iar. Thus, on many an occasion, investigators have announced the design of a new study by identifying both the problem to be studied and the prior selec­tion of a single source of evidence—such as “interviews”—as the focus of the data collection effort.

Triangulation: Rationale for using multiple sources of evidence. The approach to individual sources of evidence as just described, however, is not recommended for conducting case studies. On the contrary, a major strength of case study data collection is the opportunity to use many different sources of evidence (see BOX 22 and BOX 19B, earlier, for examples of such stud­ies). Furthermore, the need to use multiple sources of evidence far exceeds that in other research methods, such as experiments, surveys, or histories. Experiments, for instance, are largely limited to the measurement and record­ing of actual behavior in a laboratory and generally do not include the sys­tematic use of survey or verbal information. Surveys tend to be the opposite, emphasizing verbal information but not the measurement or recording of individual behavior. Finally, histories are limited to events in the “dead” past and therefore seldom have any contemporary sources of evidence, such as direct observations of a phenomenon or interviews with key actors.

BOX 22

A Case Study Combining Personal Experience
with Extensive Field Research

Most people across the country by now have heard of Head Start. Its development and growth into one of the most successful federal programs is traced by Zigler and Muenchow (1992). Their book is exceptionally insightful, possibly because it is based on Zigler’s personal experiences with the program, beginning with his role as its first director. However, the book also calls on other independent sources of evidence, with the coauthor contributing historical and field research, including interviews of more than 200 persons associated with Head Start. All of these multiple sources of evi­dence are integrated into a coherent if not compelling case study of Head Start. The result is a winning combination: a most readable but also well-documented book.

Of course, each of these strategies can be modified, creating hybrid strate­gies in which multiple sources of evidence are more likely to be relevant. An example of this is the evolution of “oral history” studies in the past several decades. Such studies involve extensive interviews with key leaders who have retired, on the stipulation that the interview information will not be reported until after the leader’s death. Later, the historian will join the interview data with the more conventional array of historical evidence. Nevertheless, such a modification of the traditional methods does not alter the fact that the case study inherently deals with a wide variety of evidence, whereas the other methods do not.

The use of multiple sources of evidence in case studies allows an investiga­tor to address a broader range of historical and behavioral issues. However, the most important advantage presented by using multiple sources of evidence is the development of converging lines of inquiry, a process of triangulation and corroboration emphasized repeatedly in the previous section of this chapter. Thus, any case study finding or conclusion is likely to be more convincing and accurate if it is based on several different sources of information, following a corroboratory mode (see BOX 23).

BOX 23

Triangulating from Multiple Sources of Evidence

Basu, Dirsmith, and Gupta (1999) conducted a case study of the federal govern­ment’s audit agency, the U.S. Government Accountability Office. Their case was theory oriented and examined the relationship between an organization’s actual work and the image it presents to external parties (the finding was that they are loosely coupled). The case study used an impressive array of sources of evidence— an extended period of field observations, with diaries; interviews of 55 persons; and reviews of historical accounts, public records, administrators’ personal files) and news articles—all triangulating on the same set of research questions.

Patton (2002) discusses four types of triangulation in doing evaluations— the triangulation

1. of data sources {data triangulation),
2. among different evaluators {investigator triangulation),
3. of perspectives to the same data set (theory triangulation), and
4. of methods (methodological triangulation).

The present discussion pertains only to the first of these four types {data triangulation), encouraging you to collect information from multiple sources but aimed at corroborating the same fact or phenomenon. In pursuing such corroboratory strategies, Figure 4.2 distinguishes between two conditions— when you have really triangulated the data (upper portion) and when you have multiple sources as part of the same study but that nevertheless address differ­ent facts (lower portion). When you have really triangulated the data, the events or facts of the case study have been supported by more than a single source of evidence; when you have used multiple sources but not actually tri­angulated the data, you typically have analyzed each source of evidence sepa­rately and have compared the conclusions from the different analyses—but not triangulated the data.

With data triangulation, the potential problems of construct validity also can be addressed because the multiple sources of evidence essentially provide multiple measures of the same phenomenon. Not surprisingly, one analysis of case study methods found that those case studies using multiple sources of evi­dence were rated more highly, in terms of their overall quality, than those that relied on only single sources of information (see COSMOS Corporation, 1983).

Prerequisites for using multiple sources of evidence. At the same time, the use of multiple sources of evidence imposes a greater burden, hinted at earlier, on yourself or any other case study investigator. First is that the collection of data from multiple sources is more expensive than if data were only collected from a single source (Denzin, 1978, p. 61). Second and more important, each inves­tigator needs to know how to carry out the full variety of data collection tech­niques. For example, a case study investigator may have to collect and analyze documentary evidence as in history, to retrieve and analyze archival records as in economics or operations research, and to design and conduct surveys as in survey research. If any of these techniques is used improperly, the opportunity to address a broader array of issues, or to establish converging lines of inquiry, may be lost. This requirement for mastering multiple data collection tech­niques therefore raises important questions regarding the training and exper­tise of the case study investigator.

Unfortunately, many graduate training programs emphasize one type of data collection activity over all others, and the successful student is not likely to have a chance to master the others. To overcome such conditions, you should seek other ways of obtaining the needed training and practice. One such way is to work in a multidisciplinary research organization rather than being limited to a single academic department. Another way is to analyze the methodological writings of a variety of social scientists (see Hammond, 1968) and to learn of the strengths and weaknesses of different data collec­tion techniques as they have been practiced by experienced scholars. Yet a third way is to design different pilot studies that will provide an opportunity for practicing different techniques.

No matter how the experience is gained, every case study investigator should be well versed in a variety of data collection techniques so that a case study can use multiple sources of evidence. Without such multiple sources, an invaluable advantage of the case study strategy will have been lost. Worse, what started out as a case study may turn into something else. For example, you might overly rely on open-ended interviews as your data, giving insuffi­cient attention to documentary or other evidence to corroborate the inter­views. If you then complete your analysis and study, you probably will have done an “interview” study, similar to surveys that are entirely based on ver­bal reports that come from open-ended interviews—but you would not have done a case study. In this interview study, your text would constantly have to point out the self-reported nature of your data, using such phrases as “as reported by the interviewees,” “as stated in the interviews,” or “she/he reported that….” and the like.

EXERCISE 4.3 Seeking Converging Evidence

Name a particular incident that occurred recently in your everyday life. How would you go about establishing the “facts” of this incident, if you wanted now (in retrospect) to demonstrate what had happened? Would you inter­view any important persons (including yourself)? Would there have been any artifacts or documentation to rely on?

2. Principle 2: Create a Case Study Database

A second principle has to do with the way of organizing and documenting the data collected for case studies. Here, case studies have much to borrow from the practices followed by the other research methods defined in Chapter

Their documentation commonly consists of two separate collections:

1. the data or evidentiary base and
2. the report of the investigator, whether in article, report, or book form.

With the advent of computer files, the distinction between these two collec­tions has been made even clearer. For example, investigators doing psycho­logical, survey, or economic research may exchange data files and other electronic documentation that contain only the actual database—for example, behavioral responses or test scores in psychology, itemized responses to vari­ous survey questions, or economic indicators. The database then can be the subject of separate, secondary analysis, independent of any reports by the orig­inal investigator.

However, with case studies, the distinction between a separate database and the case study report has not yet become an institutionalized practice. Too often, the case study data are synonymous with the narrative presented in the case study report, and a critical reader has no recourse if he or she wants to inspect the raw data that led to the case study’s conclusions. The case study report may not have presented adequate data, and without a case study data­base, the raw data may not be available for independent inspection. A major exception to this is where ethnographic studies have separated and stored data on their fieldwork, to make these data available to new research investigators. The practice is sufficiently important, however, that every case study project should strive to develop a formal, presentable database, so that in principle, other investigators can review the evidence directly and not be limited to the written case study reports. In this manner, a case study database markedly increases the reliability of the entire case study.

The lack of a formal database for most case studies is a major shortcoming of case study research and needs to be corrected. There are numerous ways of accom­plishing the task, as long as you and other investigators are aware of the need and are willing to commit the additional effort required to build the database. At the same time, the existence of an adequate database does not preclude the need to pre­sent sufficient evidence within the case study report itself (to be discussed further in Chapter 6). Every report should still contain enough data so that the reader of the report can draw independent conclusions about the case study.

Nevertheless, the problem of initially establishing a case study database has not been recognized by most of the books on field methods. Thus, the subsec­tions below represent an extension of the current state of the art. The problem of developing the database is described in terms of four components: notes, documents, tabular materials, and narratives.

Case study notes. For case studies, your own notes are likely to be the most common component of a database. These notes take a variety of forms. The notes may be a result of your interviews, observations, or document analysis. The notes may be handwritten, typed, on audiotapes, or in word-processing or other electronic files, and they may be assembled in the form of a diary, on index cards, or in some less organized fashion.

Regardless of their form or content, these case study notes must be stored in such a manner that other persons, yourself included, can retrieve them effi­ciently at some later date. Most commonly, the notes can be organized accord­ing to the major subjects—as outlined in the case study protocol—covered by a case study; however, any classificatory system will do, as long as the system is usable by an outside party. Only in this manner will the notes be available as part of the case study database.

This identification of the notes as part of the case study database does not mean, however, that you need to spend excessive amounts of time in rewriting interviews or making extensive editorial changes to make the notes pre­sentable. Building such a formal case record, by editing and rewriting the notes, may be a misplaced priority. Any such editing should be directed at the case study report itself, not at the notes. The only essential characteristics of the notes are that they be organized, categorized, complete, and available for later access.

Case study documents. Many documents relevant to a case study will be col­lected during the course of a study. Chapter 3 indicated that the disposition of these documents should be covered in the case study protocol and suggested that one helpful way is to have an annotated bibliography of these documents. Such annotations would again facilitate storage and retrieval, so that later investigators can inspect or share the database.

The single, unique characteristic of these documents is that they are likely to require a large amount of physical storage space, unless you trouble to make portable document format (PDF) copies and store them electronically. In addi­tion, the documents may be of varying importance to the database, and you may want to establish a primary file and a secondary file for such documents. The main objective, again, is to make the documents readily retrievable for later inspection or perusal. In those instances in which the documents have been relevant to specific interviews, one additional cross-reference is to have the interview notes cite the documents.

Tabular materials. The database may consist of tabular materials, either col­lected from the site being studied or created by the research team. Such mate­rials also need to be organized and stored to allow for later retrieval.

The materials may include survey and other quantitative data. For example, a survey may have been conducted at one or more of the case study sites as part of an embedded case study. In such situations, the tabular materials may be stored in computer files. As another example, in dealing with archival or observational evidence, a case study may have called for “counts” of various phenomena (see Miles & Huberman, 1994). The documentation of these counts, done by the case study team, also should be organized and stored as part of the database. In brief, any tabular materials, whether based on surveys, observational counts, or archival data, can be treated in a manner similar to the way they are handled when using other research methods.

Narratives. Certain types of narrative, produced by a case study investigator upon completion of all data collection, also may be considered a formal part of the database and not part of the final case study report. The narrative reflects a special practice that should be used more frequently: to have case study investigators compose open-ended answers to the questions in the case study protocol. This practice has been used on several occasions in multiple-case studies designed by the author (see BOX 24).

BOX 24

Narratives in the Case Study Database

A series of 12 case studies was done on personal computer use in schools (Yin, 2003, chap. 3). Each case study was based on open-ended answers to about 50 protocol questions concerning matters such as the number and location of the personal com­puters (an inventory question requiring tabular and narrative responses), the rela­tionship between the computer units and other computational systems within a school district, and the training and coordination provided by the district.

After data collection has finished, the case study investigator’s first responsibility was to answer these 50 questions as completely as possible, citing specific sources of evidence in footnotes. These answers were unedited but served as the basis for both the individual case reports and the cross-case analysis. The availability of the data­base meant that other members of the case study team could determine the events at each site, even before the case study reports were complate.

In such a situation, each answer represents your attempt to integrate the avail­able evidence and to converge upon the facts of the matter or their tentative inter­pretation. The process is actually an analytic one and is the start of the case study analysis. The format for the answers may be considered analogous to that of a comprehensive “take-home” exam, used in academic courses. You the investigator are the respondent, and your goal is to cite the relevant evidence—whether from interviews, documents, observations, or archival evidence—in composing an ade­quate answer. The main purpose of the open-ended answer is to document the con­nection between specific pieces of evidence and various issues in the case study, generously using footnotes and citations.

The entire set of answers can be considered part of the case study database. You, along with any other interested party, can then use this database to com­pose the actual case study report. Or, if no reports are composed concerning the individual cases (see Chapter 6 for such situations), the answers can serve as the database for the subsequent cross-case analysis. Again, because the answers are part of the database and not of the final report, you should not spend much time trying to make the answers presentable. In other words, you need not per­form the standard editing and copyediting chores. (However, for an example of a case study that was written entirely in the form of narrative answers to the protocol questions and in which such editing was done, see Yin 2003, chap. 2.) The most important attribute of good answers is that they indeed connect the pertinent issues—through adequate citations—to specific evidence.

EXERCISE 4.4 Practicing the Development of a Database

For the topic you covered in Exercise 4.3, write a short report (no more than two double-spaced pages) that adheres to the following outline: Start the report by stating a major question you were attempting to answer (about the facts of the incident recalled from your everyday life). Now provide the answer, citing the evidence you had used (your format should include formal citations and foot­notes). Repeat the procedure for another research question (or the questions from your hypothetical case study protocol). Envisage how this question-and- answer sequence might be one of many in your total case study “database.”

3. Principle 3: Maintain a Chain of Evidence

Another principle to be followed, to increase the reliability of the informa­tion in a case study, is to maintain a chain of evidence. Such a principle is based on a notion similar to that used in forensic investigations.

The principle is to allow an external observer—in this situation, the reader of the case study—to follow the derivation of any evidence from initial research questions to ultimate case study conclusions (see Figure 4.3). Moreover, this external observer should be able to trace the steps in either direction (from con­clusions back to initial research questions or from questions to conclusions). As with criminological evidence, the process should be tight enough that evidence presented in “court”—the case study report—is assuredly the same evidence that was collected at the scene of the “crime” during the data collection process.

Conversely, no original evidence should have been lost, through carelessness or bias, and therefore fail to receive appropriate attention in considering the “facts” of a case. If these objectives are achieved, a case study also will have addressed the methodological problem of determining construct validity, thereby increasing the overall quality of the case study.

Imagine the following scenario. You have read the conclusions in a case study report and want to know more about the basis for the conclusions. You therefore want to trace the evidentiary process backward.

First, the report itself should have made sufficient citation to the relevant portions of the case study database—for example, by citing specific docu­ments, interviews, or observations. Second, the database, upon inspection, should reveal the actual evidence and also indicate the circumstances under which the evidence was collected—for example, the time and place of an interview. Third, these circumstances should be consistent with the specific procedures and questions contained in the case study protocol, to show that the data collection had followed the procedures stipulated by the protocol. Finally, a reading of the protocol should indicate the link between the content of the protocol and the initial study questions.

In the aggregate, you have therefore been able to move from one part of the case study process to another, with clear cross-referencing to methodological procedures and to the resulting evidence. This is the ultimate “chain of evidence” that is desired.

Source: Yin K Robert (2008), Case Study Research Designs and Methods, SAGE Publications, Inc; 4th edition.