Selecting Material for Analysis in Content Analysis

To select textual material to include in the content analysis, evaluators may find it easiest to think first about a population of documents. For some assignments, this population may already exist, as in the Stars and Stripes evaluation. For other assignments, evaluators have to collect data into a database. This happened when GAO evaluators used focus groups to obtain responses to food assistance programs on Indian reservations. (GAO, 1990)

1. Defining a Document

A document should be physically separable, minimally sized, and self-contained textual information. A letter is a document. Each daily edition of Stars and Stripes is a document. A file folder is not a document because it contains within it smaller it~ms that are physically separable, some of which are self-contained. A book is somewhat ambiguous as a document. Most books could be considered documents, but an edited book in which each chapter had separate authors might better be thought of as an aggregate of documents. A transcription of an open-ended interview would probably be defined as a document. However, if the scope of the evaluation were limited to responses to just one interview question, then a transcription of just the pertinent answer might be the document. Thus, evaluators have latitude in defining a document. The guiding principle is to let the evaluation’s purpose and needs determine the definition.

2. Choosing a Sampling Method

Sampling is necessary when a document population is too large to be analyzed in its entirety. Two broad options are available, probability sampling and nonprobability sampling. Probability sampling may be the right choice if the evaluation question implies the need to generaliz~ from the sample to the population and if the procedures required for probability sampling are practical UI).der the circumstances. N onprobability sampling, sometimes called judgment or purposeful sampling, may be the right choice if generalization is not necessary or if probability sampling procedures are not practical. Examples of probability sampling and nonprobability sampling are GAO (1992d) and Patton (1990, pp. 169-83),respectively.

In some assignments, multistage sampling is appropriate. For example, in a study of federal personnel actions, one might first select a probability sample of personnel folders-an aggregate of self-contained documents-and then, in the second stage, sample “action” ‘documents within the folders. Sampling a document’s segments may also be useful. For example, in a study of recommendations from GAO reports, we might probabilistically select one recommendation (that is, the recommendation itself plus its supporting material) from each of several reports. Weber (1990) recommends that documents be sampled in their entirety in order to preserve semantic coherence. However, the sampling of segments may be a good strategy when a document contains substantial amounts of material not relevant to the study or when it is desirable to draw information from a large number of lengthy documents.

The Stars and Stripes content analysis used sampling. Since two editions of the paper had been published-dailies in Europe and the Pacific-a reasonable population of documents wolild be all issues of Stars and Stripes published in the decade ending in 1988. (The Congress had made its study request in 1987.) During the decade 1978-88, each edition contained 28 pages, so the document population was much too large to be studied in its entirety.

To reduce the textual material to manageable proportions, the evaluators chose a nonprobabil isticsample of documents. Specifically, they selected all issues of both the Pacific and European editions that ublished in March 1987.2 For content analysis,
they chose only news stories. For comparison purposes, they also identified all AP and UPI stories that dealt with DOD and the U.S. military and with sensitive topics otherwise cited in the allegations of censorship.

Source: GAO (2013), Content Analysis: A Methodology for Structuring and Analyzing Written Material: PEMD-10.3.1, BiblioGov.