Selecting and Managing Documents in Content Analysis

1. Using All the Documents

Even though the population of documents may seem conceptually clear, assembling them for coding generally has three problems: missing documents, inappropriate documents, and uncodable documents. There may be a discrepancy between the supposed population of documents and those actually located. For example, in an evaluation of international development projects in existence over a 10-year period, the documents sought were project evaluation reports, but reports could not be found for all the projects. When documents are missing even after a persistent search, evaluators should note the probable reasons before proceeding with the content analysis. When substantial numbers of documents are missing, the content analysis must be abandoned.

An inappropriate document is one that does not match the definition of document required for the analysis. Almost inevitably, upon inspection, some documents prove inappropriate for the content analysis. For example, in the international development study, some reports that had been labeled and indexed as project evaluation reports did not actually fit that description. Inappropriate documents should be discarded but a record should be kept of the reasons.

Some documents might match the requirements of the analysis but turn out to be uncodable. For example, missing pages or ambiguity of content raise such severe doubts about the quality of the data that it would be better not to include such documents in the analysis.

Once the set of working documents has been determined, the person in charge of coding should record each document in a log. Each document should be given a unique number, and as the coding proceeds, the following minimal information should be recorded: the coder it was assigned to, the date it was coded, and unusual problems.

2. Using a Sample of Documents

When the documents to be coded are a sample of a population, the sample should be chosen from the working population identified in the procedures above. See chapter 3 on selecting material for analysis for some of the sampling considerations.

Source: GAO (2013), Content Analysis: A Methodology for Structuring and Analyzing Written Material: PEMD-10.3.1, BiblioGov.