The Potential for Coding Error in Content Analysis

The four interrelated potential sources of coding inaccuracy in most applications of content analysis are (1) deficiencies in the documents, (2) ambiguity in the judgment process, (3) coder bias, and (4) coder error. (Orwin, 1994) For example, a poorly written document may lead to a coder’s making ambiguous decisions, or ambiguity in the judgment process may set the stage for coder bias.

1. Deficiencies in the Documents

If a document is vague, the coder may become uncertain and make mistakes. Deficiencies in the original documents cannot usually be remedied, but coding conventions can help achieve coder consistency. For example, ambiguity in a Stars and Stripes article about weaponry may lead a coder to doubt whether to code it as a “strategic issue” or to not code it because it is really about tactics. In this case, the evaluators would do well to establish coding conventions in the coding manual and to address them during coder training.

2. Ambiguity in Making Judgments

In all but the most straightforward of variables, coders have to exercise judgment, and judgment opens the door for error. For example, in a study of the evaluation reports from international aid projects, coders used a five-point scale to rate the extent to which the objectives of the various aid projects had been met. At first, short phrases defined the points on the scale; at the highest level, for example, objectives were “fully achieved” or “almost fully achieved.” Practice coding sessions  revealedinconsistencies among the coders, so some coders suggested that a numerical scale would be better-objectives were “90- to 100-percent achieved”-but some inconsistency still occurred .. When a third scale provided both word and numeric definitions, the result was coder consistency at the necessary level. The changes to the coding instrument and the training probably both contributed to this improvement.

3. Coder Bias

It is hard to imagine a topic about which a coder would have no preconceptions. As Orwin notes, “Ambiguities in the judgment process and coder bias are related in that ambiguity creates a hospitable environment for bias to creep in unnoticed” (1994, p.142). Training helps coders stay on guard against unintentional bias, and the trainers may be able to spot coders whose bias is intentional. It also helps if documents are assigned to coders randomly.

4. Coder Error

Coders are bound to occasionally apply the coding criteria incorrectly or just write down the wrong code. Such error can be system_atic, tending to favor or disfavor certain categories, or merely random. Wise choices in constructing category labels can help avoid such mistakes, as can proper training.

4.1. Intercoder Reliability

Consistency is often referred to as “intercoder reliability.” It means the degree to which different coders assign the same codes to segments of text. Much inconsistency can generate misleading data. In many circumstances, evaluators can make numerical estimates of intercoder reliability and use the results to judge the readiness of coders to proceed from training to actual coding (see appendix III). To check intercoder reliability during practice, either coders should examine the same documents or else a subset of the documents should be the same for all coders.

4.2. Systematic Error

Even when coders are relatively consistent from one to another, coding can still produce systematic error: the coders as a group tend to make the same errors in assigning category codes to segments of text. In general, gauging the extent of systematic error is more difficult than checking intercoder reliability because it implies that someone knows the “true” codes for text segments. No one in fact has such knowledge. However, evaluators may be able to detect gross levels of systematic error during training and then redefine the variables’ categories and modify the coding manual.

Source: GAO (2013), Content Analysis: A Methodology for Structuring and Analyzing Written Material: PEMD-10.3.1, BiblioGov.