Practical Matters: Creating an Organized Protocol for Coding in Meta-Analysis

Once you have decided what study characteristics to code, the next step is to plan to code them (likely coding effect sizes at the same time, as described in Chapter 5). The guidance for this coding comes from a coding protocol, which consists of both the interface coders used to record information from the studies as well as a coding manual providing instructions for this coding process (see Wilson, 2009). Through using this coding protocol, your goal is to create a usable database for later meta-analyses. There are several consid­erations for each aspect, which I describe next.

1. Coding Interface

Considering first the interface coders use to record information, three options include using paper forms that coders complete, using a computer­ized form to collect this information, or coding directly into the electronic format to be used for analyses. Part of an example paper coding form (from a meta-analysis of the association between relational aggression and peer rejection described throughout this book; Card et al., 2008) is shown in Figure 4.2.

Using paper forms would require coders to write information into pre­defined questions (e.g., “Sample age in years:___ ”), which would then be transferred into an electronic database for analyses. The advantages of this approach are (1) that coders need training only in the coding process (guided by the manual of instructions described in Section 4.4.2) rather than proce­dures for entering data into a computer, and (2) the information is checked for plausibility when entered into the computer.

A computerized form would present the same information to coders but would require them to input the coded data electronically, perhaps using a relational database program (e.g., Microsoft Access). This type of interface would require only a small amount of training beyond using paper forms and would reduce the time (and potentially errors) in transferring informa­tion from paper to the electronic format. However, this advantage is also a disadvantage in that it bypasses the check that would occur during this entry from paper forms.

A third option with regard to a coding interface is to code information directly into an electronic format (e.g., Microsoft Excel, SAS, SPSS) later used for analysis. This option is perhaps the most time-efficient of all in reducing the number of steps, but it is also the most prone to errors. I strongly discour­age this third method if multiple coders will be coding studies.

2. Coding Manual

A coding manual is a detailed collection of instructions describing how information reported in research reports is quantified for inclusion in your meta-analysis. Creating a detailed coding manual serves three primary pur­poses. First, this coding manual provides a guide for coders to transfer infor­mation in the study reports to the coding interface (e.g., paper forms). As such, it should be a clear set of instructions for coding both “typical” studies and more challenging coding situations. Second (and relatedly), this coding manual aims to ensure consistency across multiple reporters⁹ by providing a clear, concrete set of instructions that each coder should study and have at hand during the coding process. Third, this coding manual serves as docu­mentation of the coding process that should guide the presentation of the meta-analysis or be made available to others to ensure transparency of the coding (see beginning of this section).

With regard to the coding manual, the amount of instruction for each study characteristic coded depends on the level of inference of the coding: low-inference coding requires relatively little instruction, whereas high-
inference coding requires more instruction. In addition, the coding manual is most often a work in progress. Although an initial coding manual should be developed before beginning the coding, ambiguities discovered during the coding process likely will force ongoing revision.

Turning again to the example coding form of Figure 4.2, we should note that this form would be accompanied by a detailed coding manual that all coders have been trained in and have present while completing this form. To provide illustrations of the type of information that might be included in such a manual, we can consider two of the coded study characteristics. First, item 5 (mean age) might be accompanied by the rather simple instruction “Record the mean age of the sample in years.” However, even this relatively simple (low-inference) code requires fairly extensive elaboration: “If study analyzed a subset of the data, record the mean age of the subset used in analy­ses. If study reported a range of ages but not the mean, record the midpoint of this range.” My colleagues and I also had to change the coding protocol rather substantially when we found that many studies failed to report ages, but did report the grades in school of participants. This led us to add the “grade” code (item 6) along with instructions for entering this information in the database: “If sample age is not reported in the study, then an estimated age can be entered from grade using the formula Age = Grade + 5.”¹⁰ A second study characteristic shown in Figure 4.2 that illustrates typical instruction is item 10 (aggression—source of information). The coding manual for this item specifies the choices that should be coded (self-report, peer nomination, peer rating, teacher report, parent report, researcher observations, or other) and definitions of each code.

3. Database for Meta-Analysis

The product of your coding should be an electronic file with which to con­duct your meta-analysis. Table 4.2 provides an example of what this database might look like (if complete, the table would extend far to the right to include other coded study characteristics, coded effect sizes [Chapter 5], information for any artifact corrections [Chapter 6], and several calculations for the actual meta-analysis [Chapters 8-10]). Although the exact variables (columns) you include will depend on the study characteristics you decide to code, the gen­eral layout of this file sould be considered. Here, each row represents a single coded study, and each column represents a coded study characteristic.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.