Practical Matters: When (and How) to Correct: Conceptual, Methodological, and Disciplinary Considerations in Meta-Analysis

1. General considerations

As I described earlier, one consideration in deciding whether to correct for artifacts is the expected magnitude of effects these artifacts have on the results. Given the numerous artifact adjustments described in the previous section, you might reasonably choose to correct only for those that seem most pressing within the primary studies being synthesized.

How pressing a particular type of artifact is within a meta-analysis is partly a conceptual question and partly an empirical question. First, you must consider the collection of primary studies in light of your conceptual expertise of the area. Relevant questions include the following: How valid are the measures within this research in relation to the construct I am inter­ested in? How representative are the samples relative to the population about which I want to draw conclusions? Again, there is not a statistical answer to such questions; rather, these questions must be answered based on your understanding of the field.

In addition to conceptual considerations, you might also base conclu­sions on empirical grounds. Specifically, you can consider the data reported in primary studies to draw conclusions about the presence of important arti­facts. For example, I recommend coding the internal consistencies of relevant measures within the primary studies, meta-analyzing these reliabilities (see Chapter 7), and determining (1) whether the collection of studies has gener­ally high or low reliabilities of measures and (2) whether substantial variabil­ity exists across studies in these reliabilities. Similarly, if many studies use similar measures of a variable (i.e., with the same scale), then you could code and evaluate standard deviations across studies (see Chapter 7) to determine whether some studies suffer from restricted ranges. In short, for each of the potential artifacts described in the previous section, you should consider the available empirical evidence to determine whether this artifact is uniformly or inconsistently present in the primary studies being analyzed. If a particu­lar artifact is uniformly present, then correcting for it will yield more accurate overall effect size estimates (among latent constructs). If a particular artifact is present in some studies but not in others (or present in differing degrees across studies), then correcting for this artifact will reduce less interesting (i.e., artifactual) variability across studies and allow for a clearer picture of substantively interesting variability in effect sizes.

2. Disciplinary Considerations

Whereas I view the conceptual and empirical considerations as most impor­tant in deciding whether and how to correct for artifacts, the reality is that these corrections are more common in some fields than in others. This means that one meta-analyst working within one field might be expected to correct for certain artifacts, whereas another meta-analyst working within another field might be met with skepticism if certain (or any) corrections were to be performed. These disciplinary practices are unfortunate, especially because they are more often due to those who are influential in a field more so than consideration of particular needs of a field. Nevertheless, it is useful to recog­nize the common practices within your particular field.

Notwithstanding recognition of these disciplinary practices, I want to encourage you to not feel restricted by these practices. In other words, do not base your decision to perform or not perform certain artifact corrections only on common practices within your field. Instead, carefully consider the conceptual and empirical basis for making certain corrections, and then use (or not) these corrections to obtain results that best answer your research questions.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.