Identifying Goals and Research Questions for Meta-Analysis

In providing a taxonomy of literature reviews (see Chapter 1), Cooper (1988, 2009a) identified the goals of a review to be one of the dimensions on which reviews differ. Cooper identified integration (including drawing generaliza­tions, reconciling conflicts, and identifying links between theories of dis­ciplines), criticism, and identification of central issues as general goals of reviewers. Cooper noted that the goal of integration “is so pervasive among reviews that it is difficult to find reviews that do not attempt to synthesize works at some level” (1988, p. 108). This focus on integration is also central to meta-analysis, though you should not forget that there is room for addi­tional goals of critiquing a field of study and identifying key directions for future conceptual, methodological, and empirical work. Although these goals are not central to meta-analysis itself, a good presentation of meta-analytic results will usually inform these issues. After reading all of the literature for a meta-analysis, you certainly should be in a position to offer informed opin­ions on these issues.

Considering the goal of integration, meta-analyses follow one of two¹ general approaches: combining and comparing studies. Combining studies involves using the effect sizes from primary studies to collectively estimate a typical effect size, or range of effect sizes. You will also typically make inferences about this estimated mean effect size in the form of statistical significance testing and/or confidence intervals. I describe these methods in Chapters 8 and 10. The second approach to integration using meta-analysis is to compare studies. This approach requires the existence of variability (i.e., heterogeneity) of effect sizes across studies, and I describe how you can test for heterogeneity in Chapter 8. If the studies in your meta-analysis are het­erogeneous, then the goal of comparison motivates you to evaluate whether effect sizes found in studies systematically differ depending on coded study characteristics (Chapter 4) through meta-analytic moderator analyses (Chap­ter 9).

We might think of combination and comparison as the “hows” of meta­analysis; if so, we still need to consider the “whats” of meta-analysis. The goal of meta-analytic combination is to identify the average effect sizes, and meta­analytic comparison evaluates associations between these effect sizes and study characteristics. The common component of both is the focus on effect sizes, which represent the “whats” of meta-analysis. Although many different types of effect sizes exist, most represent associations between two variables (Chapter 5; see Chapter 7 for a broader consideration). Despite this simplicity, the methodology under which these two-variable associations were obtained is critically important in determining the types of research questions that can be answered in both primary and meta-analysis. Concurrent associations from naturalistic studies inform only the degree to which the two variables co-occur. Across-time associations from longitudinal studies (especially those controlling for initial levels of the presumed outcome) can inform temporal primacy, as an imperfect approximation of causal relations. Associations from experimental studies (e.g., association between group random assignment and outcome) can inform causality to the extent that designs eliminate threats to internal validity. Each of these types of associations is represented as an effect size in the same way in a meta-analysis, but they obviously have different implications for the phenomenon under consideration. It is also worth noting here that a variety of other effect sizes index very different “whats,” including means, proportions, scale reliabilities, and longitudinal change scores; these possibilities are less commonly used but represent the range of effect sizes that can be used in meta-analysis (see Chapter 7).

Crossing the “hows” (i.e., combination and comparison) with the “whats” (i.e., effect sizes representing associations from concurrent naturalistic, lon­gitudinal naturalistic, quasi-experimental, and experimental designs, as well as the variety of less commonly used effect sizes) suggests the wide range of research questions that can be answered through meta-analysis. For exam­ple, you might combine correlations between X and Y from concurrent natu­ralistic studies to identify the best estimate of the strength of this association. Alternatively, you might combine associations between a particular form of treatment (as a two-group comparison receiving versus not receiving) and a particular outcome, obtained from internally valid experimental designs, to draw conclusions of how strongly the treatment causes improvement in functioning. In terms of comparison, you might evaluate the extent to which X predicts later Y in longitudinal studies of different duration in order to evaluate the time frame over which prediction (and possibly causal influence) is strongest. Finally, you might compare the reliabilities of a particular scale across studies using different types of samples to determine how useful this scale is across populations. Although I could give countless other examples, I suspect that these few illustrate the types of research questions that can be answered through meta-analysis. Of course, the particular questions that are of interest to you are going to come from your own expertise with the topic; but considering the possible crossings between the “hows” (combination and comparison) and the “whats” (various types of effect sizes) offers a useful way to consider the possibilities.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.