Dimensions of Literature Reviews, Revisited in writing Meta-Analytic Results

Before I turn to specific recommendations for writing the results of your meta-analysis, it is important for you to recognize that there is no single “right” way to write these results. As I described in Chapter 1 (see also Coo­per, 1988), literature reviews vary along several dimensions. Before you begin to write the results of your meta-analysis, you should have a clear under­standing of the goals, organization, and audience for this report.

1. Goals of Meta-Analysis

You began the meta-analysis with some goal in mind, and it is important that you keep this goal in mind as you write your report. As I described in Chap­ter 2 (see also Cooper, 1988), the goal of conducting a meta-analytic review (indeed, most literature reviews) is usually that of integration. However, this general goal of integration entails at least two subgoals (see Cooper, 1988).

One aspect of integration is generalization from specific instances. For example, the example meta-analysis I have described throughout this book (involving the association between relational aggression and peer rejection) relied on a number of studies, each specific in terms of age of the sample, method of measuring relational aggression, and a number of other features. By combining results (Chapters 8 and 10) across these specific instances (studies), it is possible to make statements that are more generalized, albeit within the bounds of the population defined by the studies represented in the meta-analysis. This generalization is not made uncritically, however. Through the comparison of studies (i.e., moderator analyses; Chapter 9) that differ on conceptually relevant characteristics, it is possible to empirically evaluate where findings can (absence of moderation) and cannot (presence of moderation) be generalized.

A second aspect of integration involves the resolution of conflicting find­ings or conclusions. Often, conflicting conclusions come from only seemingly conflicting findings from the Null Hypothesis Significance Testing (NHST) Framework, as I illustrated in Chapter 5. In these cases, meta-analysis, which focuses on effect sizes across studies rather than conclusions regarding sta­tistical significance, typically provides considerable clarity. In other cases, conflicting findings (and resulting conflicting conclusions) might not really be conflicting, but simply due to sampling fluctuations. Here, formal tests of heterogeneity of effect sizes (Chapter 8) will provide clearer conclusions about whether findings are truly conflicting. Finally, results might truly be conflicting (effect sizes are heterogeneous); here, meta-analytic results still have much to offer. One approach would be to accept this conflicting evi­dence (i.e., heterogeneous effect sizes), yet still offer the best generalizable answer through random-effects models (Chapter 8). Alternatively, you might use meta-analytic approaches to go beyond the existence of conflicting find­ings (i.e., reporting the random-effects mean) to evaluate the sources of con­flicting findings through moderator analyses (Chapter 9).¹

Although the goal of your meta-analysis likely involves one or both of these aspects of integration, this does not have to be your only goal in writ­ing the results of your review. Other goals of literature reviews include (1) critiquing the body of research that you have reviewed and (2) identifying key directions for future conceptual, methodological, and empirical work (see Chapter 2 and Cooper, 1988). Although neither of these goals is directly met by the techniques of meta-analysis, they are certainly goals that you, the author (and the person who has just carefully studied the available litera­ture), can certainly address in your writing.

2. Organization of the Meta-Analysis

The results of simple meta-analyses (i.e., those reporting only mean effect sizes and a limited number of moderators analyses from a single sample of studies) have less flexibility as to how they can be organized. However, more complex meta-analytic reviews (i.e., those with many moderator analyses or those comprised of several discrete meta-analyses of different samples of empirical literature) can be organized in various ways. Cooper (1988) stated that literature reviews are commonly organized in three ways: his­torically (i.e., studies reviewing the progress of a field of study across time), conceptually (i.e., studies addressing a common idea or question are orga­nized together), or methodologically (i.e., studies with similar methodologi­cal or measurement approaches are organized together). Although each of these organizational approaches is an option, you are most likely to organize the results of your meta-analytic review either conceptually or methodologi­cally. To illustrate a conceptual organization, the manuscript containing the example meta-analytic review I have used throughout this book (Card et al., 2008) reported results of eight separate meta-analyses: one meta-analysis investigating gender differences in relational aggression, a second meta­analysis investigating the association of relational aggression with overt forms of aggression, and six smaller meta-analysis investigating associations of relational aggression with six distinct adjustment correlates. To illustrate a methodological organization, a meta-analysis might separately report results of concurrent naturalistic, longitudinal naturalistic, and experimental stud­ies of a particular effect.

3. Audience for the Meta-Analysis

Given that I have characterized the writing of your meta-analysis as “pre­senting the results to the world,” it makes sense that you would want to have in mind who is in that world—in other words, your intended audience. The potential audience for meta-analyses varies in terms of both their knowledge of the topic you have focused on and their familiarity with meta-analytic tech­niques. Scientists specializing in the area of your review are likely familiar with the terminology and theoretical perspectives, so they typically need less introduction and guidance in these areas (though you should not neglect this entirely). However, they may be unfamiliar with meta-analytic techniques, depending on the prevalence of meta-analyses in your particular field. Scien­tists outside of your specialized area will need more introduction to the topic area and may or may not be familiar with meta-analytic techniques. Practi­tioners, policymakers, and educated laypeople will almost universally need more didactic explanation of your topic and meta-analytic techniques.

Complicating matters even further, it is likely that your presentation will reach multiple audiences. If you decide that the only readers you care to inform are specialists in your field who are familiar with meta-analysis—and you write your report only for this audience—you should realize that you are probably targeting a very small audience, and the likelihood that your report will be published in a widely read outlet is small. Even if you decide to target a broader range of scientists within your field, you should recognize that others (e.g., educators, practitioners, policymakers) may read your report. Therefore, you are diminishing the potential impact of your review if it is not accessible to a broader audience of readers.

Conversely, you should be aware that some of the details that can be confusing and intimidating to readers unfamiliar with meta-analysis would be the very details that some readers (those very familiar with meta-analysis) will expect to see. The challenge, then, is to effect a balance between (1) providing enough technical details for content experts familiar with meta­analysis to evaluate your work, versus (2) not overwhelming other readers with too much technical detail. Although this can be a difficult line to walk, and it is likely that you cannot make 100% of readers 100% happy, I do think the following principles can help achieve this balance.

First, ask yourself what you find more discouraging when you read a report: (1) when you simply cannot understand what the authors have done, or (2) the authors provide what seems to be excessive detail of what they have done. My own reaction, and I suspect the reaction of many of you, is that it is better to be bored by too much detail than confused by too little. Following this principle, my suggestion is that it is better to report a potentially impor­tant piece of information than to omit it.

My recommendation that you err on the side of reporting too much rather than too little comes with a corollary: You do not have to report everything in the narrative text of your manuscript. Depending on the editorial style of your publication outlet, it may be preferable to place some details in tables, footnotes, appendices, or supplemental online documents. Doing so allows interested readers to evaluate these details, but does not distract attention for other readers. If space restrictions at your publication outlet preclude these options, then noting that full results are available upon request (and then providing them upon request) is an option.

My third recommendation is to write at multiple levels. What I mean by writing at multiple levels is that your text has pieces that make it under­standable to audiences with a broad range of background in your topic and in meta-analysis. How you accomplish this is to provide a clear, jargon-free statement that is understandable to a broad audience in tandem with more technical details. For example, technical details can be placed in parenthe­ses, as in the following: “Associations between relational aggression and peer rejection are stronger among studies using peer reports of relational aggression than those using observations (mean rs = .34 versus .09, X²(df=1) = 21.05, p < .001).” Similarly, you might ensure that each paragraph con­taining technical information consists of (1) a clear first sentence of what you evaluated, (2) one or more sentences reporting the detailed (techni­cal) results, and (3) a clear final sentence or two stating what you found in jargon-free terms. I do not intend these to be absolute rules; rather they are my own suggestions for accomplishing the difficult task of writing at multiple levels.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.