Basic Terminology in Meta-Analysis

Before further discussing meta-analysis, it is useful to clarify some relevant terminology. One clarification involves the distinction of meta-analysis from primary or secondary analysis. The second clarification involves terminology of meta-analysis within the superordinate category of a literature review.

1. Meta-Analysis versus Primary or Secondary Analysis

The first piece of terminology to clarify are the differences among the terms “meta-analysis,” “primary analysis,” and “secondary analysis” (Glass, 1976). The term “primary analysis” refers to what we typically think of as data anal- ysis—when a researcher collects data from individual persons, companies, and so on,² and then analyzes these data to provide answers to the research questions that motivated the study. The term “secondary analysis” refers to re-analysis of these data, often to answer different research questions or to answer research questions in a different way (e.g., using alternative analytic approaches that were not available when the data were originally analyzed). This secondary data analysis can be performed either by the original research­ers or by others if they are able to obtain the raw data from the researchers. Both primary and secondary data analysis require access to the full, raw data as collected in the study.

In contrast, meta-analysis involves the statistical analysis of the results from more than one study. Two points of this definition merit consideration in differentiating meta-analysis from either primary or secondary analysis. First, meta-analysis involves the results of studies as the unit of analysis, spe­cifically results in the form of effect sizes. Obtaining these effect sizes does not require having access to the raw data (which are all-too-often unavailable), as it is usually possible to compute these effect sizes from the data reported in papers resulting from the original, primary or secondary, analysis. Second, meta-analysis is the analysis of results from multiple studies, in which indi­vidual studies are the unit of analysis. The number of studies can range from as few as two to as many as several hundred (or more, limited only by the availability of relevant studies). Therefore, a meta-analysis involves drawing inferences from a sample of studies, in contrast to primary and secondary analyses that involve drawing inferences from a sample of individuals. Given this goal, meta-analysis can be considered a form of literature review, as I elaborate next.

2. Meta-Analysis as a Form of Literature Review

A second aspect of terminological consideration involves the place of meta­analysis within the larger family of literature reviews. A literature review can be defined as a synthesis of prior literature on a particular topic. Literature reviews differ along several dimensions, including their focus, goals, per­spective, coverage, organization, intended audience, and method of synthe­sis (see Cooper, 1988, 2009a). Two dimensions are especially important in situating meta-analysis within the superordinate family of literature reviews: focus and method of synthesis. Figure 1.1 shows a schematic representation of how meta-analysis differs from other literature reviews in terms of focus and method of synthesis.

Meta-analyses, like other research syntheses, focus on research out­comes (not the conclusion reached by study authors, which Rosenthal noted are “only vaguely related to the actual results” (1991, p. 13). Reviews focusing on research outcomes answer questions such as “The existing research shows X” or “These types of studies find X, whereas these other types of studies find Y.” Other types of literature reviews have different foci. Theoretical reviews focus on what theoretical explanations are commonly used within a field, attempt to explain phenomena using a novel theoretical alternative, or seek to integrate multiple theoretical perspectives. These are the types of reviews that are commonly reported in, for example, Psychological Review. Survey reviews focus on typical practices within a field, such as the use of particu­lar methods in a field or trends in the forms of treatment used in published clinical trials (e.g., Card & Little, 2007, surveyed published research in child development to report the percentage of studies using longitudinal designs). Although reviews focusing on theories or surveying practices within the lit­erature are valuable contributions to science, it is important to distinguish the focus of meta-analysis on research outcomes from these other types of reviews.

However, not all reviews that focus on research outcomes are meta­analyses. What distinguishes meta-analysis from other approaches to research synthesis is the method of synthesizing findings to draw conclusions. The methods shown in the bottom of Figure 1.1 can be viewed as a continuum from qualitative to quantitative synthesis. At the left is the narrative review. Here, the reviewer evaluates the relevant research and somehow draws con-
elusions. This “somehow” represents the limits of this qualitative, or narra­tive, approach to research synthesis. The exact process of how the reviewer draws eonelusions is unknown, or at least not artieulated, so there is eon- siderable room for subjectivity in the research conclusions reached. Beyond just the potential for subjective bias to emerge, this approach to synthesizing research taxes the reviewer’s ability to process information. Reviewers who attempt to synthesize research results qualitatively tend to perceive more inconsistency and smaller magnitudes of effects than those performing meta­analytic syntheses (Cooper & Rosenthal, 1980). In sum, the most common method of reviewing research—reading empirical reports and “somehow” drawing conclusions—is prone to subjectivity and places demands on the reviewer that make conclusions difficult to reach.

Moving toward the right, or quantitative direction, of Figure 1.1 are two vote-counting methods, which I have termed informal and formal. Both involve considering the significance of effects from research studies in terms of significant positive, significant negative, or nonsignificant results, and then drawing conclusions based on the number of studies finding a particu­lar result. Informal (also called conventional) vote counting involves simply drawing conclusions based on “majority rules” criteria; so, if more studies find a significant positive effect than find other effects (nonsignificant or sig­nificant negative), one concludes that there is a positive effect. A more formal vote-counting approach (see Bushman & Wang, 2009) uses statistical analy­sis of the expected frequency of results given the type I error rates (e.g., Given a traditional type I error rate of .05, do significantly more than 5% of studies find an effect?). Although vote-counting methods can be useful when infor­mation on effect sizes is unavailable, I do not discuss them in this book for two reasons (for descriptions of these vote-counting methods, see Bushman & Wang, 2009). First, conclusions of the existence of effects (i.e., statistical significance) can be more powerfully determined using meta-analytic proce­dures described in this book. Second, conclusions of significance alone are unsatisfying, and the focus of meta-analysis is on effect sizes that provide information about the magnitude of the effect.³

At the right side of Figure 1.1 is meta-analysis, which is a form of research synthesis in which conclusions are based on the statistical analysis of effect sizes from individual studies.⁴ I reserve further description of meta-analysis for the remainder of the book, but my hope here is that this taxonomy makes clear that meta-analysis is only one approach to conducting a literature review. Specifically, meta-analysis is a quantitative method of synthesizing empirical research results in the form of effect sizes. Despite this specific­ity, meta-analysis is a flexible and powerful approach to advancing scientific knowledge, in that it represents a statistically defensible approach to synthe­sizing empirical findings, which are the foundation of empirical sciences.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.