Meta-analysis in management research

"Meta-analysis is the statistical combination of results from two or more separate studies" (Deeks et al, 2019, chapter 10). When the treatment effect (or effect size) is consistent from one study to the next, meta-analysis can be used to identify this common effect. When the effect varies from one study to the next, meta-analysis may be used to identify the reason for the

The Need for Research Synthesis in the Social Sciences

Isaac Newton is known to have humbly explained his success: “If I have seen further it is by standing upon the shoulders of giants” (1675; from Columbia World of Quotations, 1996). Although the history of science suggests that Newton may have been as likely to kick his fellow scientists down as he was to

Basic Terminology in Meta-Analysis

Before further discussing meta-analysis, it is useful to clarify some relevant terminology. One clarification involves the distinction of meta-analysis from primary or secondary analysis. The second clarification involves terminology of meta-analysis within the superordinate category of a literature review. 1. Meta-Analysis versus Primary or Secondary Analysis The first piece of terminology to clarify are

A Brief History of Meta-Analysis

In this section, I briefly outline the history of meta-analysis. My goal is not to be exhaustive in detailing this history (for more extensive treatments, see Chalmers, Hedges, & Cooper, 2002, Hedges, 1992, and Olkin, 1990; for a his­tory intended for laypersons, see Hunt, 1997). Instead, I only hope to provide a basic overview

The Scientific Process of Research Synthesis

Given the importance of research syntheses, including meta-analyses, to the progression of science, it is critical to follow scientific standards in their preparation. Most scientists are well trained in methods and data-analytic techniques to ensure objective and valid conclusions in primary research, yet methods and data-analytic techniques for research synthesis are less well known.

Identifying Goals and Research Questions for Meta-Analysis

In providing a taxonomy of literature reviews (see Chapter 1), Cooper (1988, 2009a) identified the goals of a review to be one of the dimensions on which reviews differ. Cooper identified integration (including drawing generaliza­tions, reconciling conflicts, and identifying links between theories of dis­ciplines), criticism, and identification of central issues as general goals of

The Limits of Primary Research and the Limits of Meta-Analytic Synthesis

Perhaps no statement is more true, and humbling, than this offered as the opening of Harris Cooper’s editorial in Psychological Bulletin (and likely stated in similar words by many others): “Scientists have yet to conduct the flawless experiment” (Cooper, 2003, p. 3). I would extend this conclusion further to point out that no scientist

Critiques of Meta-Analysis: When Are They Valid and When Are They Not?

As I outlined in Chapter 1, attention to meta-analysis emerged in large part with the attention received by Smith and Glass’s (1977) meta-analysis of psy­chotherapy research (though others developed techniques of meta-analysis at about the same time; e.g., Rosenthal & Rubin, 1978; Schmidt & Hunter, 1977). The controversial nature of this meta-analysis drew criticisms,

2.4 Practical Matters: The Reciprocal Relation between Planning and Conducting a Meta-Analysis

My placement of this chapter on identifying research questions for meta­analysis before chapters on actually performing a meta-analysis is meant to correspond to the order you would follow in approaching this endeavor. As with primary research, you want to know your goals and research questions, as well as potential limitations and critiques, of your

1 Comment

Developing and Articulating a Sampling Frame for Meta-Analysis

Given that meta-analysis uses the individual study as its unit of analysis, it is useful to think of your meta-analysis as consisting of a sample of studies, just as primary analyses sample people or other units (e.g., families, businesses) comprising its sample. In primary analyses, we typically wish to make infer­ences to a larger

Inclusion and Exclusion Criteria for Meta-Analysis

The inclusion criteria, and conversely the exclusion criteria, are a set of explicit statements about the features of studies that will or will not (respectively) be included in your meta-analysis. Ideally, you should specify these criteria before searching the literature so that you can then determine whether each study identified in your search should

Finding Relevant Literature for Meta-Analysis

After specifying inclusion/exclusion criteria, the next step is to begin search­ing for empirical studies that fit within this sampling frame. In searching for this relevant literature, you have many options, each with advantages and limitations over the others. Although it is not always necessary to use all of the options I list next, it

Reality Checking: Is My Meta-Analysis Search Adequate?

Regardless of what methods of searching the literature you rely upon, the most important question is whether your search is adequate. You can think of the adequacy of your search in three ways. First, is the sample of studies you have obtained representative of the population of studies, or is it instead biased (as

Practical Matters: Beginning a Meta-Analytic Database

Aside from perhaps persistence and patience, the most import virtue you can have for searching the literature for a meta-analysis is organization. As you have likely inferred, searching for studies is a time-intensive process, and you certainly do not want to add to this time by repeating work because of poor organization. A good

Identifying Interesting Moderators in Meta-Analysis

Decisions about which study characteristics to code need to be heavily informed by your knowledge of the content area in which you are performing a meta-analytic review. Nevertheless, I describe two sets of general consider­ations that I believe apply to meta-analytic reviews across fields: considering the research questions you are interested in and considering

Coding Study “Quality” in Meta-Analysis

Some have recommended that meta-analysts code for study quality and then either (1) include only studies meeting a certain level of quality or (2) evalu­ate quality as a moderator of effect sizes.2 This recommendation is problem­atic, in my view, because it assumes (1) that “quality” is a unidimensional construct and (2) that we are

Evaluating Coding Decisions in Meta-Analysis

Once you have decided what study characteristics to code, the next step, of course, is to do it—to carefully read obtained reports and to record informa­tion about the studies. The information recorded is that regarding both the study characteristics you have decided to code (see previous two sections) and the effect sizes. I defer

Practical Matters: Creating an Organized Protocol for Coding in Meta-Analysis

Once you have decided what study characteristics to code, the next step is to plan to code them (likely coding effect sizes at the same time, as described in Chapter 5). The guidance for this coding comes from a coding protocol, which consists of both the interface coders used to record information from the

1 Comment

The Common Metrics in Meta-Analysis: Correlation, Standardized Mean Difference, and Odds Ratio

1. Significance Tests Are Not Effect Sizes Before describing what effect sizes are, I describe what they are not. Effect sizes are not significance tests, and significance tests are not effect sizes. Although you can usually derive effect sizes from the results of significance tests, and the magnitude of the effect size influences the

Computing r from Commonly Reported Results

You can compute r from a wide range of results reported in primary studies. In this section, I describe how you can compute this effect size when primary studies report correlations, inferential statistics (i.e., £-tests or F-ratios from group comparisons, x2 from analyses of contingency tables), descriptive data, and probability levels of inferential tests.