Developing and Articulating a Sampling Frame for Meta-Analysis

Given that meta-analysis uses the individual study as its unit of analysis, it is useful to think of your meta-analysis as consisting of a sample of studies, just as primary analyses sample people or other units (e.g., families, businesses) comprising its sample. In primary analyses, we typically wish to make infer­ences to a larger

Inclusion and Exclusion Criteria for Meta-Analysis

The inclusion criteria, and conversely the exclusion criteria, are a set of explicit statements about the features of studies that will or will not (respectively) be included in your meta-analysis. Ideally, you should specify these criteria before searching the literature so that you can then determine whether each study identified in your search should

Finding Relevant Literature for Meta-Analysis

After specifying inclusion/exclusion criteria, the next step is to begin search­ing for empirical studies that fit within this sampling frame. In searching for this relevant literature, you have many options, each with advantages and limitations over the others. Although it is not always necessary to use all of the options I list next, it

Reality Checking: Is My Meta-Analysis Search Adequate?

Regardless of what methods of searching the literature you rely upon, the most important question is whether your search is adequate. You can think of the adequacy of your search in three ways. First, is the sample of studies you have obtained representative of the population of studies, or is it instead biased (as

Practical Matters: Beginning a Meta-Analytic Database

Aside from perhaps persistence and patience, the most import virtue you can have for searching the literature for a meta-analysis is organization. As you have likely inferred, searching for studies is a time-intensive process, and you certainly do not want to add to this time by repeating work because of poor organization. A good

Identifying Interesting Moderators in Meta-Analysis

Decisions about which study characteristics to code need to be heavily informed by your knowledge of the content area in which you are performing a meta-analytic review. Nevertheless, I describe two sets of general consider­ations that I believe apply to meta-analytic reviews across fields: considering the research questions you are interested in and considering

Coding Study “Quality” in Meta-Analysis

Some have recommended that meta-analysts code for study quality and then either (1) include only studies meeting a certain level of quality or (2) evalu­ate quality as a moderator of effect sizes.2 This recommendation is problem­atic, in my view, because it assumes (1) that “quality” is a unidimensional construct and (2) that we are

Evaluating Coding Decisions in Meta-Analysis

Once you have decided what study characteristics to code, the next step, of course, is to do it—to carefully read obtained reports and to record informa­tion about the studies. The information recorded is that regarding both the study characteristics you have decided to code (see previous two sections) and the effect sizes. I defer

Practical Matters: Creating an Organized Protocol for Coding in Meta-Analysis

Once you have decided what study characteristics to code, the next step is to plan to code them (likely coding effect sizes at the same time, as described in Chapter 5). The guidance for this coding comes from a coding protocol, which consists of both the interface coders used to record information from the

1 Comments

The Common Metrics in Meta-Analysis: Correlation, Standardized Mean Difference, and Odds Ratio

1. Significance Tests Are Not Effect Sizes Before describing what effect sizes are, I describe what they are not. Effect sizes are not significance tests, and significance tests are not effect sizes. Although you can usually derive effect sizes from the results of significance tests, and the magnitude of the effect size influences the

Computing r from Commonly Reported Results

You can compute r from a wide range of results reported in primary studies. In this section, I describe how you can compute this effect size when primary studies report correlations, inferential statistics (i.e., £-tests or F-ratios from group comparisons, x2 from analyses of contingency tables), descriptive data, and probability levels of inferential tests.

Computing g from Commonly Reported Results

As when computing r, you can compute standardized mean differences from a wide range of commonly reported information. Although I have presented three different types of standardized mean differences (g, d, and gGlass), I describe only the computation of g in detail in the following. If you are inter­ested in using gGlass as the effect

Computing o from Commonly Reported Results

When you are interested in computing the odds ratio (o, sometimes denoted by OR), or the association between two dichotomous variables, the range of typically reported data is usually more limited than that described in the pre­vious two sections. In this section, I describe computing an odds ratio from three common situations: studies reporting

1 Comments

Comparisons among r, g, and o

I have emphasized the importance of basing the decision to rely on r, g, or o on conceptualizations of the association involving two continuous vari­ables, a dichotomous and a continuous variable, or two dichotomous vari­ables, respectively. At the same time, it can be useful to understand that you can compute values of one effect

Practical Matters: Using Effect Size Calculators and Meta-Analysis Programs

As I described in Chapter 1, several computer programs are designed to aid in meta-analysis, some of which are available for free and others for purchase. All meta-analytic programs perform two major steps: effect size calculation and effect size combination. Effect size combination (as well as comparison) is the process of aggregating results across

The Controversy of Correction to effect Sizes in Meta-Analysis

There is some controversy about correcting effect sizes used in meta-analyses for methodological artifacts. In this section I describe arguments for and against correction, and then attempt to reconcile these two positions. 1. Arguments for Artifact Correction Probably the most consistent advocates of correcting for study artifacts are John Hunter (now deceased) and Frank

Artifact Corrections to Consider in Meta-Analysis

Hunter and Schmidt (2004; see also Schmidt, Le, & Oh, 2009) suggest several corrections to methodological artifacts of primary studies. These corrections involve unreliability of measures, poor validity of measured variables, arti­ ficial dichotomization of continuous variables, and range restriction of vari­ables. Next I describe the conceptual justification and computational details of each of

Practical Matters: When (and How) to Correct: Conceptual, Methodological, and Disciplinary Considerations in Meta-Analysis

1. General considerations As I described earlier, one consideration in deciding whether to correct for artifacts is the expected magnitude of effects these artifacts have on the results. Given the numerous artifact adjustments described in the previous section, you might reasonably choose to correct only for those that seem most pressing within the primary

Describing Single Variables in Meta-Analysis

There are relatively few instances of meta-analyzing single variables, yet this information could be potentially valuable. At least three types of information regarding single variables could be important: (1) the mean level of individu­als on a continuous variable; (2) the proportions of individuals falling into a particular category of a categorical variable; and (3)

When the Metric Is Meaningful: Raw Difference Scores

Paralleling the situation when you might want to meta-analyze means and standard deviations—that is, when included studies share a common (or comparable) scale for variable X—there may also be instances when we are interested in comparing two groups on a variable measured on a common scale across studies. For example, studies may all compare