Practical Matters: Creating an Organized Protocol for Coding in Meta-Analysis

Once you have decided what study characteristics to code, the next step is to plan to code them (likely coding effect sizes at the same time, as described in Chapter 5). The guidance for this coding comes from a coding protocol, which consists of both the interface coders used to record information from the

1 Comments

The Common Metrics in Meta-Analysis: Correlation, Standardized Mean Difference, and Odds Ratio

1. Significance Tests Are Not Effect Sizes Before describing what effect sizes are, I describe what they are not. Effect sizes are not significance tests, and significance tests are not effect sizes. Although you can usually derive effect sizes from the results of significance tests, and the magnitude of the effect size influences the

Computing r from Commonly Reported Results

You can compute r from a wide range of results reported in primary studies. In this section, I describe how you can compute this effect size when primary studies report correlations, inferential statistics (i.e., £-tests or F-ratios from group comparisons, x2 from analyses of contingency tables), descriptive data, and probability levels of inferential tests.

Computing g from Commonly Reported Results

As when computing r, you can compute standardized mean differences from a wide range of commonly reported information. Although I have presented three different types of standardized mean differences (g, d, and gGlass), I describe only the computation of g in detail in the following. If you are inter­ested in using gGlass as the effect

Computing o from Commonly Reported Results

When you are interested in computing the odds ratio (o, sometimes denoted by OR), or the association between two dichotomous variables, the range of typically reported data is usually more limited than that described in the pre­vious two sections. In this section, I describe computing an odds ratio from three common situations: studies reporting

1 Comments

Comparisons among r, g, and o

I have emphasized the importance of basing the decision to rely on r, g, or o on conceptualizations of the association involving two continuous vari­ables, a dichotomous and a continuous variable, or two dichotomous vari­ables, respectively. At the same time, it can be useful to understand that you can compute values of one effect

Practical Matters: Using Effect Size Calculators and Meta-Analysis Programs

As I described in Chapter 1, several computer programs are designed to aid in meta-analysis, some of which are available for free and others for purchase. All meta-analytic programs perform two major steps: effect size calculation and effect size combination. Effect size combination (as well as comparison) is the process of aggregating results across

The Controversy of Correction to effect Sizes in Meta-Analysis

There is some controversy about correcting effect sizes used in meta-analyses for methodological artifacts. In this section I describe arguments for and against correction, and then attempt to reconcile these two positions. 1. Arguments for Artifact Correction Probably the most consistent advocates of correcting for study artifacts are John Hunter (now deceased) and Frank

Artifact Corrections to Consider in Meta-Analysis

Hunter and Schmidt (2004; see also Schmidt, Le, & Oh, 2009) suggest several corrections to methodological artifacts of primary studies. These corrections involve unreliability of measures, poor validity of measured variables, arti­ ficial dichotomization of continuous variables, and range restriction of vari­ables. Next I describe the conceptual justification and computational details of each of

Practical Matters: When (and How) to Correct: Conceptual, Methodological, and Disciplinary Considerations in Meta-Analysis

1. General considerations As I described earlier, one consideration in deciding whether to correct for artifacts is the expected magnitude of effects these artifacts have on the results. Given the numerous artifact adjustments described in the previous section, you might reasonably choose to correct only for those that seem most pressing within the primary

Describing Single Variables in Meta-Analysis

There are relatively few instances of meta-analyzing single variables, yet this information could be potentially valuable. At least three types of information regarding single variables could be important: (1) the mean level of individu­als on a continuous variable; (2) the proportions of individuals falling into a particular category of a categorical variable; and (3)

When the Metric Is Meaningful: Raw Difference Scores

Paralleling the situation when you might want to meta-analyze means and standard deviations—that is, when included studies share a common (or comparable) scale for variable X—there may also be instances when we are interested in comparing two groups on a variable measured on a common scale across studies. For example, studies may all compare

Regression Coefficients and Similar Multivariate Effect Sizes in Meta-Analysis

1. Regression Coefficients In many areas of study, researchers are interested in associations of one variable (X), with another variable (Y) controlling for other variables (Zs). For example, education researchers might wish to understand the relation between ethnicity and academic success, controlling for SES. Or a develop­mental researcher might be interested in whether (and

Miscellaneous Effect Sizes in Meta-Analysis

As I hope is becoming increasingly clear, you can include a wide range of options for effect sizes in your meta-analyses. Although this section on miscellaneous effect sizes could include dozens of possibilities, I limit my description to two that seem especially useful: scale internal consistency and longitudinal change scores. 1. Scale Internal Reliability

Practical Matters: The Opportunities and Challenges of Meta-Analyzing Unique Effect Sizes

1. The Challenges of Meta-Analyzing unique Effect Sizes Meta-analyzing unique effect sizes carries a number of challenges. In this section, I describe some challenges to meta-analyzing unique effect sizes. These challenges apply not only to the effect sizes I have described in this chapter, but to a nearly unlimited range of other advanced effect

1 Comments

The Logic of Weighting

Although the democratic process of giving equal weight to each study has some appeal, the reality is that some studies provide better effect size esti­mates than others, and therefore should be given more weight than others in aggregating results across studies. In this section, I describe the logic of using different weights based on

Measures of Central Tendency in Effect Sizes

1. Choices of Indices of Central Tendency Turning momentarily away from the topic of weighting, I now consider the ways in which you can represent the central tendency of effect sizes from a series of studies in your meta-analysis. As with representing central tendency within a primary data analysis, you can consider the mode,

Inferential Testing and Confidence Intervals of Average Effect Sizes

The key to making inferences regarding statistical significance about, or computing confidence intervals around, this (weighted) mean effect size is to compute a standard error of estimate. Here, I am referring to the standard error of estimating the overall, average effect size, as opposed to the standard error of effect size estimates from each

Evaluating Heterogeneity among Effect Sizes

In Figure 8.1, all of the studies had confidence intervals that contained the vertical line representing the overall population effect size. This situation is called homogeneity—most of the studies capture a common population effect size, and the differences that do exist among their point estimates of effect sizes (i.e., the circles in Figure 8.1)

Practical Matters: Nonindependence among Effect Sizes

An important qualifier to the analyses I have described in this chapter (and those I will describe in subsequent chapters) is that they should be per­formed with a set of independent effect sizes. In primary data analysis, it is well known that a critical assumption is of independent observations; that each case (e.g., person)