Practical Matters: When (and How) to Correct: Conceptual, Methodological, and Disciplinary Considerations in Meta-Analysis

1. General considerations As I described earlier, one consideration in deciding whether to correct for artifacts is the expected magnitude of effects these artifacts have on the results. Given the numerous artifact adjustments described in the previous section, you might reasonably choose to correct only for those that seem most pressing within the primary

Describing Single Variables in Meta-Analysis

There are relatively few instances of meta-analyzing single variables, yet this information could be potentially valuable. At least three types of information regarding single variables could be important: (1) the mean level of individu­als on a continuous variable; (2) the proportions of individuals falling into a particular category of a categorical variable; and (3)

When the Metric Is Meaningful: Raw Difference Scores

Paralleling the situation when you might want to meta-analyze means and standard deviations—that is, when included studies share a common (or comparable) scale for variable X—there may also be instances when we are interested in comparing two groups on a variable measured on a common scale across studies. For example, studies may all compare

Regression Coefficients and Similar Multivariate Effect Sizes in Meta-Analysis

1. Regression Coefficients In many areas of study, researchers are interested in associations of one variable (X), with another variable (Y) controlling for other variables (Zs). For example, education researchers might wish to understand the relation between ethnicity and academic success, controlling for SES. Or a develop­mental researcher might be interested in whether (and

Miscellaneous Effect Sizes in Meta-Analysis

As I hope is becoming increasingly clear, you can include a wide range of options for effect sizes in your meta-analyses. Although this section on miscellaneous effect sizes could include dozens of possibilities, I limit my description to two that seem especially useful: scale internal consistency and longitudinal change scores. 1. Scale Internal Reliability

Practical Matters: The Opportunities and Challenges of Meta-Analyzing Unique Effect Sizes

1. The Challenges of Meta-Analyzing unique Effect Sizes Meta-analyzing unique effect sizes carries a number of challenges. In this section, I describe some challenges to meta-analyzing unique effect sizes. These challenges apply not only to the effect sizes I have described in this chapter, but to a nearly unlimited range of other advanced effect

1 Comments

The Logic of Weighting

Although the democratic process of giving equal weight to each study has some appeal, the reality is that some studies provide better effect size esti­mates than others, and therefore should be given more weight than others in aggregating results across studies. In this section, I describe the logic of using different weights based on

Measures of Central Tendency in Effect Sizes

1. Choices of Indices of Central Tendency Turning momentarily away from the topic of weighting, I now consider the ways in which you can represent the central tendency of effect sizes from a series of studies in your meta-analysis. As with representing central tendency within a primary data analysis, you can consider the mode,

Inferential Testing and Confidence Intervals of Average Effect Sizes

The key to making inferences regarding statistical significance about, or computing confidence intervals around, this (weighted) mean effect size is to compute a standard error of estimate. Here, I am referring to the standard error of estimating the overall, average effect size, as opposed to the standard error of effect size estimates from each

Evaluating Heterogeneity among Effect Sizes

In Figure 8.1, all of the studies had confidence intervals that contained the vertical line representing the overall population effect size. This situation is called homogeneity—most of the studies capture a common population effect size, and the differences that do exist among their point estimates of effect sizes (i.e., the circles in Figure 8.1)

Practical Matters: Nonindependence among Effect Sizes

An important qualifier to the analyses I have described in this chapter (and those I will describe in subsequent chapters) is that they should be per­formed with a set of independent effect sizes. In primary data analysis, it is well known that a critical assumption is of independent observations; that each case (e.g., person)

Categorical Moderators in meta-analysis

1. Evaluating the Significance of a categorical Moderator The logic of evaluating categorical moderators in meta-analysis parallels the use of ANOVA in primary data analysis. Whereas ANOVA partitions variability in scores across individuals (or other units of analysis) into variability existing between and within groups, categorical moderator analysis in meta-analysis partitions between-study heterogeneity into

Continuous Moderators in meta-analysis

Continuous moderators in meta-analysis are coded study variables that can be considered to vary along a continuum of possible values. For example, mean characteristics of the sample (age, SES, percentage of ethnic minorities, percentage male or percentage female) or methodology (e.g., dose of a drug, number of therapy sessions in intervention) might be evaluated

A General Multiple Regression Framework for Moderation in meta-analysis

After considering the regression approach to analyzing continuous modera­tors (previous section), you are probably wondering whether this approach allows for evaluation of multiple moderators—it does. However, before con­sidering inclusion of multiple moderators, I think it is useful to take a step back to consider how a regression approach can serve as a general approach

1 Comments

An Alternative SEM Approach for Moderation in meta-analysis

Cheung (2008) described an approach to meta-analysis within an SEM frame­work that can be used for moderator analyses as described in this chapter, as well as estimating fixed-effects means as described in Chapter 8 and more complex models (random- and mixed-effects models) described in Chapter 10. You should be aware that this is not

Practical Matters: The Limits of Interpreting Moderators in Meta-Analysis

Notwithstanding the considerable flexibility of a regression framework and the SEM approach for moderator analysis in meta-analysis, you should con­sider three potential limits when drawing conclusions from moderator analy­ses. 1. Empirically confounded Moderators Just as you want to avoid highly correlated predictors in a multiple regression analysis of primary data, it is important to

1 Comments

Differences among fixed-, random-, and Mixed-Effects Models

It is easiest to begin with the simple case in which you are interested only in the mean effect size among a set of studies, both in identifying the mean effect size and in computing its standard errors for inferential testing or for computing of confidence intervals. Even in this simple case, there are

Analyses of Random-Effects Models

A random-effects model in meta-analysis can be estimated in four general steps: (1) estimating the heterogeneity among effect sizes, (2) estimating pop­ulation variability in effect sizes, (3) using this estimate of population vari­ability to provide random-effects weights of study effect sizes, and (4) using these random-effects weights to estimate a random-effects mean effect size

Mixed-Effects Models

Mixed-effects models, sometimes called conditionally random models, com­bine the (fixed-effects) moderator analyses of Chapter 9 with the estimation of variance in population effect sizes (random-effects) described earlier in this chapter. These models are useful when you want to evaluate moderators in meta-analysis, and you (1) either want the generalizability provided by random-effects models, or

A Structural Equation Modeling Approach to Random- and Mixed-Effects Models

In Chapter 9, I introduced an alternative approach to meta-analysis based on Cheung’s (2008) description of meta-analysis within the context of struc­tural equation modeling. Here, I extend the logic of this approach to describe how it can be used to estimate random- and mixed-effects models (follow­ing closely the presentation by Cheung, 2008). As when