Practical Matters: The Opportunities and Challenges of Meta-Analyzing Unique Effect Sizes

1. The Challenges of Meta-Analyzing unique Effect Sizes

Meta-analyzing unique effect sizes carries a number of challenges. In this section, I describe some challenges to meta-analyzing unique effect sizes. These challenges apply not only to the effect sizes I have described in this chapter, but to a nearly unlimited range of other advanced effect sizes that we might consider.

One challenge of using unique effect sizes in meta-analysis is that the pri­mary studies might often fail to report the necessary data. When I described basic effect sizes in Chapter 5, I mentioned that these effect sizes are often reported, or else sufficient information to compute such effect sizes are typi­cally reported. In this chapter, I have focused attention on some effect sizes that are likely to be commonly reported (e.g., internal consistency), but this information is still less likely to be reported in all relevant studies. If you are using unique effect sizes (either those I have described here or others), it will be important for you to contact authors of studies that could provide relevant data that are not reported. Often, you will need to give explicit instructions to these authors on how to compute these unique effects, which might be less familiar to researchers than more basic effect sizes (e.g., correlations).

A related challenge involves the inconsistencies in analytic methods and reporting of advanced effect sizes. Earlier in this chapter, I described this challenge when using independent effect sizes, such as regression coefficients or semipartial correlations from multiple regression analyses in which differ­ent studies include different predictors/covariates. We can also imagine how this inconsistency would pose obstacles to the use of other effect sizes. For example, imagine that you wanted to meta-analytically combine results of exploratory factor analyses, such as factor loadings and commonality. If you looked at the relevant literature, you would find tremendous variability in the use of principal components versus true factor analysis models, methods of extraction, the way authors determined the number of factors to extract or interpret, and methods of rotation. Given this diversity, it would be difficult, if not impossible, to attempt to meta-analytically combine these results. This example illustrates the challenge of meta-analyzing unique effect sizes from studies that might vary in their analytic methods and reporting.

As I will discuss further in Chapter 8, meta-analysis of an effect size involves not only obtaining an estimate of that effect size for each study, but also computing a standard error for each effect size estimate for weighting. In other words, it is not enough to simply be able to find sufficient data in the primary study to compute the effect size, but you must also determine the correct formula and find the necessary information in the study to com­pute the standard error. Some readers might agree that the equations just to compute effect sizes are daunting; the formulas to compute standard errors are usually even more challenging and are typically difficult to find in all but the most advanced texts (and in some cases, there is no consensus on what an appropriate standard error is). Furthermore, you typically need more information to calculate the standard errors than the effect sizes, and this information is more often excluded from reports (and more often puzzling to authors if you request this information). In short, you need to remember that, to use an advanced effect size in a meta-analysis, you must be able to compute both its point estimate and its standard error from primary studies.

2. Balancing the Challenges with the Opportunities of Meta-Analyzing Unique Effect Sizes

Although the use of unique effect sizes in meta-analysis poses several chal­lenges, their use also offers several opportunities. Namely, if only unique effect sizes answer the questions you want to answer, then it is worth facing these challenges to answer these questions. How can you weigh the potential reward versus the cost of using unique effect sizes? Although this is a dif­ficult question to answer, I offer some thoughts next.

First, I suggest asking yourself whether the question you want to answer in your meta-analysis (see Chapter 2) really requires reliance on unique effect sizes. Can your question be effectively answered using traditional effect sizes such as r, g, or o? Is it possible that the question you are asking is similar to one involving these unique effect sizes? If so (to the last question), you might consider coding both the basic and the unique effect sizes from the studies included; you then can attempt to proceed using the unique effect sizes, but can revert to the basic effect sizes if you have to. One special consideration involves questions where you are truly interested in multivariate effect sizes, such as independent associations from multiple regression-type analyses. In these situations, you may want to read Chapter 12 before proceeding, and decide whether you might better answer these questions through multivari­ate meta-analysis of basic effect sizes rather than through univariate meta­analysis of multivariate effect sizes.

Second, you will want to determine how readily available the necessary information is within the included effect sizes. It is invaluable to examine some of the primary studies that will be included in your planned meta­analysis to get a sense of what sort of information the authors report. When doing so, sample a few studies from different authors or research groups, as their reporting practices likely differ. If you find that the necessary informa­tion is usually reported, then this can be taken as encouragement to proceed with meta-analysis of unique effect sizes. However, if the necessary infor­mation is rarely or inconsistently reported, you need to assess whether you will be able to obtain this information. Consider both your own willingness to solicit this information from authors and the likely response you will get from them. If you think that the availability of this information will be incon­sistent, then consider both (1) the expected total number of studies from which you could get the necessary information, and (2) the degree to which these studies are representative of all studies that have been conducted.

Finally, you need to realistically consider your own expertise with both meta-analysis and the relevant statistical techniques. If this is your first meta­analysis, I recommend against attempting to use unique effect sizes. Perform­ing a good meta-analytic review of basic effect sizes is challenging enough, so I encourage you to get some experience using these before attempting to meta-analyze unique effect sizes (at a minimum, be sure to code both basic and unique effect sizes). If you feel ready to try to meta-analyze unique effect sizes, consider your level of expertise in that particular statistical area (i.e., that regarding the unique effect size). Do you feel you are fluent in computing the effect size from commonly reported information? Are you familiar with the relevant standard errors and believe you can consistently calculate these from reported information? Do you feel comfortable in guiding researchers through the appropriate analyses when you need to request further informa­tion from them?

This section might seem discouraging, but I do not intend it to be. Using unique effect sizes in your meta-analysis can provide exciting opportunities to answer unique research questions. At the same time, it is important that you are realistic about your ability to use these unique effect sizes, and pro­ceed with caution (but do proceed).

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.