Practical Matters: Using Effect Size Calculators and Meta-Analysis Programs

As I described in Chapter 1, several computer programs are designed to aid in meta-analysis, some of which are available for free and others for purchase. All meta-analytic programs perform two major steps: effect size calculation and effect size combination. Effect size combination (as well as comparison) is the process of aggregating results across studies, the topic of Chapters 8-10 later in this book. Effect size calculation is the process of taking results from each study and converting these into a common effect size, the focus of this chapter.

Relying on an effect size calculator found in meta-analysis programs to compute effect sizes (as well as to combine results across studies) can be a time-saving tool. However, I discourage beginning meta-analysts from rely­ing on them. All of the calculations described in this chapter can be per­formed with a simple hand calculator or spreadsheet program (e.g., Excel), and the meta-analytic combination and comparison I describe later in this book can be performed using these spreadsheets or simple statistical analysis software (e.g., SAS or SPSS). In other words, I see little need for specific soft­ware when conducting a meta-analysis.

Having said both that these programs can save time but that I recom­mend not using them initially, you may wonder if I think that you have too much time on your hands. I do not. Instead, my concern is that these pro­grams make it easy for beginning meta-analysts who are less familiar with the calculations to make mistakes. The value of struggling with the equations in this chapter is that doing so forces you to think about what the values mean and where to find them within the research report. The danger of using an effect size calculator is of mindless use, in which users put in whatever values they can find in the report that look similar to what the program asks for.

At the same time, I do not entirely discourage the use of these meta­analysis programs. They can be of great use in reducing the burden of tedious calculations after you understand these calculations. In other words, if you are just beginning to perform meta-analyses, I encourage you to compute some effect sizes by hand (i.e., using a calculator or spreadsheet program) as well as using one of these programs. Inconsistencies should alert you that either your hand calculations are inaccurate or that you are not providing the cor­rect information to the program (or that the program is inaccurate, though this should be uncommon with the more commonly used programs). After you have confirmed that you obtain identical results by hand and the pro­gram, then you can decide if using the program is worthwhile. I offer this same advice when combining effect sizes, which I discuss later in this book.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.