Identifying Interesting Moderators in Meta-Analysis

Decisions about which study characteristics to code need to be heavily informed by your knowledge of the content area in which you are performing a meta-analytic review. Nevertheless, I describe two sets of general consider­ations that I believe apply to meta-analytic reviews across fields: considering the research questions you are interested in and considering coding certain specific aspects of studies.

1. Considering Research Questions of Interest

Just as planning a primary research study requires you to select variables based on your research questions, planning a meta-analysis requires that you base your decisions about which study characteristics to code on the research questions that you wish to answer. If your research questions are exclusively about average effect sizes across studies (i.e., combining studies), then you might not need to code much beyond effect sizes, sample sizes, and informa­tion for any artifact corrections you wish to make. I qualify this statement by noting that it is still valuable to be able to provide basic descriptive infor­mation about this sample of studies to inform the generalizability of your review. Nevertheless, the number of study characteristics that you will need to code to address this research question adequately is small.

In contrast, if at least some of your research questions involve com­paring studies (i.e., identifying whether studies with certain features yield larger effect sizes than studies with other features), then it will be much more important to code many study characteristics. Obviously, if you put forth a research question about a specific characteristic moderating effect sizes (e.g., do studies with this characteristic yield larger effect sizes than studies without this characteristic?), then it will be necessary to code this specific characteristic. However, you should also consider what study characteristics might commonly co-occur with the characteristic you are interested in, and code these. For example, if you are interested in investigating whether stud­ies with certain types of samples yield different effect sizes (e.g., children vs. adults), you should carefully consider the other study characteristics that are likely to differ across these types of samples (e.g., studies of adults might frequently rely on self-reports, whereas studies of children might frequently rely on parent reports, observations, etc.). If you fail to code these other study characteristics, then you cannot empirically rule out the possibility that your results involving the coded study characteristic of interest are not really due to these co-occurring characteristics. In contrast, if you do code these char­acteristics, then you are able to evaluate empirically such competing explana­tions (see Chapter 9).

As a more extreme version of research questions involving specific mod­erators, some meta-analysts aim to predict all heterogeneity in effect sizes by coded study characteristics. Although this goal tends to be quite exploratory, and you would therefore view the findings of moderation by specific charac­teristics cautiously, it nevertheless is a goal you might consider. If so, then you will necessarily code a large number of study characteristics; specifically, you will code any study characteristics that meet two conditions. First, the study characteristics are consistently reported in many or even most studies; this is necessary to avoid a preponderance of missing data when you evalu­ate the coded characteristic as a moderator. The second condition is that the study characteristic varies across at least some studies; this variability across studies is necessary for the study characteristic to covary with effect sizes. You would then enter these coded study characteristics into some large pre­dictive model (e.g., forward stepwise regression) to explore relations between them and variation in effect sizes.

2. Considering Specific Aspects of Studies

As I mentioned, the exact study characteristics you code will depend on your research questions and be informed by your knowledge of the topic area. Nevertheless, four general types of characteristics should be considered in any meta-analysis in the social sciences: characteristics of the sample, mea­surement, design, and source (see also Lipsey, 2009; Lipsey & Wilson, 2001, pp. 83-86). These are summarized in Table 4.1.

2.1. Sample Characteristics

Potentially relevant characteristics of the sample that you might consider include aspects of the sampling procedure and the demographic features of the sample. For instance, you might code sampling procedures such as whether the sample was drawn from unique settings (e.g., from a univer­sity setting, some sort of clinical setting, a correctional facility, or specific other settings relevant to the area), whether the study attempted to draw a sample representative of a larger population (e.g., a nationally representative sample) versus relying on a convenience sample, and the country from which the sample was drawn. Potentially relevant demographic features to consider include the gender or ethnic composition of the sample, the mean socioeco­nomic status or age of the sample, or any other potentially relevant descrip­tors (e.g., average IQ). Although you will not necessarily code all of these possible characteristics, either because you do not believe they are relevant or because the primary studies do not consistently report these features, I believe that most meta-analyses in the social sciences should at least consider coding some sample characteristics.

2.2. Measurement Characteristics

In many areas of social science, there exist multiple approaches to measure­ment and multiple specific measures of the variables that comprise your effect sizes. For this reason, you may want to code the measurement characteris­tics of either or both variables comprising your effect size. Potential aspects that can be coded include both the source of information (e.g., self-report; some other reporter such as a spouse, parent, or teacher; observations by the researcher) and specific features of the measurement process (e.g., covert ver­sus overt observations, timed versus untimed performance on a test). In areas where a small number of measurement instruments are widely used, you might also consider coding the specific measure used. In survey research, you might code whether the original version of an instrument, a shortened form, or a translated form of the scale was used. These suggestions repre­sent just a few of the possibilities you might consider. A thorough knowledge of the strengths and limitations about measurement processes and specific measures in your field will be extremely influential in guiding your decisions about the measurement characteristics you might decide to code.

2.3. Design Characteristics

You might also consider coding both broad and narrow characteristics of the designs of studies included in your meta-analysis. At the broad level, you might code, for example, whether studies used experimental group compari­sons, quasi-experimental group comparisons, single-group pre-post com­parisons, or regression discontinuity designs. At a narrower level, you could consider specific design features; for example, if you were conducting a meta­analysis of treatment studies, you might code various aspects of the control groups (e.g., no contact, attention only, treatment as usual, placebo).

2.4. Source Characteristics

Finally, in some instances coding characteristics of the research report itself may be valuable. As described in Chapter 11, you should always code whether or not the report is published (and potentially more nuanced codes such as publication quality) to evaluate evidence of publication bias. There may be instances when it is useful to code the year of publication (or year of pre­sentation for conference presentations, year of defense for dissertation, etc.), which might serve as a proxy for the year the data was collected.1 Evalua­tion of this year as a moderator might illuminate historic trends in the effect sizes across time. It might also be useful to evaluate whether or not studies were funded, or perhaps the specific sources of funding, if you suspect that these factors could bias the results. A fourth set of source characteristics to consider are the potentially relevant characteristics of the researchers them­selves (e.g., discipline, gender, ethnicity). Evaluating these in relation to effect sizes might indicate either the presence of uncoded methodological features (related to, e.g., disciplinary styles) or systematic differences in results poten­tially caused by biases of the researchers (e.g., different magnitudes of gender differences found by male versus female researchers).

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.

Leave a Reply

Your email address will not be published. Required fields are marked *