The inclusion criteria, and conversely the exclusion criteria, are a set of explicit statements about the features of studies that will or will not (respectively) be included in your meta-analysis. Ideally, you should specify these criteria before searching the literature so that you can then determine whether each study identified in your search should be included in your meta-analysis. Practically speaking, however, you are likely to find studies that are ambiguous given your initial criteria, so you will need to modify these criteria as these unanticipated types of studies arise.
1. The Importance of Clear Criteria
Developing an explicit set of inclusion and exclusion criteria is important for three reasons. First, as I noted earlier, these criteria should reliably guide which studies you will (or will not) include in your meta-analysis. This guidance is especially important if others are assisting in your search. Even if you are conducting the search alone, however, these criteria can reduce subjectivity that might be introduced if the criteria are ambiguous.
The second reason that explicit criteria are important is that these criteria define the population to which you can make conclusions. A statement of exclusion (i.e., an exclusion criterion) limits your conclusions not to involve this characteristic. For example, in the example meta-analysis I will present throughout this book (considering various effects involving relational aggression), my colleagues and I excluded samples with an average age of 18 years or older. It would therefore be inappropriate to attempt to draw any conclusions regarding adults from this meta-analysis. A statement of inclusion (i.e., an inclusion criterion) implies that the population is defined—at least in part—by this criterion. For example, a criterion specifying that included studies must use experimental manipulation with double-blind procedures would mean that the population is of studies with this design (and any other inclusion criteria stated).
The third reason that explicit criteria are important relates to the goal of transparency, which is an important general characteristic to consider when reporting your meta-analysis (see Chapter 13). Here, I mean that your inclusion/exclusion criteria should be so explicit that a reader could, after performing the same searches as you perform, come to the same conclusions regarding which studies should be included in your meta-analysis. To illustrate, imagine that you perform a series of searches that identify 100 studies, and based on your inclusion/exclusion criteria you decide that 60 should be included in your meta-analysis. If another person were to evaluate those same 100 studies using your inclusion/exclusion criteria, he or she should—if your criteria are explicit enough—identify the same 60 studies as appropriate for the review. To achieve this level of transparency in your meta-analysis, it is important to record and report the full set of inclusion/ exclusion criteria you used.
2. Potential Inclusion/Exclusion Criteria
The exact inclusion/exclusion criteria you choose for your meta-analysis should be based on the goals of your review (i.e., What type of studies do you want to make conclusions about?) and your knowledge of the field. Nevertheless, there are several common elements that you should consider when developing your inclusion/exclusion criteria (from Lipsey & Wilson, 2001, pp. 18-23):
2.1. Definitions of Constructs of Interest
The most important data in meta-analyses are effect sizes, which typically are some index of an association between X and Y.3 In any meta-analysis of these effect sizes, it is important to specify criteria involving operational definitions of both constructs X and Y. Although it is tempting for those with expertise in the area to take an “I know it when I see it” approach, this approach is inadequate for the reader and for deciding which studies should be included. One challenge is that the literature often refers to the same (or similar enough) construct by different names (e.g., in the example meta-analysis, the construct I refer to as “relational aggression” is also called “social aggression,” “indirect aggression,” and “covert aggression”). A second challenge is that the literature sometimes refers to different constructs with the same name (e.g., in the example meta-analysis, several studies used a scale of “indirect aggression” that included such aspects as diffuse anger and resentment that were inconsistent with the more behavioral definition of interest). By providing a clear operational definition of the constructs of interest, you can avoid ambiguities due to these challenges.
2.2. Sample Characteristics
It is also important to consider the samples used in the primary studies that you will want to include or exclude. Here, numerous possibilities may or may not be relevant to your review, and may or may not appear in the literature you consider. Some basic demographic variables to consider include gender (e.g., Will you include studies sampling only males or only females?), ethnicity (e.g., Will you include only representative samples, or those that sample one ethnic group exclusively?), and age (e.g., Will you include studies sampling infants, toddlers, children, adolescents, young adults, and/or older adults?). It is also worth considering what cultures or nationalities will be included. Even if you place no restrictions on nationality, you will need to exclude reports written in languages you do not know,4 which likely precludes many studies of samples from many areas of the world. Beyond these examples, you might encounter countless others—for example, samples drawn from unique settings (e.g., detention facilities, psychiatric hospitals, bars), selected using atypical screening procedures (e.g., certain personality types), or based on atypical recruitment strategies (e.g., participants navigating to a website). Although it is useful if you can anticipate some of these irregular sample characteristics in advance, many will invariably arise unexpectedly and you will have to deal with these on a case-by-case basis.
2.3. Study Design
A third consideration for inclusion/exclusion criteria for almost every metaanalysis is the type of research design that included studies should have. Some obvious possibilities are to include only experimental, quasi-experimental, longitudinal naturalistic, or concurrent naturalistic designs. Even within these categories, however, there are innumerable possibilities. For example, if you are considering only experimental treatment studies, should you include only those with a certain type of control group, only those using blinded procedures, and so on? Among quasi-experimental studies, are you interested only in between-group comparisons or pre-post designs? Answers to these sorts of questions must come from your knowledge of the field in which you are performing the review, as well as your own goals for the meta-analysis.
2.4. Time Frame
The period of time from which you will draw studies is a consideration that may or may not be relevant to your meta-analysis. By “period of time,” I mean the year in which the primary study was conducted, for which you might use the proxy variable year of publication (or completion, presentation, etc., for unpublished works). For many phenomena, it might be of more interest to include studies from a broad range of time and evaluate historic effects through moderator analyses (i.e., testing whether effect sizes vary regularly across time; see Chapter 9) rather than a priori excluding studies. However, in some situations it may make sense to include only those studies performed within a certain time period. These situations might include when you are only interested in a phenomenon after some historic changes (e.g., correlates of unprotected sex after the AIDS crisis) or when the phenomenon has only existed during a certain period of time (e.g., studies of cyberbullying have only been performed since the popularity of the Internet has increased).
2.5. Publication Type
The reporting format of the studies is another consideration for potential inclusion/exclusion criteria. Although including only published studies is generally considered problematic (due to the high possibility of publication bias; see Chapter 11), it is important to consider what types of reports will be included. Possibilities include dissertations, other unpublished written reports (e.g., reports to funding agencies), conference presentations, or information that the researcher provides you upon request.
2.6. Effect Size Information
Finally, a necessary inclusion criterion is that the studies provide sufficient information to compute an effect size.5 In most situations, this will be information provided in the written report that allows you directly to compute an effect size (see Chapter 5). However, you should also consider whether you would include studies that provide only enough information to compute a lower-bound estimate (e.g., probability ranges such as p < .05, statements that results were nonsignificant; see Chapter 5). When studies do not report sufficient information to compute effect sizes, you should contact the study authors to request more information; here, a necessary inclusion criterion is that the authors supply this information.
3. Relative Advantages of Broad versus Narrow Inclusion Criteria
In developing inclusion/exclusion criteria, specifying both broad and narrow sets of criteria has notable advantages. By broad criteria, I refer to a set of criteria that include most possible studies and exclude few, whereas narrow criteria will exclude many studies and include few. Of course, these two choices represent end points along a continuum. Selecting a set of criteria that falls along this continuum has several implications for your meta-analysis.
Perhaps the most important consideration in weighing a broad versus narrow set of criteria is that of the population of studies about which you want to draw conclusions. Put simply: Would you prefer to make conclusions about a very specific, well-defined population, or would you rather make more gener- alizable conclusions about a potentially messy population (i.e., one with likely fuzzy boundaries, likely inconsistent representation in your sample of studies, and possibly undistinguished subpopulations)? Specific to the issue of study quality (see Chapter 4) is the question of whether you want to include only the most methodologically rigorous studies or are willing to include methodologically flawed studies (risking the “garbage in, garbage out” criticism described in Chapter 2). There is not a universal “right answer” to these questions, just as there is not a right answer to the issue of level of generalization to the “apples and oranges” problem described in Chapter 2. If you choose a narrow set of criteria, you should be cautious to draw conclusions only about this narrowly defined population. In contrast, if you choose a broad set of criteria, it is probably advisable to code for study characteristics that contribute to this breadth and to evaluate these as potential moderators of effect sizes (see Chapter 9).
A second consideration is the number of studies that will ultimately be included in your meta-analysis by specifying a broad versus narrow set of criteria. Broad criteria will result in a meta-analysis of more studies that are more diverse in their features, whereas narrow criteria will result in fewer studies that are more similar in their features. Having fewer studies will sometimes result in inadequate power to evaluate the average effect size (see Chapters 8 and 10), will usually preclude thorough consideration of study characteristics that account for differences in effect sizes (i.e., moderator analyses; see Chapter 9), and might even lead your audience to view your review as too small to be important to the field. In contrast, having more studies increases the amount of work involved in the meta-analysis (especially the coding of studies), perhaps to the point where a meta-analysis of the full collection of studies is impossible.6 Therefore, one consideration is to specify inclusion/exclusion criteria that yield a reasonable number of studies given your research questions, your available time and resources, and typical practices in your field. This is not the only, or even primary, consideration, but it is a realistic factor to consider.
Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.
24 Aug 2021
24 Aug 2021
25 Aug 2021
24 Aug 2021
25 Aug 2021
24 Aug 2021