My placement of this chapter on identifying research questions for metaanalysis before chapters on actually performing a meta-analysis is meant to correspond to the order you would follow in approaching this endeavor. As with primary research, you want to know your goals and research questions, as well as potential limitations and critiques, of your meta-analysis before you begin.
However, such an ordering is somewhat artificial in that it misses the often reciprocal relation between planning and conducting a meta-analytic review. At a minimum, someone planning a meta-analysis almost certainly has read empirical studies in the area that would likely be included in the review, and conclusions that the reader takes from these studies will undoubtedly influence the type of questions asked when planning the meta-analysis.
Beyond this obvious example, I think that much of the process of conducting a meta-analysis is less linear than is typically presented, but more of an iterative, back-and-forth process among the various steps of planning, searching the literature, coding studies, analyzing the data, and writing the results. I do not view this reality as problematic; although we should avoid the practice of “HARKing” (Hypothesizing After Results are Known; Kerr, 1998), we do learn a lot during the process of conducting the meta-analysis that can refine our initial questions. Next, I briefly describe how each of the major steps of searching the literature, coding studies, analyzing the data, and writing the results can provide reasons to revise our initial plans of the meta-analysis.
As I discuss in detail in Chapter 3, an important step in meta-analysis is specifying inclusion/exclusion criteria (i.e., what type of studies will be included in the literature) and searching for relevant literature. This process should be guided by the research questions you wish to answer, but the process might also change your research questions. For example, finding that there is little relevant literature to inform your meta-analysis research questions—either too few studies to obtain a good estimate of the overall effect size or too little variation over levels of moderators of interest—might force you to broaden your questions to include more studies. Conversely, finding that so many studies are relevant to your research question that it is not practical to include all of them might cause you to narrow your research question (e.g., to a more limited sample, type of measure, and/or type of intervention).7
Research questions can also be modified after you begin coding studies (see Chapters 4-7). Not only might your careful reading of the studies lead you to new or modified research questions, but also the more formal process of coding might necessitate changes in your research questions. If studies do not provide sufficient information to compute effect sizes consistently, and it is not possible to obtain this information from study authors, then it may be necessary to abandon or modify your original research questions. If your research questions involve comparing studies (i.e., moderator analyses), you may have to alter this research question if the studies do not provide adequate variability or coverage of certain characteristics. For example, if you were interested in evaluating whether an effect size differs across ethnic groups, but during the coding of studies found that most studies sampled only a particular ethnic group, then you would not have adequate variability across the studies and would have to abandon this particular research question (or else modify it in some way to make it more tractable).
Analyzing the data (see Chapters 8-12) is probably where the most modifications to original study questions will occur. Although you should thoroughly investigate your original research questions, and you should avoid entirely exploratory “fishing expeditions,” you will invariably form new research questions during the data analysis phase. Some of these new questions will be formed as you learn answers to your original questions (e.g., “Having found this, I wonder if . . . ?”), whereas other questions will come from simply looking at the data (e.g., thinking about why a particular study, or set of studies, has discrepant effect sizes). Although both approaches are post hoc, the latter is certainly more exploratory—and therefore more likely to capitalize on chance—than the former. However, both approaches to creating new research questions are valuable, as long as you are upfront about their source when presenting and drawing conclusions from your metaanalysis (see Chapter 13).
As is true of analyzing the data, the process of writing your results may lead to refinement of research questions or even the development of new ones. Furthermore, the process of presenting your findings to colleagues— through either conference presentations or the peer review process—is likely to generate further refinement and creation of research questions.
Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.
very nice submit, i definitely love this website, keep on it