The Scientific Process of Research Synthesis

Given the importance of research syntheses, including meta-analyses, to the progression of science, it is critical to follow scientific standards in their preparation. Most scientists are well trained in methods and data-analytic techniques to ensure objective and valid conclusions in primary research, yet methods and data-analytic techniques for research synthesis are less well known. In this section, I draw from Cooper’s (1982, 1984, 1998, 2009a) description of five⁵ stages of research synthesis to provide an overview of the process and scientific principles of conducting a research synthesis. These stages are formulating the problem, obtaining the studies, making decisions about study inclusion, analyzing and interpreting study results, and present­ing the findings from the research synthesis.

As in any scientific endeavor, the first stage of a literature review is to formulate a problem. Here, the central considerations involve the question that you wish to answer, the constructs you are interested in, and the popu­lation about which you wish to draw conclusions. In terms of the questions answered, a literature review can only answer questions for which prior liter­ature exists. For instance, to make conclusions of causality, the reviewer will need to rely on experimental (or perhaps longitudinal, as an approximation) studies; concurrent naturalistic studies would not be able to provide answers to this question. Defining the constructs of interest seems straightforward but poses two potential complications: The existing literature may use differ­ent terms or operationalizations for the same construct, or the existing litera­ture may use similar terms to describe different constructs. Therefore, you need to define clearly the constructs of interest when planning the review. Similarly, you must consider which samples will be included in the literature review; for instance, you need to decide whether studies of unique popula­tions (e.g., prison, psychiatric settings) should be included within the review. The advantages of a broad approach (in terms of constructs and samples) are that the conclusions of the review will be more generalizable and may allow for the identification of important differences among studies. However, a nar­row approach will likely yield more consistent (i.e., more homogeneous, in the language of meta-analysis) results, and the quantity of literature that must be reviewed is smaller. Both of these features might be seen as advantages or disadvantages, depending on the goals (e.g., to identify average effects versus moderators) and ambition (in terms of the number of studies one is willing to code) of the reviewer.

The next step in a literature review is to obtain the literature relevant for the review. Here, the important consideration is that the reviewer is exhaus­tive, or at least representative, in obtaining relevant literature. It is useful to conceptualize the literature included as a sample drawn from a population of all possible studies. Adapting this conceptualization (and paralleling well- known principles of empirical primary research) highlights the importance of obtaining a representative sample of literature for the review. If the literature reviewed is not representative of the extant research, then the conclusions drawn will be a biased representation of reality. One common threat to all literature reviews is publication bias (also known as the file drawer problem). This threat is that studies that fail to find significant effects (or that find effects counter to what is expected) are less likely to be published, and therefore less likely to be accessible to the reviewer. To counter this threat, you should attempt to obtain unpublished studies (e.g., dissertations), which will either counter this threat or at least allow you to evaluate the magnitude of this bias (e.g., evaluating whether published versus unpublished studies find different effects). Another threat is that reviewers typically must rely on literature writ­ten in a language they know (e.g., English); this excludes literature written in other languages and therefore may exclude most studies conducted in other countries. Although it would be impractical to learn every language in which relevant literature may be written, you should be aware of this limitation and how it impacts the literature on which the review is based. To ensure the transparency of a literature review, the reviewer should report the means by which potentially relevant literature was searched and obtained.

The third, related, stage of a literature review is the evaluation of stud­ies to decide which should inform the review. This stage involves reading the literature obtained in the prior stage (searching for relevant literature) and drawing conclusions regarding relevance. Obvious reasons to exclude works include investigation of constructs or samples that are irrelevant to the review (e.g., studies involving animals when one is interested in human behavior) or failure of the work to provide information relevant to the review (e.g., it treats the construct of interest only as a covariate without providing sufficient information about effects). Less obvious decisions need to be made for works that involve questionable quality or methodological features differ­ent from other studies. Including such works may improve the generalizabil- ity of the review but at the same time may contaminate the literature basis or distract from your focus. Decisions at this stage will typically involve refining the problem formulated at the first stage of the review.

The fourth stage is the most time-consuming and difficult: analyzing and interpreting the literature. As mentioned, there exist several approaches to how reviewers draw conclusions, ranging from qualitative to informal or formal vote counting to meta-analysis. For a meta-analysis, this stage involves systematically coding study characteristics and effect sizes, and then statisti­cally analyzing these coded data. As I describe later in this book (Chapter 2) there are powerful advantages to using a meta-analytic approach.

The final stage of the literature review is the presentation of the review, often in written form. Although I suspend detailed recommendations on reporting meta-analyses until later in the book, a few general guidelines should be considered here. First, we should be transparent about the review process and decisions taken. Just as empirical works are expected to present sufficient details so that another researcher could replicate the results, a well- written research synthesis should provide sufficient detail for another scholar to replicate the review. Second, it is critical that the written report answers the original questions that motivated the review, or at least describes why such answers cannot be reached and what future work is needed to provide these answers. A third, related, guideline is that we should avoid a simple study-by-study listing. A good review synthesizes—not merely lists—the lit­erature. Meta-analysis provides a powerful way of drawing valuable informa­tion from multiple studies that goes far beyond merely listing their individual results.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.