Finding Relevant Literature for Meta-Analysis

After specifying inclusion/exclusion criteria, the next step is to begin search­ing for empirical studies that fit within this sampling frame. In searching for this relevant literature, you have many options, each with advantages and limitations over the others. Although it is not always necessary to use all of the options I list next, it is useful to consider at least most of them and how reliance on some but not others might bias the sample of studies you obtain for your meta-analysis.

Before describing these search options, it is useful to consider the con­cepts of recall and precision (see White, 2009). Recall is the percentage of studies retrieved from those that should be retrieved (i.e., the number of stud­ies meeting your inclusion criteria that actually exist); it is a theoretical value that can never be known because you never know how many studies actually exist. Precision is the percentage of retrieved studies that are relevant (i.e., actually meet your inclusion criteria). Ideally, we would like both to be 100%, such that our search strategies yield every available study that meets our criteria and none that do not. In reality, we can never meet this goal, so you must balance the relative costs of one or the other being less than 100%. The cost of imperfect recall is that you will miss studies that should have been included, resulting in reduced statistical power and potentially biased results if the missed studies differ from those you included. The cost of imperfect precision is that we will waste our resources retrieving and reading studies that will not be included in our meta-analysis. Although this might not seem like a tremendous cost, it is if it means that you cannot complete your meta- analysis.⁷ The goal of your search strategy should be to achieve high recall without diminishing precision beyond an unacceptable level, where “unac­ceptable” depends on your available resources and the expected benefits of increasing recall in terms of statistical power and reducing bias.

1. Electronic Databases

Modern electronic databases, available via the Internet through most uni­versity libraries (or available for subscription for others), have made the task of searching for relevant studies much easier than in the early days of meta­analysis. Electronic databases exist in many fields, such as economics (Econ- Lit), education (ERIC), medicine (Medline), psychology (PsycINFO), and sociology (Sociological Abstracts), to name just a few. These databases often have wide coverage (though see cautions below) and therefore serve as one of the primary search tools in modern meta-analysis. In fact, these databases are typically the first searches performed by meta-analysts, and I would con­sider them necessary (though not sufficient) for your meta-analysis.

Despite their power and apparent simplicity, using electronic databases is a more complex process than might be initially apparent (see Reed & Baxter, 2009). I next describe three considerations in using these databases, attempt­ing to consider these generically rather than focusing on any one database.

1.1. What Is Included and What Is Excluded?

The first question you should ask before using any electronic database is “What is included (and what is excluded) from this database?.” Answering this question requires you to read the documentation of the databases you are considering; consulting with librarians in your topical area is invaluable, as they have considerable expertise on this question.

Some databases include dissertations and other unpublished works, whereas others do not. If the database you plan to use does not include dis­sertations, you should certainly supplement your search of this database with one that includes dissertations (such as Proquest dissertation and the­sis database). If the database does not include other unpublished work, and your inclusion criteria allow for this work, then you will need to ensure that other search strategies will find these works. If the database does include unpublished works, you should investigate how these works are selected for inclusion; databases that include works unsystematically (e.g., primary study authors being willing to submit works to the database) should be treated cau­tiously as the sample of unpublished work may be biased.⁸

Another consideration is the breadth of published work included in the database. Prominent journals are more likely to be included than peripheral journals, and books by larger publishers are more likely to be included than those by lesser-known publishers. If it is plausible that the results (effect sizes) could differ in studies published in outlets included (e.g., prominent journals) versus excluded (e.g., periphery journals) in the database(s) you are using, then reliance on this database may yield a biased sample of studies.

1.2. Key Words

After researching the databases you will use to understand their coverage, you then search the databases for relevant studies. To perform this search, you generally enter key words, for which the search engine will return records containing these key words. Selection of appropriate key words goes far in increasing recall and precision, so you should consider these key words care­fully and report them in your meta-analytic review.

A first consideration is the key words you select. You can select key words based on your knowledge of the literature in your area, by examining the key words specified in studies that you know contain data about the phe­nomenon of interest, and through thesauri available in some electronic data­bases. Your goal is to create a list of words or phrases that (1) are as specific to the phenomenon you are investigating as possible and (2) cover the range of terms used to describe the phenomenon. Considering the example meta­analysis involving associations of relational aggression with various other constructs (e.g., gender, peer rejection), our goal was to search for all studies of relational aggression. Terms such as “aggression” were too broad, as these would identify studies investigating constructs aside from that in which we were interested. Using the term “relational aggression” was more specific, but by itself would have been inadequate because different researchers use dif­ferent terms for this construct. We ultimately developed a list of four terms to use in our search (“relational aggression,” “social aggression,” “indirect aggression,” and “covert aggression”) that represent the terms typically used by primary study authors investigating this construct.

Wildcard marks (e.g., “*” in PsycINFO) are useful in combination with key words. Wildcard marks are used in conjunction with a stem, specify­ing that the search engine returns all studies containing the specified stem followed by any characters where the wildcard mark is typed. For example, submitting the phrase “relational agg*” would return studies containing the phrases “relational aggression,” “relational aggressor,” and so on. Using wildcard marks can also return unexpected and unwanted findings, how­ever, (e.g., the example stem and wildcard would also return any studies that used the phrase “relational aggravation”). These can generally be recognized quickly and skipped, or you can modify the wildcard search term or use the Boolean statement “not” as described next.

Boolean statements are a tremendous asset when you are searching elec­tronic databases. These statements include “or,” “and,” and “not” in most databases. The use of “or” is especially valuable in combining alternative key words for the same construct; for example, we connected the four terms for the construct of interest using “or” in our example meta-analysis (i.e., the search phrase was: “relational aggression” or “social aggression” or “indi­rect aggression” or “covert aggression”). The logical statement “and” is useful for either limiting the studies returned or specifying two construct associa­tions that are of interest in many meta-analyses. For example, in the example meta-analysis, we could have combined the above search (various key words for relational aggression combined using “or”) with a phrase limiting the samples to childhood or adolescence (“child* or adolesc*”) using the “and” statement.⁹ Similarly, if we were only interested in studies reporting associa­tions between relational aggression and peer rejection (one of the examples I use commonly throughout the book), we could have used “and” to link the phrases for relational aggression with a set of phrases for peer rejection. Finally, you can use the key word “not” either to exclude unwanted wildcard phrases (e.g., in the example above, I could specify “not ‘relational aggrava­tion’ ” to remove the unwanted studies using this term), or to specify exclu­sion criteria (e.g., specifying “not ‘adult’ ”).

1.3. Cautions

Electronic databases are incredibly powerful and time-efficient tools for searching for relevant studies, and I believe that every modern meta-analysis should use these databases. However, at least three cautions merit consider­ation.

First, as I described earlier, you should carefully consider what is not included in the electronic databases you use. If a database does not include (or if it has poor rates of inclusion) unpublished works or studies published in peripheral outlets, then reliance on this database alone would result in diminished recall. This diminished recall can threaten your meta-analysis by decreasing statistical power and, if the studies not included in the database systematically differ from those included (e.g., publication bias, Chapter 11), by producing biased results. To avoid these problems, you should identify alternative electronic databases and other search strategies that are likely to identify relevant studies not included in the electronic database you are using.

A related caution comes from the fact that most electronic databases are discipline specific. Although the databases vary in the extent to which they include works in related disciplines, this disciplinary specificity suggests that you should not rely on only a single database within your discipline. Many, if not most, phenomena that social scientists study are considered within mul­tiple disciplines. For example, research on relational aggression might appear not only in psychology (e.g., in the PsycINFO database), but also in criminal justice, education, gender studies, medicine, public health, and sociology (to name just a few possibilities). I recommend that you consider searching at least one or two databases outside of your primary discipline to explore how much literature might be obtained from other disciplines.

A third caution in using electronic databases relates to their very nature: You perform a search and a list of studies is provided, but you have no idea how many potentially relevant studies were not provided. In other words, rely­ing only on electronic databases provides no information about what stud­ies were not identified in your search, so the possibility remains that some studies—and possibly even some very well-known studies—did not match your specified search criteria. You can address this problem in several ways. One possibility is to perform some additional searches within your selected database(s) that use broader terms (e.g., “aggression” rather than more spe­cific terms such as “relational aggression”) and visually scan results to see if any additional relevant studies could be identified with broader search cri­teria. Second, you can rely on additional search procedures besides the elec­tronic database. I return to this topic of assessing the adequacy of your search (including the adequacy of electronic database searches) in Section 3.4.

1.4. Conclusions about the Use of Electronic Databases

Electronic databases of journal articles, books and chapters, and often some unpublished works exist in most social science disciplines. These searchable databases can provide an efficient method of searching for studies to include in your meta-analysis if you carefully consider the coverage of the databases you use and select appropriate key words along with wildcard marks and Boolean statements. These electronic databases should not be your only method of searching the literature, however, as several cautions need to be considered when using them. Nevertheless, the electronic databases are likely to be one of the primary ways you will search for studies, and every modern meta-analysis should use these tools.

2. Bibliographical Reference Volumes

Bibliographical reference volumes are printed works that provide informa­tion similar to electronic databases (e.g., titles, authors, abstracts), often list­ing studies by broad topics and/or including an index of key words. These volumes were frequently published by large research societies and were intended to aid literature searches in specific fields in much the same way that electronic databases now do in most fields. For example, the Ameri­can Psychological Association published Psychological Abstracts from 1927 to 2006. In many fields, publication of these printed reference volumes has been discontinued in favor of online electronic versions (though exceptions may exist).

Searching these reference volumes is not nearly as convenient as search­ing electronic databases, and few meta-analysts currently rely on these vol­umes as their primary search instrument (though you are likely to see them used when you read older meta-analytic reviews). Nevertheless, there still may be instances when you would consider using these printed volumes. Spe­cifically, if studies potentially relevant for your meta-analysis include older studies, and the electronic databases that you use have not yet incorporated all of these older studies, then it may be useful to consult these reference vol­umes to ensure that you do not systematically exclude these older studies.

3. Listings of Unpublished Works

As I mentioned briefly in Chapter 2, and describe in detail in Chapter 11, one of the most challenging threats to many meta-analyses is that of publication bias (a.k.a. the “file drawer problem”). The extent to which you can avoid and evaluate this threat depends on your searching for and including unpub­lished studies in your meta-analysis. I have already mentioned the value of searching electronic databases that include dissertations as one method of obtaining unpublished studies. Next I list three additional listings that might allow you to find more unpublished studies. For each, I suggest searching with the same careful rigor I suggested for searching electronic databases.

3.1.  Conference Programs

A potentially valuable way to find unpublished studies is to search the pro­grams of academic conferences in which relevant work is likely to be pre­sented. Dedicated meta-analysts often have shelves of these programs, though even this idea is becoming antiquated as more conference programs are archived and searchable online. In this approach, you search the titles of presentations listed in conference books (larger conferences typically have at least crude indices) and request copies of these works from authors (whose contact information is usually listed in these books).

From my experience, it is usually possible to identify a large number of unpublished works by searching conference programs; however, retriev­ing copies of these presentations for coding can be more difficult. Typically, you are better able to contact authors and more likely to receive requested presentations if you make your request shortly after the conference rather than several years later. Therefore, studies obtained through conference pro­grams probably underrepresent older studies. Some other tips I have learned through experience include: (1) whenever you request a conference presenta­tion, provide exact details such as the title of the presentation and the year and conference where it was presented; (2) contact coauthors if you do not receive a response from the first author, as some authors of the presentation may have graduated or left academia; (3) tell the author why you are request­ing this information (I will elaborate on this piece of general advice below).

Although I think conference presentations are a valuable source of unpublished studies, there are some limitations and cautions to consider. First, your search for conference presentations should be systematic. If you decide to search the programs of a particular conference, you should make reasonable efforts to search the programs’ books across a reasonable number of years (vs. the years you attended but not the years in between when you did not attend), and you should certainly search for works within the entire conference book (vs. just the presentations you happened to attend). Second, you should recognize that the response rate to your requests might be low (you should track this response rate as it might be useful to report), and you should consider the possibility that responses might be systematically related to effect sizes.¹⁰ Finally, you should anticipate that conference presentations will often present information needed for study coding (Chapter 4) and effect size calculation (Chapters 5-7) in less detail than other formats (e.g., journal articles). It is still better to code what you can from these studies than not to consider them at all, and it is possible to request further information from study authors.

3.2. Funding Agency Lists

Another valuable way to obtain unpublished studies is to search funding award listings from relevant funding agencies (e.g., National Institutes of Health, National Science Foundation, private foundations). Because funding decisions are made before results are known, studies obtained through this approach will not likely be subject to biases in findings of significance/non- significance. Furthermore, searching these listings is likely to yield studies that have been started but have not yet gone through the publication process (i.e., more recent studies).

3.3. Research Registries

Some fields of clinical science have established listings in which researchers are expected to register a study before conducting it. To encourage registra­tion, some journals will only publish results from studies registered prior to conducting the study. Such registries, by creating a listing of studies in advance of knowing the results, should yield a collection of results unbiased by the findings (e.g., nonsignificant or counterintuitive findings). If the field in which you are performing your meta-analysis has such registries, these will be a very valuable search avenue for obtaining an unbiased set of stud­ies.

4. Backward Searches

After accumulating a set of studies for potential inclusion in your meta­analysis, you will begin the process of coding these studies (see Chapters 4-8). You should read these articles completely (vs. going straight for the method and results sections where most information you will code appears), searching for cited studies that might be relevant for your review that you did not identify through your other strategies. Similarly, you should care­fully read prior reviews (narrative or meta-analytic) searching for potentially relevant studies.

This process of searching for relevant studies cited in the works you have found is referred to as “backward searching” (sometimes also called “foot­note chasing”); that is, you are working from the studies you have “back­ward” in time to identify previously conducted studies cited in these works. This approach is especially useful in identifying older studies, whereas it is unlikely to identify newer studies that have not yet been cited. An impor­tant potential bias of this approach comes from the possibility that studies yielding certain “favorable” results (e.g., significant findings, effects favoring expectations) are probably more likely to be cited than studies with “unfa­vorable” results (e.g., null findings, counterintuitive findings).

Despite the potential biases of backward searches, I believe that they represent a valuable method of searching. My own experience is that many studies come from this approach even with what I consider quite thorough initial searches using other means. This approach is also valuable in iden­tifying literature that might have been missed in other search approaches due to failures to use appropriate key words or to search literatures in other disciplines.

5. Forward Searches

Whereas backward searches attempt to find studies cited in the studies you have, forward searches attempt to find studies that cite the studies you have. Forward searches are often performed using special databases for this purpose (e.g., Social Science Citation Index), though some field-specific databases are incorporating this approach (e.g., the psychology database PsycINFO now has this capacity). To perform a forward search, you enter information for a study you know is relevant to the topic of your meta-analysis, and the search engine finds works that cite this study. Because these citing studies necessar­ily occur after the cited study, the search is moving “forward” in time and is more likely to find newer articles than a backward search.

There are various degrees of intensity in engaging in forward searches. A less intense approach is to identify several of the earliest and most seminal works on the topic, then perform forward searches to identify studies citing these seminal papers. At the other end of the spectrum, you could perform forward searches of all works that you have determined meet the inclusion criteria for your review.

Forward searches are likely to yield high recall, as it is unlikely that many relevant studies would fail to cite at least some of the seminal works in the area. However, my experience¹¹ is that forward searches are often quite low in precision. This is because many papers will cite a seminal work in an area when this area is of tangential interest to the paper.

6. Communication with Researchers in the Field

The final search approach that I will describe is to consult experts/researchers in the field in which you are performing your meta-analysis. This approach actually consists of several possibilities.

At a minimum, you should ask some experts to examine your inclu- sion/exclusion criteria and the list of studies you have identified, request­ing that they note additional studies that should have been included. If you examine these suggested studies and some do meet your inclusion criteria, then you should not only include these studies, but also consider why your search strategy failed to identify these studies (and revise your search strat­egy accordingly). I recommend that you consult colleagues who have a some­what different perspective in the field than your own (i.e., different “camps”) to provide a unique perspective.

Another valuable approach to communicating with researchers is simply to e-mail those individuals who conduct research in the area of your meta­analysis, asking them if they have any additional studies on the topic. This effort can also vary in intensity, ranging from e-mailing just the most active researchers in the field to e-mailing every author of studies you have identi­fied through other means. Although you will have to identify an approach that works best for you given your field and relationships with other research­ers, some practices that I have found valuable are: (1) to clearly state why I am requesting studies (e.g., “I am conducting a meta-analytic review of the associations between X and Y”); (2) to provide a small number of the most critical inclusion criteria (e.g., “I am interested in obtaining studies involving children or adolescents”); and (3) to state the various ways that they could provide the requested information to me (e.g., “I would like the correlation between X and Y, but can compute this from most other statistics you might have available, such as £-tests, ANOVA results, or raw means and standard deviations. I am also happy to compute this correlation myself, if you are willing to share the raw data with the agreement that I will delete this data file after computing this correlation.”).

A related but less targeted approach is to post requests on listservs, web­pages, or similar forums. Many of the same practices that are valuable when e-mailing are useful in such postings, though the standards of particular forums might necessitate briefer requests.

These communications with researchers are extremely valuable, though several considerations are important. First, my impression is that the response rates vary widely for different meta-analysts, with some receiving almost no responses but others receiving tremendous responses. I suspect that the fac­tors that improve response rates include your ability to convince others that your request is important and worth their time, your ability to minimize the burden on the researchers, and the quality of relationships you have with these colleagues. A second consideration is the obvious fact that the more widespread your requests (i.e., numerous e-mails or public postings), the more people know that you are conducting this particular meta-analysis, which is a consideration in terms of the review process. Perhaps the most important consideration, however, is one that I believe means that you absolutely must, to at least some degree, involve colleagues in the area of your meta-analysis: Meta-analytic reviews synthesize the body of knowledge in an area of study and typically provide the foundation for the next wave of empirical study in this area. Thus, the research community has a vested interest in this process, and the meta-analyst has an obligation to consider their input. This statement does not mean that you need to send the initial draft of your meta-analysis to everyone in your field (you should not), nor that your review needs to sup­port the conclusions of everyone in your field (your conclusions are hopefully empirically driven). Instead, by soliciting input from others in your field, whether by simply including the full body of their empirical results in your review or obtaining input from a smaller number of colleagues, your meta­analysis will benefit from this collective knowledge.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.