Practical Matters: Beginning a Meta-Analytic Database

Aside from perhaps persistence and patience, the most import virtue you can have for searching the literature for a meta-analysis is organization. As you have likely inferred, searching for studies is a time-intensive process, and you certainly do not want to add to this time by repeating work because of poor organization.

A good organizational scheme for the literature search will include sev­eral key components. First, you should have a clear, written statement of the inclusion/exclusion criteria that you will use in evaluating studies found through this search. Toward this end, it might be useful to record stud­ies identified in your search that were excluded for one reason or another (recording why they were excluded). Second, you should have a clear list of methods for searching the literature, with enough details to replicate these searches. For example, you might have a list that begins:

Step 1: Read the following review papers and chapters (listing these works).

Step 2: Search the PsycINFO database using the following key words (listing the key words, including any wildcard marks and logical operations).

Step 3: Search the ERIC database using the following key words (listing the same set of key words as the step 2 search, unless there is reason to use other key words or logical operations).

You then record the dates—and names, if multiple people are conducting the searches—of each search.

During the course of these searches, you will scan many titles and abstracts in an attempt to determine whether each study is relevant for your meta-analysis. I suggest that you be rather inclusive during this initial screen­ing, retaining any studies that might meet your inclusion criterion. You should also retain any nonempirical works, such as reviews or theoretical papers; although these do not provide empirical results for your meta-analysis, it will be worthwhile to read them (1) to identify additional studies cited in these papers, and (2) to inform interpretation of results of your meta-analysis.

As you are identifying works that you will retain, it is critical to have some way of organizing this information. I use spreadsheets such as that shown in Table 3.1. (I have shown only four studies here, your spreadsheet will likely be much larger.) Although you should develop an approach that meets your own needs, this example spreadsheet contains several pieces of information that I recommend recording. The first column contains a number for each paper (article, chapter, dissertation, etc.) identified in the search. The num­ber is arbitrary, but it is useful for filing purposes (as the number of papers becomes large, it is useful to file them by number rather than, e.g., author name). The next four columns contain citation information for the paper. This information is useful not only for citing the paper in your write-up, but in identifying repetitive papers during your multiple search strategies (for this purpose, having this information in a searchable spreadsheet is useful). The sixth column contains the abstract, which is useful if you want to search for specific terms within your spreadsheet. I recommend copying this informa­tion into your spreadsheet if it is electronically available, but it probably is not worth the time needed to type this in manually. The seventh column identi­fies where and when the paper was found; recording the date is important because (1) you might want to update the search near the completion of your meta-analysis, and (2) you should report the last search dates in your presen­tation of your meta-analysis. The two rightmost columns (columns eight and nine) contain information for retrieving and coding the reports. One column indicates whether you have the report, or the status of your attempt to retrieve it (e.g., the third paper notes that I had requested this dissertation through my university’s interlibrary loan system). The last column will become relevant when you begin coding the studies (see Chapters 4-8). Here, I have recorded the person (BS = Brian Stucky, the second author on this paper) who coded this report and the date it was coded. Recording both pieces of information are valuable in case you later identify a problem in the coding (e.g., if one coder was making a consistent error) or if you revise the coding protocol (you then need to modify the coding of all studies coded before this change). In this column, I also record when studies are excluded for a particular reason; for instance, the fourth study was excluded because it used an adult sample (which was one of the specified exclusion criteria in this review).

Of course, you may use a different way of organizing information from your literature search. The point is that you should have some way of organiz­ing information that clearly records important information and avoids any duplication of effort.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.