Practical Matters: What Impact Do Sampling Biases Have on Meta-Analytic Conclusions?

The short answer to the question, “What impact do sampling biases have on the conclusions of a meta-analysis?” is “I don’t know.” As a meta-analyst you do not know, your readers do not know, and it is not possible to know unless you could obtain every study that has ever been conducted on the topic of the meta-analysis. Because obtaining every study is almost never possible (and if you did, there is by definition no bias because you have obtained the popula­tion of studies), this question is impossible to answer.

The magnitude of sampling bias likely varies considerably from field to field and even from one meta-analysis to another. So, it is appropriate to always be concerned about the extent to which publication bias impacts the findings of a meta-analysis. Does this mean every meta-analysis should be viewed as untrustworthy and uninformative? Absolutely not. You should remember that the available literature is all that we as scientists have, so if you dismiss this literature as not valuable, then we have nothing on which to base our empirical sciences. Moreover, it is important to remember that a meta-analytic review is no more subject to sampling bias than other lit­erature reviews. In fact, meta-analysis offers two advantages over traditional approaches to literature review that allow us to face the challenge of sam­pling bias. First, meta-analysts typically are more exhaustive in searching the literature than those performing narrative reviews, and the search proce­dures are made transparent in the reporting of meta-analyses. Second, only meta-analysis allows you to evaluate and potentially correct for publication/ sampling bias. Although there is no guarantee that these methods will per­fectly fix the problem, they are far better than simply ignoring it.

Source: Card Noel A. (2015), Applied Meta-Analysis for Social Science Research, The Guilford Press; Annotated edition.