Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews

D. Herrmann, P. Sinnett, J. Holmes, S. Khan, C. Koller, M. Vassar

Research output: Contribution to journalReview article

5 Citations (Scopus)

Abstract

Background: Publication bias is an over-representation of statistically significant results in the published literature and may exaggerate summary effect estimates in oncology systematic reviews. Omitting non-significant results in systematic reviews may therefore affect clinical decision-making. We investigate ways that systematic reviewers attempted to limit publication bias during the search process as well as the statistical methods used to evaluate it. For a subset of reviews not reporting publication bias evaluations, we carried out our own assessments for publication bias to determine its likelihood among these reviews.

Design: We examined systematic reviews from the top five highest impact factor oncology journals published between 2007 and 2015. Systematic reviews were screened for eligibility and qualifying reviews (n = 182) were coded for relevant publication bias study characteristics by two authors. A re-analysis of reviews not initially evaluating for publication bias was carried out using Egger's regression, trim-and-fill, and selection models.

Results: Of the 182 systematic reviews, roughly half carried out a hand search to locate additional studies. Conference abstracts were the most commonly reported form of gray literature, followed by clinical trials registries. Fifty-one reviews reported publication bias evaluations. The most common method was the funnel plot (80%, 41/51) followed by Egger's regression (59%, 30/51) and Begg's test (43%, 22/51). Our publication bias evaluations on non-reporting reviews suggest that the degree of publication bias depends on the method employed.

Conclusion: Our study shows publication bias assessments are not frequently used in oncology systematic reviews. Furthermore, evidence of publication bias was found in a subset of non-reporting reviews. Systematic reviewers in oncology are encouraged to conduct such analyses when appropriate and to employ more robust methods for both mitigating and evaluating publication bias.

Original languageEnglish
Pages (from-to)931-937
Number of pages7
JournalAnnals of oncology : official journal of the European Society for Medical Oncology
Volume28
Issue number5
DOIs
StatePublished - 1 May 2017

Fingerprint

Publication Bias
Medical Oncology
Research
Journal Impact Factor
Registries

Keywords

  • Egger’s
  • funnel plot
  • meta-analysis
  • oncology
  • publication bias
  • systematic reviews

Cite this

@article{35649c902857411dbe3d3e7c9a375bf9,
title = "Statistical controversies in clinical research: publication bias evaluations are not routinely conducted in clinical oncology systematic reviews",
abstract = "Background: Publication bias is an over-representation of statistically significant results in the published literature and may exaggerate summary effect estimates in oncology systematic reviews. Omitting non-significant results in systematic reviews may therefore affect clinical decision-making. We investigate ways that systematic reviewers attempted to limit publication bias during the search process as well as the statistical methods used to evaluate it. For a subset of reviews not reporting publication bias evaluations, we carried out our own assessments for publication bias to determine its likelihood among these reviews.Design: We examined systematic reviews from the top five highest impact factor oncology journals published between 2007 and 2015. Systematic reviews were screened for eligibility and qualifying reviews (n = 182) were coded for relevant publication bias study characteristics by two authors. A re-analysis of reviews not initially evaluating for publication bias was carried out using Egger's regression, trim-and-fill, and selection models.Results: Of the 182 systematic reviews, roughly half carried out a hand search to locate additional studies. Conference abstracts were the most commonly reported form of gray literature, followed by clinical trials registries. Fifty-one reviews reported publication bias evaluations. The most common method was the funnel plot (80{\%}, 41/51) followed by Egger's regression (59{\%}, 30/51) and Begg's test (43{\%}, 22/51). Our publication bias evaluations on non-reporting reviews suggest that the degree of publication bias depends on the method employed.Conclusion: Our study shows publication bias assessments are not frequently used in oncology systematic reviews. Furthermore, evidence of publication bias was found in a subset of non-reporting reviews. Systematic reviewers in oncology are encouraged to conduct such analyses when appropriate and to employ more robust methods for both mitigating and evaluating publication bias.",
keywords = "Egger’s, funnel plot, meta-analysis, oncology, publication bias, systematic reviews",
author = "D. Herrmann and P. Sinnett and J. Holmes and S. Khan and C. Koller and M. Vassar",
year = "2017",
month = "5",
day = "1",
doi = "10.1093/annonc/mdw691",
language = "English",
volume = "28",
pages = "931--937",
journal = "Annals of oncology : official journal of the European Society for Medical Oncology",
issn = "0923-7534",
publisher = "Oxford University Press",
number = "5",

}

Statistical controversies in clinical research : publication bias evaluations are not routinely conducted in clinical oncology systematic reviews. / Herrmann, D.; Sinnett, P.; Holmes, J.; Khan, S.; Koller, C.; Vassar, M.

In: Annals of oncology : official journal of the European Society for Medical Oncology, Vol. 28, No. 5, 01.05.2017, p. 931-937.

Research output: Contribution to journalReview article

TY - JOUR

T1 - Statistical controversies in clinical research

T2 - publication bias evaluations are not routinely conducted in clinical oncology systematic reviews

AU - Herrmann, D.

AU - Sinnett, P.

AU - Holmes, J.

AU - Khan, S.

AU - Koller, C.

AU - Vassar, M.

PY - 2017/5/1

Y1 - 2017/5/1

N2 - Background: Publication bias is an over-representation of statistically significant results in the published literature and may exaggerate summary effect estimates in oncology systematic reviews. Omitting non-significant results in systematic reviews may therefore affect clinical decision-making. We investigate ways that systematic reviewers attempted to limit publication bias during the search process as well as the statistical methods used to evaluate it. For a subset of reviews not reporting publication bias evaluations, we carried out our own assessments for publication bias to determine its likelihood among these reviews.Design: We examined systematic reviews from the top five highest impact factor oncology journals published between 2007 and 2015. Systematic reviews were screened for eligibility and qualifying reviews (n = 182) were coded for relevant publication bias study characteristics by two authors. A re-analysis of reviews not initially evaluating for publication bias was carried out using Egger's regression, trim-and-fill, and selection models.Results: Of the 182 systematic reviews, roughly half carried out a hand search to locate additional studies. Conference abstracts were the most commonly reported form of gray literature, followed by clinical trials registries. Fifty-one reviews reported publication bias evaluations. The most common method was the funnel plot (80%, 41/51) followed by Egger's regression (59%, 30/51) and Begg's test (43%, 22/51). Our publication bias evaluations on non-reporting reviews suggest that the degree of publication bias depends on the method employed.Conclusion: Our study shows publication bias assessments are not frequently used in oncology systematic reviews. Furthermore, evidence of publication bias was found in a subset of non-reporting reviews. Systematic reviewers in oncology are encouraged to conduct such analyses when appropriate and to employ more robust methods for both mitigating and evaluating publication bias.

AB - Background: Publication bias is an over-representation of statistically significant results in the published literature and may exaggerate summary effect estimates in oncology systematic reviews. Omitting non-significant results in systematic reviews may therefore affect clinical decision-making. We investigate ways that systematic reviewers attempted to limit publication bias during the search process as well as the statistical methods used to evaluate it. For a subset of reviews not reporting publication bias evaluations, we carried out our own assessments for publication bias to determine its likelihood among these reviews.Design: We examined systematic reviews from the top five highest impact factor oncology journals published between 2007 and 2015. Systematic reviews were screened for eligibility and qualifying reviews (n = 182) were coded for relevant publication bias study characteristics by two authors. A re-analysis of reviews not initially evaluating for publication bias was carried out using Egger's regression, trim-and-fill, and selection models.Results: Of the 182 systematic reviews, roughly half carried out a hand search to locate additional studies. Conference abstracts were the most commonly reported form of gray literature, followed by clinical trials registries. Fifty-one reviews reported publication bias evaluations. The most common method was the funnel plot (80%, 41/51) followed by Egger's regression (59%, 30/51) and Begg's test (43%, 22/51). Our publication bias evaluations on non-reporting reviews suggest that the degree of publication bias depends on the method employed.Conclusion: Our study shows publication bias assessments are not frequently used in oncology systematic reviews. Furthermore, evidence of publication bias was found in a subset of non-reporting reviews. Systematic reviewers in oncology are encouraged to conduct such analyses when appropriate and to employ more robust methods for both mitigating and evaluating publication bias.

KW - Egger’s

KW - funnel plot

KW - meta-analysis

KW - oncology

KW - publication bias

KW - systematic reviews

UR - http://www.scopus.com/inward/record.url?scp=85042513389&partnerID=8YFLogxK

U2 - 10.1093/annonc/mdw691

DO - 10.1093/annonc/mdw691

M3 - Review article

C2 - 28039176

AN - SCOPUS:85042513389

VL - 28

SP - 931

EP - 937

JO - Annals of oncology : official journal of the European Society for Medical Oncology

JF - Annals of oncology : official journal of the European Society for Medical Oncology

SN - 0923-7534

IS - 5

ER -