Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals

Riley J. Hedin, Blake A. Umberham, Byron N. Detweiler, Lauren Kollmorgen, Matt Vassar

Research output: Contribution to journalArticle

17 Citations (Scopus)

Abstract

BACKGROUND: Systematic reviews and meta-analyses are used by clinicians to derive treatment guidelines and make resource allocation decisions in anesthesiology. One cause for concern with such reviews is the possibility that results from unpublished trials are not represented in the review findings or data synthesis. This problem, known as publication bias, results when studies reporting statistically nonsignificant findings are left unpublished and, therefore, not included in meta-analyses when estimating a pooled treatment effect. In turn, publication bias may lead to skewed results with overestimated effect sizes. The primary objective of this study is to determine the extent to which evaluations for publication bias are conducted by systematic reviewers in highly ranked anesthesiology journals and which practices reviewers use to mitigate publication bias. The secondary objective of this study is to conduct publication bias analyses on the meta-analyses that did not perform these assessments and examine the adjusted pooled effect estimates after accounting for publication bias. METHODS: This study considered meta-analyses and systematic reviews from 5 peer-reviewed anesthesia journals from 2007 through 2015. A PubMed search was conducted, and full-text systematic reviews that fit inclusion criteria were downloaded and coded independently by 2 authors. Coding was then validated, and disagreements were settled by consensus. In total, 207 systematic reviews were included for analysis. In addition, publication bias evaluation was performed for 25 systematic reviews that did not do so originally. We used Egger regression, Duval and Tweedie trim and fill, and funnel plots for these analyses. RESULTS: Fifty-five percent (n = 114) of the reviews discussed publication bias, and 43% (n = 89) of the reviews evaluated publication bias. Funnel plots and Egger regression were the most common methods for evaluating publication bias. Publication bias was reported in 34 reviews (16%). Thirty-six of the 45 (80.0%) publication bias analyses indicated the presence of publication bias by trim and fill analysis, whereas Egger regression indicated publication bias in 23 of 45 (51.1%) analyses. The mean absolute percent difference between adjusted and observed point estimates was 15.5%, the median was 6.2%, and the range was 0% to 85.5%. CONCLUSIONS: Many of these reviews reported following published guidelines such as PRISMA or MOOSE, yet only half appropriately addressed publication bias in their reviews. Compared with previous research, our study found fewer reviews assessing publication bias and greater likelihood of publication bias among reviews not performing these evaluations.

Original languageEnglish
Pages (from-to)1018-1025
Number of pages8
JournalAnesthesia and Analgesia
Volume123
Issue number4
DOIs
StatePublished - 1 Oct 2016

Fingerprint

Publication Bias
Anesthesiology
Meta-Analysis
Guidelines
Peer Review
Resource Allocation

Cite this

Hedin, Riley J. ; Umberham, Blake A. ; Detweiler, Byron N. ; Kollmorgen, Lauren ; Vassar, Matt. / Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals. In: Anesthesia and Analgesia. 2016 ; Vol. 123, No. 4. pp. 1018-1025.
@article{6d83b621b9f943579fddda259e277056,
title = "Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals",
abstract = "BACKGROUND: Systematic reviews and meta-analyses are used by clinicians to derive treatment guidelines and make resource allocation decisions in anesthesiology. One cause for concern with such reviews is the possibility that results from unpublished trials are not represented in the review findings or data synthesis. This problem, known as publication bias, results when studies reporting statistically nonsignificant findings are left unpublished and, therefore, not included in meta-analyses when estimating a pooled treatment effect. In turn, publication bias may lead to skewed results with overestimated effect sizes. The primary objective of this study is to determine the extent to which evaluations for publication bias are conducted by systematic reviewers in highly ranked anesthesiology journals and which practices reviewers use to mitigate publication bias. The secondary objective of this study is to conduct publication bias analyses on the meta-analyses that did not perform these assessments and examine the adjusted pooled effect estimates after accounting for publication bias. METHODS: This study considered meta-analyses and systematic reviews from 5 peer-reviewed anesthesia journals from 2007 through 2015. A PubMed search was conducted, and full-text systematic reviews that fit inclusion criteria were downloaded and coded independently by 2 authors. Coding was then validated, and disagreements were settled by consensus. In total, 207 systematic reviews were included for analysis. In addition, publication bias evaluation was performed for 25 systematic reviews that did not do so originally. We used Egger regression, Duval and Tweedie trim and fill, and funnel plots for these analyses. RESULTS: Fifty-five percent (n = 114) of the reviews discussed publication bias, and 43{\%} (n = 89) of the reviews evaluated publication bias. Funnel plots and Egger regression were the most common methods for evaluating publication bias. Publication bias was reported in 34 reviews (16{\%}). Thirty-six of the 45 (80.0{\%}) publication bias analyses indicated the presence of publication bias by trim and fill analysis, whereas Egger regression indicated publication bias in 23 of 45 (51.1{\%}) analyses. The mean absolute percent difference between adjusted and observed point estimates was 15.5{\%}, the median was 6.2{\%}, and the range was 0{\%} to 85.5{\%}. CONCLUSIONS: Many of these reviews reported following published guidelines such as PRISMA or MOOSE, yet only half appropriately addressed publication bias in their reviews. Compared with previous research, our study found fewer reviews assessing publication bias and greater likelihood of publication bias among reviews not performing these evaluations.",
author = "Hedin, {Riley J.} and Umberham, {Blake A.} and Detweiler, {Byron N.} and Lauren Kollmorgen and Matt Vassar",
year = "2016",
month = "10",
day = "1",
doi = "10.1213/ANE.0000000000001452",
language = "English",
volume = "123",
pages = "1018--1025",
journal = "Anesthesia and Analgesia",
issn = "0003-2999",
publisher = "Lippincott Williams and Wilkins Ltd.",
number = "4",

}

Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals. / Hedin, Riley J.; Umberham, Blake A.; Detweiler, Byron N.; Kollmorgen, Lauren; Vassar, Matt.

In: Anesthesia and Analgesia, Vol. 123, No. 4, 01.10.2016, p. 1018-1025.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Publication Bias and Nonreporting Found in Majority of Systematic Reviews and Meta-analyses in Anesthesiology Journals

AU - Hedin, Riley J.

AU - Umberham, Blake A.

AU - Detweiler, Byron N.

AU - Kollmorgen, Lauren

AU - Vassar, Matt

PY - 2016/10/1

Y1 - 2016/10/1

N2 - BACKGROUND: Systematic reviews and meta-analyses are used by clinicians to derive treatment guidelines and make resource allocation decisions in anesthesiology. One cause for concern with such reviews is the possibility that results from unpublished trials are not represented in the review findings or data synthesis. This problem, known as publication bias, results when studies reporting statistically nonsignificant findings are left unpublished and, therefore, not included in meta-analyses when estimating a pooled treatment effect. In turn, publication bias may lead to skewed results with overestimated effect sizes. The primary objective of this study is to determine the extent to which evaluations for publication bias are conducted by systematic reviewers in highly ranked anesthesiology journals and which practices reviewers use to mitigate publication bias. The secondary objective of this study is to conduct publication bias analyses on the meta-analyses that did not perform these assessments and examine the adjusted pooled effect estimates after accounting for publication bias. METHODS: This study considered meta-analyses and systematic reviews from 5 peer-reviewed anesthesia journals from 2007 through 2015. A PubMed search was conducted, and full-text systematic reviews that fit inclusion criteria were downloaded and coded independently by 2 authors. Coding was then validated, and disagreements were settled by consensus. In total, 207 systematic reviews were included for analysis. In addition, publication bias evaluation was performed for 25 systematic reviews that did not do so originally. We used Egger regression, Duval and Tweedie trim and fill, and funnel plots for these analyses. RESULTS: Fifty-five percent (n = 114) of the reviews discussed publication bias, and 43% (n = 89) of the reviews evaluated publication bias. Funnel plots and Egger regression were the most common methods for evaluating publication bias. Publication bias was reported in 34 reviews (16%). Thirty-six of the 45 (80.0%) publication bias analyses indicated the presence of publication bias by trim and fill analysis, whereas Egger regression indicated publication bias in 23 of 45 (51.1%) analyses. The mean absolute percent difference between adjusted and observed point estimates was 15.5%, the median was 6.2%, and the range was 0% to 85.5%. CONCLUSIONS: Many of these reviews reported following published guidelines such as PRISMA or MOOSE, yet only half appropriately addressed publication bias in their reviews. Compared with previous research, our study found fewer reviews assessing publication bias and greater likelihood of publication bias among reviews not performing these evaluations.

AB - BACKGROUND: Systematic reviews and meta-analyses are used by clinicians to derive treatment guidelines and make resource allocation decisions in anesthesiology. One cause for concern with such reviews is the possibility that results from unpublished trials are not represented in the review findings or data synthesis. This problem, known as publication bias, results when studies reporting statistically nonsignificant findings are left unpublished and, therefore, not included in meta-analyses when estimating a pooled treatment effect. In turn, publication bias may lead to skewed results with overestimated effect sizes. The primary objective of this study is to determine the extent to which evaluations for publication bias are conducted by systematic reviewers in highly ranked anesthesiology journals and which practices reviewers use to mitigate publication bias. The secondary objective of this study is to conduct publication bias analyses on the meta-analyses that did not perform these assessments and examine the adjusted pooled effect estimates after accounting for publication bias. METHODS: This study considered meta-analyses and systematic reviews from 5 peer-reviewed anesthesia journals from 2007 through 2015. A PubMed search was conducted, and full-text systematic reviews that fit inclusion criteria were downloaded and coded independently by 2 authors. Coding was then validated, and disagreements were settled by consensus. In total, 207 systematic reviews were included for analysis. In addition, publication bias evaluation was performed for 25 systematic reviews that did not do so originally. We used Egger regression, Duval and Tweedie trim and fill, and funnel plots for these analyses. RESULTS: Fifty-five percent (n = 114) of the reviews discussed publication bias, and 43% (n = 89) of the reviews evaluated publication bias. Funnel plots and Egger regression were the most common methods for evaluating publication bias. Publication bias was reported in 34 reviews (16%). Thirty-six of the 45 (80.0%) publication bias analyses indicated the presence of publication bias by trim and fill analysis, whereas Egger regression indicated publication bias in 23 of 45 (51.1%) analyses. The mean absolute percent difference between adjusted and observed point estimates was 15.5%, the median was 6.2%, and the range was 0% to 85.5%. CONCLUSIONS: Many of these reviews reported following published guidelines such as PRISMA or MOOSE, yet only half appropriately addressed publication bias in their reviews. Compared with previous research, our study found fewer reviews assessing publication bias and greater likelihood of publication bias among reviews not performing these evaluations.

UR - http://www.scopus.com/inward/record.url?scp=84983059472&partnerID=8YFLogxK

U2 - 10.1213/ANE.0000000000001452

DO - 10.1213/ANE.0000000000001452

M3 - Article

C2 - 27537925

AN - SCOPUS:84983059472

VL - 123

SP - 1018

EP - 1025

JO - Anesthesia and Analgesia

JF - Anesthesia and Analgesia

SN - 0003-2999

IS - 4

ER -