Are results from randomized trials in anesthesiology robust or fragile? An analysis using the fragility index

Kale Goerke, Matthew Parke, Jarryd Horn, Chase Meyer, Kody Dormire, Brad White, Trevor Bright, Corbin Walters, Matt Vassar

Research output: Contribution to journalArticle

Abstract

Aim: In anesthesiology, the findings from randomized controlled trials often underpin guidelines influencing clinical decision-making and therefore directly affect patient care. The aim of this study is to evaluate the fragility index and fragility quotient of randomized controlled trials published in the eight highest ranked anesthesiology journals. In addition, we assess the extent to which risk of bias scores, loss to follow-up, Web of Science Citation Index, and journal impact factor influence fragility index and fragility quotient. Methods: We included randomized trials published between 2014 and 2016 from the eight highest ranking anesthesiology journals based on Clarivate Analytics' Science Citation Index and Google Scholar Metrics: Anesthesiology subcategory. We included journals that published general anesthesia topics and omitted specialty anesthesia journals. The fragility index and fragility quotient for all included trials were calculated. Risk of bias for each trial was evaluated using the Cochrane 'risk of bias' Tool 2.0. Results: One hundred and thirty one randomized control trials were included in this analysis. The median fragility index was 3 (interquartile range 1.0-5.5) with a fragility quotient of 0.03 (interquartile range 0.01-0.08). In 11% (14/131) of trials, the number of patients lost to follow-up was greater than the corresponding fragility index. Weak correlations were found between fragility index and total sample size (r=0.13) and between fragility index and event frequency (r=0.19). A near-negligible correlation was found between 5-year impact factor and fragility index (r=-0.03) and, similarly, between fragility index and Science Citation Index (r=-0.05). Ten trials were at high risk of bias with the randomization process found to be the domain at the highest risk of bias. Conclusion: In assessing the fragility of randomized controlled trials published in the top eight anesthesiology journals, our study suggests that statistically significant results in these journals are disconcertingly fragile. The median fragility index calculated from our 131 primary studies reveals that only three nonevents must be replaced with events to negate statistical significance. Although a current scale does not exist for fragility index ranges, many trials published by the top journals in anesthesiology are based on concerning methodology and highly fragile outcomes. With small median sample sizes and few patient events characterizing a large number of these trials, many of today's current guidelines and clinical practices may be founded on research containing statistical significance but lacking clinical significance.

Original languageEnglish
JournalInternational Journal of Evidence-Based Healthcare
DOIs
StateAccepted/In press - 1 Jan 2019

Fingerprint

Anesthesiology
Randomized Controlled Trials
Sample Size
Journal Impact Factor
Lost to Follow-Up
Random Allocation
Practice Guidelines
General Anesthesia
Patient Care
Anesthesia
Guidelines
Research

Keywords

  • anesthesiology
  • fragility index
  • randomized control trial
  • research methodology
  • statistics

Cite this

@article{0c55357db71b43989583d19654ad4e2c,
title = "Are results from randomized trials in anesthesiology robust or fragile? An analysis using the fragility index",
abstract = "Aim: In anesthesiology, the findings from randomized controlled trials often underpin guidelines influencing clinical decision-making and therefore directly affect patient care. The aim of this study is to evaluate the fragility index and fragility quotient of randomized controlled trials published in the eight highest ranked anesthesiology journals. In addition, we assess the extent to which risk of bias scores, loss to follow-up, Web of Science Citation Index, and journal impact factor influence fragility index and fragility quotient. Methods: We included randomized trials published between 2014 and 2016 from the eight highest ranking anesthesiology journals based on Clarivate Analytics' Science Citation Index and Google Scholar Metrics: Anesthesiology subcategory. We included journals that published general anesthesia topics and omitted specialty anesthesia journals. The fragility index and fragility quotient for all included trials were calculated. Risk of bias for each trial was evaluated using the Cochrane 'risk of bias' Tool 2.0. Results: One hundred and thirty one randomized control trials were included in this analysis. The median fragility index was 3 (interquartile range 1.0-5.5) with a fragility quotient of 0.03 (interquartile range 0.01-0.08). In 11{\%} (14/131) of trials, the number of patients lost to follow-up was greater than the corresponding fragility index. Weak correlations were found between fragility index and total sample size (r=0.13) and between fragility index and event frequency (r=0.19). A near-negligible correlation was found between 5-year impact factor and fragility index (r=-0.03) and, similarly, between fragility index and Science Citation Index (r=-0.05). Ten trials were at high risk of bias with the randomization process found to be the domain at the highest risk of bias. Conclusion: In assessing the fragility of randomized controlled trials published in the top eight anesthesiology journals, our study suggests that statistically significant results in these journals are disconcertingly fragile. The median fragility index calculated from our 131 primary studies reveals that only three nonevents must be replaced with events to negate statistical significance. Although a current scale does not exist for fragility index ranges, many trials published by the top journals in anesthesiology are based on concerning methodology and highly fragile outcomes. With small median sample sizes and few patient events characterizing a large number of these trials, many of today's current guidelines and clinical practices may be founded on research containing statistical significance but lacking clinical significance.",
keywords = "anesthesiology, fragility index, randomized control trial, research methodology, statistics",
author = "Kale Goerke and Matthew Parke and Jarryd Horn and Chase Meyer and Kody Dormire and Brad White and Trevor Bright and Corbin Walters and Matt Vassar",
year = "2019",
month = "1",
day = "1",
doi = "10.1097/XEB.0000000000000200",
language = "English",
journal = "International Journal of Evidence-Based Healthcare",
issn = "1744-1595",
publisher = "Lippincott Williams and Wilkins Ltd.",

}

TY - JOUR

T1 - Are results from randomized trials in anesthesiology robust or fragile? An analysis using the fragility index

AU - Goerke, Kale

AU - Parke, Matthew

AU - Horn, Jarryd

AU - Meyer, Chase

AU - Dormire, Kody

AU - White, Brad

AU - Bright, Trevor

AU - Walters, Corbin

AU - Vassar, Matt

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Aim: In anesthesiology, the findings from randomized controlled trials often underpin guidelines influencing clinical decision-making and therefore directly affect patient care. The aim of this study is to evaluate the fragility index and fragility quotient of randomized controlled trials published in the eight highest ranked anesthesiology journals. In addition, we assess the extent to which risk of bias scores, loss to follow-up, Web of Science Citation Index, and journal impact factor influence fragility index and fragility quotient. Methods: We included randomized trials published between 2014 and 2016 from the eight highest ranking anesthesiology journals based on Clarivate Analytics' Science Citation Index and Google Scholar Metrics: Anesthesiology subcategory. We included journals that published general anesthesia topics and omitted specialty anesthesia journals. The fragility index and fragility quotient for all included trials were calculated. Risk of bias for each trial was evaluated using the Cochrane 'risk of bias' Tool 2.0. Results: One hundred and thirty one randomized control trials were included in this analysis. The median fragility index was 3 (interquartile range 1.0-5.5) with a fragility quotient of 0.03 (interquartile range 0.01-0.08). In 11% (14/131) of trials, the number of patients lost to follow-up was greater than the corresponding fragility index. Weak correlations were found between fragility index and total sample size (r=0.13) and between fragility index and event frequency (r=0.19). A near-negligible correlation was found between 5-year impact factor and fragility index (r=-0.03) and, similarly, between fragility index and Science Citation Index (r=-0.05). Ten trials were at high risk of bias with the randomization process found to be the domain at the highest risk of bias. Conclusion: In assessing the fragility of randomized controlled trials published in the top eight anesthesiology journals, our study suggests that statistically significant results in these journals are disconcertingly fragile. The median fragility index calculated from our 131 primary studies reveals that only three nonevents must be replaced with events to negate statistical significance. Although a current scale does not exist for fragility index ranges, many trials published by the top journals in anesthesiology are based on concerning methodology and highly fragile outcomes. With small median sample sizes and few patient events characterizing a large number of these trials, many of today's current guidelines and clinical practices may be founded on research containing statistical significance but lacking clinical significance.

AB - Aim: In anesthesiology, the findings from randomized controlled trials often underpin guidelines influencing clinical decision-making and therefore directly affect patient care. The aim of this study is to evaluate the fragility index and fragility quotient of randomized controlled trials published in the eight highest ranked anesthesiology journals. In addition, we assess the extent to which risk of bias scores, loss to follow-up, Web of Science Citation Index, and journal impact factor influence fragility index and fragility quotient. Methods: We included randomized trials published between 2014 and 2016 from the eight highest ranking anesthesiology journals based on Clarivate Analytics' Science Citation Index and Google Scholar Metrics: Anesthesiology subcategory. We included journals that published general anesthesia topics and omitted specialty anesthesia journals. The fragility index and fragility quotient for all included trials were calculated. Risk of bias for each trial was evaluated using the Cochrane 'risk of bias' Tool 2.0. Results: One hundred and thirty one randomized control trials were included in this analysis. The median fragility index was 3 (interquartile range 1.0-5.5) with a fragility quotient of 0.03 (interquartile range 0.01-0.08). In 11% (14/131) of trials, the number of patients lost to follow-up was greater than the corresponding fragility index. Weak correlations were found between fragility index and total sample size (r=0.13) and between fragility index and event frequency (r=0.19). A near-negligible correlation was found between 5-year impact factor and fragility index (r=-0.03) and, similarly, between fragility index and Science Citation Index (r=-0.05). Ten trials were at high risk of bias with the randomization process found to be the domain at the highest risk of bias. Conclusion: In assessing the fragility of randomized controlled trials published in the top eight anesthesiology journals, our study suggests that statistically significant results in these journals are disconcertingly fragile. The median fragility index calculated from our 131 primary studies reveals that only three nonevents must be replaced with events to negate statistical significance. Although a current scale does not exist for fragility index ranges, many trials published by the top journals in anesthesiology are based on concerning methodology and highly fragile outcomes. With small median sample sizes and few patient events characterizing a large number of these trials, many of today's current guidelines and clinical practices may be founded on research containing statistical significance but lacking clinical significance.

KW - anesthesiology

KW - fragility index

KW - randomized control trial

KW - research methodology

KW - statistics

UR - http://www.scopus.com/inward/record.url?scp=85070805472&partnerID=8YFLogxK

U2 - 10.1097/XEB.0000000000000200

DO - 10.1097/XEB.0000000000000200

M3 - Article

AN - SCOPUS:85070805472

JO - International Journal of Evidence-Based Healthcare

JF - International Journal of Evidence-Based Healthcare

SN - 1744-1595

ER -