The Results of Randomized Controlled Trials in Emergency Medicine Are Frequently Fragile

Research output: Contribution to journalArticle

3 Citations (Scopus)

Abstract

Study objective: Randomized controlled trials govern evidence-based clinical practice, and it is therefore critical that their results be robust. We aim to investigate the fragility of randomized controlled trials in emergency medicine by determining how often significance would be nullified with small changes in outcomes using the fragility index. Methods: We conducted a methodological systematic review of randomized controlled trials in emergency medicine published in the top 10 general medicine journals and the top 10 emergency medicine journals. Inclusion criteria required that trials be emergency medicine studies structured with a 2-arm or 2-by-2 factorial design and report at least 1 statistically significant dichotomous outcome. Results: A total of 180 trials met inclusion criteria. The median fragility index across all trials in emergency medicine was 4 (interquartile range [IQR]2 to 10)and the median sample size was 140 (IQR 69.5 to 286). For trials from general medicine journals (n=32), the median fragility index was 9 (IQR 4 to 16.5)and the median sample size was 415.5 (IQR 219.5 to 901); for trials from emergency medicine journals (n=148), the median fragility index was 4 (IQR 1 to 9)and the median sample size was 119 (IQR 60 to 227.25). One third of all trials (62/180)had a loss to follow-up that was greater than or equal to the fragility index. There was a modest correlation between fragility index and total number of events (r=0.36; 95% confidence interval [CI]0.23 to 0.48)and a weak correlation between fragility index and total sample size (r=0.26; 95% CI 0.12 to 0.39). There was no correlation between fragility index and either P value (r=–0.14; 95% CI –0.28 to –0.006)or Science Citation Index (r=0.07; 95% CI –0.08 to 0.22). Conclusion: The statistical significance of the results of randomized controlled trials in emergency medicine was often contingent on a small number of events. Until frequentist interpretation of clinical trials is replaced with Bayesian analysis, the fragility index may have utility as a tool to aid clinicians in assessing the robustness of randomized controlled trials in emergency medicine when considered in conjunction with the fragility quotient and other reported metrics.

Original languageEnglish
Pages (from-to)565-576
Number of pages12
JournalAnnals of Emergency Medicine
Volume73
Issue number6
DOIs
StatePublished - 1 Jun 2019

Fingerprint

Emergency Medicine
Randomized Controlled Trials
Sample Size
Confidence Intervals
Medicine
Bayes Theorem
Evidence-Based Practice
Clinical Trials

Cite this

@article{4f647b425b4f4cf19d2673adf3c3a7e8,
title = "The Results of Randomized Controlled Trials in Emergency Medicine Are Frequently Fragile",
abstract = "Study objective: Randomized controlled trials govern evidence-based clinical practice, and it is therefore critical that their results be robust. We aim to investigate the fragility of randomized controlled trials in emergency medicine by determining how often significance would be nullified with small changes in outcomes using the fragility index. Methods: We conducted a methodological systematic review of randomized controlled trials in emergency medicine published in the top 10 general medicine journals and the top 10 emergency medicine journals. Inclusion criteria required that trials be emergency medicine studies structured with a 2-arm or 2-by-2 factorial design and report at least 1 statistically significant dichotomous outcome. Results: A total of 180 trials met inclusion criteria. The median fragility index across all trials in emergency medicine was 4 (interquartile range [IQR]2 to 10)and the median sample size was 140 (IQR 69.5 to 286). For trials from general medicine journals (n=32), the median fragility index was 9 (IQR 4 to 16.5)and the median sample size was 415.5 (IQR 219.5 to 901); for trials from emergency medicine journals (n=148), the median fragility index was 4 (IQR 1 to 9)and the median sample size was 119 (IQR 60 to 227.25). One third of all trials (62/180)had a loss to follow-up that was greater than or equal to the fragility index. There was a modest correlation between fragility index and total number of events (r=0.36; 95{\%} confidence interval [CI]0.23 to 0.48)and a weak correlation between fragility index and total sample size (r=0.26; 95{\%} CI 0.12 to 0.39). There was no correlation between fragility index and either P value (r=–0.14; 95{\%} CI –0.28 to –0.006)or Science Citation Index (r=0.07; 95{\%} CI –0.08 to 0.22). Conclusion: The statistical significance of the results of randomized controlled trials in emergency medicine was often contingent on a small number of events. Until frequentist interpretation of clinical trials is replaced with Bayesian analysis, the fragility index may have utility as a tool to aid clinicians in assessing the robustness of randomized controlled trials in emergency medicine when considered in conjunction with the fragility quotient and other reported metrics.",
author = "Jamin Brown and Aaron Lane and Craig Cooper and Matt Vassar",
year = "2019",
month = "6",
day = "1",
doi = "10.1016/j.annemergmed.2018.10.037",
language = "English",
volume = "73",
pages = "565--576",
journal = "Annals of Emergency Medicine",
issn = "0196-0644",
publisher = "Mosby Inc.",
number = "6",

}

The Results of Randomized Controlled Trials in Emergency Medicine Are Frequently Fragile. / Brown, Jamin; Lane, Aaron; Cooper, Craig; Vassar, Matt.

In: Annals of Emergency Medicine, Vol. 73, No. 6, 01.06.2019, p. 565-576.

Research output: Contribution to journalArticle

TY - JOUR

T1 - The Results of Randomized Controlled Trials in Emergency Medicine Are Frequently Fragile

AU - Brown, Jamin

AU - Lane, Aaron

AU - Cooper, Craig

AU - Vassar, Matt

PY - 2019/6/1

Y1 - 2019/6/1

N2 - Study objective: Randomized controlled trials govern evidence-based clinical practice, and it is therefore critical that their results be robust. We aim to investigate the fragility of randomized controlled trials in emergency medicine by determining how often significance would be nullified with small changes in outcomes using the fragility index. Methods: We conducted a methodological systematic review of randomized controlled trials in emergency medicine published in the top 10 general medicine journals and the top 10 emergency medicine journals. Inclusion criteria required that trials be emergency medicine studies structured with a 2-arm or 2-by-2 factorial design and report at least 1 statistically significant dichotomous outcome. Results: A total of 180 trials met inclusion criteria. The median fragility index across all trials in emergency medicine was 4 (interquartile range [IQR]2 to 10)and the median sample size was 140 (IQR 69.5 to 286). For trials from general medicine journals (n=32), the median fragility index was 9 (IQR 4 to 16.5)and the median sample size was 415.5 (IQR 219.5 to 901); for trials from emergency medicine journals (n=148), the median fragility index was 4 (IQR 1 to 9)and the median sample size was 119 (IQR 60 to 227.25). One third of all trials (62/180)had a loss to follow-up that was greater than or equal to the fragility index. There was a modest correlation between fragility index and total number of events (r=0.36; 95% confidence interval [CI]0.23 to 0.48)and a weak correlation between fragility index and total sample size (r=0.26; 95% CI 0.12 to 0.39). There was no correlation between fragility index and either P value (r=–0.14; 95% CI –0.28 to –0.006)or Science Citation Index (r=0.07; 95% CI –0.08 to 0.22). Conclusion: The statistical significance of the results of randomized controlled trials in emergency medicine was often contingent on a small number of events. Until frequentist interpretation of clinical trials is replaced with Bayesian analysis, the fragility index may have utility as a tool to aid clinicians in assessing the robustness of randomized controlled trials in emergency medicine when considered in conjunction with the fragility quotient and other reported metrics.

AB - Study objective: Randomized controlled trials govern evidence-based clinical practice, and it is therefore critical that their results be robust. We aim to investigate the fragility of randomized controlled trials in emergency medicine by determining how often significance would be nullified with small changes in outcomes using the fragility index. Methods: We conducted a methodological systematic review of randomized controlled trials in emergency medicine published in the top 10 general medicine journals and the top 10 emergency medicine journals. Inclusion criteria required that trials be emergency medicine studies structured with a 2-arm or 2-by-2 factorial design and report at least 1 statistically significant dichotomous outcome. Results: A total of 180 trials met inclusion criteria. The median fragility index across all trials in emergency medicine was 4 (interquartile range [IQR]2 to 10)and the median sample size was 140 (IQR 69.5 to 286). For trials from general medicine journals (n=32), the median fragility index was 9 (IQR 4 to 16.5)and the median sample size was 415.5 (IQR 219.5 to 901); for trials from emergency medicine journals (n=148), the median fragility index was 4 (IQR 1 to 9)and the median sample size was 119 (IQR 60 to 227.25). One third of all trials (62/180)had a loss to follow-up that was greater than or equal to the fragility index. There was a modest correlation between fragility index and total number of events (r=0.36; 95% confidence interval [CI]0.23 to 0.48)and a weak correlation between fragility index and total sample size (r=0.26; 95% CI 0.12 to 0.39). There was no correlation between fragility index and either P value (r=–0.14; 95% CI –0.28 to –0.006)or Science Citation Index (r=0.07; 95% CI –0.08 to 0.22). Conclusion: The statistical significance of the results of randomized controlled trials in emergency medicine was often contingent on a small number of events. Until frequentist interpretation of clinical trials is replaced with Bayesian analysis, the fragility index may have utility as a tool to aid clinicians in assessing the robustness of randomized controlled trials in emergency medicine when considered in conjunction with the fragility quotient and other reported metrics.

UR - http://www.scopus.com/inward/record.url?scp=85058189836&partnerID=8YFLogxK

U2 - 10.1016/j.annemergmed.2018.10.037

DO - 10.1016/j.annemergmed.2018.10.037

M3 - Article

C2 - 30551894

AN - SCOPUS:85058189836

VL - 73

SP - 565

EP - 576

JO - Annals of Emergency Medicine

JF - Annals of Emergency Medicine

SN - 0196-0644

IS - 6

ER -