The fragility of statistically significant results in otolaryngology randomized trials

Mason Skinner, Daniel Tritz, Clayton Farahani, Andrew Ross, Tom Hamilton, Matt Vassar

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

Objectives: The American Academy of Otolaryngology–Head and Neck Surgery regards randomized controlled trials as class A evidence. A novel method to determine the robustness of outcomes in trials is the fragility index. This index represents the number of patients whose status would have to change from a non-event to an event to make a statistically significant result non-significant. Methods: Investigators included otolaryngology journals listed in the top 10 of one or both of Google Scholar Metrics and Clarivate Analytics’ Journal rankings. For inclusion, a randomized controlled trial needed to report a one-to-one random assignment of participants to condition, contain two parallel arms or have used a two-by-two factorial design, and report at least one statistically significant dichotomous outcome. Results: Sixty-nine trials met inclusion criteria. The median fragility index was three events (interquartile range 1–7.5). Median sample size was 72 (interquartile range 50–102.5). Modest correlations were observed between fragility index and total sample size (r = 0.27) and fragility index and event rate (r = 0.46). Investigators found no correlation between fragility index and impact factor or Science Citation Index. In 39% (27/69) of trials, the number lost to follow-up was equal to or greater than the fragility index. Conclusion: A median fragility index of 3 indicates that three people, on average, are needed to alter the outcomes in otolaryngology trials. This indicates that the results of two-group randomized controlled trials reporting binary endpoints published in otolaryngology journals may frequently be fragile.

Original languageEnglish
Pages (from-to)61-66
Number of pages6
JournalAmerican Journal of Otolaryngology - Head and Neck Medicine and Surgery
Volume40
Issue number1
DOIs
StatePublished - 1 Jan 2019

Fingerprint

Otolaryngology
Randomized Controlled Trials
Sample Size
Research Personnel
Lost to Follow-Up
Arm
Neck

Cite this

@article{138a7347866f4f8e8e53fae286d7bfe1,
title = "The fragility of statistically significant results in otolaryngology randomized trials",
abstract = "Objectives: The American Academy of Otolaryngology–Head and Neck Surgery regards randomized controlled trials as class A evidence. A novel method to determine the robustness of outcomes in trials is the fragility index. This index represents the number of patients whose status would have to change from a non-event to an event to make a statistically significant result non-significant. Methods: Investigators included otolaryngology journals listed in the top 10 of one or both of Google Scholar Metrics and Clarivate Analytics’ Journal rankings. For inclusion, a randomized controlled trial needed to report a one-to-one random assignment of participants to condition, contain two parallel arms or have used a two-by-two factorial design, and report at least one statistically significant dichotomous outcome. Results: Sixty-nine trials met inclusion criteria. The median fragility index was three events (interquartile range 1–7.5). Median sample size was 72 (interquartile range 50–102.5). Modest correlations were observed between fragility index and total sample size (r = 0.27) and fragility index and event rate (r = 0.46). Investigators found no correlation between fragility index and impact factor or Science Citation Index. In 39{\%} (27/69) of trials, the number lost to follow-up was equal to or greater than the fragility index. Conclusion: A median fragility index of 3 indicates that three people, on average, are needed to alter the outcomes in otolaryngology trials. This indicates that the results of two-group randomized controlled trials reporting binary endpoints published in otolaryngology journals may frequently be fragile.",
author = "Mason Skinner and Daniel Tritz and Clayton Farahani and Andrew Ross and Tom Hamilton and Matt Vassar",
year = "2019",
month = "1",
day = "1",
doi = "10.1016/j.amjoto.2018.10.011",
language = "English",
volume = "40",
pages = "61--66",
journal = "American Journal of Otolaryngology - Head and Neck Medicine and Surgery",
issn = "0196-0709",
publisher = "W.B. Saunders Ltd",
number = "1",

}

The fragility of statistically significant results in otolaryngology randomized trials. / Skinner, Mason; Tritz, Daniel; Farahani, Clayton; Ross, Andrew; Hamilton, Tom; Vassar, Matt.

In: American Journal of Otolaryngology - Head and Neck Medicine and Surgery, Vol. 40, No. 1, 01.01.2019, p. 61-66.

Research output: Contribution to journalArticle

TY - JOUR

T1 - The fragility of statistically significant results in otolaryngology randomized trials

AU - Skinner, Mason

AU - Tritz, Daniel

AU - Farahani, Clayton

AU - Ross, Andrew

AU - Hamilton, Tom

AU - Vassar, Matt

PY - 2019/1/1

Y1 - 2019/1/1

N2 - Objectives: The American Academy of Otolaryngology–Head and Neck Surgery regards randomized controlled trials as class A evidence. A novel method to determine the robustness of outcomes in trials is the fragility index. This index represents the number of patients whose status would have to change from a non-event to an event to make a statistically significant result non-significant. Methods: Investigators included otolaryngology journals listed in the top 10 of one or both of Google Scholar Metrics and Clarivate Analytics’ Journal rankings. For inclusion, a randomized controlled trial needed to report a one-to-one random assignment of participants to condition, contain two parallel arms or have used a two-by-two factorial design, and report at least one statistically significant dichotomous outcome. Results: Sixty-nine trials met inclusion criteria. The median fragility index was three events (interquartile range 1–7.5). Median sample size was 72 (interquartile range 50–102.5). Modest correlations were observed between fragility index and total sample size (r = 0.27) and fragility index and event rate (r = 0.46). Investigators found no correlation between fragility index and impact factor or Science Citation Index. In 39% (27/69) of trials, the number lost to follow-up was equal to or greater than the fragility index. Conclusion: A median fragility index of 3 indicates that three people, on average, are needed to alter the outcomes in otolaryngology trials. This indicates that the results of two-group randomized controlled trials reporting binary endpoints published in otolaryngology journals may frequently be fragile.

AB - Objectives: The American Academy of Otolaryngology–Head and Neck Surgery regards randomized controlled trials as class A evidence. A novel method to determine the robustness of outcomes in trials is the fragility index. This index represents the number of patients whose status would have to change from a non-event to an event to make a statistically significant result non-significant. Methods: Investigators included otolaryngology journals listed in the top 10 of one or both of Google Scholar Metrics and Clarivate Analytics’ Journal rankings. For inclusion, a randomized controlled trial needed to report a one-to-one random assignment of participants to condition, contain two parallel arms or have used a two-by-two factorial design, and report at least one statistically significant dichotomous outcome. Results: Sixty-nine trials met inclusion criteria. The median fragility index was three events (interquartile range 1–7.5). Median sample size was 72 (interquartile range 50–102.5). Modest correlations were observed between fragility index and total sample size (r = 0.27) and fragility index and event rate (r = 0.46). Investigators found no correlation between fragility index and impact factor or Science Citation Index. In 39% (27/69) of trials, the number lost to follow-up was equal to or greater than the fragility index. Conclusion: A median fragility index of 3 indicates that three people, on average, are needed to alter the outcomes in otolaryngology trials. This indicates that the results of two-group randomized controlled trials reporting binary endpoints published in otolaryngology journals may frequently be fragile.

UR - http://www.scopus.com/inward/record.url?scp=85056810621&partnerID=8YFLogxK

U2 - 10.1016/j.amjoto.2018.10.011

DO - 10.1016/j.amjoto.2018.10.011

M3 - Article

C2 - 30472124

AN - SCOPUS:85056810621

VL - 40

SP - 61

EP - 66

JO - American Journal of Otolaryngology - Head and Neck Medicine and Surgery

JF - American Journal of Otolaryngology - Head and Neck Medicine and Surgery

SN - 0196-0709

IS - 1

ER -