The fragility of statistically significant results in otolaryngology randomized trials

Objectives: The American Academy of Otolaryngology–Head and Neck Surgery regards randomized controlled trials as class A evidence. A novel method to determine the robustness of outcomes in trials is the fragility index. This index represents the number of patients whose status would have to change from a non-event to an event to make a statistically significant result non-significant. 

Methods: Investigators included otolaryngology journals listed in the top 10 of one or both of Google Scholar Metrics and Clarivate Analytics’ Journal rankings. For inclusion, a randomized controlled trial needed to report a one-to-one random assignment of participants to condition, contain two parallel arms or have used a two-by-two factorial design, and report at least one statistically significant dichotomous outcome. 

Results: Sixty-nine trials met inclusion criteria. The median fragility index was three events (interquartile range 1–7.5). Median sample size was 72 (interquartile range 50–102.5). Modest correlations were observed between fragility index and total sample size (r = 0.27) and fragility index and event rate (r = 0.46). Investigators found no correlation between fragility index and impact factor or Science Citation Index. In 39% (27/69) of trials, the number lost to follow-up was equal to or greater than the fragility index. 

Conclusion: A median fragility index of 3 indicates that three people, on average, are needed to alter the outcomes in otolaryngology trials. This indicates that the results of two-group randomized controlled trials reporting binary endpoints published in otolaryngology journals may frequently be fragile.