TY - JOUR
T1 - Circular analysis in systems neuroscience
T2 - The dangers of double dipping
AU - Kriegeskorte, Nikolaus
AU - Simmons, W. Kyle
AU - Bellgowan, Patrick S.
AU - Baker, Chris I.
N1 - Funding Information:
We would like to thank P.A. Bandettini, R.W. Cox, J.V. Haxby, D.J. Kravitz, A. Martin, R.A. Poldrack, R.D. Raizada, Z.S. Saad, J.T. Serences and E. Vul for helpful discussions. This work was supported by the Intramural Research Program of the US National Institute of Mental Health.
PY - 2009/5
Y1 - 2009/5
N2 - A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, 'double dipping', the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.
AB - A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. Here we argue that systems neuroscience needs to adjust some widespread practices to avoid the circularity that can arise from selection. In particular, 'double dipping', the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we apply widely used analyses to noise data known to not contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. We suggest a policy for avoiding circularity.
UR - http://www.scopus.com/inward/record.url?scp=67649135107&partnerID=8YFLogxK
U2 - 10.1038/nn.2303
DO - 10.1038/nn.2303
M3 - Article
C2 - 19396166
AN - SCOPUS:67649135107
SN - 1097-6256
VL - 12
SP - 535
EP - 540
JO - Nature Neuroscience
JF - Nature Neuroscience
IS - 5
ER -