Why most discovered true associations are inflated
- PMID: 18633328
- DOI: 10.1097/EDE.0b013e31818131e7
Why most discovered true associations are inflated
Erratum in
- Epidemiology. 2009 Jul;20(4):629
Abstract
Newly discovered true (non-null) associations often have inflated effects compared with the true effect sizes. I discuss here the main reasons for this inflation. First, theoretical considerations prove that when true discovery is claimed based on crossing a threshold of statistical significance and the discovery study is underpowered, the observed effects are expected to be inflated. This has been demonstrated in various fields ranging from early stopped clinical trials to genome-wide associations. Second, flexible analyses coupled with selective reporting may inflate the published discovered effects. The vibration ratio (the ratio of the largest vs. smallest effect on the same association approached with different analytic choices) can be very large. Third, effects may be inflated at the stage of interpretation due to diverse conflicts of interest. Discovered effects are not always inflated, and under some circumstances may be deflated-for example, in the setting of late discovery of associations in sequentially accumulated overpowered evidence, in some types of misclassification from measurement error, and in conflicts causing reverse biases. Finally, I discuss potential approaches to this problem. These include being cautious about newly discovered effect sizes, considering some rational down-adjustment, using analytical methods that correct for the anticipated inflation, ignoring the magnitude of the effect (if not necessary), conducting large studies in the discovery phase, using strict protocols for analyses, pursuing complete and transparent reporting of all results, placing emphasis on replication, and being fair with interpretation of results.
Comment in
-
Curses--winner's and otherwise--in genetic epidemiology.Epidemiology. 2008 Sep;19(5):649-51; discussion 657-8. doi: 10.1097/EDE.0b013e318181b865. Epidemiology. 2008. PMID: 18703928 Review.
-
Transposed conditionals, shrinkage, and direct and indirect unbiasedness.Epidemiology. 2008 Sep;19(5):652-4; discussion 657-8. doi: 10.1097/EDE.0b013e318181b3e3. Epidemiology. 2008. PMID: 18703929
-
The search for truth must go beyond statistics.Epidemiology. 2008 Sep;19(5):655-6; discussion 657-8. doi: 10.1097/EDE.0b013e318181b877. Epidemiology. 2008. PMID: 18703930
Similar articles
-
Implications of small effect sizes of individual genetic variants on the design and interpretation of genetic association studies of complex diseases.Am J Epidemiol. 2006 Oct 1;164(7):609-14. doi: 10.1093/aje/kwj259. Epub 2006 Aug 7. Am J Epidemiol. 2006. PMID: 16893921
-
Part 1. Statistical Learning Methods for the Effects of Multiple Air Pollution Constituents.Res Rep Health Eff Inst. 2015 Jun;(183 Pt 1-2):5-50. Res Rep Health Eff Inst. 2015. PMID: 26333238
-
The reassessment of trial perspectives from interim data--a critical view.Stat Med. 2006 Jan 15;25(1):23-36. doi: 10.1002/sim.2180. Stat Med. 2006. PMID: 16220517
-
Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects.J Clin Epidemiol. 2011 Oct;64(10):1060-9. doi: 10.1016/j.jclinepi.2010.12.012. Epub 2011 Mar 31. J Clin Epidemiol. 2011. PMID: 21454050 Review.
-
Effect of formal statistical significance on the credibility of observational associations.Am J Epidemiol. 2008 Aug 15;168(4):374-83; discussion 384-90. doi: 10.1093/aje/kwn156. Epub 2008 Jul 8. Am J Epidemiol. 2008. PMID: 18611956 Review.
Cited by
-
Putting Psychology to the Test: Rethinking Model Evaluation Through Benchmarking and Prediction.Adv Methods Pract Psychol Sci. 2021 Jul-Sep;4(3):10.1177/25152459211026864. doi: 10.1177/25152459211026864. Epub 2021 Sep 23. Adv Methods Pract Psychol Sci. 2021. PMID: 38737598 Free PMC article.
-
Is it Time to Abandon the Use of P-Values in Early Phase Translational Trials: Why (Effect) Size Matters.JACC Basic Transl Sci. 2024 Feb 26;9(2):278-279. doi: 10.1016/j.jacbts.2024.01.008. eCollection 2024 Feb. JACC Basic Transl Sci. 2024. PMID: 38510723 Free PMC article. No abstract available.
-
A genome-wide association study of neutrophil count in individuals associated to an African continental ancestry group facilitates studies of malaria pathogenesis.Hum Genomics. 2024 Mar 15;18(1):26. doi: 10.1186/s40246-024-00585-w. Hum Genomics. 2024. PMID: 38491524 Free PMC article.
-
Parental education and income are linked to offspring cortical brain structure and psychopathology at 9-11 years.JCPP Adv. 2024 Feb 6;4(1):e12220. doi: 10.1002/jcv2.12220. eCollection 2024 Mar. JCPP Adv. 2024. PMID: 38486948 Free PMC article.
-
Deep phenotyping of post-infectious myalgic encephalomyelitis/chronic fatigue syndrome.Nat Commun. 2024 Feb 21;15(1):907. doi: 10.1038/s41467-024-45107-3. Nat Commun. 2024. PMID: 38383456 Free PMC article.
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources
Other Literature Sources
Medical