An exploratory test for an excess of significant findings
- PMID: 17715249
- DOI: 10.1177/1740774507079441
An exploratory test for an excess of significant findings
Abstract
Background: The published clinical research literature may be distorted by the pursuit of statistically significant results.
Purpose: We aimed to develop a test to explore biases stemming from the pursuit of nominal statistical significance.
Methods: The exploratory test evaluates whether there is a relative excess of formally significant findings in the published literature due to any reason (e.g., publication bias, selective analyses and outcome reporting, or fabricated data). The number of expected studies with statistically significant results is estimated and compared against the number of observed significant studies. The main application uses alpha = 0.05, but a range of alpha thresholds is also examined. Different values or prior distributions of the effect size are assumed. Given the typically low power (few studies per research question), the test may be best applied across domains of many meta-analyses that share common characteristics (interventions, outcomes, study populations, research environment).
Results: We evaluated illustratively eight meta-analyses of clinical trials with >50 studies each and 10 meta-analyses of clinical efficacy for neuroleptic agents in schizophrenia; the 10 meta-analyses were also examined as a composite domain. Different results were obtained against commonly used tests of publication bias. We demonstrated a clear or possible excess of significant studies in 6 of 8 large meta-analyses and in the wide domain of neuroleptic treatments.
Limitations: The proposed test is exploratory, may depend on prior assumptions, and should be applied cautiously.
Conclusions: An excess of significant findings may be documented in some clinical research fields.
Comment in
-
Comments on 'An exploratory test for an excess of significant findings' by JPA loannidis and TA Trikalinos.Clin Trials. 2007;4(3):254-5; discussion 256-7. doi: 10.1177/1740774507079437. Clin Trials. 2007. PMID: 17715250 No abstract available.
Similar articles
-
The appropriateness of asymmetry tests for publication bias in meta-analyses: a large survey.CMAJ. 2007 Apr 10;176(8):1091-6. doi: 10.1503/cmaj.060410. CMAJ. 2007. PMID: 17420491 Free PMC article.
-
Publication bias was not a good reason to discourage trials with low power.J Clin Epidemiol. 2009 Jan;62(1):47.e1-10. doi: 10.1016/j.jclinepi.2008.02.017. Epub 2008 Jul 14. J Clin Epidemiol. 2009. PMID: 18620841
-
Evaluation of excess statistical significance in meta-analyses of 98 biomarker associations with cancer risk.J Natl Cancer Inst. 2012 Dec 19;104(24):1867-78. doi: 10.1093/jnci/djs437. Epub 2012 Oct 22. J Natl Cancer Inst. 2012. PMID: 23090067
-
Effect of formal statistical significance on the credibility of observational associations.Am J Epidemiol. 2008 Aug 15;168(4):374-83; discussion 384-90. doi: 10.1093/aje/kwn156. Epub 2008 Jul 8. Am J Epidemiol. 2008. PMID: 18611956 Review.
-
Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects.J Clin Epidemiol. 2011 Oct;64(10):1060-9. doi: 10.1016/j.jclinepi.2010.12.012. Epub 2011 Mar 31. J Clin Epidemiol. 2011. PMID: 21454050 Review.
Cited by
-
ABO and Rhesus blood groups and multiple health outcomes: an umbrella review of systematic reviews with meta-analyses of observational studies.BMC Med. 2024 May 20;22(1):206. doi: 10.1186/s12916-024-03423-x. BMC Med. 2024. PMID: 38769523 Free PMC article.
-
Human papillomavirus infection and the risk of cancer at specific sites other than anogenital tract and oropharyngeal region: an umbrella review.EBioMedicine. 2024 Jun;104:105155. doi: 10.1016/j.ebiom.2024.105155. Epub 2024 May 14. EBioMedicine. 2024. PMID: 38744109 Free PMC article. Review.
-
The psychological reality of the learned "p < .05" boundary.Cogn Res Princ Implic. 2024 May 3;9(1):27. doi: 10.1186/s41235-024-00553-x. Cogn Res Princ Implic. 2024. PMID: 38700660 Free PMC article.
-
The maternal factors associated with infant low birth weight: an umbrella review.BMC Pregnancy Childbirth. 2024 Apr 25;24(1):316. doi: 10.1186/s12884-024-06487-y. BMC Pregnancy Childbirth. 2024. PMID: 38664680 Free PMC article.
-
Evidence that publication bias contaminated studies on visual event boundaries and anchoring effects.Proc Natl Acad Sci U S A. 2024 Mar 26;121(13):e2320278121. doi: 10.1073/pnas.2320278121. Epub 2024 Mar 18. Proc Natl Acad Sci U S A. 2024. PMID: 38498712 Free PMC article. No abstract available.
MeSH terms
Substances
LinkOut - more resources
Full Text Sources
Medical