Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review
- PMID: 23861749
- PMCID: PMC3702538
- DOI: 10.1371/journal.pone.0066844
Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review
Abstract
Background: The increased use of meta-analysis in systematic reviews of healthcare interventions has highlighted several types of bias that can arise during the completion of a randomised controlled trial. Study publication bias and outcome reporting bias have been recognised as a potential threat to the validity of meta-analysis and can make the readily available evidence unreliable for decision making.
Methodology/principal findings: In this update, we review and summarise the evidence from cohort studies that have assessed study publication bias or outcome reporting bias in randomised controlled trials. Twenty studies were eligible of which four were newly identified in this update. Only two followed the cohort all the way through from protocol approval to information regarding publication of outcomes. Fifteen of the studies investigated study publication bias and five investigated outcome reporting bias. Three studies have found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes (range of odds ratios: 2.2 to 4.7). In comparing trial publications to protocols, we found that 40-62% of studies had at least one primary outcome that was changed, introduced, or omitted. We decided not to undertake meta-analysis due to the differences between studies.
Conclusions: This update does not change the conclusions of the review in which 16 studies were included. Direct empirical evidence for the existence of study publication bias and outcome reporting bias is shown. There is strong evidence of an association between significant results and publication; studies that report positive or significant results are more likely to be published and outcomes that are statistically significant have higher odds of being fully reported. Publications have been found to be inconsistent with their protocols. Researchers need to be aware of the problems of both types of bias and efforts should be concentrated on improving the reporting of trials.
Conflict of interest statement
Figures
![Figure 1](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g001.gif)
![Figure 2](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g002.gif)
![Figure 3](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g003.gif)
![Figure 4](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g004.gif)
![Figure 5](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g005.gif)
![Figure 6](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g006.gif)
![Figure 7](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g007.gif)
![Figure 8](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g008.gif)
![Figure 9](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g009.gif)
![Figure 10](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g010.gif)
![Figure 11](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g011.gif)
![Figure 12](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g012.gif)
![Figure 13](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g013.gif)
![Figure 14](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g014.gif)
![Figure 15](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g015.gif)
![Figure 16](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g016.gif)
![Figure 17](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g017.gif)
![Figure 18](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g018.gif)
![Figure 19](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g019.gif)
![Figure 20](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g020.gif)
![Figure 21](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g021.gif)
![Figure 22](https://cdn.statically.io/img/www.ncbi.nlm.nih.gov/pmc/articles/instance/3702538/bin/pone.0066844.g022.gif)
Similar articles
-
Systematic review of the empirical evidence of study publication bias and outcome reporting bias.PLoS One. 2008 Aug 28;3(8):e3081. doi: 10.1371/journal.pone.0003081. PLoS One. 2008. PMID: 18769481 Free PMC article. Review.
-
Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas.Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article.
-
Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions.Cochrane Database Syst Rev. 2014 Oct 1;2014(10):MR000035. doi: 10.1002/14651858.MR000035.pub2. Cochrane Database Syst Rev. 2014. PMID: 25271098 Free PMC article.
-
The future of Cochrane Neonatal.Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
-
Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials.Cochrane Database Syst Rev. 2014 Apr 29;2014(4):MR000034. doi: 10.1002/14651858.MR000034.pub2. Cochrane Database Syst Rev. 2014. Update in: Cochrane Database Syst Rev. 2024 Jan 4;1:MR000034. doi: 10.1002/14651858.MR000034.pub3. PMID: 24782322 Free PMC article. Updated. Review.
Cited by
-
Supporting study registration to reduce research waste.Nat Ecol Evol. 2024 Jun 5. doi: 10.1038/s41559-024-02433-5. Online ahead of print. Nat Ecol Evol. 2024. PMID: 38839851 Review.
-
Demystifying Open Science in health psychology and behavioral medicine: a practical guide to Registered Reports and Data Notes.Health Psychol Behav Med. 2024 May 29;12(1):2351939. doi: 10.1080/21642850.2024.2351939. eCollection 2024. Health Psychol Behav Med. 2024. PMID: 38817594 Free PMC article.
-
A systematic analysis of diagnostic performance for Alzheimer's disease using structural MRI.Psychoradiology. 2022 Mar 9;2(1):287-295. doi: 10.1093/psyrad/kkac001. eCollection 2022 Mar. Psychoradiology. 2022. PMID: 38665142 Free PMC article.
-
Linguistic spin in randomized controlled trials about age-related macular degeneration.Front Epidemiol. 2022 Oct 31;2:961996. doi: 10.3389/fepid.2022.961996. eCollection 2022. Front Epidemiol. 2022. PMID: 38455287 Free PMC article.
-
Primary outcome reporting in clinical trials for older adults with depression.BJPsych Open. 2024 Mar 7;10(2):e60. doi: 10.1192/bjo.2023.650. BJPsych Open. 2024. PMID: 38450491 Free PMC article. Review.
References
-
- Song F, Parekh S, Hooper L, Loke YK, Ryder J, et al... (2010) Dissemination and publication of research findings: an updated review of related biases.. Health Technol Assess 14. - PubMed
-
- Rothstein HR, Sutton AJ, Borenstein M (2005) Publication Bias in Meta-Analysis. Prevention, Assessment and Adjustments: Wiley.
-
- Dickersin K, Min YI (1993) NIH clinical trials and publication bias. Online J Curr Clin Trials Doc No 50. - PubMed
-
- Ioannidis JP (1998) Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 279: 281–286. - PubMed
Publication types
MeSH terms
Grants and funding
LinkOut - more resources
Full Text Sources
Other Literature Sources