Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2012;7(10):e46363.
doi: 10.1371/journal.pone.0046363. Epub 2012 Oct 8.

Inappropriate fiddling with statistical analyses to obtain a desirable p-value: tests to detect its presence in published literature

Gary L Gadbury  1 , David B Allison
Affiliations

Inappropriate fiddling with statistical analyses to obtain a desirable p-value: tests to detect its presence in published literature

Gary L Gadbury et al. PLoS One. 2012.

Abstract

Much has been written regarding p-values below certain thresholds (most notably 0.05) denoting statistical significance and the tendency of such p-values to be more readily publishable in peer-reviewed journals. Intuition suggests that there may be a tendency to manipulate statistical analyses to push a "near significant p-value" to a level that is considered significant. This article presents a method for detecting the presence of such manipulation (herein called "fiddling") in a distribution of p-values from independent studies. Simulations are used to illustrate the properties of the method. The results suggest that the method has low type I error and that power approaches acceptable levels as the number of p-values being studied approaches 1000.

PubMed Disclaimer

Conflict of interest statement

Competing Interests: I have read the journal’s policy and have the following conflicts. Dr. Allison has, anticipates, or has had financial interests with the Frontiers Foundation; Vivus, Inc; Kraft Foods; University of Wisconsin; University of Arizona; Paul, Weiss, Wharton & Garrison LLP; and Sage Publications. This study was funded by Coca Cola. There are no declarations of employment, consultancy, patents, products in development or marketed products to declare. This does not alter our adherence to all the PLoS ONE policies on sharing data and materials, as detailed online in the guide for authors.

Figures

Figure 1
Figure 1. Boxplots of OBJ and DIFF for the conditions of fiddling or no fiddling.
Boxplots compare the distributions (from 1000 simulations) of comparison statistics from mixture models fitted to a distribution of p-values for which no fiddling has occurred (i.e., a level 2 null distribution) and to a distribution of p-values for which fiddling did occur. OBJ is the objective function calculated at maximum likelihood estimates of parameters of the model, and DIFF is a difference in the fitted model to 0.05 versus 0.1.
Figure 2
Figure 2. Steps A – F for the simulation procedure as described in section 5.3.
The steps are repeated 1000 times for various sample sizes given in step H from section 5.3. For the final step I in section 5.3, the dashed line above for step B is redirected to instead connect the P.alt P-values to fitting the mixture model.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Cowles M, Davis C (1982) On the origins of the 0.05 level of statistical significance. Am Psychol 37: 553–558.
    1. Rothstein HR, Sutton AJ, Borenstein M, editors (2005) Publication Bias in Meta-Analysis – Prevention, Assessment and Adjustments. Chichester: John Wiley & Sons.
    1. Emerson GB, Warme WJ, Wolf FM, Heckman JD, Brand RA, et al. (2010) Testing for the presence of positive-outcome bias in peer review. Arch Intern Med 170: 1934–1939. - PubMed
    1. Ridley J, Kolm N, Freckelton RP, Gage MJG (2007) An unexpected influence of widely used significance thresholds on the distribution of reported P-values. J Evol Biol 20: 1082–1089. - PubMed
    1. Stern JM, Simes RJ (1997) Publication bias: evidence of delayed publication in a cohort study of clinical research projects. Br Med J 315: 640–645. - PMC - PubMed

Publication types