Sample-Size Planning for More Accurate Statistical Power: A Method Adjusting Sample Effect Sizes for Publication Bias and Uncertainty

SUMMARY DOCTORAL DISSERTATION: Psychology is facing a “replication crisis”. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In this dissertation, we investigated potential indicators of problems in the published psychological literature. In Part I of this dissertation, we looked at inconsistencies in reported statistical results in published psychology papers. To facilitate our research, we developed the free tool statcheck; a “spellchecker” for statistics. In Part II, we investigated bias in published effect sizes. We showed that in the presence of publication bias, the overestimation of effects can become worse if you combine studies. Indeed, in meta-analyses from the social sciences we found strong evidence that published effects are overestimated. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. Some of the solutions we propose are preregistration, replication, and transparency. We argue that to select the best strategies to improve psychological science, we need research on research: meta-research.

These results suggest that cognitive training procedures based on dot-probe paradigms are probably not adequate to modify attentional patterns in individuals with depressive symptoms. The theoretical and practical implications are discussed.

There is currently a replication crisis in many fields of neuroscience and psychology, with some estimates claiming up to 64% of research in psychological science is not reproducible. Three common culprits which have been suspected to cause the failure to replicate such studies are small sample sizes, "hypothesizing after the results are known," and "p-hacking." Here, we introduce accurate stimulus timing as an additional possibility. Accurate stimulus onset timing is critical to almost all psychophysical research. Auditory, visual, or manual response time stimulus onsets are typically sent through wires to various machines that record data such as: eye gaze positions, electroencephalography, stereo electroencephalography, and electrocorticography. These stimulus onsets are collated and analyzed according to experimental condition. If there is variability in the temporal accuracy of the delivery of these onsets to external systems, the quality of the resulting data and scientific analyses will degrade. Here, we describe an approximately $200 Arduino based system and associated open-source codebase which achieved a 5.34 microsecond delay from the inputs to the outputs while electrically opto-isolating the connected external systems. Using an oscilloscope, the device is configurable for different environmental conditions particular to each laboratory (e.g. light sensor type, screen type, speaker type, stimulus type, temperature, etc). This low-cost open-source project delivered electrically isolated stimulus onset Transistor-Transistor Logic triggers with a median precision of 5.34 microseconds and was successfully tested with 7 different external systems that record eye and neurological data. EEG | SEEG | ECoG | fMRI | TTL | Opto-isolation | Stimulus Tracking | Psychophysics | DIY | Open-source | Arduino | MultimodalC. Davis and J. Martin were with the