Failing replication studies in psychology often wrongfully blame contextual variation
Replication studies often fail to replicate findings of original studies, which is central to the ‘reproducibility crisis’ in science. One often heard explanation of this disparity in findings is that the effect under investigation is heterogeneous, that is, varying across contexts. However, most of these studied effects actually lack heterogeneity, say Tilburg University researchers who investigated them. Their study has just been published in the prestigious journal Psychological Bulletin.
Evidence for heterogeneity of effect sizes was studied when only minor changes to sample population and settings were made between studies. The research included all 68 pre-registered multi-lab direct replication projects in social and cognitive psychology up to now.
The researchers found only limited heterogeneity; most findings (72%) were most likely to have zero to small heterogeneity. This suggests that minor changes in sample population and settings are unlikely to affect research outcomes in these fields of psychology. Hence the often heard explanation ‘effects vary across contexts’ of the disparity of studies’ findings should not be the default explanation of this disparity, according to the researchers.
In addition, the research showed that showed that if the overall effect is zero (which means no effect of the treatment or no association between variables), then it is likely nowhere to be found when only changing sample population or settings. Only when the overall effect is non-zero, differences in findings of direct replications may be likely.
Publication:
Olsson-Collentine, A., van Assen, M. A. L. M., & Wicherts, J. M. (2020, June 30). Postprint - Heterogeneity in direct replications in psychology and its association with effect size.
Note to editors
For more information please contact Prof. Marcel van Assen at tel. +31 13 466 2362 / M.A.L.M.vanAssen@tilburguniversity.edu.