How to improve psychological science
Psychology is facing a 'replication crisis'. Many psychological findings could not be replicated in novel samples, which lead to the growing concern that many published findings are overly optimistic or even false. In her dissertation, Michèle Nuijten investigated potential indicators of problems in the published psychological literature and came up with some solutions.
One suggested solution to this problem is sharing data. If we share data, or perhaps make data sharing compulsory, we create the opportunity to run analyses again.
Video-interview with Michèle Nuijten, associate professor Methodology and Statistics and Hylke Annema, head Research Support
Use the free tool Statcheck
Most conclusions in psychology are based on statistics, so it is important that statistical results are reported correctly. Nuijten looked at inconsistencies in the statistical results in published psychology papers. To facilitate her research, she developed the free tool statcheck; a 'spellchecker' for statistics. Using statcheck, Nuijten found that roughly half of the articles published in psychology contained at least one inconsistency. Moreover, in one in eight papers she found at least one gross inconsistency that may have affected the statistical conclusion. Against her expectations, she did not find evidence that articles that shared their raw data had a lower probability of inconsistencies. To prevent statistical reporting inconsistencies, Nuijten advises editors of scientific journals to use statcheck to check submissions for any potential errors.
Success stories are overestimated
Statistical reporting inconsistencies are only one aspect of the problems currently affecting psychological science. One other major problem is publication bias: studies that find statistically significant effects have a higher chance of being published than studies that do not.
Michèle Nuijten: "When only the 'success stories' are published, we get a biased view of the effects in the scientific literature. Counterintuitively for most researchers, this problem is not solved when you combine information from multiple, comparable studies. On the contrary, in the presence of publication bias, the overestimation of effects can even become worse if you combine studies."
Indeed, Nuijten analyzed studies from the social sciences in general and from intelligence research, and found strong evidence that published effects are overestimated. What is more, in intelligence research the sample sizes are systematically too small. In our analyses She could not identify any specific study characteristics that are related to a stronger overestimation of effects.
Solutions: preregistration, encourage replication and share data
In this dissertation Nuijten found evidence for a high prevalence of statistical reporting inconsistencies and overestimated effects in psychological science. These are worrying findings, and it is important to think about concrete solutions to improve the quality of psychological research. One of the solutions is preregistration: researchers publish their entire research plan online, before they start collecting data. This way, studies that did not find any effects can not disappear into the file drawer unnoticed, and publication bias is countered.
Furthermore, replication research should be encouraged, to ensure that overestimated effects are corrected in the long run. Finally, researchers should more often share raw data and research materials, so that any errors can more easily be detected and corrected.
International Data Seal of Approval
In addition, Tilburg University offers Dataverse, a data repository for researchers to store their data and other materials like questionnaires and syntaxis for future safekeeping. Tilburg University is the first Dutch university awarded the Data Seal of Approval, an international mark of approval indicating compliance with strict quality demands, protocols and procedures.
Hylke Annema, head Research Support at Tilburg University: "Dataverse complies with the international FAIR standards, stating data should be Findable to everyone in the world, Accessible, Interoperable using different systems and Re-usable. We guarantee that for the next 25 years."
Michèle Nuijten: "In the end it all revolves around the question: how can we improve the quality of psychological research? To select the best strategies to do that, we need research on research: meta-research. If we use scientific methods to study how we can improve science, I predict a great future for psychology."
Michèle Nuijten recently defended her PhD-thesis, entitled Research on Research, a metascientific study of problems and solutions in psychological science.
Press release PhD Defence Michèle Nuijten (in Dutch)