Will Eliminating Questionable Research Practices Solve the Replication Crisis?

Posted: April 4th, 2019

Will Eliminating Questionable Research Practices Solve the Replication Crisis?. For this discussion we will explore some unsettled issues in Psychology. Each topic has a ‘yes’ side and a ‘no’ side, And this is where you should start.

Step 1: Read the textbook introduction on the issue, the textbook articles, and the ‘exploring the issue’ section for the issue.

Step 2: Do some of your own research to find support for the yes and no sides of the issue. Use the library databases (Medline, Psychinfo, etc.) and Google Scholar to find primary research journal articles, review journal articles, book chapters, etc. on the issue. The textbook provides some additional suggested resources but your own research should expand beyond this and will have the advantage of exploring recent research on the issue.

Step 4: Post your discussion answer. Your discussion answer should be at least four paragraphs with references and in text citations (in APA format): introduction, (yes) research summary, (no) research summary, and a conclusion paragraph. Everything should be in YOUR OWN WORDS. In the first paragraph, you will introduce the issue as you see it, including any definitions and/or critical points you think we need to consider. Then, you should summarize the research you found that supports the ‘yes’ side of the issue (including at least one source outside the textbook) and the research you found on the ‘no’ side of the issue (including at least one source outside the textbook). Finally, you should give us your conclusion – which side do you support based on the research? Is there common ground? Do you have any criticism on the research that’s been conducted?

Will Eliminating Questionable Research Practices Solve the Replication Crisis?
Introduction
There has been much debate in recent years about questionable research practices (QRPs) and their potential role in contributing to the replication crisis in psychology and other fields. QRPs refer to actions that, although possibly common in everyday research, deliberately or inadvertently increase the likelihood of obtaining statistically significant and potentially false-positive results (John et al., 2012). Some key examples of QRPs include selectively reporting only studies with significant results, continuing data collection until results are significant, and failing to report all measured variables or operationalizations. While individual instances of QRPs may seem minor, it has been argued that the widespread use of such practices could systematically bias the literature and contribute to difficulties replicating findings (Simmons et al., 2011). However, others argue that eliminating QRPs alone will not solve deeper methodological issues underlying replication problems. In this discussion, I will explore arguments on both sides of this issue and draw my own conclusions based on the available evidence.
Yes, Eliminating QRPs Could Help
Research suggests that QRPs are common in psychology and other fields. For example, in a survey of 2,000 psychologists, over half admitted to having failed to report all dependent measures or report unexpected findings of an exploratory nature at least once (John et al., 2012). Similarly, a meta-analysis found that studies with significant results were about twice as likely to be published as non-significant studies, suggesting publication bias (Franco et al., 2014). If QRPs systematically inflate the number of false-positive results in the literature, this could contribute to difficulties replicating those findings. As Simmons et al. (2011) argue, even small increases in false-positive rates due to QRPs could have major consequences when multiplied across thousands of researchers and studies. Eliminating QRPs could help “self-correct” the literature and improve replicability.
No, Deeper Issues Also Need Addressing

However, others argue that QRPs alone do not fully explain replication problems. Methodological and statistical issues may play a larger role. For example, low statistical power in psychology means that studies are underpowered to detect true effects, increasing chances of false negatives as well as false positives (Button et al., 2013). Additionally, many replications involve direct rather than conceptual replications, and operationalizations may differ in ways that influence outcomes (Schmidt, 2009). Even if QRPs were eliminated, these deeper issues could still contribute to replication difficulties. While transparency around QRPs is important, it may not be a panacea for replication problems without also addressing issues like power, precision in operationalizations, and directness of replication attempts (Lindsay, 2015).
Conclusion
In summary, while QRPs appear common and could inflate false-positive results, their elimination alone may not solve replication problems facing psychology and other fields. Deeper methodological issues related to power, precision, and replication design also need to be addressed. Greater transparency around research practices could help correct some biases in the literature. However, true progress on replication will also require efforts to improve power and standardize operationalizations in original studies as well as best practices for direct and conceptual replications. Overall, the evidence suggests that QRPs are part of a complex problem requiring attention to numerous factors.
References

Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org/10.1038/nrn3475
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953
Lindsay, D. S. (2015). Replication in Psychological Science. Psychological Science, 26(12), 1827–1832. https://doi.org/10.1177/0956797615616374
Schmidt, S. (2009). Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
In this discussion, I have explored arguments on both sides of the issue of whether eliminating questionable research practices (QRPs) alone could solve replication problems in psychology. While research suggests QRPs are common and could inflate false positives, methodological issues related to power, precision of operationalizations, and replication design may also contribute significantly to replication difficulties. Greater transparency around research practices could help correct some biases, but true progress will require attention to numerous factors through efforts to improve original studies as well as best practices for replication attempts. Overall, the evidence indicates QRPs are part of a complex problem with no single solution. Both eliminating questionable practices and addressing deeper methodological issues will be important for improving replicability.