Decision-Making in Research Tasks with Sequential Testing

Background: In a recent controversial essay, published by JPA Ioannidis in PLoS Medicine, it has been argued that in some research fields, most of the published findings are false. Based on theoretical reasoning it can be shown that small effect sizes, error-prone tests, low priors of the tested hypotheses and biases in the evaluation and publication of research findings increase the fraction of false positives. These findings raise concerns about the reliability of research. However, they are based on a very simple scenario of scientific research, where single tests are used to evaluate independent hypotheses. Methodology/Principal Findings: In this study, we present computer simulations and experimental approaches for analyzing more realistic scenarios. In these scenarios, research tasks are solved sequentially, i.e. subsequent tests can be chosen depending on previous results. We investigate simple sequential testing and scenarios where only a selected subset of results can be published and used for future rounds of test choice. Results from computer simulations indicate that for the tasks analyzed in this study, the fraction of false among the positive findings declines over several rounds of testing if the most informative tests are performed. Our experiments show that human subjects frequently perform the most informative tests, leading to a decline of false positives as expected from the simulations. Conclusions/Significance: For the research tasks studied here, findings tend to become more reliable over time. We also find that the performance in those experimental settings where not all performed tests could be published turned out to be surprisingly inefficient. Our results may help optimize existing procedures used in the practice of scientific research and provide guidance for the development of novel forms of scholarly communication.

Tags
Data and Resources
To access the resources you must log in

This item has no data

Identity

Description: The Identity category includes attributes that support the identification of the resource.

Field Value
PID https://www.doi.org/10.1371/journal.pone.0004607
PID pmc:PMC2643008
PID pmid:19240797
URL https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0004607&type=printable
URL https://dash.harvard.edu/bitstream/handle/1/4506434/2643008.pdf?sequence=1
URL http://dx.doi.org/10.1371/journal.pone.0004607
URL http://dash.harvard.edu/handle/1/4506434
URL http://europepmc.org/articles/PMC2643008/
URL https://www.ncbi.nlm.nih.gov/pmc/articles/pmid/19240797/?tool=EBI
URL http://nrs.harvard.edu/urn-3:HUL.InstRepos:4506434
URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2643008/
URL http://europepmc.org/articles/PMC2643008
URL https://paperity.org/p/61491900/decision-making-in-research-tasks-with-sequential-testing
URL https://doaj.org/toc/1932-6203
URL https://ideas.repec.org/a/plo/pone00/0004607.html
URL https://pubmed.ncbi.nlm.nih.gov/19240797/
URL http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0004607
URL https://academic.microsoft.com/#/detail/2116399790
URL http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0004607
URL http://dx.plos.org/10.1371/journal.pone.0004607
Access Modality

Description: The Access Modality category includes attributes that report the modality of exploitation of the resource.

Field Value
Access Right Open Access
Attribution

Description: Authorships and contributors

Field Value
Author David Rand, 0000-0001-8975-2783
Author David Rand, 0000-0002-2821-468X
Author Thomas Pfeiffer, 0000-0002-0592-577X
Publishing

Description: Attributes about the publishing venue (e.g. journal) and deposit location (e.g. repository)

Field Value
Collected From PubMed Central; ORCID; UnpayWall; DOAJ-Articles; Crossref; Microsoft Academic Graph; Digital Access to Scholarship at Harvard
Hosted By Europe PubMed Central; PLoS ONE; Digital Access to Scholarship at Harvard
Publication Date 2009-01-01
Publisher Public Library of Science
Additional Info
Field Value
Country United States
Description Engineering and Applied Sciences
Description Psychology
Language English
Resource Type Article
keyword keywords.General Biochemistry, Genetics and Molecular Biology
keyword Q
keyword R
system:type publication
Management Info
Field Value
Source https://science-innovation-policy.openaire.eu/search/publication?articleId=dedup_wf_001::e1fe741fd029d962653fde6245d87f73
Author jsonws_user
Last Updated 26 December 2020, 21:45 (CET)
Created 26 December 2020, 21:45 (CET)