Large-scale analysis of the quality of scientific reports

JMIR Publications recently published “Institutional Scoring with the Rigor and Transparency Index: Large-Scale Analysis of the Quality of Scientific Reporting” in the Journal of Internet Medical Research (JMIR)who reported that improving stringency and transparency measures should lead to improvements in reproducibility in the scientific literature, but evaluating transparency measures tends to be very difficult if performed manually by reviewers.

The general objective of this study is to establish a quality metric for scientific reports that can be used in institutions and countries, as well as to highlight the need for high-quality reports to guarantee replicability within biomedicine, making use of manuscripts from the Project of Reproducibility: Cancer Biology.

The authors discuss an enhancement to the Rigor and Transparency Index (RTI) presented above, which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using graded manuscripts based on criteria found in the reproducibility guidelines (eg. g., NIH, MDAR, ARRIVE).

Using the work of the Reproducibility Project: Cancer Biology, the authors were able to determine that the replication studies scored significantly higher than the original articles, which the project said all required additional information from the authors to begin replication efforts.

Unfortunately, the RTI measures for journals, institutions, and countries currently score lower than the replication study average. If they take the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure that the average manuscript contains enough information for replication attempts.

Dr. Anita Bandrowski of the University of California, San Diego said, “Research reproducibility is necessary for scientific progress. However, over the past decade, numerous reports of research irreproducibility have shed light on a problem. persistent, which is proving to be problematic and costly.

“Research reproducibility is necessary for scientific progress. However, over the past decade, numerous reports of research irreproducibility have shed light on a persistent problem, which is proving to be problematic and costly.”

In an effort to encourage reproducibility, numerous scientific organizations and journals have adopted the Promoting Transparency and Openness guidelines, which focus on establishing better practices at the individual journal level.

In a similar vein, the Publisher-Driven Materials Design, Analysis, and Reporting Framework is a multidisciplinary research framework designed to improve reporting transparency in life sciences research at the individual manuscript level.

This framework provides a checklist of minimal and consistent reporting whose criteria were used, in part, to create the first RTI, a daily quality metrics focused on research methodologies and reporting transparency.

Specifically, the authors here present the latest version of the RTI, which represents the average SciScore over a subset of articles, and demonstrate how it can be used to assess reporting transparency within research institutions.

While it is impractical to simply describe all ‘2’ articles as non-replicable and all ‘8’ articles as replicable, given the numerous fields and their subsequent best practices, it can be argued that the higher scores are associated with more methodological detail and as such are likely to be easier to use for attempting replication.

Provided by JMIR Publications