LegiStorm

Summary:
What GAO Found

The National Institutes of Health (NIH), the National Science Foundation (NSF), and the National Aeronautics and Space Administration (NASA) are the three largest federal funders of basic scientific research in the United States. According to leading experts GAO interviewed, these agencies could do more to increase the rigor and transparency of the research they fund by taking actions to better align awards and recognition for researchers with more rigorous and transparent research practices. Experts suggested, for example, that agencies could incentivize or mandate that researchers preregister their studies as a means to share their research plans before the research is conducted. Doing so would enable other researchers to comment on and strengthen the methodology and analysis plans. Experts further suggested that agencies help improve standards for data repositories where research data are stored publicly, encourage the publication of null research results, and support training in statistical analysis and study design. Although the scientific community has developed many such practices to enhance research reliability, GAO found that they are not widely adopted because of researcher misconceptions and misaligned incentives in funding and publishing, among other things.

Role of Rigor and Transparency in Research Reliability



NIH, NSF, and NASA have taken steps to promote and support additional rigor and transparency in research, such as establishing requirements for researchers to disclose research results and associated data publicly. However, these agencies largely rely on grant application reviews and the prepublication peer review process to help ensure research rigor. GAO found that these agencies do not evaluate the rigor and transparency of the research they fund to help identify strategies for improvement. Specifically, they do not collect indicators of rigorous study design and transparency of research results such as study sample size, adherence to research plans, or the extent to which research data are findable, accessible, and usable. As a result, the agencies lack information to support changes to the grant making process and research funding priorities. Federal guidance and Standards for Internal Control in the Federal Government call for agencies to prioritize making federally funded research more rigorous and transparent and to use quality information to achieve agency objectives. Without this information on the research they fund, agencies are limited in their ability to take effective actions to improve research reliability, like those the experts described to GAO.

Why GAO Did This Study

In 2019, the U.S. government funded more than $42 billion in basic scientific research across a wide range of scientific disciplines. Unsuccessful attempts to reproduce and replicate research results have been documented across many scientific disciplines, including those funded by NASA, NIH, and NSF. The scientific community has expressed concern over the difficulty of replicating prior research results.

GAO was asked to review strategies to improve the reliability of federally funded research. Among other things, this report (1) examines what actions, according to experts, federal agencies could take to foster rigor and transparency in the research they fund; and (2) assesses the extent to which selected federal science funding agencies have taken actions to improve rigor and transparency. GAO conducted a literature review; reviewed NIH, NSF, and NASA documents; and conducted four roundtable discussions with 22 experts. GAO also interviewed agency officials as well as stakeholders from academia, professional societies, publishing, and other parts of the scientific community.