LegiStorm

Summary:
What GAO Found

GAO found that machine learning, a type of artificial intelligence (AI) that uses algorithms to identify patterns in information, is being applied to forecasting models for natural hazards such as severe storms, hurricanes, floods, and wildfires, which can lead to natural disasters. A few machine learning models are used operationally—in routine forecasting—such as one that may improve the warning time for severe storms. Some uses of machine learning are considered close to operational, while others require years of development and testing.

GAO identified potential benefits of applying machine learning to this field, including:

Reducing the time required to make forecasts by replacing components of models that are slow and that increase the cost of modeling.
Increasing model accuracy by more fully exploiting available data, using other data that traditional models cannot, and creating synthetic data to fill gaps.
Reducing the uncertainty of model output by improving ensemble modeling—the processes of generating combined predictions from numerous models—and making better use of historical data.
Forecasting natural disasters using machine learning



GAO also identified challenges to the use of machine learning. For example:

Data limitations hamper the training of machine learning models and can reduce accuracy in some regions, such as rural areas where weather observations are sparse.
A lack of trust and understanding of the algorithms as well as concerns about bias can make forecasters and other users hesitant to use machine learning models.
Limited coordination and collaboration create challenges for fully developing some machine learning models. For example, some forecasters told us they lack opportunities to interact with researchers and convey their needs.
Workforce and resource gaps also create challenges. For example, the upfront costs to develop and run machine learning models are high, and some companies working on these models do not fully understand the data and phenomena they are modeling, according to academic researchers.
GAO identified five policy options that could help address these challenges. These options are intended to inform policymakers, including Congress, federal and state agencies, academic and research institutions, and industry of potential policy implementations. The status quo option illustrates a scenario in which government policymakers take no additional actions beyond current ongoing efforts.

Policy Options to Help Address Challenges to the Use of Machine Learning in Natural Hazard Modeling

Policy Option
Opportunities
Considerations
Facilitate improved data collection, sharing, and use (report p. 37).
Government policymakers could expand use of existing observational data and infrastructure to close gaps, expand access to certain data, and (in conjunction with other policymakers) establish guidelines for making data AI-ready.

Efforts to address gaps in data sets can improve machine learning model performance.
 Expanded access to existing data would improve the ability of researchers and groups to develop and test machine learning technologies.
Adopting standards for AI-ready data could reduce resources needed to curate data and facilitate more efficient modeling.


Expanding observational infrastructure can be expensive and could divert limited resources from other efforts.
Agencies need to weigh the benefits of greater data sharing against any increase in risks related to data security and privacy.
Strict data standards may slow research and innovation if they burden or constrain machine learning researchers.

Expand education and training (report p. 38).
Government policymakers could update education requirements to include machine learning-related coursework and expand learning and support centers, while academic policymakers could adjust physical science curricula to include more machine learning coursework.

Updating education requirements would better prepare students to use machine learning in government.
More robust education can better prepare both researchers and end users in fields like meteorology and climatology to develop and use machine learning.


Education and training reforms may need to be repeatedly adjusted, as technological change in this space can be rapid and unpredictable.
Establishing and expanding professional development and training opportunities throughout government may require substantial investment.

Address hiring and retention barriers and certain resource shortfalls (report p. 39).
Government policymakers could address pay scale limitations for positions that include machine learning expertise and work with private sector policymakers to expand the use of public-private partnerships (PPP).

Providing workforce incentives to government employees for machine learning development could allow agencies to bolster workforce capacities.
Expanding PPPs might help agencies overcome computational resource shortfalls and help industry draw on government expertise.


Increasing salary limits for some employees would require agency budget increases or cuts to other budget items.
Expanding the use of PPPs could magnify resource disparities between government and private industry.
PPPs that entail hosting government data on collaborator systems may pose security risks that would need to be considered and addressed.

Take steps to mitigate bias and foster trust in data and machine learning models (report p. 40).
Policymakers could establish efforts to better understand and mitigate various forms of bias, support inclusion of diverse stakeholders for machine learning models, and develop guidelines or best practices for reporting methodological choices.

Sustained efforts to address bias in data sets can reduce the likelihood of models negatively affecting certain communities.
Acquiring diverse stakeholder perspectives throughout machine learning models’ life cycles can help reduce certain types of bias in data and models.
Fostering machine learning model transparency could improve end-user and decision-maker trust.


Embedding efforts to address bias throughout the model life cycle may increase model costs and slow model development.

Maintain status quo efforts (report p. 41).
Government policymakers could maintain existing policy efforts and organizational structures, along with existing strategic plans and agency commitments.

Some agency efforts are underway to address the challenges described.


The extent to which agencies will meet their commitments under status quo efforts is unclear. Status quo efforts may not address challenges specific to the use of AI in natural hazard modeling.

Source: GAO. | GAO-24-106213

Why GAO Did This Study

Natural disasters cause on average hundreds of deaths and billions of dollars in damage in the U.S. each year. Forecasting natural disasters relies on computer modeling and is important for preparedness and response, which can in turn save lives and protect property. AI is a powerful tool that can automate processes, rapidly analyze massive data sets, enable modelers to gain new insights, and boost efficiency.

This report on the use of machine learning in natural hazard modeling discusses (1) the emerging and current use of machine learning for modeling severe storms, hurricanes, floods, and wildfires, and the potential benefits of this use; (2) challenges surrounding the use of machine learning; and (3) policy options to address challenges or enhance benefits of the use of machine learning.

GAO reviewed the use of machine learning to model severe storms, hurricanes, floods, and wildfires across development and operational stages; interviewed a range of stakeholder groups, including government, industry, academia, and professional organizations; convened a meeting of experts in conjunction with the National Academies; and reviewed key reports and scientific literature. GAO is identifying policy options in this report.

For more information, contact Brian Bothwell at (202) 512-6888 or bothwellb@gao.gov.