Introduction

In the late 1990s, the U.S. National Oceanic and Atmospheric Administration (NOAA) began publishing a tally of weather and climate disasters that each resulted in more than $1 billion in damage, noting that the time series had become “one of our more popular web pages”1. Originally, the data was reported in current-year U.S. dollars. In 2011, following criticism that the dataset was misleading, NOAA modified its methods to adjusted historical losses to constant-year dollars by accounting for inflation (https://www.washingtonpost.com/blogs/capital-weather-gang/post/2011-billion-dollar-weather-disaster-record-legit-or-bad-economics/2012/01/12/gIQADocztP_blog.html).

By 2023, the billion dollar disaster time series had become a fixture in NOAA’s public outreach, was highlighted by the U.S. government’s U.S. Global Change Research Program (USGCRP) as a “climate change indicator” (https://storymaps.arcgis.com/collections/ad628a4d3e7e4460b089d9fe96b2475d?item=1), was a cited as evidence in support of a “key message” of the Fifth U.S. National Climate Assessment showing that “extreme events are becoming more frequent and severe” (https://nca2023.globalchange.gov/chapter/2/). The time series is often cited in policy settings as evidence of the effects of human-caused climate change to increase the frequency and intensity of extreme weather events and associated economic damage, including in federal agencies, Congress and by the U.S. President (https://www.congress.gov/bill/118th-congress/house-bill/598/text; https://www.whitehouse.gov/briefing-room/statements-releases/2023/11/14/fact-sheet-biden-harris-administration-releases-fifth-national-climate-assessment-and-announces-more-than-6-billion-to-strengthen-climate-resilience-across-the-country). In addition to being widely cited in justifications of policy, as of March, 2024, NOAA’s billion dollar dataset has been cited in almost 1000 articles according to Google Scholar (https://scholar.google.com/scholar?hl=en&as_sdt=0%2C6&q=%22billion+dollar+disasters%22&btnG=).

This paper evaluates the billion dollar disaster time series by applying criteria of NOAA’s Information Quality and Scientific Integrity policies. The evaluation finds that billion dollar disaster time series fails to meet NOAA’s criteria for “information quality,” specifically, NOAA’s criteria of traceability, transparency, presentation, and substance.

Thus, the billion dollar disaster dataset is not simply an insufficient basis for claims of the detection and attribution of changes in climate variables (or a consequence of such changes), but the dataset is inappropriate for use in such research. Throughout, I use the terms “detection” and “attribution” as defined by the Intergovernmental panel on Climate Change (IPCC)2. Climate data should be the basis for claims of detection and attribution of changes in climate variables, not economic loss data. Because of the shortfalls in scientific integrity documented in this evaluation, policy makers and the public have been misinformed about extreme events and disasters in the United States.

Results

Evaluation of policy or program performance is among the most common and influential practices in applied policy research. Policy evaluation tells us if actions by government programs and agencies are meeting their stated goals and provides insight into reasons for successes and failures. As such, evaluation offers important input that empowers policy makers to correct course and supports efforts by the public to hold governments democratically accountable. A systematic evaluation includes four distinct intellectual tasks3,4: (a) identification of goals to be achieved, (b) metrics which can be used to assess progress (or lack thereof) with respect to goals, (c) data or evidence related to such metrics, and finally, if possible, (d) judgments of responsibility for observed outcomes.

NOAA’s billion dollar disaster time series is considered a “fundamental research communication” under the Public Communications order of NOAA’s parent agency, the Department of Commerce (https://www.osec.doc.gov/opog/dmp/daos/dao219_1.html). NOAA defines a “fundamental research communication” to be “official work regarding the products of basic or applied research in science and engineering, the results of which ordinarily are published and shared broadly within the scientific community” (https://www.noaa.gov/sites/default/files/legacy/document/2021/Feb/202-735-D.pdf). NOAA further identifies an important subset of “fundamental research communications” to be “influential information,” which “means information the agency reasonably can determine will have or does have a clear and substantial impact on important public policies or private sector decisions” (https://www.noaa.gov/organization/information-technology/policy-oversight/information-quality/information-quality-guidelines). The billion dollar disaster dataset is also what the Office of Management and Budget defines as “Influential Scientific Information” (https://www.govinfo.gov/content/pkg/FR-2005-01-14/pdf/05-769.pdf).

NOAA’s Information Quality and Scientific Integrity policies set forth the criteria to be used for evaluating “fundamental research communications,” including the subset of “influential information.” Specifically, NOAA’s Information Quality Guidelines identify three criteria of information quality: utility, objectivity, and integrity (https://www.noaa.gov/organization/information-technology/policy-oversight/information-quality/information-quality-guidelines).

Utility refers to “the usefulness of research to its intended users, including the public,” with an emphasis on “transparency.” NOAA’s Scientific Integrity Policy provides further guidance: “Transparency, traceability, and integrity at all levels are required” in order for the agency “to achieve” its mission (https://www.noaa.gov/sites/default/files/legacy/document/2021/Feb/202-735-D.pdf).

  • Traceability: “The ability to verify sources, data, information, methodology, results, assessments, research, analysis, conclusions or other evidence to establish the integrity of findings.”

  • Transparency: “Characterized by visibility or accessibility of information.”

Objectivity refers to presentation and substance:

  • Presentation: “includes whether disseminated information is presented in an accurate, clear, complete, and unbiased manner and in a proper context.”

  • Substance: “involves a focus on ensuring accurate, reliable, and unbiased information. In a scientific, financial, or statistical context, the original and supporting data shall be generated, and the analytic results shall be developed, using sound statistical and research methods.”

Integrity refers to “security ‑ the protection of information from unauthorized access or revision, to ensure that the information is not compromised through corruption or falsification.” Integrity will not be further considered as part of this evaluation.

NOAA’s Scientific Integrity Policy also states that it will “ensure that data and research used to support policy decisions undergo independent peer review by qualified experts” (https://sciencecouncil.noaa.gov/scientific-integrity-commons/sic-integrity-policy/). OMB requires that agencies develop “a transparent process for public disclosure of peer review planning, including a Web-accessible description of the peer review plan that the agency has developed for each of its forthcoming influential scientific disseminations” (https://www.govinfo.gov/content/pkg/FR-2005-01-14/pdf/05-769.pdf). There is no such plan in place for the NOAA “billion dollar” dataset and the methods, which have evolved over time, and results have not been subject to any public or transparent form of peer review.

The evaluation conducted here thus focuses on traceability and transparency (as elements of utility) and presentation and substance (as elements of objectivity).

Traceability and transparency

The NOAA billion dollar disaster dataset is intransparent in many ways, including its sources, input data and methodologies employed to produce results. The intransparency includes elements of event loss estimation, additions to and subtractions of events from the database, and adjustments made to historical loss estimates. There have been an unknown number of versions of the dataset, which have not been documented or made publicly available. Changes are made to the dataset more frequently than annually, suggesting that there have been many dozens of versions of the dataset over the past decades. Replication of the dataset or changes made to it is thus not possible by any independent researcher, as is verification or evaluation of the dataset itself.

Seven examples illustrate the lack of transparency and lack of traceability.

First, NOAA states that it utilizes more than “a dozen sources” to “help capture the total, direct costs (both insured and uninsured) of the weather and climate events” (https://www.ncei.noaa.gov/access/billions/faq). However, NOAA does not specifically identify these sources in relation to specific events, how its estimates are derived from these sources, or the estimates themselves. Almost all data sources that NOAA cites that it relies on for loss estimates are public agencies that produce data released to the public. Insured losses for specific events are aggregated and typically made available to the public, such as by the Florida Office of Insurance Regulation (https://www.floir.com/home). Aggregated data provides no information on specific businesses or individuals.

NOAA also states that it includes in it loss estimates various indirect losses such as business interruption, wildfire suppression and others. NOAA does not provide the data or methods for its estimation of such indirect losses. Smith and Matthews5 (who also have created and maintained the dataset as NOAA employees) also identify livestock feeding costs as a function of national feedstock trends as a variable used in compiling the dataset. Livestock feeding costs are not considered a disaster cost in conventional disaster accounting methods (such as by NOAA Storm Data or SHELDUS), as these are not direct losses due to a local or regional extreme event, but rather an estimate of national market changes in commodity prices which are influenced by many more factors than an extreme event. It is unclear what other measures of indirect costs are included in the NOAA tabulation.

Second, consider the case of Hurricane Idalia, which made landfall in the Big Bend Region of Florida in late September 2023. Initial catastrophe model estimates suggested insured losses of $2.5 to 5 billion (https://www.insurancejournal.com/news/national/2023/09/05/738970.htm). The initial NOAA estimate reported on its billion dollar disaster website in the immediate aftermath of the storm was $2.5 billion. However, actual insured losses have been far less than was estimated in the storm’s aftermath, totaling officially about $310 million through mid-November 2023 (https://www.floir.com/home/idalia). The historical practice of NOAA’s National Hurricane Center for estimating total direct hurricane damage was to double insured losses to arrive at an estimate of total direct losses6. Even accounting for some additional insurance claims to be made, it is unlikely that Idalia would reach $1 billion in total direct losses under the NHC methodology. Yet by December 2023 NOAA had increased its loss estimate for Idalia to $3.6 billion. What is the basis for NOAA’s estimate of Idalia’s total losses being ~12 times insured losses? That is unknown.

Third, similarly unknown is why historical events are periodically added and removed from the dataset. For instance, from a version of the dataset available in December 2022 to an update published in July 2023, 10 new events were added and 3 were deleted (Fig. 1). A later comparison with yet another version of the dataset indicates 4 additional historical events were added (not shown in Fig. 1). There is no documentation or justification for such changes, I am only aware of them through the happenstance of downloading the currently available dataset at different times.

Fig. 1
figure 1

Undocumented changes to disaster counts made by NOAA between two different versions of the billion dollar disaster dataset, one downloaded in 2022 and another in 2023.

Full size image

Fourth, a comparison of event loss estimates from the 2022 dataset and the 2023 version shows that each individual event has been adjusted by a different amount. According to NOAA, the only annual adjustment acknowledged is for inflation based on the Consumer Price Index (CPI). From 2022 to 2023, most of the adjustments made to individual events are between 4.5% and 6% but nine events are adjusted from 6.6% to 145%, and one is a reduction of about 75%. An annual adjustment for CPI should be constant across all events. No documentation is provided to explain these various adjustments and why they are unique to each event.

Fifth, NOAA states that they perform “key transformations” of loss data estimates by “scaling up insured loss data to account for uninsured and underinsured losses, which differs by peril, geography, and asset class.” NOAA makes no details available on the methodology or basis for such transformations, nor their impact on loss estimates, nor how these transformations may change over time.

Similarly, Smith and Matthews5 reference an overall bias correction that has been applied to the dataset, as well as an additional correction for crop insurance losses. Smith and Katz6 reference other adjustments, such as an adjustment to U.S. flood insurance participation rates, but neither the methodologies nor results of these various adjustments are documented, nor has the baseline data to which the adjustments are applied. Table 3 from Smith and Katz7 suggests an open-ended formulaic approach to loss estimation, but none of the data that would be used in such formulas is available. Nor is it clear that NOAA currently applies the formula to loss estimation. If so, it should be straightforward to provide sources, data and methods for each iteration of the dataset.

Sixth, the number of smaller disasters ranging from $1 to $2 billion was fairly constant from 1980 to 2007 and then sharply increased starting in 2008 (Fig. 2). NOAA states that “we introduce events into the time series as they “inflate” their way above $1B in costs in today’s dollars. Every year, this leads to the introduction of several new events added from earlier in the time series” (https://sciencecouncil.noaa.gov/scientific-integrity-commons/sic-integrity-policy/). However, the December 2023 dataset shows a net change of zero events from $1-2 billion for the period of 1980–2000 and a net increase of such 2 events from 2001–2023. NOAA’s statement that it elevates disasters from <1 billion in losses to the billion dollar disaster database also indicates that NOAA has another dataset with sub-billion dollar events that is not publicly available.

Fig. 2
figure 2

Increasing disaster counts costing $1-2 billion in a version of NOAA’s 2023 dataset.

Full size image

The sharp discontinuity in the counts of $1-2 billion events starting in 2008 is suggestive of a change in disaster accounting methods, however, the lack of transparency into the creation of the dataset makes it impossible to know the reasons that may underlie this discontinuity.

Seventh, a comparison of 2023 CPI-adjusted official losses of NOAA’s National Hurricane Center (NHC)20 to the loss estimates of the 2023 NOAA billion dollar dataset (BDD), for significant hurricanes shows large differences (Table 1).

Table 1 A comparison of NOAA Billion Dollar Disasters (BDD) CPI-adjusted billion dollar disaster estimates for certain significant hurricanes, 1983 to 2017, with CPI-adjusted estimates from NOAA NHC for the same storms
Full size table

The NOAA billion dollar disaster estimates are in all cases except Hurricane Andrew substantially higher than the CPI-adjusted estimates based on the official estimates of NHC. There is no obvious pattern to the differences and the lack of methodological and data transparency makes it impossible to understand why there are such large differences and why these differences vary by such a great deal.

These seven examples indicate clearly that the NOAA billion dollar dataset fails with respect to NOAA’s scientific integrity criteria of traceability and transparency. The many issues and questions raised above cannot be answered because it is impossible to verify sources, data or methodology to establish the integrity of findings. These seven examples are just a small subset of issues that I have raised in public forums about the provenance, methods, and publicly communicated results of the application of these methods. The billion dollar dataset thus does not meet NOAA’s requirement that data be transparent and traceable.

Presentation and substance

Even in the absence of the issues documented above, the NOAA billion dollar disaster dataset is potentially misleading, because it has been represented by NOAA and U.S. government officials as evidence of the detection of trends in extreme weather phenomena and the attribution of those trends to human-caused climate change due to the emission of greenhouse gases.

For instance:

The point here is not to call into question the reality or importance of human-caused climate change – it is real, and it is important. Rather, the question is whether the NOAA billion dollar disaster time series provides evidence of detection or attribution of changes in the climate of extreme weather events in the United States, as frequently claimed.

Economic loss data is not suitable for detection and attribution of trends in extreme weather events because losses involve more than just climatic factors. It is well understood that a disaster occurs at the intersection of an extreme event and a vulnerable and exposed society (IPCC)8. NOAA acknowledges that a combination of risk, vulnerability and exposure is necessary for a disaster to occur (https://www.ncei.noaa.gov/access/billions/faq), but it fails to take any of these factors into account in its methodologies prior to making claims of detection and attribution. Of note, NOAA performs such a GDP normalization for disasters at the state level but does not do so for its national billion dollar disaster database. In a June, 2023 insurance industry Webinar, the lead scientist responsible for the NOAA dataset identified the absence of a national GDP-based normalization to be a major challenge for interpreting the database, and suggested that this would be added to the dataset in the future (https://www.catmanagers.org/event-details/put-past-losses-in-their-proper-context-1). Smith and Katz7 explain that “the billion-dollar dataset is only adjusted for the CPI over time, not currently incorporating any changes in exposure (e.g., as reflected by shifts in wealth or population)”.

Over time, population and wealth have increased dramatically in the United States (and globally), meaning that when an extreme climate or weather event occurs, there is more to be damaged and invariably, more damage occurs even if there is no underlying trend in the frequency or intensity of extreme weather. Consequently, there is a large literature that seeks to “normalize” historical loss data to account for changes in exposure and vulnerability (e.g., a recent literature review identified more than 60 such papers9, other relevant studies discuss the importance of the spatial dimensions of land use change10,11,12,13).

A common approach to disaster normalization adjusts historical losses based on GDP, as a proxy for increasing population and wealth14,15,16,17,18. Figure. 3 shows loss per disaster in the NOAA 2023 dataset as a percentage of US GDP (https://fred.stlouisfed.org/series/RGDPNAUSA666NRUG). According to a simple linear trend, losses per disaster are down by about 80% since 1980, as a proportion of GDP. This is likely due to a combination of actual decreasing losses as a proportion of GDP, as has been documented in many rich countries, as well as the sharp increase in small disasters included in NOAA’s dataset (see Fig. 2).

Fig. 3
figure 3

Losses per disaster in NOAA’s billion dollar disaster dataset (the version downloaded in July 2023), 1980 to 2022.

Full size image

In comparison, weather and climate disasters losses as a percentage of U.S. GDP, show no increase over the period of record, which is 1990–2019 based on these data (Fig. 4).

Fig. 4: United States weather-related catastrophe losses as a percentage of U.S. GDP, 1990–2019.
figure 4

Sources: Spatial Hazard Events and Losses Database for the United States (SHELDUS) at Arizona State University, which has made public aggregate losses from 1990 to 2019. Data on GDP from the U.S. Office of Management and Budget.

Full size image

Other, more sophisticated and granular approaches to the normalization of U.S. weather and climate related disaster losses robustly confirm the aggregate downward trend in losses, once population growth and wealth are properly accounted6,18,19,20,21,22. Hurricane, flood and tornado losses have all decreased as a proportion of GDP on climate time scales, and as these are responsible for the majority of direct losses, so too have aggregate disaster losses.

NOAA’s failure to consider changes in exposure and vulnerability is significant. Consider for example Hurricane Andrew in 1992. The NOAA 2023 billion dollar disaster loss estimate for Andrew is $58.9 billion, but a 2023 normalized loss estimate is more than twice that at $119.9 billion (based on Weinkle et al.). For comparison, in 2022, Swiss Reinsurance estimated that a recurrence of Hurricane Andrew would result in $120 billion in total damage (https://www.abcactionnews.com/news/price-of-paradise/experts-say-modern-day-hurricane-andrew-could-cost-florida-120-billion). Thus, these estimates differ by ~100%.

By adjusting for inflation, but not for changes in exposure or vulnerability, the NOAA billion dollar dataset introduces a bias into the time series, as the upwards trend in losses in the billion dollar disaster time series is a result of growth in population and wealth, and not climate trends. As Smith and Katz7 acknowledged more than a decade ago of the increase in billion dollar disasters, “the magnitude of such increasing trends is greatly diminished when applied to data normalized for exposure.”

Thus, any claim that the NOAA billion dollar disaster dataset indicates worsening weather or worsening disasters is incomplete at best and misleading at worst. When U.S. disaster losses are considered in the context of exposure changes it becomes clear that as the absolute costs of disasters has increased, the impact relative to the economy has diminished over past decades, which is exactly the opposite of claims made by NOAA, the U.S. National Climate Assessment, the USGCRP, and the president of the United States, among many others.

The most appropriate data for investigating detection and attribution of changes in climate variables will always be climate data, and not economic data. IPCC has assessed research on the detection and attribution of trends in extreme weather events and has only low confidence in the emergence of signals of climate-impact drivers for river floods, heavy precipitation and pluvial flood, landslide, drought, fire weather, tropical cyclones, hail, severe wind storms and heavy snowfall2 – that is, each of the elements of the billion dollar disaster dataset. The IPCC does express confidence in some regions in the detection and attribution of changes in heat extremes and in extreme precipitation2, neither of which is an element of the billion dollar disaster database. The IPCC is explicit in warning against conflating changes in extreme precipitation with changes in pluvial flooding2.

NOAA makes strong claims of detection and attribution contrary to the conclusions of the IPCC but provides no analyses in support of these claims. For instance, NOAA states of its time series:

“The increases in population and material wealth over the last several decades are an important factor for higher damage potential. These trends are further complicated by the fact that many population centers and infrastructure exist in vulnerable areas like coasts and river floodplains, while building codes are often insufficient in reducing damage from extreme events. Climate change is also playing a role in the increasing frequency of some types of extreme weather that lead to billion-dollar disasters.”

However, NOAA makes no effort to quantify the roles of increasing population and material wealth, nor does it substantiate its claims that climate change has increased the frequency of some types of extreme weather.

NOAA does not acknowledge a large literature on disaster “normalization” that seeks to quantify the roles of population, material wealth, mitigation, building practices, etc. on increasing losses and also ignores literature on the detection and attribution of trends in various forms of extreme weather2,9.

Thus, any claim that the NOAA billion dollar disaster dataset indicates the detection trends in climate variables and the attribution of those trends to human-caused climate change is contrary to the most recent assessment of the IPCC. NOAA has provided no evidence or research to support claims that human-caused changes in climate are driving the increase in billion dollar disaster counts. Similarly, the opposite claim, that increasing billion dollar disasters are evidence of changes in the frequency of some extreme events resulting from human-caused climate change is also unsupported. NOAA’s claims are also circular – one claim is that climate change causes increasing billion dollar disasters and the second claim is that increasing billion dollar disasters indicate climate change. The billion dollar dataset fails to meet NOAA’s criteria of presentation and substance.

To summarize: the NOAA billion dollar disaster dataset falls short of NOAA’s guidelines for scientific integrity. The shortfalls documented here are neither small nor subtle. They represent a departure from NOAA’s long-term history of scientific integrity and excellence, which has saved countless lives and supported the nation’s economy.

Discussion

Identifying the reasons why NOAA’s billion dollar disaster dataset has departed so significantly from the agency’s own standards of scientific integrity goes well beyond the scope of this paper. However, the steps necessary to bring the dataset back into conformance with NOAA’s information quality criteria are straightforward (https://www.noaa.gov/organization/information-technology/policy-oversight/information-quality/information-quality-guidelines):

  • Publish all data, including all versions of the dataset;

  • Document and publish baseline loss estimates and their provenance;

  • Clearly describe all methodologies employed to adjust baseline data;

  • Document every change made to the dataset, give each successive version of the dataset a unique name, and publish all version of the data;

  • Maintain all historical versions of the dataset in a publicly accessible archive;

  • Subject the methods and results to annual peer review by experts, including economists and others with subject matter expertise, who are independent of NOAA. Make the peer review reports public;

  • Align NOAA’s practices with federal government policies for disseminating statistical information that are applied to other agencies (https://www.federalregister.gov/documents/2002/06/04/02-13892/federal-statistical-organizations-guidelines-for-ensuring-and-maximizing-the-quality-objectivity);

  • Align claims with IPCC methods and standards for any claims of detection and attribution, or justify why the claims are at odds with those of the IPCC.

NOAA is a crucially important agency that sits at the intersection of science, policy and politics. It has a long and distinguished history of providing weather, climate, water, ocean and other data to the nation. These data have saved countless lives, supported the economy and enabled significant scientific research. The agency is far too important to allow the shortfalls in scientific integrity documented in this paper to persist. Fortunately, science and policy are both self-correcting.

Methods

Policy evaluation

The analysis in this paper follows the logic of policy evaluation, which compares policy implementation with respect to criteria, with a goal of identifying progress or lack thereof towards goals (sources). Identifying progress requires identification of specific metrics of progress and data relevant to those metrics.