A recent report by the National Urban Area Initiative Association attempts to demonstrate that the Urban Area Security Initiative (UASI) grant program, created in 2003, is an effective homeland security program. Administered by the Federal Emergency Management Agency (FEMA), the UASI program allocates grant funding to help high-risk, high-density urban areas develop the capacity “to prevent, protect against, respond to, and recover from acts of terrorism.” While the National UASI Association report attempts to demonstrate the effectiveness of UASI grants, the report’s findings must be interpreted with caution. First, the report’s methodology has significant shortcomings. Second, the report cannot be viewed as an independent, objective assessment of the grant program’s effectiveness.
Methodology problems. To demonstrate effectiveness, an appropriate methodology would measure the net impact of the grants using control groups while controlling for alternative explanations. Such a methodology is hard to implement given the nature of the grants. One would have to compare outcomes of grant-funded areas to a control group consisting of non-grant-funded areas. In addition, one would have to rule out alternative explanations, such as the effect of other homeland security grant programs on the outcomes being assessed. While the nature of the UASI grants make rigorous scientific evaluations easier said than done, the report falls short of providing credible evidence for the effectiveness of UASI grants.
Measuring results. Too frequently, the report asserts that the UASI program is effective without providing any quantifiable outcome measures. For example, the report asserts that the “UASI program is enhancing regional collaboration and coordination” without providing any quantitative evidence to support the conclusion. How much has collaboration and coordination increased in areas receiving funding? Other than stating that there has been an increase, the report does not provide any outcome measures. In a 2009 report that contradicts this finding, the Government Accountability Office (GAO) concluded that FEMA “does not have measures to assess how UASI regions’ collaborative efforts have built preparedness capabilities”—the primary goal of the program.
Money does not equal effectiveness. Similarly, the report concludes that the “UASI program has been essential to strengthening security at critical infrastructure across the Nation.” The report mentions that 67 percent of urban areas receiving funding use UASI grants for critical infrastructure activities. However, this finding does not mean that the grants were effectively used. Like too many other parts of the report, the amount of federal taxpayer dollars spent is equated with effectiveness. Unfortunately, the amount of money spent does not provide any assessment of how the grants have strengthened the protection of critical infrastructures.
Further, the report asserts that the “value and effectiveness” of the UASI grants “can literally be measured in lives saved” without, ironically, measuring or estimating the number of lives saved as a result of the program. Instead, the report credits the UASI program with the dramatic increase in urban search-and-rescue teams since 2001. While the number of rescue teams has increased, the report does not determine how much of the increase resulted from the grants. There is no doubt that the expansion of these rescue teams is essential for responding to disasters. However, the report fails to demonstrate how much of the increase resulted from the UASI program. Have other national, state, or local funding sources contributed to the expansion?
Objectivity (or lack thereof). While the report does provide some useful information, it cannot be considered an objective, independent assessment of effectiveness. The National UASI Association is a lobbying organization set up by UASI grantees to encourage more spending on the UASI program. Promoting their own self-interest, special interest groups rarely, if at all, produce reports that are not self-flattering.
