What GAO Found
The Veterans Benefits Administration (VBA)—within the Department of Veterans Affairs—measures and reports the accuracy of its disability compensation claim decisions in two ways: (1) by claim and (2) by disabling condition, though its approach has limitations. When calculating accuracy rates for either measure through its Systematic Technical Accuracy Review (STAR), VBA does not always follow generally accepted statistical practices, resulting in imprecise performance information. For example, VBA does not adjust its accuracy estimates to reflect that it samples the same number of claims for review from each regional office—despite their varying workloads—and thus produces imprecise estimates of national and regional accuracy. Further, VBA reviews about 39 percent (over 5,000) more claims nationwide than is necessary to achieve its desired precision in reported accuracy rates, thereby diverting limited resources from other important quality assurance activities, such as targeted reviews of error-prone cases. In addition to issues with its statistical practices, VBA's process for selecting claims for STAR review creates an underrepresentation of claims that are moved between regional offices, which may inflate accuracy estimates because these claims have had historically lower accuracy rates. Finally, VBA has not clearly explained in public reports the differences in how its two accuracy measures are calculated or their associated limitations, as suggested by best practices for federal performance reporting.
VBA has taken steps to enhance and coordinate its other quality assurance activities, but GAO found shortcomings in how VBA is implementing and evaluating these activities. To improve local accuracy, VBA created regional office quality review teams (QRTs) with staff dedicated primarily to performing local accuracy reviews. QRTs assess individual claims processor performance and conduct special reviews to forestall certain types of errors. In addition, VBA began using questionnaires for assessing decision-making consistency, which are more efficient to administer than VBA's prior approach to conducting consistency studies. VBA also coordinates quality assurance efforts by disseminating national accuracy and consistency results, trends, and related guidance to regional offices for use in training claims processors. Further, VBA uses STAR results to inform other quality assurance activities, such as focusing certain QRT reviews on commonly made errors. However, GAO identified implementation shortcomings that may detract from the effectiveness of VBA's quality assurance activities. For example, contrary to accepted practices for ensuring the clarity and validity of questionnaires, VBA did not pre-test its consistency questionnaires to ensure the clarity of questions or validity of the expected results, although VBA officials indicated that they plan to do so for future questionnaires. In contrast with federal internal control standards that call for capturing and distributing information in a form that allows people to efficiently perform their duties, staff in the four regional offices that we visited had trouble finding the guidance they needed to do their work, which could affect the accuracy as well as the speed with which staff decide claims. Federal standards also call for knowing the value of efforts such as quality assurance activities and monitoring their performance over time; however, VBA has not evaluated the effect of its special QRT reviews or certain consistency studies on improving targeted accuracy rates, and lacks clear plans to do so.
Why GAO Did This Study
With a backlog of disability compensation claims, VBA faces difficulties in improving the accuracy and consistency of the claim decisions made by staff in its 57 regional offices. To help achieve its goal of 98 percent accuracy by fiscal year 2015, VBA recently implemented a new way of measuring accuracy and changed several quality assurance activities to assess the accuracy and consistency of decisions and to provide feedback and training to claims processors. GAO was asked to examine VBA's quality assurance activities.
This report evaluates (1) the extent to which VBA effectively measures and reports the accuracy of its disability compensation claim decisions and (2) whether VBA's other quality assurance activities are coordinated and effective. GAO analyzed VBA claims and STAR accuracy data from fiscal year 2013 (the most recent fiscal year for which complete data are available); reviewed relevant federal laws, VBA guidance, and other documents relevant to quality assurance activities; and interviewed VBA staff from headquarters and four VBA regional offices (selected to achieve variety in geography, workload, and accuracy rates), as well as veteran service organization officials.
What GAO Recommends
GAO is making eight recommendations to VA to improve its measurement and reporting of accuracy, review the multiple sources of policy guidance available to claims processors, enhance local data systems, and evaluate the effectiveness of quality assurance activities. VA concurred with all of GAO's recommendations.
For more information, contact Daniel Bertoni at (202) 512-7215 or bertonid@gao.gov.
