Fourth Validation Compendium Report

Executive Summary

National Audit Office Value for Money Report

1. Public Service Agreements (PSAs) were first introduced in 1998. They were introduced to improve the accountability of Government spending by defining the key outcomes expected for a given level of investment.
   
2. In the Spending Review in 2004 the Government set over 100 PSAs for central Government Departments. These established the key aims and goals for Government performance in 2005-08. This report examines the quality of the data systems used to measure progress against these targets.
   
3. Good quality data are critical if performance measures and targets are to be used effectively to improve public sector delivery and accountability. Good quality data help Departments to: improve programme management and performance; assess whether they need to revise policies and programmes; allocate resources and make other policy decisions; and report reliably to the public and Parliament on their achievements.
   
4. The National Audit Office (NAO) was asked by Her Majesty’s Government to report on the quality of the data systems underlying PSAs. This report examines the data systems used by eleven Departments to monitor and report progress against their 2005-08 PSA targets. Combining these findings with the results of the other six Departments, published in December 2006 in the Third Validation Compendium Report [Footnote 1], we present an overall picture of the quality of the data systems supporting all 2005-08 PSAs. We conducted a follow-up exercise of the six Departments previously validated, to ensure the analysis was updated where necessary.
   
5. We reviewed a total of 237 data systems operated by 17 Departments to support progress against PSAs set for the 2005-08 period. The overall results are summarised in Figure 1.
   
   

    

   
   
   
6. Overall our examination shows that Departments have successfully taken steps to improve the quality of their data systems. Departments have been less successful at transferring the learning to the development of new data systems and there are still improvements that can be made to increase the relevance and reliability of data used in reporting progress.
   
   

    

   Recommendations

    

7. Specification of data systems
   
   • Departments should explicitly consider measurement issues when setting new PSAs. Specifically, Departments should ensure that all the elements of performance they are targeting are properly defined and measurable.
     
   • Departments should evaluate existing data sources to assess their suitability for PSA monitoring purposes before incorporating them into PSA data systems.
     
   • Where PSA targets represent longer-term outcomes to be achieved beyond the Spending Review period, Departments should agree appropriate milestones which allow interim progress against the target to be judged, and remedial action to be taken if necessary.
     
   • Departments should ensure that performance data are available frequently enough to inform managers and front-line personnel of their progress against target, and help them make decisions about how best to pursue the target.

   
   

8. Operation of data systems
   
   • Reports to management boards should disclose limitations to data quality, and present all the information necessary to place performance information in context.
     
   • Departments should introduce a management function to challenge and approve data quality.
     
   • Accountability for data system quality should be clearly established, and should be separated from performance accountability to avoid perverse incentives.
     
   • Departments should ensure that systems are in place to detect errors in outturn data.
     
   • Where the data system contains subjective elements, guidelines, training and/or a challenge function should be put in place to ensure that judgements by different people are consistent.
     
   • When collecting data from external sources, Departments should ensure that they are aware of significant risks to data quality, and take steps to manage them.

   
   

9. Reporting of data
   
   • Technical Notes should properly define the PSA and disclose limitations to data quality, ensuring the reader has all the information necessary to place performance information in context.
     
   • Performance reports should contain the information needed for readers to correctly understand performance.
     
   • Departments should report latest outturn data for all elements of the target specified in the Technical Note.
     
   • Criteria for achieving success against the target should be clearly specified. In cases when there is more than one performance indicator counting towards a target, Departments should disclose the decision matrix for judging whether the target has been achieved (e.g. four out of seven indicators need to be positive; all must show improvement).

   
   

10. Part four of this report discusses developments to the PSA framework introduced through the Comprehensive Spending Review 2007 (CSR2007). It describes how recommendations arising from our validation work have been addressed by the Treasury in CSR2007 and discusses the ongoing risks to data quality.
    

1.  [back from footnote 1] Third Validation Compendium Report: Volumes 1 and 2 2006-07 (HC 127).