Third Validation Compendium Report

Executive Summary

National Audit Office Value for Money Report

1. Good quality data are crucial if performance measures and targets are to be used effectively to improve public sector delivery and accountability. Good data help Departments to: improve programme management and performance; assess whether they need to revise policies and programmes; allocate resources and make other policy decisions; and report reliably to the public and Parliament on their achievements.
    
2. In 2004, the Government announced 110 Public Service Agreement (PSA) targets for 17 Departments covering the period 2005-08. PSA targets express the priority outcomes that Departments are seeking to achieve nationally and internationally, and cover key aspects of the Government’s social, economic and environmental policy. Large sums of public money are devoted to the programmes designed to deliver them.
    
3. The National Audit Office (NAO) previously reported on the quality of the data systems underpinning Departments’ PSA targets covering the period 2003-06[Footnote 1]. We are now examining the data systems used by Departments to monitor and report progress against their 2005-08 PSA targets.
    
4. During 2005-06 we validated the data systems operated by six Departments: Cabinet Office, Department for Culture, Media and Sport (DCMS), Department for Education and Skills (DfES), Department for the Environment, Food and Rural Affairs (Defra), Ministry of Defence (MOD) and HM Treasury. This report (Volume 1) summarises the results. Detailed findings for each data system are presented in Volume 2. We will report on the data systems for all 2005-08 PSA targets in 2007-08.
    
5. For the six Departments included in this report, we reviewed a total of 65 data systems used to measure progress against all 46 of their 2005-08 PSA targets[Footnote 2]. For five of these systems (eight per cent), it was too early to assess the strength of departmental controls. For the remaining systems, we found a mixed picture (Figure 1).
    
   
6. Nearly three-quarters of the systems developed by Departments provided a broadly appropriate basis for measuring progress against their PSA targets, but less than half of these were fully fit for the purpose. Most required some action to strengthen measurement or reporting arrangements:
    
   • 23 per cent of systems had weaknesses in either their design or operation. These weaknesses may not have resulted in unreliable data or may have affected only an element of the data system, but in practice Departments did not know the actual levels of error that may have resulted. Departments should address these weaknesses to reduce the likelihood of error in the future.
      
   • In 17 per cent of systems, we found that Departments had not explained adequately in their public performance reports the impact of unavoidable limitations in the quality of their PSA data.
      
7. For nearly 20 per cent of systems, we found significant limitations in Departments’ control arrangements: eight per cent of Departments’ systems were not fit for monitoring and reporting progress on the key elements of their PSA targets, while for another 11 per cent Departments had not established the data systems needed to report progress at the time of our review.
    
8. In 2004, we examined the systems underpinning the same six Departments’ PSA targets covering the 2003-06 period. Figure 2 compares the results from the two periods and shows that there has not been a major improvement in Departments’ overall approach to the management of PSA data quality.
    
   
9. In practice, PSA targets themselves change over time and Departments develop their measurement systems accordingly. Nearly 30 per cent of the 2005-08 data systems that we examined were new or significantly different from the 2003-06 systems. Looking at just those systems that were common to both PSA periods we found that the Departments had responded to our previous recommendations and made a number of improvements, for example: better system documentation, greater checks on the quality of data obtained from external bodies, and clearer disclosure of data limitations in public performance reports. However, these improvements were offset by limitations in the new data systems established for the 2005-08 period.
    
10. The lack of overall improvement highlighted in Figure 2 raises questions about the priority attached by Departments to data quality when developing, monitoring and reporting against their PSA targets. In our previous reports, we provided guidance on how Departments can strengthen their approach. Looking forward, we expect the Government to announce new PSA objectives in 2007 as part of the Comprehensive Spending Review. This provides an excellent opportunity to build in data quality management from the start for the new round of measures and targets.
     
11. This report summarises our validation findings for the six Departments and highlights the risks that Departments should consider when establishing or reviewing their different PSA data systems. We also highlight other actions needed to raise the importance placed by Departments on the control of key performance data, and strengthen Departments’ overall approach to PSA data quality management.
     
12. Consequently we recommend action at three different levels: Departmental staff responsible for managing individual PSA data systems; central Departmental staff responsible for establishing Departments’ policy towards PSA data quality; and HM Treasury, as the department responsible for oversight of the PSA framework across all Departments.
     

Recommendations

1. Consider the implications for measurement and data systems when designing PSA measures and targets.
    
2. Explicitly assess the risks to data quality in PSA data systems, taking into account differences between types of data.
    
3. Ensure data systems and the associated controls are adequately documented and ensure supporting records are kept up to date.
    
   For staff responsible for the Department’s data quality policy
    
4. Raise the profile of PSA data systems across the Department by setting out clear expectations for data quality and reporting standards.
    
5. Actively monitor PSA data quality and ensure that there is adequate challenge to outturn data.
    
6. Assign responsibilities and resources to address identified data system weaknesses.
    
7. Formalise the role of statisticians and other data specialists in the quality assurance of PSA data systems.
    
   For HM Treasury
    
8. Challenge Departments’ measurement arrangements early in the process of developing new PSA measures and targets.
    
9. Review Departments’ response to NAO validation recommendations for strengthening PSA data systems.
    
10. Provide measurement guidance to Departments setting out HM Treasury’s expectations for PSA data quality.
     

1.  [back from footnote 1] Public Service Agreements: Managing Data Quality – Compendium Report 2004-05 (HC 476); Second Validation Compendium Report: 2003-06 PSA data systems 2005-06 (HC 985).
2.  [back from footnote 2] For the purposes of validation, PSA targets may rely on more than one data system, for example, if they include multiple, unrelated aspects of performance or comprise separate sub-targets.