Executive Summary

National Audit Office Value for Money Report

Good quality data are crucial if performance measures and targets are to be used effectively to improve public sector delivery and accountability. Good data help Departments to: manage delivery against priorities; assess whether they need to revise policies and programmes; and report reliably on their achievements.

In 2002, Government announced 122 Public Service Agreement (PSA) targets for all 18 Departments[Footnote1] for the period 2003-06, including 18 targets shared by two or more Departments. This report outlines the findings from the National Audit Offices validation of the data systems used to monitor and report progress against all these PSA targets.

In March 2005, the C&AGs report Public Service Agreements: Managing Data Quality (HC 476 Session 2004-05) summarised findings from our validations conducted during 2004, covering seven Departments and the Sure Start programme. This report adds to that analysis, incorporating the findings for a further eleven Departments data systems validated during 2005.

Figure 1 summarises the results: while 77 per cent of data systems provided a broadly appropriate basis for measuring progress, in two-thirds of data systems we looked at, Departments had encountered problems to varying degrees:

• In just over 20 per cent of cases we found Departments faced measurement problems that could not be addressed cost-effectively. However, in 18 per cent we found Departments need to explain further the implications of these limitations to readers of their public performance statements.
   
• For 29 per cent, we found weaknesses in the systems that need addressing. The most common encountered were poor controls over data collection and documentation, and shortcomings in Departments checks on data obtained from external bodies. These weaknesses may not have resulted in unreliable data but Departments did not know the actual levels of error that exist. Departments should strengthen the controls in these systems to reduce the risk of error in the future.
  
  

  

  
   
• For 12 per cent of systems, we found that the arrangements that Departments had were not fit for monitoring and reporting progress against their PSA targets. Most commonly, this was the result of design problems, where the systems established did not measure adequately the aspects of performance included in the target.
   
• For 6 per cent of 2003-06 targets, Departments had not established the necessary data systems to report progress at the time of our review.

 Our examinations showed that there is much opportunity for Departments to take further action to ensure that data systems for all PSA targets are robust. The demands of developing data systems for complicated targets and over-reliance on existing data streams can explain many of the difficulties that Departments face in developing better data systems. But Departments can address many of the problems we encountered in their 2003-06 data systems. As part of this report we conducted a follow-up exercise with five Departments who were validated in 2004. By the end of 2005, Departments had already addressed weaknesses in 20 out of 46 systems reviewed in response to our recommendations, demonstrating the scope for Departments to tackle the weaknesses identified in their 2003-06 data systems and, going forward, to strengthen arrangements in place for their 2005-08 targets.

This report identifies common challenges faced by Departments and highlights ways to strengthen their approach to establishing and operating PSA data systems and reporting results to Parliament and the public. Our validations of the 2003-06 PSA data systems also provide general lessons which should inform Departments actions:

• Departments should develop a more systematic approach to data quality assurance. For example they could:
   
  • introduce a formal process of risk assessment for key performance data and, where necessary, include data quality risks in their corporate risk registers;
     
  • allocate clear responsibilities for data quality management, including active oversight of and challenge to systems;
     
  • formalise the role of Departmental statisticians and other data specialists in the quality assurance of PSA data systems to ensure standards and checks are applied consistently; and
     
  • develop a clear policy on the disclosure of data limitations for reporting out-turn data for all PSA targets.
     
• They should plan and co-ordinate the data needs for new systems. Many weaknesses stem from inadequate attention to data issues when PSA targets are selected and specified.
   
• When setting PSA targets, Departments should consider their capability to measure progress and judge when success has been achieved. Departments should define the quality of data needed for effective progress monitoring, and then assess whether existing or new data systems can best meet the requirement. This process should involve staff from the relevant business areas, statisticians and analysts, and the providers of data whether within or outside the Department.
   
• Systems must be adequately documented and updated for any significant changes. Clear definitions of terms, well-documented controls and unambiguous criteria for judging success enable systems to operate consistently over time and provide the foundations for making robust judgements of performance. Where Departments revise systems for PSA targets they should update documentation and agree major changes with HM Treasury and explain them in their Technical Notes.
   
• Managers should check that data obtained from other organisations are fit for purpose. Many PSA data systems rely on data that are produced by other organisations. Managers need to discuss with these organisations to assure themselves that the data are appropriate and that any limitations are clearly understood.
   
• Departments should make users of performance data aware of limitations in underlying systems. When reporting progress, Departments should explain the implications of any data limitations that might affect how out-turn figures are interpreted. This approach builds trust in public reporting by helping users make informed assessments of reported results.

1. [back from footnote 1]Including Sure Start, the only cross-cutting government programme with its own Public Service Agreement.