Delivering High Quality Impact Assessments

Executive Summary

National Audit Office Report

1. The purpose of Impact Assessments (IAs) is to assess the need for, and likely impact of, proposed regulations. Departments typically introduce over 300 new or amended regulatory proposals each year. The Government has given a commitment to improve the design of new regulations and the Better Regulation Executive (BRE) works with departments and regulators to establish whether new regulation is necessary. Impact Assessments help policy makers understand the consequences of proposed regulation and consider how to achieve the desired outcomes while minimising costs. Impact Assessments are also used to seek the views of stakeholders and communicate the policy decision. They are required for all Government interventions that impose or reduce costs on businesses, the third sector or the public sector.
2. During 2007, in response to the findings of previous NAO reports, the BRE phased in new arrangements for preparing Impact Assessments, including a new template, guidance and training. The changes aimed to improve the quality and consistency of IAs. We evaluated the impact of these changes by comparing the ‘new style’ IAs published in the first six months of 2008 with the standard of IAs published in 2006. Part 2 sets out the results of this analysis. Our 2007 report also highlighted the importance of robust scrutiny processes in the development of high quality IAs. We therefore examined whether departments had established effective scrutiny arrangements to encourage the development of IAs that are evidence-based, influential and fulfil their intended objectives (Part 3). Full details of the Report’s scope and methodology are provided in Appendix 1.
   
   

   Key findings

   
   

   The quality of Impact Assessments

3. The new IA process has helped to improve the standard of IAs. The introduction of a new template, guidance and training has improved the clarity of presentation in IAs. As a result, there is greater consistency in providing the requested information, for example on the proposed timing of post-implementation review. There was also a greater incidence of quantification: the proportion of IAs which quantified costs increased from 56 per cent in 2006 to 67 per cent in 2008, and the proportion that quantified benefits increased from 40 per cent to 60 per cent.
4. The standard of IAs still varied widely. There were marked differences between the best and worst IAs, and variations in quality between different elements within individual IAs. The strengths included the statement of the policy problem; the use of consultation; and the clarity of recommendations. The analysis of costs and benefits of a range of options was the weakest area in 2008: whilst more final IAs quantified the costs and benefits of the preferred option, only 20 per cent presented the results of an evaluation of a range of options – although it is possible that other options were considered, and dismissed, earlier in the development of the IA. The consideration of implementation and enforcement issues, which was the weakest area in 2006, has shown an improvement. Some 80 per cent of IAs stated who will monitor and enforce the new regulation, although there was still insufficient analysis of compliance and enforcement issues; for example, only 20 per cent of IAs included or referenced a detailed implementation plan.
5. There was insufficient analysis of evidence in the weaker IAs. In some cases, the new requirements have led policy officials to provide only the information specifically requested rather than present an analysis of a range of regulatory proposals. There is evidence of a wider use of economic techniques and a greater involvement of specialists in an increasing number of IAs, but many still contained a superficial evidence base or standard of analysis. The level of analysis should be proportionate to the scale of the problem under consideration but 60 per cent of IAs included £0 for either the costs or benefits, which may be justifiable in some cases but approximately one fifth provided no justification for this assessment.
   
   

   The effectiveness of scrutiny arrangements

6. The BRE’s introduction of a new IA process provided a catalyst for change and departments have strengthened scrutiny processes. Departments have increased the frequency of review and the use of economists in developing IAs, including a more prominent role for Chief Economists in reviewing the standard of analysis. The majority of departments have introduced formal scrutiny processes and are making greater use of peer review and challenge panels. These changes are contributing to the development of stronger scrutiny arrangements. Some departments, such as BERR, have gone further than others in seeking to embed challenge into the development of IAs. There are also variations in the thresholds used for the scrutiny of IAs and the extent to which scrutiny processes are applied in practice.
7. The BRE undertakes less real-time external challenge to the development of IAs than its predecessor body. The BRE believes that departments should have primary responsibility for the quality assurance of IAs. It seeks to engage with departments on the implementation of regulatory reform initiatives and more directly with policy officials in the early stages of development of regulatory proposals. Ten of the thirteen departments we surveyed, however, told us that the possibility of external scrutiny was a strong motivator for policy officials in developing IAs and nine departments believed that the BRE provides the most effective form of external scrutiny.
   
   

   Conclusion on value for money

8. The BRE’s introduction of a new IA process, and departments’ steps to strengthen scrutiny arrangements, have improved the quality of published IAs in several respects. There is sharper presentation of results, better planning for post implementation review, and a greater incidence of quantification of costs and benefits. But there remains wide variation between the best and worst IAs, and fewer presented an analysis of the costs and benefits for a range of options or summarised an implementation plan. Further improvements in the quality of analysis and evidence are needed for IAs fully to play their part in helping to ensure that new regulations deliver the intended benefits in a cost-effective manner.
   
   

   Recommendations

9. Good impact assessment is integral to better policy making and essential to achieving value for money. Our recommendations are intended to help departments improve the quality of IAs and embed them in policy formation.
   
   

   For departments:

   1. There are still wide variations in the quality of IAs. Departmental Better Regulation Units should analyse the results of the NAO’s examination of their IAs to understand their department’s strengths and weaknesses; to determine what is driving this performance; and identify how to raise standards.
   2. Only 20 per cent of IAs included a detailed plan setting out how the proposed regulation would be implemented. The BRE’s new IA process and departments’ revised scrutiny arrangements focus on improving the standard of economic analysis. Departments should extend this effort to consider implementation and enforcement issues more prominently. They should improve the standard of implementation plans by providing more specific information on the costs of enforcement; the anticipated compliance issues; and the arrangements for post-implementation review.
   3. IAs do not differentiate sufficiently between the scale of regulatory intervention under consideration. IAs cover a wide range of policy proposals, in terms of size, origin and intention, and the level of analysis should be proportionate to the regulatory proposal. To help ensure resources are directed to the most significant regulatory proposals, departments should make an early judgement on the level and type of analysis expected in each specific case, the impact tests that are applicable and any requirement for expert support.
   4. Whilst more 2008 IAs quantified costs and benefits of the preferred option, only 20 per cent presented the results of an evaluation of a range of options. Departments should give full consideration to a range of options and conduct an appropriate level of analysis to inform the policy decision.
      
      

      For the BRE:

   5. The prospect of external scrutiny is the most effective motivator for departments to produce high quality IAs. To improve the design of new regulations, the BRE and departments should bring external challenge to the development of regulations. In order to target poor performance in departments and raise the standards of IAs, and their use in policy formation, the BRE should:
      • periodically review departmental approaches to producing IAs, with an assessment of the standard of a sample of IAs, to develop an understanding of their strengths and weaknesses; and
      • where problems are identified in departmental approaches, hold more frequent meetings with departments to ensure that minimum standards of scrutiny are met; common criteria applied; and best practice disseminated.
   6. The new summary sheet has helped raise awareness and improve presentation but, in some cases, has led to a more superficial approach to producing IAs. It is important, therefore, that the new template is used to summarise the analysis rather than become the IA itself. Before the introduction of regulatory budgets, the BRE should re-emphasise the need to focus on the evidence base in IAs and revise the guidance to better explain the role and use of the summary sheet. In particular, departments should:
      • provide sufficient evidence to justify the inclusion of costs or benefits as £0;
      • appraise fully a range of options for achieving the desired policy outcome; and
      • ensure all sections of the summary sheet are completed.