Allocation and management of risk in Ministry of Defence PFI projects

Executive Summary

National Audit Office Value for Money Report 

 

1. The Ministry of Defence (the Department) undertakes a wide range of activities, including front line military capability and essential support functions. To deliver these activities efficiently, the Department uses various forms of relationship with the private sector, with differing approaches to sharing risk. The Private Finance Initiative (PFI) represents one important form of relationship. In PFI projects the Department usually defines the outputs it requires and invites the private sector to design and deliver the required service outputs for an agreed annual price.
   
   
2. Risk management is important to all forms of procurement undertaken by the Department. Effective risk allocation and management is particularly important to delivering value for money in PFI contracts. These contracts seek to optimise value for money by agreeing an allocation of risks between the public and private sectors over the course of a long-term contract. This is not the sole contributor to value for money but it is very unlikely that the best possible PFI outcome will be achieved without effective risk allocation and management.
   
   
3. This study examines whether there has been effective allocation and management of risk in the Department’s PFI projects. Our findings are based on a detailed examination of eight PFI case study projects. This analysis is supported by a census of all the Department’s PFI contracts let in 2007 and consultation with the Department’s staff, contractors and advisers. We selected the case studies to reflect the diverse nature of the Department’s PFI portfolio (Figure 1). Appendix 1 contains detailed findings from each case study. Our examination has also drawn on our previous experience of examining risk management in PFI projects (see Methodology, Appendix 2).
   
   
   Figure 1 ("The eight case studies projects") is unavailable in this version of the executive summary.
   
   
4. Our analysis of these projects focussed on ten key risks. In doing so we recognised two important features of the Department’s portfolio of PFI contracts. Firstly, the Department’s approach to procuring PFI deals has developed over time. Our case study projects were relatively early contracts entered into between 1997 and 2002. These were chosen to provide a good level of operational experience to examine. Since these contracts were let the Department has achieved a greater standardisation of contractual terms and conditions. It has also improved its internal scrutiny of PFI deals and all large projects in the light of previous experiences. Secondly, we recognise that the Department has faced particular challenges arising from the fact that its PFI portfolio is less homogenous than those of other departments. 
   
   
5. The Department’s portfolio of more than 50 PFI projects represents a small, but important, part of the Department’s annual spending. The total private sector capital investment in Defence is over £9 billion. In 2007-08 the Department spent £1.3 billion in PFI service charges. The projects range from small projects with a capital value of under £50 million to larger projects with a capital value of over £1 billion. The larger projects include the service accommodation contract Allenby and Connaught let in March 2006 (capital value £1.3 billion, total contract value £8 billion) and the Future Strategic Tanker Aircraft, the Department’s largest PFI deal completed in March 2008 (capital value £2.6 billion, total contract value £13 billion). The capital value of the case study projects we examined, focussing on projects in full service in 2007, was just over £900 million (Figure 1, page 5)
   
   

   Findings

    
6. The Department has achieved a good service delivery on a broad and diverse portfolio of PFI projects. Across its whole PFI portfolio of more than 50 projects most have reached full service delivery on time, for the cost set out in the contract and are delivering services satisfactorily. These new projects have enabled the Department to achieve considerable benefits from a range of services. Some of the projects are delivering new equipment and training which are contributing to improving the effectiveness of military personnel. Others are providing support services which are helping the Department to carry out its work more efficiently. 
   
   
7. In the case study projects we examined most of the risks were being well managed by the Department with the projects delivering value for money but there were exceptions. We concluded that in nine out of the ten risk categories we examined there was either a low risk to value for money or moderate risk to value for money (Figure 2).
   
   
   In one risk category, the specification of the asset or service, we concluded there was significant risk to value for money to the procurement phase although not to the subsequent management of the projects in their operational phase. The specification issues had contributed to problems on two procurements: the Armoured Vehicle Training Service project which was cancelled during its procurement and the Defence Animal Centre where the contract will need to be renegotiated. In the other six case study projects we examined the risks had generally been well managed contributing to value for money. The Defence Animal Centre has a capital value of £11 million and is therefore, under current Treasury policy, below the threshold of projects which would now need to be procured under PFI. Since these deals the Department has taken a number of steps to address the risk of inadequate specification of assets or services (Figure 3). 
   
   
    
8. The Department has developed commercial disciplines for scrutinising the value for money of its PFI procurements and has extended these into other projects. The Department has over time developed appropriate processes for scrutinising PFI procurements. The Department has an experienced Private Finance Unit which provides support and guidance to PFI projects. Large PFI projects also need to satisfy the Department’s Investment Approvals Board. Recently the Department has used its PFI experience to establish a further review process for its large projects under all forms of procurement. The new review process draws on the checking, known as due diligence, which the private sector funders carry out on PFI deals. It aims to improve assurance that the proposed commercial arrangements will enable the project to meet operational requirements with appropriate contract terms. 
   
   
9. The Department is using these disciplines to take tough decisions on some PFI projects although these decisions could have been made on a more timely basis. Although most of the Department’s PFI projects have delivered the services required the Department is prepared to take decisions to abandon inappropriate PFI project proposals or to renegotiate or terminate PFI contracts which are not delivering the required services. These decisions are taken in the interests of achieving value for money, but in some cases the Department has taken a long time to decide on the appropriate action. For example, the Department took the right decision not to proceed with a proposed PFI procurement for Armoured Vehicle Training but only after spending six years in developing the project. The Department is rightly seeking to renegotiate its Defence Animal Centre contract but has also taken six years to reach this position despite ongoing dissatisfaction with the service delivery.  
   
   
10. The Department took on average 37 months to procure the projects we surveyed, where data was available, but large projects often took longer. The Department’s methods for overseeing PFI procurements aim to avoid inappropriate deals being completed. It is right that time is spent on undertaking such assessment. But it is also important that procurements are efficient so that service delivery is not unduly delayed and bid costs, which are likely to be factored into contract prices in the long term, are kept within reasonable limits. The average procurement time in the Department’s PFI projects we surveyed, in the 77 per cent of projects where the Department held such data, was 37 months. These statistics compare with the average PFI experience across government of 34 months in a 2006 NAO survey (Improving the PFI Tendering Process HC 149 2006-07). Larger projects often took longer to procure, the average for the Department’s PFI projects with a capital value of over £50 million, where data was available, being 45 months. This longer time reflects: the special requirements of the Department’s projects compared with repeat projects such as hospitals or schools; the range of the Department’s in-house stakeholders who are involved in decisions about the projects and the assessment of the deals; and some scope, despite these special considerations, for the Department to improve the speed at which it closes larger deals, which it is seeking to do through improvements to the oversight of its capital procurements. 
    
    
11. The Department’s efforts to allocate and manage risk at the outset of the projects that we examined were often hampered by a lack of data on the services required. Lack of data represented a significant risk to value for money in the eight case study projects we examined. Insufficient service information creates a risk because the Department may not get the service which matches its needs or the contractors may increase the pricing of deals because of the resulting uncertainty. Procurement times have also been affected by the need to clarify specification issues. In particular lack of data on the Department’s service requirements was a factor in the cancelled Armoured Vehicles Training PFI project. Some issues, which could have been identified earlier, only emerged after contract letting. For example, in the Field Electrical Power Generators project the Department had to pay the contractor £7.3 million without competition to modify the generators when the Department found that some of its vehicles, when pulling the generators, had manoeuvrability problems and could not turn corners safely. 
    
    
12. The Department’s PFI contracts have flexibility to deal with changes but there are risks to maintaining value for money where changes are required. The Department works in a fast changing, often unpredictable, environment. It often finds that changes are needed to its projects either in the run-up to contract letting or once the project is in service because: 
    
    
    • the operational needs of its military staff may change, particularly if they need to be deployed in new territories;
    • new technology used in equipment or training may be developed; or 
    • in some cases, changes are needed because the Department’s needs are initially difficult to define precisely across its large complex organisation.

    Any change after appointing a contractor creates a risk to value for money as it may involve added costs in a situation where competition is absent. The NAO report Making Operational Changes in PFI Projects (HC 205, 2007-08) analysed the risks arising from contract changes, and described how Departments and agencies can best manage these risks. The capacity for the Department’s needs to change, sometimes at short notice, does not mean that the PFI is an inappropriate form of procurement: well-designed PFI contracts are flexible and can adapt the services provided as circumstances change. It does mean that changes, and the pricing of them, have to be carefully negotiated between the Department and their contractors.
    
    

13. There are instances where contract management could be improved, especially in assessing performance. The Department’s PFI projects in service are managed by teams who are generally addressing contract management issues in an appropriate manner and building effective working relationships with their contractors. The Department’s Private Finance Unit has also already made a useful step in evaluating the Department’s overall experience of using PFI through its 2005 review of the operational experience of its PFI projects. Our examination has, however, identified the following areas for improving project management:
    
    
    • User satisfaction feedback had not been obtained in 25 per cent of the projects in our census.
    • Risk management processes such as risk registers were not used consistently. 
    • In some projects post contract evaluations, to consider the costs and efficiency of the procurements, were either not carried out or were not done on a timely basis.
    • In the Defence Fixed Telecommunications project, the Department had to recover £1.3 million from the supplier BT after it became apparent that employees of BT had inflated the number of calls being answered within the required time limit by calling each other. This fraud was not detected initially because there was no adverse consequence for the Department’s staff using the telecommunications services. The Department and BT have taken steps to address the circumstances which led to this fraud. The Department and BT have now imposed a new management structure and governance arrangements on the contract. There is more detailed performance reporting by BT which is subject to audit by the Department. BT has also made changes to the staffing of the project.
      
      
14. Appropriate skills are required for managing PFI contracts so that value for money is not eroded during the contract’s life. There is a particular need for the knowledge of the contract and the aims of the project to be transferred to those who will manage the contract. There was a lack of staff continuity on some of the case study projects we examined, but other case study projects had benefited from retaining at least one member of the contract negotiation team in post for the first year or two of the operational phase.
    
    

    Value for money conclusion

    
    
15. In six of the eight case studies we examined the Department has procured and managed successful PFI solutions. One project was cancelled before contract letting and one other contract we examined will need to be renegotiated. In the PFI projects we examined the Department has therefore generally achieved effective allocation and management of risk. The allocation and management of risk, which has been the focus of this examination, is however not the sole contributor to value for money. Effective competitions, or strong benchmarking processes where there is a single supply source, are also needed to obtain the best prices for the risks transferred. The case study contracts we examined were all procured through competitive tendering.
    
    

Recommendations

These recommendations relate to the Department’s portfolio of existing PFI projects and any further PFI procurements they may undertake. The recommendations may also have relevance to other forms of public/private partnerships which the Department may develop in the future.

1. A lack of robust data for project teams to specify their requirements and the risks being transferred to contractors has been an issue for a number of the case studies examined. Project teams should ensure that the initial planning stage of each project includes the production of suitable data on any existing use of the required service, forecast usage and the condition of assets being transferred to the private sector. The Department’s Private Finance Unit should check that this information is available before bidding competitions commence.
2. PFI contracts are long term contracts which have the potential flexibility to deal with the Department’s changing requirements. But change requires negotiations between the parties which need to be managed effectively to ensure value for money is not eroded during the contract period. The Department’s PFI project teams should assess the likely impact of future changing circumstances on the PFI contracts they propose to enter into. In particular, they should:
   
   
   • be fully conversant with the prescribed processes for dealing with change, including Treasury and Departmental guidance;
   • only enter into contracts that set out clearly how changes will be made to the project including processes for benchmarking the price of variations;
   • continue, during the service period, to undertake regular assessments (at least yearly) of factors that could affect future requirements and the impact these may have;
   • actively manage supplier relationships with the private sector to successfully incorporate changing requirements on terms which are value for money; and
   • when considering a possible change in the use of an asset delivered by the private sector, remember that the Department may become liable for repairs arising from the change.
      
3. Some of the Department’s projects with early PFI contracts may experience difficulties in enforcing the performance they require because of a lack of clarity in the drafting of the contract. Where the Department’s projects experience service problems because the contract does not set out their requirements clearly they should either renegotiate the contract or seek, through their relationship with the contractor, a mutually agreed working arrangement to overcome the contract deficiencies. Terminating a non-performing contract represents an extreme option. It imposes significant transaction costs on the Department, but these costs may be lower than the ongoing costs of poor performance. The Department should not rule out termination of non-performing contracts on the grounds of transaction costs alone.
    
4. Although it is normal practice in PFI projects for the private sector to record performance and the public sector to monitor performance levels there is a risk that contractors might misrecord performance to avoid payment deductions. The Department’s project teams should assess the appropriateness of the systems used to validate contractors’ service performance. In particular they should be alert to the risk that the extent of successful service delivery could be overstated without any adverse effect on users of the service that would draw their attention to the situation. Project teams should carry out audit work on the performance monitoring systems, consider whether the performance data being provided is adequate and carry out spot checks of the authenticity of the underlying data.
    
5. Good contract management involves retaining knowledge about the project, monitoring risks and carrying out regular evaluations. There was a lack of staff continuity on some of the case study projects we examined, which meant that it was harder to achieve a high standard of contract management. To improve the management of PFI projects the Department’s PFI project teams should:
   
   
   • keep at least one senior member of the team in post for the first year after the contract has been let, so there is a suitable transfer of knowledge to the team who will manage the contract;
   • have staff with appropriate contract management skills acquired through either previous experience or appropriate training; n capture project risks on formal risk registers in both the procurement and in-service phases;
   • undertake user satisfaction assessments on a systematic basis; and
   • carry out post contract evaluations and subsequent annual reviews of overall contract performance. The NAO framework for evaluating PFI projects may assist this process (available from the NAO  website www.nao.org.uk).
   • identifying and disseminating lessons from its project teams’ post contract evaluations and subsequent annual reviews;
   • analysing information on the internal and external costs of procuring recent and current PFI deals to identify action points for improving the efficiency of the procurement of future deals; and
   • recording and monitoring the main risks affecting the successful delivery of services across the Department’s PFI portfolio taking account of the project evaluations and the issues identified in this report.