Benchmarking and market testing the ongoing services component of PFI projects

Executive Summary

National Audit Office Value for Money Report

1. Around 500 Private Finance Initiative (PFI) projects out of the 600 [Footnote 1]  that have been let, are now in the operational phase. At least half of these projects have provisions in their contracts that require the value of certain services, such as catering and cleaning, to be tested at intervals, typically every five to seven years. The services that are subject to this value testing are often a significant part of the total cost of a PFI contract (Figure 1) and so the process of value testing is an important aspect in seeking to achieve value for money from a PFI contract which may run for 25 or 30 years or more. Value testing may involve comparing information about the current service provider’s provision with comparable sources [benchmarking] or alternatively, inviting other suppliers to compete with the incumbent in an open competition [market testing].
   
   
2. We examined the contractual provisions for value testing within a sample of 34 PFI contracts in order to assess their expected effectiveness. We also examined the early experience of the 11 PFI projects in England that had carried out value testing at the time of our study in summer 2006. We found that in some of these initial cases the value testing had demonstrated that value for money was being achieved, but in other cases the outcome was uncertain. Our methodology is set out in Appendix 1.
   
   
3. Although the projects examined were all those that had used value testing up to summer 2006, the number of projects is only around two per cent of all PFI projects in operation. In addition, there have, over time, been improvements in contract terms for value testing since some of the contracts we examined and there is now detailed Treasury guidance to supplement the previous guidance about value testing in the Treasury’s Standardisation of PFI Contracts (SoPC). For these reasons the results of our examination of the early examples of value testing are not necessarily indicative of how the arrangements will work in future in other PFI projects. Nevertheless, we considered it important to examine these initial examples to highlight lessons drawn from the first practical experiences of applying value testing. These lessons will help the many projects that will be using benchmarking or market testing in the coming years.
   
   Figure 1 ("Example of the proportion of the total annual costs of a PFI contract which are subject to value testing")  is unavailable in this version of the executive summary.
     
4. The Treasury issued new guidance in October 2006 [Footnote 2] which took account of our initial findings and additional research conducted by Partnerships UK (PUK) on behalf of the Treasury. Current best practice as set out in Treasury guidance addresses issues described in our study and gives the prospect of improved value for money in the future. Our other main findings were:
   
   

The mechanics for carrying out value testing in the early PFI contracts
 

1. The early PFI contracts, let before the start of contract standardisation in 1999, contain a range of provisions for benchmarking or market testing. Lawyers, Nabarro, examined on our behalf a sample of 34 contracts and found that the value testing terms were often expected to have limited effectiveness, although these have yet to be tested in practice. Some of these contracts, where value testing would be appropriate, have no contractual provisions. Where contract clauses are absent, or are expected to have limited effectiveness, it may nevertheless be possible for the parties to carry out effective and value for money value testing if they develop suitable processes.
   
   
2. Projects will now benefit from the Treasury’s 2006 guidance which is more detailed than that available in previous versions of Standardisation of PFI Contracts. We agree with the Government’s view expressed in the new guidance that, because of the potential benefits of competition, transparency, and flexibility for re-assessing the service provision, market testing, if it can be applied successfully, is generally the mechanism most likely to give a better outcome on value for money grounds.
   
   
3. Although only three market tests had been completed at the time of our study, which does not allow wide-ranging conclusions to be drawn about this process, these first three market tests were competitive processes and produced beneficial outcomes. Although not necessarily indicative of future experiences, one was won by an in-house bid and the other two by the incumbent supplier. External suppliers bid but were not successful in these competitions. To maintain the competitive benefits of market testing, suppliers must continue to be interested in bidding against incumbent suppliers. The Treasury is seeking to facilitate a market in benchmarking and market testing through publishing details of ongoing services that will be put out to competition. The Treasury intends to make this available on PUK’s website.
   
   

   The application of the processes
   
   

4. In projects where ineffectual or vague contract clauses were identified, authorities and their private sector counterparts have had to work together in order to produce an effective project plan to manage the process.
   
   
5. Value testing can be a lengthy process. We found that where projects had completed the process it had taken nine to 25 months; similar to the time taken to re-let service contracts in conventional procurement. But one project examined had yet to complete the process, having taken, up to March 2007, 37 months. Agreeing how the process will be conducted can be time consuming and there have often been detailed negotiations before a final price adjustment has been agreed. In particular, projects have experienced difficulties in finding suitable benchmark data with which to compare the services. The Treasury is liaising with departments to draw up a central database of benchmarking and market testing information.
   
   
6. By allowing prices to be renegotiated in line with market rates these processes enable the public sector to benefit if market prices fall and they also limit the uncertainty faced by the private sector by giving them an opportunity to obtain a price rise when costs increase.
   
   
7. The two telecommunication projects which had completed value testing had achieved value for money through price reductions of 19 per cent and 37 per cent after using benchmarking to take account of falling prices in the very competitive communications sector.
   
   
8. In the seven building projects we examined, five of which were hospitals, where value testing had been completed the final price adjustments were mainly -2 to +6 per cent (Figure 2) although in one school project the final price increase was 14 per cent. The authorities had been involved in negotiations to arrive at these price changes after the value tests initially suggested that, in most cases, upward price changes would be required, with the changes mainly in the range -1 to +19 per cent (Figure 2).
    

   Figure 2 ("Price adjustments arising from value testing the seven PFI buliding projects") is unavailable in this version of the executive summary.
    

9. These price changes were separate from the contractual arrangements allowing the private sector an annual price increase for general inflation. The price changes initially proposed reflect various market factors, including salary costs that had risen more than had been expected since contract letting and that some initial contracts may have been priced competitively at below normal market rates. These projects were the first PFI building projects to use value testing processes and the price changes reflect cost changes in the market for facilities management services up to 2006. The resulting price changes are not, therefore, indicative of the price changes that may arise in future uses of value testing in PFI building projects.
   
   
10. The negotiations, initiated by the authorities, were a significant factor in arriving at the final price changes. As part of the negotiations, in three of the seven projects, the authorities made minor reductions to their service specification in order to keep the price affordable. These authorities considered that the service levels were previously over-specified and do not expect the reductions in specifications to compromise the service delivered to the public although it is too early to judge the outcome conclusively.
    
    
11. The price changes which have arisen from these value testing exercises should not be viewed in isolation but are part of the overall cost of procuring facilities services to the standards specified in the PFI contracts over an extended period. The Treasury requires project teams to make a value for money decision on whether to include or exclude these services from the contract before embarking on a PFI procurement. We are not, however, aware of any systematic overall comparison to date between the cost and quality experiences of facilities services procured under the PFI with conventional outsourcing. There are difficulties in making these comparisons as the conventional examples may not be comparable to the PFI deals in terms of the required services or standard of performance. There is also limited experience to date of the price changes arising from using the value testing arrangements in PFI contracts.
    
    
12. We considered the value for money of the completed value tests by reference to the resulting changes to the price and service specification, and the effectiveness of the value testing (Figure 3).
    
    Figure 3 ("National Audit Office assessment of the value for money of the nine completed value tests") is unavailable in this version of the executive summary. 

    
    
    Lessons learned as a result of the early applications of benchmarking and market testing
    

13. The projects that have completed benchmarking and market testing have learned important lessons such as: the need for realistic timescales, the benefit of the early engagement of the private sector, the need for an effectively designed project plan, access to good comparable benchmarking information and the effective role that advisors can play in the process. These lessons have been incorporated into the Treasury’s 2007 Standardisation of PFI Contracts version 4 (SoPC4) and Operational Taskforce Note 1, Benchmarking and Market Testing guidance.
     

Recommendations

1. Departments should ensure that their PFI project teams are familiar with, and adopt, the new Treasury guidance on benchmarking and market testing. Project teams should take appropriate advice including consulting their Private Finance Units, the Treasury Operational Taskforce managed by Partnerships UK, and other projects which have already undertaken these processes. Project teams should also consider the skills they will need and seek to identify who in their departments or agencies might have them before using external advisors.
   
   
2. For the potential benefits of market testing to be realised there needs to be strong competition. Project teams and their respective departments have a role to play in keeping the market active and competitive, for example by keeping suppliers informed of future bidding opportunities.
   
   
3. The Treasury should continue to liaise with departments to draw up a central database of benchmarking and market testing information. The Treasury and departments should then consider whether further data is needed and how such data can best be obtained.
   
   
4. The price changes arising from value testing are part of the overall cost of procuring facilities services to the standards required under a PFI contract over an extended period. Departments should ensure that project teams follow Treasury guidance to assess the value for money case for including or excluding facilities services before letting a PFI contract and, to inform this process, should take steps to compare the cost and quality experience to date of procuring facilities services under the PFI with conventional outsourcing.
   
   
5. Project teams need to consider, in identifying affordability limits for PFI projects, that the contract price may increase after contract letting for factors which include service and price changes arising from value testing the ongoing service provision.
   
   
6. Where service amendments need to be made in conjunction with benchmarking or market testing exercises the amendments need to take into account the needs of users, opportunities for innovation and the ongoing demonstration of value for money and affordability.
   
   
7. In benchmarking the costs of PFI services for their particular projects authorities should be aware that if the project’s investors are managing a number of similar projects or, if the facilities services providers are working on similar projects, the costs of the services being provided should reflect economies of scale which would not be available to other private sector parties without such a volume base.
   
   
8. Departments should bear in mind that the lessons in this PFI report, and the related Treasury guidance, relating to benchmarking and the recompetition of services through market testing may have relevance to testing the value of services in other long-term service delivery arrangements. These include conventional outsourcing, partnering and joint ventures, both during the life of these arrangements and on any subsequent recompetition of the service requirement.

1.  [back from footnote 1] As of April 2007 the list of PFI projects published by HM Treasury has been updated and reduced from over 750 signed PFI deals to 600. This reflects the large data validation exercise HM Treasury has carried out as: some projects had concluded or been terminated; some projects had changed their contractual structures that meant they were no longer classed as PFI; some projects had been contractually merged; and departments have stopped collecting data on some very small projects in order to reduce reporting burdens.
2.  [back from footnote 2] Operational Taskforce Note 1: Benchmarking and market testing guidance, HM Treasury, October 2006.