INTOSAI Working Group on the Audit of Privatisation

Guidelines on Best Practice for the Audit of Public/Private Finance and Concessions

Contents

Part 2: The Guidelines

Section 2: Scoping the project

Glossary

FOREWORD

by Sir John Bourn
Comptroller and Auditor General of the United Kingdom
Chairman of the Working Group on the Audit of Privatisation

The use of private finance for public infrastructure or the letting of concessions to private sector operators to provide public services can be seen in the same light as privatisation. Both privatisation and the private finance or concession approach involve the state ceasing to perform activities which may traditionally have been seen as exclusively the responsibility of the state. In the case of privatisation the state may sometimes entirely give up any particular role in relation to the privatised entity, but in following the private finance or concession approach the state will continue to take an interest in the activities in question. Accordingly the state will be concerned both with establishing such arrangements on a basis which is likely to provide value for money and with ensuring that they continue to do so.

The guidelines on the audit of public/private finance and concessions set out in this document therefore also deal with both of these aspects. The guidelines represent the experience of the many SAIs who are members of the Working Group on the Audit of Privatisation and of those who have commented on drafts of the guidelines. The result is therefore an authoritative statement of an approach to auditing such arrangements which can be used by SAIs throughout the world.

John Bourn
Seoul
October 2001

1. Following a decision taken by the XIV INCOSAI in Washington in 1992, the INTOSAI Working Group on the Audit of Privatisation was formally established by the Governing Board in May 1993 under the Chairmanship of Sir John Bourn, Head of the United Kingdom National Audit Office.

2. The membership of the Working Group comprises representatives from the SAIs of:

   Albania	India
   Antigua and Barbuda	Israel
   Argentina	Lithuania
   Australia	New Zealand
   Austria	Norway
   Bahamas	Oman
   Chile	Paraguay
   Czech Republic	Peru
   Denmark	Poland
   Ecuador	Russia
   Egypt	Saudi Arabia
   El Salvador	Turkey
   Estonia	United Kingdom (Chair)
   Germany	Uruguay
   Hungary	Yemen
   	Zambia

3. In recent years Governments have been experimenting with a variety of ways in which the public and private sectors can work together to get better value for money for the taxpayer in delivering public services. This new development depends on harnessing private sector business skills and capital to identify scope for innovation in the delivery of such services and is now being carried forward in a series of partnerships between the public and private sectors. Therefore in November 1998 the XVI INCOSAI invited the Working Group to develop guidance on the audit of an important example of such public/private partnerships, the public/private finance and concession approach.
4. The public/private finance and concession approach represents a departure from the traditional procurement of assets where the public sector pays for the construction or development of an asset and then makes separate arrangements for the on-going maintenance and operation of this asset. It is also different to traditional outsourcing which simply involves the provision of services. Instead the public/private finance and concession approach involves the provision of services to the public sector by the private sector, which also takes responsibility for the construction / development and financing of any assets needed to provide the required services. Also, the services provided may extend beyond those support services which traditionally the private sector has provided to the public sector, such as building maintenance, to services whose delivery in the past has been the responsibility of the public sector itself, such as the incarceration of prisoners.
5. Thus public/private finance and concession contracts typically involve public sector clients specifying services which they wish to purchase and, through competition, selecting private sector suppliers to provide them. Alternatively, they can involve the award of a concession to a private sector supplier who then charges the general public for the use they make of the services provided. Sometimes it is expected that the private sector will need to construct new physical assets, such as buildings, to provide the specified services. And sometimes existing public sector assets may be transferred or rented to the private sector supplier, to enable the supplier to provide the services required.
6. This new approach to providing services offers the prospect of better value for money than traditional publicly financed methods. It is however also both new and more complicated than the traditional methods and brings with it new pitfalls and new risks to value for money. The approach has also been extended from its initial use in infrastructure projects such as roads and bridges into areas previously seen as core state activities, such as the provision of custodial services for the incarceration of offenders. All this means that securing the potential benefits of the public/private finance and concession approach is requiring new skills on the part of the public sector.
7. The purpose of these guidelines is to provide a logical framework for SAIs wishing to audit these agreements to establish whether the public sector has got the best possible deal for the taxpayer. The guidelines aim to cover all the issues potentially involved on such deals, although not all the guidelines will be relevant in each case and an SAI may not have the powers necessary to follow each guideline in full. The Working Group has explicitly recognised that as the public/private finance and concession approach is new, it may be very unfamiliar in some countries and therefore to some SAIs. For that reason, the guidelines may need to be applied in such countries with appropriate simplifications and modifications to take account of the state of development of the approach there. None of the issues addressed in the guidelines contained is straightforward; all require the SAI to be well informed and balanced in reaching judgements on how well the body procuring the services managed the process, faced with uncertainty and, for many, a new method of procurement, and with conflicting objectives. Each guideline therefore has been drawn up in a format designed to bring out the reasoning and experience underlying it. To this end, there are two parts to each guideline:

   the guideline itself; and

   the reasons for the guideline.

   ---

   The Guidelines: Summary

8. These guidelines and also supplementary guidance, giving more detailed advice on how to follow the guidelines, is available on the Working Group's website www.nao.gov.uk/intosai/wgap.
9. The guidelines themselves are grouped in five sections.

Section 1: The general approach of the SAI

10. If the SAI is to carry out well thought through performance audits, leading to worthwhile recommendations and the spread of good practice, it needs to determine its remit in examining a public/private finance and concession contract and plan its audit thoroughly. It will also need access to a range of specialist skills.

Guidelines

1. SAI responsibilities
2. Acquiring the necessary skills
3. Involvement of the SAI
4. Planning the Audit

Section 2: Scoping the project

11. This section covers the SAI’s examination of how a procuring organisation decided the scope of a public/private finance and concession deal. In reviewing this subject the SAI will need to have a clear understanding of how the organisation chose this project ahead of other possible uses of its resources and what its objectives were in doing this. The SAI will also need to examine how the organisation structured the project to meet its needs and the role the organisation’s assessment of the private sector’s capabilities had in shaping this structure.

Guidelines

5. Selection of the project
6. Definition of project requirements
7. Private sector capabilities
8. Evaluation of potential benefits
9. Wider policy objectives
10. Selection of the most suitable form of partnership
11. Innovation
12. Risk assessment
13. Affordability and likely value for money
14. Outline business case

Section 3: Project management

12. This section covers the SAI’s examination of how a procuring organisation managed the process of awarding a public/private finance and concession contract. Thus the SAI will need to examine whether the organisation established a team, with the necessary skills and expertise, with responsibility for delivering the project successfully and whether it established a proper system of cost control. The SAI will also need to review the planning that the organisation undertook to ensure that there was an effective competition for the award of the contract and that any agreed deal offered value for money.

Guidelines

15. Project team
16. Market investigation
17. Contractual matters
18. Tender strategy
19. Project timetable
20. Public sector comparator
21. Tender list
22. Specification of requirements
23. Maintaining competition
24. Regular reviews
25. Budgets for project costs
26. Appointment of advisers
27. Cost management

Section 4: Contract

13. This section covers the SAI’s examination of how a procuring organisation selected a private sector partner and then negotiated the final contract with them. The SAI will need to examine that the organisation properly evaluated all aspects of the bids received, such as the allocation of risk between itself and the private sector and the price quoted, and then chose as preferred bidder that which offered it best value. The SAI will also then need to review the negotiations between the organisation and its preferred bidder to identify the effects any changes agreed in these negotiations had on the deal’s value for money.

Guidelines

28. Bidders’ design proposals
29. Bidders’ operational proposals
30. Financing the deal
31. Bidders’ suggestions for varying the specification
32. Allocation of risk
33. Financial assessment
34. Assessment of risk allocation
35. Assessment of bidders’ financial capabilities
36. Quality of service proposed by bidders
37. Choice of bidder
38. Changes during negotiations with winning bidder

Section 5: Value for money evaluation

14. This final section covers the SAI’s examination of how a procuring organisation gained reassurance, before committing itself by signing the contract, that the deal met its objectives, was affordable, and offered value for money.

Guidelines

39. Achievement of objectives
40. Evaluation of alternatives
41. Ensuring service delivery
42. Confirmation of affordability

Section 6: Managing the Contract

15. This final section covers the SAI's examination of how a procuring organisation managed the contract once signed. Thus the SAI will need to examine what steps the audited body took to ensure that it acted intelligently in managing the contract and that it paid the contractor according to their performance. The SAI will also need to review how the audited body managed changes and the allocation of risk between itself and the contractor and how, if applicable, the body managed its exit from the contract.

Guidelines

43. Managing the contract intelligently
44. Payment for performance
45. Management of change
46. Management of risk
47. Management of the contract’s expiry

Conclusion

16. Since the public/private finance and concession approach is a fairly recent development, and bearing in mind that good practice in successfully implementing such projects is still evolving, the guidelines proposed should be seen as suggestions and advice, put forward in the light of experience so far. They are not laws or set procedures which every SAI should apply in their entirety to every study. Their purpose is essentially to offer guidance to ensure that an audit of such deals follows a professional and structured approach and identifies useful lessons for future deals. We believe nevertheless that many of the issues addressed are likely to remain valid and relevant, however the public/private finance and concession approach develops.

PART 2: THE GUIDELINES

SECTION 1: THE GENERAL APPROACH OF THE SAI

In nearly all countries the responsibility for auditing the state agencies letting public/private finance and concessions contracts rests with the SAI. Such contracts can also however be awarded by regional or local government. Contracts let by these bodies may or may not fall within the remit of the SAI, depending on the auditing framework in place within a particular country. In any case the SAI needs to be clear who was responsible for what in awarding any contract and what is the SAI's remit for examining the deal.

In the field of public private finance and concessions, it can be particularly important that the state agencies letting such contracts exercise well-informed judgement and discretion. The SAI's responsibilities should not lead to the SAI substituting its own judgement for that of the audited body. On the contrary, it should be the aim of the SAI to encourage audited bodies to exercise their own discretion reasonably and wisely. In doing so, the SAI may well draw on lessons learned from audit examination of other cases in which state agencies have exercised their discretion.

Supplementary Guidance

The defining characteristic of public/private finance projects and concessions is that private sector entities become intimately involved in the delivery of services commissioned by the state: a private sector approach to a public sector problem. To provide the degree of looked for assurance to parliament and to the public about the value for money of such arrangements the SAI is likely to need a wider range of skills than might normally be deployed in the audit of a purely public sector project.

Thus, in addition to the more usual general performance audit skills required to review any complex or high-risk assignment, the skills necessary will include specific expertise relating to the service being put out to contract and the marketplace. The SAI may need to recruit staff with such specialist skills or otherwise to supplement its in-house skills with external advice from, for example, consultants in specialist fields offering up-to-date expertise.

Supplementary Guidance

The SAI faces a dilemma when reviewing public/private finance and concessions deals as to when to carry out its examination. In many cases the deal envisages that the private sector supplier will provide services over many years, even decades, in the future. Therefore it will only really be possible to make a final assessment of whether or not the deal has achieved value for money at the end of the contract in question. However this is too long a timescale if lessons are to be learned and improvements implemented in subsequent projects - and indeed in the project under examination. The need to demonstrate accountability also requires that such deals be examined sooner rather than later. The SAI should therefore examine public/private finance and concessions deals prior to the end of the contract. This will inevitably involve the SAI taking a view on how well the deal is likely to succeed in meeting the public sector client's requirements in the future. The SAI and the audited body will need to achieve the requisite degree of co-ordination in their activities so as to ensure the effectiveness of the audit at all stages.

A good stage for first examining these projects may be soon after the contract has been awarded. Examining the deal soon after contract signature has the merit that the terms of the deal are fixed; prior to this the terms may be constantly # changing as they are subject to re-negotiation.

In certain circumstances, however, and where this is constitutionally permissible, it may be necessary or desirable for the SAI to examine the deal before the contract is awarded; for example if concerns are being expressed about the probity or likely value for money of the procurement process. Examining a deal prior to contract award has the advantage that any weaknesses identified by the SAI can be corrected before the contract is signed and so more serious difficulties avoided at a later stage. The SAI may therefore choose, if the project poses significant audit risks, to examine it at each of the most significant stages in its procurement.

In carrying out an examination before the contract is signed, the SAI will need to manage the risks involved in such an early intervention. For example, the risk that the SAI’s examination could have an adverse impact on the tender process itself as the audited body may divert its scarce resources from negotiating the best deal possible for the public sector to dealing with the SAI examination, or the risk to the SAI itself that it might face a conflict of interest when commenting at a later date on some aspect of the deal which came about because of advice it gave on an earlier examination.

The early review by the SAI of a contract, before or just after its award, does not prevent the SAI from returning to examine it again at a later date. The management of a public/private finance and concessions contract, once signed, is also important in ensuring value for money.

Supplementary Guidance

Without good planning the SAI risks undertaking an audit that is ill-focused and lacking in the breadth and depth of evidence needed to secure a credible report. To form a view on a deal’s value for money the SAI will need access to the public sector body letting the contract. However, as public/private finance and concessions projects typically involve a wide range of third parties in addition to the public sector body letting the contract, the SAI will also need to obtain the views of these other parties if it is to reach sound conclusions about the deal.

The aim of any examination should be to identify lessons for the future so that either deals are procured more efficiently or better deals are reached. If the SAI is to be effective in this aim, its planning of the audit should include how best any lessons identified should be presented so that they are acted on in future.

It is important that the SAI's examination should, so far as legally permissible, take into account all relevant information, including material that may be commercially sensitive or confidential. Even if such material may not be or should not be published in the SAI's eventual report, it may well have an important bearing on the SAI's conclusions.

Supplementary Guidance

SECTION 2: SCOPING THE PROJECT

The use of public/private finance or concessions may be a means through which a project can be delivered earlier than if it had been financed as a purely public sector project. While this may be an advantage, it can mean that poor quality projects are developed simply because they can be financed in this way and that high quality projects, which can only be financed as public projects, are deferred. It therefore makes sense for the SAI to examine how the audited body considered its priorities amongst competing projects.

Supplementary Guidance

It is important for the successful delivery of any project that an organisation states its requirements clearly from the start. Otherwise there is the risk that it will end up with a project that does not meet its real needs. For example, on one rail deal the audited body did not resolve early enough its requirements for concessionary travel for certain categories of passengers. Consequently the government had to negotiate arrangements with the contractor after the award of the contract when its negotiating position was weaker.

At the start the organisation’s requirements will only be in outline, covering both current and forecast needs, but will then be developed in more detail as the project progresses. In order to avoid wasting both its own and the private sector’s resources in planning for projects which were never affordable from the start, the organisation will need to identify clear cost limits as to what it can afford.

The public/private finance and concessions approach can offer better value for money than conventional public procurement because it offers the private sector scope to offer innovation in areas such as service delivery. However the scope for such innovation is limited if the client fails to specify its requirements in output terms and instead concentrates on inputs and the mechanics of delivery, thereby placing unnecessary constraints on how the private sector can carry out the project. It is important therefore that an organisation specifies their needs in terms of deliverables. There may be good reasons why some particular constraints should apply, for example, there may be a need for charges to the public to be regulated, or the private sector may be required to take over certain public assets for a given price.

Supplementary Guidance

In determining how to express the desired outcomes, an organisation should have regard to private sector capabilities. This should ensure that any project is presented to the market in such a way that the subsequent procurement can progress in an orderly way. In some cases, for example a payroll service, the private sector may be very familiar with the type of outcome likely to be required and therefore easily capable of providing it. In others, such as novel applications of information technology, the private sector may need more help in understanding what the requirements are. The public sector partner should carefully consider, in each potential project, in what terms the private sector will best understand its requirements.

The SAI should therefore examine what initial soundings of the market an audited body made to ascertain how the private sector would best understand its requirements and of the extra help that the private sector would need to gain such an understanding.

Supplementary Guidance

At the early stages the audited body needs to make a preliminary assessment of the benefits it is seeking from the project for the provision of the service concerned. It should also identify any potential constraints as regards achieving these benefits. Among the benefits and constraints to be considered are the likely contract length and any requirements for flexibility within this period. Such an assessment will enable the body to set meaningful criteria for its later evaluation of tenders received. The audited body should not have established its criteria at the end of the procurement process simply to justify the deal reached. See also guideline 13 (affordability and likely value for money).

Supplementary Guidance

In many areas of government business, there can be policy or regulatory reasons which place constraints on how a particular service is to be delivered. Consequently there may have to be trade-offs between these general political requirements and the other financial or quality of service objectives specific to the project in question. For example, a government may require the protection of the pension and employment rights of those public sector staff transferring to the private sector under a public/private finance and concessions deal. Procuring bodies need to identify such policy imperatives and understand fully the basis for these so that they can include them in their specified requirements and communicate them clearly to the private sector. In one deal, for example, the audited body’s failure to consider sufficiently early their policy objectives on the protection of employees’ rights resulted in the need to ask for revised tenders as the initial tenders received did not address this issue satisfactorily. The SAI should consequently examine how an audited body took account of these policy objectives when planning the project and what weight it gave to these.

Supplementary Guidance

In some cases, it may be possible to leave open to the market exactly what form of partnership between the public and private sectors would best meet the procuring organisation’s objectives. More usually, however, it will be impractical to seek bids on such an open basis: giving bidders complete freedom might unduly complicate their work and make it difficult for the procuring organisation to make fair and thorough comparisons of one bid with another. This means that the organisation will probably have to limit the range of options before seeking bids. In so limiting the options there is a risk that the procuring organisation might inadvertently compromise value for money, for example by viewing as the procurement of services what is in reality the sale of a business opportunity. The SAI should thus review whether the audited body examined the possible use of a range of alternative project mechanisms before it chose to procure the project using the public/private finance and concessions approach.

Supplementary Guidance

The public/private finance and concessions approach is likely to produce improved value for money when scope is maximised for the private sector to develop innovative ways of delivering the services or outputs the procuring audited body requires (Guideline 6). For this to be achieved, it is highly important for the procuring organisation to ensure that any restrictions it may impose on innovation are justified. The SAI should therefore examine whether the audited body undertook a preliminary assessment of the scope for possible innovation to ensure that any restrictions on this that it identified were justifiable. This assessment will then form the basis of a proper system for the evaluation of any such innovation proposed by bidders.

Supplementary Guidance

The SAI should review whether the audited body investigated in advance the appropriate allocation of project risks between the public sector and private sector parties affected by the project.

Appropriate allocation of risk between the public and private sectors is a key requirement if the public/private finance and concessions approach is to deliver better value for money than conventional procurement. In many cases the private sector is better placed to manage risks which traditionally the public sector has borne. For this reason any costs attached to such risks should be lower if responsibility for them is given to the private sector. If however the public sector seeks to transfer a risk which the private sector cannot manage, value for money will reduce as the private sector seeks to charge a premium for accepting such risks. The public sector should therefore seek not the maximum but rather the optimum transfer of risk; that is, it should not attempt to transfer all risks at any cost but rather to allocate individual risks to those best placed to manage them.

For certain risks the public sector remains the best placed. In other cases it may make sense for risks to be shared, for example by allocating a risk to the private sector but with the public sector offering some kind of guarantee if that risk materialises. (In such a case, the SAI will need to examine the rationale for the guarantee and the basis for deriving the amount.) Also relevant to risk allocation may be the arrangements for the private sector to take over any public assets as part of the deal. Procuring organisations therefore need to think through in advance whether there are any aspects of the deal they are proposing to put to the market which are likely to stand in the way of allocating risks optimally; for example by creating new risks which would fall on the private sector supplier.

The accounting treatment of the project in the accounts of the audited body may well depend on the extent of risk transfer to the private sector. There can therefore be an incentive for audited bodies to transfer risk to secure a particular accounting treatment - even if such risk transfer is poor value for money. It is therefore particularly important for the SAI to examine the reasonableness of the transfer of risk in the project.

Supplementary Guidance

It is a waste of time and money to go out to competition for a project which could have been seen to be unaffordable or bad value for money even before bids were sought. Although the procuring organisation might reasonably challenge the market by seeking novel approaches, including novel approaches to risk transfer, it is often possible, with appropriate external advice, to predict when the market is unlikely to be able to deliver a proposed requirement at an affordable price. Prior to going out to tender the procuring organisation will therefore need to demonstrate that the proposed project is likely to provide value for money when compared to other options, that it can afford to pay for the project (Guideline 6) and that it is likely that the private sector can provide the services required (Guidelines 7 and 12).

Value for money consists not simply of the lowest cost but the best combination of cost over the whole life of a project and quality which meets an organisation’s requirements. The evaluation of value for money will therefore involve the assessment not only of financial benefits and costs but also non-financial factors.

Supplementary Guidance

It is good practice to bring together formally the case for proceeding with a public/private finance and concessions deal, so that an organisation’s senior management has clear evidence on which to take any decision to proceed. Without such a business case, there is the risk that a deal will be done because it can be done rather than because it should be done. At this early stage such a business case will only be in outline with a more detailed being prepared prior to the decision to award any contract.

Supplementary Guidance

SECTION 3: PROJECT MANAGEMENT

Just as the use of public/private finance or concessions may present the SAI with the need to deploy additional skills (Guideline 1), so may the audited body itself have needed to acquire special skills in order to negotiate such a contract. Whether the audited body was equipped with the right skills at the right time can have a major impact on the quality of the resulting contract. The reporting and decision making procedures within a project team will also need to be clear if the contract’s procurement is to proceed efficiently and effectively.

Impropriety within the project team can pose a major risk to value for money. The SAI will therefore need to identify the measures the audited body took to safeguard the impartiality and integrity of the procurement. For example, the project team members can be required to comply with the provisions of a published code of conduct.

Supplementary Guidance

As part of the planning of the procurement the audited body should investigate the market for the services in question. The purpose of this investigation is to inform the way the project is presented to the market, so that potential suppliers will be able to appreciate the full extent of the business opportunity it offers. It is therefore useful to discuss a potential project with potential bidders to discover what aspects of the project are likely to appeal to them and what may not be so attractive. This is particularly important where the project is of a new type. However, when holding such discussions, the audited body will need to have regard to possible legal constraints imposed by any procurement regulations.

Supplementary Guidance

The outcome of the procurement will be a contract with the private sector partner. It is therefore sensible in the planning stage of the procurement to consider what issues are likely to arise in negotiating the contract. The SAI should therefore examine whether early in the project the audited body identified the contractual issues that were likely to arise (for example, early termination of the contract, variations to the contract price) and drafted a proposed contract which reflected its preferred position on each issue. The aim should be to obtain as early as possible a contract which adequately protects the public sector’s position but which at the same time is acceptable to the private sector bidders and their bankers. Early legal advice will therefore be essential for the audited body.

Supplementary Guidance

Competition is essential if a procuring organisation is to get a deal which provides value for money. To ensure maximum competition the organisation will need to plan the procurement with this aim in mind. A key part of this planning is the tender strategy, which sets out the number of tender rounds and bidders at each stage, how the organisation will communicate with bidders, and the timetable for the tender exercise.

The principle which should govern the tendering strategy is to maximise competitive tension throughout so that tenderers will always feel under pressure to put forward their best possible bids. Competitive tension will be greatest when bidders are competing keenly with each other. Bidders are however unlikely to be willing to compete strongly if a procuring organisation asks an excessive number of firms to submit bids and commit the substantial resources that the preparation of a full bid involves. It will therefore make sense for an organisation, when preparing its strategy, to be sensitive to the time and costs that bidders incur in preparing bids and to limit the number of rounds to the minimum consistent with a successful competition.

Supplementary Guidance

When planning the procurement the audited body should have updated the project timetable in its outline business case (Guideline 14) to reflect the tender timetable contained in its tender strategy (Guideline 18). The project timetable will be more comprehensive than that for the tender exercise as it will need to reflect the detailed work to be done by the project team and the contractor before both the award of the contract and the start of service delivery. Where possible this timetable should contain milestones against which progress can be measured and set out the various points in the timetable where the audited body will pause to review critically whether it was still worth proceeding with the project.

Supplementary Guidance

The decision on whether a proposed public/private finance and concessions deal represents value for money will be better informed if the procuring organisation prepares details of a publicly funded alternative, known as the public sector comparator, against which the costs and benefits (both financial and non- financial) of the public/private finance and concessions deal can then be measured. Because of the importance of this comparison, the audited body will need formally to document this assessment. The SAI will also need to review the details of this comparison thoroughly, especially the calculation of the public sector comparator, in order to establish whether the costs and benefits are reasonable.

Supplementary Guidance

Competition is central to getting good value from public/private finance and concessions deals. One of the key conditions for a successful competition is a good tender list of firms invited to bid. Such a list cannot be expected to emerge by itself. A procuring organisation needs actively to stimulate interest in the proposed project, to publicise the procurement competition in accordance with the relevant law and regulations, and to give all necessary guidance to potential tenderers on how to submit good bids. The procurer may also need to assist the process through which firms can become aware of each other’s interest and come together into consortia. All of this must, of course, be done in accordance with the relevant legal framework.

After stimulating the market’s interest, the procuring organisation will need to select for its tender list firms with good technical, managerial and organisational capabilities. Without these capabilities, it is unlikely that a firm will be capable of providing the required services should it be awarded the contract. The procuring organisation should also, however, seek to assess, when selecting its tender list, a firm’s willingness and ability to submit a bid. Experience has shown that firms who express an interest and are then invited to tender sometimes fail to submit bids, resulting in a reduction in the extent of competition.

Supplementary Guidance

Another condition for a successful competition is a clear specification of the procuring organisation’s requirements. Tenderers cannot submit good bids unless they know clearly what is that the client wants. The lack of a clear specification may result in the submission of only a very limited number of bids that adequately meet the requirements, thus effectively reducing the extent of competition. However clarity in itself is insufficient. A clear specification may still result in poor value for money if it does not state the procuring organisation’s real needs or if it contains excessive detail, particularly detail focussed on inputs rather than desired outputs or outcomes. Specifying inputs or the methods of service delivery will limit innovation (Guideline 6). In determining whether a specification is sufficiently clear the SAI will therefore need to consider whether it addressed the organisation’s real needs and was sufficiently output-based. Particular attention will need to be paid to the quality of service identified as being required where one of the main objectives of putting a service out to concession is to improve its quality.

Supplementary Guidance

The maintenance of competitive tension throughout the procurement process is essential if the procurement is to be truly competitive. Competitive tension draws out of tenderers their best bids. Often though the terms of a deal can alter to the detriment of the audited body when it is left negotiating with a single potential supplier and competitive tension has effectively disappeared. Most procurements of public/private finance and concessions projects end with such negotiations between the audited body and their preferred bidder.

Supplementary Guidance

It is very important that a procuring organisation should not lose sight of the original purpose of a project during the course of detailed negotiations, to avoid continuing with a project that is no longer worthwhile and thus wasting management time and incurring of unnecessary costs. For this reason someone not directly involved in the negotiations should be responsible for considering, as the negotiations evolve, whether the expected outcome is likely to be fully consistent with the objectives originally set. In a number of cases, external consultants have been specifically appointed to carry out this monitoring role; in other cases internal audit have carried it out.

Any important departure should be reported to the procuring organisation’s senior management as soon as it is clearly established (or highly probable) and a decision should be sought on whether to change the objective or abort the deal. Senior management should in any event keep in touch with progress on significant projects.

The SAI should therefore examine whether, after the preparation of the outline business case (Guideline 14), the audited body’s project team regularly reviewed the project to identify whether it was still worthwhile and whether, at certain key points as procurement progressed, the body’s senior management sought assurance on this issue from the project team.

Supplementary Guidance

The SAI should examine whether the audited body set and controlled realistic budgets for all project costs, including internal and external resources.

As a major project in its own right, the procurement of a public/private finance and concessions deal often involves the procuring organisation in heavy costs. The principles of cost control apply. Consequently the audited body should have set budgets for its costs as early as possible. This may have been difficult because the body may be inexperienced in procuring public/private finance and concessions and have little knowledge of the amount of work that this involves. Although, however, it may be difficult to set a firm budget for costs at the outset of the procurement, that is no reason for failing to set an initial budget and revising it later when more is known about the work that needs to be done.

Budgets should have been set and controlled on a project basis and should have included not only expenditure on external advisers but also in-house resources. For external advisers it is good practice to set budgets for the individual tasks that they have to perform.

Supplementary Guidance

Good quality external advisers can be essential to the successful negotiation of a public/private finance and concessions deal. The best way to secure such advisers is after a proper competition since one can then choose from a range of good quality firms. The principle behind such competition is to obtain advisers who will provide the best value for money. They will not necessarily be the cheapest advisers, although that does not mean that there is no role for competitive pressure on the fees advisers wish to charge. Thus when choosing advisers the audited body should have taken into account not only the price but also the likely quality of advice. Once appointed the performance of advisers should have been monitored to ensure that they were providing the quality of advice required.

Supplementary Guidance

Cost control consists not only of setting budgets (Guideline 25) but also of the regular monitoring of costs incurred against these budgets. The audited body should therefore have carried out such monitoring for all costs, including both internal and external resources, and only increased budgets for well-defined additional tasks. The body should also have clearly identified the work it expected from its advisers in return for their fee. Unclear or incomplete specifications of the scope of the work required from an adviser can result in the need for an additional contractual agreement and the payment of extra fees.

As part of any examination of the management of project costs the SAI should review the appropriateness of the basis that the audited body chose for the advisers’ remuneration. The body should have chosen a basis which gave the advisers an incentive to meet the project’s objectives, while controlling its own costs. At the same time, if the external advisers were engaged to provide advice on the merits and method of proceeding with the project, the remuneration basis chosen should not create conflicts of interest or bias the advice given. If advisers were commissioned to carry out a significant volume of work, the SAI will wish to examine whether they were pressed to offer a discount to their normal fee rates.

Supplementary Guidance

SECTION 4: CONTRACT

One of the main ways in which the use of public/private finance or concessions can improve the value for money received by the public sector is through obtaining innovations. In examining the effectiveness of the procurement, it therefore makes sense for the SAI to see how far the audited body succeeded in eliciting from bidders a number of different and innovative solutions to the problem of delivering what the audited body wants. If there is not a good range of solutions, this may be a sign that the audited body’s specification was expressed too much in terms of inputs, thus restricting the scope for innovation. The SAI should therefore assess whether bidders suggested a variety of practical innovative solutions in the areas of asset design, service operation (Guideline 29), financing (Guideline 30), variations to the specification (Guideline 31), and the allocation of risk between the public and private sector parties (Guideline 32).

Design is the area with the greatest scope for innovation, not least because it has a large impact on the level of building and operating costs. The SAI should therefore check the amount of freedom bidders had to suggest different designs. For example, if the audited body provided bidders with suggested designs, the potential for innovation may have been constrained. Additionally, the audited body should not have rejected innovative designs simply because they challenged presumptions or established practices used in the public sector comparator.

Supplementary Guidance

Another area for possible innovation is how the eventual contractor will operate and deliver the required services. The extent of innovative service delivery will depend in part on the nature of the project. For example, in one country, prison projects using the public/private finance and concessions approach have required the contractors not only to provide support services but also to take over the custodial services in the prisons. In contrast on hospital projects the private sector is only responsible for support services while the core, clinical services continue to be provided by the public sector.

Where private sector proposals have been invited the SAI will need to assess whether the audited body was open to new ideas while at the same time checking whether these ideas were deliverable. The initial period of operations - for example, the opening of a new prison or the introduction of a new IT system - can pose particular risks. The SAI will therefore need to satisfy itself that the audited body had sought sufficient information from the bidders as to how they would deal with these risks.

Supplementary Guidance

Financing charges are usually a significant part of a contractor’s expected costs and new forms of financing public/private finance and concessions projects which offer reduced costs, such as the use of bond financing, are gradually being developed. The SAI should therefore check that the audited body had sought bids which were based on obtaining finance through competitive procedures and which took into account the best sources of finance the market could offer and had received bids which contained a number of different proposals for financing.

Supplementary Guidance

Although the audited body should have clearly expressed in the specification what they required to be delivered (Guidelines 6 and 22), bidders may have been able to add value by suggesting alternative deliverables. These alternatives could include, for example, the delivery of different requirements, different bases for measuring the aspect of performance required, or different methods of remuneration for successful performance. The audited body should be open to such alternatives and so far as may be legally permissible should conduct the procurement in such a way that alternative proposals from bidders can be considered.

The SAI will need to assess how the audited body balanced the need to be open to innovative suggestions with the need to comply with procurement law. The audited body will also need to have considered the desirability of treating all bidders fairly. The SAI should check whether, if the audited body decided to pursue a bid which did not conform to the original specification, it did so only because the variants contained in this bid offered the prospect of better value for money. The body should have obtained appropriate legal advice to support this decision.

Supplementary Guidance

Value for money will be maximised where risks are allocated to the parties best able to manage them (Guideline 12). The SAI will therefore need to identify whether the audited body received bids which contained an allocation of risks between the public and private sectors which was clear, likely to achieve value for money, and unlikely to give rise to major renegotiations during the closing stages of the procurement process.

In certain circumstances where there is a particular risk which is unusual, or where there is uncertainty about which allocation will produce the best value for money, the SAI should identify whether the audited body obtained information from bidders on the effect on prices of alternative risk allocations. This would have enabled it to evaluate whether or not it was worth transferring a particular risk or risks to the private sector. In general however the audited body should avoid asking bidders to prepare a large number of alternative pricing calculations based on different allocations of risk as this would risk deterring bidders by making large demands on the bidders’ time and by increasing their bidding costs.

Supplementary Guidance

The next stage after the receipt of bids is their evaluation by the audited body. The evaluation of bids on a public/private finance and concessions project is more complex than in a conventional procurement as it will need to take full account of all relevant financial and non-financial aspects of bidders’ proposals which could significantly affect the value for money of the deal. Thus the criteria for evaluation will not just be price, or price subject to technical compliance with the service specification. The audited body will need also to take account of such matters as the degree of risk-taking by the potential suppliers, the extent of innovation, and trade-offs between price and quality.

Because of the complexity of the evaluation it is highly likely that the exercise of informed judgement will be required. The exercise of this judgement should however be within the context of an evaluation framework which ensured that the method of evaluation was consistent with the audited body’s objectives for the project and with what tenderers had been told about their requirements. To be effective the audited body should have set up this framework early in the procurement (Guideline 8).

It may make sense at the start of the bid evaluation process for the audited body to evaluate separately each of these factors, for example through the use, pragmatically and without undue bureaucracy, of separate working groups to examine each major aspect of the evaluation. This helps to avoid any risk that the initial evaluation of the technical and quality of service aspects of the bids might be unduly biased by knowledge of the prices proposed. In line with this, this guideline covers the need for the auditor to examine the audited body’s separate evaluation of the financial implications of the bids, a great determinant of whether or not value for money has been obtained. Guidelines 34 to 36 contain guidance for auditors on the examination of the other separate strands of evaluation.

Supplementary Guidance

Another significant factor affecting the value for money offered by a bid is the proposed risk allocation. The audited body should therefore have compared the extent to which the different bidders were prepared to assume risk. Where bidders were prepared to accept different levels of risk the audited body should have assessed what effect this had on their proposed pricing of the contract. In some cases individual bidders may have provided alternative prices for the contract assuming different levels of risk transfer (Guideline 32).

The SAI should therefore check that the audited body had appropriate procedures for evaluating the value for money of bids that were based on differing levels of risk transfer. In addition, where a bidder had not been prepared to accept certain risks, the SAI should also check that the audited body had taken into account the cost to the public sector of retaining and managing these risks.

Supplementary Guidance

Guideline 21 highlighted the importance of a firm’s technical, managerial and organisational capabilities to the successful delivery of the services required. The financial viability of a bidding consortium and its members is, however, also critical to the success of a public/private finance and concessions project. The required service will not be delivered if, for example, the necessary financing fails to materialise. The SAI should assess what steps the audited body took, in addition to reviewing the bidder’s financial model to assess its financeability (Guideline 33), to examine that each bidding consortium had obtained commitments of funding in support of its financing proposals for the project (Guideline 30), and to check the financial viability of the members of the consortium. The audited body should have placed particular emphasis on the assessment of the financial position of those consortium members engaged in either construction or IT systems development work as a financial failure of such a member could result in serious delays to the provision of the required service.

Supplementary Guidance

The SAI should check what steps the audited body took to evaluate the quality of service proposed by bidders (Guidelines 28, 29 and 31). This evaluation should have included a technical assessment of whether a bidder’s proposed solution would deliver the services required. Given that this is an area where users of the service – such as the clinical staff in a proposed new hospital – may be able to contribute to the evaluation, the SAI should examine how the audited body involved these in this process. The SAI should also see whether the audited body established minimum levels of service quality and how it assessed the merits of bids which proposed a higher quality service but at higher cost.

Supplementary Guidance

As noted in Guideline 33, in the case of public/private finance and concessions projects the evaluation of bids is more complex than in a conventional procurement as it needs to take full account of all relevant financial and non-financial aspects of bidders’ proposals which could significantly affect the value for money of the deal. The final choice of bid is likely to turn on the trade-offs between price and quality proposed by the bidders. In such circumstances, the conclusions of the separate analyses of the aspects of the bids received (Guidelines 33 to 36) need to be brought together into a comprehensive ranking. In doing this the procuring organisation will then be able to address these trade-offs explicitly in their final evaluation of bids by taking account of both financial and non-financial factors and seeking to identify the bid that offered the best combination of these.

In the event that only one bid is received, it will be impossible for any such ranking to be constructed. The situation will have moved directly from an invitation to bid to negotiations with a single bidder. Guideline 38 will therefore apply.

When putting together the comprehensive ranking of bids, procuring organisations should avoid uncritically following some arithmetical scheme of weighting since it is very unlikely that such an evaluation scheme can accurately represent the value of differing bids in such a complex field as public/private finance and concessions projects. Such an evaluation scheme should not be used as a substitute for the necessary exercise of informed judgement when attempting to address the necessary trade-off between price and quality. Organisations should be prepared to be flexible and open to alternative, innovative proposals. For example, they should be careful not to exclude bidders’ innovative ideas for deliverables such as payment mechanisms (Guideline 31). Despite these caveats, evaluation schemes along the lines outlined above can provide a useful discipline in helping ensure that a contract is not awarded solely on the grounds of cost but went to the bidder offering best value, that is, the combination of price, quality of service, and risk transfer which best suits their needs.

Since this final evaluation of bids is a critical phase in the procurement exercise, the SAI should be concerned to see whether the audited body selected the best available deal using consistently applied evaluation criteria.

Supplementary Guidance

Because of the complexity of public/private finance and concessions deals, bidders have often been unable to make final bids which were unconditional, resulting in an often lengthy period of exclusive negotiations with a preferred bidder. During such a period, some weakening of competitive tension is almost inevitable. Unless the procuring organisation can keep a careful control on concessions to the preferred bidder, there is a high risk of the terms of the deal changing so that they are less favourable to the procurer. Ideally these exclusive negotiations should be concerned purely with the detailed terms of the deal as the matters of principle underlying these details should have been agreed prior to the selection of the preferred bidder. However the worsening of the terms of the deal for the procurer often happens when important matters of principle are still not agreed at this selection point. For example, on one vehicle fleet project, negotiations with the preferred bidder began before a clearly superior outcome in terms of risk and return had been identified. The SAI should therefore examine how the audited body managed this phase of the procurement.

Supplementary Guidance

SECTION 5: VALUE FOR MONEY EVALUATION

A key requirement of any project is that it should do what it was intended to do. However it is possible that a deal where the contract has been awarded may not meet the audited body’s objectives and so not offer value for money. This can arise because a public/private or concessionary project is often highly complicated and takes many months to negotiate. Consequently there is a risk that the audited body may have lost sight of its original objectives. Alternatively the audited body’s objectives may have changed during this period. It may also be the case that the terms necessary to meet the objectives were simply not available in the market.

To avoid continuing with a deal which was unlikely to meet its objectives, the audited body’s senior management should have been reviewing at regular intervals during the procurement whether the deal was still worthwhile (Guideline 24). It is especially important that the body undertook such a review before it signed the contract since the audited body will not be easily able to back out of a deal after this point. The SAI should therefore check that, immediately prior to signing the contract, the audited body’s senior management reviewed the deal negotiated, to ensure that it met its objectives, either those originally set or the latest ones if these have been amended. The review should also have taken full account of any policy changes that were likely to occur in the foreseeable future. The SAI can then take assurance from their examination of this review that the deal signed met the audited body’s objectives.

An audited body should be wary of changing its objectives in a way which might be seen by potential tenderers as fickle, as that will reduce the public sector’s future prospects of getting good public/private finance and concessions deals. If therefore an audited body changed its objectives during the procurement, the SAI should check whether it took appropriate advice and examine whether any legal problems might arise from such a change for example the risk of legal challenge by the bidders.

Supplementary Guidance

To establish that a deal is good value necessarily requires a procuring organisation to satisfy itself that it is better than the realistic alternative option or options. To do that it will need to carry out a systematic, comprehensive and thorough comparison. Such a comparison will have begun in the outline business case (Guideline 14), and have been worked up in more detail during the planning phase of the procurement (Guideline 20). The comparison should then be maintained throughout the procurement process so that at the end the procuring organisation can be sure that the contract it wishes to sign is the best way of meeting its objectives.

Such a comparison may make use of financial modelling, but should not be limited to that. The SAI should examine whether the audited body had analysed and understood all the key differences between the proposed deal and the alternatives. As there is a particular risk that financial models will be erroneous, will be used erroneously, or that more weight will be given to their output than the uncertainty of the assumptions warrants, the SAI should pay particular attention to these aspects of the evaluation.

Supplementary Guidance

If a contract is to ensure the delivery of the services required, the arrangements for monitoring the contract and the contractor’s performance will be crucial. The SAI should therefore assess the adequacy of the audited body’s arrangements for these. The SAI will also need to review the provisions in the signed contract for termination and handover, the audited body’s rights of access to the contractor’s records, and the payment regime, to assess whether the contractor has sufficient incentives, and the audited body adequate safeguards, to ensure that the service will be delivered to the required standards. Where the contract provides for the public sector to share in surplus income arising from the private sector’s operation of the project, the SAI will need to examine the mechanism for identifying and accounting for such surplus. The SAI should also assess the contract provisions for handling any changes to the audited body’s requirements and other variations to the contract price, such as indexation and market-testing. These provisions should include adequate procedures for resolving any disputes over the pricing and any other aspect of proposed changes.

Supplementary Guidance

Public/private finance and concessions deals represent a contractual commitment by a procuring organisation to pay for services, provided they are delivered. As such, these deals reduce the future flexibility of the organisation to manage its expenditure. It would therefore be irresponsible for the organisation to enter into such a deal without verifying that it would have access to the funds needed to meet the contractual commitment. This verification would have to cover both any capital costs, which might arise in the short-term, and long-term ongoing commitments to pay the private sector partners.

Supplementary Guidance

SECTION 6: MANAGING THE CONTRACT

Guideline 43
Managing the contract intelligently

Guideline

The SAI should examine the steps the audited body has taken to ensure that it manages the contract intelligently.

Reasons for the guideline

Savings and other benefits do not automatically flow from a Public/Private Finance Concession contract. Whatever the value for money of such a contract when it is first signed, the subsequent poor management of this contract can result in higher costs, wasted resources, impaired performance and public concern. The SAI should therefore examine whether the audited body established a team that was sufficiently resourced and with the necessary expertise to manage the contract intelligently. The audited body should understand what team the private sector is providing and match those responsibilities. The team established by the audited body needs an understanding of the contract and its context, an understanding of the business and its objectives, and strong negotiation skills.

The audited body should have maintained that team’s expertise during the period of the contract and regularly reviewed its performance in managing the contract, as well as periodically re-evaluating the value for money offered by the contract. The SAI should look for evidence that, as a result of lessons identified in such reviews, the audited body sought improvements in existing and subsequent contracts, where possible.

Confusion over who is responsible for what can also result in ineffective management of a contract. The SAI should therefore review whether the roles and responsibilities for the management of the contract, and the procedures to be followed, have been clearly defined. However the correct application of procedures and compliance with agreed responsibilities will not, by themselves, be sufficient to guarantee value for money. Delivery of the contractual requirements depends also on the individuals involved. The management of the relationship between the audited body and the contractor will also need to be examined by the SAI.

Guideline 44
Payment for performance

Guideline

The SAI should examine whether the audited body has monitored the contractor’s performance adequately and paid the contractor accordingly.

Reasons for the guideline

In public/private finance and concession projects the payments made to the contractor during the course of the contract are dependent of the contractor’s performance in delivering the service required. The monitoring of contractor performance is therefore crucial if such a contract is to provide value for money. The SAI should therefore review the audited body’s monitoring of contractor performance and identify the actual level of performance achieved and whether payments to the contractor have been adjusted accordingly. The SAI will also need to examine whether there has been proper financial control over the payments made.

On one project there were marked deficiencies in a department’s payment control system which put at risk some of the objectives in signing the contract in the first place.

Guideline 45
Management of change

Guideline

The SAI should review the audited body’s management of changes during the contract period to its requirements and to the provisions of the contract itself.

Reasons for the guideline

Given the long-term nature of public/private finance and concession projects it is inevitable that there will be change during the course of such contracts. There will be changes in the requirements of the client for whom the services are being provided. It may also be necessary to negotiate with the contractor changes to the contract provisions. Such changes can clearly affect the value for money of the contract, not always for the worse. For example, on one prison project the contractor had been able to re-finance the project more cheaply because they had succeeded in building and operating this more successfully than had been assumed at the signing of the contract. As the refinancing of the deal required changes to the terms of the contract, the government client was able to share in the benefits of the refinancing in return for its agreement to the necessary contract changes.

The SAI should examine the audited body’s management of changes to identify whether there has been proper control and no adverse impact on value for money. It is possible to build into the original contract considerable scope for change. In those cases where the changes made have required the contract itself to be amended or renegotiated, the SAI should examine the reasons why.

Guideline 46
Management of risk

Guideline

The SAI should examine whether the audited body has succeeded in maintaining the allocation of risk laid down in the signed and contract and has managed effectively those risks it retained.

Reasons for the guideline

The proper allocation of risk is one of the key determinants of the value for money offered by a public/private finance and concession contract when signed. However the allocation of risk is not static and can change during the course of a contract. Such change can be to the detriment of the audited body if not managed effectively. The SAI should therefore examine the audited body’s management of the allocation of risk. In addition, it is unlikely that the private sector will accept all risks. Responsibility for some risks will remain with the audited body and the achievement of value for money from the contract is also dependent on how effectively the body manages the risks it retains. The SAI should examine the audited body’s management of these risks. As the management of risk in general is both a key responsibility of the senior management of public sector bodies and also a matter on which SAIs expect to focus, the SAI should consider, in examining risk management in the context of the deals to which these guidelines apply, whether the audited bodies have applied appropriate formal risk management methodologies.

Guideline 47
Management of the contract’s expiry

Guideline

The SAI should examine whether the audited body planned and managed effectively its exit from the contract on its expiry.

Reasons for the guideline

The final risk to the value for money comes at a contract’s end when poor management can adversely affect the value for money of any subsequent arrangement for service delivery that the audited body enters into. It is important therefore that the audited body carries out adequate planning towards the end of the contract, so that it can make informed decisions about the options available to it on the contract’s conclusion, and then manage that exit properly.

GLOSSARY

• Return to Intosai Working Group home page