INTRODUCTION

1. Following a decision taken by the XIV INCOSAI in Washington in 1992, the INTOSAI Working Group on the Audit of Privatisation was formally established by the Governing Board in May 1993 under the Chairmanship of Sir John Bourn, Head of the United Kingdom National Audit Office.
2. The membership of the Working Group comprises representatives from the SAIs of:

Albania	India
Antigua and Barbuda	Israel
Argentina	Lithuania
Australia	New Zealand
Austria	Norway
Bahamas	Paraguay
Chile	Peru
Czech Republic	Poland
Denmark	Russia
Ecuador	Saudi Arabia
Egypt	Turkey
El Salvador	United Kingdom (Chair)
Estonia	Uruguay
Germany	Yemen
Hungary	Zambia

 

3. In recent years Governments have been experimenting with a variety of ways in which the public and private sectors can work together to get better value for money for the taxpayer in delivering public services. This new development depends on harnessing private sector business skills and capital to identify scope for innovation in the delivery of such services and is now being carried forward in a series of partnerships between the public and private sectors. Therefore in November 1998 the XVI INCOSAI invited the Working Group to develop guidance on the audit of an important example of such public/private partnerships, the public/private finance and concession approach.
4. Public/private finance and concession contracts typically involve public sector clients specifying services which they wish to purchase and, through competition, selecting private sector suppliers to provide them. Alternatively, they can involve the award of a concession to a private sector supplier who then charges the general public for the use they make of the services provided. Sometimes it is expected that the private sector will need to construct new physical assets, such as buildings, to provide the specified services. And sometimes existing public sector assets may be transferred or rented to the private sector supplier.
5. This new approach to providing services offers the prospect of better value for money than traditional publicly financed methods. It is however also both new and more complicated than the traditional methods and brings with it new pitfalls and new risks to value for money. The approach has also been extended from its initial use in infrastructure projects such as roads and bridges into areas previously seen as core state activities, such as the provision of custodial services for the incarceration of offenders. All this means that securing the potential benefits of the public/private finance and concession approach is requiring new skills on the part of the public sector.
6. The purpose of these guidelines is to provide a logical framework for SAIs wishing to audit these agreements to establish whether the public sector has got the best possible deal for the taxpayer. The guidelines aim to cover all the issues potentially involved on such deals, although not all the guidelines will be relevant in each case and an SAI may not have the powers necessary to follow each guideline in full. None of the issues addressed in the guidelines contained is straightforward; all require the SAI to be well informed and balanced in reaching judgements on how well the body procuring the services managed the process, faced with uncertainty and, for many, a new method of procurement, and with conflicting objectives. Each guideline therefore has been drawn up in a format designed to bring out the reasoning and experience underlying it. To this end, there are two parts to each guideline:

• the guideline itself; and
• the reasons for the guideline.

These parts are contained in INTOSAI’s published guidance [INSERT EVENTUAL TITLE OF INTOSAI PUBLICATION]. This document gives more detailed advice on how to follow the guideline and is only available on the INTOSAI website. The guidelines themselves are grouped in five sections.

7 If the SAI is to carry out well thought through performance audits, leading to worthwhile recommendations and the spread of good practice, it needs to determine its remit in examining a public/private finance and concession contract and plan its audit thoroughly. It will also need access to a wide variety of relevant skills which are unlikely to be available within the SAI itself.

8 This section covers the SAI’s examination of how a procuring organisation decided the scope of a public/private finance and concession deal. In reviewing this subject the SAI will need to have a clear understanding of how the organisation chose this project ahead of other possible uses of its resources and what its objectives were in doing this. The SAI will also need to examine how the organisation structured the project to meet its needs and the role the organisation’s assessment of the private sector’s capabilities had in shaping this structure.

9 This section covers the SAI’s examination of how a procuring organisation managed the process of awarding a public/private finance and concession contract. Thus the SAI will need to examine whether the organisation established a team, with the necessary skills and expertise, with responsibility for delivering the project successfully and whether it established a proper system of cost control. The SAI will also need to review the planning that the organisation undertook to ensure that there was an effective competition for the award of the contract and that any agreed deal offered value for money.

10 This section covers the SAI’s examination of how a procuring organisation selected a private sector partner and then negotiated the final contract with these. The SAI will need to examine that the organisation properly evaluated all aspects of the bids received, such as the allocation of risk between itself and the private sector and the price quoted, and then chose as preferred bidder that which offered it best value. The SAI will also then need to review the negotiations between the organisation and its preferred bidder to identify the effects any changes agreed in these negotiations had on the deal’s value 4for money.

11 This final section covers the SAI’s examination of how a procuring organisation gained reassurance, before committing itself by signing the contract, that the deal met its objectives, was affordable, and offered value for money.

12 Since the public/private finance and concession approach is a fairly recent development, and bearing in mind that good practice in successfully implementing such projects is still evolving, the guidelines proposed should be seen as suggestions and advice, put forward in the light of experience so far. They are not laws or set procedures which every SAI should apply in their entirety to every study. Their purpose is essentially to offer guidance to ensure that an audit of such deals follows a professional and structured approach and identifies useful lessons for future deals. We believe nevertheless that many of the issues addressed are likely to remain valid and relevant, however the public/private finance and concession approach develops.

SECTION 1: THE GENERAL APPROACH OF THE SAI

Guideline 1
SAI responsibilities

The SAI should identify what its audit responsibilities are in relation to public/ private finance projects and concessions and decide how to carry these out.

None.

Return to Guideline

The SAI should identify and secure the core in-house skills it needs to carry out authoritative studies of public/private finance projects and concessions, and supplement these skills with expert external support as necessary.

Supplementary guidance

Where the SAI carries out performance audits of the contract letting process, the skills required can be divided into two groups. First, there are the general performance audit skills required to undertake the effective review of any complex or high-risk assignment. Since public/private finance and concession contracts are a recent development and good practice in the area is still evolving, auditing these deals will require the exercise of considerable judgement. Therefore the SAI will generally need to put its ablest performance auditors on this work.

Second, because of the complexity and specialist nature of many of the areas involved in achieving a successful public/private finance and concessions deal, the SAI needs to have access to specific skills relating to the service being put out to contract and the marketplace. These will include, in addition to contractual and legal skills, knowledge of the financial markets, in particular to project financing and the conditions and information required by financiers, together with financial, analytical and economic skills.

It is unlikely that the SAI will have in-house all the necessary skills and expertise in these specialist areas to discharge its responsibilities. In this situation the SAI will need to buy in the particular specialist skills. Figure 1 lists some of the consultancy skills the SAI may need to draw upon when examining public/private finance and concession deals and the areas in which consultants may be able to help. The actual consultants used will depend on the nature of the project being examined and the SAI’s own analysis of what skills are required and the extent to which these are already available in-house.

Experience suggests that the selection of such consultants should place at least as much emphasis on the individual consultants who will be providing the advice as on the consultancy firms who employ them. Individuals with good prior experience of the public/private finance and concessions approach are preferable. The consultants selected will need to have the respect of the audited body as well as the SAI if their findings are to carry weight. Regardless of the size and prestige of the firm selected, they will still require careful management by the SAI.

It may also make sense for SAIs to recruit experts in some or other of the areas of advice covered by consultancies, especially for example where the state is embarking on a major programme of service procurement using the public/private finance and concessions approach, because SAIs will also need to develop their range of core in-house audit skills in response. The SAI’s consultants could help with this. SAIs have also found it very helpful to exchange experiences and information with each other, and to exchange staff on secondments, as part of this learning process. It can also be valuable for the SAI to second some of its staff to external specialist firms (and vice versa) to obtain the necessary expertise.

Return to Guideline

The SAI should examine the public/private finance and concessions deal as soon as is practical, usually after the contract is awarded.

None

Return to Guideline

In planning the audit of a public/private finance and concessions contract, the SAI should plan to cover all major aspects of the deal that have a bearing on value for money, as set out in the following guidelines, to identify the key parties to the deal and, where possible, to take evidence from them, and to be alert to identifying lessons for the future.

The SAI will need access to the public sector body letting the contract as soon as possible after the contract award since the contract team may be dispersed to other work, especially where the contract is a unique event for the body concerned. Without such access the SAI will be unable to examine such issues as the project’s objectives, key criteria for success and the views of those who negotiated the contract on lessons for the future.

It will also be important for the SAI to plan to seek, where possible, the views of other parties with an interest in the project. public/private finance and concessions projects typically involve a wide range of third parties in addition to the public sector body letting the contract. These parties can include for example those representing the end users of the new facilities (for instance, doctors and nursing staff on hospital projects, and road user groups on roads), local organisations such as local government bodies, and representatives of staff providing the service prior to the deal being concluded. A particularly important group will be the firms involved either successfully or unsuccessfully in tendering for the contract. Figure 2 (over) lists potential topics for discussion with these firms. The SAI must quickly establish who the interested parties are and plan effective means of taking evidence from them about the project. Techniques to identify the views of these parties include interviews, structured questionnaires and attitudinal surveys.

If there is a gap between the award of the contract and the SAI’s report, the SAI will need to be alert to post-contract award events which may cast a light to the extent to which the audited body met their project objectives. For example, the private sector contractor may fail to begin delivery of the services required by the date given in the contract or fail to meet the required performance standards when service delivery begins.

As part of its planning of the audit the SAI will need to consider how best to present any lessons identified during the examination so that they are acted on in future. For example the SAI could choose to present any lessons learned in the form of a management letter to the audited body. Alternatively, it could publish these as a formal report to parliament. In deciding on the method of disseminating these lessons, the SAI will also need to take into account the effect this might have on its relations with parliament and the government.

Return to Guideline

SECTION 2: SCOPING THE PROJECT

The SAI should examine how the audited body prioritised potential projects and whether it implemented projects in that priority order.

In many public/private finance and concessions projects, the private sector finances the construction of a major physical asset, operates it, and recovers the cost over time through charges either to the client organisation for services rendered or through charges to the general public for their use of the asset involved (eg toll bridges). Consequently the client does not need to find all the money for the capital asset up-front during its construction. For this reason, the public/private finance and concessions approach can be very attractive to public sector organisations since it allows them to undertake projects which they might be unable to finance conventionally.

In this attraction, however, there lies the risk that the organisation’s priorities may be distorted in favour of those projects which are capable of being run as public/private finance and concessions and against those which meet the body’s real needs. Therefore, when selecting any project to be undertaken, organisations need to establish some means of choosing projects by reference to wider programme objectives which offset such potential distortions. They need to think hard about the volume and quality of services they require and the full costs of their provision over the long-term, regardless of the eventual method of procurement.

The SAI should therefore identify whether the audited body established as part of its strategic planning a clear need for the services eventually procured. Since it is possible that such planning may have identified many potential projects, not all of which could be afforded, the SAI may also need to consider whether the audited body prioritised and selected the projects identified in accordance with criteria which reflected its policy objectives and overall business strategy.

Return to Guideline

The SAI should examine whether the audited body stated its requirements clearly from the start and expressed these in output terms.

Although at the early stages it is not usually necessary for an organisation to specify its requirements in detail, there should be formal objectives for the project setting out in outline the requirements. The SAI should therefore examine whether an audited body set such objectives and whether these requirements were based on a proper analysis of its needs. This analysis should cover not only current needs but also some consideration of how these needs might change over time.

The objectives set should have specified the body’s needs in terms of deliverables (ie outputs, not inputs) if they are to maximise the scope for private sector innovation. The SAI should therefore examine whether the deliverables specified by the audited body focussed on what was required and not on how they should be achieved. This can be difficult for the audited body since traditionally many such bodies have themselves been managers or providers of services and directly responsible for the construction of capital assets. This means that they have in the past necessarily focussed on inputs (numbers of staff, physical specifications of buildings and so forth). So the audited body needs to ensure that those of its staff responsible for negotiating public/private finance and concession deals are trained to do so.

As part of its setting of objectives for the project, the audited body should have identified what money it had available to pay for the services to be procured and set cost limits accordingly. This is important as public/private finance and concessions deals represent a contractual commitment by a procuring organisation to pay for services, provided they are delivered. As such, these deals reduce the future flexibility of the organisation to manage its expenditure. For example, it will be harder than under traditional procurement for the organisation to cut back on the level of maintenance of the assets involved, when faced with demands to reduce its spending, as such cutbacks will need to be agreed with the contractor. The SAI should therefore check that the audited body formed a view early in the project of its likely cost and whether it would have sufficient funds to meet this.

Finally, as part of specifying the outputs required, the audited body should have given close consideration to the level of their likely use of the services being sought. In a public/private finance and concessions deal an organisation is usually committed to a certain level of service under the contract for a number of years. Gaining release from this obligation will need the contractor’s agreement and may well prove expensive. In comparison, under traditional procurement, if the organisation ceases to need a particular service, it can stop it immediately and sell any assets involved in the provision of this. (There are however limits to the level of flexibility available under traditional procurement. The ability to dispose of unwanted assets, which, under traditional procurement are paid for up-front, was often constrained because of the poor condition of these due to the cutbacks in their maintenance. Public sector assets are often highly specialised and, in the case of public sector accommodation, often in locations which are not attractive to the property market. Such assets can therefore be difficult to dispose of or have little market value.)

It is possible to build into a public/private finance and concessions deal some flexibility on usage. For example, on some past property deals, the public sector have negotiated the right to vacate in the future a certain percentage of the accommodation currently occupied for no extra cost on top of their usual service charge. The audited body should therefore have identified in advance the likely level of flexibility in respect of its usage of services it required from the private sector. The body’s examination of this issue should have been thorough since, if the body’s actual level of usage is different to that forecast, it could end up either paying for unnecessary services as its level of usage is lower then expected or for unrequired flexibility if usage is higher.

Return to Guideline

The SAI should review whether the audited body made a preliminary assessment of the private sector’s capabilities for delivering the requirements.

None

Return to Guideline

The SAI should identify whether the audited body made a preliminary evaluation of the benefits it sought.

The SAI should examine whether the audited body prior to going out to tender undertook a preliminary assessment of the benefits it was seeking from the project and used the results of this to establish criteria for the evaluation of the benefits offered by the bids. Figure 3 lists some of the relevant criteria for such evaluations. The SAI should check that the audited body used these criteria as the basis of its evaluation of tenders and did not draw up new criteria at the end of the procurement process simply to justify the deal reached.

Return to Guideline

The SAI should review how the audited body assessed the impact any wider policy objectives might have on the project.

None

Return to Guideline

The SAI should examine whether the audited body examined a range of alternative ways of meeting its needs, such as public sector traditional procurement or privatisation, before choosing the public/private finance and concessions option.

Figure 4 gives a number of alternative project mechanisms which may be available to an audited body when considering exactly what form of partnership with the private sector would best meet their objectives. The choice of partnership will in part be influenced by who is to pay for the service provided. For example the project may involve a public sector body procuring and paying for a service itself. Alternatively it may involve the granting of a concession to a contractor who then charges the general public for the use of the services provided.

Return to Guideline

The SAI should review whether the audited body identified the room for innovation in advance in areas such as design and construction, operation, and project financing.

Figure 5 lists possible areas for innovation and what points the SAI needs to be aware of in each when considering the scope for innovation in a project and the limits placed on this by an audited body.

Return to Guideline

The SAI should review whether the audited body investigated in advance the appropriate allocation of projects risks between the public sector and private sector parties affected by the project.

The SAI should check whether an audited body identified at the start the risks associated with the project and the scope for their transfer to the private sector. This preliminary identification of risk will be closely linked to the body’s early consideration of the contractual terms they will be seeking from the private sector (Guideline 17) as it is the contract which enshrines the eventual agreed allocation of risk between the public and private sectors.

On the scope for risk transfer, the body should have made some initial contact with the market in order to identify what risk transfer the private sector were likely to be willing to accept. The SAI may need the help of an experienced external consultant in assessing whether the transfer of risk initially proposed was reasonable. A good indication of whether the original proposals were unreasonable would be if the actual risk transfer differed greatly from that originally identified.

There are a number of individual risks the audited body should have paid particular attention to. For example, on demand risk, Guideline 6 mentioned the need for the body to consider the likely level of the services delivered under a public/private finance and concessions contract. As part of its examination of risk allocation, the body should have considered the extent to which it would seek to transfer this risk to the private sector and the basis on which it should do this. The basis of the risk transfer can vary. For example, payments to the contractor can be linked in whole or in part to the level of usage of the services supplied, rather than to their availability. Alternatively the audited body could require the right to reduce the scope of the services supplied without incurring a financial penalty (Guideline 6). The audited body could also negotiate the right to exit the whole of the contract, regardless of whether the contractor has defaulted, at set points within the contract period without the need to pay the contractor compensation. It is particularly important that the audited body sounds out the market early on the extent of transfer of demand risk and the basis for this as, on many projects, the private sector will refuse to accept this risk or will only do so by including a large premium for it in the quoted service charge. The audited body should therefore have considered very carefully whether its plans to transfer demand risk are likely to result in value for money.

The audited body should have considered closely the risk of fraud. The body should therefore have included in its planned contract terms provisions covering the occurrence of fraud and corruption.

The SAI should also examine how thoroughly the audited body considered the extent of risks retained in the public sector as for certain risks this may offer the best value for money. In identifying risks remaining with themselves the audited body will need to have examined closely any warranties or indemnities sought by the private sector from themselves as these can effectively limit the extent of any risk transfer to the private sector. (There is relevant guidance also in Guideline 12 of the Guidelines on Best Practice for the Audit of Privatisations.)

One risk retained by the public sector is contract renewal risk. This risk here is that the audited body is unable to relet the contract without proper competition at its natural expiry or early termination. The SAI therefore should check that the audited body considered, even at an early stage, what was to happen at the end of the contract. The body should have identified in advance the core skills which needed to be retained in-house, and the provisions that needed to be included in the draft contract, to ensure its ability to relet the contract competitively if necessary.

The SAI should also be aware that often the steps taken to manage an identified risk can result in the creation of a further, secondary risk. The SAI should therefore check that the audited body’s identification of the risks associated with the project also included an examination of such secondary risks.

Effective risk management involves not only the identification of each risk involved and who will take responsibility for this but also the formulation of a strategy for dealing with that risk. The SAI should therefore examine whether the audited body, after identifying the risks involved, considered the arrangements for managing the risks that it retained. These arrangements will be closely linked to those introduced to manage the public/private finance and concessions contract, when awarded, and the contractor’s performance under this.

For their management arrangements to be effective the audited body should have identified the core skills that it needed to retain in-house to perform this management function. The body should have put these management arrangements in place prior to the award of the contract to ensure that the staff who are responsible for monitoring the contract and managing retained risks are familiar with these and receive any necessary training. The audited body should have also planned for the arrangements to come into effect not only once service delivery begins but also prior to this during the development or construction stage. This should help ensure that service delivery begins on time.

Return to Guideline

The SAI should examine the extent to which the audited body considered, before starting the procurement process, whether the project was likely to be affordable and offer value for money.

The SAI should examine whether before going out to tender the audited body prepared an outline appraisal which demonstrated the project’s value for money. This appraisal should have included the following elements:

• an estimate of the likely capital and operating costs using traditional procurement;
• the economic impacts on the public sector if the project were to proceed using the public/private finance and concession approach; and
• the likely project management costs.

The SAI should also check that, in addition to the above financial factors, the audited body identified the non-financial benefits and disadvantages of each option in order to make a complete assessment of the value for money of each.

The audited body should also have confirmed that it could still afford to pay for the project by comparing the costs identified above for each option with the likely level of funding available in the short and long term (Guideline 6). It should also have identified that it was likely that the private sector could provide the services required (Guidelines 7 and 12).

Return to Guideline

The SAI should examine whether the audited body prepared a proper business case to support the decision to begin the project’s procurement.

The SAI should examine whether an audited body produced an outline business case before proceeding with the project. This case should summarise the planning work previously undertaken and which the SAI should have already examined in depth. Figure 6 lists the contents usually found in such a case.

Return to Guideline

SECTION 3: PROJECT MANAGEMENT

The first step in the thorough planning of the procurement of a public/private finance and concessions deal is to assemble in good time a properly qualified project team. This is crucial if any contract is to be awarded successfully. Ideally the team should be set up once the procuring organisation has decided that the project is feasible and that procurement should proceed.

An organisation should only put such a team in place after it has identified what skills it requires at the various stages of the project in order to ensure its success. project management is important to the successful completion of any project since a project manager not only sets timetables for all stages of the project, and monitors progress against these, but is also responsible for seeking to resolve problems as they arise so that they do not cause delay. The team should also include at least one member with knowledge of and responsibility for ensuring compliance with any relevant procurement rules. The SAI should therefore review what skills and expertise the audited body made use of in the procurement.

Having considered what skills it is likely to require, the procuring organisation should identify the sources of these skills. In a complex procurement like a public/private finance and concessions deal it is highly unlikely that all the necessary skills and expertise will be available in-house. To the extent that they are not, there will therefore be a need for external advisers with previous experience of negotiating public/private finance and concessions or other partnership types of deals to complement the in-house expertise (Figure 4: Alternative project mechanisms - Guideline 10). Such advisers should be appointed early in the project since their commercial experience should help prevent the project being developed in a way which is unattractive to the market. Good legal and financial advice is particularly important, and the SAI should check whether the audited body actively took up the references of prospective advisers.

In certain circumstances, for example where an organisation is procuring a series of public/private finance and concessions deals, it may not be necessary to set up a completely new team for a project as one may already exist from a previous deal. The team will therefore be experienced in many of the issues involved in negotiating such deals. However an organisation will still benefit from an analysis of the expertise required for a new project since each public/private finance and concessions deal will have unique features. For example, although the new project may involve the provision of accommodation as on previous projects, the ownership of the site in question may be less straightforward than before, requiring legal advice in new areas.

Finally the audited body should have established reporting and decision making procedures within the team to ensure proper control of the procurement. To ensure speedy decision-making, clear levels of delegated authority should have been established not only within the project team and but also with users. A member of the body’s staff should have been designated as "project sponsor" to provide the focal point for the body’s input into the project. There should also have been a project champion within the body’s senior management to help the team cut through problems which could not be resolved without reference to senior management.

Return to Guideline

The SAI should examine how the audited body investigated the market prior to beginning the formal procurement, to establish that there were suppliers who were willing to tender for the project.

The SAI should check whether the audited body investigated the market sufficiently early in the project to ensure that there were suppliers who were willing and competent to bid for the work. In examining this consultation, the SAI should also take into account any investigation of the market that the audited body undertook when assessing the project’s feasibility (Guideline 7). If the body has undertaken such consultation, it will need to have avoided conferring preferential treatment on any potential bidder or bidders by such action.

Return to Guideline

The SAI should examine whether the audited body identified the contractual issues that were likely to arise during the procurement and drew up a draft contract, setting out initial proposals on each issue.

Figure 7 lists the main contractual issues that one would expect a procuring organisation to address early on. The SAI should examine whether the audited body addressed these issues sufficiently early on and what legal advice it took when doing this. The SAI will also need to form a view as to whether the source of this advice had sufficient expertise to give good quality advice on the issues concerned. Guidance on the likely acceptability of their contractual proposals to the private sector should have been available to the audited body from its initial contacts with the market on the likely allocation of risk (Guideline 12).

In deciding on the opening position on the above contractual issues, a procuring organisation can save effort and expense by drawing, where relevant, on previous experience with similar public/private finance and concessions projects or on standard terms and conditions for such deals issued by any relevant central body. There is however a risk that in doing so that they may include contract terms that are not relevant to this particular project. The audited body should therefore have considered carefully whether it should replicate the provisions of past or standard contracts in their entirety, given possible changes in circumstances.

Return to Guideline

The SAI should identify whether the audited body prepared a tendering strategy, covering the number of tender rounds to be held, the number of bids to be invited at each tender stage, the body’s approach to communicating with bidders, and a realistic timetable for the tender process.

The SAI should check that, as part of its planning of the procurement, the audited body drew up a tender strategy before going out to tender. Since in many cases there will need to be several rounds of bidding, the body will need to have considered how many rounds it intended to hold and the number of firms it proposed to invite to tender at each round, and the overall tender timetable in which these tender rounds were to be held. Although it is good practice to map out such matters in advance, the body should be sufficiently flexible to vary the strategy in the light of changing circumstances, and possibly at short notice, if it believes that better value for the taxpayer will result.

In drawing up its tender strategy the audited body should have been sensitive to the time and costs that bidders incur in preparing bids and accordingly limited the number of rounds to the minimum consistent with a successful competition. Because of the substantial costs involved it should similarly have limited the number of bidders expected to submit full bids. The tender timetable should also be sufficiently brisk to maintain competitive tension, but long enough to permit the important contractual issues to be settled before the selection of the winning bidder.

As part of its tender strategy the audited body should have considered its plans for communicating with bidders. Such a communications strategy should cover how reasonably probable and competent bidders are to be made aware of the project, and what information will be released to them. As regards the possible release of the tender strategy itself to bidders with its details on the planned number of tender stages and bidders, there is much to be said for letting bidders know what is likely to be involved. This will enable them to make informed decisions on their own bidding strategy. Releasing such information however could cause problems at a later date should the procuring organisation decide to vary its plans. The SAI should therefore check whether the audited body took steps not to restrict unduly its freedom of action in this, for example by making it clear to bidders that it reserved the right to vary the tender strategy if necessary. Changes in tender strategy should be communicated to bidders in an even handed way.

In preparing the tender strategy the audited body should have considered how to structure the tendering plans in such a way as to allow early discussions with bidders. Such discussions are necessary if the public sector is to benefit from private sector innovation in the shaping of its requirements and the proposed solutions to these. Not allowing scope for such discussions may result in the audited body having to request further, unplanned, bids from tenderers in order to secure the services required. In the European Union, for example, allowing for such discussions will mean that the body should carefully consider using the "negotiated" or "concession" procedures, rather than the "restricted" procedure, because the former provide greater scope for negotiations with bidders earlier in the process.

Finally, in cases where an audited body is letting more than one contract to a common timetable, the SAI will need to examine whether the body considered at an early stage offering more than one contract to a bidder. Awarding contracts to the same bidder in this way may result in lower prices as the successful firm may pass on savings arising from economies of scale. The audited body will however need to have checked that its plans complied with any relevant procurement regulations and that potential bidders had the capacity to handle more than one contract simultaneously. Alternatively the audited body may have taken a policy decision to award its contracts to different bidders in order to strengthen the market for public/private finance and concessions deals if for example initial soundings of the market had shown that it was weak in the particular sector concerned.

Return to Guideline

The SAI should identify whether the audited body prepared a credible project timetable which identified milestones against which progress could be measured, and points within the process at which the body was to review the project’s continued viability.

None

Return to Guideline

The SAI should examine whether the audited body assessed the project against any publicly funded alternative, and review the adequacy of this public sector comparator.

The audited body will need to establish the public sector comparator as part of the planning of the procurement if it is to be used later as a benchmark against which to judge bids received. The outline business case prepared earlier in the process (Guideline 14) will have set out the key basis for comparison. The body should then revise the figures in this as the procurement progresses, to reflect more up to date information on public sector costs, and if necessary revisit the assumptions underlying the comparator. As much of the comparator as possible should be completed before the receipt of best and final offers from bidders if it is to help in the assessment of bids.

When reviewing the calculation of the public sector comparator the SAI should focus in particular on the assumptions underlying the comparator and the variables which will have the most significant impact on the total cost of the publicly funded alternative. Variables likely to be important, and therefore worthy of closer examination, include the completeness and reasonableness of the valuation of the possible financial impact of the various risks involved (Guideline 12), the discount rates used to calculate the net present values, and the forecasts of the volume of services required. It may be necessary when examining these assumptions and variables to commission the help of external consultants with relevant technical or financial expertise.

When examining the publicly funded alternative the SAI will need to consider the amount of detail required in this comparator. The level of detail required will depend on the impact that the comparator is likely to have on the assessment of value for money. For example, where private sector provision clearly offers much better value for money, there is less need for a precise calculation of the comparator’s value.

Return to Guideline

The SAI should examine whether the audited body succeeded in creating a good tender list.

The SAI should first examine what action the audited body took to stimulate the market. The previous investigation of the market (Guidelines 7 and 16) should have identified the likely interest in the project and what, if any, stimulation might be necessary. If some stimulation of the market were necessary, this should have been done before formal procurement started in order to maximise the number of good quality bids submitted. Stimulation can involve direct mailing of potential bidders, advertisements in the trade press, contacts with trade associations, and the holding of bidders’ conferences.

The SAI should also check how the body advertised the project. It should pay particular attention to the wording of any advert of formal announcement of the project since too specific a statement of requirements in such notices can deter potential bidders and restrict the opportunity for innovation by the bidders during the competition. An examination of the guidance on expected quality and technical standards provided to bidders will also be necessary. Clear guidance will aid bidders in the compilation of their bids. If however this guidance is too restrictive, innovation could be stifled

A competition will only have been effective if it included good quality firms. The SAI will thus need to examine how the audited body selected the list of firms invited to tender. In doing this, the body should have focused on firms’ technical, financial and managerial capabilities. The body should have been concerned with not only whether a firm was capable of providing the required services should they be awarded the contract but also whether, prior to the award of the contract, they are likely to remain in the competition and not withdraw at some point. Assessment of a firm’s management of its workflow will be particularly important. Such an assessment should cover not only work on other contracts that the firm will have during the construction or development phase of this contract but also the other contracts that the firm is bidding for at the same time as tendering for this contract. Because of the complexities and costs of preparing a public/private finance and concessions bid bidders have sometimes withdrawn from one competition to concentrate on another which they consider they have more chance of winning.

Because of the size and complexity of many public/private finance and concessions deals, firms often come together as consortia to compete for this work. Where a consortium expresses an interest in their proposed deal, the audited body’s assessment of capabilities should cover all members of the consortium as often these possess different skills and expertise. For example, on accommodation projects it is common to find construction and facilities management firms in a consortium. Once a consortium has been included on the tender list the audited body should continue to monitor the composition of this as it can often change with members leaving and new ones joining. Where there are changes of membership the audited body should have satisfied themselves again that the consortium still possesses the capabilities needed to deliver the required services.

A consortium may decide that it does not need to possess itself all the skills and expertise that are needed to deliver the required service but will instead sub-contract these in when necessary. If the audited body is content with this approach, it will still need to assess whether the sub-contractor, when eventually chosen, possesses the required capabilities to ensure service delivery. The body should continue monitoring the use of sub-contractors which are not members of the consortium during the asset construction / development stage. The project’s successful implementation could be jeopardised by a change of sub-contractor in this late stage

The SAI should also identify what help from its external advisers the audited body received when drawing up the tender list. These advisers are likely to have much more knowledge of the market place than the audited body on matters such as the possible existence of cartels between bidding firms.

Finally, the best check on whether the tender list did indeed provide sufficient competition is to verify the number of bids actually received, compare this with the number that the body planned to receive, and seek explanations for any difference between the two. If the number of actual bids was less than expected, it will be useful to seek the opinions of bidders or potential bidders as to why this was the case. Another indication of the strength of competition is the quality of the bids received, as evidenced for example by the audited body’s own evaluation of bids received.

Return to Guideline

The SAI should consider whether the audited body set out a clear specification of the requirements.

In determining whether the specification of requirements clearly stated the audited body’s real needs, the SAI will need to examine how the specification was developed. The body should have involved the users in the identification of `requirements but controlled their input to ensure that users’ expectations were realistic and matched the organisation’s real needs. Discussion of requirements with bidders will also be useful to identify whether these requirements were practicable and whether there was scope for improvement.

This consultation should have resulted initially in the production of an outline specification of requirements. These documents are often highly complex and the SAI may well need external advice in understanding them. The SAI should verify that the contents of this specification were consistent with the project deliverables identified when the objectives for the project were set (Guideline 6) and were themselves sufficiently output-based and covered the level of performance required of the contractor when eventually awarded the contract. In deciding whether the specification was sufficiently clear, it will also be necessary to seek the views of all the firms which bid for the contract, and perhaps also any who decided to withdraw from the bidding.

Return to Guideline

The SAI should identify whether, through the establishment of a competitive final shortlist, the audited body succeeded in maintaining competitive tension to contract award, and managed the negotiations with the preferred bidder well.

A good way for a procuring body to achieve competitive tension is to maintain time pressure throughout the procurement. Such pressure can be maintained by notifying the procurement timetable to bidders so that all parties are aware of the target dates that need to be met, and also providing bidders with model frameworks for contracts and other guidance on the audited body's requirements so that time is saved. The SAI should therefore check the information the audited body released to bidders on its timetabling requirements.

One indicator of the successful maintenance of competitive tension is the creation of a competitive shortlist of firms invited to submit final bids. The SAI should therefore verify the number of final bids actually received, compare this with the number that the audited body planned to receive and seek explanations for any difference between the two. If the number of final bids was less than expected, it will be helpful to seek bidders’ opinions as to why this was the case. The views of the bidders on the procurement will also provide evidence as to whether competition was effective.

Given the lack of competitive pressure during the negotiations with the preferred bidder and the changes to the deal’s terms often experienced in this stage, the audited body should have minimised the scope for negotiation by keeping this period of exclusive negotiations short and by achieving as much improvement to the terms of the bid as possible before the preferred bidder are informed of their selection. The SAI should therefore be concerned if these exclusive negotiations took many months and/or there were changes during this period to the proposed price, scope of services, allocation of risk, or contractual terms.

Return to Guideline

The SAI should identify whether during the procurement the audited body regularly reassessed that the project continued to offer value for money.

The audited body will find it useful if it continued to assess the deal’s value for money periodically once the contract has been awarded. It will then be better informed as to whether or not to exercise any right it may have negotiated in the contract to exit it, regardless of whether the contractor has defaulted (Guideline 12).

Return to Guideline

The SAI should examine whether the audited body set realistic budgets for all project costs, including internal and external resources.

None.

Return to Guideline

The SAI should check that the audited body appointed good quality external advisers after competition.

None.

Return to Guideline

The SAI should review how the audited body monitored and managed its project costs, including internal and external resources.

None.

Return to Guideline

SECTION 4: CONTRACT

The SAI should examine whether the range of solutions proposed by bidders included design variants.

None.

Return to Guideline

The SAI should review whether bidders put forward operational variants in their proposals.

One of the government’s wider policy objectives (Guideline 9) may be the protection of the rights of those public sector staff transferring to the private sector under any public/private finance and concessions deal. If this is the case, the SAI will need to check, as part of its review of bidders’ operational proposals, whether the audited body, when evaluating bids, assessed the proposed arrangements for employees’ pension and employment rights and union recognition.

Return to Guideline

The SAI should investigate whether bidders’ proposals included different ways of financing the project.

In assessing the value for money offered by the financing element of the winning bidder’s proposals, the SAI will need to examine how the audited body assured itself that the proposed rates of return were reasonable, for example by the use of benchmarking. The SAI may need to commission a financial consultant to undertake its own comparison of the financing of the deal with that in other deals.

The SAI will also need to identify whether the financing relied to any extent on the unusual treatment of taxation. A deal is unlikely to offer value for money to the taxpayer if its viability is dependent on dubious practices which reduce the tax intake. The audited body’s evaluation of the financing proposals should have covered this issue. However if necessary the SAI may need to seek reassurance on the treatment of taxation from its own financial consultant.

Return to Guideline

The SAI should examine whether bidders suggested amendments to the requirements identified in the specification by the audited body.

None.

Return to Guideline

The SAI should identify whether bidders submitted different proposals for the allocation of risk between themselves and the audited body.

The audited body’s earlier soundings of the market (Guideline 12) should have enabled it to seek bids on the basis of a risk allocation which would achieve value for money and be acceptable to bidders. Lengthy negotiations on contract terms after the appointment of a preferred bidder may however be a sign either that the information on risk allocation previously requested from bidders was lacking or that the audited body had initially proposed a transfer of risk which was unacceptable to the private sector, despite its prior market soundings.

Return to Guideline

The SAI should identify whether the audited body carried out an assessment of the financial aspects of bids.

In order to identify the financial ranking of bids the audited body should have converted bidders’ proposed annual contract payments into a single net present value figure, using an appropriate discount rate. Because these net present values form a key part of the evaluation of alternative bids the SAI should satisfy themselves that the calculations were correctly carried out. Common areas of error are in the treatment of future inflation and the value of assets existing at the end of the contract (the "terminal" or "residual" value). The SAI should therefore check that the audited body discounted correctly any allowance for this built into the cashflows.

The audited body should not have confined the financial evaluation to an assessment of the overall price of the bids received. In particular the SAI should identify whether the audited body and its advisers reviewed the financial models prepared by bidders in support of their bids in order to gain assurance that these models accurately reflected bidders’ proposals and that the bids could be financed. Failure to spot potential financing problems prior to the selection of preferred bidder could result in higher financing charges being passed on to the public sector if the preferred bidder proves unable subsequently to raise finance on the terms they originally proposed.

The SAI will also benefit from carrying out its own review of the successful bidder’s financial model. Such a review will, for example, help in their understanding of the bidder’s approach to key areas of the deal, such as the identification of opportunities for third party revenues and their assumptions about the disposal of any surplus assets. It will also help in the assessment of a deal’s value for money, especially where there has been little competition in the procurement.

Return to Guideline

The SAI should examine how the audited body assessed the allocation of risk proposed by each bidder and its effect on the proposed contract price.

When evaluating the allocation of risk, the SAI should pay close attention to the payment regime as the two are closely linked. For example, the level of risk transfer to the private sector will be reduced if there are only small variations to the service charge for poor service provision or service unavailability. The public sector will also have retained a higher level of risk if the contract allows for many circumstances in which the service charge can be legitimately increased.

Return to Guideline

The SAI should check the audited body’s assessment of the financial stability of bidders and members of bidding consortia.

None.

Return to Guideline

The SAI should assess the audited body’s evaluation of the quality of service proposed by each bidder.

None.

Return to Guideline

The SAI should review how the audited body assessed the results of the financial and non-financial evaluation so as to select the bid offering best value.

As the final evaluation of bids is a critical phase in the procurement exercise, the audited body should have clearly documented its assessment of each bid and its reasons for choosing the preferred bidder. In checking whether the audited body evaluated each bid consistently the SAI should obtain any internal guidelines which were issued for the evaluation of bids. These guidelines may have established the weightings to be applied to the evaluation criteria, minimum thresholds for quality, and maximum affordability limits. The guidelines may also have indicated how conflicts between different evaluation criteria such as price and quality would be resolved. As mentioned above, in certain circumstances it is acceptable for the audited body to exercise judgement in selecting their preferred bidder in a way that would give a different result than would have been reached simply by following the evaluation guidelines. An example would be where the weightings applied to the various evaluation factors did not produce results which reflected the audited body’s current priorities.

A key consideration in the bid evaluation will be cost (Guideline 33). However this is not necessarily the overriding consideration. The aim of any bid evaluation should be to identify the bid offering best value for money. Value for money is the best combination of cost over the life of the project and quality which meets an organisation’s requirements. There will therefore be situations where value for money and other objectives may be better achieved by the selection of a bidder whose proposed price is not the lowest - for example, where non-financial factors, such as the quality of service to users, were very important or where there were doubts about the deliverability of a low price bid. In the latter case the SAI should examine whether the audited body assessed that bidders planned to devote sufficient resources to the project to enable it to be delivered to a satisfactory standard and on time at the low price quoted.

Other factors which the audited body should have taken into account in its final bid evaluation include the bidder’s track record (taking into account any problems which have arisen on other similar projects previously undertaken by the bidder) (Guideline 7), design features (which should have been subject to a separate technical evaluation - Guideline 28), operational features (Guideline 36) and the impact of risk (Guideline 34).

Return to Guideline

The SAI should review how the audited body minimised changes to the terms of the deal during the final negotiations with the winning bidder.

When managing this phase of the procurement, the audited body should have attempted to limit the scope for change, for example by making it clear to the bidder at the start of the negotiations that it was seeking to finalise the contract within a short period of time and that it was prepared to walk away from the deal if it no longer provided value for money. It could also have asked one of the unsuccessful bidders to remain involved with the project until the contract had been let, although this would only have been reasonable if it had asked the unsuccessful bidder to remain involved in this way for only a short period. In practice, longer than two months is unlikely to be acceptable to a reserve bidder.

Where changes were agreed, the audited body should have assessed the impact on the value for money of the deal of each of these. It should have been wary of agreeing to changes which put the winning bidder below the next best bidder, and avoided any changes which rendered the deal poor value for money as compared to an alternative option.

In assessing the audited body’s actual performance during these negotiations the SAI will need to identify the extent of changes agreed to. Changes may have occurred in a number of areas but one of the most important of these is the financial aspect of the deal. Provided there were no changes to the service specification, the tender price should not normally have changed after the preferred bidder’s appointment. An acceptable exception is for the effect of interest rate movements to be factored into the tender price up to the date of contract letting. The SAI should seek full explanations for any price changes which occurred after the preferred bidder had been appointed and obtain evidence that the audited body assessed the value for money implications of the price variations.

Risk allocation is another area where the preferred bidder or their financiers might have sought changes in the closing stages. The SAI will be able to identify such changes by comparing the signed contract with the draft version current at the selection of the preferred bidder. If changes have occurred, there will be implications for the deal’s value for money because, even if the contract price remained unchanged, there are likely to be additional costs to the audited body if it agreed to bear certain risks which it had previously sought to transfer to the contractors. As with price changes the SAI will need to understand the nature of any changes in risk allocation which occurred and to see evidence that the audited body correctly assessed the value for money implication of the changes.

The SAI should not rely on the audited body’s explanations as to how the risks were finally allocated between it and the preferred bidder. Instead the SAI should examine the contract to verify how the risks have been dealt with and to identify any additional contractual risks which the audited body may not have been aware of. In doing this the SAI may need to obtain input from a legal adviser with public/private finance and concessions experience.

Finally the SAI should assess the implications of any other significant changes to the project which occurred after the preferred bidder was appointed. If the audited body made any changes to the deliverables they required, the SAI should ensure that the body treated all bidders fairly and sought legal advice on this point, if necessary. Design changes or operational changes should have been kept to a minimum at this stage of the project as these matters should have been fully dealt with before the appointment of the preferred bidder. The audited body should have resisted any attempt by the preferred bidder to change the framework for sanctions and the SAI should check that all the main provisions for measuring performance and for dealing with shortfalls in performance quality were agreed and documented before the contract was let.

Return to Guideline

SECTION 5: VALUE FOR MONEY EVALUATION

The SAI should examine whether the deal meets the project’s original objectives, or the latest circumstances if these objectives changed during the procurement.

None.

Return to Guideline

The SAI should examined whether the audited body confirmed that the proposed deal offered best value for money, compared to reasonable alternatives, before awarding the contract.

The SAI should examine whether the audited body confirmed before it signed the contract not only that the preferred option was better than the other bids submitted but also that it represented better value for money than other alternative procurement methods and the option of walking away from the deal and not signing the contract. Experience shows that mistakes can easily be made when such comparisons are done by those not experienced in carrying them out. In some cases audited bodies have appointed external consultants to carry out the technical analysis or to review that performed by their own staff. Despite this, the SAI should still check the calculation of these comparisons thoroughly, using the guidance given in Guideline 20.

Experience also shows that it is possible to construct over-elaborate financial and economic models to support the appraisal of options. The SAI should therefore be aware that it is a waste of money for these models to be more complex than is required or for them to provide spurious precision. In many cases, there will be some uncertainty about the underlying assumptions which will limit the degree of precision it is sensible to seek from the financial or economic model. In such cases, it is good practice for the audited body to examine the sensitivity of the conclusions to variations in the key assumptions.

Return to Guideline

The SAI should review whether the contract’s provisions on matters such as performance measurement and supplier remuneration will be likely to ensure delivery of the service required.

More detailed guidance on the arrangements for contract monitoring, termination and handover are given in Guideline 12. Similarly Guideline 34 contains more information on reviewing the payment regime.

It is important that the signed contract reflects the deal that has been negotiated. The SAI will therefore need to check that that the contract clearly sets out the deal negotiated and contains no ambiguities or omissions. The SAI should also check that the audited body’s understanding of the risk allocated achieved is consistent with that actually contained in the contract.

On public/private finance and concessions deals, in addition to the main contract between the successful firm or consortium and the public sector, there are likely to be separate contracts (or sub-contracts) between the consortium (or firm) and separate service providers. Thus there are separate construction and facilities management contracts. The SAI will also need to examine whether these additional contracts are consistent in their allocation of risk with the main public/private finance and concessions contract and that the audited body checked that they did not transfer risk back to the public sector.

In reaching an opinion on the adequacy of the contract provisions in all the areas mentioned above, it is unlikely that the SAI will possess the necessary expertise themselves to review the contract but will need to enlist the help of legal consultants.

Return to Guideline

The SAI should identify whether there are sources for any capital funds required to implement the project and for the ongoing payments payable under the contract.

Given the importance of the deal’s affordability (Guideline 6), although the audited body should have been reviewing constantly the affordability of the proposals as the project progressed (Guideline 24), the SAI will need to identify whether the audited body made one last check, before it signed the contract, on the availability of any capital funds required and of the revenue funding needed to meet the payments under the contract once service delivery began. The SAI should also examine whether any problems with the project’s affordability had arisen since the award of the contract. If problems had occurred, the SAI should establish the reasons for these and assess whether they were reasonably foreseeable at the time of the contract’s award.

Return to Guideline

GLOSSARY

• Return to Intosai Working Group home page