Report

by the Comptroller
and Auditor General

Principles Paper: Managing provider failure

HC 89 Session 2015-16 21 July 2015

Our vision is to help the nation spend wisely.

Our public audit perspective helps Parliament hold government to account and improve public services.

The National Audit Office scrutinises public spending for Parliament and is independent of government. The Comptroller and Auditor General (C&AG), Sir Amyas Morse KCB, is an Officer of the House of Commons and leads the NAO, which employs some 810 people. The C&AG certifies the accounts of all government departments and many other public sector bodies. He has statutory authority to examine and report to Parliament on whether departments and the bodies they fund have used their resources efficiently, effectively, and with economy. Our studies evaluate the value for money of public spending, nationally and locally. Our recommendations and reports on good practice help government improve public services, and our work led to audited savings of £1.15 billion in 2014.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 21 July 2015

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

14 July 2015

This paper explores the principles departments should use to manage provider failure. We define a provider as any organisation that delivers services to or on behalf of government and failure as any event which requires an intervention above and beyond normal performance management.

© National Audit Office 2015

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.gsi.gov.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

10760 07/15 NAO

The National Audit Office study team consisted of:
Eleanor Murray and Henry Midgley, under the direction of Oliver Lodge.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Summary

About this paper

1 Many organisations provide public services, and sometimes these organisations fail. The failure of a provider is not necessarily something that should be avoided at all costs. It can be the necessary price of innovation or come from effective competition, keenly priced contracts and robust contract management. However, the failure of a provider can have serious consequences and departments must manage failure effectively, in order to ensure continuity of services, and to protect the interests of people who use them. Our experience of auditing government over many years suggests that the way failure of providers is considered and managed leaves room for improvement. This paper draws together common findings from our work, to assist departments in meeting this challenge.

2 Managing the failure of a provider can be difficult. Departments may at times need to move quickly to respond to changing external pressures or new ministerial priorities. However, departments will be best placed to minimise negative impacts if they have planned how they would respond to a failure. Planning should start with deciding how a public service will be provided – by a public sector body, through a contract, via a market or by another approach, which should be informed by how much risk of failure a department is willing to tolerate.

3 It can be very difficult to decide how much risk to transfer to providers, and how such risk transfers will work in reality, given that the department retains ultimate responsibility for providing a service. We have seen many instances where departments believed that in contracting out services they had transferred the risks to the provider through financial penalties. But when a contract fails the provider can walk away, and government cannot. Departments have to recognise this position and refine their judgements on these and many other issues as they proceed.

4 There can also be a difference between a department’s stated and revealed appetite for the failure of a provider. For example, market mechanisms and payment by results can seem significantly more politically attractive when they are introduced (and offer the potential for lower costs and improved outcomes through competition) than when they result in providers leaving the market. Failure regimes have often been established, but we have observed very few occasions when government allowed providers to actually fail. This change in risk appetite can be exacerbated by the 24 hour news cycle and the role of social media, which provide both an opportunity for departments to access unfiltered public feedback on a provider, and a challenge as failures unfold quickly and publicly, generating demands for instant action in complex situations.

5 In our experience, departments are very rarely able to specify at the beginning of a programme everything that they will require over its lifetime. There are practical steps departments can take to increase the flexibility they have to introduce changes, but these will come at a cost. In the past, we have seen departments jeopardise value for money by entering into long term but inflexible agreements, which need to be revisited part way through at significantly increased cost.

6 We have also observed that government has a poor record of learning from past failures. Departments face similar issues and often make similar mistakes in designing failure regimes. We have identified, over a number of reports, the need for government to take a more coordinated approach to learning about delivery models and this is especially true when thinking about this topic.

7 This principles paper draws on a review of our previous work, a wide ranging literature review and interviews with stakeholders across the private, public and third sectors. It sets out principles departments should consider when planning for and managing failure. We have included case studies to show how applying these principles can either assist departments in attaining ministerial objectives or, if they fail to apply them, lead to failing to achieve those objectives and reputational damage. Badly managed failure can discredit particular models of providing public services. It can also harm the users’ interests, either because they experience a break in services or anxiety about losing their service. Figure 1 sets out the structure of the rest of this paper.

Part One

Context

1.1 This part describes:

what provider failure is; and

why it is increasingly important.

What is a provider?

1.2 In this guide, a provider is any organisation that provides a service to or on behalf of government. This can include a contractor that provides an IT system for a government department, an arm’s-length body managing delivery and a hospital providing services to a local community. It includes providers from the private sector, charities and the public sector. Providers may work directly with a department or be subcontracted under another provider to provide part of a service (Figure 2). Although it is not the focus of this guide, government may on occasion also need to manage the failure of a provider that does not provide services to or on behalf of government. For example, government intervened to manage the nationalisation of Northern Rock, because of risk of instability in the financial system.1

1.3 We draw mainly in this paper on examples from our previous work on commissioning within departments. However, the lessons are relevant to all public sector commissioners, including local authorities, police and crime commissioners and the NHS.

What we mean by failure

1.4 In this paper, failure is any situation requiring an intervention above and beyond normal performance management. This may include the provider:

not providing a service for financial reasons (financial failure);

not governing itself according to public expectations (governance failure); or

not meeting an expected minimum performance standard (performance failure).

Departments will need to define the particular dimensions of provider failure in their sector. In many cases, departments will find it easy to define failure. In others departments will need to define the boundary between underperformance and failure. In doing so they will need to consider how far they will let a provider deal with a failure in performance through internal mechanisms and when they should step in.

1.5 Departments may also need to consider how to handle a provider exiting a market for reasons other than failure. This could happen if, as a commercial entity, the provider does not feel there are enough opportunities for profit.

A changing delivery landscape

1.6 Provider failure is a more important issue now because of the open public services agenda (Figure 3).2 Increasing the amount of innovation in public service delivery models has the potential to deliver better public services at lower costs, but it also increases the risks of failure.

1.7 Even with opening up public services, government is responsible for failure and for the system as a whole, and it has intervened in the activities of providers; for example, intervening in local schools in Birmingham following allegations arising from the ‘Trojan Horse’ letter. Local authorities are primarily accountable to a local electorate, however the Department for Communities and Local Government has intervened in instances where there were serious failures; for example the Department appointed commissioners to oversee certain functions in Tower Hamlets Council following irregularities in the controls over expenditure of public money. The Committee of Public Accounts expects accounting officers to discharge their responsibilities to Parliament for public services, even when services are contracted out.3

1.8 Since public services have been opened up, consolidation of the market has led to a number of providers developing large portfolios of work with several departments. Should one of these providers fail, the consequences would be experienced by a number of different departments and by many different users. Government as a whole has set up mechanisms to manage commercial providers at the centre through the Crown Commercial Service and the Crown Representatives.

1.9 These principles are intended to support departments in balancing their aim to transfer responsibility for operational delivery to providers while retaining their accountability to Parliament and users of public services.

1 Comptroller and Auditor General, HM Treasury, The Nationalisation of Northern Rock, Session 2008-09, HC 298, National Audit Office, March 2009.

2 Cabinet Office, Open public services white paper, Cm 8145, July, 2011.

3 HC Committee of Public Accounts, Accountability for public money, Twenty-eighth Report of Session 2010-11, HC 740, April 2011.

Part Two

Strategy and Planning

2.1 This part sets out what departments should think about when planning for failure:

Strategy: appetite for failure.

Planning:

selecting the delivery model; and

planning for the event of a failure.

Strategy: the appetite for failure

2.2 The department should set out its definition of failure in its plan (paragraph 1.4). It should then define its appetite for failure. The consequences of a provider failing will be much more serious in some circumstances than others. In some cases, government may not wish to tolerate any performance failure. In others, departments may be able to design contingency plans that allow for the failure of a provider without a significant impact on service users. We have identified some of the factors to consider when assessing the likely impact of a failure, and therefore how great a risk of failure to design into a delivery model (Figure 4).

2.3 Departments might choose to use different types of information to assess their level of exposure to failure. They could include: feedback from potential users; academic studies of potential users; provider forums; scenario testing; evaluations of previous programmes; or consultations. They may also need to consider the political implications of failure.

2.4 The appetite for failure may change as policy priorities or a delivery area evolves, and the factors affecting the impact of a failure may also change over time, as may a department’s view of them. In which case, departments may have to update plans for preventing and managing failure. When a department introduces a new policy, it should consider how this affects the sector concerned, including established providers (Figure 5), as well as considering how it will affect its other policy priorities.

Planning

Selecting a delivery model

2.5 Different service delivery models imply different appetites for failure. It is therefore important for the department to choose a delivery model consistent with their appetite for failure. Some delivery models, such as markets or payment by results, involve providers being incentivised by a risk of financial failure. As we stated in our report on social care markets, “the risk of going out of business can act as a powerful incentive on providers to continue to perform well.”4 But departments will need to be conscious of the extent to which there is political will to allow a provider to fail as this can often be a key consideration when a provider failure occurs. In other models, where the department closely manages delivery, providers have less latitude to fail. However, this may also mean the provider has less room to innovate or that government is paying a premium for the service because the price it is paying includes a ‘risk premium’, but it is not actually transferring very much risk.

2.6 We have seen a number of instances where failure regimes have been set up but on relatively few occasions has government allowed providers to actually fail. Based on our experience, departments will need to carefully think through how they would respond to a failure before assuming they have a high appetite for it. The clearer and more accurate departments can be about their appetite for failure before starting a programme, the more able they will be to select the correct delivery model. If a department has a high tolerance for failure, it can focus its efforts on managing the system or market as a whole, and may not need to monitor individual providers.

2.7 Alternatively, if a department has a low appetite for failure (the failure of any individual provider would have serious consequences), then it is important to monitor individual providers. There is a risk that departments might initially think they have a high appetite for failure and so manage the system as a whole, but when an individual provider looks like it will fail, they rethink (and lower) their risk appetite (Figure 6).

2.8 Different delivery models have different advantages and constraints, and to get the best from them, it is important that departments have people with the skills to manage them effectively. We have found in the past that contract management has been under-resourced in central government.5 Departments may need to consider whether the model remains the most cost-effective, given the costs of monitoring and managing the market effectively.

2.9 Delivery models vary in their flexibility, which will affect options for managing a provider failure. In our experience, departments often need to manage changing priorities over the life of a programme. There are steps departments can take to increase flexibility, for example inserting break clauses into contracts. Whilst this may increase costs, it needs to be balanced against the fact that inflexible arrangements may lead to poor value for money when the department makes changes. For example, procurement rules may limit a department’s flexibility to change approach during a procurement or to adjust their thinking after they have procured a service. Contract terms might constrain a department’s ability to gain control of assets in an insolvency.

2.10 We have reported on procurement in government, and published guidance on good practice in contract management.6,7 Service users may have views on the selection of a delivery model – for example, a delivery model which requires a service user to research and select one provider from a number of choices may not be appropriate for all service users in all circumstances. We have guidance on the risks and opportunities involved in the use of markets and payment by results.8,9

Designing a delivery model

2.11 Having chosen a delivery model, the next challenge is to refine it to apply to the department’s particular circumstances and objectives. For example by setting prices, deciding on the amount of risk to transfer, setting a timetable or establishing what performance to expect, and how to monitor it.

2.12 Figure 7 sets out some relevant pitfalls which can result in issues for the commissioner of a service with their providers, as our case study in Figure 8 illustrates. As the case in Figure 8 illustrates, attempts to transfer too much risk can be self defeating as they may contribute to the failure of a provider, at which point, the risk reverts to the department.

2.13 In our experience, departments can find it difficult to fully transfer risk. Commercial entities in particular may limit their liability through the creation of special purpose vehicles which shield the main entity from excessive losses. Departments will always hold the ultimate risk because they are responsible for service delivery to the public and are usually the largest party in the agreement.

Planning: Plan for the event of a provider failure

2.14 Failure to recognise and plan for the possibility of provider failure can leave departments exposed and give them limited scope to respond (Figure 9).

2.15 Departments will be better able to deal with a provider failure if they develop contingency plans. The level of detail of the planning should be proportionate to the likelihood and impact of a failure (and the department’s confidence that they understand the likelihood and impact of failure). As far as possible, plans should be realistic, consistent with the chosen delivery model, focused on departmental priorities and take account of past experience (both within the department and across government).

2.16 Failure can happen very suddenly and quickly and may require resources which are not typically used in normal contractual or other forms of management to be deployed quickly. Departments will be in a far stronger position if they identify these resources in their plans, (for example legal advice, alternative providers, communications experts), and how they can be made available quickly. Where there are very specialised services, departments may be competing with others for them in a crisis.

2.17 In the event of a failure, there is likely to be a lot of uncertainty among a range of stakeholders. The department is responsible for ensuring that the service continues and stakeholders will look to the departments in the event of a provider failure. It will be easier to manage this if the department has a plan for managing communication with stakeholders, particularly the users of the service concerned, and are proactive in communicating how they will manage the situation (Figure 10).

2.18 In some circumstances, all providers might be at risk of failure at the same time. This may happen for example when an external event (such as a changing economic situation) affects the service’s outcomes, leading to performance or financial failure for providers. Where this is the case, departments should consider how to prevent it or address it.

2.19 Due diligence work can reduce the risk of failure on beginning a relationship with a provider, but it also imposes costs and so may limit the available pool of providers. Some departments now require all providers to go through an external governance audit before they commence providing services.10 Such checks can mitigate some risks but may also affect the number of providers available. It is very difficult to design a process that excludes all undesirable potential providers while retaining all good ones. For example, small and medium-sized enterprises may be excluded by due diligence requirements. Departments may also consider how dynamic their measures of risk for each provider are and how well they reflect the knowledge held across government about the provider. Departments will need to come to a judgement based on their appetite for risk, cost and innovation.

2.20 Government commissions from a number of large providers who hold many contracts with a variety of different departments. Consolidation of the market in this way may mean that an individual failure has consequences for many government departments and needs to be dealt with at a strategic level. Departments and government as a whole may choose to assess these providers on a portfolio basis, with a coordinated response to any potential failure. Such a response might involve prioritising where to put effort, ensuring that each commissioner is aware of their place in the overall strategy for the provider’s failure and ensuring that resources are deployed effectively.

4 Comptroller and Auditor General, Oversight of user choice and provider competition in care markets, Session 2010–2012, HC 1458, National Audit Office, September 2011, p. 29.

5 Comptroller and Auditor General, Transforming government’s contract management, Session 2014-15, HC 269, National Audit Office, September 2014, p. 7.

6 Comptroller and Auditor General, Improving government procurement, Session 2012-13, HC 996, National Audit Office, February 2013.

7 Office for government commerce and National Audit Office, Good Practice Contract Management Framework, National Audit Office, December 2008.

8 National Audit Office, Delivering public services through markets; principles for achieving value for money, National Audit Office, 2012.

9 Comptroller and Auditor General, Outcome-based payment schemes: government’s use of payment by results, Session 2015-16, HC 86, National Audit Office, June 2015.

10 Comptroller and Auditor General, Transforming government’s contract management, Session 2014-15, HC 269, National Audit Office, September 2014.

Part Three

Measurement and Implementation

3.1 This part sets out what departments might do when a programme begins.
It covers:

measurement: introducing appropriate oversight;

Implementation:

Clarity with suppliers during normal operation of the programme.

Balancing the need to be consistent with the need to respond to changing circumstances.

Measurement: Put in place appropriate oversight arrangements

Accounting officer responsibilities

3.2 Irrespective of whether and how far services are devolved, accounting officers must satisfy themselves that they are securing good value for money from departmental spending.11 In 2011, government committed to publishing accountability system statements. These set out the system for each department’s decentralised funding so accounting officers can give assurances to Parliament. Government said that departments should set out their arrangements for dealing with failure or underperformance in these statements.12

3.3 Departments bear ultimate responsibility for how providers use public money. Departments must assure Parliament of the propriety and regularity with which money is spent, whether it is spent directly by a department or by a provider on their behalf. These terms are defined in guidance from HM Treasury, Managing public money (Figure 11).

Monitoring the provider

3.4 After selecting a provision model and a plan for provider failure, departments need ways to measure whether providers are at risk of failing. These need to be proportionate to departments’ appetite for failure, and appropriate for their model. For example, departments may need to monitor more where the impact of a failure would be larger and its likelihood higher. They might balance this imperative with the costs of monitoring, to themselves and to providers.

3.5 There are a variety of different mechanisms departments can use to monitor risks and identify early warning signs of failure. These include requesting regular performance returns, inspection regimes, feedback from users or staff, regular discussions with providers, and many other methods. An effective monitoring regime is likely to include a mix of proactive monitoring (where the department asks for information) and reactive monitoring (where the department creates channels for people to raise concerns). Figure 12 sets out some of their advantages and disadvantages.

3.6 Reactive monitoring can allow departments to discover ‘unknown unknowns’. Sometimes whistleblowers have uncovered important abuses, so it is essential that staff and service users can communicate through these channels of communication (Figure 13). Monitoring systems should, as far as possible, cover the full range of activities for which departments must account to Parliament. Even where departments have set up regulators to regulate providers, they retain ultimate responsibility for the system (immediate responsibility for the provider rests with the governance systems of that provider). Regulators do not always identify provider failure so it is worth considering other types of intelligence about performance (Figure 13).

3.7 It may be necessary to adjust systems to respond to changes in risk, accepting that this may result in additional cost. For example, departments may choose to do the following:

adopt dynamic measures which change as the service and the delivery environment evolves:

recognise that performance may rise over time so expected minimum service levels from a provider may change during their relationship with the department.

recognise the department’s appetite for failure may change during a long contract or service agreement, with new ministers and senior officials having a different appetite for risk and therefore, where possible, build flexibility into agreements.

recognise that after a provider fails, the department may need to consider whether that changes incentives for all providers in its delivery chain (see Part Four). This may alter the department’s type of monitoring.

raise the level of monitoring to respond to known changes in risk – for example during economic distress or periods of rising service demand and when capacity is under strain.

change the monitoring system to reflect the expertise within the department. If departmental expertise erodes over time, the department may need to hire or consult with experts in the field, or ask for more data about performance and benchmark the data it has more effectively.

3.8 Departments will be better placed to spot emerging risks if they are able to monitor prospectively as well as retrospectively. For example, departments should monitor the projected financial performance of providers – examining income and cash flow projections – as well as the past performance of providers through accounts. Departments may choose to test their providers’ forecasts for unexpected events.

3.9 A department’s ability to monitor a provider and their provision of services depends on the rights it has to get information from the provider. In establishing rights to information, departments should ensure that Parliament and its auditors can follow the public pound. We have reported on open book accounting in relationships with the private sector and how to monitor for financial impropriety elsewhere.13,14 Every major government contract needs a supply chain assurance strategy.

3.10 Effective monitoring systems may benefit providers. Lenders, for example, may take assurance from the department’s monitoring systems in providing cheaper credit to providers (Figure 5).

Implementation

Clarity with providers and users about what constitutes failure and what would happen as a consequence of it

3.11 It will be easier to identify and manage failure if departments can agree with providers and users about what constitutes failure and what happens in the event of a failure. However, we recognise that this is not straightforward to do, as what is considered to be a failure is likely to change over time, as political and public priorities shift: for example delivering a service successfully may affect expectations of that service, leading to a higher level of provision being classified as failing. Departments should be as honest as possible with providers about their changing service expectations, particularly in the context of a long-term relationship, so that providers are better able to meet those expectations.

3.12 A clear and transparent approach to sharing information about failure can help providers to deal with failure before it happens. Departments may choose to:

share their assessment with a provider who they believe is close to failing;

share knowledge with a provider who is close to failing about options for successfully turning around their business, including inviting another (more successful) provider to assist or mentor them;

appoint new senior staff to a provider’s board to assist them; or

discuss emerging risks with a provider so they are better able to manage them if they materialise.

Departments may consider whether these approaches are consistent with their view of the way that failure acts as an incentive in their system. In some delivery models, such as markets, such an approach may lead to providers believing the department will not let them fail, and thus not responding to the incentive.

3.13 Different approaches will be suitable for different types of providers and departments should consider the implications of any help they give to an unsuccessful provider, in the same way as they would consider the impact of rescuing a failing provider. They should also have regard to relevant regulations such as procurement law and state aid rules. However, as we have stated in the context of commercial providers, good provider management requires effective relationship management.15,16

3.14 This is particularly important when several providers provide a service together. In this situation, departments should be particularly clear about where accountability lies for each part of the service. Departments should remain alert to any confusion about accountability and address it speedily (Figure 14).

3.15 Departments may choose to discuss with providers:

what would lead to the department suspending the service;

what they want to safeguard;

what legal responsibilities they have should a provider fail; and

who will perform particular roles in the event of a failure, both inside the department and in the provider:

who will take decisions; and

who will be the point of contact for the provider, the department and for users during the process of dealing with the failure.

Responding to failure

3.16 When a failure occurs, a department will need to respond, balancing its plan, any relevant regulations (such as procurement law) and the requirements of the moment. Sometimes that response will just be monitoring service continuity for users and learning lessons, for example in a market where users can find alternative provision themselves. In others the department will need to respond more substantially to safeguard the taxpayer interest and users of the service – for example, a department might need to take over the provider and run services itself, replace senior management in the provider or fund the provider until it can source alternative provision. As we stated in our report on lessons from transport programmes, departments “must be realistic and responsive to unexpected events” in how they implement plans.17

3.17 Departments may need to respond quickly to such events, to:

reassure the public about service continuity and achieve their own objectives (Figure 15); and

safeguard their legal position, for example by establishing a channel for discussions with the administrators of a bankrupt company.

3.18 Departments may discover that circumstances have changed since they made their plans. This may include:

increasingly vulnerable users – for example because of an economic recession or changes in their characteristics;

increasing numbers of people using the service; and

the service arriving at a time of maximum stress: for example, parts of the health service are particularly busy over winter, which may again affect service continuity arrangements.

3.19 The incentives operating on providers, whether in the public or private sector, may also have changed. For example:

Local accountability mechanisms (through police and crime commissioner elections or local council elections) can set different priorities to those of national government. These may limit local bodies’ capacity, capability or incentives to deliver a service in line with departmental objectives.

If a provider has failed there may be weaker competition within the market, which may lead to falls in the quality of provision and rising costs.

If providers are running at capacity already, then they may be less eager to take on further work. This may be a particular risk if a provider is regarded as having failed because it was unable to make money on its particular area of service delivery.

As private companies provide more services, departments need to become more alert to the way in which the ownership and general strategy of a private provider may affect its long-term financial sustainability. In addition, they need to be aware that acquisitions may transform a market: turning it from a competitive market into a potential monopoly with increased risks from failure. Both these situations can be seen in the case explained in Figure 16.

11 HC Committee of Public Accounts, Accountability for public money, Twenty-eighth Report of Session 2010-11, HC 740, April 2011.

12 Department for Communities and Local Government, Accountability: adapting to decentralisation, September 2011, pp. 15, 23–24.

13 Comptroller and Auditor General, Open-book accounting and supply-chain assurance, Session 2015-16, HC91-I, National Audit Office, July 2015.

14 HM Treasury and National Audit Office, Good practice in tackling external fraud, National Audit Office, June 2008.

15 Her Majesty’s Inspectorate of Constabulary and National Audit Office, Private sector partnering in the police service, July 2013, p. 45.

16 Comptroller and Auditor General, Transforming government’s contract management, Session 2014-15, HC 269, National Audit Office, September 2014, p. 35.

17 Comptroller and Auditor General, Lessons from major rail infrastructure programmes, Session 2014-15, HC 267, National Audit Office, October 2014 p. 8.

Part Four

Evaluation and feedback

4.1 This part covers:

evaluation:

consider how the department’s response has altered incentives
in the market;

after a failure, it is important to reconsider the health of other providers; and

feedback: share lessons about failure within and outside the department.

Evaluation

Consider, how the department’s response has altered incentives in the market

4.2 Providers’ behaviour is likely be affected by changes in incentives created by the department’s response to a failure. Understanding changes in provider behaviour can help departments to redesign their own approach to the sector. Regulators in the financial sector design rescue arrangements with regard to their potential effects on ‘moral hazard’ (the risk that rescuing one provider creates an incentive for other providers to take risks – for which they keep the rewards – in the expectation that if they get into difficulty they will also be rescued). Such effects are also present in the public sector. Providers may presume that a department will continue to finance them should they be unable to meet an agreed specification (whether contractual or other). In the railway sector, the Department for Transport calculated that rescuing a franchisee would lead to greater costs for the public sector, as all the other private sector franchisees asked for support too (Figure 17).

4.3 The Office of Fair Trading, a predecessor to the Competition and Markets Authority, has been explicit: “if there is a suspicion that authorities will prop up weak firms to ensure service continuity, such firms may be tempted to continue in the market, thereby risking eventual financial or operational failure and disruptive exit”.18 It may be, after rescuing a failed provider, that if a department uses a delivery model where they expect providers to fail, then departments will need to consider how to adjust the delivery model or how to restore the credibility of their previous failure regime.

Wider work after a failure, consider the implications for the rest of the market

4.4 Once a provider has failed, the department may need to consider what the failure says about its system of service delivery. Sometimes the provider failure may reveal problems that affect all providers, and may signal that a department needs to re-evaluate its risk assessment for other providers. In others it may be a sign of a healthy market. Departments might consider in particular, the realism of their expectations of performance for providers and the costing of the provision. In our paper on Principles of Price Setting in Public Markets, we found that local authorities and departments were more successful when they set prices which sustained supply in the short and long term.19 We have also found issues in the past when similar programmes are launched together at the same time and providers struggle to obtain enough referrals.

4.5 If a department considers that the service is unsustainable, it has options. These include raising the price the department pays for the service, changing the definition of the goals of the service or abandoning the programme altogether (Figure 18 illustrates the last option). The department may consider the impact of any action upon its own priorities, the interests of users and the sustainability of a market for provision in that sector.

Feedback

Share lessons about failure within and outside the Department

4.6 Failure can be extremely expensive to the department and to users. It is therefore important that departments share what they have learned as widely as practicably possible, to avoid government paying several times to learn the same lesson. Equally, where departments successfully manage a provider failure it may be beneficial for other departments to reflect on whether there are lessons for their own contingency planning. This is particularly important for innovative projects. In our 2013 report on the Troubled Families programmes, we noted that more innovative projects require effective evaluation mechanisms during the project and after their completion so that learning is identified and not lost.20

4.7 Departments can also share lessons with providers, as often failures take place within the provider as well as the commissioning organisation. This may reduce the risk that government has to manage a similar provider failure in future.

18 Office of Fair Trading Orderly Exit: designing continuity regimes in public markets, December 2012, p. 8.

19 National Audit Office, Deciding prices in public services markets: principles for value for money, December 2013, pp. 24, 31.

20 Comptroller and Auditor General, Programmes to help families facing multiple challenges, Session 2013-14, HC 878, National Audit Office, December 2013, p. 8.

Appendix One

Guidance and useful resources

Part Two: Strategy and Planning

HM Treasury, The Green Book: appraisal and evaluation in central government, November 2014.

Office for Government Commerce and National Audit Office, Good practice contract management framework, National Audit Office, December 2008.

National Audit Office, Deciding prices in public services markets: principles for value for money, December 2013.

National Audit Office and Chartered Institute of Management Accounts, Delivering value and accountability: How finance can help, NAO and CIMA costing paper, October 2014.

Part Three: Measurement and Implementation

National Audit Office and HM Treasury, Good practice in tackling external fraud, National Audit Office, June 2008.

National Audit Office, Intelligent monitoring: An element of Financial Relationships with Third Sector Organisations, June 2009.

Part Four: Evaluation and Feedback

Cabinet Office and HM Treasury, The Magenta Book: Guidance for evaluation, April 2011.

Design and Production by NAO Communications
DP Ref: 10760-001