Report

by the Comptroller
and Auditor General

Cross-government

Projects leaving the Government Major Projects Portfolio

HC 1620 Session 2017–2019 19 October 2018

Our vision is to help the nation spend wisely.

Our public audit perspective helps Parliament hold government to account and improve public services.

The National Audit Office scrutinises public spending for Parliament and is independent of government. The Comptroller and Auditor General (C&AG), Sir Amyas Morse KCB, is an Officer of the House of Commons and leads the NAO. The C&AG certifies the accounts of all government departments and many other public sector bodies. He has statutory authority to examine and report to Parliament on whether departments and the bodies they fund, nationally and locally, have used their resources efficiently, effectively, and with economy. The C&AG does this through a range of outputs including value-for-money reports on matters of public interest; investigations to establish the underlying facts in circumstances where concerns have been raised by others or observed through our wider work; landscape reviews to aid transparency; and good-practice guides. Our work ensures that those responsible for the use of public money are held to account and helps government to improve public services, leading to audited savings of £741 million in 2017.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 18 October 2018

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

17 October 2018

This study considers whether there is evidence that projects that have left the Government Major Projects Portfolio have delivered their intended benefits; and whether accountability for and transparency of major project delivery has improved.

© National Audit Office 2018

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.gsi.gov.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

11709 10/18 NAO

The National Audit Office study team consisted of:
Damian Burns, Sandy Gordon, Matt Hemsley, Matt Phelps, Riaz Rahman, Helen Roberts, under the direction of Geraldine Barker.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Key facts

Summary

1 Government departments deliver many of their strategic priorities and objectives through programmes or projects. Major government projects are often large-scale, novel, delivered by multiple stakeholders and present a risk that many commercial organisations would not take on.1 In April 2011, the Major Projects Authority, now the Infrastructure and Projects Authority (the Authority), created the Government Major Projects Portfolio (the Portfolio) to improve the delivery of government’s biggest and riskiest projects by increasing transparency and providing independent assurance.2 As at September 2017, the Portfolio consisted of 133 projects with a planned total of around £420 billion of whole-life costs and more than £650 billion of benefits (Figure 1). Between April 2011 and September 2017, 430 projects had joined the Portfolio and of these, 302 had subsequently left.

2 In 2016, we reported on whether the establishment of the Authority and the Portfolio had improved government’s performance in delivering major projects. The turnover of projects, the limited data published and the lack of systematic monitoring of whether projects had realised benefits made it difficult to conclude on trends in performance across the Portfolio.3 The Committee of Public Accounts (the Committee) expressed concern: about the quality of data, including that on benefits and the apparent gap in responsibilities for reporting on the delivery of benefits; and that some projects leave the Portfolio without a review to ensure that they are on track to deliver their benefits.

3 The Committee identified that improving the delivery of benefits should be a priority, recommending that the Authority should do more to determine whether government projects were likely to achieve their stated benefits before they leave the Portfolio. Poor measurement of what projects achieve reduces accountability and transparency for government and Parliament, and makes it difficult to assess whether the costs of projects are justified. It also means that government is missing an opportunity to learn about what constitutes success.

4 This report examines:

why we examined projects leaving the Portfolio (Part One);

whether there is evidence that projects which have left the Portfolio have delivered their intended benefits (Part Two); and

whether accountability for, and transparency of, major project delivery has improved (Part Three).

We do not conclude on whether individual projects have provided value for money. We set out our audit approach in Appendix One and evidence base in Appendix Two.

Key findings

On delivery of benefits

5 For the 302 projects that have left the Portfolio since its inception in 2011, the Authority does not have complete data on the reasons why they had left and what they had delivered by the time of their departure. Before 2016 the processes for entering and leaving the Portfolio lacked clarity and were based on negotiations which were not always fully recorded. Transparency has improved since 2016, with the Authority reporting in 2018 that 20 out of 29 projects that left between October 2016 and September 2017 had delivered their original objectives. It is still not always clear, however, what has been delivered. This is in part because it may take years for a project to deliver its strategic outcomes. For example, the project to renovate and modernise the headquarters of the Foreign & Commonwealth Office left the Portfolio when the department had moved into the new building, but it was too early to assess other intended benefits, such as financial savings and a more sustainable environment (paragraphs 2.1 to 2.6 and 3.6 to 3.9).

6 Since the Portfolio’s inception projects have entered and left and not all have done so because they have reached key stages in the project lifecycle. We followed up 48 projects that had left the Portfolio; 35 of these had reached a significant delivery milestone when they left including IT projects that were being rolled out and initiatives where the department funded activity for a period of time, which was complete, as well as projects where construction was complete. The remaining 13 left because they had been cancelled, merged with or replaced by other projects, disaggregated or the Authority agreed that the project no longer needed to report to it because departmental assurance was sufficient and they were no longer high-risk (paragraphs 2.6 to 2.12 and Figure 5).

7 There is a varied picture as to whether projects have delivered successfully after they leave the Portfolio. Once projects leave the Portfolio, the Authority is no longer responsible for monitoring progress in delivering benefits, it is up to departments to provide this oversight. In Spring 2018, we followed up 48 projects which had left the portfolio and found that:

in 12 cases departments and other sources provided good evidence that the project had achieved its outcome. For example, the Super-Connected Cities Initiative was a £150 million fund to improve internet connectivity, help with small businesses’ broadband costs, Wi-Fi in public buildings and other capital projects. It initially covered 22 cities. The scope was widened to 50 cities and the initiative had spent £121 million to  2015-16 when it left the Portfolio;

four projects had been cancelled because it was clear that they would not achieve their objectives, or because there was a more cost-effective alternative or because government policy had changed. As we have said in other reports, cancellation of a programme is not necessarily a poor outcome, if circumstances or priorities have changed. It is still important to learn lessons from cancelled projects to support future decision-making;

six projects delivered less than intended because they were scaled back after problems with development or implementation;

for 22 projects it was not possible to say whether outcomes had been achieved. In some cases this was because they were still being rolled out, and it was too early to say; and

for four projects, it was unclear what had been delivered because departments had stopped monitoring them, due to either a change of policy or because the department had decided to deliver them in different ways, which resulted in project teams being disbanded and so departments were unable to answer our questions.

Taking these points together, it is not possible to say in aggregate to what extent projects have realised the intended benefits (paragraphs 2.13 to 2.18).

8 Evaluating projects too soon or determining that a project is a success because it has delivered short- or medium-term goals can be misleading. The Ministry of Justice’s Prison Unit Cost Programme’s objectives were to maximise savings from public sector prisons by reducing operating costs while supporting the safety, security and decency of public prisons. It cost £115 million and was expected to save £550 million. The programme broadly achieved the planned cost savings, staff reductions and prison closures by 2016. However, since 2017, staff numbers have increased to improve safety in prisons. This demonstrates that longer-term tracking is needed to show whether a project’s outcomes are sustainable (paragraph 2.15).

9 Measuring success against a revised scope may also present an inaccurate picture of the success of a project, as it is not making judgements against the business case that was first approved. The Mobile Infrastructure Project was a Department for Digital, Culture, Media & Sport programme to fill gaps in commercial mobile telephone coverage. It initially committed to building 575 mobile telephone masts to expand coverage to 60,000 premises, at an estimated cost of £150 million. However, following approval, it built 75 masts against a revised target of 40, reaching 7,199 premises at a cost of £36 million. The Authority’s exit review gave the project a green delivery confidence rating against the new baseline. The subsequent evaluation used only the revised baselines and reported that they had been achieved (paragraphs 2.17, 3.14 and Case study 2).

10 In some cases it was not clear what the intended benefits were. Seven of the projects we examined did not have a business case against which to measure benefits. For instance, the Household Energy Efficiency programme delivered against its target of improving energy efficiency in one million homes, but did not have targets or measurable goals for its wider objectives such as saving energy. This not only reduces transparency around benefits but is a missed opportunity to instil greater discipline to the management of government initiatives (paragraphs 2.14, 2.18 and Case study 1).

On accountability and transparency

11 Senior responsible owners and departmental accounting officers have clear accountabilities for projects while they are in progress, but this accountability risks dilution once a project passes to business-as-usual. Since early 2018, accounting officers report to Parliament on the results of their assessments of projects’ regularity, propriety, value for money and feasibility. After implementation is complete, accountability for achieving the outcomes of projects generally passes to an operational business unit and ownership of benefits can be lost (paragraphs 3.1 to 3.4).

12 Accountability is also affected by the point at which a project is no longer treated as a major project. The Authority has agreed to remove projects from the Portfolio when they are delivered by a third party and the department has completed a limited role, such as helping to negotiate contracts. For instance, as agreed between the Department for Business, Energy & Industrial Strategy and the Authority at the outset, the project to enable investment in the Hinkley Point C nuclear power station left the Portfolio when the Department had achieved its objectives of identifying investors and signing a construction contract. This Department remains the project sponsor, is responsible for continuing oversight of the developer and has risks to manage but is not reporting on these to the Authority. Other such examples, such as the same department’s Smart Meters Implementation Programme under which energy suppliers must replace customers’ electricity and gas meters, remain in the Portfolio (paragraph 3.5).

13 Poor records and incomplete reporting of the process for leaving reduce transparency, particularly for the early years of the Portfolio. We accept that there is a need for flexibility on when and why projects enter and leave the Portfolio. But lack of transparency increases the risk and perception that projects are removed inappropriately. This is all the more important as changes in policy have led to categories of projects, such as ongoing privately financed projects and many defence projects, no longer being classified as government major projects. It is also not always clear why some programmes have been included in the Portfolio. There are instances of policy initiatives being included to ensure they had central scrutiny, such as the then Department of Health’s programme to increase the number of health visitors and the Civil Service Reform Programme, a programme of government actions to modernise the civil service, but many other such initiatives were not included (paragraphs 2.10, 2.12 and 3.15).

14 In 2016 the Authority introduced a standard process for deciding when projects should leave the Portfolio, addressing many of the shortcomings above (paragraph 5). Since 2016, the Authority has introduced a process which requires projects to have achieved their objectives, completed their core activity, to have a form of exit review and a positive delivery confidence assessment. Most projects now have an exit review and only exceptional cases have left with poor delivery confidence. The Authority has also asked departments to report more detailed data on the expected benefits of projects, which it uses to provide feedback and to engage with departments, but reporting by departments is variable and this information is not published (paragraphs 3.9 to 3.13 and 3.17 to 3.22).

Concluding remarks

15 The Portfolio represents the government’s biggest and riskiest projects, which are also intended to produce £650 billion of benefits. The Portfolio was set up to provide more transparency around their performance, as well as to provide extra assurance. Although there is evidence that most projects that have recently left the Portfolio have implemented their planned scope, it is less clear whether they achieve the intended outcomes. For example, there is no corresponding figure for what proportion of the £650 billion of benefits have been realised. Weaknesses in the Authority’s processes in the past have reduced the degree of transparency around many of the 302 projects that have left the Portfolio. Recent improvements are welcome but there is still a need for the Authority to develop its oversight at exit and for departments to monitor and evaluate projects and their outcomes more consistently, so that performance improves and maximum value is derived from projects.

Recommendations

16 To improve clarity about what projects deliver when they leave the Portfolio, the Authority should:

a be more disciplined in applying exit procedures. It should use the Portfolio’s existing categories of projects to determine standard points when projects would be expected to leave the Portfolio and use these to form the starting point for any subsequent negotiations; and

b improve the collection of benefits data so that mature estimates of benefits can be published alongside other Portfolio project reporting. Projects may enter the Portfolio before forecasts of benefits are robust. However, mature estimates of benefits, such as those in approved full business cases, could be included alongside forecasts of whole-life costs when Portfolio data are published.

17 The Authority and HM Treasury should:

c require projects in the Portfolio to have a business case and maintain cost and benefit estimates to reflect the project’s status. Projects may enter the Portfolio before they have a mature business case but by the time they leave, a full business case should be in place and cost and benefit estimates should be updated to reflect any changes to the scope of the project. The Authority and HM Treasury in its role as finance ministry need to work together to realise their ambitions for benefits realisation, cost discipline and selecting the right projects for future funding.

18 Departments should:

d manage delivery of the benefits of major projects until it is clear what the projects have achieved. Departments should identify a small number of key measures for each project that indicate whether strategic objectives have been met, and monitor progress towards these. Where benefits are likely to become apparent over time or where the full impact can only be assessed over a prolonged period, departments may delay or reassess evaluations; and

e publish a statement on project closure that assesses what has been delivered. Departments are responsible for delivering the benefits of projects and such a statement would close the gap in visibility of major projects. This may be an evaluation planned as part of the project but there should be reasonable consistency in how this is done. Publication of evaluations would help departments learn lessons from major projects, including the realism of expected benefits.

1 Comptroller and Auditor General, Assurance for high-risk projects, Session 2009-10, HC 85-1, National Audit Office, June 2010.

2 Cabinet Office, Overview of the Major Projects Authority, June 2011.

3 Comptroller and Auditor General, Delivering major projects in government: a briefing for the Committee of Public Accounts, Session 2015-16, HC 713, National Audit Office, January 2016.

Part One

Introduction

About the Government Major Projects Portfolio

1.1 In April 2011 the then Major Projects Authority, now the Infrastructure and Projects Authority (the Authority), created the Government Major Projects Portfolio (the Portfolio). The Portfolio aims to improve the delivery of government’s biggest and riskiest projects by increasing transparency and providing independent assurance.4 As of September 2017, the Portfolio consisted of 133 projects in four broad categories (Figure 1). These had a total of £423 billion of budgeted whole-life costs and £657 billion of benefits. There are no corresponding figures for outturn costs or benefits. Major projects are important instruments, managed by temporary organisations, for delivering specific outputs to help achieve government objectives and strategic priorities. It is usually the responsibility of the permanent organisation, such as a government department, to deliver longer-term benefits. The Portfolio represents a fraction of government’s project activity. It does not include projects that are smaller-scale or in early development, nor any EU Exit implementation projects.

1.2 Projects enter the Portfolio following agreement between departments, HM Treasury and the Authority. Typically these are projects that require approval from HM Treasury because their budget exceeds a department’s delegated authority level, could create pressures leading to breaches in financial limits or could make contractual commitments that amount to a significant level of future expenditure. Projects may also be on the Portfolio for reasons other than cost, such as they are novel, contentious, or require primary legislation.

Changes in the Portfolio’s composition

1.3 The Portfolio changes continuously with a flow of projects joining and leaving. Of the 430 projects that had joined the Portfolio between April 2011 when it was created and September 2017, 302 had left (Figure 2). As at September 2017, the number of projects was the lowest in the Portfolio’s history and was one-third fewer than its peak of around 200 in 2014. Part Two covers the reasons projects leave the Portfolio.

Reporting and assurance for projects on the Portfolio

1.4 The Authority requires departments to provide quarterly data returns about their projects, and provides assurance and assessment of deliverability while projects are on the Portfolio by organising independent reviews at key stages in a project’s life (Figure 3). It also aims to provide Parliament and the taxpayer with more transparency about whether projects are likely to be successful by reporting annually on their status.

The importance of benefits in project delivery

1.5 The success of all projects is judged by whether they are delivered to planned time and cost and achieve their objectives, which in government are often defined as benefits to users and society more generally. In 2016 we reported that:

it was difficult to form conclusions about trends in performance across the Portfolio because of the amount of turnover, the limited data published and because there was no systematic monitoring of benefit realisation;

projects often left before they were complete or had achieved their objectives; and

the Authority monitors benefits when a project is on the Portfolio but is not in a position to monitor whether they are realised once it leaves.5

1.6 The Committee of Public Accounts recommended that the Authority should push departments to state project benefits clearly and to establish appropriate data systems to measure them; should certify that benefits were on target to be achieved before a project left the Portfolio; and needed to make it clear to departments that once a project left, the department was responsible for realising the agreed benefits. With benefits estimated at more than £650 billion, even small improvements in the rate of achievement would have a large impact on public services and value for money. Poor measurement of what projects have achieved reduces accountability and transparency for government and Parliament. It also means that government is missing an opportunity to learn about what constitutes success.

What we did

1.7 This report examines:

what happened to projects that left the Portfolio, and whether there is evidence that they achieved their intended benefits; and

whether accountability and transparency have improved.

1.8 Our approach involved analysing data on the 302 projects that left the Portfolio between its inception in April 2011 and September 2017. As delivery outcomes are not reported after projects have left the Portfolio, we followed up on a selection of 48 projects from across 17 departments to examine what happened to them and whether they delivered the intended benefits.

4 Cabinet Office, Overview of the Major Projects Authority, June 2011

5 Comptroller and Auditor General, Delivering major projects in government: a briefing for the Committee of Public Accounts, Session 2015-16, HC 713, National Audit Office, January 2016

Part Two

The outcome of projects leaving the Portfolio

2.1 The Infrastructure and Projects Authority’s (the Authority) original guidance of 2012 expected projects to leave with the agreement of the Authority and, typically, 12 months after the final Gate 5 assurance review (which is intended to confirm that the project is in operation and achieving benefits).6 The Authority did not start recording the stage of projects until 2015-16. Taking all 302 projects that left the Government Major Projects Portfolio (the Portfolio) between 2011 and 2017, we found that there was no information about their stage on exit for 220 projects.

2.2 Of the remaining 82 projects (Figure 4), only 39 left at the benefits realisation or operational phases as would be expected if they followed the cycle set out in Figure 3. Instead, the Authority and departments agree to move projects off the Portfolio, based on the project’s particular characteristics, the level of assurance required and taking account of required project activity and the Authority’s assessment of the likelihood of success.

2.3 The Authority does not report on what projects have delivered when they leave the Portfolio. We examined;

the status of projects when they leave the Portfolio;

what evidence there is that projects have achieved their intended benefits; and

how departments monitor projects after they leave the Portfolio.

Status of projects when they leave the Portfolio

2.4 We followed up 48 projects that had left the Portfolio between 2011 and 2017 with 17 departments to obtain additional information to that held by the Authority. We have not independently evaluated their success, but draw on the evidence departments have provided about what they have achieved to date.

Limitations in data and record-keeping reduce transparency, particularly for the early years of the Portfolio

2.5 Limitations with the data and the lack of documentation makes it difficult to draw conclusions, for example on whether projects left the Portfolio at the right time and for the right reasons. Gaps include:

no single view of which projects had exit reviews, nor was the Authority able to locate in all cases such reviews for requested projects;

thirty-four projects left in 2011-12 without a delivery confidence assessment being recorded; and

documentation on inclusion on the Portfolio and exit from it. For example, we saw very limited documentation to support the Authority’s agreement to remove a large number of projects from the Portfolio on three occasions (see paragraph 2.10).

This reduces transparency. Similarly, departments often had difficulties in responding to our questions about sample projects when relevant people had moved on, indicating a lack of central record-keeping on projects within departments.

Projects leave the Portfolio for reasons other than completion

2.6 Figure 5 shows that 35 out of the 48 projects we examined had reached a significant delivery milestone when they left the Portfolio. Most commonly, the trigger for a project leaving the Portfolio is the project producing a deliverable such as a building or letting a contract, rather than the realisation of stated benefits, such as savings or operational improvements. For example, the project to renovate and modernise the headquarters of the Foreign & Commonwealth Office came off the Portfolio when the Department had moved into the new building. It was too early to assess other intended benefits, such as improved individual and team productivity, financial savings and a more sustainable environment.

2.7 Thirteen of the 48 projects in our selection left for reasons other than delivery. In five cases, the project left because the Authority and departments considered that it had made sufficient progress and no longer needed to report to it, or because a project’s scope had reduced. Four programmes were cancelled either because of not being likely to achieve their objectives, or the availability of a more cost-effective alternative, or government policy changed, or a combination of these reasons. As we have previously stated, it is acceptable for a project to be cancelled, providing that lessons are learned from it and, in the case of a failing project, that it is quickly brought to an end.7

2.8 There were also four projects that left as they were merged or disaggregated into different projects. For example, the Department for Work & Pensions’ IT programme was separated into two smaller projects that subsequently joined the Portfolio. The Priority School Building Programme, a Department for Education project, had multiple iterations on the Portfolio as it was merged or disaggregated following discussions between the Authority and the Department (Figure 6).

2.9 The sale of some of the student loan book was included in the Portfolio, but removed in June 2017 before the sale had been agreed.8 The Authority decided that the project no longer needed to report to it, although the Authority and the sponsor department rated delivery confidence as amber-red because of uncertainty about the sale price. The Authority considered, however, that there was sufficient evidence that the sale could proceed. The only costs were advisers’ fees and there were only two possible outcomes: a successful sale or no sale. HM Treasury continued to monitor the sale.

2.10 There have been three occasions when whole groups of projects have been removed from the Portfolio (Figure 7). When the Portfolio was created there were no pre-existing criteria for including projects, other than requiring HM Treasury approval, and it was not always clear what should be reported as a single programme. Departments and the Authority made pragmatic arrangements, which sometimes changed with experience or as their collective view changed over time. An oversight group (paragraph 3.12) now decides which projects enter the Portfolio.

In quarter 4 of 2011-12, following discussion with the Ministry of Defence, the Authority agreed to remove 38 projects, partly to reduce the reporting burden on the Ministry and also to reflect the merger and consolidation of some projects.

Between quarter 2 of 2012-13 and quarter 2 of 2013-14, seven rail franchising projects came off the Portfolio as the Department for Transport and the Authority agreed to report on one overall rail franchising programme.

In quarter 4 of 2014-15, the then Department of Health lobbied successfully to remove hospital construction projects from the Portfolio, as responsibility for these projects lay with hospital trusts and not the Department, following reorganisation. These projects included the Midland Metropolitan and Royal Liverpool University hospitals, two of the four public sector contracts on which Carillion faced large losses before its collapse.9 While the Authority did not scrutinise the two hospital projects, the Cabinet Office monitored Carillion as part of its risk management system for strategic suppliers. Once it was aware of the company’s financial difficulties in July 2017, the Cabinet Office developed detailed contingency plans.

2.11 Projects may also leave because the Authority agrees that they no longer meet the criteria for a major project and that a department’s own assurance arrangements are sufficient. For example, the Authority agreed that a shared service managed by the Department for Transport could be assured in-house, as following a significant change in its scope, it no longer met the definition of a major project.

2.12 While we accept that there is a need for flexibility on when and why projects leave the Portfolio, the poor recording of the reasons reduces transparency and so undermines one of the objectives of the Portfolio. It also increases the risk and perception that projects are removed inappropriately.

Delivery of projects after they leave the Portfolio

Most projects had delivered the intended outputs, even if longer-term success is not clear

2.13 We assessed that, of our selection of 48 projects, in 12 cases the project had produced the intended outputs, for example, infrastructure had been built, an IT system had been implemented or a business change had been completed, and there was good evidence that it had achieved its outcomes. For example, the Super-Connected Cities Initiative was a £150 million fund to improve internet connectivity, help with small businesses’ broadband costs, Wi-Fi in public buildings and other capital projects. It initially covered 22 cities. The scope was widened to 50 cities and the initiative spent £121 million to 2015-16 when it left the Portfolio. It had clear metrics for measuring success. An impact review concluded that the project had delivered more than intended.

2.14 In 22 further cases, the project had delivered the intended output but we cannot say that it has yet achieved its stated aims. The main reasons for this were that:

achievement might depend on benefits that have yet to be fully delivered, especially for programmes delivering services due to run for a lengthy period. For example, the then Department of Health’s programme to increase the number of health visitors had reached its recruitment target, but we cannot say whether it has yet achieved the intended health outcomes, including improving the health and well-being of under-fives;

in some cases it was not clear what the intended benefits were. Seven of the projects we examined did not have a business case. For example, the Household Energy Efficiency programme (Case study 1) delivered against its target of improving energy efficiency in one million homes, but did not have targets or measurable goals for its wider objectives, such as saving energy;

in five cases, the department was responsible for enabling investment, not for delivering the required infrastructure or system. Responsibility for the longer-term outcomes lay elsewhere, such as with the generating company in the case of the Hinkley Point C nuclear power station, where the Department for Business, Energy & Industrial Strategy was responsible for identifying an investor. All these projects were still under way but were not part of the Portfolio; and

four projects were still being implemented or rolled out. These had left the Portfolio, and it was too early to assess whether benefits had been delivered.

2.15 In one case, the Ministry of Justice’s Prison Unit Cost Programme had broadly met its objectives when it left the Portfolio, but subsequent events cast doubt on its success. The Programme sought to maximise savings from public sector prisons by reducing operating costs while supporting the safety, security and decency of public prisons. It cost £115 million and was expected to save £550 million. The programme broadly achieved the planned cost savings, staff reductions and prison closures by 2016. However, since 2017, the government has sought to increase staff numbers to improve safety in prisons. This demonstrates that longer-term tracking is needed to show whether a project’s outcomes are sustained.

Some projects had achieved less than intended

2.16 As indicated in Figure 5, four of the projects we selected were cancelled. They were replaced in full or in part by other projects. For example, the Immigration Case Work Programme was replaced following the disbanding of the Border Agency, when the then Home Secretary commissioned a new plan to modernise IT across the immigration system building on the capabilities built to date. This and two other projects had spent some £350 million in total before they left the Portfolio. The fourth cancelled programme was Building Schools for the Future, where the incoming government had decided to terminate the programme in 2010. It entered the Portfolio, even though the Department for Education was winding it down with the aim of completing the buildings for which contracts had been let and making savings in ongoing projects. Expenditure to 2014-15 was some £6.2 billion. It left the Portfolio in 2014-15, following its closure. The planned evaluations were cancelled, and there was no plan to realise benefits.

2.17 Six programmes in our selection were scaled back because of problems with development or implementation, or due to changes in the external environment. For example, the Mobile Infrastructure Project reduced its scope and cost because the initial targets were unattainable, although the programme reported success against its revised baseline (Case study 2). Three of these programmes delivered the revised scope, but others, such as the Common Agricultural Policy Delivery Programme, only partially delivered their revised objectives.10

Sometimes it is not clear what the project has achieved

2.18 In some cases, such as the Ministry of Defence’s civilian workforce programme, we could not determine whether the project had delivered what was intended, because the programme was closed and responsibility transferred elsewhere before completion. In three other cases, departments did not record what was delivered. As a result of reorganisation, or in the transition to business-as-usual, responsibility for the project and its outcomes was lost. The departments reported actual spend on these projects of £250 million before they were reorganised.

Departments did not always monitor benefits

2.19 We examined whether departments or the Authority had carried out a review of the projects that had left the Portfolio and whether they were tracking the delivery of the planned benefits of the projects. Departments are not likely to know if projects have delivered their intended objectives or benefits, or what other outcomes they have achieved, unless they do at least one of these. We also asked departments to report on whether they considered the intended outcomes reported to the Authority to have been achieved.

2.20 Departments had either put arrangements in place to monitor the delivery of benefits in 37 of the 48 cases we examined, or were monitoring a successor project. In nine cases the department told us there was no monitoring, either because the project had been cancelled, or because a department’s activities were reorganised and the project had been merged or subsumed into other projects that were ongoing and/or not part of the Portfolio. Departments were unable to answer our questions about two further projects, and we therefore conclude that there is no ongoing activity to track them.

2.21 In addition to ongoing monitoring of benefit delivery, departments had commissioned some form of evaluation or post-project review for 23 out of 48 projects. This included 12 independent reviews or evaluations and a range of internal reviews or project closure reports. In the cases of the Thames Tideway Tunnel, which is under construction, and the Department for Education’s Priority School Building Programme to fund new school buildings, the departments told us it was too early to conduct a review. There were eight projects (including three cancelled projects) for which there was no evidence of either monitoring or review.

2.22 Because some projects deliver their benefits over a long timescale, departments can only demonstrate that they have achieved their aims if they track outcomes over the long term. Departmental practice seems to vary. The Prison Unit Costs example in paragraph 2.15 shows that if monitoring stops after the immediate target is met, it is not clear whether the outcome is sustainable. By contrast, the Electoral Registration Transformation Programme had an external review that demonstrated the impact of the programme up to 2015, but electoral registration is subject to regular reporting by the Electoral Commission, which allows longer-term impact to be tracked.

2.23 We followed up on project monitoring processes with a range of departments. We found variability in their processes. We looked for evidence that there was a clear process, with roles and responsibilities set out and supported by robust guidance. The Ministry of Defence has detailed documented processes for monitoring and evaluating projects. The Ministry of Justice also has centralised guidance. The other departments are developing guidance, and the Department for Digital, Culture, Media & Sport is seeking to develop an Evaluation Centre of Excellence.

6 Major Projects Authority, Government Major Projects Portfolio Guidance, May 2012.

7 Comptroller and Auditor General, Innovation across central government, Session 2008-2009, HC 12, National Audit Office, March 2009.

8 Comptroller and Auditor General, The sale of student loans, Session 2017–2019, HC 1385, National Audit Office, July 2018.

9 Comptroller and Auditor General, Investigation into the government’s handling of the collapse of Carillion,
Session 2017-19, HC 1002, National Audit Office, June 2018.

10 Comptroller and Auditor General, Early review of the Common Agricultural Policy Delivery Programme, Session 2015-16, HC 606, National Audit Office, December 2015; Progress on the Common Agricultural Policy Delivery Programme, Session 2016-17, HC 727, National Audit Office, October 2016.

Part Three

Accountability and transparency

Accountability

3.1 Robust accountability is an essential part of good public management and democratic government. In 2014, the Cabinet Office revised the guidance on accountability for select committees (the Osmotherly Rules) with the aim of sharpening accountability and improving the delivery of major projects.11 Departments issued revised appointment letters in 2015 to senior responsible owners stating that they have personal responsibility for project implementation, including ensuring that the project meets its objectives and that the projected benefits are deliverable. Senior responsible owners are expected to appear before select committees to account for their decisions.

3.2 Despite this development, in 2016, the Committee of Public Accounts (the Committee) commented that there appeared to be a gap in responsibilities for reporting the delivery of benefits. While it was clear that departments were responsible for realising benefits and evaluating success, the Committee reported that there were underlying weaknesses in the way departments articulated benefits and the data the Infrastructure Projects Authority (the Authority) received on them.

3.3 In response to the Committee’s report on accountability12 in 2016, the government agreed that accounting officers should:

prepare assessments of the regularity, propriety, value for money and feasibility of projects on the Government Major Projects Portfolio (the Portfolio), starting at the outline business case approval stage. These were to be produced from April 2017, with a summary of the key points from the assessment published. We have seen one such assessment to date; and

provide greater assurance over major projects while they are being implemented. While responsibility for the implementation of major projects can be delegated, accounting officers cannot delegate personal accountability. Further assessments of the project would be expected after the above approval stage if the project exceeds tolerances in terms of costs, benefits, timescales and risk. We have seen one reassessment in response to a delay to a project on the Portfolio. It concluded that the programme met the standards.

The Committee concluded that accounting officers lacked the cost and performance data required for effective oversight. There is now an expectation that departments will identify how accountability systems align with financial data and performance indicators.

3.4 While these developments are improving the accountabilities for delivering projects to cost and time, accountability for delivering benefits is less clear. We examined how five departments, which spanned a mix of project types and volumes of projects leaving the Portfolio, managed their portfolios of projects.13 All five departments told us that there was a clear distinction between monitoring projects that are in progress and monitoring those that have completed or become business-as-usual. Each department had a team responsible for monitoring its project portfolio, which captured data on the projects’ performance, including delivery of expected benefits, and presented information to oversight groups such as investment committees. However, once projects were delivered, the responsibility for monitoring benefits passed to the part of the business responsible for the work and reporting to these oversight groups ceased.

3.5 Accountability is also affected by the point at which a project is no longer treated as a government major project. Projects may be delivered by a third party and the department’s role is to fund local providers or help to negotiate contracts. In some cases, the Authority has agreed that departments can stop reporting when these tasks are complete. For example, while the Hinkley Point C project was on the Portfolio, its objective was only to identify an investor and sign a contract to construct the new nuclear power plant, as agreed between the department and the Authority at the outset, and did not include the related construction project. It left the Portfolio when this objective was achieved. It would be inappropriate to conclude that the project’s removal from the Portfolio means the wider project of Hinkley Point C is complete. The Department for Business, Energy & Industrial Strategy remains the project sponsor, and is responsible for continuing oversight and ongoing monitoring of the developer and has risks to manage. Other such examples, including the same department’s Smart Meters Implementation Programme under which energy suppliers must replace customers’ electricity and gas meters, remain in the Portfolio as the Department remains the project sponsor, and has a leadership and coordination role.

Transparency

The Authority now publishes more information about projects leaving the Portfolio

3.6 We and the Committee commented in 2016 that transparency about project performance was limited by the information published about the Portfolio. The Authority’s annual report assesses the state of projects currently on the Portfolio through high-level findings on costs and through delivery confidence assessments, which examine the likelihood of the projects being successful.14 While the Authority collects information on benefits, costs incurred and schedule, these are not published. It also did not report on why projects left the Portfolio.

3.7 In 2016 the Authority started reporting in its annual report on why projects left the Portfolio. In 2018, it reported that 29 projects had left the Portfolio between October 2016 and September 2017, as they no longer required Authority support:

twenty had delivered successfully against original objectives;

six had delivered successfully following a reshape of objectives;

one was stopped;

one was closed and replaced by another Portfolio project; and

one no longer met the criteria for being on the Portfolio.

3.8 Because the Portfolio consists of ongoing projects, reporting on it cannot show the final outcome of the projects. It would be possible to be more confident about the delivery of Portfolio projects if the reasons they left the Portfolio were clear and if there was a consistent process to confirm that they would achieve their intended benefits.

The Authority introduced a standard leaving process in November 2016

3.9 Until November 2016 there was no documented process or established criteria for when projects were expected to leave the Portfolio. Projects left with the agreement of the Authority and HM Treasury when they became operational or were otherwise deemed no longer to require the Authority’s oversight. The Authority told us that its individual operations teams decided whether a project had an exit review (Figure 3) and the type of review that would take place. Its guidance states that Gate 5 reviews of operations or benefits realisation should be held six to 12 months after projects moved into the operational phase. In practice, the Authority uses a range of different types of review, for example Gate 5 reviews or project assurance reviews, to examine departure from the Portfolio. From 2014, the Authority introduced the concept of a distinct exit review, but other kinds of review continue to be used and categorised as ‘a form of exit review’.

3.10 We analysed all the reviews of various types that the Authority considered to be exit reviews. Our analysis of these reviews shows that, of the 302 projects that left between April 2011 and September 2017, 176 (58%) did so without a form of exit review (Figure 8).

3.11 The Authority told us that one reason for the low number of exit reviews in the early years was that both departments and the Authority were adapting to the nature and obligations of the Portfolio. Projects came on and off the Portfolio as departments’ understanding grew of reporting requirements and whether projects ought to be on the Portfolio (paragraph 2.10). The Authority was also less likely to undertake an exit review if a project was cancelled or reconfigured into a revised project.

3.12 In response to the Committee’s concerns about projects leaving without an exit review, in 2016 the Authority introduced a centralised process for managing the exit of projects from the Portfolio. Although leaving is still a negotiation between departments and the Authority, an oversight group within the Authority made up of the Authority’s operations managers and chaired by the operations director exists to provide structure and consistency in decisions on when projects join and leave the Portfolio. It uses the following criteria for leaving:

The project has achieved its objectives and completed its core project activity.

The project should have closed or be about to close.

A form of exit review should have been completed.

Only in exceptional cases can a project leave the Portfolio if it has a delivery confidence assessment of either red or amber-red.

3.13 The new process addresses many of the Committee’s concerns. In particular, there is now a defined point of completion, and an independent review of the project takes place at that point to confirm that benefits are likely to be realised. This increases confidence that projects leave the Portfolio for appropriate reasons. Figure 8 shows that, in more recent years, projects have been more likely to leave with an exit review. Since November 2016, there have been instances where projects left the Portfolio without an exit review, however, we found these were for reasons such as that the project was subsumed or merged into an existing project or the Authority agreed than an exit review could be undertaken at a later date.

The Authority is now monitoring but not reporting on scope change

3.14 As stated in Part Two, changes to the scope of projects are to be expected, but can make it more difficult to determine whether programmes are delivering what was intended. From 2015-16 the Authority asked departments to report changes to the scope of their projects. Of the 82 projects that have left since then, nine reported a reduction in scope and seven reported an increase. The Authority does not report on these changes to scope.

Some programmes are not subject to the usual project management disciplines

3.15 In Part Two we explained that some programmes are included in the Portfolio even though they do not have the typical characteristics of projects, such as a dedicated project team and a clear end point. Manifesto commitments or ministerial announcements that increase funding for existing operations, such as HM Revenue & Customs’ organised crime programme and the then Department of Health’s health visitor programme, have come on to the Portfolio. This was because HM Treasury and the Authority wanted to monitor their progress, although other such initiatives are not included. In the case of the health visitor programme the Department benefited from it being on the Portfolio as it introduced programme management disciplines.

3.16 The inclusion of programmes in the Portfolio, however, does not always result in project management disciplines being introduced. Projects do not necessarily have a business case, clearly defined outcomes, a benefits realisation plan or clear lines of responsibility once the policy is in place (Case study 3). Without these elements, it is not possible to conclude that the programmes have achieved what they intended. For example, out of the 302 projects that left, the Authority recorded 48 as having no business case.15 Good practice would require the department to develop a business case to inform decision-makers and this would have to be updated as information and assumptions are developed and refined. However, HM Treasury may agree with departments that writing a business case is not required, as funding has already been committed because of a ministerial decision or policy announcement.

The Authority’s delivery assessments do not always show that projects are on track when they leave the Portfolio

3.17 The Authority and departments use delivery confidence assessments to indicate a project’s ability to deliver its aims and objectives. The assessments consider risks and whether the project has the resilience to overcome threats or shortcomings. Figure 9 shows that 44 projects (15%) out of the 302 projects that left the Portfolio up to September 2017 left with a red or amber-red rating, indicating that successful project delivery was unachievable or in doubt. A further 63 (21%) projects had an amber rating, which meant that significant issues had to be addressed.

3.18 Since November 2016, when the Authority updated its guidance so that projects with a delivery confidence assessment of either red or amber-red should only leave the Portfolio in exceptional cases, only one project (the student loans sale referred to in paragraph 2.9) has left with an amber-red rating. Incomplete projects can leave the Portfolio with amber ratings, which mean that there may be significant issues, but successful delivery is still feasible if they are addressed promptly.

Since 2016 the Authority collects more detailed benefits data, but it is incomplete

3.19 In 2016, the Committee concluded that improving the delivery of projects’ benefits was a priority. The Authority told the Committee it was working to improve the quality of the data on benefits it receives from departments. Since 2015-16 departments report both expected monetised and qualitative benefits on a quarterly basis while projects are on the Portfolio. Prior to this only total financial benefits were reported. In addition, since 2015-16, the Authority has also requested more detail on the nature of monetised benefits to allow distinction between cash savings, non-cash savings and wider economic benefits, allowing a more granular analysis of benefits data. It uses these data to provide feedback to and to engage with departments.

3.20 The nature of the benefits reported in the Portfolio is wide-ranging; examples of qualitative benefits include: meeting modern technology standards; enabling new scientific discoveries to be made; or increasing access to services for users. Defence capability projects do not typically report monetised benefits. For September 2017 portfolio data, we found that all projects had reported at least one intended outcome, with some reporting up to 10.

3.21 Our analysis of information on forecast monetised benefits held by the Authority shows a small improvement in reporting of monetised benefits: the 302 projects that left the Portfolio between 2011 and June 2017 showed that, up to 2014-15, only about half of the projects (111 of 220) reported any monetised benefits, but this proportion increased to over two-thirds (56 of 82) from 2015-16. But we are unable to determine whether departments were not expecting any monetised benefits, or whether they were just not reporting them to the Authority.

3.22 As well as working to improve data on benefits for projects that are on the Portfolio, the Authority and HM Treasury have also sought to raise awareness of benefits realisation across government. In October 2017, the Authority published guidance for managing benefits effectively.16 In March 2018, HM Treasury also refreshed its guidance on appraising and evaluating policies, programmes and projects. The new guidance placed greater emphasis on building monitoring into projects throughout their lifecycle.

11 HM Government, The Civil Service Reform Plan, June 2012.

12 HC Committee of Public Accounts, Accountability to Parliament for taxpayers’ money, Thirty-ninth report of Session 2015-16, HC 732, May 2016.

13 The Ministry of Defence; the Department of Health & Social Care; the Department for Digital, Culture, Media & Sport; the Ministry of Justice; and the Department for Work & Pensions.

14 Under its Prime Ministerial mandate, the Authority is required to produce an annual report on the Portfolio. It published its first annual report in 2013.

15 The Portfolio reported 39 projects where no business case was required and a further nine projects that did not report the latest business case stage.

16 Infrastructure and Projects Authority, Guide for Effective Benefits Management in Major Projects: Key benefits management principles and activities for major projects, October 2017.

Appendix One

Our audit approach

1 In 2016, we could not conclude whether the establishment of the Infrastructure and Projects Authority (the Authority) and the Government Major Projects Portfolio (the Portfolio) had improved government’s performance in delivering major projects. In part because limited data were published and because there was no systematic monitoring of whether projects realised their benefits.

2 The Committee of Public Accounts (the Committee) identified that improving the delivery of benefits should be a priority. It commented that there appeared to be a gap in responsibilities for reporting the delivery of benefits and expressed concerns around the quality of data, including benefits, that some projects left the Portfolio without having a review to ensure that they are on track to deliver their benefits.

3 This report follows up on the issues raised by the Committee by examining whether:

there is evidence that projects which were on the Portfolio have delivered their intended benefits; and

there are improvements in accountability for, and transparency of, major project delivery.

4 We consider the process and data available around projects leaving the Portfolio and for delivery of intended benefits we followed up on a selection of 48 projects that had exited the Portfolio, as the Authority only reports on projects that are on the Portfolio.

5 This report does not seek to evaluate the success of each individual project but rather examine the evidence available that projects have delivered their intended benefits and the arrangements departments have in place for monitoring and evaluating project delivery.

6 Figure 10 outlines our audit approach. Our evidence base is described in Appendix Two.

Appendix Two

Our evidence base

1 We reached our findings following an analysis of evidence collected between February and July 2018. Our main methods are outlined below:

Processes to manage the exit of projects from the Portfolio

Analysis of Infrastructure Projects Authority (the Authority’s) and Government Major Projects Portfolio (the Portfolio) data of departmental quarterly returns of major projects between quarter 1 of 2011-12 and quarter 2 of 2017-18 to identify all projects that left the Portfolio between this period and information such as costs, benefits, project stage and delivery confidence assessment.

Analysis of the Authority’s datasets to identify which projects had an exit review.

Review of Authority’s published guidance of assurance reviews while projects are on the Portfolio.

Review of documents that set out the Authority’s mandate and its wider operating context, including characteristics of projects expected to come onto the Portfolio.

Review of the Authority’s guidance to understand the expected processes for projects exiting the Portfolio and interviews with Authority staff as to exiting arrangements and how these have changed.

Interviews with Authority staff about processes applied for exiting the Portfolio and review of supporting documentation, including where available exit reviews.

Delivery of intended benefits of projects

We selected 48 projects from 17 departments or their agencies (Appendix Three). The selection of our sample included all organisations that had a project leave the Portfolio since its inception to quarter 1 of 2017-18. We selected a sample to provide a spread of projects encompassing different project categories, costs and delivery confidence assessment.

Interviews with Authority staff on our sample of 48 projects about processes applied for exiting the Portfolio, performance of projects and review of supporting documentation, including where available exit reviews.

We sent questionnaires to departments or bodies responsible for our sample projects, requesting information, such as why projects left the Portfolio, current status, what it had delivered and whether it was monitored or evaluated.

Literature review of documents of our sample, including National Audit Office reports, published evaluations or performance reports, documentation provided by departments and Authority reviews.

From our selected projects we used Authority and departmental information to determine which projects had left due to project completion, rather than removal due to other reasons such as cancellation. For these projects, we requested, and reviewed where available, latest business cases from departments for our sample of projects to identify the intended benefits and what departments were monitoring.

Departments’ monitoring and evaluation arrangements

For our 48 selected projects we issued questionnaires to departments to understand how they were monitoring and evaluating these projects.

Interviews with project and programme management staff from the Department for Digital, Culture, Media & Sport; the Department of Health & Social Care; the Department for Work & Pensions; the Ministry of Justice; and the Ministry of Defence to gather insight more broadly around benefits monitoring, portfolio management and engagement with the Authority’s exit processes. Supplemented with review of departmental guidance documents around processes for project monitoring and benefits realisation.

Review of central government guidance of mandated and recommended practice, including HM Treasury guidance on appraisal and evaluation of programmes and projects and the Authority’s guide for effective benefits management in major projects.

Appendix Three

About our sample

1 Figure 11 shows the list of projects in our sample.

To illustrate the scale of the projects we selected, we list here the forecast whole-life cost of each project reported in the last Government Major Projects Portfolio return before it left the Portfolio. For cancelled projects this is the cost of the programme had it continued. Where forecasts differ from known actual costs, we have reported this in a footnote.

Design and Production by NAO External Relations
DP Ref: 11709-001