Report

by the Comptroller
and Auditor General

Department for Business, Innovation & Skills

BIS’s capital investment in science projects

HC 885 Session 2015-16 10 March 2016

Our vision is to help the nation spend wisely.

Our public audit perspective helps Parliament hold government to account and improve public services.

The National Audit Office scrutinises public spending for Parliament and is independent of government. The Comptroller and Auditor General (C&AG), Sir Amyas Morse KCB, is an Officer of the House of Commons and leads the NAO, which employs some 810 people. The C&AG certifies the accounts of all government departments and many other public sector bodies. He has statutory authority to examine and report to Parliament on whether departments and the bodies they fund have used their resources efficiently, effectively, and with economy. Our studies evaluate the value for money of public spending, nationally and locally. Our recommendations and reports on good practice help government improve public services, and our work led to audited savings of £1.15 billion in 2014.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 8 March 2016

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

4 March 2016

This report examines whether the Department for Business, Innovation & Skills (BIS) can demonstrate it is achieving value for money from its investment in science projects.

© National Audit Office 2016

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.gsi.gov.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

10905 03/16 NAO

The National Audit Office study team consisted of:
Kevin Coughlan, Cornelius Halladay-Garrett, Rachel Kift, Sonia Sousa and Heather Thompson, under the direction of Peter Gray.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Key facts

Summary

1 The government invests in science to support economic growth, improve national productivity and help the UK take the lead in new markets. The Department for Business, Innovation & Skills (BIS) has overall responsibility for the government’s spending on science, technology and engineering. It also provides funding for a wide range of scientific disciplines and industry sectors with the aim of developing and maintaining the UK’s science and research capability.

2 In 2014-15, BIS allocated £1.1 billion of capital funding to science (Figure 1). This covered its expenditure on major national projects such as oceanographic research ships, supercomputers and research institutes, capital funding for large national research facilities such as particle accelerators, and the UK’s participation in international programmes such as the European Space Agency. It also covered capital funding allocated by the Higher Education Funding Council for England (HEFCE) for laboratories and research facilities in universities.

3 In December 2014 BIS announced plans for a further £5.9 billion of capital expenditure on science between 2016 and 2021. This included £800 million for new projects, more than £1.2 billion for ongoing national and international projects and a £900 million fund to respond to new challenges as they emerge. It also included plans to spend around £3 billion on the underlying laboratory infrastructure in universities or research institutes. The 2015 Spending Review confirmed this level of future funding. Under the Haldane principle, the government does not take decisions on whether to fund specific projects or individual research proposals. However, ministers have the final say in decisions that involve large-scale, long-term commitments such as the construction of large research facilities.

4 We last reported on the government’s spending on large science facilities in 2007.1 The report was generally positive about BIS’s delivery of a significant capital programme and how it and the research councils had set about prioritising project proposals. Our report did, however, raise concerns about how BIS assessed the ongoing costs of projects and the impact of meeting those costs on the balance of activities funded by the research councils. We also concluded that BIS needed to give more attention to specifying from the start how to assess and measure the success of individual projects.

5 A report published in 2013 by the House of Lords Science and Technology Committee2 raised concerns that the cost of running large-scale infrastructure had not been budgeted for. It also concluded that the potential of the UK’s large-scale scientific resources is being compromised by the lack of a long-term strategic investment plan. In response, BIS launched a public consultation in 2014 to identify strategic priorities for science investment.

6 Reports published in 2015 by the House of Commons Science and Technology Committee and by Sir Paul Nurse acknowledged that the UK research base is world-leading, producing excellent research in a competitive system.3,4 However, they also emphasised the importance of ensuring capital investments are accompanied by sufficient resource funding. Between 2010 and 2014, science resource funding fell in real terms. The 2015 Spending Review committed to maintain science resource at £4.7 billion in real terms up to 2021.

Scope and approach

7 This report covers BIS’s investment in large facilities and other national and international capital projects managed by the research councils, and capital projects managed by HEFCE through its UK Research Partnership Investment Fund (UKRPIF). Together, spending on these projects totalled £638 million in 2014-15, more than half of BIS’s total capital expenditure on science. The remaining capital funding, not examined in this report, comprises funding allocated to smaller projects by the research councils and the funding HEFCE allocates directly to universities for smaller research facilities.

8 The report examines whether BIS has robust arrangements in place to select the projects most likely to add value; whether it has adequate information on project performance once projects are operational; and how BIS evaluates whether projects are delivering benefits. Part One covers the government’s plans for investing in capital science projects, Part Two focuses on how BIS has decided its capital spending priorities and Part Three examines BIS’s assessment of the performance and impact of operational projects. The merits of the scientific case underpinning these projects do not form part of this report.

9 Full details of our scope and audit approach are set out in Appendices One and Two. Appendix Three lists all capital projects we identified as being within the report scope.

Key findings

Deciding capital spending priorities

10 BIS has carried out a partial assessment of the state of UK science infrastructure but a broader assessment would provide BIS with consolidated information to inform its decisions on spending priorities. BIS consulted with the research community to determine how much to invest in major projects and how much to spend on underlying laboratory infrastructure. HEFCE’s assessment of the condition of infrastructure in the higher education sector helped inform understanding about the scale of investment needed, and some of the research councils have assessed the condition of the facilities they fund. But BIS had not carried out or commissioned a broader assessment of the extent that facilities funded by BIS or other government departments, or international facilities, meet the UK’s needs (paragraphs 2.2 to 2.4).

11 Since 2010, processes for sifting project proposals to identify investment priorities have not been supported by good information:

Prior to 2014 BIS did not have a plan for prioritising its capital investment in science projects. Because the 2010 Spending Review resulted in a significant reduction in funding for new science capital projects, the research councils did not, after 2010, continue their well-established exercise to recommend projects for funding. Instead, they developed a strategic framework which set out priorities for investment in science but did not identify specific projects for funding. When extra funding did become available, usually at short notice, BIS had to quickly identify projects where funding could begin to be spent but did not have a plan to help it prioritise projects in a structured way. These proposals were subject to business case approval (assessed in paragraph 12 below) (paragraphs 2.5 to 2.7, 2.21 to 2.23).

In 2014 BIS undertook a prioritisation exercise that identified 15 new projects involving capital expenditure of £800 million up to 2021. However, there were weaknesses in how it prioritised projects. BIS carried out a public consultation with the research community and agreed the criteria it would use to prioritise projects, but did not specify the information it needed from respondents. As a result, it did not have good-quality information to assess and prioritise new projects. A further 4 projects were announced without being assessed. BIS informed us this was because it had identified them as crucial to the UK and its international standing. BIS informed us that, in all cases, decisions to proceed would be subject to a satisfactory business case (paragraphs 2.8 to 2.11, Figure 9, and paragraph 2.24).

12 The analysis supporting recent business cases has not always been complete. We reviewed 20 business cases approved between 2008 and 2015 and found that some of the more recently approved business cases lacked key analysis, such as an assessment of alternative options, estimates of what projects could cost to run, or assurance on how ongoing costs would be funded. Running costs of science infrastructure can be substantial. BIS has committed £3.2 billion of capital expenditure to 56 projects since 2007. We estimate that these projects may cost some £2 billion to run between 2015-16 and 2020-21. BIS believes that its 2015 Spending Review resource settlement will cover the costs of running projects but we have not seen analysis to support this (paragraphs 2.12, 2.14 to 2.16, Figure 10).

13 HEFCE’s approach to prioritising and approving capital projects in higher education institutions has, in most respects, been robust. HEFCE’s UKRPIF tends to be used to fund smaller projects than some of those managed by the research councils. Nevertheless, we reviewed 8 projects and found that HEFCE had gained assurance that all were sustainable and would deliver scientific and economic impacts (paragraphs 2.17 to 2.19, Figure 14).

Performance and impact

14 Many projects were delivered on time and within budget, with few exceptions. Despite the technical risks involved with cutting-edge science projects, there are examples of complex projects that were delivered on time and within budget. Of the 10 projects we examined in 2007, 5 were delivered on budget including the HECToR supercomputer (£65 million). Three exceeded their budgets by more than 10%. These included the Halley VI Antarctic research station, which was 4 years late and £15 million (46%) over budget because of reported difficulties with the design specification, the quality of construction and the challenging location. Of 20 subsequent projects that are now operational, none were delayed by more than a year and 16 were delivered within budget (paragraph 3.2).

15 Among the projects that have been operational for some time, many are in high demand and have produced benefit to science and society. The Diamond Light Source, a particle accelerator, has been operational since 2007 and has enabled scientific achievements in a number of fields. For example, scientists at Diamond have worked with car manufacturers to understand how the structure of steel can be manipulated to make faster and more streamlined cars. The Royal Research Ship James Cook began operations at sea in 2006 and has been used for climate change research (paragraph 3.9, Figure 18 and Figure 19).

16 Of the operational projects we examined, 1 had run significantly below capacity due to resourcing constraints. The number of days the ISIS neutron source, a particle accelerator, was available for experiments was below the capacity of the facility between 2006 and 2014. This was because its funding was not sufficient to cover power costs or the number of technicians needed. The availability of the facility for scientific work has improved since BIS increased its resource funding in 2015-16 (paragraph 3.11).

17 BIS and the research councils do not have a common systematic framework for assessing whether operational projects are delivering expected benefits. Research councils and projects use a range of different approaches to assess the impact of individual projects including case studies to illustrate benefits achieved and reviews of the impacts of research. However, taking a more systematic approach would help BIS assess whether projects across its portfolio are delivering what was expected and inform future spending decisions (paragraphs 3.13 to 3.14).

18 Few of the operational projects we examined have calculated the economic impact of projects. Our 2007 report highlighted the importance of measuring the economic impact of science capital projects. The Biotechnology and Biological Sciences Research Council has measured the economic impact arising from its £137 million investment into the Babraham Research Campus, calculating that it has helped create 6,673 jobs and generated £298 million of value to the UK economy. There are also plans to assess the economic impacts of the European Bioinformatics Institute and the UK Data Service. While it will not always be proportionate to carry out a full economic appraisal, in some cases an assessment would allow BIS to demonstrate economic benefits have been achieved and support the case for further investment (paragraph 3.14).

Conclusion on value for money

19 In 2014-15, BIS’s capital spending on science was above £1 billion and the 2015 Spending Review confirmed that this level of spending would be maintained up to 2021. Many projects have been delivered on time and within budget, have high levels of demand and have made notable scientific impacts. However, since 2010, BIS has lacked a clear process for deciding which projects are investment priorities, and BIS’s processes for prioritising projects and taking spending decisions have not been consistently supported by good-quality information such as what projects could cost to run. BIS also lacks adequate analysis of whether its investment in a portfolio of science capital projects is optimising scientific and economic benefits. We regard these shortcomings as avoidable and undermining of BIS’s ability to prioritise and deliver value for money across the range of its capital funding of scientific research.

Recommendations

20 BIS needs to develop a more systematic and informed approach to investing in science projects. In particular, BIS should:

a Set out a more structured and strategic process for proposing projects, identifying priorities and taking funding decisions, potentially as part of its plans for the recently proposed integrated Research UK organisation. BIS’s aim should be to optimise the value of its portfolio of investments. To ensure decision-making is soundly based, the prioritisation process should be supported by robust analysis of, for example, the likely costs of running projects and the anticipated economic and scientific benefits.

b Conduct a systematic analysis of the existing infrastructure. To take informed decisions on capital investment, BIS needs to ensure there is an adequate picture of the existing infrastructure and its ability to support BIS’s science strategy, including current gaps and emerging priorities, the need for future upgrades and renewals, and the extent to which international facilities can meet UK requirements. To gain this picture, BIS should draw on existing information and analysis held by its partner organisations and other sector bodies.

c Ensure that decisions to invest in capital projects are not taken without a robust assessment of the costs likely to be incurred over the life of the projects. At a time when available resources are limited, taking decisions without sufficient information on what projects could cost to run may have long-term consequences for how the UK science budget is spent.

d Optimise the value from its investment decisions by carrying out an appropriate level of analysis before committing to individual projects. In particular, BIS should consider what options are available to achieve desired outcomes, analyse the demand for projects and assess the scientific and economic impact expected from the project.

e Take a more systematic approach to evaluating the impact of operational projects. BIS’s current approach may not be capturing all the economic and scientific benefits of the projects it has funded. While the extent of analysis that is possible will depend on the nature and scale of each project, assessing projects in a more structured way will help to inform BIS’s future investment decisions.

f Work with HM Treasury to consider how best to provide a predictable funding framework for planning scientific capital investment as part of any review of future spending. Funding allocations for science projects were unpredictable between 2010 and 2014. This led to projects being selected, often at short notice, to match funding that became available unexpectedly. In 2014, the government committed capital funding for science up until 2021, allowing BIS to plan which projects to fund. Decisions about investment priorities are likely to be better informed if decision-making takes place in a more predictable framework for funding longer-term projects.

1 Comptroller and Auditor General, Big Science: Public investment in large scientific facilities, Session 2006-07, HC 153, National Audit Office, January 2007.

2 House of Lords Select Committee on Science & Technology, Scientific Infrastructure, 2013.

3 House of Commons Select Committee on Science & Technology, The science budget, 2015.

4 Sir Paul Nurse, A review of the UK research councils, Ensuring a successful UK science endeavour, November 2015.

Part One

Background

The government’s plan for investing in science

1.1 The Department for Business, Innovation & Skills (BIS) has overall responsibility for the government’s expenditure on science, technology and engineering. BIS’s investment covers a wide range of scientific disciplines and industry sectors, and aims to develop and maintain the UK’s science and research capability.

1.2 In 2014 BIS published Our plan for growth: science and innovation, which highlighted that the UK’s ability to capitalise on its science base will be critical for future prosperity and well-being.5 It identified investment in infrastructure, training and skills, and commercialisation of science and technology as key to economic growth. It also set out the ‘8 Great Technologies’, the key sectors where the government has decided to invest to exploit the UK’s science capabilities and influence economic growth.

Science and research infrastructure

1.3 In order to support and advance the UK’s science and research capabilities, BIS invests in a range of infrastructure from small-scale laboratories, focused on discrete research projects, to large national and international facilities (Figure 2). Scientific infrastructure is used to carry out a wide range of pure and applied research across many scientific sectors and disciplines. The results of research can be used to inform scientific knowledge, leading to benefits for both the economy and society (Figure 3 and Figure 4). In particular, the government invests in science to encourage innovation, create new jobs, increase productivity and allow the UK to take the lead in new markets.

BIS’s funding of science

1.4 This report focuses on two main channels for BIS’s science capital funding: (1) the research councils and the UK Space Agency; and (2) the Higher Education Funding Council for England (HEFCE) (Figure 5).

1.5 The capital budget managed by the research councils includes:

funding provided for specific projects – Decisions to fund the projects are taken by BIS;

international subscriptions – The level of annual capital funding for international projects reflects the UK’s existing commitments and decisions taken by BIS to participate in new international projects;

funding for large facilities – BIS decides the level of capital funding required for the 3 large national facilities, the Diamond Light Source, the ISIS neutron source and the Central Laser Facility; and

research councils’ core budget – The research councils are free to decide how to spend the remaining capital budget, which may be used for minor upgrades, capital grants and estates maintenance. Except for projects with a capital value greater than £2 million, this element of total capital expenditure is outside the scope of this report.

1.6 The capital funding HEFCE allocates to higher education institutions includes:

UK Research Partnership Investment Fund – This fund, introduced in 2012, is designed to enhance research facilities in higher education institutions through funding large-scale projects that can attract co-investment from private sources; and

other capital grants – As autonomous institutions, universities can spend the capital grants allocated by HEFCE according to their own priorities – unless these grants have been awarded competitively, when specific terms and conditions apply. This element of total capital expenditure is outside the scope of this report.

1.7 Since 2012-13, BIS’s annual capital expenditure on science has risen from less than £600 million a year to more than £1 billion a year (Figure 6). At the 2010 Spending Review, the capital element of the science budget, which was previously protected, was reduced by 41% from £873 million in 2010-11 to £514 million in 2011-12. The capital budget was expected to remain at that level over the spending review period. However, in 2011-12, and in the following years, extra capital funding became available for new projects and capital spending on science increased beyond what was originally planned. The 2015 Spending Review committed £5.9 billion of capital expenditure between 2016 and 2021.

1.8 We estimate that since we reported in 2007,6 BIS has spent or committed to spending a total of £3.2 billion on 56 major capital projects.7 In addition it has spent or committed to spend £500 million8 through HEFCE’s UK Research Partnership Investment Fund (UKRPIF).

1.9 Over the same period, 2010-11 to 2014-15, resource funding for research activity and other ongoing costs of operating projects remained constant, while rising as a percentage of BIS’s total resource expenditure (Figure 7). BIS believes that the 2015 Spending Review settlement, which committed to maintain funding for science resource budgets in real terms until 2021, will cover the costs of running current and new projects.

1.10 This report examines whether BIS has robust arrangements in place to select the projects most likely to add value; whether BIS receives adequate information on project performance once they are operational; and how BIS evaluates whether they are delivering benefits. Part Two focuses on how BIS has decided its capital spending priorities and Part Three examines BIS’s assessment of the performance and impact of operational projects.

5 HM Treasury, Department for Business, Innovation & Skills, Our plan for growth: science and innovation, December 2014.

6 Comptroller and Auditor General, Big Science: Public investment in large scientific facilities, Session 2006-07, HC 153, National Audit Office, January 2007.

7 For the purpose of this project, we have defined major projects as national or international projects with capital costs to BIS of greater than £2 million. The £3.2 billion does not include smaller projects, expenditure funded by the research councils for minor upgrades, capital grants and estates, or other funding allocated by HEFCE.

8 Does not include capital funding allocated to universities other than through the UKRPIF.

Part Two

Deciding capital priorities

2.1 This part examines:

the Department for Business, Innovation & Skills’ (BIS’) information on the state of UK science infrastructure;

BIS’s approach to deciding capital investment priorities; and

whether investment decisions are supported by robust evidence and analysis and a clear process.

Understanding the state of UK science infrastructure

2.2 In order to make well-informed decisions on capital investment, BIS needs sufficient information, collected in a cost-effective way, on the state of existing infrastructure, as well as a good understanding of the future scientific challenges the UK may face. This information is needed to help BIS identify gaps in current or future capability, and to inform its decisions about whether to develop an existing facility, invest in a new national facility, or join an international project. Such information might include:

an inventory of the UK’s research infrastructure, with an assessment of its state of repair and fitness for purpose;

information on the expected lifetime of projects and the optimal or planned timetable for upgrades and renewals;

utilisation rates of current facilities; and

the extent to which existing or planned international facilities will meet the UK’s current and future needs.

2.3 BIS has carried out a partial assessment of the state of UK science infrastructure. Some information has been collected, for example, Higher Education Funding Council for England (HEFCE) has assessed the condition of infrastructure in the higher education sector, and some of the research councils have assessed the infrastructure for specific sectors or types of equipment. In addition, an EU-funded initiative completed in 2012 mapped research infrastructure across the EU providing a partial inventory of the UK’s stock of larger facilities.9

2.4 There is scope to further expand this partial assessment of the state of UK science infrastructure. Much information already exists on the condition of national facilities, and the extent to which current or planned international facilities or research facilities managed by other government departments meet the UK’s needs. However, this information is held by a variety of organisations and has not been brought together and assessed as a whole. A broader assessment would provide BIS with consolidated information to inform its spending decisions.

The impact of funding uncertainty on BIS’s planning and decision-making

2.5 Since 2010, uncertainty over the amount of capital funding available for science has shaped the way BIS has decided priorities for science capital funding. Following the 2010 Spending Review, funding was no longer allocated for any new science capital projects. As a consequence, the research councils did not continue to develop the large facilities roadmap that had previously identified projects for capital funding. Instead, they developed a strategic framework for capital investment, which set out which areas of science were priorities for investment but did not identify specific projects as candidates for funding.10

2.6 The absence of a prioritised roadmap meant that BIS did not have a prepared investment plan to support its decision-making. Between 2011 and 2014 additional funding for science became available at short notice and BIS was often required to identify projects where funding could be spent quickly. In 2013 the House of Lords Science and Technology Committee raised concerns about the absence of a funded long-term strategic plan for investing in scientific infrastructure.11

2.7 In total, BIS announced nearly £1.7 billion of additional spending on capital projects that it had not originally expected to be available (Figure 8). These proposals were subject to business case approval (assessed in Figure 10).

BIS’s consultation on capital projects

2.8 Recognising the need for a more structured approach to science capital funding, in April 2014, BIS launched a public consultation to identify strategic priorities for long-term science and research capital investment. It asked respondents:

What balance should we strike between meeting capital requirements at the individual research project and institution level, relative to the need for large-scale investments at national and international levels?

What should be the UK’s priorities for large-scale capital investments in the national interest, including where appropriate collaborating in international projects?

2.9 Although the consultation primarily sought views on future priorities, BIS also asked for views on potential new projects; agreed the criteria it would use to sift project proposals; and used the consultation results to select new projects. In December 2014, BIS announced plans for allocating £5.9 billion of capital expenditure to major projects and underlying laboratory infrastructure between 2016 and 2021.12,13 This included around £800 million funding for 15 new capital projects subject to business case approval (Figure 9).

2.10 We identified weaknesses in how BIS used the responses it received from the consultation exercise to sift project proposals. In particular:

BIS did not specify the information it needed respondents to provide, such as estimated capital and resource costs. As a result, it did not have the information it needed to assess all of the projects suggested by respondents. For the projects it did assess, it did not have all of the information it needed to score projects against some of the assessment criteria; and

BIS did not have a clear rationale for how it scored the projects. It did not set out scoring criteria and, although BIS considered some criteria more important than others, it did not weight the scores for those criteria.

2.11 Furthermore, at the point BIS sifted project proposals and announced its capital spending plans, it did not have robust information on what some projects would cost to run so did not know what projects would cost to run in total. It was also not certain how much resource funding would be available to support new projects, so did not know whether the portfolio as a whole was affordable. BIS plans to assess the costs of running individual projects at the business case stage. Specific issues include:

of the 15 new projects that BIS selected from the consultation responses, over half were not scored on what they would cost to run (Figure 9);

information on running costs may not have been provided on a consistent basis. For example, the capital consultation document stated that running costs for the Engineering Structures and Systems project (a project for testing innovative engineering technologies) would be £105 million over 3 years. The scorer judged that the project’s running costs ‘should be affordable’. However, BIS informed us that this figure included the value of research grants for the project and actual running costs are estimated at £5 million over 3 years; and

BIS has committed to some projects without determining how it will fund ongoing costs. In 2014, BIS committed to joining two international programmes: the European Spallation Source and the European X-ray Free Electron Laser. This involved a total capital investment of £195 million. While BIS expected participation to enhance national capability, when it committed to join the projects BIS had not decided how it would fund the ongoing costs of at least £14 million per year or whether the UK would need to withdraw from another international programme to meet these new commitments.

2.12 The cost of operating scientific infrastructure can be substantial. For example, since 2007 we estimate that BIS has spent around £3.2 billion on science capital projects. We estimate that it may cost BIS some £2 billion to meet the running costs of these projects over the 6-year period between 2015 and 2021.14 This includes the costs of operating the infrastructure, but not the costs of carrying out research. BIS believes that, given research grants account for a large proportion of the research councils’ expenditure, there is considerable flexibility for the research councils to meet the running costs of future projects.

2.13 Our work in other areas has highlighted the problems that can emerge if departments do not maintain control over the total long-term costs of procurement programmes. In particular, our work on the Ministry of Defence’s (MoD’s) equipment plan has examined the challenges MoD has previously faced in dealing with an affordability gap in its equipment plan. In 2015 we reported that the 2015 equipment plan looked more stable than the previous year’s plan but identified the need for the MoD to make room in its budget for the support costs of a range of new equipment currently being procured.15

Information supporting investment decisions

Major projects

2.14 Investment decisions on many of the 15 new projects still depend on satisfactory business cases. The quality of these business cases will be crucial to addressing the gaps we identified in the information supporting the December 2014 prioritisation. To test whether business cases were likely to be of a satisfactory quality, we reviewed a sample of 20 business cases approved between 2007 and 2015. Details of our approach are set out in Appendix Two.

2.15 Based on the business cases we reviewed, there are indications of a decline in the level of information in the business cases that BIS approved after late 2011. Most of the business cases approved before November 2011 included evidence that HM Treasury considers necessary for a soundly based investment decision.16 However, some business cases we examined that were approved after November 2011 lacked adequate analysis in some areas (Figure 10).

2.16 Specific areas where some business cases lacked analysis include:

in 7 out of 20 business cases we reviewed, BIS and the research councils took investment decisions without adequate consideration of what projects could cost to run or an assessment of whether running costs were affordable (Figures 11 and 12). In some cases, the assumption was that running costs would be funded from the research councils’ core budgets but the research councils were not able to confirm whether the additional expenditure would be affordable because they did not yet know how much funding they would receive in future spending review periods. Our published guide, Initiating successful projects, highlights the importance of committing to projects that are affordable;17

6 out of the 20 business cases we reviewed did not include adequate analysis of alternative options. These cases included the £189 million investment into Big Data and Energy-Efficient Computing, where the range of options in the business case was limited by the need to find projects that would use the funding within the current spending review period (Figure 13);

5 out of the 20 business cases did not adequately assess the potential demand for the project or its capability. Whether or not a project is well-used, either by publicly funded researchers or by industry, will be a factor in its ability to deliver the planned scientific and economic benefits; and

14 of the 20 business cases we examined did not include adequate sensitivity analysis. HM Treasury’s Green Book recommends testing the sensitivity of costs and benefits to changes in assumptions. Sensitivity analysis helps decision-makers understand the extent that estimates of costs and benefits are vulnerable to uncertainties.

UK Research Partnership Fund (UKRPIF)

2.17 We also examined the information that HEFCE uses when it allocates capital science funding to universities through its UKRPIF. We reviewed 8 bids approved between 2012 and 2014.

2.18 We found that HEFCE had collected the evidence it needed to support its investment decisions in all of the cases we reviewed. In particular, HEFCE requires evidence that the project’s funding sources for operating and research costs are sustainable. It also requires projects to estimate the return on investment to show that they can deliver value for money (Figure 14).

2.19 However, given the uncertainties in predicting future running costs, HEFCE would have better information to assess the robustness of estimates if bidders were required to provide a range of costs rather than single-figure estimates. Sensitivity analysis of costs and benefits would give HEFCE a clearer understanding of the level of risk in the project and bring its investment decision process closer to the best practices recommended by HM Treasury.

The process for taking investment decisions

Identifying priorities

2.20 Clarity over roles and responsibilities is important if BIS is to have a robust process for taking investment decisions. Under the Haldane principle, the government does not take decisions on whether to fund specific projects or individual research proposals, and therefore BIS delegates detailed decision-making to its partner organisations. In 2010 the government made a statement clarifying the Haldane principle and, in particular, stated that ministers have the final say in decisions that involve long-term and large-scale commitments, including the construction of large research facilities where business cases require ministerial approval.

2.21 In 2007 the research councils used a well-delineated staged approach for identifying priorities for capital investment aimed at developing a large facilities roadmap (Figure 15), whereby the research councils consulted with the research community to identify candidates for funding, and prioritised projects according to agreed criteria. Ministers took the final decision on which projects should be funded.

2.22 As described earlier in this report, the 2010 Spending Review resulted in a significant reduction in funding for new science capital projects and the approach for prioritising large projects was discontinued. Instead, when additional capital funding became available at short notice, BIS consulted the research councils to identify opportunities to fund projects and also responded to project proposals from other parties. For example, HM Treasury provided funding specifically for a number of projects including the National Graphene Institute and the Alan Turing Institute.

2.23 While all project proposals were subject to business case approval (assessed at Figure 10), the investments were often announced before business cases had been developed and approved. Consequently, from 2011, BIS instructed the research councils to take a ‘lighter touch’ to business cases. BIS informed us that where funding had already been identified for specific projects, business cases were generally shorter, with fewer alternative options. BIS advised the research councils that the length of the business case should recognise where investments had already been announced and the strategic case had already been implicitly accepted. For example, BIS informed us that it would not have been relevant to assess alternative options to the National Graphene Institute because the investment had already been announced.

2.24 Projects were also announced in advance of BIS’s 2014 exercise to prioritise capital projects following a consultation with the research community, as described earlier in this report. Four projects announced in early 2014 were not assessed in the sifting exercise because funding had already been allocated. BIS informed us this was because it had decided the projects were crucial to the UK and its international standing.

9 European Science Foundation, Mapping of the European Research Infrastructure Landscape. Available at: www.esf.org/serving-science/ec-contracts-coordination/meril-mapping-of-the-european-research-infrastructure-landscape.html

10 Research Councils UK, Investing for Growth, 2012.

11 House of Lords Select Committee on Science and Technology, Scientific Infrastructure, 2013.

12 HM Treasury, Department for Business, Innovation & Skills, Our plan for growth: science and innovation, December 2014.

13 The £5.9 billion comprised: (a) the £2.9 billion Grand Challenges fund for large-scale investments in science, which includes £1.2 billion of spending already announced (on projects including the Square Kilometre Array, the new polar research ship, the European Spallation Source and the Plato Space Mission), £800 million to fund new projects and a £900 million agility fund to respond to new challenges as they emerge; and (b) £3 billion for individual research projects and facilities at research institutions, including funding to cover the capital elements of the UK’s subscriptions to international projects.

14 This estimate includes projects with approved business cases only and is based on information available in February 2016. It assumes that, in most cases, running costs will remain constant during the 6-year period, including BIS’s contributions for running international projects. It does not include running costs met by bodies other than BIS or the research councils, such as higher education institutions or charities. Not all of these costs will be additional as some new projects will replace old equipment, for which running costs will no longer be required.

15 Comptroller and Auditor General, Major Projects Report 2015 and the Equipment Plan 2015 to 2025, Session 2015-16, HC 488-I, National Audit Office, October 2015.

16 HM Treasury, The Green Book of appraisal and evaluation in central government, 2013.

17 National Audit Office, Initiating successful projects, December 2011.

Part Three

Performance and impact

3.1 This Part examines:

the Department for Business, Innovation & Skills’ (BIS) performance in overseeing the delivery of projects to time and budget;

the performance of projects that are currently operational; and

how BIS evaluates whether projects are achieving scientific benefits or economic impacts.

Delivery performance

3.2 By their nature, science projects are cutting edge and will, in some cases, suffer cost and time overruns. Despite the technical risks involved, there are examples of complex projects that were delivered on time and within budget. Of the 10 projects we examined in 2007, 5 were delivered on budget including the HECToR supercomputer (£65 million). Three projects exceeded their original budgets by more than 10%. These included the Halley VI Antarctic research station (Figure 16). Of 20 subsequent projects which are now operational, none were delayed by more than a year and 16 were delivered within budget.

3.3 Specific issues have affected 3 projects in development:

The Francis Crick Institute for biomedical research was approved in 2011. It has received £330 million of capital funding from the Medical Research Council and a further £400 million from universities and charities for a new building in central London. The building was originally expected to be operational by August 2015 but, due to slower than expected progress in commissioning the building work, is not expected to be fully operational until the end of 2016. The Institute is expected to cost the research council around £42 million to run annually, but may face operational challenges if the closely located second Crossrail project interferes with its sensitive equipment, leading to inaccurate experimental results. The Institute and its funders have lodged objections to the proposed route.

Life Study, a study largely funded by the Economic and Social Research Council (ESRC) to collect and monitor data about babies throughout their lives, was approved in 2011. When it encountered challenges recruiting participants, ESRC decided to discontinue the study, which will now end in early 2016. Total sunk costs are still to be confirmed but expected to be around £9 million of the total planned investment of £33.5 million.

The Mega Amp Spherical Tokamak (MAST) upgrade project to enhance the UK’s role in international fusion research received £30 million of BIS funding. The upgrade was due to complete by early 2016 but there have been delays in assembling the infrastructure and costs have increased by £8 million (27%). The cost overruns on the project have led to funding cuts in the wider fusion research programme.

Operational performance

3.4 We examined the arrangements BIS and the research councils have in place to monitor operational performance and how operational projects have performed, including the extent that they are used.

Oversight arrangements

3.5 We found that there are a variety of arrangements for overseeing operational projects.

3.6 In some cases, research councils maintain close oversight over operational projects. For example, the Engineering and Physical Sciences Research Council receives quarterly information on the ARCHER supercomputer to allow it to assess whether the project’s contractors are meeting service requirements, while the Biotechnology and Biological Science Research Council oversees the performance of the institutes it funds by being represented on project boards.

3.7 In other cases, research councils have less direct involvement in monitoring whether the project is achieving the intended impacts and benefits. For example, the Engineering and Physical Sciences Research Council is not represented on boards established for overseeing the performance of the National Graphene Institute. The Natural Environment Research Council (NERC) had representatives on the Halley VI project board during the delivery stage of the project but, now that the research station is operational, the British Antarctic Survey is responsible for the day-to-day management of operations and we did not see any evidence that NERC monitors information on the operational performance and impact of the project.18

3.8 There is also variation in the type of information on performance that projects report. We found that while most of the operational projects monitor their performance against performance indicators, targets have not always been set for these indicators. In most cases, previous performance levels are used as a baseline against which current performance levels can be assessed (Figure 17). While in some cases measuring performance against a baseline may be appropriate, in other cases measuring performance against a target would be more informative.

Operational performance of projects

3.9 We examined the performance of those projects that have been operational for some time. Where possible, we compared current performance levels with past performance or with what the project team informed us was the desired level of performance. All but one of the projects we examined in our 2007 report are operational. Among these projects, most have been in high demand and are meeting performance expectations (Figure 18). Notable successes include the Diamond Light Source (Figure 19), the Laboratory of Molecular Biology, which has developed a new method for making human antibodies, and Royal Research Ship James Cook, which allows scientists to explore climate change.

3.10 Of more recent projects we examined, some are showing early signs of good performance while performance information is not yet available for other projects (Figure 20). For example:

in operation since November 2013, ARCHER is the UK’s most powerful computer, with 3.5 times the speed of HECToR, its predecessor. In its first year ARCHER met service level and user satisfaction targets. Utilisation rates are gradually increasing but were below optimal (71% across the year, against 80% to 85%) so the project team is making it easier for less experienced users to use the facility; and

the UK Data Service has been operational since 2012. It is a resource for social science researchers and policy-makers who need access to government surveys and census data. The service is well-used, with performance against indicators such as data downloads and website hits increasing.

3.11 Funding constraints have reduced the availability and reliability of facilities in a small number of cases. The ISIS neutron source is a particle accelerator used to study bio-materials and advanced materials. While the particle accelerator’s technical performance has been at the required levels since 2008, the availability of the facility for experiments has been below expectations. Even allowing for planned shutdowns, total scheduled user days have been around 120 days per year compared with an optimum of 180 user days per year (Figure 21). Some of the facility’s experimental stations were only partially available in 2014-15. The Science and Technology Facilities Council attributes the shortfall to insufficient funding for power and technical staff. The project team informed us that increased funding for 2015-16 has improved availability.

3.12 We found that the technical hub at the European Bioinformatics Institute is not being used to capacity in all areas. The hub has a number of functions including a training suite for running courses on using bioinformatics data. Demand for training outstrips supply, but funding constraints have affected the Institute’s capacity to use the training suite to deliver face-to-face training. On-site training has declined since 2012 while online training courses have increased. The Institute intends to request resources to expand on-site training further as part of future funding applications.

How BIS evaluates whether scientific and economic benefits are being achieved

3.13 Evaluating the operational performance and impact of science projects is challenging. The scientific, societal and economic returns from investment in science projects are often long-term and it is difficult to put a value on ground-breaking research. There are also costs associated with carrying out in-depth assessments of impacts achieved. However, taking a systematic approach to evaluating impact will help BIS assess whether projects are delivering what was expected, learn lessons and inform future spending decisions.

3.14 We found that BIS does not have a systematic approach to assessing whether operational projects are delivering benefits across its portfolio. While some research councils are strengthening how they assess whether projects realise their intended benefits, and research councils undertake a range of activities to assess impact, we identified opportunities for BIS to improve evaluation by taking a more consistent approach:

Out of 20 business cases we reviewed, 9 did not include a plan for realising benefits once the project was operational. If benefits realisation plans are developed at the outset, it will be easier to assess whether projects are delivering what was intended.

Many projects use case studies to evaluate the scientific impact. While case studies are a good way of illustrating specific successes, they may not always allow BIS to measure whether the project has achieved what it set out to.

Many projects are expected to lead to economic benefits but BIS does not always calculate the economic benefits arising. The Biotechnology and Biological Sciences Research Council’s analysis of the impact of its £137 million investment into the Babraham Research Institute and campus development suggests that it has helped to create 6,673 jobs and generated £298 million of Gross Value Added to the UK economy. There are plans to assess the economic impacts of the European Bioinformatics Institute and the UK Data Service. The HECToR supercomputer was subject to an impact assessment, helping make the case for its successor, ARCHER, but economic benefits were not quantified. While it may not always be proportionate or possible to carry out a full economic appraisal, in some cases an assessment will allow BIS to demonstrate that economic benefits are being realised and support the case for further investment.

BIS’s ‘Gateway 5 review’ is usually carried out only 1 year after the project becomes operational. The Gateway 5 review is intended to confirm that the desired benefits are being achieved. But while it provides an early indication of how projects are performing, the review may be too soon in a project’s lifetime to evaluate impact. Furthermore, it is unclear how rigidly the Gateway 5 process is applied. Of the 17 projects we examined, 3 have so far been subject to a Gateway 5 review.

3.15 BIS uses an economic model to project the long-term economic impact of its investment in science projects and to support the case for future investment. BIS assumes that the general rate of return for publicly funded scientific research will be 20%, based on an academic review published in 2014. BIS’s model takes into account project-specific factors such as the level and profile of government and private sector funding, but does not quantify the specific economic benefits that may arise due to the features of individual projects.

3.16 BIS’s model allows a calculation of the potential return from investing in a portfolio of projects and was accepted by HM Treasury as support for its 2015 Spending Review bid for capital science funding. However, it differs from a framework designed to evaluate large science infrastructure projects, which has been used to evaluate the economic impact of the Large Hadron Collider at CERN. BIS could draw on this approach in cases where an economic assessment of an individual project is needed.19

18 The British Antarctic Survey is an institute of NERC.

19 M Florio and E Sirtori, ‘The Evaluation of Science infrastructures: A Cost-Benefit Analysis Framework’, Working Paper n. 2014-10, Department of Economics, Management and Quantitative Methods, University of Milan, December 2014. Study sponsored by the European Investment Bank Institute as part of its University Research Sponsorship programme.

Appendix One

Our audit approach

1 We examined whether the Department for Business, Innovation & Skills (BIS) can demonstrate that it is achieving value for money from its investment in science projects. The report covered:

large facilities and other national and international capital projects managed by the research councils and UK Space Agency, including projects we examined in our 2007 report, as well as subsequent major projects managed by the research councils.20 We did not cover the research councils’ capital expenditure on smaller projects, minor upgrades, capital grants or their estates; and

projects managed by the Higher Education Funding Council for England (HEFCE) through its UK Research Partnership Investment Fund (UKRPIF).21 We did not cover the capital funding HEFCE allocates directly to higher education institutions for smaller research facilities or equipment.

2 Together, the expenditure within the scope of the study comprises just over half of BIS’s total annual capital science budget in 2014-15.

3 While other government departments (including the Ministry of Defence, the Department of Health, the Department for Environment, Food & Rural Affairs, the Department for Transport, the Department of Energy & Climate Change and the Department for International Development) also fund research to meet their own policy objectives, this report focuses on science projects funded by BIS through its partner organisations, in particular the research councils and HEFCE. Catapult centres managed by Innovate UK are outside the scope of the report. The merits of the scientific case underpinning science projects do not form part of this report.

4 We examined:

whether BIS has a long-term strategy for prioritising its investment in science capital projects that clearly articulates its objectives and future direction;

whether BIS has prioritised investment in the projects most likely to add value and which are consistent with its strategy; and

BIS’s evaluation of whether science capital projects are achieving anticipated benefits and delivering long-term impact.

5 Our audit approach is summarised in Figure 22. Our evidence base is described in Appendix Two.

20 We have broadly defined major capital projects as those exceeding a total of £2 million capital value.

21 The UKRPIF supports large-scale capital projects from higher education institutions with a track record of research excellence, providing that they secure at least double funding from co-investment sources (business, charities or endowments). Capital funding from UKRPIF varies between £10 million and £35 million for any individual project.

Appendix Two

Our evidence base

1 We reached our independent conclusions on whether the Department for Business, Innovation & Skills’ (BIS) investment in science projects delivers value for money following our analysis of evidence collected between May and November 2015. Our audit approach is outlined in Appendix One.

2 We identified whether BIS has a strategy for deciding its scientific capital spending priorities:

We drew on evidence from our previous work, namely our 2007 report Big Science: Public investment in large scientific facilities.22 This allowed us to understand BIS’s approach to prioritising capital spending on major scientific facilities until 2007.

We reviewed documents from BIS, the research councils and other published documents to understand how BIS has decided capital spending priorities from 2007 onwards. These included roadmaps published between 2007 and 2010, budget statements, long-term plans for science (for example, the ‘8 Great Technologies’), the research councils’ 2012 strategic framework for scientific capital investment, and strategies and roadmaps setting out investment priorities for the ‘8 Great Technologies’ (where available).

We reviewed the documents supporting the 2014 capital consultation to understand BIS’s approach to prioritising scientific capital spending since 2014. This included reviewing the documents that set out the capital consultation; the documents supporting the prioritisation exercise, including the scoring process against decision-making criteria and minutes of the meetings of the Ministerial Advisory Group; as well as the resulting 2014 capital roadmap and the 2014 science and innovation plan for growth.

We reviewed Higher Education Funding Council for England (HEFCE’s) documents supporting the appraisal and approval process of the UK Research Partnership Investment Fund (UKRPIF).

We carried out semi-structured interviews with BIS’s departmental officials and research councils’ representatives to help us understand the capital consultation process.

We spoke with representatives from stakeholder groups, namely the Royal Academy of Engineering and The Royal Society.

3 We assessed the quality of information supporting investment decisions. This comprised: (a) an examination of a sample of 20 business cases for major projects that were approved and funded between 2007 and 2015; and (b) the information that HEFCE used to support decision-making for its UKRPIF.

a We took the following approach to examining business cases supporting decisions on major projects:

In the absence of a comprehensive list of capital science projects funded by BIS and the research councils, we developed a list of projects with reference to a variety of sources including published roadmaps and announcements of new projects. We sought the views of the research councils on the accuracy and completeness of the list that we developed.

We used the list to select a sample of 20 projects (36% of the total number identified) which had been approved for funding since publication of our previous report in 2007. We selected the sample to cover a broad range of projects. Specifically, the sample included some projects that are already operational as well as projects that are not yet operational and were approved at various points between 2007 and 2015. We also ensured that our sample included a range of projects of different types and value, and covered projects managed by each research council and the UK Space Agency. For example, our sample included projects to build research institutes, upgrades to existing facilities, investments in survey instruments, supercomputers and ships, and participation in international as well as UK projects. We shared our sample with BIS and, for each project, requested the final business case supporting the investment decision. The 20 projects we selected are listed at Figure 10.

We examined whether the business cases contained key information, specifically considering: (1) whether alternative options were identified and assessed; (2) whether the costs of running the project once operational had been estimated and assessed as affordable; (3) whether the potential demand for the project had been considered; (4) whether a plan had been developed for measuring whether benefits had been realised; (5) whether the return on investment had been calculated; and (6) whether sensitivity analysis had been conducted on costs and benefits. We selected these categories with reference to HM Treasury’s Green Book and its supplementary guidance, which emphasises the importance of well-prepared business cases to support evidence-based decisions.23

For each category, we developed a set of criteria and ratings to assess and evaluate the completeness of information in each business case (Figure 23). The objective of rating the business cases was to give an indication of relative completeness and trends over time. We recognise that rating the business cases required judgement so to ensure that we assessed information in a consistent manner, we developed a standard template to capture the results of our examinations. This was followed by a moderation exercise where we reviewed the assessments for consistency. In addition, our assessment criteria and ratings have been subject to an internal quality assurance review.

b To review the information that HEFCE used to support decision-making for its UKRPIF, we selected a sample of 8 bids approved between 2012 and 2014. The sample included a range of projects, across funding rounds, of different types and scale. The projects were:

Cancer Research Development at the University of Manchester;

Multidisciplinary Characterisation Facility at the University of Manchester;

Graphene Engineering Innovation Centre at the University of Manchester;

Oxford Target Discovery Institute at the University of Oxford;

Big Data Institute at the University of Oxford;

National Automotive Innovation Campus at the University of Warwick;

Casting and Simulation Research Facility at the University of Birmingham;

Research and Translation Hub at Imperial West Technology Campus, Imperial College of London.

c We conducted semi-structured interviews with officials at BIS, research councils and HEFCE to support and triangulate the evidence collected from the review of business cases and bids for the UKRPIF.

4 We reviewed data on project delivery performance:

We compared actual project costs with the costs expected at project approval and actual delivery dates with planned delivery dates.

As the report focused on wider issues, while we reviewed the data we did not carry out a detailed validation of the data BIS reported.

5 We assessed how performance and impact of operating facilities are evaluated.

We analysed performance data for 17 operational facilities. These are listed in Figure 18 and Figure 20.

The sample was not selected to be representative of the full population. Instead, we wanted to cover most of the operational projects we reviewed in our 2007 report, plus a selection of subsequent operational projects. These additional projects included a range of projects of different types and size, and covered projects managed by each research council. We shared and discussed our selection with BIS and the research councils.

Where data were available, we examined performance over time against baseline and target.

We examined oversight and monitoring of performance and project delivery.

We examined whether impact has been assessed and how it has been done.

We carried out 7 site visits or phone interviews with facility managers to supplement our analysis of performance data. This gave us views on the possible reasons for variations between actual and expected performance, as well as a better understanding of the facilities’ oversight and monitoring process.

We conducted semi-structured interviews with BIS analysts to understand BIS’s approach to assessing the impact and value of its investments.

We reviewed published literature on how other organisations, in the UK and overseas, assess long-term scientific and economic impacts.

We conducted a rapid evidence assessment of the published literature, which followed the same structure of a systematic literature review but made use of a defined list of inclusion and exclusion criteria to make it less resource-intensive.

We contrasted BIS’s approach to assessing long-term impacts with the European Investment Bank-funded framework for assessing investments in large-scale research projects. This case example was selected because it: (a) provided information about the types of benefits and costs relevant to assessing research facilities; (b) offered guidance on how to value costs and benefits; and (c) could be used as a tool for evaluating research facilities already in operation.

22 Comptroller and Auditor General, Big Science: Public investment in large scientific facilities, Session 2006-07, HC 153, National Audit Office, January 2007.

23 HM Treasury, Delivering public value from spending proposals, 2013.

Appendix Three

Scope of the review

1 This appendix provides a list of all projects identified as being within the scope of the study. It provides details of planned and actual capital investment, planned and actual year of operation, and annual operating costs where applicable.

2 The following figures include data on the projects we examined in 2007 (Figure 24), data on subsequent capital projects funded by the Department for Business, Innovation & Skills (BIS) or its partner organisations (Figure 25), data on international projects where BIS has provided or committed to provide capital funding (Figure 26), and the Higher Education Funding Council for England’s (HEFCE’s) capital investment in the UK Research Partnership Investment Fund (Figure 27). Project information was provided by BIS and the research councils during the course of our study and confirmed as accurate as at February 2016.

Design and Production by NAO Communications
DP Ref: 10905-001