UK Research and Innovation’s management of the Industrial Strategy Challenge Fund

UK Research and Innovation
and the Department for Business,
Energy & Industrial Strategy

REPORT

by the Comptroller
and Auditor General

Session 2019–2021

5 February 2021

HC 1130

The National Audit Office (NAO) scrutinises public spending for Parliament and is independent of government and the civil service. We help Parliament hold government to account and we use our insights to help people who manage and govern public bodies improve public services.

The Comptroller and Auditor General (C&AG), Gareth Davies, is an Officer of the House of Commons and leads the NAO. We audit the financial accounts of departments and other public bodies. We also examine and report on the value for money of how public money has been spent.

In 2019, the NAO’s work led to a positive financial impact through reduced costs, improved service delivery, or other benefits to citizens, of £1.1 billion.

We are the UK’s independent public spending watchdog.

We support Parliament in holding government to account and we help improve public services through our high-quality audits.

UK Research and Innovation’s management of the Industrial Strategy Challenge Fund

UK Research and Innovation
and the Department for Business,
Energy & Industrial Strategy

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 3 February 2021

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Gareth Davies
Comptroller and Auditor General
National Audit Office

2 February 2021

© National Audit Office 2021

Value for money reports

Our value for money reports examine government expenditure in order to form a judgement on whether value for money has been achieved. We also make recommendations to public bodies on how to improve public services.

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.org.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

008001 02/21 NAO

This report can be found on the National Audit Office website at www.nao.org.uk

If you need a version of this report in an alternative format for accessibility reasons, or any of the figures in a different format, contact the NAO at enquiries@nao.org.uk

The National Audit Office study team consisted of:

Rachel Burden, Terry Caulfield, Emily Stanyon and Anna Sydorak-Tomczyk, under the direction of Peter Gray.

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

020 7798 7400

www.nao.org.uk

@NAOorguk

Key facts

Summary

1 The Department for Business, Energy & Industrial Strategy (the Department) has overall responsibility for the government’s spending on science, research and innovation. One of its objectives is to deliver the UK’s Industrial Strategy by, among other things, promoting investment in science, research and innovation to make sure the government’s ambition of the UK becoming the ‘most innovative country’ is fulfilled.

2 Established by the Department, the Industrial Strategy Challenge Fund (the Fund) supports the aim set out in the government’s Industrial Strategy to raise long-term productivity and living standards. An “industry-led” programme, announced in late 2016 and funded from the National Productivity Investment Fund (NPIF) introduced by HM Treasury, the Fund aims to ‘tackle […] major industrial and societal challenges’ through supporting four ‘grand challenges’ outlined in the Industrial Strategy (future mobility; clean growth; artificial intelligence and data; and the ageing society). The Fund contributes to the government’s aim for the UK to spend 2.4% of its gross domestic product on research and development (R&D). Recently, the government has looked to the Fund to contribute to its aim to achieve net zero carbon emissions by 2050. The government is also considering how its ‘levelling up’ agenda – whereby it aims to create opportunity for everyone in all regions and address disparities in economic and social outcomes – will apply to R&D and innovation spending across the UK.

3 The government’s approach to distributing the Fund has changed since it was introduced. Most recently, UK Research and Innovation (UKRI), a non-departmental public body reporting to the Department and which is responsible for the Fund, has invited potential bidders from business and academia to identify important societal and industrial ‘challenges’ faced by the UK that might merit financial support from the Fund. Each challenge is intended to contribute to one of the four grand challenges identified in the Industrial Strategy. Once a challenge is approved by ministers, organisations are invited to bid for individual projects that will contribute to that challenge. Those bidding need to demonstrate they can contribute their own funds to the project - known as co-investment - alongside public funding. The selection process, as it now exists, is outlined in Figure 1. By 2019-20, UKRI and the Department had overseen three rounds of funding, known as Waves 1, 2 and 3.1

4 Current challenges range from supporting the UK’s development of low-carbon technologies, to looking to support the better detection of disease and to identify new ways to tackle cyber security threats. Individual projects within the challenges range from the mass production of vaccines, support for new approaches to constructing houses, the development of batteries required for electric vehicles and the establishment of a national satellite test facility.

5 The Department has set UKRI five objectives for the Fund, to:

increase UK businesses’ investment in R&D, while also improving R&D capability, capacity and technology adoption;

increase multi- and inter-disciplinary research;

increase engagement between academia and industry on targeted innovation activities;

increase collaboration between new small companies and those that are established; and

increase overseas investment in R&D in the UK.

6 By January 2021, the Fund was supporting 1,613 projects contributing to one of the 24 approved challenges, each linked to one of the four grand challenges.2 UKRI has so far spent around £1.2 billion of the eight-year budget of £3.0 billion funding projects. To date, industry has contributed £567 million against the Fund’s co-investment target of £2.8 billion. UKRI currently forecasts it will meet this target over the life of the current challenges.

7 UKRI, established in 2018 (bringing together the seven research councils, Innovate UK and Research England), has overall responsibility for the Fund. The Department scrutinises the affordability of proposals and then approves spending from the Fund. It also advises ministers on policy alignment between Fund challenges and Departmental objectives such as the Industrial Strategy. HM Treasury scrutinises and approves, from a value for money perspective, business cases (a condition of funding from the NPIF).

Scope of this report

8 The Fund has a number of characteristics which can make the assessment of its value for money challenging. It is looking to support innovative projects, some of which by their nature will fail; the impact of its support may only become obvious over the long term; and it is looking to contribute to a broad range of objectives. Regardless of these challenges, we consider that there are certain key elements which need to be in place to support the achievement of value for money – clear objectives, an approach which aligns resources with the achievement of these objectives, and consideration of progress and performance.

9 This report examines whether the Fund has been set up in a way likely to optimise value for money. The report examines:

the establishment of the Fund, in particular whether it has attracted sufficient good-quality bids, whether the selection processes have been efficient and whether the budget is managed effectively (Parts One and Two); and

the approach to monitoring and evaluating the Fund’s performance, as well as its performance to date (Part Three).

10 Full details of our scope and audit approach are set out in Appendices One and Two.

Key findings

Establishment of the Fund

11 The Fund is an ambitious programme, and government is looking for it to contribute to an increasing number of objectives. The Fund contributes to addressing the government’s four grand challenges set out in its Industrial Strategy. To do so, it has established 24 challenges, each of which has a set of objectives. Under these, UKRI is funding and then maintaining oversight of 1,613 projects. The government is also looking to the Fund to contribute to its target for the UK to spend 2.4% of its gross domestic product on R&D and, more recently, to its ambitions around net zero. It is also considering the Fund’s role in contributing to its ‘levelling up’ agenda (paragraphs 1.5 to 1.9 and Figures 2, 3 and 17).

12 The Department and UKRI worked quickly to establish the Fund and attract sufficient interest from bidders. Since 2017, UKRI has received almost 2,700 bids for 61 competitions for project funding across 16 challenges. Of these, about one in four bidders – 699 – were successful. Almost 60% of the competitions received at least two bids for every project awarded funding. We found that, over time, UKRI has developed its approach to how it identifies challenges to focus more on the needs of industry and academia. Stakeholders we interviewed, including applicants and other organisations involved in promoting investment in R&D and innovation, were positive about, for example, the support provided to industry through the Fund (paragraphs 2.2 to 2.6 and Figures 5 and 6).

13 The Fund has succeeded in attracting winning bids from small and micro companies although larger companies accounted for a growing proportion of projects in the latest funding Wave. Small and micro companies accounted for more than 40% of the project awards in both of the first two Waves. The third funding Wave, however, which started in 2019-20, saw a rapid increase in the proportion of projects awarded to large companies (from 20% to 29%) compared with the second funding Wave, and a rapid fall (from 44% to 31%), in the number of small- and micro-businesses winning funding. UKRI has found no evidence of in-built bias towards larger companies during the selection process. A number of factors are likely to be impacting on the number of smaller companies applying, including their awareness of the Fund, the time and effort required to apply and the requirement – increased for the third funding Wave – to bring co-investment (paragraphs 2.7 to 2.9 and Figure 7).

14 Funding awards have been distributed unevenly across the country. By the end of March 2020 just over 63% of the funding awarded had gone to organisations registered in London, the South East and West Midlands. Of the total funding awarded, 44% had gone to organisations registered in London and the South East, mainly in the health and life sciences sector. The West Midlands had attracted another 19% of the awarded funding, mainly in the manufacturing and materials sector. Our analysis suggested that the geographical distribution of funding was not explained by the distribution of businesses undertaking R&D activities in the economy. To date, UKRI has not had an explicit objective to consider the regional balance in its awards. In July 2020, the government stated that it was considering how spending on R&D and innovation should contribute to its ‘levelling up’ agenda (paragraphs 2.10 to 2.12 and Figures 8 and 9).

15 Lengthy processes for agreeing challenges and then projects leads to delays in funding projects. A balance needs to be struck between making sure proposals for challenges and then bids from prospective grant recipients are of sufficient quality and approved quickly. Overly long processes might delay the impact of projects and might deter applicants. We identified lengthy approvals processes at both key stages of the Fund – selecting and approving challenges and then selecting and approving projects (paragraphs 2.15 to 2.19 and Figures 1 and 10). For the third Wave of funding, for example:

it took UKRI, the Department and HM Treasury 72 weeks to move from identifying ideas for new challenges (expression of interest stage) to the approval of those challenges; and

UKRI then took on average 31 weeks to assess applications for project funding and then make an offer for funding to the applicant.

16 Grant recipients and stakeholders consulted by us consistently identified the lengthy approvals process as a potential deterrent to prospective applicants. A number of factors contributed to the time taken to award funding, including capacity constraints in UKRI and a need in the early stages of the Fund to improve the quality of business cases submitted by UKRI to support ideas for new challenges. However, lengthy approval processes at both official and ministerial level in the Department and HM Treasury, carried out in sequence, have added to the time taken. Businesses and representative bodies told us they had concerns about the impact of these elongated approval processes (paragraphs 2.13 to 2.19 and Figure 10).

17 UKRI has faced difficulty recruiting staff to help administer the Fund. Having fewer staff than UKRI estimates is required to manage the Fund – could have an impact on a number of areas - such as its oversight and evaluation of performance. In 2019, UKRI estimated that its staffing to administer the second funding Wave was 40% under capacity. For the third Wave this had deteriorated further to around 60%, equating to approximately 42 full-time equivalents. More recently, UKRI informed us that the shortfall had improved with eight positions covering the Wave 2 and 3 challenges still vacant in January 2021. The appointment of Challenge Directors within UKRI is critical to setting the direction for and then successfully implementing each challenge. Since the start of the Fund, it took on average over 37 weeks to appoint permanent Challenge Directors for the duration of the challenge, although in 2019-20 this had shortened to 24 weeks.3 UKRI informed us that the delays were partly due to the difficulty of finding and hiring staff at an appropriate level who have a mix of science and industry experience. The recruitment is not delivering diversity at Challenge Director level, which is important given that diversity of thinking is likely to be important to the success of a Fund seeking to encourage innovative ideas. Out of 20 Challenge Directors at the end of August 2020, three were female (paragraphs 2.20 to 2.22 and Figure 11).

18 UKRI is underspending against the Fund budget, which may put pressure on future years’ budgets. In total, by March 2020 UKRI had spent £1,024 million against the overall Fund budget to that point of £1,146 million (an 11% underspend). Delays in getting new challenges approved and up and running has had a knock-on impact on UKRI’s ability to start spending on newly approved challenges. These delays are likely to push spending into the following years as commitments build up. This pressure, exacerbated by the impact of COVID-19 which will have slowed progress on some approved projects, may increase the risk of UKRI having to re-profile its future spending to fit within its annual budgets. UKRI and the Department are currently reviewing the Fund’s multi-year budget profile (paragraphs 2.26 to 2.30 and Figure 12).

19 UKRI and the Department reacted quickly to support COVID-19 related R&D whilst supporting other projects unable to progress as planned. In May 2020, the government announced that the Vaccines Manufacturing and Innovation Centre funded by the Fund would receive a further £93 million to accelerate its construction and to make sure a vaccine could be produced quickly and in large quantities. At the same time, UKRI agreed to re-profile £165 million from the current budget into future years for 20 challenges. Some businesses, particularly small- and medium-sized enterprises, struggled to meet the co-investment requirements due to the impact of COVID-19 on their operations. Some of those businesses have had to re-scope or pause work on their funded projects (paragraphs 2.31 and 2.32).

Monitoring performance

20 There is currently no clear link between the objectives set for the Fund and the performance of the Fund’s projects, making it more difficult for UKRI to measure the long-term impact of the Fund as a whole. The five Fund objectives, set by the Department, focus on who receives support from the Fund, such as business and academic bodies, rather than the impacts projects may deliver (which are set at a challenge level). While UKRI reports performance against Fund objectives, no expectations or baselines were set at the start and reporting at Fund level does not focus on impact. UKRI is looking to enhance its evaluation of the Fund’s performance against its objectives and to consider its wider social and economic impact. UKRI has, for each challenge, established ‘benefit maps’, which set out in detail the outcomes that the challenge is intended to deliver as well as how the projects supported by the challenge will contribute. It is currently piloting its approach to collecting performance data in this area for two challenges, with the aim of having this approach in place for all challenges by April 2022 (paragraphs 3.9 to 3.12).

21 UKRI’s assessment of performance suggests projects and challenges are performing well although there has been some fluctuation over the last six months. Challenge Directors’ most recent assessment of performance (as at January 2021) put 16 of the 24 challenges in the top two categories (meaning that performance was either in line with expectations or an issue which might have a negative impact on the challenge had been identified but could be mitigated). In June 2020, their assessment was that 12 challenges were performing at that level; by September 2020 this figure was 19. The fluctuating assessments over recent months have reflected, for example, the impact of COVID-19 on individual challenges. UKRI also monitors project progress. Considered against six criteria, such as cost and risk management, projects have consistently been classified as being on track to deliver their planned objectives, or having some scope for improvement to make sure that all objectives would be met. However, we have identified some limitations in UKRI’s performance data (paragraphs 3.3 and 3.4).

22 UKRI’s ability to generate meaningful performance information on the Fund’s projects has, until June 2020, been hindered by its reliance on systems inherited from its predecessors. Until June 2020, UKRI was reliant on the manual compilation of data drawn from these systems, leading to sometimes incomplete and untimely management information which places constraints on its ability to monitor project performance efficiently and in a consistent manner. UKRI now has access to more timely information that draws directly from these different systems. Even so, the research councils monitor projects using different processes and systems compared to Innovate UK. For projects initiated by the research councils, some of the data are either not included at all in the routine management information or still have to be added manually (paragraph 3.5).

Conclusion

23 In the three years since the Fund – £3.0 billion of support to industry and academia to help solve the economy’s and society’s most complex issues – was introduced, UKRI and the Department have worked well to generate interest from industry and academia. Over that period, government has enhanced its engagement with industry to seek out challenges which might benefit most from taxpayer support.

24 UKRI’s own assessment shows that the Fund’s key components – challenges and projects – are broadly performing well. To sustain this position, the Department and HM Treasury, working with UKRI, need to place more emphasis on the outcomes and impact its funding secures at the Fund level. The increasing number of challenges supported by the Fund, each with their own objectives, and range of different objectives at Fund level risk obscuring priorities and will make the assessment of value for money in the longer term more difficult. UKRI, the Department and HM Treasury need to look again at the drawn-out process for selecting and approving challenges and projects, to ensure that good applicants are not deterred from putting forward bids.

Recommendations

25 We have identified a number of areas where UKRI has sought to learn lessons and improve its approach. For example, externally it has changed how it identifies challenges so that it is more focused on the needs of industry and academia. Internally, it has taken steps to develop more timely and accurate management information from the bodies which were brought together into UKRI. And its approach to performance assessment is evolving to consider the impact the projects it funds are achieving.

26 To support the next steps in the Fund’s development we recommend:

a UKRI, working with the Department and HM Treasury, should re-examine the arrangements for measuring the performance of the Fund, in particular whether the array of current objectives provide a coherent direction and have sufficient focus on the impacts to be delivered from the money spent;

b UKRI, working with the departments, should streamline the current arrangements for selecting new challenges to help shorten the time taken to arrive at decisions. Lengthy approval timescales can have practical implications for applicants hoping for support, UKRI’s ability to manage its budget effectively, and ultimately the delivery of impacts. UKRI and the departments should assess whether the right staffing capacity is in place in UKRI at the times needed and whether the approval processes in the departments could be shortened without undermining the rigour of the exercise;

c In line with the government’s ambition for R&D to support the wider ‘levelling up’ agenda, UKRI should examine the factors that may be driving the current geographical distribution of funding. This might include issues such as awareness of the Fund across the UK, and the intensity of involvement in the Fund across the industrial sectors known to be investing in R&D and innovation, for example, arising through the choice of challenge funds; and

d UKRI should review the impact of changes in its conditions for co-investment funding on the size of business applying for support through the Fund.

1 Wave 1 was funded in two stages (Wave 1a and 1b). In Wave 1a, to get the Fund up and running, the research councils and Innovate UK identified fundable projects from recently completed competitions which matched Industrial Strategy ambitions. Wave 1b followed the challenge structure used in Waves 2 and 3. Due to the lack of readily available data on Wave 1a projects these have not been included in some of our analyses.

2 The Industrial Strategy Challenge Fund is currently made up of what UK Research and Innovation (UKRI) describes as 21 challenges and three programmes. Programmes do not follow the challenge model so do not have a Challenge Director and are monitored and evaluated differently. This is due to the fact that in Wave 1, UKRI’s predecessors were required to spend the money quickly and provided funding to investment-ready programmes. We refer to both as challenges in this report.

3 This analysis looked at the time taken until the permanent Challenge Director started in their role at UKRI. It includes the time taken for recruitment and the notice period for the appointee when leaving their previous role (if relevant).

Part One

Background

1.1 This Part of the report considers:

the government’s main aim for research and development (R&D) and innovation;

the purpose of the Industrial Strategy Challenge Fund (the Fund); and

the governance arrangements for the Fund.

The government’s aim for R&D and innovation

1.2 The Department for Business, Energy & Industrial Strategy (the Department) has overall responsibility for the government’s spending on science, research and innovation. The Department is taking forward the government’s Industrial Strategy, Building a Britain fit for the future,4 published in 2017, which set out a plan to improve productivity and shape the economy after exiting the EU.

1.3 One of the Department’s key priorities has been to focus on solving what the government has described as the four grand challenges facing society. These are set out in the Industrial Strategy.5

Future of mobility – focusing on how people, goods and services move around the country, identifying ways, for example, to improve the efficiency in the way these are moved leading to reductions in pollution and congestion.

Clean growth – supporting UK industry’s move to clean growth through the development, manufacture and use of low-carbon technologies, systems and services that cost less than high-carbon alternatives.

Artificial intelligence and data – artificial intelligence can help industry and others use datasets to identify better ways of doing complex tasks – such as helping doctors diagnose medical conditions more effectively.

Ageing society – an ageing population creates new demand for technologies, products and services, including new care technologies and new housing models.

1.4 The Department plans to address these issues by:

increasing productivity and improving lives by tackling the UK’s grand challenges in life sciences, artificial intelligence, automation and space;

making the UK a science superpower through backing ideas and supporting talent from home and abroad; and

investing in R&D and innovation to drive discovery and unleash potential.

1.5 The government has a target to increase the UK’s public and private investment in R&D to 2.4% of gross domestic product by 2027.6 To contribute to this, the government established the National Productivity Investment Fund (NPIF). The government uses the NPIF to target areas that it considers critical to improving productivity – for example, transport, housing, digital infrastructure, and R&D. The Department has been allocated £7.1 billion from the NPIF over the period 2017-18 to 2021-22 to “enhance the UK’s position as a world leader in science and innovation and improve productivity and growth through the Industrial Strategy”.

The Industrial Strategy Challenge Fund

1.6 The government introduced the Industrial Strategy Challenge Fund (the Fund) at the end of 2016 to raise long-term productivity and growth of jobs. Potential bidders from business and academia were invited to identify challenges that might merit financial support from the Fund. The Department expected these challenges to contribute to one of the Industrial Strategy’s grand challenges. Ideas for new challenges were considered by UK Research and Innovation (UKRI) and the Department against the Fund’s objectives and their alignment with the grand challenges. If a challenge idea was selected, UKRI, a non-departmental public body sponsored by the Department, prepared a challenge business case for approval by HM Treasury. When the business case received approval UKRI proceeded to invite bids for specific projects. Successful bidders received funding and other support to carry forward each project. The Department earmarked £3.0 billion from its NPIF allocation to the Fund.

1.7 The Department has given UKRI five objectives for the Fund to:

increase UK businesses’ investment in R&D, while also improving R&D capability, capacity and technology adoption;

increase multi- and inter-disciplinary research;

increase engagement between academia and industry on targeted innovation activities;

increase collaboration between new, small companies and those that are established; and

to increase overseas investment in R&D in the UK.

In Part Three of this report, we look at UKRI’s performance against these objectives as well as its assessment of how well projects and challenges are performing.

1.8 In July 2020, the government announced that, alongside the four grand challenges, the Fund would also contribute to ambitions such as achieving net zero carbon emissions by 2050. At the same time, the government announced that it was considering how R&D and innovation spending could contribute to its ‘levelling up’ agenda – whereby it aims to create opportunity for everyone in all regions and to address disparities in economic and social outcomes – as it looks to develop a new R&D strategy focused on local growth.

1.9 As at January 2021, UKRI had established 24 challenges (Figure 2, with Figure 3 providing examples of the objectives underpinning some of the challenges).7 The Fund had provided support to 1,613 projects. Projects range widely in size and levels of funding, from £24,000 for a small feasibility study to explore the potential for simulated experiences through virtual reality to £111 million for a battery industrialisation centre in Coventry. Figure 4 provides further detail on some of the supported projects.

1.10 By January 2021, UKRI had invested £1.2 billion in projects over three years using the Fund. In addition, these projects have to date contributed £567 million of co-investment from industry against its co-investment target of £2.8 billion. UKRI has forecast that industry will realise almost £3 billion in co-investment over the life of the current challenges.

Oversight of the Fund

1.11 There are three public bodies involved in oversight of the Fund:

UKRI is the Department’s arm-length body responsible for the Fund’s implementation and performance, and for its engagement with the Department. Its chief executive officer advises the Department and ministers on new challenges to support the Fund and is accountable for its operation. A Steering Board (described in more detail at paragraph 1.13) oversees the Fund’s operations.

The Department has overall responsibility for the government’s expenditure on science, research and innovation. It scrutinises and approves spending from the Fund and ensures policy alignment between Fund challenges and Departmental objectives such as the Industrial Strategy.

HM Treasury scrutinises and approves, from a value for money perspective, spending as part of the NPIF. As part of the HM Treasury requirement for NPIF programmes, it and the Department approved the business case for the Fund, including challenges selected, to ensure they are aligned with wider government priorities. Individual challenges required individual business cases in line with HM Treasury’s Managing Public Money guidance.

1.12 The Fund was initially administered directly by Innovate UK and seven research councils, with the Department selecting the projects under Wave 1a to fund, pending the creation of UKRI in 2018 (Figure 5). UKRI brings together the research councils, Innovate UK and Research England. UKRI was established to invest in and facilitate research and innovation activities across the UK and, through Research England, to support higher education providers in England to carry out research and knowledge exchange activities. In 2019-20, UKRI spent £7.8 billion, 55% of the Department’s total expenditure of £14.1 billion, of which £510 million was via the Fund. The remaining £7.3 billion was invested in other R&D programmes, supporting science infrastructure, funding higher education institutions and overseeing other NPIF programmes.

1.13 The Department requires UKRI to have in place governance and control mechanisms for the Fund. UKRI has appointed a Steering Board for the Fund comprising key UKRI personnel, including the executive chairs of the research councils (where they are also the senior responsible officers for a challenge) and Innovate UK, and senior officials from the Department and HM Treasury. The Steering Board oversees the performance of the Fund, in part by considering the performance of projects and of challenges and then recommending appropriate action in the light of this information. The Steering Board also considers progress against its five objectives. Alongside considering performance the Steering Board takes strategic decisions on the delivery of individual challenges, including consideration of certain aspects of the business case and programmes. It is responsible for approving major changes to the budget and is accountable to the UKRI Board.

4 HM Government, Industrial Strategy, Building a Britain Fit for the Future, white paper CM 9528, November 2017.

5 In addition to the four grand challenges, the Industrial Strategy focuses on what are described as five foundations of productivity: ideas, people, infrastructure, business environment and places.

6 In 2018, the public sector contributed 30% and the private sector (including private non-profit bodies) 70% to total research and development spend.

7 The Fund is currently made up of what UKRI describe as 21 challenges and three programmes. Programmes do not follow the challenge model so do not have a Challenge Director and are monitored and evaluated differently. We refer to the programmes as challenges in this report.

Part Two

The establishment of the Industrial Strategy Challenge Fund

2.1 This Part examines whether UK Research and Innovation (UKRI) and the Department for Business, Energy & Industrial Strategy (the Department) have:

attracted sufficient competitive interest in the Industrial Strategy Challenge Fund (the Fund);

managed the selection process efficiently; and

managed the budget for the Fund effectively.

Attracting good-quality bids

2.2 The success of the Fund in meeting the objectives set by government depends crucially on UKRI’s ability to attract sufficient high-quality bids for project funding to enable effective competition across the full range of challenges. Competition means that UKRI has a choice of projects to fund, which should help support value for money. We examined the extent to which the Fund is now attracting interest from bidders, the views of bidders on the challenge process and the diversity of applicant organisations receiving funding.

2.3 The Department and UKRI were under pressure to get the Fund up and running very quickly. When the Fund was announced in November 2016, UKRI had yet to be established. The Department was asked by ministers to get the Fund operational for the 2017-18 financial year (the first Wave of funding). The Department decided to support projects which met the Fund’s criteria, which could be started quickly, and which could commit to co-investment funding.8

2.4 Government’s approach to approving challenges and then projects has evolved between each Wave of funding. For the second and third Waves of funding starting in 2018-19 and 2019-20, UKRI and the Department significantly developed their approach, drawing upon feedback from stakeholders within and outside of government. For the second Wave of funding, UKRI and the Department identified the challenges they would support; later on they held public competitions to help identify additional challenges. For the third Wave of funding, they asked for expressions of interest from industry and academia on potential new challenges, to add to those already approved in the second Wave. For both Waves, industry and academia were able to bid through a competitive process for funding under each new challenge. Figure 6 provides an overview of the key differences between the three Waves of funding.

2.5 Some of the stakeholders we consulted – successful applicants and representative bodies from industry – were positive about, for example, UKRI’s communication and the support to industry it provided through the Fund, as well as the encouragement it gave to interdisciplinary working. Successful applicants suggested the Fund had supported projects which might not otherwise have taken place. While applicants were positive, some voiced concerns about the length of time taken to make funding awards and some had concerns about the ability of smaller companies to meet the co-investment requirements set for funding.

2.6 To support the selection of good quality projects, UKRI assesses each bid against criteria such as the project’s relevance to the Fund’s objectives and takes into account the views of the relevant Challenge Director. Decisions are made in the context of UKRI’s focus on supporting innovative projects. Since 2017, UKRI has received almost 2,700 bids for 61 competitions for project funding across 16 challenges.9 Of these, about one in four – 699 – were successful. Each of the 61 competitions received at least one bid, and almost 60% had at least two bids for every project awarded funding and 8% of challenges had 10 or more bids for each successful project.

Attracting bids from small companies

2.7 Our examination of winning bids for projects suggested that UKRI initially succeeded in attracting a range of different-sized companies to participate, although recent data suggest that larger companies now account for a growing proportion of funded projects. One of the Fund objectives (see paragraph 1.7) is to increase collaboration between younger, smaller companies and larger more-established companies, putting an emphasis on funding micro-, small- and medium-sized enterprises. Our analysis of UKRI’s data suggests that small and micro companies have accounted for more than 40% of project awards in both of the first two Waves (Figure 7). The third Wave, however, saw a rapid increase in the proportion of projects awarded to large companies and a rapid fall in the number of small- and micro-businesses winning funding.

2.8 UKRI has been examining trends in the size of company applying for and winning funding.

In January 2020, UKRI found the proportion of projects associated with small- and medium-sized enterprises was 5% lower for projects funded through the Fund than those supported by other UKRI programmes.

In October 2020, UKRI looked at the characteristics of those applying for funding. The analysis showed that the proportion of applications from large businesses increased for Wave 3 compared with previous Waves. At the same time, the proportion of small- and micro-sized businesses applying decreased for Wave 3. UKRI compared the size of companies making applications with those receiving funding and found no difference in the distribution. It concluded that there was no evidence of an in-built bias in favour of larger companies during the selection process.

2.9 We identified several reasons why smaller companies may find it challenging to apply for funding from the Fund. For example, in Wave 3, responding to a requirement from the Secretary of State for Business, Energy & Industrial Strategy, UKRI increased the co-investment requirement from industry (the ratio of public investment to private investment increased from 1:0.45 in Wave 1 to 1:1.5 in Wave 3). This change may have made it more challenging for smaller and medium-sized enterprises with smaller budgets to be involved. In addition we identified two more general impacts.

The application process. Some fund recipients we interviewed told us that the application process for funding was lengthy. This may particularly affect small- and medium-sized enterprises which have less resource to put into the application process and, for example, may be unfamiliar with what is expected of an application.

Awareness of the Fund. Stakeholders informed us that small- and medium-sized enterprises may not be aware of the funding available from UKRI, potentially limiting the number of small and micro businesses applying.

Attracting bids from across the UK

2.10 In July 2020 the government stated its intention that spending on research and development (R&D) and innovation should contribute to its ‘levelling up’ agenda. Prior to this UKRI was not given an explicit objective on how to consider the regional balance in its awards. However, the Industrial Strategy did include a focus on what it described as “place” – “prosperous communities across the UK”. We examined the extent to which the Fund is attracting winning bids from across the UK.

2.11 Our analysis of funding awards made since the start of the Fund and October 2020 suggested that awards had been concentrated in London, the South East and West Midlands (Figure 8). Just over 44% of the total funding awarded had gone to organisations registered in London and the South East, mainly in the health and life science sector. The West Midlands had attracted another 19% of the awarded funding, mainly in the manufacturing and materials sector. We have previously reported on the concentration of R&D funding going to particular parts of the UK. In a 2013 memorandum for the House of Commons Science and Technology Committee, we reported that R&D activity was concentrated in three regions – London, the South East and the East of England – which together accounted for £14 billion of activity in 2011 (52% of the total R&D carried out in the UK).

2.12 We completed further analysis to look at how the amount awarded to projects in each region compared with that region’s population and the number of claims for R&D tax credits (as a proxy for the amount of R&D activity). Our analysis found that the geographical distribution of funding was not necessarily explained by the population or distribution of businesses undertaking R&D activities (Figure 9).

The average amount of funding per capita was £19, and ranged from £5 in Yorkshire and the Humber to £47 in the West Midlands.

The average amount of funding per R&D tax credit claim was £23 and ranged from £6 in Yorkshire and the Humber to £51 in the West Midlands.

The efficient management of the selection process

2.13 While stakeholders we interviewed were positive about many aspects of the challenge process a number raised concerns about the time taken to ultimately approve projects. A number suggested that an elongated timetable for approvals is not aligned with supporting innovative projects and has had a negative impact on a project’s ability to start. It also made planning less certain, and may have deterred some from bidding. We examined the extent to which the bidding processes were being managed efficiently.

2.14 The bidding and selection processes followed a number of stages.

Expressions of interest. At the start of Wave 2 and Wave 3, expressions of interest were sought for challenges to include in the Fund.10 These challenges were intended to be issues, linked to the grand challenges, that the bidders wished to address but which, in the absence of public money, might not otherwise receive investment.

Shortlisting of challenges by UKRI. UKRI selected, against criteria linked to the grand challenge, a shortlist of potential challenges for support.

Shortlist of challenges approved by the Department. UKRI discussed the shortlisted challenges with officials from the Department and ministers. The Secretary of State decided on which challenges to approve after considering UKRI recommendations.

Preparation of a Wave level business case by UKRI and approval by HM Treasury and the Department. Once the challenges were shortlisted, UKRI produced a Wave level business case and submitted it to the Department and subsequently to HM Treasury for approval. This business case included details of the Fund’s objectives, the rationale for funding, information on the shortlisted challenges, a target for co-investment and a plan for management, monitoring and evaluation.

Approval of challenge level business cases. For Waves 2 and 3, the Department delegated the authority to approve individual business cases for challenges to UKRI, with a few exceptions. These were also approved by HM Treasury in accordance with National Productivity Investment Fund (NPIF) rules.

Running competitions to identify projects. Once the stages above were complete, UKRI could begin competitions for projects within each challenge and grant funding awards.

2.15 Figure 10 shows that it took 43 weeks for Wave 2 and 72 weeks for Wave 3 to move from the expressions of interest stage to the approval of Wave business cases.11 The business cases for individual challenges were approved either at the same time as the Wave business case or shortly afterwards with a few exceptions. Funding for individual projects can only begin once the challenge business case is approved.

2.16 The length of time taken to complete the selection processes increased from Wave 2 to 3. The time allowed for applicants to submit ideas for challenges accounted for three and seven weeks of the total time for approval respectively for Waves 2 and 3. Officials in UKRI and the Department identified a number of factors holding up the processes.

Quality of business cases. The Wave 2 business case (see paragraph 2.14) prepared by UKRI required additional work before the Department and HM Treasury would approve it. UKRI was required to improve the value for money case; be more specific about the objectives the challenge was seeking to achieve and the proposed approach to monitoring and evaluation; and to establish the link between the proposal and other government initiatives. For Wave 3 the Department and HM Treasury reported an improvement in the quality of business cases submitted for approval. UKRI informed us it recognised further improvements were needed.

Processes for engaging with stakeholders. In Wave 3, at the request of the Secretary of State, UKRI sought to increase its active engagement with industry to identify challenges and to consult on its planned increase in co-investment. This led to raising the profile of the Fund and generating a set of strong, industry-driven proposals. UKRI received 252 responses to its call for expressions of interest for challenges. However, officials informed us that it was also a very resource-intensive and time-consuming process. The UKRI Steering Board agreed, in a subsequent review, that the approach to engagement used on Wave 3 should not be replicated.

Approval processes in both the Department and HM Treasury. The approval of business cases by officials and then by ministers in the Department and HM Treasury all took place in sequence – four stages in all. This process was unchanged between Waves 2 and 3.

Capacity to manage the process within UKRI. At the start of Waves 2 and 3, UKRI faced significant challenges recruiting staff to oversee and manage the challenge programmes. While UKRI has reallocated staff from its other activities we identified examples where this had led to delays in administration, monitoring and evaluation. This issue is considered further at paragraph 2.20.

2.17 UKRI and the Department have to strike a balance between identifying and then selecting the best bids for challenges and preparing robust business cases, and the time and effort taken in doing so. There are a number of potential consequences of having an elongated approval process.

Delays may begin to favour larger applicants, who are more likely to have the personnel and resources to engage with the process over an extended period.

Businesses hoping to exploit opportunities in faster moving competitive markets may lose a competitive edge if they are unable to embark on potentially fruitful projects.

Delays are likely to have led to the Fund underspending against its budgeted profile. The overall underspend figure for the Fund increased from under 10% in the early years to 14% in 2019-20 (see paragraph 2.27). Underspending could limit the value for money secured from the Fund if component projects are not taken forward at the time considered right to achieve impact.

2.18 Along with the time taken to approve challenges, stakeholders we consulted also noted that the time to approve project applications was too long and did not align with the innovative nature of the projects. UKRI has multiple stages to determine which projects to select and to formally approve the funding offer. This involves project assessments where up to five experts assess applications against 10 questions, including looking at how well the project aligns with the challenge objectives, and project setup which includes financial checks on applicant and project costs, accepting the funding conditions and agreeing collaboration agreements between project participants where relevant.

2.19 We looked at the total time taken from when applications were submitted to when funding was offered. Using data for 236 projects with applications submitted in late 2018 or after, this took more than 31 weeks on average (with the quickest taking 16 weeks and the longest 53 weeks).12 While UKRI does not have a target for the whole process, it does have a target time of 90 days for a successful applicant to receive a grant letter after they have been notified they have been successful. We found that on average it took 154 days from notification to grant offer letter approval, with a range of 53 days to 302 days. UKRI’s 90-day target is dependent on quick action within UKRI and prompt responses from project participants.

Recruitment of staff to oversee challenges

2.20 UKRI estimated that it needed 186 full-time equivalent staff to administer the Fund in 2019. This complement includes Challenge Directors who have responsibility for the oversight and management of individual challenges, as well as those responsible for day-to-day monitoring and the administration of challenges. In June 2019, 103 out of the 186 positions required were vacant (55%). UKRI estimated that its staffing to administer Wave 2 challenges was 40% under-capacity. For Wave 3 this had increased further to around 60% under-capacity, equating to approximately 42 full-time equivalents in the Wave 3 challenge teams (out of the 67.5 staff required). By January 2021, the situation had improved – UKRI identified four vacancies in each of Waves 2 and 3 against a complement of 68 and 63 full-time equivalents.

2.21 UKRI’s appointment of Challenge Directors is critical to setting the direction for and then successfully implementing each challenge.13 Our analysis shows that UKRI took more than 37 weeks on average to appoint a permanent Challenge Director for the duration of the challenge, and more than a year to appoint directors for five of the challenges (Figure 11).14 UKRI informed us that the delays were partly due to the difficulty of finding and hiring staff at an appropriate level who have a mix of science and industry experience. UKRI also faced restrictions on the salaries it could offer.15 UKRI believes it has improved its recruitment processes. Our analysis suggests that the time to appoint Challenge Directors has improved from Wave 1 to Wave 3 from 52 to 24 weeks.

2.22 UKRI has faced difficulty in securing diversity amongst its Challenge Directors. Out of 20 Challenge Directors at the end of August 2020, only three were female. Having a diversity of thinking and perspective is likely to be critical to the delivery of the challenge programmes, enabling broader insight to be brought to the identification of challenges and the encouragement and engagement of a wider range of applicant organisations. UKRI has, over the past year, sought to address equality, diversity and inclusion issues within the organisation – it agreed a plan to tackle these issues, assigning advocates, allocating resources to promote equality, diversity and inclusion issues and providing teams with training. At the time of our fieldwork, this thinking had yet to carry over into how it monitors diversity among the lead applicants to the Fund.

Leveraging investment to support R&D

2.23 A key objective of the Fund is to lever increased investment in R&D and innovation in the UK. Applicants to the Fund are expected to invest alongside any public funding committed through the Fund. UKRI have different co-investment requirements depending on the organisation type, grant type and Wave. Small- and medium-sized enterprises, for example, are often required to co-invest less than larger companies as they have smaller budgets and need more support. By the end of June 2020, industry contributed around £567 million to projects, against a target of £2.8 billion. UKRI has forecast industry will contribute £3.0 billion in co-investment, slightly more than its target, over the life of the current challenges.

2.24 The ratio of public investment to private investment required has increased from 1:0.45 in Wave 1 to 1:1.5 in Wave 3. The Department decided that for Wave 3 co-investment should increase to allow the available public funding to be distributed across a wider range of challenges and also increase the private investment levered by the Fund.

2.25 As noted in paragraph 2.9 some stakeholders we consulted were concerned that the increased co-investment requirements might deter smaller organisations from participating. Our evidence suggests that larger businesses had begun to win a larger proportion of competitions for funding (see analysis in paragraph 2.7 and Figure 7).

Management of the budget

2.26 UKRI was allocated a budget of £3.0 billion for the Fund to be spent over the period 2017-18 to 2024-25. The medium-term budget has provided UKRI with a degree of financial certainty within which to plan the investments made by the Fund. To use the overall budget effectively UKRI needs to manage the overall profile of spending over the eight-year period.

2.27 UKRI has successfully managed within the budgets set so far although there is a risk that underspending early in the eight-year period may create difficulties later. Over the first three years of operation, the Fund budget has been underspent by 11% – in total, by March 2020 UKRI had spent £1,024 million against its budget to that point of £1,146 million. Figure 12 sets out details of budget and spending for each of the first three years of the Fund’s existence.

2.28 Expenditure each year now reflects a combination of spending on challenges established in previous years of the Fund plus spending on new challenges approved in-year. Our analysis suggested that delays in getting new challenges approved and up and running was having a knock-on impact on UKRI’s ability to spend against the monies allocated for the new challenges and that the underspend had increased between Waves 2 and 3.

Wave 2 – UKRI underspent by £37.3 million, 41% of the allocation earmarked for the new challenges approved in Wave 2 (2018-19).

Wave 3 – UKRI underspent £73 million, 84% of the allocation earmarked for the new challenges approved in Wave 3 (2019-20).

2.29 UKRI considers the nature of spending in the Fund to be inherently volatile reflecting decisions taken by multiple partners, the availability of co-investment and the inherent riskiness of some of the projects/challenges. A number of issues have contributed to underspending in addition to the delayed approval of challenges and the staffing issues referred to above, for example there is the need to pause some work to secure industry buy-in and clarify what can be delivered.

2.30 Higher than projected underspends when new challenges are approved may increase the pressure on budgets in the following years as commitments build up. This pressure may increase the risk of UKRI having to re-profile its spending to fit within its annual budgets. The financial re-profiling of a challenge could have an impact on the co-funding from partner organisations by altering planned activity. Although UKRI has mitigated the risk by engaging actively with its partner bodies, it will need to keep the risk manageable. UKRI and the Department are currently reviewing the Fund’s multi-year budget profile. The Fund was part of a one-year settlement in the spending review in November 2020.

2.31 The impact of COVID-19 on the level of activity undertaken on funded projects may put pressure on future budgets. During 2020-21 UKRI agreed to re-profile £165 million from the current budget into future years for 20 challenges. Some businesses, particularly small- and medium-sized enterprises struggled to meet the co-investment requirements due to the impact of COVID-19. Some of those businesses have had to re-scope or pause work on their projects supported by the Fund in response to COVID-19; some also struggled to meet their co-investment targets. UKRI is also considering whether all challenges still remain relevant given the changed environment – it is prepared to stop challenges that are no longer relevant or where planned outcomes can no longer be achieved.

2.32 The medicines manufacturing challenge in particular has been affected by COVID-19 with some projects completely altered in scope to focus on coronavirus-related medicines. One of these projects is the Vaccines Manufacturing and Innovation Centre (VMIC). UKRI originally allocated £65 million to build VMIC, with the project starting in December 2017 and completion proposed for 2020-21. However, delays (due to the need, for example, to improve bid quality and to negotiate with the project’s lead partner on issues such as co-investment) pushed the start of the project to September 2018, with completion proposed for summer 2022. Following the emergence of COVID-19, the government approved a further £93 million of funding to support an opening date in summer 2021 and to significantly increase manufacturing capacity to 70 million doses of a pandemic vaccine over a four- to six-month time period. VMIC is now expected to be fully operational in the first quarter of 2022, with the equipment required to dispense the vaccine into vials (known as ‘fill and finish’) expected to be operational from December 2021. UKRI informed us that the global impact of the pandemic has led to increased supplier lead times and supply shortages for fill and finish equipment.

8 Co-investment funding helps support the government’s ambition for the UK to invest 2.4% of gross domestic product in research and development by 2027.

9 Due to the limitations in accessing data systems across the research councils, this analysis only includes data from Innovate UK-managed projects.

10 The bodies responsible for the Industrial Strategy Challenge Fund before UK Research and Innovation was created (see paragraph 1.12) did not run an expressions of interest phase for Wave 1 as they wanted to start funding projects as soon as possible.

11 Challenges under Wave 1 were approved in 2017 as part of the business case for the Industrial Strategy Challenge Fund as a whole and before creation of UK Research and Innovation in April 2018.

12 UKRI did not provide detailed date data for applications pre-late 2018 due to IT system limitations. In late 2018 and early 2019, IT systems were updated and the Innovation Funding Service was implemented which enabled more detailed tracking and recording of project setup activity.

13 Due to the seniority of the Challenge Director role, all roles had to be approved by a Senior Civil Service Panel

14 Eighteen of the 20 challenges had an interim Challenge Director in place before the permanent Challenge Director was recruited. In seven of those 18 challenges, the interim Challenge Director was appointed in the permanent role.This analysis looked at the time taken until the permanent Challenge Director started in their role at UKRI. It includes the time taken for recruitment and the notice period when leaving their previous role (if relevant).

15 HM Treasury, Guidance for approval of senior pay: applicable from 1 January 2018 – updated November 2018. Available at: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/756196/Guidance_for_approval_of_senior_pay_final.pdf.

Part Three

Measuring the performance of the Industrial Strategy Challenge Fund

3.1 This Part considers UK Research and Innovation’s (UKRI’s) approach to:

assessing performance at project and challenge level;

assessing performance against the Industrial Strategy Challenge Fund’s (the (Fund’s) objectives; and

evaluating long-term performance.

Assessing performance at project level

3.2 Measuring the performance of the Fund is not straightforward. The Fund’s investments are intended to support projects that will contribute to the challenges chosen, in many instances delivering benefits to society as well as commercial and other benefits to participating organisations. A number of factors complicate interpretation of performance:

The heterogenous nature of challenges. There are 24 challenges. Most are in different sectors, and have very different objectives. It can therefore be difficult to bring their performance together to provide a perspective across the portfolio.

The innovative nature of the projects. UKRI expects its investment in research and development (R&D) and innovation to be risky and the outcomes uncertain. Such uncertainty makes it challenging to predict and measure the outcomes from the projects. If a project is not completed, it will not necessarily mean that it was a failure given the context in which the Fund is operating.

Long-term outcomes. Many of the expected outcomes from projects could take 10–15 years to come to fruition, making it hard to track and evaluate the outcomes coming from different projects and the programme.

3.3 To assess the performance of the Fund, UKRI has introduced processes to measure performance at project and challenge level (Figure 13). For example, for the majority of projects it funds, UKRI relies on independent project monitoring officers contracted to conduct quarterly project-by-project assessments against six criteria (scope, cost, time exploitation, risk management and project management).16 These assessments are currently brought together to provide an insight into progress and performance for each of the 24 challenges. In addition, UKRI’s Challenge Directors report against each challenge on a quarterly basis to senior managers against three criteria (budget, time and people).

3.4 By January 2021, UKRI was reporting broadly favourable progress across its projects and challenges:

Scores from the independent monitoring officers on individual projects show that 90% of projects were assessed at 3.0 or over (out of 5). A score of 3 or more indicates that UKRI is content with the project’s performance and progress, taking into account the innovative nature of the projects supported, while identifying room for improvement. Average project monitoring officer scores by challenge lay between 3.17 and 4.06.

Performance assessment varied considerably in one category, cost, highlighting an area of concern. In this category, some challenges were assessed as exceeding expectations (5.0) and some as very poor (2.0) or unacceptable (1.0).

Challenge Directors’ assessments of the performance at challenge level put 16 challenges in the top two categories (meaning that performance was either in line with expectations or where an issue which might have a negative impact on the project had been identified but could be mitigated). These assessments have fluctuated over recent months. In July 2020, 12 challenges were in these categories. By October 2020, 19 of the challenges were rated in the top two categories. UKRI informed us that the reason for the decrease from 19 to 16 was primarily due to the impact of COVID-19 on individual challenges.

3.5 UKRI’s ability to extract meaningful performance information has been hindered by its reliance on systems inherited from Innovate UK and the research councils. Until June 2020, it was reliant on the manual compilation of data drawn from these systems, leading to sometimes incomplete and untimely management information. UKRI now has access to more timely information which draws directly from these different systems. Even so, the research councils monitor the projects using different processes and systems compared to Innovate UK, which provide the core system for the Fund. As the processes and systems are not aligned for projects initiated by the research councils, some of the data are either not included at all in the management information or still have to be added manually. In addition, UKRI does not report to the Steering Board on the 596 projects funded in Wave 1a as management information on these projects is not readily available. In August 2020, the value of these and research council-managed projects was £682 million.

3.6 The nature of the challenges and projects supported by the Fund inevitably carry some risk. UKRI set out its risk appetite with respect to the Fund for the first time in September 2020:

With respect to funding decisions, UKRI characterises its risk appetite as ‘bold’ – the riskiest category in its risk framework. This reflects the decisions UKRI needs to take to fund what it describes as “cutting edge and innovative” projects.

With regard to programme and project delivery, UKRI characterises its risk appetite as ‘open’ – its second riskiest category. UKRI’s open category provides challenges with the scope to consider all potential delivery options to meet their objectives within a reasonable level of risk balanced against the potential for reward.

3.7 UKRI considered its risk exposure against these categories in October 2020 and concluded all risks were within the risk appetite it had defined, other than for the impact of COVID-19 where challenges were starting to see increased disruption as a result of local lockdowns.

Monitoring progress at Fund level

3.8 In 2020, UKRI started to consider its performance against the Fund’s objectives (paragraph 1.7). In January 2020, it looked at the Fund’s progress against its five objectives by comparing its performance with that of other funds for which it had oversight. This comparison was favourable for four of the five objectives for which UKRI was able to collect data. Since then, it has reviewed performance on a quarterly basis by considering the proportion of the Fund which meets each objective’s measure of success (Figure 14). However, no targets have been set, making it difficult to determine whether reported performance is adequate.

3.9 While UKRI has established processes to track the performance of projects and the current performance of challenges we found it difficult to draw a clear link between these and the five Fund objectives set by the Department. The five objectives, as currently set, are heavily focused on the types of collaboration receiving support from the Fund – multidisciplinary, spanning business and academia, and so on – rather than the outcomes those collaborations might deliver in terms of addressing the challenges and benefits to the UK economy which are set at challenge level. The mismatch between the objectives, as currently stated, together with the increasing number of challenges the Fund is seeking to address is making the task of reporting on progress at Fund level more difficult.

Evaluating performance

3.10 In 2019, UKRI set out in detail the benefits that each challenge was intended to deliver and how the projects supported would contribute. These are known as ‘benefits maps’. By May 2019 these were established across all challenges. It is currently piloting its approach to collecting performance data in this area for two challenges, and is looking to have adopted this approach for all challenges by April 2022.

3.11 To be able to assess whether the planned benefits are delivered, UKRI needs to be able track planned and unplanned impacts arising from its investments. UKRI has already set out a thorough approach for evaluating impact although it faces challenges putting this into practice. UKRI’s approach reviews and evaluates performance across the three levels of the Fund; project level, challenge level and Fund level (Figure 15). UKRI has created a Fund evaluation team who help with the challenge evaluations and lead on the portfolio level evaluations.

3.12 Our review suggested that UKRI faces a number of challenges implementing its approach to evaluation:

Timings of the evaluations which have been commissioned to date do not take sufficient account of the long-term nature of what the Fund is trying to achieve. Some outcomes may take many years to achieve after the project has completed, yet evaluations it has commissioned to date will only cover the first few years after the projects are completed. UKRI’s approach does allow for future evaluation activity beyond the life of the programme.

Challenge Directors are responsible for the evaluation of their challenges which they commission from independent evaluators. We identified examples where they did not have the necessary knowledge or the capacity within their team to complete the evaluations required in a timely way.

UKRI’s central evaluation team has oversight of evaluations to ensure consistency. External evaluators commissioned by UKRI commented on the team’s capacity to coordinate evaluations across multiple challenges, creating a risk to the consistency of approach being taken.

Delays to the evaluations being produced – to date, UKRI has completed process and baseline evaluations for seven of the 11 challenges which make up Waves 1 and 2. It informed us that it expects to receive the same process and baseline evaluations for two of the remaining four challenges by March 2021 in line with its expectations, with these two types of evaluations for the two remaining challenges expected in July 2021, four months behind schedule. Given the baseline is supposed to show the state of the market when the challenge began, delaying this means the evaluation has to be carried out retrospectively and may be less accurate. The delay also pushes back when UKRI is able to implement any recommendations from the process or baseline evaluations.

16 Almost 90% of projects funded by the Industrial Strategy Challenge Fund originate through Innovate UK rather than the research councils.

Appendix One

Our audit approach

1 This report examines UKRI’s management of the Industrial Strategy Challenge Fund (the Fund). The report is structured as follows:

Part One provides the background on the Fund.

Part Two looks at the establishment of the Fund, in particular whether it has attracted sufficient good-quality bids, whether the selection processes have been efficient and whether the budget is managed effectively.

Part Three looks at the approach to monitoring and evaluating the Fund’s performance.

2 Our audit approach is summarised in Figure 16.

Appendix Two

Our evidence base

1 We reached our independent conclusions after analysing a variety of evidence sources collected between December 2019 and January 2021. Our audit approach is outlined in Appendix One.

2 In designing and carrying out our work, we took account of previous relevant National Audit Office (NAO) reports including our 2016 report BIS’s capital investment in science projects.17 We also considered the House of Commons Business, Energy and Industrial Strategy Committee 2019 report Industrial Strategy: Sector Deals,18 and the House of Commons Science and Technology Committee’s 2019 report Balance and effectiveness of research and innovation spending.19

3 The scope of this report was informed by the ‘discovery and design’ sessions the NAO organised with UK Research and Innovation (UKRI) and the Department for Business, Energy & Industrial Strategy (the Department) at the beginning of 2020. Our aim for these sessions was to build an early understanding of the challenges that UKRI and the Department faced in implementing the Industrial Strategy Challenge Fund (the Fund).

4 We interviewed officials from UKRI, the Department and HM Treasury. Those we interviewed included:

representatives from UKRI’s executive team and senior staff who have formal decision-making powers for and oversight of the Fund;

UKRI’s Challenge Directors, members of challenge teams and project monitoring officers;

members of UKRI’s central teams including evaluation leads and officials responsible for the management information; and

officials at the Department and HM Treasury with responsibility for decision-making relevant to and for oversight of the Fund.

5 We interviewed external stakeholders, including:

representatives from industry and academia; and

fund recipients from a range of different-sized businesses and from academia.

6 We interviewed independent evaluation providers responsible for undertaking the process and impact evaluations for several challenges.

7 We reviewed relevant documents including:

minutes and papers from the Fund Steering Board meetings, including portfolio updates and finance reports;

documents from five challenge case studies including programme board and advisory group meeting minutes and papers, project monitoring officer reports, risk registers, evaluation reports and business cases;

business cases for each Wave and evidence of approvals;

performance and management information including reports from UKRI’s DELPHI system (its interactive data-processing and visualisation tool);

data provided by UKRI on the projects funded by the Fund; and

documents relating to UKRI’s response to the impact of COVID-19 on the Fund.

8 We undertook analysis of UKRI data including:

actual spend against budget;

distribution of funding by type and location of recipient;

length of time to approve business cases and project applications;

length of time to approve the appointment of Challenge Directors; and

performance data at project, challenge and Fund level.

9 In reaching our independent conclusions we are aware of the following limitations to our study:

Due to the creation of UKRI from nine different bodies in April 2018, the data systems used for managing and reporting on the Fund’s projects have evolved over time and are not yet completely linked. This limited the data available for a number of our analyses. Often we were only able to see a subset of the projects (for example, if data were only available for Innovate UK-managed projects and do not include research council-managed projects, or Wave 1a projects).

Wave 1a projects are included in the total number of projects (1,613 – see paragraphs 6, 11 and 1.9), current spend (£1.2 billion – see paragraphs 6 and 1.10), realised co-investment figures (£567 million – see paragraphs 6, 1.10 and 2.23) and Fund expenditure (see Figure 12). However, due to availability of the data, Wave 1a projects are not included in the analysis of funding by size of recipients (Figure 7) and by region (Figures 8 and 9).

17 Comptroller and Auditor General, BIS’s capital investment in science projects, Session 2015-16, HC 885, National Audit Office, March 2016.

18 House of Commons Business, Energy and Industrial Strategy Committee, Industrial Strategy: Sector Deals, Seventeeth Report of Session 2017-19, HC 663, March 2019.

19 House of Commons Science and Technology Committee, Balance and effectiveness of research and innovation spending, Twenty-first Report of Session 2017–19, HC 1453, September 2019.

Appendix Three

Structure of the Industrial Strategy Challenge Fund

1 Figure 17.

Design and Production by NAO External Relations
DP Ref: 008001-001