© National Audit Office 2022

Financial modelling in government

Cross-government

REPORT

by the Comptroller
and Auditor General

Session 2021-22
27 January 2022
HC 1015

The National Audit Office (NAO) scrutinises public spending for Parliament and is independent of government and the civil service. We help Parliament hold government to account and we use our insights to help people who manage and govern public bodies improve public services.

The Comptroller and Auditor General (C&AG), Gareth Davies, is an Officer of the House of Commons and leads the NAO. We audit the financial accounts of departments and other public bodies. We also examine and report on the value for money of how public money has been spent.

In 2020, the NAO’s work led to a positive financial impact through reduced costs, improved service delivery, or other benefits to citizens, of £926 million.

We are the UK’s independent public spending watchdog.

We support Parliament in holding government to account and we help improve public services through our high-quality audits.

Financial modelling in government

Cross-government

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 25 January 2022

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Gareth Davies
Comptroller and Auditor General
National Audit Office

19 January 2022

Value for money reports

Our value for money reports examine government expenditure in order to form a judgement on whether value for money has been achieved. We also make recommendations to public bodies on how to improve public services.

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.org.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

009100 01/22 NAO

This report can be found on the National Audit Office website at www.nao.org.uk

If you need a version of this report in an alternative format for accessibility reasons, or any of the figures in a different format, contact the NAO at enquiries@nao.org.uk

The National Audit Office study team consisted of:

Phil Bradburn, Catherine Hayes, Sarah Hipkiss, Linh Nguyen and Elliott White, with assistance from Finian Bamber, Laura Cole, Andrea Jansson, Tanya Khan, Phillip Li, Zahara Lloyd and Farhan Subhan, under the Direction of Ruth Kelly.

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

020 7798 7400

www.nao.org.uk

@NAOorguk

Key facts

Summary

1 Analysis is at the heart of how the government runs its business. Government relies on financial models for its day-to-day activities including: estimating costs; distributing funding within organisations; and testing policy options. In recent years departments have used models to plan NHS test and trace services, set allocations for teacher training places, and estimate the cost of the financial settlement when leaving the EU.

2 Financial models use information or data to provide insight into a question or to better understand a problem. Using models helps government to select policy options, understand the impact of these options and improve the value for money of government spending. For example, UK TIMES, a bottom-up, cost optimisation model of the whole UK energy system, produces an estimate of all greenhouse gases, under different planning assumptions. Government uses this model to provide important evidence supporting its plans to tackle climate change, such as the net zero target decision. Models also underpin decisions which affect people’s lives. In December 2020, we reported on the epidemiological modelling by NHS Test and Trace, which it used to help plan staff and testing capacity at a time of inherent uncertainty. We found that underestimating demand in September 2020 led to difficulties in meeting higher than expected demand for tests, increasing turnaround times and limiting tests available to the public.1

3 After the collapse of the West Coast Main Line franchise competition in 2012 – where errors in models played a role in the incorrect information given to bidders – HM Treasury (HMT) initiated a review of how the government produces and uses models, known as the Macpherson Review. This review was published in 2013 and made eight recommendations to extend the pockets of good practice it found across the whole of government. Following the review, HMT took action to improve the quality of models, such as setting up a working group to produce guidance. Separately, in 2013, the government introduced cross-government functions to provide professional support to departments. The two functions most related to financial modelling are the Analysis Function and the Finance Function.

4 Supported by the board, the accounting officer of each central government organisation is responsible for overseeing the use and quality assurance of models within that organisation. Models will vary in their importance to the organisation, and some will qualify as ‘business-critical models’.2

Scope and purpose of this report

5 We have examined the roles that HMT, the Office for Budget Responsibility (OBR), the Analysis Function and the Finance Function have in improving modelling across government. We considered how well the principles set out in the Macpherson Review, Managing Public Money and other modelling guidance are embedded across government and applied to business-critical financial models. Our audit approach is based on the National Audit Office’s (NAO’s) Framework to review models (Figure 1) and the report examines:

how the responsibility for modelling is organised across government (Part One);

the quality assurance processes across government and how organisations provide assurance that models are fit for use (Part Two); and

how uncertainty is assessed, communicated and taken into account when developing plans (Part Three).

6 This report reviews models used for financial planning, but many of the recommendations will be sensible principles to follow for all models across government. We use the term ‘models’ and ‘modelling’ to refer to financially focused business-critical models. This includes models used to inform debate on the costs of potential policies as well as models more directly tied to budget bids and financial reporting. We used 12 case studies across four departments to understand the processes these departments use for managing business-critical models. The report does not conclude on the reasonableness or robustness of any individual model reviewed as part of the study. Our methods and evidence base are described in Appendix Two.

Key findings

Governance of business-critical models

7 It is unclear who is ultimately accountable for upholding modelling standards and for driving improvement across government. The Analysis Function, Finance Function and HMT all have an interest in how the 962 business-critical models across departments are managed and used. We have, however, been unable to identify any single body responsible and accountable for updating and maintaining guidance, monitoring and assuring whether the guidance has been implemented, or driving cross-government improvement by learning from others. We have reported before on the importance of clear aims, expectations, roles and responsibilities, especially where multiple government organisations are involved(paragraphs 1.7 and 1.8, Figure 2 and Figure 4).3

8 The centre of government and departments have worked together to improve understanding and oversight of models.4 Following the Macpherson Review, HMT updated Managing Public Money to provide detail on accounting officers’ responsibilities for the quality assurance of models and set up the Quality Assurance Working Group to promote good practice across government. The Aqua Book is one of the working group’s core products. Published in 2015, it introduced guidance across government on how to produce high-quality analysis. The working group assessed actions since the Macpherson Review and found all departments had made progress in implementing governance and assurance processes and improving the robustness and resilience of models (paragraphs 1.4 to 1.6 and Figure 3).

9 The Analysis Function has yet to agree with HMT the funding it considers necessary to support efforts to improve modelling in government. In 2020-21 the Analysis Function received £1.3 million in funding from the Office for National Statistics. For the 2020 Spending Review, the Analysis Function prepared a bid for £4.9 million to cover its planned activities in 2021-22. However, because the scope of the Spending Review changed, HMT did not review the bid and the Function remained funded at the original rate for 2021-22. At the 2021 Spending Review, HMT did not allocate funding specifically for the Function, in part because the Function was in the process of working out its scope and governance arrangements. HMT agreed to consider the 2022-23 funding for the Analysis Function as part of the main estimate funding round in February 2022. This will determine the level of funding available to the Function and be a crucial step in enabling the Function to refine and then deliver its plans, including on modelling in government (paragraphs 1.5 and 1.6).

10 Departments take different approaches to managing their business-critical models. A department’s accounting officer is ultimately responsible for the use and quality assurance of models in his or her department. This responsibility is usually delegated to the department’s director of analysis. Government guidance sets out high-level principles and it is left to departments and arm’s-length bodies (ALBs) to interpret and apply this. This means that departments have developed at least six different definitions of business-critical models, customised their own guidance, and taken variable approaches to monitoring and improving the quality of models (paragraphs 1.12 to 1.16 and Figure 5).

11 Departments take different approaches to overseeing and supporting ALBs. We reported in 2021 that the risks in relation to ALBs are not well understood, and that there is no collective understanding of the oversight appropriate for different types of ALBs.5 ALBs produce, quality assure, and provide outputs from their models for their department. There is no guidance for departments on the level of scrutiny on modelling they should apply to their ALBs. Our survey highlighted that the oversight of ALBs’ models continues to be variable across government, with nine out of 15 departments sharing their resourcing and training with their ALBs and 14 departments giving responsibility for the quality of models to their ALBs. (Paragraphs 1.17 to 1.19 and Figure 7).

12 It is difficult for Parliament and the public to access information about business-critical models. Transparency supports scrutiny and quality assurance and Managing Public Money states that “transparency should be the norm in the development and use of all models”. In practice, we found this is not usually the case. For a sample of 75 models, we found no information available for 45 of these models. For the remaining 30, we found a range of information, from basic details on the model through to extensive details of the model published. Only nine departments out of 17 have published their register of business-critical models since the Macpherson Review published the full list in 2013. Only four of these registers have been updated since January 2017 (paragraphs 1.20 to 1.22, Figure 8 and Figure 9).

Assurance of data, assumptions, methods and calculations

13 Departments do not consistently use quality assurers who are independent of the modelling team, which leads to a risk of self-review. The Aqua Book and the Analysis Functional Standard both expect that models are independently reviewed. In our case studies, we saw examples of models being reviewed by a second analyst before use. However, the assuring analyst was usually located in the same team as the primary analyst, and the separation between duties was not always clear. In our audit work across government, we regularly find errors in departments’ models. For example, our audit of a department’s 2020-21 accounts identified errors of £800 million and £45 million in the calculations of two financial models. The department corrected these errors as part of the financial audit process and so they did not affect the published annual report and accounts. Before our audit, the models had not been independently verified, which could have identified the errors. Our case study departments told us that there are barriers to independent review, and they are taking various actions to address these (paragraphs 2.5 to 2.7).

14 Assurance of input data and assumptions is variable. We saw examples of good practice in departments: in some cases they tested their updated assumptions with stakeholders and in others they routinely compared forecast results to actual events. On the other hand, for some models, we found backlogs in the routine work of updating assumptions, and gaps in documentation and supporting evidence. This makes it more difficult to keep track of, assure and validate assumptions. Poor-quality inputs can have serious impacts: our 2021 report Optimising the defence estate found forecasts were initially based on assumptions which proved unachievable. This contributed to the potential net benefits being overstated. Expected savings have fallen by 73% since 2016. We reported it was uncertain whether the expected benefits would have still exceeded the costs if the department had considered all relevant costs and appropriate risk contingency.6 Controls for the quality management and input of data also vary within and between departments. Our report Challenges in using data across government found that a lack of common data models and standards makes it difficult and costly to combine data, and data quality is often inadequate. In December 2020, government produced a framework to improve the quality of its data (paragraphs 2.8 to 2.13, Figure 10 and Figure 11).

15 There is room for improvement in model documentation. Effective quality assurance of business-critical models requires clear and proportionate documentation. In our 12 case studies, we found examples of good quality documentation but also some notable gaps: some models lacked technical guides, analytical assurance plans, assurance records or written succession plans. Gaps in model documentation make complex models difficult to interpret, revisit or review. As a result, senior responsible owners may lack the necessary information to make informed decisions on the risks of using their model’s results (paragraphs 2.14 to 2.17).

Managing uncertainty

16 Model producers do not adequately assess or communicate the uncertainty in their models. Models cannot exactly represent what we observe or predict the future with perfect accuracy. Uncertainty is inherent in modelled information and should be considered as part of all analysis. This is emphasised by HM Government’s Orange Book, which describes how analysis of risks provides the foundation to identify and manage risks and uncertainties. In our case studies we found limited evidence of detailed analysis of uncertainty and departments generally present outputs as best estimates. Where analysts do perform uncertainty analysis, this is often basic, for example, sensitivity testing of the main assumptions. We saw pockets of good practice in communicating uncertainty, such as including a confidence interval around a best estimate, but also found examples where uncertainty was often described only in qualitative terms or where it was not routinely presented to users (paragraphs 3.2 to 3.9 and Figure 12 and Figure 13).

17 Senior decision-makers need to use uncertainty analysis to manage risks to value for money. Models are used widely across government to support financial planning, risk management and decision-making for major projects and programmes. Decision-makers need information on the range of outcomes that may occur and their relative likelihoods to manage risks to value for money. In our case studies, we found departments often use best estimates as a basis for their financial and business plans. We found limited evidence of departments using uncertainty analysis or developing contingency plans to respond effectively to unintended but plausible events. Our report Lessons learned from Major Programmes found that many programmes we reviewed have not sufficiently recognised the inherent uncertainties and risks in early estimates.7 For example, our report on Completing Crossrail found the decision-making in the latter stages of the project was dominated by achieving a fixed completion date.8 Some of the decisions taken drove unnecessary cost into the programme. Furthermore, we found in our report Learning for government from EU Exit preparations that the civil service can improve how it deals with uncertainty.9 This was also demonstrated in our report Initial learning from the government’s response to the COVID-19 pandemic, which found that government lacked a script for many aspects of its response. This reduced the government’s ability to respond to the emergency (paragraphs 3.1 to 3.3, 3.7 to 3.11).

18 There are opportunities for HMT and the OBR to improve their use of business-critical model outputs from departments and ALBs. Departments and ALBs present outputs from their models to HMT and the OBR as part of the spending review and budget process. HMT and the OBR use these outputs for forecasting, budget planning and to monitor emerging risks. Departments typically provide a best estimate and do not routinely provide a range of uncertainty around this best estimate in their initial submissions to HMT and OBR. HMT spending teams and the OBR told us they request further analysis from departments on uncertainty on a case-by-case basis. HMT and OBR would have greater insight from departments by routinely requesting the range of plausible outcomes. (Paragraphs 3.12 to 3.16).

Conclusion

19 Financial modelling is at the heart of how the government understands its spending, performance and risks and makes business-critical decisions. Outputs from models underpin decisions made by departments and ALBs that often have very real impacts on people’s lives. Errors in government models have directly caused significant losses of public money and delays to critical public programmes. Since the completion of the Macpherson Review of the quality assurance of models, the government has made progress through publishing cross-government guidance. Separately, the government introduced the Analysis Function and the Finance Function. Departments and ALBs have implemented new governance and assurance procedures.

20 Although progress has been made, there remain significant weaknesses in how government produces and uses models. There is scope for better leadership from the centre of government to drive further progress, uphold standards and support greater transparency around models that departments use to make decisions. Although we saw examples of good practice, the level of quality assurance that departments apply to business-critical models remains variable. The analysis of uncertainty is often a peripheral activity despite it being extensively recommended in government guidance and despite the risks to long-term value for money of not doing so. Taken as a whole, the government is overly reliant on best estimates from models which do not fully reflect the inherent uncertainty and risks. Without further progress, government plans will continue to be developed with weaknesses that place value for money at risk.

Recommendations

21 Accounting officers, supported by directors of analysis, are ultimately responsible for the quality of models in their organisations. Our recommendations are directed both to accounting officers and HMT, the OBR and the Functions. They are aimed at improving the clarity of requirements and the provision of oversight and incentives to support accounting officers in their role.

22 Accounting officers should:

a Oversee the use of models within their organisation and ensure an appropriate quality assurance framework is in place and used for all business-critical models.

23 HMT should:

b re-emphasise accounting officer responsibilities for business-critical models as set out in Managing Public Money, and the importance of publishing lists of such models on gov.uk by specifying this requirement in the guidance HMT issues on annual reports and accounts;

c put in place processes to assure itself that outputs from departments’ and ALBs’ business-critical models, which HMT uses, have been quality-assured in line with modelling standards. This should include clarifying in all relevant guidance that all models must comply with the Aqua Book;

d build on its current approach to quantifying uncertainty and risk analysis by requiring departments to present HMT with a range of plausible outcomes from business-critical models as a matter of routine. This range should be driven by key inputs and model parameters in each case to take account of where there might be material uncertainties around best estimates; and

e agree with the Analysis Function on responsibilities for ownership and maintenance of the Aqua Book, including appropriate sign-off arrangements between the Function and HMT for Aqua Book updates.

24 The Analysis Function should:

f set out the appropriate governance structure for the ownership, maintenance, monitoring and assurance of analytical modelling standards and guidance, as presented in the Analysis Functional Standard. As part of this, the Function should work with the Cabinet Office to develop an appropriate assessment framework to provide the necessary processes to monitor departments’ and accounting officers’ implementation of the Analysis Functional Standard;

g update its Functional Standard and relevant guidance to include clear principles for departments and ALBs to follow on independent review of business-critical models, and on publication of a model’s inputs, methodology, assumptions, and outputs; and

h work with departments, ALBs and other stakeholders such as the Quality Assurance Working Group on guidance and training to facilitate system-wide learning and improvement. This should include sharing good practice on how business-critical models are managed and practical advice on how to analyse and communicate uncertainty.

25 HMT and the Analysis Function should:

i agree the funding and capacity implications of the proposed governance structure in relation to analytical modelling standards and guidance.

26 The Cabinet Office is working on common standards for departmental sponsorship of ALBs. As part of this work, it should:

j include guidance for departments on overseeing the production and assurance of models in ALBs, based on expert input from the Analysis Function.

27 The Finance Function should work with the Analysis Function to:

k strengthen the requirements in the Finance Functional Standard on departments to apply the Analysis Functional Standard and the Aqua Book to financial planning and reporting. This should include guidance on how accountants should analyse, manage and communicate uncertainty; and

l include appropriate elements relating to analysis and modelling from the Finance Functional Standard in the Finance Function’s self-assessment tools to measure compliance of functional members with requirements on modelling.

28 The OBR should:

m require departments, as a matter of routine, to analyse and present the range of plausible outcomes driven by key inputs and model parameters in each case to take account of where there might be material uncertainties around best estimates.

1 Comptroller and Auditor General, The government’s approach to test and trace in England – interim report, Session 2019–2021, HC 1070, National Audit Office, December 2020.

2 The Macpherson Review criteria for judging if a model is business-critical are based on the extent to which: the model drives essential financial and funding decisions; the model is essential to achievement of business plan actions and priorities; errors could engender serious financial, legal, or reputational damage or penalties.

3 National Audit Office, Improving operational delivery in government: A good practice guide for senior leaders, March 2021.

4 We use the term centre of government to refer to the Cabinet Office, HM Treasury and the senior leadership of the Analysis Function and the Finance Function.

5 Comptroller and Auditor General, Central oversight of arm’s-length bodies, Session 2021-22, HC 297, National Audit Office, June 2021.

6 Comptroller and Auditor General, Optimising the defence estate, Session 2021-22, HC 293, National Audit Office, June 2021.

7 Comptroller and Auditor General, Lessons learned from Major Programmes, Session 2019–2021, HC 960, National Audit Office, November 2020.

8 Comptroller and Auditor General, Completing Crossrail, Session 2017–2019, HC 2106, National Audit Office, May 2019.

9 Comptroller and Auditor General, Learning for government from EU Exit preparations, Session 2019–2021, HC 578, National Audit Office, September 2020.

Part One

Governance and assurance

1.1 This part examines how responsibility for business-critical models is organised across government and the roles that HM Treasury (HMT), the Office for Budget Responsibility (OBR), the Analysis Function and the Finance Function have in improving modelling across government. This part also examines the governance of business-critical models in departments.

Business-critical models across government

1.2 A model is a way of analysing or representing some aspect of the real world, usually using a quantitative approach to apply financial, economic, or mathematical theories and assumptions. A model will take input data and process them into outputs which estimate the real world. Government relies on thousands of models for its day-to-day activities, such as simulating policy options, estimating future costs, or allocating funding within organisations. Models will vary in their importance to the organisation, and some will qualify as ‘business-critical models’. Across 17 central government departments alone, there are nearly 1,000 business-critical models in use (Figure 2). This does not include business-critical models owned by arm’s-length bodies (ALBs).

1.3 This report builds on our good-practice framework for reviewing models (Figure 1) and focuses on models used for financial planning. Throughout the report, we use the terms ‘models’ and ‘modelling’ to refer to financial-focused business-critical models. This includes models used to inform debate on the costs of potential policies as well as models more directly tied to budget bids and financial reporting. The report reviews whether the governance arrangements around these models are sufficiently robust to support the development and execution of credible plans. We examined models in 12 case studies across four departments (Department for Business, Energy & Industrial Strategy (BEIS), Department for Education (DfE), Department for Work & Pensions (DWP) and HM Revenue & Customs (HMRC)) to better understand the processes these departments use for managing business-critical models. The report does not conclude on the reasonableness or robustness of any individual model reviewed as part of the study.

Progress in improving modelling across government

1.4 After the collapse of the West Coast Main Line franchise competition in 2012 – where errors in models played a role in the incorrect information given to bidders –HMT initiated a review of how the government produces and uses models, known as the Macpherson Review. Following this review, government took actions to improve the quality of model assurance (Figure 3), including reviewing departments’ actions against the Macpherson Review recommendations and publishing the Aqua Book, providing cross-government guidance on how to produce quality analysis.

1.5 In 2013, Cabinet Office introduced 11 cross-government functions, with the aim of building specialist capability and professionalising the workforce. To further this initiative, Cabinet Office established the Analysis Function in 2017. The Function’s role is to lead the analytical community, improve analytical capability and share best practice, including in relation to modelling. In 2020-21, the Analysis Function received £1.3 million in funding from the Office for National Statistics. In the 2020 Spending Review (which would have allocated funding for 2021-22), the Analysis Function submitted a bid for £4.9 million to fund 71 full-time equivalent staff. To support its 2020 Spending Review bid, the Analysis Function set out its planned activities which included publishing an updated Functional Standard and developing capability and capacity across government. However, the scope of the Spending Review changed because of the COVID-19 pandemic. HMT did not review this bid, so the Analysis Function remained funded at the original rate for 2021-22.

1.6 Since its inception in 2017, the Analysis Function’s remit has evolved to provide further support across government. In September 2021, the Function set up a new Analysis Function Strategy and Delivery Division to strengthen support for analysts across government. In the 2021 Spending Review (which allocated funding for 2022-23), HMT did not decide on the funding level for the Function, in part because of the ongoing changes to its structure, governance arrangements and scope. HMT agreed to consider the 2022-23 funding position for the Analysis Function as part of the 2022-23 main estimate funding round in February 2022. This will be a crucial step in enabling the Function to refine and then deliver its plans, including on modelling across government.

Oversight, accountability and monitoring against standards

1.7 We have been unable to identify any single body responsible and accountable for upholding standards and improving modelling across government (Figure 4). We found three bodies which have some responsibility:

HMT is responsible for setting budgets in discussion with departments, and maintaining guidance including Managing Public Money and the Green Book. It does not, however, consider itself responsible for maintaining the Aqua Book despite publishing it in 2015. It has no plans for follow-up work on the Macpherson Review, last reviewed in 2015. It also does not know to what extent departments and accounting officers are implementing modelling standards within their organisations.

The Analysis Function aims to improve the analytical capability of the civil service. It has limited central visibility of modelling across government. It does not have routine processes to monitor assurance done within departments and ALBs, nor the resources to do the work. The Function recognises that its lack of central oversight and visibility is a gap. To address this gap, in November 2021 the Function’s senior leadership accepted plans to develop governance arrangements for the Analysis Functional Standard. This includes a self-assessment framework to assess performance against the functional standard, and a review to identify core guidance and to assign responsibility for updating and promoting it.

The Finance Function aims to improve financial management. Although modelling is a core part of these activities, its strategy and annual review do not refer to modelling.

1.8 The gaps and lack of clarity described in Figure 4 indicate there is no comprehensive oversight and accountability for the quality of modelling across government. As the quality of modelling is so important to government as a whole, and involves aligning so many organisations, it is all the more important to have clear aims, expectations, roles and responsibilities, and an environment that values quality, learning and improvement. This was a key finding in our report Improving operational delivery in government: A good practice guide for senior leaders.10

Monitoring arrangements in HMT and the OBR

1.9 Both HMT and OBR use outputs from departments’ models to monitor fiscal risks. In addition, HMT uses the outputs to support budget settlements and spending reviews, and OBR uses the outputs to produce government’s fiscal forecasts. Any errors in departments’ models can have implications for these processes in HMT and the OBR.

1.10 HMT and the OBR monitor the quality of models through three main routes:

Fiscal events: HMT. Scrutiny and challenge through routine fiscal events such as budgets and spending reviews. To support their bids, departments present HMT with outputs from models. HMT does not routinely see the models themselves. HMT budget and spending review teams told us the team discusses the reasonableness of the assumptions with the department and carries out sense-checking of the outputs and the key drivers of the model. In some cases, HMT runs a parallel model alongside that maintained by the department. These models are managed on a risk basis and are for those areas of spending which are of higher risk in terms of size or sensitivity. However, there is a lack of comprehensive scrutiny and challenge of the department’s assurance and quality arrangements for the models. HMT’s processes cannot be relied on to identify all issues from business-critical models within departments and ALBs.

Fiscal events: OBR. Departments present OBR with model outputs as part of the fiscal forecast. As with HMT, OBR does not routinely see the models themselves. OBR discusses the outputs with departments including through a challenge process. OBR told us this includes examining: the performance of the model against outturn; whether the model reflects the economic forecast adequately; and whether any modelling changes introduced increase the accuracy of the model. OBR can change any forecast it deems necessary.

Green Book investment approval process. As part of the HMT approval process for all new funding outside delegated authority limits, HMT scrutinises business cases and the information supporting them. This information is often reliant on outputs from models, but the process does not routinely examine the models themselves. Assurance through the Green Book will only apply to new spending and investments and will not identify issues that occur through business-as-usual activities.

Forecast evaluation reports. OBR produces forecast evaluation reports as part of its approach to gaining assurance. As part of these reports the OBR examines the performance of some of the departments’ models, and its own models, against outturns.

1.11 Current assurance processes provide HMT and the OBR with the ability to challenge the high-level outputs from departmental models that they see. They are not designed to monitor if departments’ assurance processes for models are consistent and effective across the range of their activity. Except in a small number of cases, under this delegated model HMT and the OBR rely on departments to guarantee a model’s fitness for purpose (see Part Two) and the level of uncertainty in the estimates they provide (see Part Three).

Governance and management of business-critical models in departments

1.12 The Macpherson Review and Aqua Book set out high-level recommendations and guidance for the oversight of models. Managing Public Money states that a department’s accounting officer, supported by the board, is ultimately responsible for the use and quality assurance of models in their department. In practice, this responsibility is usually delegated to the departmental director of analysis. Finance directors also have responsibility for supporting their accounting officer in respecting these standards. Departments and ALBs decide how to interpret and apply the guidance within their organisations. We surveyed 17 central government departments and found numerous examples of departments applying the guidance in different ways to identify, monitor and quality-assure business-critical models.

1.13 The Macpherson Review sets out a generic set of criteria for determining if a model is business-critical.11 Departments have tailored these criteria to be specific to their needs, for example, by placing a figure on the monetary value that will trigger a business-critical classification (Figure 5). We estimate there are at least six different definitions of business-critical models across government. Departments will need to use a definition which reflects their priorities and the wide-ranging nature of business-critical models. Our survey showed that 14 out of 17 departments used all three elements of the Macpherson Review criteria, and 11 of those 14 departments clarify the criteria in some way.

1.14 All departments we surveyed had a register of their business-critical models. Departments take a range of approaches to actively manage their registers. DfE told us that its business-critical register is a live document that analysts update on a near real-time basis. Other departments update their register at fixed intervals: each month BEIS requests information from analysts on the models on its register and all models are updated at least quarterly; HMT updates its list every six months; and HMRC once a year.

1.15 In the absence of central good practice operational guidance on how to apply the high-level principles in the Macpherson Review and the Aqua Book, departments have developed customised guidance. This has led to a duplication of effort across departments, and numerous guidance documents. A degree of customisation is appropriate depending on an organisation’s remit and level of risk. However, total autonomy for departments to create guidance weakens incentives to share good ways of working between organisations.

1.16 Analysts and senior responsible owners of models are responsible for interpreting and applying their department’s quality assurance guidance. Our case studies revealed departments take a range of approaches to monitor and improve the quality assurance of models. We found that BEIS sets a benchmark ‘QA score’ for all models. Business-critical models are expected to achieve a score of at least 90%. Information on these QA scores is collected at the centre of the department and reported against its register of business-critical models (Figure 6). DfE also uses this ‘QA score’ approach but has not formally set a benchmark for business-critical models. DWP and HMRC use a series of checklists to guide quality assurance. Lead analysts in DWP are responsible for ensuring appropriate quality assurance and can use their own customised approach. A central team collects summaries of the quality assurance that analysts have applied. In HMRC, those responsible for a model self-report compliance against internal guidance once a year.

Governance of business-critical models in ALBs and third parties

1.17 ALBs produce models, quality assure them, and provide the modelled outputs for their use and their parent department’s use.12 This includes significant models such as the COVID-19 loan guarantees model. Our survey of 15 departments with ALBs showed that oversight of models is usually delegated to ALBs. Departments usually place responsibility for the quality assurance of models in ALBs on those bodies, increasing the likelihood of different approaches. Fourteen out of 15 departments expect their ALBs to take the necessary quality steps. Two departments support their ALBs by providing guidance specifically for the ALB, and nine departments give all their ALBs access to the department’s resources (Figure 7).

1.18 There is no specific guidance in Managing Public Money, the Aqua Book or the Analysis Functional Standard on how departments should oversee models in their ALBs, nor the appropriate level of support or scrutiny. Our report, Low-carbon heating of homes and businesses and the Renewable Heat Incentive, highlighted problems resulting from a lack of oversight of models in a non-ministerial body. BEIS relied on Ofgem, a regulator, to estimate the value of overpayments due to fraud and non-compliance. However, BEIS did not review Ofgem’s estimate and was unaware of the weaknesses in the selection of the audit sample and the key assumptions. As a result, BEIS could not reliably estimate the amount it had overpaid to participants.13 The lack of clarity is not constrained to the governance of models in ALBs. In our 2021 report Central oversight of ALBs we found that the risks in relation to ALBs are not well understood, and that there was no collective understanding of the oversight appropriate for different types of ALBs.14 Without clear guidance on what ‘good’ looks like there will continue to be significant variation in the way departments oversee modelling within their ALBs. Cabinet Office is now taking forward work on common standards for departmental sponsorship of ALBs.

1.19 We examined a model covering three BEIS loan guarantee schemes for this report and found BEIS had taken an active approach to overseeing one of its ALBs. As with the Renewable Heat Incentive, BEIS is ultimately accountable for funds made available through these loan guarantee schemes. It engaged with the British Business Bank (a BEIS ALB) and a third-party contractor to design and build a model to estimate expected losses. BEIS provided guidance to the British Business Bank on the level of assurance it expected, and it commissioned the Government Actuary’s Department to do an external audit of the model. It used this to provide itself with assurance that the British Business Bank’s work was sufficiently robust for estimating expected losses from these loan guarantee schemes and disclosing in its annual report and accounts.

Transparency of business-critical models

1.20 Managing Public Money sets out principles for how accounting officers should handle public funds. It states that “transparency should be the norm in the development and use of all models”. Transparency is important to support effective scrutiny and can be a powerful quality assurance tool, particularly where analysis is highly complex. Our report School funding in England highlights the benefits of improved transparency: DfE now publishes more details on the funding allocations for schools, including the methodology and underlying values for the formula each year. This makes it easier for schools, academy trusts and local authorities to understand how their funding allocation has been calculated and why allocations varied.15

1.21 In practice, we found departments are not routinely transparent about their models. For BEIS, DfE, DWP and HMRC, we searched for publicly available information for a sample of one quarter of their business-critical models (equating to 75 models). We found no information for 45 (60%) of these models. For the remaining 30 (40%), we found a variety of information available, from basic details on the model through to extensive details of the models published (see Figure 8). As the Macpherson Review describes, the appropriate degree of transparency will vary for each model, but increased transparency at any stage is a powerful tool.

1.22 The Macpherson Review included a list of all business-critical models across most government departments. It recommended accounting officers confirm in their annual report that their department or ALB has an up to date, and publicly available, list of business-critical models. Since this government-wide register was published in 2013, only nine departments have updated and republished their register of business-critical models, and only four of these have been updated since January 2017 (Figure 9).

10 National Audit Office, Improving operational delivery in government: A good practice guide for senior leaders, March 2021

11 The Macpherson Review criteria for judging if a model is business-critical are based on the extent to which: the model drives essential financial and funding decisions; the model is essential to achievement of business plan actions and priorities; errors could engender serious financial, legal, or reputational damage or penalties.

12 Arm’s-length body (ALB) is a term commonly used to cover a wide range of public bodies, including non-ministerial departments, non-departmental public bodies, executive agencies and other bodies, such as public corporations.

13 Comptroller and Auditor General, Low-carbon heating of homes and businesses and the Renewable Heat Incentive, Session 2017–2019, HC 779, National Audit Office, February 2018.

14 Comptroller and Auditor General, Central oversight of arm’s-length bodies, Session 2021-22, HC 297, National Audit Office, June 2021.

15 Comptroller and Auditor General, School funding in England, Session 2021-22, HC 300, National Audit Office, July 2021.

Part Two

Quality assurance

2.1 This part describes the quality assurance processes we observe in government departments to provide assurance that business-critical models are fit for use by decision-makers.

Essentials for quality analysis

2.2 HM Treasury (HMT) requires proportionate quality assurance for all government analysis as set out in The Aqua Book: guidance on producing quality analysis for government, alongside other guidance as described in Figure 4. The Aqua Book recommends checks to confirm that the analysis has been carried out correctly (known as verification) and that the right analysis has been performed (known as validation). HMT expects accounting officers, supported by directors of analysis, to oversee the quality of the modelling in their department.

2.3 Departments assign responsibility for specific models to senior responsible owners and analytical leads. Many departments maintain a central team responsible for modelling guidance and assurance. Departments each have their own definition of proportionate quality assurance and monitor working practices to different degrees to understand if these standards are applied (see paragraph 1.12).

Assurance of methods and calculations

2.4 We found that although the models we sampled for our case studies have been signed off for use, not all of them meet their department’s quality assurance standards. Departments were not always able to show us an agreed definition of their model’s intended use or evidence that the data, calculations and assumptions contained within the model appropriately met this intended use. Officials in several departments told us that model commissioners do not always see quality assurance as a priority, and sometimes provide too little funding and time for analysts to carry out proper quality assurance before a model’s results are used.

2.5 The Aqua Book and the Analysis Functional Standard both set out expectations that models are independently reviewed. A lack of independent review increases the likelihood that a model is not fit for purpose. However, we found departments do not consistently use quality assurers who are independent of the modelling team to review the detailed workings of business-critical models. In our case studies, we saw examples of models being verified before use by a second analyst. However, we found that the line between a model’s developers and its assurers is often blurred. The second reviewers for all models in our case studies in the Department for Work & Pensions (DWP), and most in HM Revenue & Customs (HMRC) and the Department for Education (DfE), were located in the same team as the model analysts. In some of these cases, we were told the reviewing analyst is not involved in the model’s development or day-to-day running because of the size and structure of the team, but in others, this separation does not exist.

2.6 In our audit work across government, we regularly find errors in departments’ models. For example, as part of our audit of a department’s 2020-21 accounts, we found errors of £800 million and £45 million in two forecasting models used to produce estimates in the financial accounts. The department corrected these errors as part of the financial audit process and so they did not affect the published annual report and accounts. The calculations in which we found errors had been reviewed within the modelling teams but had not been independently verified before our audit. An independent review could have identified the errors.

2.7 Departments told us that there are barriers to independent review, such as the availability of appropriately skilled people to do the work and gaps in documentation making it difficult for an independent reviewer to understand the model. However, we also saw evidence these barriers can be overcome: HMRC has a team which carries out independent reviews of a small sample of the department’s high-impact, business-critical models each year; the Department for Business, Energy & Industrial Strategy (BEIS) has created a network of independent quality assurers, and DWP has rebuilt some business-critical models using more widely available software, thereby increasing the number of people in the department with the skills to support the models. The Analysis Function told us that increased automation, reproducible analysis and good practice in software engineering all have the potential to support better peer review and auditing of model development, quality and documentation.

Assurance of data and assumptions

2.8 A model’s inputs are the information it processes to create an estimate. Data and assumptions are both inputs. A model will take input data and – using a set of assumptions – process them into outputs which estimate the real world. Business-critical models often rely on information sourced from multiple data producers, and modellers invest a considerable amount of time checking input data and assumptions. Controls for the quality management and input of data vary within and between departments.

2.9 Our 2019 report Challenges in using data across government found that having good data is not seen as a priority.16 A lack of common data models and standards makes it difficult and costly to combine data, and data quality is often inadequate (Figure 10). The Boardman Review of Government Procurement in the COVID-19 pandemic, reporting in 2021, found limited interoperability of data and systems was a repeated theme and there should be a greater focus on the review of legacy IT.17 The Data Standards Authority was established in April 2020 with the aim of identifying, improving and helping implement data standards that meet user needs. In response to concerns raised over the quality of government’s data, in December 2020 government produced a data-quality framework which sets out the approaches organisations should take. Improving the quality and usability of government’s data would make its business-critical models easier to assure.

2.10 In our 12 case studies, we saw awareness of the importance of getting inputs right. We found examples of good practice in all four departments, such as review of data prior to use, testing the validity of updated assumptions with internal stakeholders, and routine testing of outturns to forecasts (Figure 11). We found model producers often tested assumptions by consultation with policy and operations experts in departments and HMT officials. We also found that the Office for Budget Responsibility (OBR) was involved in testing assumptions for those forecasts it uses.

2.11 Assurance work on data varies. The time modellers spend assuring data depends on several factors, including the model commissioner’s priorities (see paragraph 2.4), the nature of the analysis and whether the modellers can benefit from investments in data quality in their department. We saw cases where modellers rely on external providers’ data without detailed testing, and other cases where modellers are aware of data-quality issues, so do quality assurance themselves. For example, the analytical unit responsible for student loans modelling has a team dedicated to making the data provided by the Student Loans Company fit for the modellers’ needs. In contrast, we also saw cases where the modellers received assurance that the data are fit for purpose directly from data providers. The developers of the Apprenticeships forecast model and the Universal Credit workload forecasts rely on datasets which are treated as a departmental resource and are maintained and quality-assured by specialists.

2.12 The use of poor-quality or inappropriate assumptions remains a source of risk. Model producers only sometimes test assumptions with stakeholders outside central government. Departments are not usually transparent about their business-critical models, including inputs and assumptions (see paragraphs 1.20 to 1.22). We also saw issues such as backlogs in the routine updating of assumptions, and gaps in documentation and supporting evidence, making it difficult to keep track of, assure and validate assumptions. For example, DWP’s demand models rely on assumptions about the list of administrative activities its staff must complete, and the time these take. However, gaps in documentation means the basis for these assumptions is not always clear. Furthermore, the COVID-19 pandemic has disrupted DWP’s planned work to gather new evidence to update its assumptions.

2.13 In our audit work, we often see cases where models with incomplete, immature or flawed inputs have misstated the benefits or value for money of a decision option. For instance, in our 2021 report Optimising the defence estate, we found the forecasts used to plan the defence estate disposal programme were initially based on assumptions which proved unachievable. This contributed to the potential net benefits being overstated. Expected savings have fallen by 73% since 2016. We reported it was uncertain whether the expected benefits would have still exceeded the costs if the department had considered all relevant costs and appropriate risk contingency. It is crucial to review and update assumptions as and when new evidence emerges. Our report found that while there will always be uncertainty in cost forecasts over time, collecting better data on costs would enable the Ministry of Defence to reassess the potential benefits of its estate optimisation programme.18

Model documentation

2.14 Model documentation is the set of records that enables the transfer of knowledge, including how a model works, its quality assurance, its limitations and what purposes its results are suitable for. For effective quality assurance to take place, models need clear and proportionate documentation. A model with high-quality documentation is likely to be more transparent, more robust and more resilient. We expect every business-critical model government uses to have documentation including a technical guide; a single record of data and assumptions, their sources and their quality; an analytical assurance plan; an analytical assurance log; and a succession plan.

2.15 In our 12 case studies, we found examples of good-quality documentation. We observed models with records of analytical assurance and models which applied version control. Our case studies included models with comprehensive user manuals, alongside other aspects of active succession planning, such as training multiple analysts. Many models we reviewed had the essentials needed to ensure qualified people could run and review the model if the usual analyst or assurer cannot.

2.16 We also found some notable gaps: data and assumptions logs were sometimes missing, and in some cases models had no analytical assurance plan. There were cases where models had no technical guide, and in others the guide was out of date. In some cases the quality assurance records were not thorough enough to demonstrate if the models had been adequately assured. Despite the succession planning activity we saw, comprehensive written succession plans were rare. We found cases in which only a single analyst was fully trained to operate a model, presenting a business continuity risk.

2.17 Gaps in model documentation can make models difficult to interpret, revisit or review. As a result, senior responsible owners may be unable to make informed decisions on the risks of using their model’s results. We heard that gaps in documentation were also a barrier to quality assurance, particularly independent review. Good documentation is also crucial for succession planning to enable a model to be transferred between analysts with minimum disruption.

16 Comptroller and Auditor General, Challenges in using data across government, Session 2017–2019, HC 2220, National Audit Office, June 2019.

17 Cabinet Office, Boardman Review of Government COVID-19 Procurement, May 2021, available at: www.gov.uk/government/publications/findings-of-the-boardman-review.

18 Comptroller and Auditor General, Optimising the defence estate, Session 2021-22, HC 293, National Audit Office, June 2021.

Part Three

Managing uncertainty

3.1 Government relies on financial forecasts and other modelling outputs to plan its spending, manage risks and make informed decisions. This part describes how government analyses and manages the uncertainty inherent in modelled information.

The need for uncertainty analysis

3.2 By their nature, models cannot exactly represent what we observe or predict the future with perfect accuracy. To plan well, manage risks and make better decisions, the government must ensure it understands the bases of estimates that models produce, and where areas of risk and uncertainty lie.19 For example, in our report High Speed Two: A progress update, we set out how High Speed 2 (HS2) Ltd used a method for calculating the contingency required in its budget that would have been more appropriate for a programme at a much greater stage of development and certainty. This led to an amount of contingency being set that was not enough to address the significant increases in cost as the design of HS2 became more detailed.20

3.3 As HM Treasury’s (HMT’s) Aqua Book explains, if uncertainty is not analysed explicitly as part of the analysis, it will be done implicitly when decisions are made. For example, the decision may be based on a best estimate, which would imply that the combined impact of all the uncertainties is assumed to be negligible. In our 2020 report Learning for government from EU Exit preparations, which summarises insights from the 28 EU Exit studies we conducted over four years, we found the civil service can improve how it deals with uncertainty.21 This could include planning for multiple scenarios, including robust contingency plans for those scenarios which will have a significant impact and could reasonably occur, even if some of these may not be the desired outcome.

3.4 HMT’s guidance and the Analysis Functional Standard recommend that uncertainty is considered as part of any analysis and during decision-making processes. Other parts of government, such as the Uncertainty Working Group (a network of analysts from across government), provide toolkits and support to apply this in practice.22

Assessing and communicating uncertainty

3.5 Across the set of models we reviewed in our case studies, we found limited evidence of detailed uncertainty analysis. Most commonly, the model produced a best estimate, with sensitivity testing of a small group of priority assumptions. For example, the Department for Education (DfE) undertakes sensitivity analysis for some, but not all, of its main assumptions for student loans. DfE told us this is largely due to the length of time needed to run models, which it is addressing by moving models to alternative software platforms. The Department for Work & Pensions (DWP) has developed ‘ready reckoners’ to allow for faster access to sensitivity testing. These types of analysis enable the consideration of some ‘what if’ questions. However, they do not provide users with a fully rounded view of uncertainty which they can then use to make plans and take informed decisions, which would come from more sophisticated uncertainty analysis. DWP told us it is progressing work on more refined scenario analysis.

3.6 We found instances of good practice when providing users with uncertainty analysis in our case studies. The UK TIMES model, owned by the Department for Business, Energy & Industrial Strategy (BEIS), is a bottom-up, cost optimisation model of the whole UK energy system. It produces an estimate of all greenhouse gases, including land use emissions, under different planning assumptions and uses extensive scenario and sensitivity analysis. It applies scenario assumptions developed by experts across government. HM Revenue & Customs’ (HMRC’s) tax credit error and fraud estimate is supported by a 95% confidence interval (Figure 12). DfE’s apprenticeships forecast model has developed a range by assessing and trying to quantify the uncertainty in the full set of assumptions behind the model. BEIS uses a weighted estimate of four scenarios to estimate expected credit losses from three COVID-19 loan guarantee schemes to businesses, using changes in several uncertain assumptions (Figure 13).

3.7 To be intelligent consumers of modelled outputs, decision-makers need information on the range of outcomes that may occur and their relative likelihoods. We asked modellers and commissioners involved in our case studies why they had not incorporated uncertainty analysis into their models. They raised several barriers, including technical barriers with the software used to run the models and the amount of time needed to do the work. They also described a lack of demand for the analysis from users in departments, HMT spending teams and OBR. In our case studies, we found that even when an output’s uncertainty is analysed, the output is often presented to users as a best estimate. This best estimate is then used to determine financial and operational plans. We were told in many cases that a model’s users have a good understanding of uncertainty in its results, often because they are in close communication with the model producers. However, without a change in culture, there is a risk that users and decision-makers do not fully appreciate the level of uncertainty in the numbers they receive. As a result they are more likely to create plans, form budgets and take decisions that are insufficiently resilient or overly optimistic.

Using uncertainty analysis to plan and monitor risks

How departments plan for and monitor risks

3.8 HM Government’s Orange Book explains that risk management must be an integral part of an organisation’s planning and decision-making.23 Analysis of risks – such as the varying of forecast scenarios and assumptions – provides the foundation to identify and ultimately manage risks. This analysis provides senior decision-makers with information to support the management of risks to value for money.

3.9 Paragraphs 3.5 to 3.7 describe how, across our 12 case studies, we found that modellers often produce best estimates from business-critical models. Departments use these best estimates to support the development of financial and operational plans and to monitor risks. We found limited evidence of more sophisticated use of uncertainty analysis within departments, where models are used for proactive operational planning or other activities. Examples include:

DWP used the scenario analysis it developed in response to the COVID-19 pandemic to plan options for handling an uncertain workload. DWP analysts produced scenario forecasts to examine how the workload of staff administering some benefits was likely to change, depending how the COVID-19 pandemic affected the economy. Planners looked at how groups of other departmental staff could be redeployed at short notice to cover the range of workloads the forecasts showed were realistic. This enabled them to train these groups in advance, so redeployed staff could change duties quickly and effectively if workloads surged; and

BEIS used its UK TIMES model to test the options it proposed for the sixth carbon budget. For each option, it ran four scenarios to understand the sensitivity of the option’s costs and benefits to assumptions about key technologies and resources. Government uses this model to provide important evidence supporting its plans to tackle climate change, such as the net zero target decision.

3.10 Senior responsible owners of models and departmental boards regularly commission updates from business-critical models. They examine updates to forecasts and how they have changed. This routine surveillance is used to monitor emerging trends and risks, but we have seen limited evidence of departments making use of this information for contingency planning.

3.11 We have reported previously on the consequences of departments not explicitly considering uncertainty.

Our report Lessons learned from Major Programmes found that many programmes we have reviewed have not sufficiently recognised the inherent uncertainties and risks in early estimates.24 This report cited the example of our report Completing Crossrail, where we found decision-making in the latter stages of the project was dominated by achieving a fixed completion date. Some of the decisions taken drove unnecessary cost into the programme.25

Our 2016 report Controlling the consumer-funded costs of energy policies: The Levy Control Framework found that the government failed to fully consider the uncertainty around its central forecasts and define its appetite for the risks associated with that uncertainty. We recommended that the government should understand the possible consequences of its central forecasts being wrong.26

Our 2021 report Initial learning from the government’s response to the COVID-19 pandemic highlighted how the government lacked a script for many aspects of its response, which reduced its ability to respond to the emergency. No script can cover all specific circumstances of every crisis, but more detailed planning of high-impact but low-likelihood events can improve the ability to respond to emergencies.27

How HMT and the Office for Budget Responsibility (OBR) use models to plan and monitor risks

3.12 HMT uses information from departments’ modelling to inform decisions on tax and government spending. It uses this information to produce government-wide budgets for the coming financial year and for spending reviews to set firm expenditure limits for departments. HMT also uses this information to support its own fiscal and balance sheet risk monitoring processes.

3.13 In the weeks leading up to a budget or spring statement, the OBR also collates evidence from departments’ modelling. It uses this information, alongside HMT’s draft costings of tax and spending measures, to produce economic and fiscal forecasts for the UK. It publishes these forecasts in its Economic and Fiscal Outlook, which gives five-year and longer-term forecasts for the UK economy and an assessment of whether the government is likely to reach its fiscal targets. OBR told us it explores uncertainties through its forecasting process and discusses the sensitivities of model outputs with departments. Estimates of sensitivity and uncertainty are presented in this publication using fan charts, sensitivities and scenarios.

3.14 We reviewed the submissions made by departments to HMT and OBR as part of the 2020 Spending Review and other budget events. We found that departments submitted the information HMT and OBR asked them to provide. This included their best estimate and explanation of how they derived the best estimate, along with evidence underpinning the model’s assumptions. Departments do not routinely provide a range of uncertainty around their best estimate in their initial submissions to HMT and OBR.

3.15 HMT and OBR take their own approaches to challenging the information that departments submit:

HMT challenges the assumptions used to derive the estimates. HMT spending teams told us they request further analysis from departments on uncertainty on a case-by-case basis. HMT’s requests for further analysis are risk-based, informed by the issues HMT teams consider most pressing and the nature of the uncertainty that different departments face. For the 2020 Spending Review, HMT told us it had a process for appraising capital projects which looked explicitly at uncertainty in cost estimates. HMT’s teams do not routinely audit or quality-assure the models underpinning the underlying submissions that departments make to HMT. For some significant areas of risk HMT teams run their own model in parallel or carry out other activities to scrutinise a department’s model.

To support the budget, OBR does a review process in four rounds, during which it asks departments to update their models with a certain set of variables. As part of this process, the OBR scrutinises and challenges the information departments provide. The OBR told us it challenges departments on the sensitivity of their forecasts to changes in economic determinants and key assumptions. The OBR also told us it requires information from departments to use in its sensitivity analysis, and that it prioritises scrutiny for those estimates which are most significant. Additionally, the OBR produces a qualitative assessment of uncertainty of policy costings, which is made available to HM Treasury and published.28

3.16 HMT and OBR have a reduced view of uncertainty in the outputs from departments’ models because departments do not routinely provide uncertainty analysis to accompany their best estimates. This reduces their opportunity to prepare for unexpected events. Greater insight from departments about their uncertainty analysis would help to better understand and address these risks.

19 HM Government’s Orange Book defines risk as the effect of uncertainty on objectives.

20 Comptroller and Auditor General, High Speed Two: A progress update, Session 2019-2020, HC 40, National Audit Office, January 2020.

21 Comptroller and Auditor General, Learning for government from EU Exit preparations, Session 2019–2021, HC 578, National Audit Office, September 2020.

22 Uncertainty Toolkit for Analysts in Government. Available at: https://analystsuncertaintytoolkit.github.io/UncertaintyWeb/index.html

23 The Orange Book is produced by the Government Finance Function and HM Treasury. It establishes the concept of risk management and provides a basic introduction to its concepts, development and implementation of risk management processes in government organisations.

24 Comptroller and Auditor General, Lessons learned from Major Programmes, Session 2019–2021, HC 960, National Audit Office, November 2020.

25 Comptroller and Auditor General, Completing Crossrail, Session 2017–2019, HC 2106, National Audit Office, May 2019.

26 Comptroller and Auditor General, Controlling the consumer-funded costs of energy policies: The Levy Control Framework, Session 2016–17, HC 725, National Audit Office, October 2016.

27 Comptroller and Auditor General, Initial learning from the government’s response to the COVID-19 pandemic, Session 2021–22, HC 66, National Audit Office, May 2021.

28 Office for Budget Responsibility, Policy costings uncertainty ratings database, October 2021, available at: https://obr.uk/forecasts-in-depth/policy-costings/

Appendix One

Our audit approach

1 This report examined the roles that HMT, the Office for Budget Responsibility (OBR), the Analysis Function and the Finance Function have in improving modelling across government. We considered how well the principles set out in the Macpherson Review, Managing Public Money and other modelling guidance are embedded across government and applied to business-critical financial models. Our audit approach is based on the National Audit Office’s (NAO’s) Framework to review models. We assessed:

how the responsibility for modelling is organised across government;

the quality assurance processes across government and how organisations provide assurance that models are fit for use; and

how uncertainty is assessed, communicated and taken into account when developing plans.

2 We applied an analytical framework with evaluative criteria, which set out what would be optimal. We reviewed:

the governance structure of responsibility and accountability for business-critical models;

the way in which data, assumptions, methods and calculations of business-critical model are assured; and

how uncertainty is analysed and communicated to users and taken into account when developing plans.

3 Our audit approach is summarised in Figure 14. Our evidence base and methods are described in Appendix Two.

Appendix Two

Our evidence base

1 To reach our conclusions on financial modelling in government, we analysed evidence collected between January and November 2021. All fieldwork took place online.

2 We have not conducted full model audits for this study. We are therefore not able to conclude – or provide assurance – on the reasonableness and robustness of any model reviewed as part of this study. However, we have drawn on evidence from the separate financial audits of five models included in our case studies: COVID-19 guarantees; the self-assessment accrued revenue receivables model; the oil and gas tax revenues forecast; the tax credits error and fraud estimate; and the student loans estimates.

Assessing the governance of business-critical models

3 We reviewed documents including:

the Review of quality assurance of government models (Macpherson Review) and follow-up reports;

the Aqua, Orange, Green and Magenta Books of HM Treasury (HMT) guidance, Managing Public Money and the Government Data Quality Framework; and

templates and guidance issued to departments by HMT for spending review budget bids, and guidance for modelling teams producing forecasts on behalf of the Office for Budget Responsibility (OBR).

4 We sent an online survey to the finance directors of 17 central government departments, of which 15 have arm’s-length bodies (ALBs), to understand the number of business-critical models they have and how they oversee them. The 17 departments consisted of 16 ministerial departments plus HM Revenue & Customs. It excluded seven other ministerial departments: the Attorney General’s Office; the Northern Ireland Office; the Office of the Advocate General for Scotland; the Office of the Leader of the House of Commons; the Office of the Leader of the House of Lords; the Office of the Secretary of State for Scotland; and the Office of the Secretary of State for Wales. Questions covered topics including the department’s management of business-critical modelling, its number of business-critical models, the definition each used to assess if a model is business-critical and the department’s relationship with its ALBs on modelling issues. The survey took place between February and June 2021 and some departments provided updates in November 2021. We received 16 written responses in total and one verbal response via interview. This represented a response rate of 100%.

5 We conducted semi-structured interviews with officials to understand how governance arrangements for modelling and analysis operate in different parts of government. We also used these interviews to identify relevant documentary evidence:

We spoke to officials in HM Treasury, including four spending review teams, those with ownership of relevant guidance, and those responsible for defining HMT’s evidence needs for the budget and spending review processes. We also spoke to officials in Cabinet Office and the OBR.

We spoke to representatives of the Finance and Analysis Functions. We reviewed their functional standards and performance management frameworks where available.

We spoke to analyst-led groups involved in producing guidance aimed at model analysts, in particular the Quality Assurance Working Group, and past and current chairs of the Uncertainty Working Group.

We spoke to teams responsible for modelling assurance and standards in four departments to understand their monitoring activities and guidance. These departments were the Department for Business, Energy & Industrial Strategy (BEIS), the Department for Education (DfE), the Department for Work & Pensions (DWP) and HM Revenue & Customs (HMRC).

Assessing transparency

6 We collected information on publication of business-critical model registers through our survey (see paragraph 4). We also reviewed published information available for a sample of 75 models listed in the BEIS, DfE, DWP and HMRC registers of business-critical models, representing one quarter of each department’s business-critical models on their register. We chose a systematic sample of every fourth model on the registers. For each of these departments, we also reviewed departmental risk registers, reporting on modelling integrity to audit and risk committees, and recent audit completion reports.

Case studies

7 We carried out case studies on nine standalone models and three groups of related models: a total of 12 case studies across four departments. We assessed if business-critical models are adequately assured in practice and if the uncertainty in their outputs is adequately analysed, communicated and factored into decisions and plans. We used this evidence to report on how, where and why working practices differ from best practice and government guidance.

8 For the model or models in each case study, we completed individual model assessments. We designed a framework based on our published Framework to review models and the UK Statistics Authority’s Administrative Data Quality Assurance Toolkit and were informed by the Aqua, Green, and Orange Books of HMT guidance.29,30 The assessment framework covered:

governance and documentation;

technical testing including assurance over methods and calculations;

assurance of inputs; and

analysing, communicating and planning amid uncertainty.

9 We gathered evidence through reviewing model documentation and outputs including related management information. We conducted semi-structured interviews with each model’s developer, quality assurer and senior responsible owner, as well as users of information produced by the models.

10 Our criteria for all case studies were that the models are business-critical, use system-wide assumptions, such as economic forecasts, and are used for financial planning. This includes models used to inform debate on the costs of potential policies as well as models more directly tied to budget bids and financial reporting.

11 We selected models with a range of characteristics, including new and long-standing models and models managed by central government, ALBs and third parties (see Figure 15 for the characteristics of each case study). We reviewed the following models:

BEIS: 2.4% research and development (R&D) model, UK TIMES model, COVID-19 guarantees (covering the Bounce Back Loan Scheme, Coronavirus Business Interruption Loan Scheme and Coronavirus Large Business Interruption Loan Scheme).

DfE: apprenticeships forecast model, advanced learner loans RAB charge model, student loans analytical pipeline (11 models).

DWP: staff demand modelling for working-age benefits administration (five models) and the parallel staff supply model.

HMRC: error and fraud (tax credits) analytical programme modelling, oil and gas tax revenues forecasting model, self-assessment revenue modelling (two models), vehicle excise duty forecast and policy costings model.

Other evidence

12 This report draws on our experience of auditing models across different parts of government over many years. We provide examples to illustrate the challenges we have identified throughout this report and how they have manifested. The summaries in this report reflect our findings at the time of the original report. They do not reflect the current status of each programme or model.

13 We drew on the expertise and experience of an expert panel consisting of Tom Dewar (CEO, Hartley McMaster), John Hopes (Past President of the Operational Research Society), Ruth Kaufman (Former Chair of Government Operational Research Service and former President of the Operational Research Society) and Bob Scott (Board Member, Operational Research Society). The panel provided valuable independent scrutiny and advice by testing our methodology, key evidence and emerging issues.

14 We gathered views from stakeholders including the community of government directors of analysis, Government Actuary’s Department, Government Internal Audit Agency, the Office for National Statistics, Office for Statistics Regulation, UK Statistics Authority and the Bank of England.

29 National Audit Office, Framework to review models, January 2022.

30 UK Statistics Authority, Administrative Data Quality Assurance Toolkit, February 2019, available at: https://osr.statisticsauthority.gov.uk/wp-content/uploads/2019/02/qualityassurancetoolkit_updated_Feb19_2.pdf

Design and Production by NAO Communications Team
DP Ref: 009100-001