Report

by the Comptroller
and Auditor General

Department for Environment, Food & Rural Affairs

Environmental metrics: government’s approach to monitoring the state of the natural environment

HC 1866 Session 2017–2019 16 January 2019

Our vision is to help the nation spend wisely.

Our public audit perspective helps Parliament hold government to account and improve public services.

The National Audit Office scrutinises public spending for Parliament and is independent of government. The Comptroller and Auditor General (C&AG), Sir Amyas Morse KCB, is an Officer of the House of Commons and leads the NAO. The C&AG certifies the accounts of all government departments and many other public sector bodies. He has statutory authority to examine and report to Parliament on whether departments and the bodies they fund, nationally and locally, have used their resources efficiently, effectively, and with economy. The C&AG does this through a range of outputs including value-for-money reports on matters of public interest; investigations to establish the underlying facts in circumstances where concerns have been raised by others or observed through our wider work; landscape reviews to aid transparency; and good-practice guides. Our work ensures that those responsible for the use of public money are held to account and helps government to improve public services, leading to audited savings of £741 million in 2017.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 15 January 2019

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

14 January 2019

In an update to our 2015 briefing, our study evaluates how far the government has an effective system for measuring progress towards its environmental objectives.

© National Audit Office 2019

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.org.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

006038 01/19 NAO

The National Audit Office study team consisted of:
Liam Blanc, James Gjertsen, Jess Hawe, Katy Losse, Munawar Mannan, Shoko Okamura, and Matt Phelps, under the direction of Keith Davis.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Key facts

Summary

1 It is essential for government to have an effective system for measuring its environmental performance, in order to:

understand whether it is on track to meet its long-term environmental goals, including those for air quality, carbon emissions and the natural environment;

assess the effectiveness of new and existing policy interventions; and

fulfil its international obligations on environmental reporting.

Robust data on performance are also essential for Parliament and the public to be able to hold government to account on how it meets its obligations and spends taxpayers’ money.1

2 A good system of environmental metrics is particularly important now, because government has just published its 25-Year Environment Plan. This sets out its ambition to improve the natural environment in England within a generation. Meeting this ambition will require significant and coordinated action across a range of different sectors of the economy, and in some areas, it will involve reversing long-term trends of environmental decline. Strong environmental monitoring is also important in the context of the UK’s exit from the European Union (EU), as it will enable stakeholders, Parliament and government itself to understand the government’s performance against its commitment that leaving the EU will not mean environmental protections are diluted.

3 Government currently collects and reports against a wide range of environmental performance metrics. Key cross-cutting indicator sets include reporting against:

the United Nations (UN) Sustainable Development Goals (the Goals);

international conventions on climate change and biodiversity;

the national Environmental Accounts, a set of satellite accounts to the main UK National Accounts, which measure the contribution of the environment to society, and the impact of economic activity on the environment; and

the Department for Environment, Food & Rural Affairs’ (Defra) internal ‘Defra group scorecard’.

Separate policy areas will also have more detailed metrics, for internal government use, or for external reporting, including reporting related to EU directives.

4 This report sets out our expectations of good practice for an effective system of performance metrics based on our experience of reviewing government approaches to managing performance (Part One). It also examines the government’s current environmental metrics (Part Two) and its plans for developing new metrics (Part Three). We focus on the metrics that relate to England or are UK-wide.

5 We prepared this report in response to a request from the Environmental Audit Committee to update our 2015 briefing on environmental and sustainability metrics. This raised concerns about the timeliness of some of government’s environmental data, these data’s alignment with the government’s objectives, and whether there were enough mechanisms to enable the government to take action if performance was poor.

Key findings

Good practice

6 Good performance metrics are reliable, relevant and cost-effective. If metrics are to be useful, they must give a fair, accurate and timely reflection of performance. It is rarely possible to have perfect performance information, and there is a balance to be struck between what is ideal and what is possible, available and affordable. There can be a legitimate trade-off between the reliability of the data and how quickly it can be produced. But organisations need to carefully consider the cost–benefit case for metrics, ensure quality and timeliness is ‘good enough’ for the circumstances, and be transparent about any caveats and limitations (paragraph 1.4 and Figure 2).

7 Performance information should be at the heart of government’s decision-making. Performance information is only useful if it actually informs decisions in practice, and we have found that this is too often an area of weakness. Performance information should be central to all policy decisions, including the introduction of new environmental policies, increases or reductions of funding for particular initiatives, and the running of established programmes. It allows policy-makers to track whether projects and programmes are achieving their objectives, and so correct delivery problems early, and prioritise effectively. To do this, metrics need to be part of a good performance system. Goals and objectives need to be clear, with appropriate arrangements for reviewing performance data, and mechanisms for ensuring that action is taken if performance is poor. Such arrangements are essential for achieving value for money from metrics: they mitigate the risk that the development of metrics becomes a distraction or a substitute for tackling underlying policy challenges (paragraph 1.5, Figure 3 and paragraph 2.11).

8 Designing the right metrics for environmental policy can be particularly challenging, but it is important to invest properly to get this right. Policy to meet environmental objectives is often complex and cross-cutting. Improving air quality, for example, will require action from a number of government departments (responsible for local government, health and transport as well as the environment) to influence a wide range of people and organisations to make significant and systemic changes. This brings practical challenges for developing a coherent and coordinated approach to tracking progress. However, difficulties are not insurmountable. We would expect policy-makers to develop a clear logic model that shows how they expect their policy initiatives in the short and medium term to influence long-term outcomes, and to measure the progress of these actions, as well as the ultimate outcomes, to help show whether steps are being taken in the right direction. New technology and analysis techniques also bring opportunities to improve coverage and quality alongside timeliness: the use of satellite data can reduce the need for physical inspections, permit near-real-time monitoring and improve the ability to analyse variations across different geographical regions (paragraphs 1.5, 1.6, Figure 4, and 3.10).

9 It is vital to have good, accessible public reporting on performance information. This allows stakeholders to test and challenge the conclusions that decision-makers draw from it. It can also help engage citizens with any behaviour changes required. If citizens have a clear understanding of progress on air quality, for example, it may build support for the switch to low emissions vehicles, walking and cycling (paragraph 1.5 and Figure 5).

Current metrics

10 There are examples of the government’s approach that are good, and which compare favourably with approaches taken in other countries. The UK is one of the countries that has progressed furthest with developing data for a set of globally agreed indicators for the UN Sustainable Development Goals, currently publishing data for 64% of the indicators. The data are published by the Office for National Statistics through an online platform that is transparent and easy to use, and which includes an opportunity for anyone to provide feedback on the indicators, including on new or alternative data sources (albeit by email rather than on the public platform, as in Finland). The Climate Change Act is widely regarded as establishing a robust framework for measuring progress on mitigating and adapting to climate change, and includes statutory responsibilities for an independent organisation (the Committee on Climate Change) to monitor and report on progress, and reporting against metrics on projected as well as current performance on greenhouse gas emissions (paragraphs 2.7, 2.8 and Figure 5).

11 However, the weaknesses we raised in 2015 remain. There remains a patchwork of sets of metrics that do not align clearly with government’s overall objectives or with each other. While the UK publishes data for most of the UN Sustainable Development Goal indicators, not all of these data are up to date. Data for one-third of the published metrics of the most environmentally focused Goals relate to 2015 or earlier. And mechanisms for taking action in response to some sets of metrics are not yet well developed, including for the Environmental Accounts, the Greening Government Commitments for the sustainability of the government estate, and the UN Sustainable Development Goals (paragraphs 2.4 to 2.7, and summary paragraph 19).

12 While government collects and reports a wide range of environmental data, there are some important gaps. We have not carried out an exhaustive review of the completeness of environmental metrics, but there are recognised gaps in government’s ability to measure soil health and the UK’s impact on biodiversity overseas. These issues are difficult to measure, but important to understand, particularly as government develops a new farming policy. Our recent reports have also highlighted some gaps in metrics for individual policy areas. On packaging recycling, we found that the methodology for the main performance metric (packaging recycling rates) was not sufficiently robust, because it did not account for undetected fraud and error. For the Renewable Heat Incentive scheme, the Department for Business, Energy & Industrial Strategy (BEIS) did not have specific goals or clear milestones to monitor progress for one of the scheme’s three objectives (helping the supply chain to develop) (paragraphs 2.9, 2.10 and Figures 10 and 13).

13 Our reports have also raised concern about how effectively metrics are used to inform decision-making in practice. On air quality, we found that Defra and the Department for Transport’s (DfT) joint air quality unit did not systematically oversee performance data on schemes run by other parts of government that include intended benefits to air quality. This meant that there was no clear single responsibility within government for knowing whether the initiatives form a coherent portfolio that delivers good value for money as a whole for air quality. Defra and DfT told us that they agree that this is an important objective, but believe that the arrangements which they currently have in place should be sufficient. For packaging recycling, Defra had not asked important questions about risks and value for money when reviewing performance against the main metric (paragraph 2.11 and Figures 11 and 12).

The future for environmental metrics

14 Government’s plans for a new framework of metrics to measure progress against its 25-Year Environment Plan are promising. On 19 December 2018, Defra published a draft framework of 65 environmental metrics to give an overview of the health of the natural environment, including the quality of air, water and wildlife diversity. Because it takes a whole-systems approach, the framework should help decision-makers understand whether government’s actions as a whole are consistent with its ambition to improve the natural environment within a generation, and it should help to highlight potential interactions between different policy areas. There are also positive signs that the framework will support accessible public reporting on the state of the natural environment, as the draft Environment Bill, would require government to report annually to Parliament on the metrics. Defra told us it plans to make the detailed, constituent data public alongside summary analysis, to promote transparency over underlying trends. There is international good practice that Defra can draw on as it develops this public reporting. For example, Germany produces an annual data report on the environment, which sets out clearly the reasoning behind each indicator and whether progress is on track (paragraphs 3.2 to 3.7 and Figure 14).

15 Defra has more to do to make sure that the framework gives an authoritative position on the state of the natural environment. Defra took the positive step of discussing its draft framework with some stakeholder groups before publication, to understand their views on whether the proposed metrics are the right ones. However, some told us that they were not given enough time to engage fully in assessing them, and Defra has not yet made any changes to the indicators in response to this feedback. Defra has now published the framework for wider public consultation. This only gave stakeholders just over five-weeks to respond, with that time period spanning Christmas and New Year. Although it included a commitment to keep the indicator framework under ‘regular review’. There are also some significant gaps in the data: Defra expects that 23% of the proposed metrics will not be ready until at least December 2019. A further 9% are likely to still need further development after that point, including on soil and sea health. To maximise their impact, environmental metrics should have a spatial element, to identify variations in different geographical areas, but Defra does not expect to publish indicators at sub-national level (paragraph 3.10 and Figure 16).

16 Defra has established arrangements that should encourage its policy-makers to use the new system of metrics to inform decisions. The new framework gives an overview of progress against ultimate ‘outcomes’, but it is also important to track more immediate ‘outputs’ (such as trees planted, or protected areas created). Defra has already started doing this through a programme office, which is responsible for assessing progress on the 230 commitments for action in the 25-Year Environment Plan. In effect, therefore, these 230 actions act as an additional, complementary set of metrics. The programme office reports on a bi-monthly basis to a dedicated board, which in turn reports to a senior environment committee. Defra told us that it plans to incorporate monitoring of progress on the 65 outcome metrics into this process once they are agreed (paragraph 3.6).

17 However, it has not yet done enough to engage other parts of government with its approach, nor to set clear accountabilities for performance. Defra has not yet engaged the DfT, BEIS and the Ministry of Housing, Communities & Local Government (MHCLG), in its oversight arrangements for progress against the 25-Year Environment Plan. This is despite these departments having a significant impact on the natural environment, both through projects intended to improve the environment, and through the potential unintended consequences of business-as-usual. This means there is no clear, single point of ownership for performance as a whole across government on the 25-Year Environment Plan. Defra has also not yet set out expectations for the scale of improvement it expects as measured by each metric over the short and medium term. Less than one-quarter of the 44 targets in the Environment Plan are entirely specific, measurable and time-bound. Defra will also need to establish clear expectations for the contribution of all parts of government to improving performance as measured by these metrics. Without these expectations, there can be an incentive to delay taking action (paragraph 3.10).

18 Defra will also need to manage the risks and opportunities associated with EU exit. EU exit presents a huge challenge for Defra and has created an unprecedented portfolio of work that it needs to deliver. This brings a risk that less immediate issues such as metrics do not get sufficient resource and senior management focus. Also, at the moment, much of government’s environmental monitoring is driven by EU requirements: the UK currently has 161 reporting obligations to European bodies, including the European Commission and the European Environment Agency. Defra will need to ensure that it maintains or improves the quality of this wider environmental monitoring, to supplement the high-level metrics in the new framework. At the same time, EU exit could bring opportunities to review this wider reporting to assess whether it all adds value in relation to UK goals (paragraphs 3.13 to 3.16).

19 It will be important to embed key environmental metrics into government’s core planning and performance monitoring. Single Departmental Plans are the main way that government conducts strategic business planning. The plans are expected to function as comprehensive, costed business plans, setting out each department’s objectives, and how it will monitor performance against them. They must be refreshed annually and require Cabinet Office and HM Treasury approval. The current plans do cover some well-established environmental policies such as climate change commitments and the Greening Government Commitments. However, coverage of the UN Sustainable Development Goals is thin: the plans highlight links between the Goals and existing policies, but do not set out in full each department’s responsibilities for achieving the Goals. The goals relating to the 25-Year Environment Plan were announced in January 2018 but not reflected in the most recent round of Single Departmental Plans across government, although Defra’s internal plan sets out a clear timetable for when it expects to produce key items such as performance metrics to support the plan’s ambitions. HM Treasury and Cabinet Office have issued draft guidance directing Departments to indicate in their 2019-20 Single Departmental Plans where their objectives or work areas support the 25-Year Environment Plan (paragraphs 2.12 to 2.14).

20 The proposed new environmental watchdog needs ‘teeth’ to provide effective scrutiny over environmental metrics. Government is establishing a new environmental watchdog to fill a potential ‘governance gap’ after EU exit, given the role that the European Commission has played in holding government to account for environmental legislation, including on air quality. The draft Environment Bill would establish the watchdog as the Office of Environmental Protection, with an obligation to publish an independent annual progress report on implementation of the 25-Year Environment Plan, and to investigate the compliance of public authorities with environmental law. It would be able to set its own strategy, and would have the power to issue formal compliance notices to public authorities and to apply for judicial review. The watchdog will be funded through Defra, with a chair appointed by the Secretary of State for Defra. While in principle this is not incompatible with it being functionally independent, it could bring risks for its independence in practice or for its perceived independence. To be effective in holding people to account the new organisation will need to have access to good-quality environmental data, and to set clear expectations over how it will use these data to determine whether and how to intervene if performance is not on track. It will also need appropriate resources and strong leadership (paragraphs 3.11 and 3.12).

Conclusion

21 Successive governments have done a lot to raise the profile of environmental issues, and the publication of a 25-year plan for improving the natural environment within a generation could mark a step-change in approach. Government’s draft framework for tracking progress against these environmental ambitions is promising. While significant challenges still need to be overcome, it is encouraging that the initial work is taking a broad, ‘whole system’ view. A critical test will be whether there is strong ‘whole of government’ ownership of the new framework of metrics, with all parts of government actually using this information to monitor progress and take action if performance is not improving as quickly as expected. To enable continuous improvement, it will be important that public reporting on progress is transparent and accessible, to engage the wider community in challenge and public debate. And the new environmental watchdog needs to be demonstrably independent to provide strong external scrutiny.

Recommendations

22 In preparing this report we have mainly sought to raise issues for the government and the Environmental Audit Committee to consider how to address. We do, however, have five direct recommendations, that government should:

a strengthen governance arrangements over environmental metrics, so that there is a single point of responsibility for each set within government. This single point should be responsible for regularly reviewing what the metrics show about performance against government’s objectives, and should have the authority to require action if performance is poor. Given the cross-cutting nature of environmental issues, it should have strong engagement with all government departments. A particular priority is to establish this for the Sustainable Development Goal indicators, and for the outcome metrics for the 25-Year Environment Plan;

b improve accountability for the 25-year plan metrics, by setting clear public expectations for the scale of improvement it expects for each of the ten goals over the short and medium term, supported by a robust internal analysis of how performance reported through the 65 indicators will need to improve to meet these expectations;

c ensure that the breadth of environmental data does not decline without good reason after EU exit. Annual progress reports to Parliament on the 25-Year Environment Plan should include an annex with links to performance data for all the environmental metrics previously reported to the EU (as at exit day), or an explanation of why the UK no longer reports against a particular metric;

d fill data gaps, particularly through greater use of geospatial data, including satellite imagery; and

e strengthen safeguards for the new environmental watchdog’s independence, by setting out how it intends to involve Parliament in choosing its chair and in determining its funding, and by setting out a clear framework document for the terms of the relationship.

1 Comptroller and Auditor General, Accountability to Parliament for taxpayers’ money, Session 2015-16, HC 849, National Audit Office, February 2016.

Part One

Essential elements of an effective system of environmental performance metrics

1.1 This part of the report covers:

good practice in developing a performance framework and performance metrics, drawing on our previous work;

challenges in measuring government’s environmental performance; and

international examples showing how other countries have overcome some of these challenges.

An effective performance framework

1.2 An effective environmental performance framework is essential for government to understand how it is performing against its objectives and to allow it to make informed decisions. Performance metrics form the basis of any effective performance framework because they allow government to:

understand whether it is on track to meet its long-term environmental goals and identify areas where it needs to take further action;

assess how effective particular policies have been and drive improvements; and

demonstrate that it is meeting its international environmental obligations.

Metrics also enable departmental boards and executive committees to understand the key risks and challenges facing government in meeting its environmental goals. They enable Parliament and civil society to better understand environmental issues and hold government to account for its environmental performance.

1.3 The government faces an important challenge in developing a robust environmental performance framework. It published its 25-Year Environment Plan in January 2018 with the stated ambition of improving the natural environment in England within a generation and ensuring that leaving the EU will not dilute current environmental protections.2 The government is also beginning to plan how it will achieve testing international commitments, having signed up to the UN’s Sustainable Development Goals (the Goals) in September 2015. The Goals cover the full range of sustainability issues (including economic and social sustainability) and include a number of environmental targets to achieve by 2030. Given the testing nature of these long-term goals, (Figure 1), it is essential that government quickly puts in place an effective performance framework to monitor and manage its environmental performance.

1.4 In our past reports we have looked at measuring performance over a wide range of policy areas. As a result, we have identified a number of good practice criteria that apply to developing performance metrics in any area. To be genuinely useful in managing performance, metrics need to provide timely and reliable information on an organisation’s main objectives (Figure 2). A key challenge in developing performance metrics and performance data is striking the right balance between what would be ideal and what is possible, available and affordable. Metrics need to be ‘good enough’ to highlight key issues and trends, but the limitations of the reported information need to be clearly understood.

1.5 Performance metrics should be at the heart of government decision-making including the introduction of new environmental policies, increases or reductions of funding for particular initiatives and the running of established programmes. On their own metrics have little value: to get real benefits from performance metrics, they must be part of a wider performance framework. A good performance framework should clearly link performance metrics to the goals and targets they support, and lay out how the metrics will be monitored and how action will be taken if they highlight problems (Figure 3). Transparent and accessible reporting is an important part of a good framework, as it allows stakeholders and the wider public to monitor government’s progress and press for change if needed. This is particularly important when it comes to environmental issues, as improvements often require changes in public behaviour. If citizens have a clear understanding of progress on air quality, for example, it may build support for the switch to low emissions vehicles, walking and cycling. Accessible performance reporting therefore has a role in engaging people outside government and helping to drive change.

The challenges to producing effective environmental metrics

1.6 Developing effective environmental metrics is challenging because environmental policies often seek to address long-term and complex issues. The complexity and variety of these issues often mean that a number of government organisations are involved in implementing policy. Improving air quality, for example, will require action from a number of government departments (responsible for local government, health and transport as well as the environment). Moreover, the influence that government has over environmental outcomes is often indirect and difficult to quantify, especially where the issues are international and the UK cannot address them in isolation. However, these complicating factors are not unique to environmental issues. Government has faced similar challenges in developing useful performance metrics in other policy areas (Figure 4) and there is international good practice that the UK can draw on (Figure 5).

2 HM Government, A Green Future: Our 25-Year Plan to Improve the Environment, January 2018

Part Two

Current environmental performance metrics

2.1 This part of the report covers:

an overview of the existing metrics collected by government;

an update on progress on environmental metrics since our last briefing; and

other progress on improving environmental metrics.

Existing environmental metrics

2.2 Government collects and reports against a wide range of environmental performance metrics (Figure 6). These are a mixture of metrics used to assess progress against domestic policy and metrics used to report against international commitments. There are four main cross-cutting metric sets, covering reporting against:

the United Nations Sustainable Development Goals (the Goals);

international conventions on climate change and biodiversity;

the national Environmental Accounts, a set of satellite accounts to the main UK National Accounts that measure the contribution of the environment to society, and the impact of economic activity on the environment; and

the Department for Environment, Food & Rural Affairs’ (Defra’s) internal ‘Defra group scorecard’.

2.3 Government does not publish an accessible summary of performance against many of these metric sets. Figure 6 provides an overview of the main sets of environmental metrics used by government, Figure 7 summarises assessments of performance for a selection of the sets and Figure 8 summarises stakeholder analysis of progress in implementing the United Nations Sustainable Development Goals.

Update on progress since our last briefing

2.4 In 2015 we produced a briefing for the Environmental Audit Committee summarising the state of the government’s environmental and sustainability metrics. This highlighted that it was good that the measures in place were a mixture of outcome and output based, and so measured both the actions being taken, such as trees planted, and the environmental outcomes, such as carbon sequestered by trees. It was also positive that the metrics sought to give long-term, as well as short-term, assessments of progress. However, our briefing raised the following concerns:

the environmental metrics could be more aligned with a hierarchy of environmental objectives and with each other;

the timeliness of some metrics was mixed, and government could do more to explain whether the time lags were justified; and

some sets of metrics were not associated with a mechanism for action if performance was poor.

2.5 There remains a patchwork of different environmental indicators, published by multiple departments and bodies, that do not clearly align with a hierarchy of objectives (Figure 6). Each set of indicators relates to a different policy or commitment with no clear links to each other or an overarching environmental strategy. This situation is set to change with the introduction of the 25-Year Environment Plan, published in 2018.3 The plan sets out government’s environmental strategy going forwards and takes a more system-level approach, with a new set of indicators pulling together information from a range of sources into a set of 15 headlines that relate directly to environmental goals in the Plan (Part Three).

2.6 We have not seen any significant improvement in the timeliness of the data underlying the metrics. For the Goals, over one third of published data for goals with an environmental focus are from 2016 or earlier (Figure 9).4 For example, the most recent data published for the UK’s ‘material footprint’ (a measure of the primary materials used to provide the goods and services consumed in the UK) and for the proportion of fish stocks within biologically sustainable levels, are five years old, from 2013, although the most recent source data are from 2014 and 2015 respectively. Some of these indicators, such as material flow, do not yet have an internationally established methodology but the majority do. There can be legitimate trade-offs between the timeliness of data and the depth of analysis and quality checks, for example some of the UK indicators differ slightly from the requirements for UN reporting and so additional work is needed to complete them and the source data will generally be available to decision-makers before it is published by the Office for National Statistics (ONS) on its online reporting platform.

2.7 Some metric sets are still not linked to effective mechanisms for taking action if performance is poor:

A wealth of environmental data, including emission statistics, energy consumption and fuel use, are collected and stored within the Environmental Accounts, a subset of the National Accounts. These data are collected in accordance with the System of Environmental Economic Accounting as set by the UN. There is currently no formal mechanism to review and act on the information in the Environmental Accounts as a whole. However, ONS does release a short bulletin that summarises key messages annually and some of the data in the accounts are used in the calculation of the UK’s carbon footprint and the Sustainable Development Goal indicators.

Arrangements for reviewing progress against the Sustainable Development Goals are not yet well-developed. The primary mechanism through which performance will be addressed is annual Single Departmental Plans, but the Goals have not been part of this process for long enough to be able to assess whether they will prove effective (see paragraphs 2.11 and 2.12). The ONS publishes the performance data for the Sustainable Development Goals through an online platform. This has received international interest, it is transparent and easy to use, and it includes the opportunity for anyone to provide feedback on the indicators, including on new or alternative data sources (albeit that this is via an email address rather than comments being visible on the platform itself, as is the case with the Finnish sustainable development indicators, see Figure 5). However, this reporting platform is not formally reviewed by policy-makers. The Department for International Development, with support from Cabinet Office, is leading a voluntary national review of progress against the Goals for the summer of 2019, which should help to raise their profile within government.

The Greening Government Commitments provide another good example where performance data are reported in a clear, transparent manner, but government’s use of the data is inconsistent. For instance, the most recent data show that three departments are missing targets to reduce the proportion of waste sent to landfill by a wide margin (the target is for 10% or less of a department’s waste to go to landfill, yet three departments reported that 35% or more of their waste was going to landfill). Defra was unable to demonstrate that these issues were acted upon effectively.

2.8 However, the Climate Change Act is a good example of a formal mechanism for action and is widely regarded as establishing a robust framework for measuring progress on mitigating and adapting to climate change. For this, regular, formal reviews of progress are required at senior levels in government and it is a requirement to respond to recommendations made by the Committee on Climate Change, an independent scrutiny body. Government must also report to Parliament on projected as well as current performance on greenhouse gas emissions.

Other issues with current metrics

2.9 We have not carried out an exhaustive review of coverage but there are some clear data gaps in some policy areas, including:

soil health, an important metric that provides several societal benefits, is not currently monitored at a national level and will need to be in order to assess progress against a key commitment of the 25-Year Environment Plan (Figure 10 overleaf); and

the UK’s biodiversity impact abroad, metrics for which are still under development. This is important to consider as improvements measured domestically would otherwise not take into account impacts from the UK’s economy in other countries.

2.10 These issues are difficult to measure, but important to understand, particularly as government develops a new farming policy. For example, policies that change food or timber production in England need to be understood in the context of the total, global impact of the country’s requirements for these resources. If policies for environmental improvement locally displace these activities overseas, they may lead to impacts on the environment in other parts of the world.

2.11 Our reports on particular environmental issues since 2015 have also raised concerns about how effectively metrics are used to inform decision-making in practice:

For air quality, government has been subject to a series of legal challenges over whether it has taken adequate action to address its failure to meet its targets for air quality. We found that Defra’s and the Department for Transport’s (DfT’s) joint air quality unit did not systematically oversee performance data on schemes run by other parts of government that included intended air quality benefits. This meant that there was no clear single responsibility within government for knowing whether the initiatives form a coherent portfolio that delivers good value for money as a whole for air quality. Defra and DfT told us that they agree that this is an important objective, but believe that the arrangements which they currently have in place should be sufficient.5

Our analysis of the metrics for air quality for this report has also highlighted that while the UK has clear performance metrics measuring air quality outcomes (i.e. levels of pollutants in the air), it does not have a comprehensive set of supporting metrics on policy ‘outputs’, to provide a more immediate and direct assessment of the progress of individual policies designed to achieve these goals. It has stated goals in the domestic, agricultural, industrial and transport sectors to take measures to improve air quality and measures the total impact of these on pollution levels through the annual emissions inventory. However, government only publishes ‘output’ metrics for some policies in the transport sector (such as take-up of ultra-low emissions vehicles); there are no equivalent published metrics to show how policies in the other sectors are progressing (Figure 11 and Figure 12).

On packaging recycling, we found that the methodology for the main performance metric (packaging recycling rates) was not sufficiently robust, because it did not account for undetected fraud and error. In addition, Defra had not asked important questions about risks and value for money when reviewing performance against this metric (Figure 13).

On the Renewable Heat Incentive, we found gaps in the Department for Business, Energy & Industrial Strategy’s (BEIS’s) monitoring of progress, as it had not set specific goals or clear milestones to measure progress on one of its three objectives for the scheme. The Renewable Heat Incentive (RHI) is a scheme to encourage a switch from fossil fuel heating systems to renewable and low-carbon alternatives in homes and business premises in Great Britain. One of BEIS’s objectives for the scheme has been to grow supply chains which can support a national transition from fossil fuel to low-carbon heating technology from the 2020s. However, BEIS had not established a monitoring plan in support of this objective, and did not include measures on developing the supply chain in its tracking of the scheme’s benefits.6 The Department has committed to publishing clear and specific goals, measures and milestones for developing the low-carbon heating supply chain within the RHI by April 2019.

The link to Single Departmental Plans

2.12 Single Departmental Plans are the primary means by which government conducts strategic business planning. We would therefore expect government’s key environmental objectives to be addressed in the plans and for subsequent performance reporting to link back to these. Departments are required to update their plans annually, setting out their key objectives, and must receive approval from the centre of government (Cabinet Office and HM Treasury). We recently published a report assessing the effectiveness of government’s business planning, which concluded that, despite recent improvements, government business planning does not effectively encourage long-term planning or break down government silos.7

2.13 We have reviewed the published Single Departmental Plans (departments also produce more detailed plans for internal use) and annual reports and note that they do include some coverage of the government’s environmental goals, such as the commitments to reduce greenhouse gas emissions and Greening Government Commitments. However, the plans do not provide convincing evidence that environmental commitments are being used to drive business planning decisions across government. In particular, the Cabinet Office told us that the plans are government’s main mechanism for monitoring the UK’s progress against the UN Sustainable Development Goals, and for prompting action if performance is poor (paragraph 2.7). Yet coverage of the Goals in the latest plans is thin. While the Goals are referenced in several of the plans, these references are to the high-level objectives rather than specific targets and indicators. This means the plans do not give a complete articulation of each department’s responsibilities for achievement of the Goals. Moreover, it is not clear whether departments effectively prioritised policies to address the Goals. Some policies intended to address a particular Goals have only an indirect or limited link to them.

2.14 The mechanisms for overseeing and taking action on the government’s environmental goals through Single Departmental Plans are not yet fully developed. 2018-19 is the first full planning round that the UN Sustainable Development Goals have been included in the process, even though the government signed up to them in 2015 and issued guidance to departments in March 2017. The goals relating to the 25-Year Environment Plan were announced in January 2018 but not reflected in the most recent round of Single Departmental Plans across government, although Defra’s internal plan sets out a clear timetable for when it expects to produce key items such as performance metrics to support the plan’s ambitions. HM Treasury and Cabinet Office have issued draft guidance directing Departments to indicate in their 2019-20 Single Departmental Plans where their objectives or work areas support the 25-Year Environment Plan. Defra and the centre of government have yet to decide how to incorporate the performance metrics into the planning process. These goals will require effective cross-government working, so it is concerning that our recent review of government planning concluded that: “government remains weak at planning and managing delivery when it cuts across organisations”. It is important that Defra and HM Treasury work together to ensure that they are adequately reflected in departments’ plans and that there are robust mechanisms to monitor performance and address issues as they arise.

3 HM Government, A Green Future: Our 25 Year Plan to Improve the Environment, January 2018.

4 Taken as goals 6, 7, 12, 13, 14 and 15, which cover water, energy, resource consumption, climate action and life on land and below water.

5 Comptroller and Auditor General, Air quality, Session 2017-2019, HC 529, National Audit Office, November 2017.

6 Comptroller & Auditor General, Department for Business, Energy & Industrial Strategy, Low-carbon heating of homes and businesses and the Renewable Heat Incentive, Session 2017–2019, HC 779, National Audit Office, February 2018.

7 Comptroller and Auditor General, HM Treasury and Cabinet Office, Improving government’s spending and planning framework, Session 2017-2019, HC 1679, National Audit Office, November 2018.

Part Three

The future for environmental performance metrics

3.1 This part of the report covers:

the government’s plans for new environmental metrics;

risks and opportunities associated with the planned changes; and

the implications of the UK’s exit from the European Union (EU).

Government plans for environmental metrics

25-Year Environment Plan metrics

3.2 As part of the 25-Year Environment Plan the Department for Environment, Food & Rural Affairs (Defra) has developed a draft framework of metrics that it intends to provide a minimum set of measures to cover all key aspects of the environment. It expects to use this framework to monitor progress against the 25-Year Environment Plan, and fulfil the requirements in the draft Environment Bill that place a duty on the Secretary of State to obtain data to monitor improvement in the environment, and to report annually to Parliament on actions taken and changes in the environment.

3.3 There are two sets of indicators, a set of 65 ‘system’ metrics that measure environmental outcomes and a set of 230 ‘actions’ that measure direct outputs. A sub set of the system indicators has been further grouped under 15 headlines that link directly to the goals of the Environment Plan (Figure 14).

3.4 Most of the system indicators are already published, or are similar to metrics that are, while around one-third still need further development before they can be used. It is not intended for the new metrics to replace any of the existing sets, but as the 25-Year Environment Plan supersedes the Natural Environment white paper, it is unlikely that the Natural Environment indicators will continue to be published.

3.5 The outcome metrics and actions of the 25-Year Environment Plan are not intended to be a static set. Developments coming from existing sets of indicators, such as those for the Sustainable Development Goals (the Goals) or biodiversity, are expected to filter into the new set and additional actions from new policies, such as the upcoming Waste and Resources Strategy, will be incorporated into the list.

3.6 Defra has set up a board to oversee the implementation of the actions and goals within the 25-Year Environment Plan, reported to by a programme office responsible for assessing progress. This board meets every other month and receives a progress dashboard covering a subset of priority actions that have been deemed to make the biggest contribution to the goals. A high-level summary of this assessment is reported to the Environment Committee, chaired by the director-general of Environment and Rural at Defra and the chief executive of the Environment Agency, every two months. Currently, just under half of the priority actions are on track for completion, with the remainder at risk of missing their final deadline. Defra told us that it plans to incorporate monitoring of progress on the 65 system indicators into this process once they are agreed.

3.7 Defra plans to generate an annual progress report, which will be laid before Parliament and will communicate progress of the Plan. Both the new outcome indicators and actions are expected to be covered in this, with the first expected to be produced in the first half of 2019.

Other planned changes to metrics

3.8 Outside the 25-Year Environment Plan other work is also being carried out to develop the environmental metrics available to government:

Defra is working with the Office for National Statistics (ONS) to continue development of a set of Natural Capital Accounts, which assess the ecosystem services provided by natural assets, to sit alongside the Environmental Accounts. This is currently due for completion in 2020.

ONS is also working to progress the metrics that sit beneath the UN Sustainable Development Goals. It currently reports data for 157 (64%) of the global indicators agreed by the UN, with disaggregated data for 71 of these.8 This means the UK is further ahead than most other countries: the Netherlands publish data for more indicators (172) but with no disaggregation; France and Germany publish data for 120 and 127 indicators respectively. ONS reports annually on progress in developing the dataset, and has published a plan for filling data gaps and increasing the levels of disaggregation.9,10 It aims to report against 75% of the global indicators by April 2020.

Risks and opportunities

3.9 Given that significant aspects of the 25-Year Environment Plan, such as interim targets and monitoring arrangements, have yet to be finalised, it is too early to conclude on whether it will create an effective performance framework for the government’s environmental ambitions. However, it has several promising features:

The draft ‘system’ metrics have been developed within a logical framework, which means that they are clearly aligned to the goals and targets published in the 25-Year Environment Plan. This should allow for more effective monitoring and clearer reporting. Figure 15 shows how existing performance metrics and new ones that will be published align with the Plan’s main objectives.

The proposed metrics cover both outputs (goods and services that government will deliver) and outcomes (the impact on the environment). This combination should enable government to help assess the effectiveness of its short-term actions and progress against its long-term goals.

Defra told us that it plans to publish both summarised and detailed data, to make the metrics accessible and transparent. Reporting summarised data (for example, by using composite performance metrics that summarise numerous sets of performance data with a single rating), should make it easier for Parliament and the public to interpret performance and assess progress towards environmental goals. More detailed data should allow stakeholders, such as conservation groups and academics, to identify significant trends in the underlying data and challenge government on more detailed aspects of environmental policy. It will, however, be important to make sure that composite performance metrics include the most important data: the proposed air quality ‘headline’ metric covers particulate matter and ammonia, but not nitrogen dioxide, despite the latter being a key area where government has failed to meet targets.

The government has made efforts to link the 25-Year Environment Plan metrics to the data being collected for the UN Sustainable Development Goals. This suggests it is taking positive steps to rationalise and streamline its process for collecting data, which is a weakness of the current system. This should reduce duplication of effort by integrating data collection and reporting against different policies. It should also allow more effective accountability by clarifying the links between performance metrics and environmental targets.

3.10 Our review of the proposed performance framework to support the 25-Year Environment Plan also identified several weaknesses and issues that government will need to address to ensure an effective performance framework:

A significant portion of the goals and targets in the 25-Year Environment Plan are currently too vague to allow government to measure and monitor performance effectively (we assess that less than one-quarter of the 44 targets are entirely specific, measurable and time-bound). While there is a risk that fixed targets can create perverse incentives, it is essential to have a clear idea of what sort of progress is expected by when. If the government does not break down its strategic objectives into clear and measurable long-term and interim goals, it will not be able to use performance data effectively to assess whether it is on track to achieve its ambitions. Defra has committed to “explore options” for including additional cross-cutting targets for environmental improvement, as part of the new framework for environmental planning, monitoring and reporting to be established by the forthcoming Environment Bill.

The ambitions in the plan will need effective inter-departmental working and active involvement from other departments with the metrics to achieve this. Defra told us that it has taken steps to engage other departments with the metrics including speaking to experts in the Department for Business, Energy & Industrial Strategy (BEIS) about climate data. However, it is not clear that this was sufficient to ensure that all departments had the opportunity to contribute to developing the metrics and to maximise the opportunity to draw on the significant amounts of data that other departments collect with environmental implications. Government has also not established clear accountability for other departments’ progress as measured by the metrics, both in terms of policies with intended environmental benefits, as well as wider policies that risk environmental harm. No department (other than Defra) is represented on the implementation board that oversees progress on the 25-Year Environment Plan (paragraph 3.5). This means there is no clear, single point of ownership for performance as a whole across government on the 25-Year Environment Plan. Other key cross-government risks and priorities, such as housing, modern slavery and the ageing society, are coordinated through Cabinet-level implementation taskforces and HM Treasury groups. There is an Inter-Ministerial Group for Environment and Clean Growth, which Defra chairs jointly with BEIS, but government has been unable to tell us how often, if at all, the Group had met.

In several areas of the 25-Year Environment Plan metrics will not be ready by December 2019, and in others existing metrics will continue to be used, at least in the short term, even though the Department does not consider them to be fully adequate (Figure 16). Defra estimates that one-quarter of the proposed metrics will not be ready until at least December 2019. A further 9% are likely to still need further development after that point. Defra has told us that significant work still needs to be done on measuring the health and diversity of our seas and on measuring the UK’s environmental impact abroad.

To maximise their impact, it is important for environmental performance metrics to have a spatial element – that is to say, that the data can be disaggregated sufficiently to identify whether there are significant variations in different geographical areas. Without this spatial element, there is a risk that the metrics will not highlight cases where there are significant regional issues and problems that need to be addressed. For example, government data on air quality is grouped into geographical zones, but does not show whereabouts within each zone breaches of air pollution limits occur. How significant a breach is in terms of public health depends very much on where this poor air quality occurs: if it is in a densely populated urban area it has much more serious implications than if it occurs away from a major population centre, but this cannot be determined from the published data. However, while Defra told us that it wanted to publish the performance data for the 25-Year Environment Plan at the greatest spatial resolution possible, some data sets (for instance, those based on national sample surveys) will not be collected in such a way to make this level of disaggregation available.

Defra acknowledges that the system of performance metrics will need to be flexible and plans to make use of new data collection techniques and technology as they become available. Satellite data are becoming increasingly widespread in private and public sectors. The benefits this can provide include cost savings, through reductions in physical inspections, near-real-time monitoring and improved spatial disaggregation. Defra and its arm’s-length bodies are already looking to increase the use of satellite data in some of their activities. The Rural Payments Agency (RPA), which pays subsidies to farmers, has been increasing remote checks on farms using open source satellite data each year while reducing the number of physical farm inspections it carries out. Defra is interested in exploiting geospatial data further in monitoring environmental land management schemes and a number of areas linked to the 25-Year Environment Plan, such as mapping habitats and woodland change.

Defra shared the draft framework of indicators with some stakeholders before publication, which is a positive step, although some told us that there was little time for them to comment on the proposed metrics, which prevented them from engaging with them fully. Defra has not yet made any changes to the indicators in response to this feedback. Given that a significant amount of environmental data is collected by non-government groups and that stakeholders have sector expertise, a formal consultation that allowed stakeholders adequate time for a considered response would have been useful to ensure that the performance metrics for the plan are as robust as possible. Defra published the framework for wider public consultation on 19 December 2018. It gave stakeholders just over five weeks to respond, with that time period spanning Christmas and New Year, although it expects to keep the indicator framework under “regular review”.

3.11 In May 2018, Defra announced plans to create a new, “world-leading, statutory and independent” environmental watchdog to hold government to account on its environmental ambitions and obligations after the UK leaves the EU. The European Union (Withdrawal) Act 2018 states that the body will have the power to take “proportionate enforcement action (including legal proceedings if necessary) where it considers that a Minister of the Crown is not complying with environmental law”. The draft Environment Bill, published in December 2018, sets out more detailed requirements and arrangements for the new body, including that it:

is known as the Office for Environmental Protection;

must publish an independent annual progress report on implementation of the 25-Year Environment Plan, and investigate the compliance of public authorities with environmental law;

has the power to set its own strategy, to issue formal compliance notices to public authorities and to apply for judicial review, but not to issue fines; and

its chair and non-executives are appointed by the Secretary of State for Defra, and it is funded through Defra.

3.12 A strong and independent watchdog is vital in ensuring that environmental performance metrics are used effectively. A regulatory body can have significant influence by providing external scrutiny to highlight key issues raised by performance data. While in principle funding and oversight through a parent department are not incompatible with a body being functionally independent, the proposed arrangements for funding, and for appointment of the Chair, could bring risks for the watchdog’s independence in practice or for its perceived independence. To safeguard its independence it will be important that the watchdog:

has sufficient resources to develop expertise and maintain awareness of the full policy portfolio;

the discretion to determine its own programme of work and to respond to emerging environmental issues; and

leadership with the right experience, seniority and experience to implement a strong culture of independence.

Implications of EU exit

3.13 The European Environment Agency (EEA) currently lists the UK as having 161 reporting obligations to itself, the European Commission and EUROSTAT. The majority of these obligations relate to specific EU directives, and the structure by which these data are handled after EU exit will depend on the UK’s future relationship with the EEA. The impact this will have on environmental monitoring is complex, as some of the reported data, such as on recycling rates or emissions, is also collected for domestic purposes or for reporting on international obligations. The Draft Withdrawal Agreement states that both the EU and the UK will ensure that the level of environmental protection is not reduced below the level provided by current standards in relation to the access to environmental information.

3.14 Although there is a risk that some data would no longer be collected if it is not required to be reported to the EU, there may also be the opportunity to simplify or innovate dataflows that are currently complex or include perverse incentives. Stakeholders have identified waste and recycling reporting as an example: recycling metrics are weight-based, which means there is an incentive to recycle more dense materials rather than lower-density materials such as plastics.

3.15 The new environmental body will be crucial in replacing the enforcement capacity of the European Commission. The threat of legal action from the Commission has, for example, been an incentive for the government to take action on air quality, having failed to meet EU concentration limits for nitrogen dioxide concentrations (Figure 12). If the new environmental body is unable to provide an equivalent incentive, there is a risk that areas of poor performance will not receive adequate funding or attention.

3.16 EU exit has significantly increased the portfolio of work Defra is required to deliver.11 Defra cites the transfer of resources to EU exit work or links to the new bills required by EU exit as the reason for delay for the majority of priority actions for the 25-Year Environment Plan that are not on track.

8 Data can be disaggregated by sex, race, religion, geography, disability, ethnicity, migrant status, age or income quintiles. This is important because in agreeing the Goals member states agreed that they should be achieved for all segments of society, with a commitment to “leave no-one behind”.

9 The ONS’s first annual report on the progress made towards measuring the global Goals in the UK was published in November 2017 and is available at: www.ons.gov.uk/economy/environmentalaccounts/articles/sustainabledevelopmentgoalstakingstockprogressandpossibilities/november2017#global-indicators, with a second annual report in November 2018. Available at: www.ons.gov.uk/economy/environmentalaccounts/articles/sustainabledevelopmentgoalstakingstockprogressandpossibilities/november2018

10 Inclusive data charter action plan. Available at: www.ons.gov.uk/economy/environmentalaccounts/methodologies/inclusivedatacharteractionplanfortheglobalsustainabledevelopmentgoals

11 Comptroller & Auditor General, Department for Environment, Food & Rural Affairs, Progress in Implementing EU Exit, Session 2017–2019, HC 1498, National Audit Office, September 2018,

Appendix One

Our audit approach

1 See Figure 17.

Appendix Two

Our evidence base

1 Our review of environmental metrics is based on evidence gathered in October and November 2018. Our audit approach is outlined in Appendix One.

Interviews

2 We conducted semi-structured interviews with stakeholders to understand their view on existing sets of environmental and sustainability metrics, what they thought to be good practice for environmental metrics, the government’s performance and the current mechanisms for action that government has. We also used these interviews to identify relevant documentary evidence. We spoke to representatives including those from: Royal Society for the Protection of Birds (RSPB), UK Environmental Law Association (UKELA) and the Committee on Climate Change.

3 We also conducted interviews with other staff across government, including various staff from the Department for Environment, Food & Rural Affairs, the Department for International Development, the Department for Business, Energy & Industrial Strategy and the Cabinet Office. In addition, we consulted the Welsh Future Generations Commissioner. We also spoke to representatives from the Supreme Audit Institutions of Finland, The Netherlands, Germany and Canada, as well as from the Welsh Future Generations Commissioner, to find further examples of good practice the UK could borrow.

Document review

4 We reviewed documentation from the departments and stakeholders and documentation in the public domain. This included Single Departmental Plans, reports prepared by the Environmental Audit Committee, online publications of metrics currently available and relevant published National Audit Office reports. We used this documentary evidence to understand the sets of metrics that are currently collected by government and to review the mechanisms for action where performance is poor.

Design and Production by NAO External Relations
DP Ref: 006038-001