Report

by the Comptroller
and Auditor General

Ofwat, Ofgem, Ofcom and the Financial Conduct Authority

Regulating to protect consumers in utilities, communications and financial services markets

HC 1992 Session 2017–2019 20 March 2019

Our vision is to help the nation spend wisely.

Our public audit perspective helps Parliament hold government to account and improve public services.

The National Audit Office scrutinises public spending for Parliament and is independent of government. The Comptroller and Auditor General (C&AG), Sir Amyas Morse KCB, is an Officer of the House of Commons and leads the NAO. The C&AG certifies the accounts of all government departments and many other public sector bodies. He has statutory authority to examine and report to Parliament on whether departments and the bodies they fund, nationally and locally, have used their resources efficiently, effectively, and with economy. The C&AG does this through a range of outputs including value-for-money reports on matters of public interest; investigations to establish the underlying facts in circumstances where concerns have been raised by others or observed through our wider work; landscape reviews to aid transparency; and good-practice guides. Our work ensures that those responsible for the use of public money are held to account and helps government to improve public services, leading to audited savings of £741 million in 2017.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 18 March 2019

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

15 March 2019

This report assesses how well Ofwat, Ofgem, Ofcom and the Financial Conduct Authority (FCA) measure and report their performance in protecting the interests of consumers.

© National Audit Office 2019

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.org.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

006342 03/19 NAO

The National Audit Office study team consisted of:
Simran Nijjar, Ivan O’Brien and Rich Sullivan-Jones, with assistance from Alice Broughton, Naimah Choudhury and Peter Langham, under the direction of Charles Nancarrow.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Key facts

Summary

1 UK households spend a total of around £140 billion a year in bills on water, energy and telecommunications, and fees and charges in financial services. These sectors provide services that are critical for security, well-being and social participation, which consumers purchase directly mostly from private companies. Each sector is overseen by a regulator to ensure that services are provided in a way that meets public policy objectives, including that markets work well for consumers.

2 The four main regulators of these sectors – Ofwat, Ofgem, Ofcom and the Financial Conduct Authority (FCA) respectively – were set up to be directly accountable to Parliament, independently of government. Their long-term high-level objectives are set out in statute, including a primary statutory duty to protect the interests of consumers. The regulators vary substantially in size, reflecting differences in their remits and the size, complexity and features of each sector. Between them, they have annual running costs totalling £855 million.

3 The regulators each have individual functions and objectives to protect the interests of consumers. These include, for example: promoting competition to encourage fair prices and meaningful choice for consumers; setting rules and limits on the prices that providers can charge where competition is not sufficient; ensuring that services are provided to an adequate standard (such as reliable broadband connections or water supply); and preventing unfair practices such as unfair terms and conditions.

4 Government, Parliament and consumer representatives have expressed concerns about whether these sectors are working as well as they can for consumers, raising questions about the effectiveness of the regulators. They have highlighted a range of issues, including long-term above-inflation price rises in utilities, difficulties for consumers in accessing the best deals, poor quality of services, and high costs in consumer credit. These issues can have significant impacts on people. Government has launched or commissioned various reviews into whether regulation is proving effective, including the 2017 ‘Regulatory Futures Review’ and an ongoing regulation study by the National Infrastructure Commission.

5 In protecting the interests of current and future consumers, regulators have to consider the often-competing needs of different stakeholder groups. For example, companies will be seeking to maintain and maximise shareholder value, while government departments will be concerned that regulators support other policy objectives such as financial stability, sustainability of resources or security of supply. The outcomes regulators are seeking to achieve can also be affected by a range of factors over which they have limited influence, such as droughts or changes in the wholesale cost of energy.

6 Faced with these challenges and differing views over their effectiveness, it is vital that regulators measure and report transparently their intentions and achievements in meeting their duties towards consumers. This means they need to ensure that they:

set out clearly their intended consumer outcomes, how they have dealt with competing incentives such as those of consumers and industry stakeholders, and any barriers or constraints they face in delivering their outcomes;

examine whether they are achieving their intended outcomes and take corrective action where necessary; and

demonstrate credibly to Parliament and other stakeholders how well they are discharging their duties and addressing the key issues for consumers.

7 In 2016, we published Performance measurement by regulators, a guide setting out best practice for regulators in measuring and reporting their performance. This was based on existing guidance, our experience from past work and input from a wide range of regulators. The guide also recognised that measuring performance is challenging for regulators, because they do not have direct control over their intended outcomes, and these are often achieved over the long term.

Scope of this report

8 This report assesses how well Ofwat, Ofgem, Ofcom and the FCA measure and report their performance in protecting the interests of consumers. We assess them against the principles set out in our 2016 guidance on performance measurement, as well as other recent work we have done on accountability and reporting in government. In particular, we examine:

whether the regulators have good insight into the key issues that consumers face, and clearly define what they want to achieve and the barriers they face in doing so (Part Two);

how well the regulators measure their own performance, understand the extent to which they are able to influence consumer outcomes, and use this information to improve their own effectiveness (Part Three); and

whether the regulators report performance publicly in a way that is useful for Parliament and other stakeholders to hold them to account (Part Four).

9 The report does not evaluate the regulatory performance of each regulator, and it does not focus on their other statutory objectives.

Key findings

Focusing on what matters to consumers

10 All four regulators have good insight into the key consumer issues in their sectors, based on their own research and consultation with other stakeholders. To focus on what matters to consumers, regulators need a clear understanding of the problems they are seeking to address. We found that each regulator routinely engages with the statutory consumer representative and complaints bodies in its sector, and conducts its own surveys, research and data monitoring. For example, Ofgem, Ofcom and the FCA conduct regular large-scale surveys into consumers’ experiences. The regulators use their consumer insight to identify key issues and set priorities, such as the FCA’s annual sector views which use data from a number of sources to analyse the actual and potential harm that consumers face in each market it regulates. We also found that the regulators share insights into consumer issues, such as the particular challenges that vulnerable consumers face, including through the UK Regulators Network (paragraphs 2.3 to 2.5).

11 The available evidence shows that consumers are facing significant issues across all four sectors. We found that the regulators are seeking to address a number of common challenges for consumers, including affordability concerns, service failures and challenges for vulnerable consumers (paragraphs 2.6 to 2.10):

Affordability and debt. The most common problem that consumers seek help from Citizens Advice within all four sectors is dealing with debt associated with paying bills and credit commitments. Affordability concerns play out differently in individual markets, but people who fall behind on their bills can often struggle across multiple services. For example, we reported in 2017 that of those seeking help with debt problems, 32% had issues in two of the four sectors and 11% in three of the four. These concerns are set against a backdrop of rising prices, including real-terms rises of 28% for gas, 37% for electricity and 6% for water since 2007.

Difficulties accessing the most appropriate deal or service, particularly for vulnerable consumers. Customers who do not switch provider typically pay more for the same service as new customers – in 2018, Citizens Advice estimated that this ‘loyalty penalty’ totals £4.1 billion a year across a range of communications and retail financial services markets. Vulnerable consumers are likely to be particularly badly affected. For example, Ofgem has found that elderly and lower social grade consumers are a third less likely than the average consumer to switch energy supplier to get a cheaper deal. The FCA has also identified that consumers in deprived areas of the country are 70% more likely to use an unarranged overdraft, which are twice as expensive as arranged overdrafts, than those in less deprived areas.

Service failures or poor quality of service. We found issues in all sectors. For example, leakage increased by 3% across the water sector between 2015-16 and 2017-18, and 36,000 homes were left without water supply for more than a day following the severe cold weather in early 2018. Ofcom found in 2018 that 15% of broadband customers had reason to complain about their service, and that by far the most common cause was connection problems.

12 Regulators have not been specific enough in defining the overall outcomes they want to achieve for consumers. Clear success criteria such as outcomes-based targets are vital so that industry, consumers and consumer representatives are clear on regulators’ expectations and priorities and how these address consumers’ key areas of concern. Regulators set broad high-level aims, such as achieving high-quality and good-value services for consumers. However, apart from targets in some specific areas, they have not defined what these high-level aims mean in practical terms (for example, with market-wide targets or other success measures), such as what level and distribution of prices or service reliability would demonstrate either good progress or a need to take further action. During our study, the regulators raised concerns that, in some areas, specific targets can have negative unintended consequences. However, they also acknowledged that there is value in being more specific about what they are trying to achieve. The regulators also do not have a joined-up way of setting objectives for issues that cut across sectors such as affordability and debt (paragraphs 2.11 to 2.15).

13 Regulators find it difficult to manage the trade-offs they face between competing objectives in protecting consumers. To meet their objectives, regulators need to manage a number of potential conflicts or trade-offs between objectives or groups of consumers, or areas where their powers are limited. Our recent report on vulnerable consumers found that some measures to promote a competitive market, which reduces prices for consumers who switch to the best deals, can conflict with objectives to protect those in vulnerable circumstances who are less likely to switch and therefore benefit from cheaper prices. It concluded that regulators and government need to be clearer about their respective responsibilities. There are areas where government formally provides direction or strategic steer, for example, introducing legislation requiring regulators to introduce price caps or universal service obligations. However, regulators report that determining how to manage many of these trade-offs remains challenging (paragraphs 2.16 to 2.18).

Measuring and improving performance

14 Regulators monitor data on consumers’ experiences and outcomes, but do not routinely use this information to assess their own performance. We found that regulators’ boards and management teams are actively using performance information to assess progress and focus on corrective action where necessary. This information focuses on project-based measures, for example progress with introducing new regulatory rules, the status of ongoing investigations, or the number of successful enforcement cases. The regulators also track a range of outcomes data, such as prices, quality of service and levels of consumer satisfaction. However, this is not typically underpinned by a clear set of detailed indicators, attributable to the regulator, which makes it difficult for either the management or the board to assess how the regulator is performing against its overall consumer objectives (paragraphs 3.4 to 3.10).

15 None of the regulators has yet developed a good overall understanding of its influence over consumer outcomes and how to exert this through different regulatory approaches. To meet their objectives, regulators rely to varying degrees on the behaviour of consumers, providers and other stakeholders. For example, consumers switching to the best deal should stimulate provider competition, and thereby lower prices and improve quality. Although difficult, regulators need to understand the extent to which they are able to influence behaviours and the resulting outcomes to make sure they intervene in the most cost-effective way. No regulator has yet made significant progress with mapping or evaluating their overall influence on outcomes, although the FCA has begun work to understand its impact and influence and has evaluated the direct impact of three recent interventions. While these require investment of resources, the FCA considers the results extremely valuable to building a picture of its influence and informing future decisions (paragraphs 3.11 to 3.14).

16 The regulators are working to improve how they measure their performance, but have further to go to be able to do so robustly. All four regulators have recognised the need to improve how they measure their performance in protecting the interests of consumers, and are committing resources to this. The regulators are at various stages of maturity and have made particular progress in different areas, but all have further to go. As the largest of the regulators, the FCA has been able to prioritise resources to evaluating its impact, and has consulted publicly on a new overall approach. It is currently working to implement this new approach and recognises that it has more to do. Ofcom and Ofgem have also made some progress in monitoring a structured set of indicators for each of their high-level intended consumer outcomes. Ofwat is at an earlier stage in the process overall, and has recently started developing a new sector-wide approach that it intends to complement its existing approach to target setting and performance monitoring for individual companies (paragraphs 3.15 and 3.16).

Reporting of consumer outcomes

17 Regulators’ public reporting does not provide a meaningful overall assessment of how well they are protecting consumers’ interests. The only place that regulators currently provide an overall assessment of performance is their annual reports. However, these do not consistently provide a clear line of sight between what they are trying to achieve, what they have done to meet these objectives, and what the actual outcomes are for consumers. Performance information is also not presented in a consistent way, meaning that year to year comparisons are not possible, and there are few benchmarks (such as expected outcomes) against which to assess progress. Consumer representatives and other stakeholders told us that regulators’ reporting would benefit from clear benchmarks and consistency from year to year (paragraphs 4.11 to 4.13).

18 Consumer representatives find regulators’ publications on specific interventions and issues useful. Our 2016 guide recommended that regulators engage with users of performance information to understand what is most useful to them. Consumer representatives, in particular, use regulators’ publications to hold them to account. Most stakeholders we interviewed, including consumer representatives, reported that the regulators engage well to understand their interests, and provide useful information on particular interventions or consumer issues. Specific examples included reports and data from Ofgem and Ofwat on vulnerability and affordability, and Ofcom’s progress reports on the roll-out of superfast broadband. However, some also highlighted gaps, including a lack of benchmarks against which to measure success, and a lack of clarity on why a certain approach was chosen over alternative options (paragraphs 4.4 to 4.7).

19 Regulators find it challenging to articulate what difference they are making to the performance of each sector. The performance of a market or sector is also influenced by other factors. For example, government policy can increase prices for consumers (such as environmental protections or government mandated projects like the roll-out of smart meters) by increasing providers’ costs, as can changing economic conditions. However, price rises may also occur if the regulator is not doing enough to keep costs and prices competitive. Regulators need to make the distinction so that there is clarity about what they can be held accountable for and what they have less control over. All four regulators have and use powers to request data from providers and report on how each market, section of a market or individual company is performing in priority areas such as prices, complaints, service disruptions, levels of switching or customers in arrears. However, regulators could improve sector reporting by, for example, supplementing it with assessments of their own performance (paragraphs 4.8 to 4.10).

20 Regulators have no common set of standards for what or how to report. Unlike for financial reporting, regulators are not subject to specific performance reporting requirements on consumer outcomes. While we would expect each regulator to report different information relevant to their role and sector, common standards would prompt reporting based on agreed best practice and make the information more accessible and consistent for users of the reports. We reported in 2018 that central government departments had found that efforts to improve reporting based on a common approach have helped them to clarify objectives and internal accountabilities and take a more professional approach to planning. Regulators have not developed equivalent standards or principles with which to drive improvements to their reporting (paragraphs 4.14 to 4.16).

Conclusion on value for money

21 Consumers of regulated services are facing a number of significant difficulties, from rising bills to the impact of service failures. The regulators who have statutory duties to protect consumers in these sectors all face common challenges in meeting their objectives. They have to balance the often-competing needs of consumer and provider interests, alongside other duties covering issues such as sustainability, security of supply, or financial stability, and they often have only limited influence over outcomes. In the context of concerns over the ability of regulators to protect consumers it is imperative that they are clear and specific about the outcomes they are seeking to achieve and are transparent in reporting their performance.

22 The regulators in this review have good insight into consumer concerns and issues. However, they are not sufficiently specific and targeted in setting out what overall outcomes they want to achieve for consumers, and therefore what information they need to evaluate and report on their overall performance robustly. Regulators are all taking steps to improve how they define, measure and report their performance in protecting consumers, but all have further to go to do so in a meaningful way. Until they achieve this, they will not be able to give consumers confidence that they are providing value for money, or adequate overall levels of protection for those who need it.

Recommendations

23 Regulators should:

a Do more to translate their high-level intended consumer outcomes into what this means in practical terms. This should be underpinned by detailed indicators or targets that, where possible, are attributable to their regulatory performance, and that can be used to measure performance in protecting the interests of consumers. It is up to regulators to determine where targets may not be appropriate (for example where they are likely to lead to unintended consequences), and to demonstrate how they plan to measure success and drive improvement.

b Work together, for example through UK Regulators Network, to develop a consistent approach to modelling and measuring regulatory influence and impact, and providing links between regulatory activities, outputs and consumer outcomes.

c Ensure that their public reporting of performance includes clear benchmarks, trends over time and analysis of the underlying factors affecting outcomes.

24 Regulators should work in consultation with government and other key stakeholders to:

d Ensure there is clarity over how best to manage trade-offs between regulatory objectives or groups of consumers, particularly in areas that overlap with government policy such as affordability and vulnerability.

e Develop principle-based standards for public reporting of regulatory performance in annual reports or other regular publications. This should include consideration of how regulatory performance can be distinguished from other impacts, such as direct government interventions which affect consumer costs and outcomes.

f Set common expectations and standards for evaluating significant regulatory decisions and interventions, particularly where regulators cannot afford to do this within existing resources.

While it is not possible to put a timescale on these recommendations, we would expect to see clear signs of progress with each annual reporting round.

Part One

Introduction

1.1 UK households spend a total of around £140 billion a year on bills in water, energy and telecommunications, and fees and charges in financial services.1 These sectors provide services that are critical for security, well-being and social participation, which consumers purchase directly mostly from private companies.

Regulators

1.2 Each sector is overseen by a regulator to ensure that services are provided in a way that meets public policy objectives, including that markets work well for consumers. The four main regulators of these sectors – Ofwat, Ofgem, Ofcom and the Financial Conduct Authority (FCA), respectively – were set up to be directly accountable to Parliament, independently of government. Their long-term high-level objectives are set out in statute, including a primary statutory duty to protect the interests of consumers (Figure 1).

1.3 The four regulators vary substantially in size, have different remits and oversee sectors of varying sizes, features and complexity. Between them, they have annual running costs totalling £855 million (which also includes other functions). The FCA is the largest with an annual spend of more than £500 million (Figure 2). Regulators are funded mainly by the sectors they regulate, for example through licence fees that companies pay to be allowed to operate. The costs of regulation are ultimately largely paid by consumers, as they are reflected in the prices that companies charge.

1.4 To protect the interests of consumers, regulators have a number of functions and objectives set out in statute at a high level. Regulators have discretion to interpret these high-level objectives when designing their regulatory approaches and establishing what outcomes they are seeking. The objectives are different for each regulator, and include:

promoting effective competition between providers in the interests of consumers (for example, to encourage fair prices and meaningful choice);

setting rules or limits on the prices that providers can charge where competition is not sufficient (for example, in natural monopolies where it would be inefficient to have more than one utility supply to every household);

ensuring that services are provided to an adequate standard, such as reliable broadband connections or water supply; and

preventing unfair practices, such as unfair terms and conditions.

1.5 In protecting the interests of current and future consumers, regulators have to consider the often-competing needs of different stakeholder groups. For example, companies will be seeking to maintain and maximise shareholder value, while government departments will be concerned that regulators support other policy objectives such as financial stability, sustainability of resources or security of supply. The outcomes regulators are seeking to achieve can also be affected by a range of factors over which they have limited influence, such as droughts or changes in the wholesale cost of energy.

Concerns over consumer issues

1.6 Consumer representatives regularly express concerns about whether these sectors are working as well as they can for consumers. They have highlighted a range of issues, such as long-term above-inflation price rises in utilities putting pressure on affordability, difficulties for consumers in accessing the best deals, poor quality of services, and high costs in some types of consumer credit. These issues can have significant impacts on people (Figure 3).

1.7 Government and Parliament have also highlighted a range of consumer issues in regulated sectors. For example, the government’s 2018 green paper on consumer markets highlighted that, while consumer satisfaction is high in some of the regulated markets, bank accounts, water, internet and gas are ranked in the bottom 10 out of 34 UK consumer markets, and all lower than EU averages.2 The Parliamentary select committees that monitor the four sectors have between them held 15 inquires in the past three years that raised concerns about the consumer experience. In the same period, the Committee of Public Accounts has similarly reported its concerns on a number of issues including energy bills, financial services mis-selling and consumers’ exposure to online fraud.

1.8 Government has in the past few years launched or commissioned various reviews into whether regulation is proving effective in light of future challenges, including the 2017 ‘Regulatory Futures Review’ and an ongoing regulation study by the National Infrastructure Commission. The Competition and Markets Authority has also conducted recent reviews into the banking and energy markets, which reported in 2016 that competition between providers was not working well in the interests of all consumers.

Performance measurement

1.9 Faced with these challenges and differing views over their effectiveness, it is vital that regulators measure and report transparently their intentions and achievements in meeting their duties towards consumers. This means they need to ensure that they:

set out clearly their intended consumer outcomes, how they have dealt with competing incentives such as those of consumers and industry stakeholders, and any barriers or constraints they face in delivering their outcomes;

examine whether they are achieving their intended outcomes, and take corrective action where necessary; and

demonstrate credibly to Parliament and other stakeholders how well they are discharging their duties and addressing the key issues for consumers.

1.10 In 2016, we published a guide setting out best practice for regulators in measuring and reporting their performance.3 This was based on existing guidance, our experience from past work and input from a wide range of regulators. The guide also recognised that measuring performance is challenging for regulators. The outcomes they are seeking often do not become evident for a number of years, and are affected by a range of factors over which the regulators have limited influence. Such factors include the conduct of regulated providers, consumer behaviour and the impact of government policy decisions (Figure 4).

Scope of this report

1.11 This report assesses how well Ofwat, Ofgem, Ofcom and the FCA measure and report their performance in protecting the interests of consumers. We assess them against the principles set out in our 2016 guidance on performance measurement, as well as other recent work we have done on accountability and reporting in government. In particular, we examine:

whether the regulators have good insight into the key issues that consumers face, and clearly define what they want to achieve and the barriers they face in doing so (Part Two);

how well the regulators measure their own performance, understand the extent to which they are able to influence consumer outcomes, and use this information to improve their own effectiveness (Part Three); and

whether the regulators report performance publicly in a way that is useful for Parliament and other stakeholders to hold them to account (Part Four).

1.12 The report does not evaluate the regulatory performance of each regulator, and it does not focus on their other statutory objectives.

1 This report only covers household and retail consumers, and does not cover businesses.

2 Department for Business, Energy & Industrial Strategy, Modernising Consumer Markets: Consumer Green Paper, April 2018.

3 National Audit Office, Performance measurement by regulators, November 2016.

Part Two

Focusing on what matters to consumers

2.1 Regulators’ statutory objectives to protect consumers are high-level and general in nature. There are other bodies in each sector with a consumer role (each sector also has independent consumer complaints organisations, as well as a statutory consumer representative), but it is the regulators that have both the statutory responsibility to protect the interests of consumers and the powers to intervene to discharge these responsibilities. To do this effectively, regulators therefore need to understand what issues consumers are facing, establish what they want to achieve in addressing those issues, and identify what barriers or limitations will affect their performance in achieving their desired outcomes.

2.2 This part examines:

whether the regulators have good insight into the key issues that consumers face, and the evidence on what these are; and

whether regulators have defined clearly the outcomes they want to achieve for consumers, and have identified the barriers or limitations that may affect their performance.

Regulators’ insight into consumer issues

2.3 To focus on what matters to consumers and develop clear objectives, regulators need a good understanding of the problems that provide the rationale for regulation. To prioritise resources on issues that will bring the biggest benefit, they need insight into the level and nature of harm to consumers, the current performance of service providers, and whether any groups of consumers are particularly at risk.

2.4 All four regulators have specific teams that gather information on the issues that consumers face in each market, and the potential harm they may experience in future. This work feeds into regulators’ processes for setting priorities and developing annual plans, such as the Financial Conduct Authority’s (FCA’s) annual sector views which use data from a number of sources to analyse the actual and potential harm that consumers face in each market it regulates. The regulators gain consumer insight through four key approaches:

Consumer surveys. Each regulator uses surveys of consumer satisfaction and experiences to gain insight on potential issues across each sector. The larger regulators, in particular, are able to invest in large-scale surveys to explore a wide range of consumer issues. For example, in 2017 the FCA started Financial Lives, a biennial survey of around 13,000 financial services consumers. Ofcom and Ofgem also conduct regular surveys on consumers’ experiences, while Ofwat directly commissions similar surveys of water companies’ customers.

Research projects. The regulators also conduct or commission in-depth research where they identify specific issues. Recent examples from 2018 include Ofcom’s research into internet users’ experiences of harm online, and Ofgem’s research into the behaviour of customers who have engaged in the energy market (such as by switching provider).

Data monitoring. All four regulators regularly gather and monitor data from a range of sources, including regulated firms, consumer complaints and social media. This provides them with data on sector-wide issues such as utility prices and tariffs, as well as specific consumer concerns.

Stakeholder consultation. Each regulator keeps up to date with prevalent or emerging consumer issues through regular engagement with other organisations, such as statutory consumer representative and complaints bodies. For example, Ofwat meets regularly with the Consumer Council for Water, and quarterly with water companies’ customer challenge groups, to discuss market intelligence, customer research, consumer complaints and other insights. In all four sectors, the statutory consumer representative told us it had a good relationship with the regulator, and that regulators’ priorities are broadly aligned with the issues of concern for consumers.

2.5 The regulators also share insights into some common consumer issues (such as the challenges that vulnerable consumers face), both bilaterally and through the UK Regulators Network, which includes all four regulators. For example, the UK Regulators Network’s 2018-19 programme included consumer-focused projects on vulnerability and comparison websites. The UK Competition Network also includes the four regulators, and has led a programme of work on how competition regulation affects different groups of consumers.

The key consumer issues

2.6 The available evidence shows that consumers are facing significant issues across all four sectors. We examined evidence on the key consumer issues in each sector from a range of sources, including regulators’ research, complaints data, consumer representative bodies, Parliamentary inquiries and press reports. We found that the most common challenges for consumers that regulators are seeking to address fall into three broad categories: affordability and debt, difficulties accessing the most appropriate deal or service, and poor-quality services. Regulators also highlighted to us sector-specific issues, such as internet users experiencing harm online.

Affordability and debt

2.7 The most common problem that consumers seek help from Citizens Advice with in all four sectors is dealing with debt associated with paying bills and credit commitments. Affordability concerns play out differently in individual markets – for example, retail lenders conduct detailed affordability assessments on potential borrowers, while Ofcom has found that only 3% of communications users have difficulty paying. However, affordability and debt are cross-sector issues. Our recent reports on vulnerable consumers and tackling problem debt have highlighted the fact that people who fall behind on their bills can often struggle with debts across multiple services.4,5 For example, of those reporting debt problems to Citizens Advice, 32% had problems in two of the four sectors in this review, while 11% had problems in three of the four.

2.8 Affordability concerns are set against a backdrop of long-term increases in price for some essential utilities, including the following:

Real-terms rises in average prices of 28% for gas, 37% for electricity and 6% for water since 2007 (Figure 5).

Real-terms increases in the average list price of phone and internet bundles between 2013 and 2018 of 7% for superfast broadband and 17% for standard broadband. Real-terms pricing trends for promotional offers have been more mixed (11% rise for standard broadband, 8% fall for superfast).

Difficulties accessing the most appropriate deal or service, particularly for vulnerable consumers

2.9 In many markets, customers who do not switch provider typically pay more for the same service as new customers who can access special deals. In 2018, Citizens Advice estimated that this ‘loyalty penalty’ totals £4.1 billion a year across five communications and financial services markets (mobile, broadband, home insurance, fixed-rate mortgages and savings accounts). Vulnerable consumers are likely to be particularly badly affected as they are less likely to access the best deals. For example:

in its 2018 consumer engagement survey, Ofgem found that while on average 18% of consumers switch energy supplier, it is much lower among those with no internet access (6%), aged 65 and over (12%) or from a lower social grade (12% for social grades D and E); and

the FCA found in 2018 that consumers in deprived areas of the country are 70% more likely to use unarranged overdrafts, which are twice as expensive as arranged overdrafts, than those in less deprived areas.

Service failures or poor quality of service

2.10 Consumers face service quality issues in all sectors. Examples of recent issues include the following:

Leakage increased by 3% across the water sector between 2015-16 and 2017-18 (with rises of more than 10% in two suppliers), and 36,000 homes were left without water for more than a day following the severe cold weather in early 2018.

Ofcom found in 2018 that 15% of broadband customers had reason to complain about their service, and that by far the most common cause was connection problems.

Data that the FCA has required banks and building societies to publish since April 2018 show that most major banks had more than 10 IT failures affecting current account customers in the nine months to December 2018.

Ten energy companies ceased trading during 2018, the largest of which had 290,000 household customers. While Ofgem’s ‘supplier of last resort’ process ensures that these closures do not lead to supply disruptions, they create uncertainty and hassle for the customers affected, and can result in higher costs.

Setting clear goals

Defining the outcomes that regulators are seeking to achieve

2.11 Our 2016 performance measurement guide highlighted that regulators need to define clear, measurable objectives and success criteria.6 This is so that teams across the regulator, and external stakeholders they work with, have a common understanding of what they are aiming to achieve and can judge progress. Specific success criteria such as outcomes-based targets are also vital so that industry stakeholders are clear on the regulator’s expectations and priorities for improvement, and consumers and their representatives can hold the regulator to account.

2.12 Regulators produce impact assessments for individual policies or major interventions, which estimate what effect they expect each policy to have on consumer outcomes. They also use targets and incentives in some specific areas to ensure that firms meet regulatory requirements. Ofgem published its first annual Consumer Impact Report in 2018, which estimates the total benefit to consumers of its decisions from the past year. It does this based primarily on its estimates of the expected benefits of those decisions, and also includes the actual impacts of its enforcement and compliance activity during the year.

2.13 The regulators have also set out overall intended consumer outcomes at a high level (Figure 6). They use these high-level outcomes to inform and articulate their current priorities and activities, which they publish each year in annual plans.

2.14 However, the regulators do not translate these high-level outcomes into a detailed, measurable view of what it would look like for markets to work well for consumers. For example, Figure 6 shows that each regulator aims to ensure that consumers receive high-quality, good-value services. However, none has defined what these high-level aims mean in practical terms (for example, with market-wide targets or other success measures), such as what level and distribution of prices or service reliability would demonstrate either good progress or a need to take further action. During our study, the regulators raised concerns that, in some areas, specific targets can have negative unintended consequences. However, they also acknowledged that there is value in being more specific about what they are trying to achieve.

2.15 Regulators also do not have a joined-up way of setting objectives for issues that cut across sectors. For example, each regulator directly or indirectly references affordability in its intended high-level outcomes. However, if someone cannot afford to pay their bills, they are likely to struggle across different sectors rather than one specific service (paragraph 2.7). The regulators do not have a common approach to considering what affordable means or how it might be measured.

Resolving trade-offs and other limitations

2.16 In defining their consumer objectives, regulators face potential conflicts or trade-offs between regulatory objectives or groups of consumers. For example, utility regulators and government encourage companies to invest in infrastructure to ensure an adequate, sustainable supply and service quality in future. However, these investments are paid for through higher prices to consumers, increasing affordability challenges. Regulators need to decide, explicitly, how they will manage these trade-offs, to determine exactly what they are aiming to achieve and the trade-offs made in establishing their desired outcomes.

2.17 There are areas where government provides direction or strategic steer in how to resolve trade-offs based on wider policy objectives or priorities. The UK and Welsh governments publish statements setting out their priorities for Ofwat, which Ofwat must act in accordance with. Similarly, at least once in each Parliament, HM Treasury is required to provide recommendations to the FCA on which aspects of economic policy it should have regard to in discharging its duties. Government also provides direction to regulators on specific issues. For example, in recent years it has introduced legislation requiring Ofgem and the FCA to introduce price caps in the energy and short-term high-cost credit markets, and Ofcom to introduce a universal service obligation to ensure every household can access broadband.

2.18 Despite these directions, regulators report that determining how to manage many of these trade-offs remains challenging. Our 2017 report on vulnerable consumers highlighted the fact that some measures intended to promote a competitive market, which reduces prices for active consumers who switch to the best deals, can conflict with objectives to protect those in vulnerable circumstances who are less likely to switch and therefore benefit from the cheaper prices.7 It concluded that regulators and government need to be clearer about their respective responsibilities in managing these conflicts. Despite progress in some areas, key questions remain: for example, what conditions would need to be in place for Ofgem to recommend removing the energy price cap before 2023, which it will review from 2020 onwards.

2.19 There are also areas where regulators know what they want to achieve, but have limited powers to do so. For example, Ofgem aims to encourage energy consumers to engage in choosing deals and switching between providers. Consumers use price comparison websites and automatic switching services to facilitate this, but these intermediary services are unregulated. Ofgem has publicly acknowledged that it has limited influence over the conduct of these companies and therefore their impact on the consumer experience.

2.20 While regulators have an overall understanding of their limitations and articulate them in specific areas, none has yet provided a detailed explanation of where their ability to achieve their objectives is restricted or requires working with others. The FCA plans to address this by publishing an annual ‘perimeter statement’ from 2019 that it hopes will provide clarity about its own limitations.

4 Comptroller and Auditor General, Vulnerable consumers in regulated industries, Session 2016-17, HC 1061, National Audit Office, March 2017.

5 Comptroller and Auditor General, Tackling problem debt, Session 2017–2019, HC 1499, National Audit Office, September 2018.

6 National Audit Office, Performance measurement by regulators, November 2016.

7 See footnote 4.

Part Three

Measuring and improving performance

3.1 Performance measurement is important in helping regulators to make sure that they are achieving their objectives, take corrective action where necessary, and make the best use of their limited resources to achieve intended outcomes. Our 2016 guide and previous work have highlighted performance measurement as a common weakness for regulators.8

3.2 This part assesses the four regulators against the key principles set out in our 2016 guidance. These are that:

regulators should have an overall performance framework that allows them to scrutinise performance and improve effectiveness, based on appropriate performance measures; and

performance information should be attributable to the actions of the regulator.

3.3 The part concludes with our overall assessment of each regulator’s progress against our performance measurement maturity model, which is also referred to in the 2016 guidance.

Using performance information to scrutinise and improve effectiveness

3.4 Performance information is only of value if it is used to assess whether objectives are being met, and to provide a mechanism through which improvements can be made. Regulators should be clear about who is involved in measuring performance, and performance information should be actively used by a regulator’s governance board to monitor progress and challenge the regulator’s management over how efficiently it is running the business and whether it is achieving its intended outcomes.

3.5 Each regulator has a dedicated central team to coordinate performance information from across the organisation, to report both to the governance board and to senior management. The central team differs in size and role between regulators, but typically brings together performance information recorded or used by individual teams across the regulator to create overall performance reports.

3.6 All four regulators routinely report performance information to their governance boards. This includes structured monthly, quarterly or half-yearly reports on performance across the regulator, as well as non-routine reports on progress with individual issues or policies. Figure 7 sets out how often the routine reports are produced and what they typically focus on. We found that the regulators’ boards are actively engaged in discussions about how to improve the information they receive so that they can improve their scrutiny of management.

3.7 Our 2016 guidance identified that regulators should use a suite of performance indicators that together summarise overall performance. This should include a range of inputs (such as staff time and costs), outputs (such as the number and size of fines issued) and outcomes. Although outcome indicators are the most challenging to measure, they are also the most valuable as they provide insight into how well each market is working for consumers and whether the regulator is achieving its overall aims.

3.8 We found that regulators produce and use good data on progress with individual projects. Most regular performance information used by regulators’ boards and management teams is focused on project-based measures of inputs and outputs, allowing them to understand whether key policies and interventions are on track, and whether corrective action may be needed. For example, regulators’ performance information typically covers progress with activities such as introducing new regulatory rules, the status of ongoing investigations or market studies, or the number of successful enforcement cases.

3.9 The regulators also collect and analyse data on consumers’ experiences and outcomes and use these to inform their activities. We found that each regulator tracks a range of data on the state of their markets, such as prices, quality of service and levels of consumer satisfaction. For example, through its sector view process, the Financial Conduct Authority (FCA) attempts to identify and quantify actual and potential levels of consumer harm in each market and use this to inform prioritisation decisions.

3.10 Despite having good data on consumers’ experiences, the regulators do not typically use this information to assess their own overall performance. The information that regulators produce is not usually underpinned by a clear and consistent set of detailed indicators attributable to the regulator or structured to support their overall intended high-level outcomes. This makes it difficult to assess how the regulators are performing against their overall consumer objectives. Ofcom and Ofgem have made the most progress in designing a structured set of consumer outcome indicators.

Ofcom introduced an outcomes-based framework in 2017 with a set of indicators to measure progress against each of its high-level intended consumer outcomes (Figure 6 on page 25), the current measure of each indicator, and comparisons with the previous year. Ofcom does not routinely report this to its board, but uses it to monitor progress.

Ofgem has similarly introduced a set of indicators for each of its high-level intended outcomes (Figure 6 on page 25). This is less detailed than Ofcom’s, but is reported to the board three times a year accompanied by commentary to provide additional context.

These indicators do not attempt to establish the regulators’ impact on outcomes, but provide consistent and structured information to present a general view of progress against their high-level intended outcomes.

Understanding regulatory influence and impact

3.11 To meet their objectives, regulators rely to varying degrees on the behaviour of consumers, providers and other stakeholders. For example, consumers switching to the best deal should stimulate provider competition, and thereby lower prices and improve quality. Regulators need to understand the extent to which they are able to influence behaviours and the resulting outcomes to make sure they intervene in the most cost-effective way.

3.12 Our 2016 guide recognised that measuring regulatory impact, and separating it out from other influences, is a particularly challenging task. It suggested a staged approach to developing this understanding over time, starting with an overall causal model or strategy map to understand the relationships between the regulator’s activities (inputs and outputs) and intended outcomes. This will provide a robust logical link between measures of inputs, outputs and outcomes. Over time, regulators could use different analytical approaches to measuring or quantifying its influence, in order to strengthen this link.

3.13 None of the regulators has yet developed a good overall understanding of what influence it has over different outcomes for consumers, and how it can exert this influence through different regulatory approaches. Regulators’ impact assessments for individual proposed interventions consider how they expect to influence change to achieve the expected outcome. This analysis, however, is typically only provided for the proposed approach, and does not provide an overall understanding of the regulator’s influence across different approaches.

3.14 The FCA has made the most progress in starting work to better understand its impact and influence with a more structured approach to measuring the direct impact of a sample of individual actions it has taken. In December 2018, it published an evaluation framework that describes its approach to evaluating whether its interventions have been effective, including attempting to isolate the direct influence it had on consumer outcomes. It has so far undertaken three evaluations of recent interventions (see Case Study 1). These analyses have required investment of resources, typically involving three full-time staff for a year. The FCA considers them extremely valuable in providing new insights into its impact, and it has now developed a rolling programme of evaluations that it hopes will support a fuller picture of its influence over time. The other regulators undertake some evaluations of their impact, but these are produced on a more ad-hoc basis, less frequently and typically not in as much detail, citing high costs and limited resources as a key barrier.

Work to improve performance information

3.15 Our 2016 guide made it clear that all the regulators needed to improve the maturity of their performance measurement. It also referred to our maturity model for organisations to assess their overall approach to performance measurement. We evaluated regulators’ progress against this maturity model, interpreted for the specific context of regulatory performance in achieving intended consumer outcomes.

3.16 The regulators’ overall performance measurement frameworks are at various stages of maturity. All four have recognised the need to improve how they measure their performance in protecting the interests of consumers, introduced changes in recent months and plan further improvements. We found that each regulator has made progress in some areas, but has further to go. Overall, the FCA is the most advanced (Figure 8). It has been developing its approach since 2016, and as the largest of the regulators has been able to prioritise more resource than smaller regulators. Ofwat is at an earlier stage in the process overall, and has recently started developing a new sector-wide approach that it intends to complement its existing approach to target setting and performance monitoring for individual companies.

8 National Audit Office, Performance measurement by regulators, November 2016.

Part Four

Reporting of consumer outcomes

4.1 External stakeholders use regulators’ public reporting, alongside other information sources, to hold them to account for their efforts to protect the interests of consumers, and to inform their own work. For example: government departments use regulators’ reporting to understand the state of the market and potential impact on consumers in designing policy; industry groups use it to benchmark their performance; and consumer representative bodies and Parliament use it to hold regulators to account against their consumer objectives.

4.2 It is therefore important for regulators to report clear, accessible information on performance to demonstrate accountability to Parliament and other key stakeholders, including consumers themselves.

4.3 This part examines:

regulators’ public reporting on specific areas, and whether this is useful to external stakeholders;

how well regulators report their overall performance in protecting consumers; and

regulators’ reporting requirements, and how these compare with central government.

Reporting on specific areas

4.4 Regulators frequently publish information on specific consumer issues, particular interventions and how each sector is performing in key areas. We interviewed key stakeholders in government, Parliament and consumer and industry representative bodies to establish how useful this reporting is.

Reporting on individual issues or interventions

4.5 Most stakeholders, including consumer representatives, told us that the regulators engage well to understand their interests regarding progress with specific areas. Our 2016 performance measurement guide recommended that regulators engage with stakeholders to understand what is most useful for them.9 All of the key stakeholder groups reported that they have a good relationship and constructive dialogue.

4.6 Regulators publish a wide range of information on specific interventions or consumer issues, which stakeholders find useful. Examples of regulators’ reporting that have been well received by stakeholders include the following:

Ofcom’s 2018 Connected Nations research. This discussed the UK’s communications infrastructure, focusing on coverage and performance of fixed broadband and mobile networks. It reported that, in 2018, 94% of premises are able to access superfast broadband with a download speed of at least 30 Mbit/s, up from 91% in 2017.

Ofgem’s annual Vulnerable Consumers report. This provided various measures of the experiences of vulnerable consumers in the energy market. In 2018 it reported, for example, that the total number of electricity and gas disconnections for people in debt fell by 92%, from 210 households in 2016 to 17 households in 2017.

Ofwat’s 2015 Affordability and Debt research paper. This presented information about vulnerable household water customers in England and Wales. It reported, for example, that national awareness of special non-financial assistance that companies provide rose from 29% in 2009 to 48% in 2014 (a figure that has stayed relatively stable since, at 43% in 2017).

The Financial Conduct Authority’s (FCA’s) 2017 feedback statement High-cost credit and review of the high-cost short-term credit price cap. This described the main issues arising from its call for input on the consumer credit sector. The report estimated that borrowers save £150 million per year because of the introduction of the FCA’s price cap on short-term high-cost credit, which meant fees and charges on a typical payday loan reduced from more than £100 to around £60.

4.7 However, stakeholders highlighted gaps in regulators’ reporting they would like to see addressed. In particular, this included a lack of benchmarks for consumer outcomes against which to measure success, and in some cases, a lack of clarity on why a certain approach was chosen over alternative options or whether there have been any unintended consequences.

Reporting on the performance of the regulated sectors

4.8 All four regulators have powers to request data from providers, which they seek to use in a way that does create undue burdens for companies. This puts them in a unique position to report, or require firms to report, on performance by each market, section of a market or even individual company. The regulators publish data or reports on how their sectors are performing in priority areas such as prices, complaints, service disruptions, levels of switching or customers in arrears. Each regulator does this differently, with some publishing a structured suite of reports annually and others mostly producing one-off reports.

4.9 Stakeholders we interviewed find this sector-level reporting useful for their own work. In particular:

industry bodies indicated that it was useful to identify benchmarks of good practice;

government departments use it to understand and monitor markets and the effects of their policies; and

consumers and their representatives can use it to choose or recommend companies that perform well or apply pressure to those that do not.

4.10 However, regulators find it challenging to report their own performance in these areas, or what other factors are affecting sector performance. The performance of a market or sector does not always reflect the regulator’s performance, because it is also influenced by other factors. For example, government policy can increase prices for consumers (such as environmental protections or government mandated projects like the roll-out of smart meters) by increasing providers’ costs, as can changing economic conditions. However, price rises may also occur if the regulator is not doing enough to keep costs and prices competitive or affordable. Regulators need to make this distinction so that there is clarity about what the regulator can be held accountable for and what it has less control over. Consumer representatives told us that regulators’ reporting does not clearly distinguish between their own performance and that of the sector. We recognise that it would be difficult to do this in detail. However, regulators could provide more context in their sector reporting, for example supplementing it with qualitative assessments of their own performance.

Reporting overall performance in protecting the interests of consumers

4.11 Performance information is only of value if it can be used to assess whether a regulator is meeting or on track to meet its objectives. While stakeholders find publications on specific issues useful, it is also important that regulators’ reporting provides a meaningful overview of their performance in protecting the interests of consumers, clearly linked to their objectives or intended outcomes.

4.12 Our analysis found that regulators’ annual reports – the only place they currently report their overall performance – do not provide a meaningful overall assessment of how well they are protecting consumers’ interests. Annual reports should give a fair, balanced and understandable analysis of an organisation’s performance. While we identified areas of good practice, regulators do not consistently provide a clear line of sight in their annual reports between what they are trying to achieve, what they have done to meet these objectives and what the actual outcomes are for consumers. Common weaknesses included a lack of benchmarks or success criteria against which to measure progress, and reporting where it was unclear how performance was being measured, if at all (examples in Figure 9).

4.13 Performance information is also not reported in a consistent way over time, meaning that year-to-year comparisons are not possible. To improve their reporting, all four regulators have recently changed or are planning to change what information they report and how they present it in their annual reports. While it is important to refine and improve performance reporting, a lack of consistency makes it difficult to understand performance over time. Key stakeholders we interviewed told us that regulators’ regular reporting would benefit from greater consistency from year to year, as well as from clearer benchmarks. The regulators acknowledged to us that these are concerns they plan to, or are already working to, address in their annual reports. Ofgem also intends to develop its Consumer Impact Report (paragraph 2.12) so that over time it presents more information on actual outcomes as well as expected benefits.

Reporting requirements

4.14 Unlike information presented in financial statements, there are no common standards with which regulators must comply in presenting performance information on consumer outcomes. While we expect each regulator to report different specific information and comply with any specific statutory requirements, common standards over the types of outcome information to report and how to do so would prompt reporting based on agreed best practice and make the information more accessible and consistent for users of the reports.

4.15 For example, we reported in 2018 on how government is working to improve planning, prioritisation and reporting for central government departments, including efforts to improve measures of the value realised from public spending.10 Each major department produces a ‘single departmental plan’ setting out key activities and objectives, and reports periodically progress against these. The accounting officers of those departments also produce ‘accountability systems statements’ setting out what they are accountable for. While there remain weaknesses in these processes, departments considered that the approach had helped them to clarify objectives and internal accountabilities and take a more professional approach to planning.

4.16 The regulators have not developed a consistent set of principles for what or how to report. Independent regulators are not subject to the same requirements as central government, and it is for each regulator’s management and board to decide on the content and presentation of its performance reporting. There is considerable scope for regulators to work together, and in consultation with government and other key stakeholders, to develop standards based on good practice, such as the examples we identified from the 2017 Building Public Trust Awards.11

9 National Audit Office, Performance measurement by regulators, November 2016.

10 Comptroller and Auditor General, Improving government’s planning and spending framework, Session 2017–2019, HC 1679, National Audit Office, November 2018.

11 National Audit Office, Good practice in annual reports 2016-17, February 2018.

Appendix One

Our audit approach

1 This study examined how well Ofwat, Ofgem, Ofcom and the Financial Conduct Authority (FCA) measure and report their performance in protecting the interests of consumers. In particular, we examined:

whether the regulators have good insight into the key issues that consumers face, and clearly define what they want to achieve and the barriers they face in doing so (Part Two);

how well the regulators measure their own performance, understand the extent to which they are able to influence consumer outcomes, and use this information to improve their own effectiveness (Part Three); and

whether the regulators report performance publicly in a way that is useful for Parliament and other stakeholders to hold them to account (Part Four).

2 We assessed the regulators against the principles set out in our 2016 guidance on performance measurement, as well as other recent work we have done on accountability and reporting in government.12

3 The study did not evaluate the regulatory performance of each regulator, and it did not focus on their other statutory objectives.

4 Our audit approach is summarised in Figure 10. Our evidence base is described in Appendix Two.

12 National Audit Office, Performance measurement by regulators, November 2016.

Appendix Two

Our evidence base

1 We reached our independent conclusions on the value for money of regulators’ approach to measuring and reporting their performance in protecting the interests of consumers by analysing evidence collected between October 2018 and January 2019.

2 We applied an analytical framework with evaluative criteria to our analysis, which considered how regulators gather insights into the key issues consumers face and good-practice criteria for performance measurement and reporting. This analytical framework was based closely on the principles set out in our 2016 guidance on performance measurement.13 Our audit approach is outlined in Appendix One.

3 To examine whether the regulators have good insight into the key issues that consumers face, and clearly define what they want to achieve and the barriers they face in doing so, we did the following:

Gathered and analysed data and information to inform our understanding of the key consumer issues in each sector. To do this, we did the following:

Analysed secondary evidence, including published research reports from regulators and other organisations on consumers’ experiences in each sector, and published data such as Citizens Advice’s consumer enquiries and ombudsman complaints data.

Conducted primary analysis of price indices for water, electricity and gas published by the Office for National Statistics (ONS) to examine long-term price trends. The ONS’s indices for these three services are components of its overall consumer price index including housing costs (CPIH) measure of inflation. We therefore deflated the indices into real terms using CPIH. Alternative inflationary measures produce different results, such as RPI (typically a higher measure of inflation, therefore indicating smaller real-terms price rises) or the GDP deflator (typically a lower measure of inflation, therefore indicating higher price rises).

Reviewed evidence submitted by the statutory consumer representative body or panel in each sector, including views on the most important current consumer issues in the water, telecoms, energy and financial services sectors. These bodies include; the Consumer Council for Water (water); the Communications Consumer Panel (telecoms); the Financial Services Consumer Panel (financial services); and Citizens Advice (energy).

Interviewed consumer advocacy organisations with a cross-sector role, such as Money Saving Expert and Citizens Advice.

Reviewed written submissions provided on request from officials at the House of Commons Library on the key issues in each sector and the most common queries they receive from Parliamentarians.

Conducted a systematic review of the key consumer issues identified in press reports in the three months to 3 December 2018.

Conducted semi-structured interviews with each regulator to understand: their approaches and process for gathering consumer insights; how they prioritise different objectives, policies and interventions; the consumer protection objectives they set; and the challenges they face.

Reviewed documentation provided by each regulator relating to: their approaches and process for gathering consumer insights; how they prioritise interventions; the consumer protection objectives they set; and the challenges they face.

Reviewed published reports providing examples of how regulators work together to share consumer insights.

Reviewed written submissions provided on request by the statutory consumer representative body or panel for each sector (listed above) providing perspectives on: how well the regulators understand the key issues and reflect them in their plans and priorities; and each body’s ongoing engagement with regulators on key consumer issues.

Conducted semi-structured interviews with representatives from the government department with policy responsibility for each of the regulated sectors, to understand their engagement with the regulators on consumer issues. These were the Department for Business, Energy & Industrial Strategy (energy), the Department for Digital, Culture, Media & Sport (telecoms); HM Treasury (financial services) and the Department for Environment, Food & Rural Affairs (water).

Drew on evidence from our previous work, in particular regarding the types of trade-offs regulators have to resolve to meet their consumer protection objectives, and issues that cut across sectors such as affordability and debt. In particular, we used evidence from our previous reports on:

Vulnerable consumers in regulated industries;14 and

Tackling problem debt.15

4 To examine how well the regulators measure their own performance, understand the extent to which they are able to influence consumer outcomes, and use this information to improve their own effectiveness, we did the following:

Conducted semi-structured interviews with a number of teams and senior officials from each regulator to understand their performance measurement frameworks and how these are used to inform decision-making.

Reviewed performance measurement documentation provided by each regulator, including routine performance reports and dashboards.

Analysed each regulator’s performance frameworks against our performance measurement maturity model, which was referred to in our 2016 guidance and originally set out in our 2009 report on performance frameworks and board reporting.16 We evaluated regulators’ progress against this maturity model, interpreted for the specific context of regulatory performance against achieving intended consumer outcomes.

Discussed regulatory performance measurement with, and reviewed documentation from, three regulators not in scope of this audit, for comparison with the regulators in scope. The three regulators were The Pensions Regulator, the Care Quality Commission and the Health and Safety Executive. We did this to ensure we had a good understanding of the challenges that regulators face when measuring and reporting performance, particularly with regard to understanding and measuring regulatory influence, and to identify examples of good practice.

Conducted semi-structured interviews with members of regulators’ governance boards, to discuss how performance information is used by senior officials, the extent to which it currently meets their needs and their views on how it could be improved.

Reviewed governance board minutes to understand how performance information is discussed in board meetings.

5 To examine whether the regulators report performance publicly in a way that is useful for Parliament and other stakeholders to hold them to account, we did the following:

Conducted semi-structured interviews with officials from each regulator to understand what performance information each regulator publishes and the factors they consider when deciding what and how to publish performance information.

Reviewed regulators’ published performance information, including annual reports, as well as a wide range of information reported on specific interventions or consumer issues.

Analysed regulators’ published performance information against good-practice criteria set out in our 2016 guidance, as well as against good practice we identified from the 2017 Building Public Trust Awards.17

Conducted semi-structured interviews with a range of stakeholders to understand whether they use regulators’ performance information and whether it meets their needs. These stakeholders included:

the government departments with policy responsibility for each sector (listed above);

the statutory consumer representative body or panel for each sector (listed above);

industry representative bodies: Energy UK, and the two Financial Services Practitioner Panels representing large businesses and small businesses respectively; and

Parliamentary staff.

Drew on evidence from our previous work, in particular to understand the performance reporting requirements for central government departments, how these have changed over time, improvements and current weaknesses.

Conducted a semi-structured discussion with representatives from HM Treasury, to understand reporting requirements for regulators.

13 National Audit Office, Performance measurement by regulators, November 2016.

14 Comptroller and Auditor General, Vulnerable consumers in regulated industries, Session 2016-17, HC 1061, National Audit Office, March 2017.

15 Comptroller and Auditor General, Tackling problem debt, Session 2017–2019, HC 1499, National Audit Office, September 2018.

16 National Audit Office, Performance Frameworks and Board Reporting I, July 2009.

17 National Audit Office, Good practice in annual reports 2016-17, February 2018.

Design and Production by NAO External Relations
DP Ref: 006342-001