British Broadcasting Corporation

BBC World Service

Report by the Comptroller and Auditor General

This report has been prepared under Clause 79 of the Broadcasting Agreement, as amended, between the Secretary of State for Culture, Media & Sport and the BBC dated July 2006

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

7 June 2016

This report examines whether the BBC World Service is meeting its plans to change significantly its services, operations and costs while managing the impact on current and future performance.

© National Audit Office 2016

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.gsi.gov.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

11159 06/16 NAO

The National Audit Office study team consisted of:
Richard Gauld, Howard Revill, Nigel Terrington and
Fatima-Zohra Yaagoub, under the direction of Peter Gray.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Key facts

Summary

Introduction

1 The BBC World Service (the Service) is an international broadcasting service run by the BBC. It provides radio, television and online services in 29 languages to an estimated global audience of 246 million adults. Its main audiences are in Africa, the Middle East and Asia. In 2014-15 it spent £254 million and employed 1,518 staff. Around 35% of its staff are based overseas, with most of the rest co-located with other BBC News Group departments in New Broadcasting House in London.

2 The World Service is established under the same Royal Charter as the BBC’s UK services. The BBC’s Executive Board is accountable to the BBC Trust for the Service’s performance. The Foreign Secretary agrees, jointly with the BBC Trust, the Service’s objectives, priorities and targets, although editorial control rests entirely with the BBC. The Service cannot open a new language service or close an existing one without the approval of the BBC Trust and Foreign Secretary.

3 The Service is facing several strategic challenges. Until recently, it broadcast mainly on radio. However, changes in technology and consumer behaviour have contributed to a long-term decline in demand for short-wave radio as audiences increasingly use online, FM radio and television news services. The Service is facing increased competition from other international and local broadcasters as international media markets develop.

4 The Service has also had to respond to financial pressures and to changes in how it is funded. In October 2010, the government announced that it was reducing the Service’s revenue funding by 16% and its capital funding by 52% over the period 2011-12 to 2016-17. In addition, the government announced that funding from the Foreign & Commonwealth Office (FCO), which had formed the main source of the Service’s income, would cease in April 2014; instead, the Service would be funded mainly from television licence fee income.

5 In November 2015, the government announced that it would make £289 million available to the Service from 2016-17 to 2019-20. This new funding is additional to the funding the Service receives from the licence fee. In May 2016, the government announced its intention to ensure that the BBC protected the licence fee funding for the Service at £254 million a year for the five years from 2017-18.

6 In response to the competitive pressures it faced and the funding reductions announced in October 2010, the Service set out to transform itself from a mainly radio-based operation to a multi-media broadcaster, while at the same time seeking to reduce its costs.

Scope and approach

7 This report examines the Service’s performance since 2010 in responding to the challenges it faces. We examine:

the Service’s progress in reducing expenditure, its investment in new delivery platforms, such as digital and television, and how it prioritises services for development (Part Two); and

the Service’s reported performance, its approach to target-setting and measuring performance, and its wider benefits for the UK (Part Three).

Key findings

Delivering change

8 The Service has used savings generated since 2010 to invest in new digital and television services. As at May 2016, all 29 language services were available across a larger number of digital platforms than in 2010. The Service had increased the number available on television from 3 to 15. Its expansion into television has taken the form of the provision of news bulletins and individual programmes delivered by partners, rather than opening its own new television channels. As at May 2016, it was planning to expand and enhance its services using the funding of £289 million announced by the government in November 2015 (paragraphs 2.14 to 2.16).

9 The Service has successfully reduced its expenditure at a lower cost than expected. Our analysis of financial data indicates that, by 2014-15, the Service had reduced its annual expenditure by £46.8 million, compared to a baseline set in December 2010. This was in line with its milestone target of £46.7 million. The Service has plans in place to achieve annual savings of £59.6 million by 2016-17. This is lower than the savings target identified in December 2010 as the Service received some extra funding subsequently. One-off implementation costs were £27 million by the end of 2014-15, 40% lower than its original estimate for that date of £45 million. The latest estimate of its implementation costs to 2016-17 is £31 million, 42% below its original estimate of £53 million (paragraphs 2.2, 2.3 and 2.7).

10 The one-off costs of reducing expenditure were lower than expected as the Service made fewer staff redundant than planned. The Service originally planned to close 650 posts to the end of 2013-14 out of a workforce in 2010-11 of 2,570. By March 2015, it had closed 630 posts, with more staff than expected leaving without taking redundancy and significant numbers being redeployed elsewhere in the BBC. The Service plans to close a further 105 posts by March 2017 (paragraph 2.4).

11 The Service reduced expenditure through increased efficiency arising from the introduction of new ways of working and closer integration, and through a reduction in services.

Almost £31 million of savings, 66% of the total up to the end of 2014-15, resulted from improvements to the Service’s efficiency, although no specific efficiency targets were set at the start of the spending reduction programme. Its move into New Broadcasting House in early 2012 and the switch to licence fee funding from 2014-15 enabled the Service to introduce new ways of working and to integrate more closely with BBC News (paragraphs 2.6 and 2.8 to 2.10).

In addition to other savings measures, the Service closed five foreign-language services completely and stopped all radio broadcasts for a further seven. Savings from reducing services to the end of 2014-15 were almost £16 million, 34% of the total delivered up to the end of 2014-15. However, as more funding was made available to the Service over time, it scaled back its plans to close services. For example, although it initially planned to stop short-wave radio programmes for all but five language services by the end of 2014-15, it was still broadcasting in short wave for 16 languages by this date. As at May 2016, it had no plans for further closures (paragraphs 2.11 and 2.12).

12 It is not clear how the Service has taken cost-effectiveness into account when recommending language services for opening or closure. Although the Service’s stated criteria for recommending which language services it should provide included cost-effectiveness, such recommendations have been driven primarily by the Service’s assessment of the strategic importance of the countries involved and their need for independent, impartial news. It is unclear how the Service took account of cost-effectiveness when recommending which language services to provide. In November 2014, the Service defined a cost-effective language as one that did not cost more than 30% of the average cost per user for similar services on similar platforms. Exceeding such a threshold does not necessarily mean that a service should not be provided. However, our analysis shows that nine language services exceeded this threshold (paragraphs 2.20 and 2.21).

Reporting performance

13 The Service estimates that its total annual audience has increased from 166 million in 2010-11 to 246 million in 2015-16, exceeding its forecasts. The Service predicted significant reductions in its audiences in forecasts it prepared in 2011 and 2012. Actual audiences have been higher as the Service has invested more in digital and television services than it originally expected, and it has cut back on the reductions it planned for short-wave radio. The Service does not, however, report on the audience reach it achieves as a share of population, either globally or for each of its language services. It also has a smaller global footprint than it had previously due to the reduction in the number of language services (paragraphs 3.2, 3.3 and 3.7, and Figure 5).

14 The Service has made a proportionate attempt to estimate the size of its audience. Given the Service’s international coverage, obtaining the data needed to estimate its audience figure can be difficult. The Service therefore obtains data from a variety of sources, including approximately 15 new audience surveys a year, on average. Most of the survey data used to estimate the audience figure in 2014-15 were reasonably up-to-date, but 10% of the data were between 5 and 10 years old. Also, because of local circumstances, there are always likely to be countries where data will not be available for television and radio audiences (paragraphs 3.11 and 3.12).

15 The Service could report more fully on the level of uncertainty of its estimated audience figure. The compilation of the audience figure involves a number of adjustments by the Service to the data collected; for example, to guard against double-counting those who consume its output on more than one platform and to only count users who actively engage with its social media content. Year-on-year changes to the figure can reflect not only changes in performance but also changes to what is included and refinements to the methodologies used to compile the total. When reporting the audience figure, the Service does not currently publish full details of how it produces the figure, the age profile of the underlying data, or the estimated uncertainty attached to the total figure (paragraphs 3.13 and 3.14).

16 The Service’s audience targets for 2016-17 were set at or below the Service’s forecast levels, and two of three had been achieved by 2014-15. In 2013, the BBC Trust and FCO jointly approved three high-level audience targets for the Service for a three-year period. The target for the Service’s estimated weekly global audience was set equal to the audience forecast of 200 million by 2016-17. The Service achieved an estimated audience of 210 million in 2014-15. Individual targets for television and digital audiences were set below their respective forecasts because the BBC Trust wanted to give the Service operational flexibility to decide how best to meet the overall audience target across different platforms. The target for television was achieved by 2014-15, and the target for digital audiences by 2015-16. Although two targets were met after the first year of a three-year period, and all three after the second year, the targets have not been formally reviewed to consider whether they should be amended. New targets will be agreed as part of the new operating licence under the new Charter (paragraphs 3.2 and 3.8 to 3.10).

17 The Service has reported that it has met its targets on audience satisfaction, but the data it uses for this does not relate specifically to the Service. The Service has reported that, in 2013-14 and 2014-15, it had met its targets of achieving higher ratings than its international competitors for ‘most trusted international news provider’ and ‘helps to keep me informed’. The results are based on data collected for BBC News as a whole. The BBC does not collect audience satisfaction data specifically for the Service as it considers that its overseas audiences tend not to differentiate between its different overseas brands. However, the 2014-15 survey over-represented European countries when considered against the Service’s overall audience. As the BBC continues to integrate its news services, it will become increasingly challenging for it to identify the performance of specific services (paragraphs 3.4, 3.15 and 3.16).

18 The FCO has not confirmed how it will assess the impact of the additional funding of £289 million. The government announced this funding on the basis that, as a provider of accurate, impartial and independent news, the Service helps to strengthen democratic accountability and governance, and promote Britain and its values around the world. As at May 2016, the FCO had not confirmed how it would assess the impact of the extra funding (paragraph 3.18).

Conclusion on value for money

19 The World Service has successfully delivered much of what it set out to do back in 2010: it has invested in new digital and television services, integrated its services more closely with the rest of the BBC, and, in so doing, delivered efficiencies, and achieved an estimated audience of 246 million in 2015-16. In addition, one-off implementation costs have been lower than expected as the Service has made fewer staff redundant than planned, even though it had closed five foreign-language services and 630 posts by March 2015.

20 This transformation has delivered value for money, but there is scope to improve the approach to target setting and performance reporting. Targets have been set at or below forecast levels, and not reviewed when they have been exceeded. The Service could be clearer about the methods and uncertainty associated with the two key approaches it uses – on audience size and satisfaction – to report on its performance, and report publicly on a broader range of measures, and in more depth.

Recommendations

a There is scope to improve the approach to target-setting and performance reporting. The Service’s audience targets were set at or below the Service’s forecast levels, and the range of measures used to report on Service performance is limited to the estimated size of its global audience and audience satisfaction with BBC News.

The BBC and FCO should provide for an arrangement to review targets, in order to maintain performance incentives while also recognising any relevant external factors that might impact on performance.

The BBC should report a broader range of measures on the Service’s performance.

b The Service should set out, alongside its published performance figures, a fuller explanation of how its figures are compiled, and the associated levels of uncertainty attached to them. In compiling its global audience figure the Service does not currently publish full details of how it produces the figure, the age profile of the underlying data, or the estimated uncertainty attached to the total figure.

c The Service should explicitly assess services against all its chosen criteria, including its cost-effectiveness criterion, when identifying which services to provide. It was unclear how the Service took account of cost-effectiveness, relative to other criteria, when recommending which language services to open or close.

d The FCO should identify how it will assess the impact of the £289 million new funding which was announced for the purposes of strengthening democratic accountability and governance, and promoting Britain and its values around the world. As of May 2016, it had not confirmed this.

Part One

Background

1.1 The BBC World Service (the Service) is an international broadcasting service run by the BBC. It provides radio, television and online services in 29 languages to an estimated global audience of 246 million adults. Its objectives are to:

provide an accurate, impartial and independent news service covering international and national developments;

be the most trusted international news provider in the world;

provide inspiring and engaging programmes and content that help its audiences understand the world and their place in it; and

reflect the UK’s people, cultures and national life.

1.2 The Service’s main audiences are in Africa, the Middle East and Asia (Figure 1). The areas where it has least coverage are South America, where historically the Service’s audiences have been low, and Europe. The Service stopped most of its broadcasts to Western Europe in the 1950s and Eastern Europe in the 2000s.

1.3 The Service spent £254 million in 2014-15 (Figure 2). Until April 2014, it was funded by grant-in-aid from the Foreign & Commonwealth Office (FCO). In April 2014, responsibility for its funding transferred to the BBC, with the Service funded mainly by television licence fee income. This change in funding arrangements was agreed in 2010, as part of a new licence fee settlement between the Department for Culture, Media & Sport and the BBC.

1.4 In November 2015, the government announced in its National Security Strategy and Strategic Defence and Security Review that it would invest £289 million in the Service from 2016-17 to 2019-20.1 This is additional to the funding that the BBC makes available to the Service from the licence fee. In May 2016 the government announced its intention to ensure that the BBC protected the licence fee funding for the Service at £254 million a year for the five years from 2017-18.2

1.5 Previously situated in its own premises in Bush House, since 2012 the Service has been co-located with other BBC News Group departments in New Broadcasting House in London, where most of its programmes are produced. It has 44 offices overseas, with staff or freelancers based in another 71 locations across the world. Around 35% of the 1,518 full-time equivalent staff that it employed in 2014-15 were based overseas.

Organisation and governance

1.6 The Service is established under the same Royal Charter as the BBC’s UK services, and contributes to the same public purposes. The agreement between the BBC and the Secretary of State for Culture, Media & Sport requires the BBC to provide a World Service to users outside the UK in languages that are approved by the Foreign Secretary. The Service cannot open a new language service or close an existing one without the approval of the BBC Trust and Foreign Secretary. The Foreign Secretary also agrees, jointly with the BBC Trust, the Service’s objectives, priorities and targets (Figure 3). Editorial control rests entirely with the BBC.

1.7 The BBC’s Executive Board is accountable to the BBC Trust for meeting the Service’s targets and ensuring it achieves value for money. From April 2014 the Service has been part of World Service Group, itself part of BBC News Group (Figure 4). The Executive Board has overall accountability for the performance of the Service, which it has then devolved to the Head of BBC News Group – the Director, BBC News and Current Affairs. News Group’s Deputy Director is responsible for leading World Service Group, but does not sit on the BBC’s Executive Board.

Strategic challenges

1.8 Until relatively recently, the Service broadcast mainly on radio. However, changes in technology and consumer behaviour have contributed to a long-term decline in demand for short-wave radio stations, as audiences increasingly use online, FM radio and television news services.3

1.9 The Service is also facing increased competition from other international broadcasters. China Central TV broadcasts five foreign-language channels across the world – English, French, Spanish, Arabic and Russian, the last two of which opened in 2009. Russia Today has opened five foreign-language news channels (Arabic, English, French, German and Spanish) in the 2000s. The Service is also facing increasing competition from local broadcasters; for example, the number of local broadcasters in Afghanistan had grown to over 170 radio and 80 terrestrial television stations by 2014.

1.10 If the Service fails to meet the challenges of changing technology and consumer behaviour and increased competition, it risks losing its audience to its competitors. The Service has therefore sought to respond to these challenges by transforming itself from a primarily radio-based operation to a multi-media broadcaster, reducing its provision of short-wave radio services, investing more in online and television services, and increasing its distribution via FM radio partners (Figure 5).

1.11 The Service has also had to respond to financial pressures. In October 2010, the government announced that it was reducing the Service’s revenue funding by 16% and its capital funding by 52% over the period 2011-12 to 2016-17. In response, the Service announced that it would close five language services and stop radio broadcasts for others.4

1 HM Government, National Security Strategy and Strategic Defence and Security Review 2015, Cm 9161, November 2015.

2 HM Government, A BBC for the future, Cm 9242, May 2016.

3 Short-wave and medium-wave transmissions are used for long distance communication and generally use radio frequencies between 0.5 and 1.6 Megahertz (medium-wave) and 4 and 21 Megahertz (short-wave). FM radio is broadcast between 87.5 and 108 Megahertz; it is capable of better sound quality but only offers local coverage.

4 The five language services that have closed are: Albanian, English for the Caribbean, Macedonian, Portuguese for Africa, and Serbian.

Part Two

Delivering change

2.1 This Part examines:

the BBC World Service’s (the Service’s) progress in reducing expenditure, through increased efficiency and reductions in its services;

its investment in new delivery platforms; and

how it prioritises services for development, including their opening and closing.

Reducing expenditure

2.2 In December 2010 the BBC Trust approved a six-year plan developed by the Service to deliver annual savings of £65.8 million by 2016-17, with implementation costs of £53 million. This was in response to the funding reductions that the government announced in October 2010 (paragraph 1.11). The Service subsequently reduced its savings target after receiving extra funding (Figure 6) and after reductions in the estimated implementation costs. Its latest target, set in June 2014, is to achieve annual savings of £60.5 million by 2016-17 (Figure 7).

2.3 Our analysis of financial data indicates that, by 2014-15, the Service had achieved annual savings of £46.8 million, in line with its milestone target of £46.7 million. One-off implementation costs were £27 million by the end of 2014-15, 40% lower than its December 2010 estimate for that date of £45 million. The latest estimate of its implementation costs to 2016-17 is £31 million, 42% below the December 2010 estimate of £53 million.

2.4 Implementation costs were lower because the Service closed fewer posts and made fewer staff redundant than planned. The Service originally planned to close 650 posts to the end of 2013-14 across the Service and the BBC out of a budgeted workforce for 2010-11 of 2,570.5 By March 2015, it had closed 630 posts, reducing the number of staff it employed to 1,518. This was 25% less than the 2,017 staff in post at March 2010 (Figure 8). In addition, more staff than expected left without taking redundancy and significant numbers were redeployed elsewhere in the BBC. The Service plans to close a further 105 posts by March 2017.

2.5 The December 2010 plan (paragraph 2.2) set out how Service expected to operate once the plan had been implemented (Figure 9). The plan included three key components for this:

achieving efficiencies by improving production methods and integrating its operations with BBC News;

reducing the scope and scale of some of its services, including substantially reducing short-wave radio services, and closing some services completely; and

investing in online and mobile services, and, when affordable, television news partnerships.

2.6 The plan did not, however, identify the extent to which efficiencies and scope reductions would each contribute to the original £65.8 million savings target. Instead, the Service adopted a phased approach to delivering these savings whereby it announced savings measures in tranches as the spending reduction programme progressed. This approach allowed the Service to decide on the exact mix of efficiency and scope reduction measures in each tranche based on the progress it had made to date in reducing expenditure and in light of latest developments.

2.7 To ensure that it delivered the required spending reductions by 2016-17, the Service front-loaded its savings measures. In January 2011 it announced the first tranche of measures, which it expected to reduce expenditure by over £29 million, almost half the total savings required, and to result in 480 post closures in the two years to 2012-13 (Figure 10). Following the decision by the BBC Trust to commit higher levels of funding to the Service from 2014-15 onwards (Figure 6), the main purpose of measures in subsequent tranches was to release funds for the Service to invest in digital and television services.

Efficiency savings

2.8 Almost £31 million of savings, 66% of the total up to the end of 2014-15, resulted from improvements to the Service’s efficiency. 67% of its £59.6 million spending reduction target up to 2016-17 is due to be delivered from improved efficiency (Figure 10). The Service will therefore have to find further efficiencies totalling over £9 million to achieve its cost reduction targets up to 2016-17.

2.9 The Service expected that the planned move from its own premises at Bush House into New Broadcasting House in early 2012 would provide opportunities for new ways of working and closer integration with BBC News which would deliver efficiencies averaging 5% annually. Additional efficiencies from further integration would also arise from 2014-15 onwards because of the switch to licence fee funding and the continuing integration of overseas offices. It is not clear how the Service calculated this 5% figure. The Service did not use this figure to set a savings reduction target for its efficiency measures, nor has it measured the extent to which it has achieved this 5%.

2.10 The Service has made progress in introducing new ways of working and integrating its operations with BBC News (Figure 11). As a result, 46% of its costs are budgeted to be incurred on its behalf by other parts of the BBC in 2015-16. However, some staff reported that the BBC still did not get the best out of them, despite the new ways of working. According to a 2015 survey of BBC staff, approximately 55% of Service staff felt that the BBC got the best out of them, compared to 64% for the BBC as a whole.

Reductions in services

2.11 The Service has reduced the scope and scale of the languages services it offers in order to reduce spending. Savings from these reductions totalled almost £16 million to the end of 2014-15, 34% of the total cost reduction of £46.8 million delivered to that date.

2.12 To deliver these savings, the Service took the following measures:

In March 2011 it closed completely five foreign-language services (paragraph 1.11)6, and stopped all radio broadcasts for seven languages7 in 2011-12. This saved £7.5 million annually.

It initially planned to stop short-wave radio programmes for all but five language services by the end of 2014-15.8 As funding became available to mitigate the impact of its spending reduction (Figure 6), it slowed the pace of closure, for example reversing some of the changes proposed to the Arabic service. As a result, it was still broadcasting in short wave for 16 languages by the end of 2014-15 and, as at May 2016, had no plans for further closures.

It introduced substantial changes to the schedule for World Service English, decommissioning some programmes and reducing the extent of non-news output. It also planned significant reductions in the number of hours broadcast in short wave, from between 7 and 19 hours a day, depending on the region, to two hours a day. Again, as mitigation funding became available, it was able to maintain the number of hours broadcast at a slightly higher level of six hours a day.

2.13 The reductions in short-wave broadcasts enabled the Service to close two of its six short-wave transmitter sites (Figure 12). These reductions were built into the new contract the Service entered into with Babcock International Group in 2012 for distribution services. This contributed to a £14 million reduction in annual distribution costs in 2016-17.

Investment in new delivery platforms

2.14 In line with its December 2010 six-year plan (paragraph 2.2), the Service has used some of the funding released by reductions in its spending to fund investment in online and mobile services and television news partnerships. The Service has been able to use more of the funding released for investment than originally planned. Its investment has increased from £2 million a year, planned in December 2010, to £7 million in 2014-15 and almost £13 million a year in 2015-16 and 2016-17. At May 2016, all 29 language services it provides were available online and could be accessed by mobile phone (Figure 13). The Service is also increasingly making its services available on social media and third-party apps.

2.15 To reduce costs and increase its audiences, the Service has made increasing use of partner organisations to broadcast its television and FM radio content, rather than broadcasting directly itself. As at May 2016, there were over 1,800 partner organisations broadcasting its content. These included, for example, 300 radio stations in Africa and nearly 700 radio stations in North America. As television is more expensive than radio, the Service’s expansion into television has taken the form of news bulletins or individual programmes, to be delivered by partners, rather than opening its own new television channels.

2.16 The Service plans to expand and enhance its services using the extra funding announced by the government in November 2015 (paragraph 1.4 and Figure 14). It is currently preparing plans to achieve this, and will seek the Foreign Secretary’s approval, as necessary, for any new language services.

Deciding priorities

2.17 Since December 1998, when the NAO last reported on it9, the Service has reduced the number of language services it provides by more than a third, from 44 to 29 (Figure 15). It has opened one new language service – a Thai Facebook news service. This opened in July 2014 after international channels, including BBC World News, were taken off-air following a military coup. It was initially intended to be provided for only three months. However, with the approval of the BBC Trust and FCO the Service decided to extend this until the restoration of an internationally recognised democratic government. As at May 2016, the service was still open.

2.18 According to the Service, it used four criteria when prioritising its existing language services in 2009-10 in the run-up to the 2010 Spending Review and when recommending which of these services to close following the funding reductions in the Spending Review settlement in 2010:

the strategic importance of the country served by the language service;

the need for independent, impartial news in that country;

the likely impact in the country in terms of audience size; and

the cost-effectiveness of the service.

These criteria reflect the Service’s objectives and priorities, and are similar to criteria used by other western national broadcasters that broadcast overseas and that have had to prioritise the languages they offer in response to significant reductions in their budgets.

2.19 When seeking the BBC Trust’s approval for both the initial opening of the new Thai service and its subsequent extension (paragraph 2.17), the Service did not formally assess the proposed service against the four criteria. The BBC Trust therefore asked the Service to develop a set of criteria to use when considering future proposals from the Service to open or close language services. In response, in November 2014, the Service agreed the following four criteria:

evidence of the lack of availability of impartial news;

the Service’s ability to meet the need for impartial news;

the likely impact on audiences; and

the Service’s ability to deliver service cost-effectively.

These criteria were different from the previous four; for example, they include no reference to the strategic importance of the country involved.

2.20 Despite its inclusion in both the original and November 2014 prioritisation criteria, it is not clear what prominence the Service attached to cost-effectiveness when recommending which language services to open or close. For example, in 2010-11, in response to the Spending Review settlement, it recommended for closure the five language services that it rated as having the lowest strategic importance, need and impact. However, the documents we obtained from the Service did not show how it took account of the relative cost-effectiveness of its existing languages when identifying those for possible closure. Similarly, the Service used the first three of the November 2014 criteria when identifying proposals for expansion in response to the government’s announcement in November 2015 of extra funding (paragraph 2.16).

2.21 In November 2014, the Service defined a cost-effective language service as one that did not cost more than 30% of the average cost per user for similar services on similar platforms. As at May 2016, our analysis showed that nine of its language services exceeded this threshold (Figure 16). This does not necessarily mean that the nine services should not be provided as their continued provision may be justified under the other criteria used by the Service when deciding on its language services. However, it is not clear what weighting the Service attaches to cost-effectiveness compared with these other criteria.

5 Composed of 2,110 budgeted posts employed by the Service and 460 budgeted posts in other parts of the BBC producing output for the Service.

6 Albanian, Macedonian, Serbian, Portuguese for Africa, and English for the Caribbean.

7 Azeri, Chinese – Mandarin, Russian, Spanish for Cuba, Turkish, Ukrainian, Vietnamese.

8 English, Burmese, French for Africa, Hausa and Uzbek.

9 Comptroller and Auditor General, BBC World Service: New Financial Arrangements, Session 1998-99, HC 26, National Audit Office, December 1998.

Part Three

Reporting performance

3.1 This Part examines the BBC World Service’s (the Service’s) performance. We assess:

the Service’s reported performance;

its approach to target-setting;

how it measures its performance; and

measuring the wider benefits it produces for the UK.

The Service’s reported performance

3.2 In 2013, the BBC Trust and Foreign & Commonwealth Office (FCO) jointly approved three high-level audience size targets for the Service for the period to 31 December 2016. By March 2016 it had achieved these (Figure 17). The Service reported in April 2016 that its total audience measured 246 million in 2015-16, 48% higher than the 2010-11 figure of 166 million. The Service does not report on the audience reach it has achieved as a percentage of the population in the countries to which it broadcasts, either globally or for individual language services.

3.3 Although the Service’s estimate of its global audience has increased since 2010-11, its global footprint is smaller. Its total potential audience has reduced by over 41 million as a result of its closure of five language services (paragraph 2.12).

3.4 In addition to the size of audience targets, the FCO and the BBC Trust set the Service two targets for audience satisfaction from 2013-14. The Service was to achieve higher ratings than its international competitors for ‘most trusted international news provider’ and ‘helps to keep me informed’. These replaced a previous target that the Service be rated more highly than its closest international competitor for awareness, reach, objectivity, relevance, value, and loyalty. The Service reported that it had met these targets in both 2013-14 and 2014-15 (Figure 18).

3.5 Although the Service has reported successful performance against both audience size and satisfaction targets, there are limitations both in how its targets are set and its performance is measured.

Target-setting

3.6 Before April 2014, the Service took a relatively simple approach to target-setting, adopting as its annual global audience target each year the audience figure it had achieved in the previous year (Figure 19). The Service had expected its audience to fall, in line with the long-term decline in its audiences for short-wave radio and its plans for further reductions in its short-wave services. It had, for example, forecast a fall of over 30 million in its global audience following the service closures announced in January 2011 (Figure 10), followed by further reductions over the course of its December 2010 six-year plan (Figure 20). It therefore believed that targets set at the previous year’s audience figure would be testing.

3.7 The audience size as measured by the Service has increased by 48% since 2010-11 (paragraph 3.2). The only year that the audience figure did not increase was 2013-14, when there was a reduction of less than 1% (Figure 19). There are a number of reasons why the forecast decline in audience numbers did not happen:

audiences figures for radio were higher than forecast as audiences for short-wave radio have not fallen as steeply as expected and the Service has cut back on the planned reduction of its short-wave broadcasts (paragraph 2.12);

the Service underestimated the increase in audience figures arising from its increased investment in television services (Figure 5 and paragraph 2.15); and

the increasing availability of the Service’s output on social media (paragraph 2.14) has resulted in higher audiences for its digital services than forecast.

3.8 After April 2014, a more refined approach to target-setting was adopted. From this point, three-year audience size targets were set on the basis of an analysis of market trends and future demand for its services. Targets were set for the total, television and online audiences (Figure 17), which the Service then cascaded down to set internal targets for 15 language services for the size of their online audiences. It did not set targets for their audiences on other delivery platforms.

3.9 The audience targets set as a result of this new approach were set at or below the Service’s forecast levels. An analysis provided by the Service to the BBC Trust in 2013 identified that, after taking account of various factors, its total 2016-17 audience could reach 200 million, while audiences for its television and digital services could reach 65 million and 33 million respectively by 2016-17. The targets proposed by the Trust, and accepted by the Service and the FCO, for television and digital audiences were lower than these forecasts. This was because the Trust considered that the Service required some flexibility and it wanted to avoid the Service focusing on the television and digital targets to the detriment of the total audience target of 200 million.

3.10 The Service reported in 2014-15 that it had already achieved two of the three audience targets, those for its total global and its television audience, after one year of the three-year target period (Figure 17). Despite this, the targets have not been formally reviewed and are still in place. As a global broadcaster, there may also be external factors that impact on the Service’s ability to achieve its targets. New targets will be agreed as part of the new operating licence under the new Charter.

Measuring performance

Audience size

3.11 The Service’s main measure for reporting on its audience is its Global Audience Measure (the Measure). This is the Service’s estimate of the number of adults worldwide who listen to its radio broadcasts, watch its television programmes, or access its online content in an average week, based on the information available to the Service on the date it calculates this estimate. Given its international coverage, obtaining the data for television and radio audiences can be difficult and, in some countries, impracticable, owing to local conditions. In May 2015, the Service calculated the 2014-15 Measure, covering 245 countries and territories, and including audience data for materials broadcast by partner organisations as well as by the Service itself.

3.12 Data for the Measure comes from a variety of sources and methods (Figure 21). In markets where these are available, the Service uses digital statistics and industry-approved ratings data. It supplements these with surveys, when this is affordable and practicable, often carried out jointly with other broadcasters. On average, over the last five years, the Service has carried out about 15 audience surveys a year. As a result, 40% of the survey data used in compiling the television and radio elements of the 2014-15 Measure had been obtained in the previous 12 months, with 10% of the data being between 5 and 10 years old. The Service excludes from the Measure survey data that are over 10 years old and countries, such as Syria, to which it broadcasts but for which it has no reliable data. The Service sometimes combines a country’s survey results if, for example, it carries out a national survey in one year and then surveys only the capital city the next year. In the most complex cases, the estimated audience for one country may be based on five or more surveys.

3.13 Year-on-year estimates for the Measure reflect year-on-year changes in how the figures are compiled as well as changes in performance. Factors impacting on the reported figures include:

Changes in the survey data

Changes in the Measure can result from the receipt of new, and the deletion of out-of-date, survey data. For example, the 2014-15 Measure included an audience of 2.7 million for Cameroon which was surveyed in 2014-15 for the first time since 2003 and whose previous data had been excluded in 2013. In contrast, data for 10 surveys was removed in 2014-15 due to the 10-year rule, reducing the Measure by 1 million.

The inclusion of data relating to new delivery media

More people are only accessing the Service’s output via social media. In 2014-15 the Service included for the first time the number of people who accessed its content on Facebook and YouTube, increasing the Measure by 3 million, and, in 2015-16, it plans to include those accessing content via other social media platforms.

Refinements to the methodologies used to guard against double and triple counting

As the Measure is a composite figure made up of figures for its audiences on different delivery platforms – radio, television and online – the Service has to adjust the data collected to guard against double-counting people who use more than one platform. The Service regularly refines how it does this in order to improve the Measure’s accuracy. For example, in the 2014-15 Measure, it attempted for the first time to account for people who accessed its content using more than one electronic device – lap-top, tablet or mobile phone.

Refinements to how it takes account of differing levels of audience engagement

The Service is unable to measure the amount of time people spend using its services. However, in 2014-15 it attempted for the first time to take account of the differing levels of engagement with its output between someone who listens to its radio programmes for several hours a day and someone who looks briefly at a Service web page. It only counted those audience members who actively engaged with its social media content by, for example, ‘liking’ or ‘commenting’ on the content. It is undertaking further research into audience behaviour to improve its calculation of the Measure in future years.

3.14 The reported figure for the Global Audience Measure is inevitably an estimate and subject to uncertainty. However, this is not sufficiently clear when reporting performance. The audience figure is reported to the BBC Trust and FCO and published in the BBC’s Annual Report and Accounts.10 The 2014-15 annual report stated that the Service’s audience totalled 209.8 million, exceeding 200 million for the first time. Although the Report noted that this was a snapshot figure, compiled from a number of data sources, it did not set out the inevitable limitations in the underlying data, such as the age profile of the data inputs (paragraph 3.12). In April 2016 when publishing its annual Global Audience Measure, the Service stated that the figures in the Measure have a standard margin of error of +/- three percentage points. It has not, however, estimated the total level of uncertainty attached to the reported total audience figure. This arises not only from the uncertainty attached to the survey and ratings data, but also how the total figure is compiled and the age profile of the data, which is not disclosed.

Audience satisfaction

3.15 Since at least 2009-10, the Service has reported that it has met its targets for audience satisfaction, but the data it uses for this do not relate specifically to the Service. The data that are reported are collected for BBC News as a whole. No separate data are collected on how the Service itself has performed. The BBC disclosed that it was using data relating to BBC News in its reported performance to the BBC Trust and FCO and in its published performance data from 2013-14 onwards.

3.16 According to the BBC, its overseas audience tends not to differentiate between the BBC’s different overseas brands and it considers BBC News to be a good proxy for the Service. However, 2014-15’s performance data were obtained from a survey of 12 countries, and European countries were over-represented within this group when considered against the countries the Service broadcasts to. European countries accounted for 25% of respondents to the survey, but only 6.5% of the Service’s audience in 2014-15. Also, the targets set by the BBC Trust and the FCO clearly specify that they relate to the Service, and the funding provided by the FCO up to April 2014 related to the Service, and not BBC News.

Wider benefits to the UK

3.17 Evidence on the benefits that the Service achieves for the UK is limited since, as a provider of accurate, impartial and independent news, the delivery of such benefits lies outside the Service’s remit and objectives. The BBC commissioned consultants in 2013 to examine retrospectively the indirect economic impact of its international output, covering both its commercial and public services. The results did not distinguish between the impact of the Service and the BBC’s commercial services. The consultants found that those who used the BBC’s services frequently had more favourable perceptions of the UK and 56% of respondents agreed that the BBC played a direct role in influencing their business decisions in favour of the UK. However, the evidence was limited to an online survey of attitudes towards the UK and the BBC among 875 ‘international business leaders’ in just three countries: the USA, India and Australia. Although these three countries accounted for 12% of the Service’s global audience in 2014-15, users of the Service’s non-English language services are under-represented in the survey.

3.18 In November 2015 the UK government announced extra funding of £289 million for the Service from 2016-17 to 2019-20 (paragraph 1.4) on the basis that, as a provider of accurate, impartial and independent news, the Service helps to strengthen democratic accountability and governance, and promote Britain and its values around the world. As at May 2016, the FCO had not confirmed how it would assess the impact of the extra funding against objectives.

10 Up to and including 2013-14, the audience figure was reported in the Service’s own Annual Review, published separately to the BBC’s Annual Report and Accounts.

Appendix One

Our audit approach

1 This study examined whether the Service is meeting its plans to change significantly its services and cost base, while managing the impact on current and future performance. It assessed whether the Service has:

delivered planned changes to its services and operations, and reduced expenditure, to achieve forecast benefits/impacts;

a clear strategy and well-evidenced understanding of the impact on value of the changes it has made; and

a clear strategy for managing risks to its influence, impact and cost-effectiveness in the face of a changing market for international broadcasting, and for achieving its targets for increasing its reach.

2 We applied an analytical framework with evaluative criteria that considered whether the Service is managed effectively. We drew on previous work on major project management across government.

3 Our audit approach is summarised in Figure 22 and our evidence base is described in Appendix Two.

Appendix Two

Our evidence base

1 We reached our conclusions on the Service after analysing evidence we collected between August and December 2015. Our audit approach is outlined in Appendix One.

Document review

2 We reviewed documentation provided by the Service to understand whether:

it had delivered planned changes to its services, operations and costs to date, and achieved the forecast benefits/impacts;

it had a logical and well-evidenced approach when planning and implementing cost reduction and changes to its services and operations; and

it had put in place adequate arrangements to managing risks to making further changes to its services, operations and costs.

Data analysis

3 We analysed the Service’s financial and performance data. This analysis allowed us to understand:

the reported performance of language services;

the Service’s estimates of the costs and benefits for each language service; and

the reliability and transparency of information underlying decisions about language services.

Interviews

4 We carried out semi-structured interviews with Service staff in London, Kiev and Cairo to understand:

how the Service has evolved in the context of a shifting digital landscape, cost reductions and increased international competition; and

how it sought to make significant cost reductions while minimising the impact on its objectives.

5 We interviewed key stakeholders, including the Foreign & Commonwealth Office (FCO), and other international broadcasters. We also liaised with specialist NAO colleagues on transformation and business modelling issues, and colleagues on the NAO team who produced the Exploiting the UK Brand Overseas report.11 These interviews allowed us to understand:

the strategic challenges faced by comparable organisations and how they have responded to these challenges; and

the Service’s relationship with the FCO.

11 Comptroller and Auditor General, Exploiting the UK Brand Overseas, Session 2015-16, HC 80, National Audit Office, June 2015.