Report

by the Comptroller
and Auditor General

Home Office

Upgrading emergency service communications: the Emergency Services Network

HC 627 Session 2016-17 15 September 2016

Our vision is to help the nation spend wisely.

Our public audit perspective helps Parliament hold government to account and improve public services.

The National Audit Office scrutinises public spending for Parliament and is independent of government. The Comptroller and Auditor General (C&AG), Sir Amyas Morse KCB, is an Officer of the House of Commons and leads the NAO, which employs some 785 people. The C&AG certifies the accounts of all government departments and many other public sector bodies. He has statutory authority to examine and report to Parliament on whether departments and the bodies they fund have used their resources efficiently, effectively, and with economy. Our studies evaluate the value for money of public spending, nationally and locally. Our recommendations and reports on good practice help government improve public services, and our work led to audited savings of £1.21 billion in 2015.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 14 September 2016

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

13 September 2016

This report examines the cross-government programme, led by the Home Office, to replace the existing emergency services mobile communications system.

© National Audit Office 2016

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.gsi.gov.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

11248 09/16 NAO

The National Audit Office study team consisted of:
John Bell, Paul Bilton, Adam Hughes, Richard Lewis, Robert Sabbarton and Kristianne Sharkey, under the direction of Louise Bladen.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Key facts

Key dates

Summary

1 Modern police, fire and ambulance services (the emergency services) rely on communications between control rooms and personnel in the field. These communications are currently provided by Airwave Solutions Limited (Airwave) through a series of contracts that now expire in 2019. These contracts cover 105 emergency services in Great Britain as well as 307 other public sector organisations.

2 In 2011, the government set up the Emergency Services Mobile Communications Programme (the programme) to look at options to replace Airwave when the contracts expire. The programme is part of the Home Office but is co-funded by the Department of Health, Scottish Government and Welsh Government. The programme’s objectives are to replace the Airwave service with one that matches it in all respects and:

makes high-speed data more readily available to the emergency services to improve their performance;

provides more flexibility to take advantage of new technologies as they emerge; and

costs less.

3 The government’s chosen option to replace the Airwave service is known as the Emergency Services Network (ESN). ESN will save money by sharing an existing commercial 4G network: the Airwave network is fully dedicated to public sector use. It will also bring better mobile-data capabilities than provided by Airwave.

4 The programme awarded the three main contracts for the provision of ESN in 2015 to Kellogg Brown and Root (KBR), Motorola Solutions Inc. (Motorola Solutions) and EE Ltd (EE). Some related contracts were awarded in June 2016 but others have yet to be awarded. In February 2016, Motorola Solutions bought the incumbent, Airwave, from an infrastructure fund managed by the Macquarie Group. The current plan is that the emergency services will start moving onto the new network in September 2017 and are due to complete this process in December 2019.

Scope of this report

5 This report examines the significant upcoming challenges that the programme will need to manage if it is to be successful, how it is managing them and why it has chosen this approach. This report looks at a live programme early in its delivery phase: it is too soon to assess whether the programme has achieved value for money. This report only looks at the services provided by Airwave that are relevant to the future programme.

6 We have defined good performance as there should be:

an agreed understanding between the programme, funding organisations and user organisations on the risks they will need to manage to deliver ESN, and similar risk appetites;

appropriate commercial arrangements in place for ESN;

best-practice processes in place for managing the delivery of ESN;

good consultation over a wide range of options before the ESN option was chosen; and

a business case which is based on strong evidence and reasonable assumptions.

7 We examined programme documentation, interviewed officials and suppliers and held workshops with programme officials and emergency services representatives. We commissioned a report on international provision of emergency service communications, which is available on our website. Full details of our methods are in the appendices.

Key findings

8 Airwave has delivered a communication service that has served the emergency services effectively in dealing with life or death situations. The Airwave network covers 97% of Great Britain, including nearly all roads and a small number of aircraft. The network has averaged 99.9% availability since April 2010 and provides capabilities for emergency service personnel to cooperate with those working in different regions or services. These capabilities exceed those available in all but two of the other G20 countries at the current time (paragraphs 1.2 to 1.8).

Risks with delivering ESN

9 ESN is inherently high risk and such an approach has not yet been used, nationwide, anywhere in the world. There are three main categories of risk associated with ESN: technical; user take-up; and commercial arrangements. These roughly align to the three major phases of the programme: design, build and test; transition; and operate. There is also an overarching risk due to the ambitious nature of the timeline adopted by the programme. Only South Korea is currently seeking to deploy a solution similar to ESN nationwide, but starts from a better base with significantly greater 4G coverage (Figure 1, paragraphs 2.1 to 2.3).

10 ESN is technically cutting edge. There are some significant technical challenges to delivering ESN.

Increasing the percentage of Great Britain’s landmass covered by EE’s network from 70% (as at July 2016) to 97%. The work to do this is shared between EE and the programme and their current projection is that sufficient coverage will be available by September 2017.

Developing handheld and vehicle-mounted devices that will work with ESN as no suitable devices currently exist.

Developing new push-to-talk software to enable ‘radio-like’ communications between emergency services personnel and control rooms.

Implementing the software and protocols that are needed to give emergency services personnel priority over commercial users of EE’s network.

Delivery by the programme against these technical challenges is by no means certain and, while total failure seems unlikely, there remains a risk that the programme will not be able to overcome these challenges for the cost or timetable proposed in the full business case, or to the satisfaction of users (paragraphs 2.3 and 2.4).

11 The programme’s success depends on the emergency services and other users choosing to take up ESN and make full use of it. The programme is not intending to force the emergency services to transition to ESN but has instead assured them that they can stay on Airwave until ESN is ‘at least as good as Airwave’. Defining this is complex and leaves room for disagreement, particularly over where is covered by the ESN service. There are also some elements of the functionality of Airwave where it is unclear how they will be matched in ESN. If even a small number of the emergency services and other users choose to delay transition, this will reduce benefits compared to the full business case. The full benefits of ESN rely on the emergency services exploiting high-speed data services by changing their operational behaviour, but supporting this is not part of the programme’s scope and the government is not yet clear on what support it may need to put in place (paragraphs 2.5 to 2.8, 4.18).

12 The commercial arrangements for ESN have separated the operational responsibilities of the emergency services from the commercial levers, which are held by the programme and therefore the Home Office. This separation has two elements. First, the majority of the cost of ESN will be paid for centrally. Second, emergency services will not have their own contractual arrangements for the full scope of ESN. Instead, they will have a call-off arrangement with one of the ESN suppliers, EE, but the terms of this are more limited than the contract they currently have with Airwave. For example, their contract with EE will give them very little direct recourse for poor service. Nor will they have a contract with most of the other ESN suppliers. Programme officials consider that, in practical terms, the arrangements under ESN are similar to those under Airwave. We have observed that under Airwave the emergency services make use of a wide range of supplementary communications services and the business case for ESN assumes these stop being needed. We consider that the commercial arrangements under ESN therefore create a risk that the emergency services feel they do not have sufficient control over the service they receive and may continue to make use of supplementary services, leading to a reduction in the benefits of ESN (paragraphs 1.14, 2.9 to 2.16).

13 Despite the inherently high level of risk, the programme has adopted a timeline for delivering ESN that is very ambitious. Programme staff and emergency services personnel all saw delivering ESN in line with the timeline in the full business case as very difficult. Programme officials told us that the current timeline contains no contingency during the design, build and test phase. Programme officials consider that it does have contingency, however, in the transition period. Emergency services personnel do not agree and told us that the transition period from September 2017 to December 2019 already gave them limited opportunity to plan or learn lessons from each other (paragraphs 2.17 to 2.22).

The programme’s approach to managing these risks

14 A 12-month delay to ESN could cost up to £475 million so the programme has put in place commercial and funding mechanisms that are designed to manage this risk. The programme’s commercial arrangements pass many elements of the technical risk to suppliers because, in the opinion of programme officials, they are best placed to manage these risks. While this is true if the risk materialises on a small scale, we consider that these arrangements could be detrimental to the overall commercial relationship between the programme and its suppliers if there are high cost increases or long delays. During transition, programme officials consider that most of the cost of delay, and benefit of achieving the existing transition plan, will fall on the emergency services, which will incentivise emergency services to transition without unnecessary delay. However, this is not certain as budgets beyond March 2020 have not yet been set (paragraphs 2.8, 2.14 to 2.18).1

15 In general, the programme has a positive delivery-focused culture that has helped it retain staff and manage issues as they have emerged. In contrast with other programmes that we have examined recently, the ESN programme has benefited from stability in staffing at both senior and junior levels. In interviews and workshops we consistently heard positive comments about the programme’s culture and focus. Staff on the programme have a strong record of delivering other projects. These factors have helped the programme manage challenges that have arisen to date. They also mean that it is well respected among stakeholders who were, for example, willing to approve investment in the programme despite wider government spending constraints. During the course of the study, the programme made changes in response to comments from us and other reviewers (paragraphs 3.2 and 3.3).

16 Nevertheless, the programme’s management of its key risks needs to improve if it is to deliver ESN successfully. For example:

The programme’s approach to technical assurance and testing needs to be better. The programme board lacks independent telecommunications expertise and the panel the programme set up to provide such assurance has not systematically analysed the risks. Furthermore, the programme’s testing plans are currently high level and there are differences of opinion between programme officials and suppliers on the scope of, and roles and responsibilities for, testing. In a programme this complex some assurance that is independent of suppliers, the programme and the emergency services would be beneficial (paragraphs 3.5 to 3.9).

User engagement could be better, particularly with police and non-emergency service users of Airwave. Emergency services representatives agreed that engagement over requirements had been good but perceptions were more mixed since then. Some emergency services representatives were unsure of the benefits of ESN to them, possibly because Airwave is currently largely paid for centrally. Some emergency services representatives also told us how programme officials do not always listen to challenges that they raise (paragraphs 3.10 to 3.13).

The circumstances in which the Airwave contract will be extended need to be more clearly set out. The programme has a clear contingency, to extend Airwave, and has agreed a cost for doing so. However, there is limited detail on how and when this contingency will be invoked and for how long. As a consequence, we found that there was not a shared understanding between programme officials, emergency services representatives and other stakeholders about contingency plans and how any delay will be funded (paragraph 3.4).

The service management arrangements once ESN is operational need to be more clearly articulated. At the moment it is unclear who in the Home Office will be responsible for ensuring ESN delivers its predicted benefits once it is operational. It is also unclear what governance will exist between that party and the emergency services to ensure that ESN continues to meet user requirements. The length of the new ESN contracts are much shorter than the Airwave contract and give the programme flexibility to change suppliers during the life of the business case (paragraphs 2.11 and 3.17).

17 The programme is behind schedule compared to the full business case and has responded by squeezing the time available rather than extending the overall time frame. The programme awarded contracts two months later than it expected in its full business case. Since contract award the programme delivered detailed designs three months late and has delayed the delivery of some elements of functionality by eight months. Overall, it is therefore between five and ten months behind the full business case. Programme officials consider that it has missed milestones due to factors outside their control. It has so far been reluctant to extend the Airwave contract and has instead reduced the time available to move the emergency services onto ESN by three months and introduced a more gradual approach to building and testing. In August 2016, the programme expected to turn off Airwave in December 2019, one month earlier than targeted in the full business case (paragraphs 2.17 to 2.22).

18 Overall, the programme, the Home Office and other sponsor bodies appear to be underrating the seriousness of the risks ESN poses. The emergency services demonstrated to us a low risk-appetite when it comes to deciding whether to transition to ESN. For example, they talked to us about plans to independently test ESN coverage because they were not convinced by the programme’s plans. By contrast, technology was not one of the top three risks raised with us by programme staff. Since the beginning of 2016, the Home Office has downgraded the risk of delivering ESN twice because it considered the risks to be under control. This meant that by June 2016 ESN did not feature on the list of risks escalated to the Home Office’s management board. We consider that, despite the programme’s mitigations, ESN remains an inherently high-risk programme that will require the highest levels of senior oversight throughout its lifetime (paragraphs 3.5 to 3.9, 3.18).

Why the programme chose to adopt these risks

19 ESN is the right direction strategically and the programme’s planned approach to delivery, if successful, will maximise benefits. Airwave is an expensive communication system costing £1,300 per handheld or vehicle-mounted device per year. Setting up ESN will cost an estimated £1.2 billion to March 2020 but after that ESN will cost an estimated £500 less than Airwave per device per year. ESN will have better data capabilities than Airwave, which should allow the emergency services to operate more effectively, and the commercial arrangements under ESN should make it easier to transition to newer technologies, such as 5G, when they arrive. From 2010 the government had a deteriorating commercial relationship with Airwave and considered that Airwave’s owners had an unsustainable debt position. Taken together, programme officials considered these factors created a strong case for moving to ESN as quickly as possible. All parties that we have spoken to, including Airwave, agree that ESN is the right long-term direction (paragraphs 1.13, 2.11, 4.2 to 4.6, 4.10 to 4.17).

20 However, we consider that in seeking to maximise benefits the programme’s planned approach to delivery has also maximised risk. The programme’s option appraisal shows that ESN carried the highest level of risk among the options it considered in detail. Further, the programme’s option appraisal did not cover options for a slower implementation of ESN to allow more time for build, testing and transition. No country yet uses 4G mobile technology for its emergency service communications and countries that are looking to implement it are planning to take a lower risk approach than that adopted by the programme. For example, South Korea, whose approach is the nearest comparator to ESN, is planning to use dedicated mobile spectrum for its emergency services communications rather than, as in Great Britain, sharing spectrum with commercial users. Australia is planning to use commercial 4G services for data first and moving to using it for voice later. All other options would have resulted in fewer benefits than the programme expects from ESN. Analysis, planning and procurement activities undertaken since inception of the programme have given programme officials more confidence in their ability to deliver ESN to time, quality and cost than when the option was agreed in 2013 (paragraphs 2.2, 4.7 to 4.9).

21 The benefits of ESN should be substantial but we consider that the business case may be overly optimistic in its valuation of these. The programme has estimated that the benefits of ESN will be worth £3.6 billion between April 2015 and March 2032. Valuing benefits is always difficult but we consider that a number of the assumptions that the programme has made in valuing these benefits may be optimistic. For example, in calculating how much ESN will save, the programme has assumed that Airwave will continue to cost the same in the future as it has done to date due to the difficult relationship they had with Airwave. Historically, however, the cost of Airwave has included designing and building the network which will not need to be repeated and we therefore consider that at least some discount should have been assumed (paragraphs 4.10 to 4.18).

Conclusion

22 The communication systems used by our emergency services can literally make the difference between life and death for members of the public and the services themselves. The existing system, provided by Airwave, works but at £1,300 per device is expensive. The need to save money and exit a difficult commercial relationship with Airwave has led the government to try and move to an approach that is not yet used nationwide anywhere in the world and carries significant implementation risk. ESN is the right direction strategically but we are concerned that the risks with getting there are under rated in the Home Office and elsewhere.

23 On the positive side, the programme has an energetic, delivery-focused culture that has helped it retain staff and manage issues as they have emerged. The programme needs to put in place more independent testing and assurance regimes for its technical solution and urgently improve its approach to engaging with the emergency services, on whose cooperation the programme depends.

Additional text requested by the Home Office

The Home Office has asked us to record that they have adopted their approach to equip the emergency services with the modern data communications capabilities they need and so welcomes the report’s key finding that ESN is the right direction strategically. The Department has also accepted the key recommendations. However, the Home Office does not agree with the NAO’s judgement about the Department’s acknowledgement of the programme’s risk, on incentives on users to transition, or the scale of benefits in the business case, considering that the programme and commercial approach are designed to maximise value for money and comply with procurement law.

Recommendations

For the programme

a The programme should improve the independence of the technical assurance arrangements it has in place. The programme should seek to recruit some external telecommunications expertise onto its programme board to provide more independent challenge of the programme. It should also look to ensure more independent testing assurance of ESN prior to transition.

b The programme needs to urgently develop a detailed contingency plan. So that the programme, the emergency services and suppliers can be clear on the circumstances in which Airwave will be extended and respond effectively to any problems, the programme should develop a detailed plan that considers some likely scenarios and responses, including funding requirements and sources.

c The programme needs to improve communications with the emergency services and other users of Airwave. To minimise the risk of unnecessary delay the programme should do more to engage with emergency services and other users. It could also do more, working with its sponsors, to clarify uncertainties around the extent to which future savings will benefit the emergency services to encourage them to move to ESN as quickly as it is safe to do so and ensuring the benefits of ESN are maximised.

d The programme needs to work with the Home Office, other sponsors and users to develop the service management arrangements for when ESN is fully operational. How the ESN service will be governed, managed and evolved during its life is currently unclear and this leads to a risk that user requirements will not be met.

For the Home Office and wider government

e The Home Office and other sponsors should work together to protect the programme from unnecessary staff turnover. The programme has benefited from stability in senior and junior roles. Lack of stability has been a problem for similar programmes in a number of our recent reports. All sponsors have a role to play in helping to maintain this. For example, they can reduce staff rotation requirements.

f When designing and approving commercial arrangements, departments and the Cabinet Office should carefully consider what will maximise the chances of successful delivery. The commercial arrangements for ESN are complex and, in our opinion, allocate risk to suppliers that they may not be best placed to manage. This has increased the risks that the ESN programme faces.

1 Unless stated otherwise the financial numbers used in this report are based on estimates provided by the programme, which we have converted into current prices using our own methodology.

Part One

Replacing the contract with Airwave

1.1 Police, fire and ambulance services (the emergency services) rely on communications between control rooms and personnel in the field to respond effectively to incidents, coordinate activities and ensure the safety of personnel in what are often life-threatening situations.

1.2 Historically, the emergency services provided these communications individually using unsecure analogue radio systems. In 2000, the then-Police Information Technology Organisation, a former arm’s-length body of the Home Office, signed a 20-year framework agreement with British Telecommunications plc (BT) to establish a new communication capability for police forces.2 Since 2001, the service has been provided through a separate company known as Airwave Solutions Limited (Airwave). Between 2007 and 2016, Airwave was owned by infrastructure funds managed by the Macquarie Group.3 In February 2016, Airwave was sold to Motorola Solutions Inc. (Motorola Solutions).

1.3 The Airwave service now covers police, fire and ambulance services, as well as other public sector users, throughout Great Britain. Separate arrangements exist in Northern Ireland. Airwave owns a telecommunications network with digital radio capabilities, known as terrestrial trunked radio (TETRA), which solely serves public sector users. These capabilities are more secure than the former analogue services and allow a greater degree of interoperability between the emergency services.

1.4 As at June 2016, 328,000 handheld, road-vehicle or helicopter devices were in use on the Airwave network across 412 public sector organisations (Figure 2). In addition to the emergency services, users include the National Crime Agency, Highways England, Transport Scotland, HM Revenue & Customs, local authorities, the Maritime and Coastguard Agency and mountain rescue services. Airwave provides a mainly voice-based communication service but can also carry limited (narrowband) data.

1.5 Airwave provides coverage to 97% of Great Britain’s landmass, including nearly all roads and built-up areas.4 It also provides coverage for a small number of aircraft and into the tunnels of metropolitan railways. Commercial mobile coverage is currently lower.5

1.6 The emergency services have benefited from very good availability of the Airwave network. In the six years between April 2010 and March 2016, the Airwave network has been available on average 99.9% of the time. The contractual target is for the network to be available 99.74% of the time in a month. Since April 2010, Airwave has only missed this target once. This was in December 2015 when performance dropped to 99.68% due to flooding in Northern England. The Airwave network is considerably more resilient in design than commercial networks.

1.7 For this report we commissioned international comparator work by Kable, a specialist ICT research company. This suggests that the current capabilities of Airwave exceed that in all but two of the other G20 countries (Figure 3).

1.8 We asked the emergency services for feedback on the current performance of Airwave. Although there were some concerns about coverage, the emergency services were generally content with the current performance.

The cost of the Airwave service

1.9 The government and emergency services expect to spend £377 million on the Airwave service in 2016-17, which is equivalent to £1,300 per device.6 In total, the government has paid £4.8 billion to Airwave between March 2001 and June 2016.7 This consists of a fixed charge paid by the Home Office and the Department of Health, supplemented by variable charges paid by the emergency services; 77% of the cost of Airwave is met centrally rather than by the emergency services themselves (Figure 4).

The programme to replace Airwave

1.10 In 2011, the government set up the Emergency Services Mobile Communications Programme (the programme) to look at options for replacing the Airwave service when the contracts end (for police this was May 2020). The programme is part of the Home Office but is co-funded by the Department of Health, Scottish Government and Welsh Government. Until April 2016, when responsibility for fire services transferred to the Home Office, it was also co-funded by the Department for Communities and Local Government.

1.11 In December 2013 the programme decided to replace the dedicated radio-based infrastructure used under the Airwave service with a mobile-data (4G) based technology using infrastructure shared with other users. It called its approach the Emergency Services Network (ESN) (Figure 5). In September 2015 it appointed Kellogg, Brown  and Root (KBR) to support the programme in implementing ESN. In December 2015 it appointed two further suppliers: Motorola Solutions and EE Ltd (EE). Additional elements are being undertaken through further procurements by the programme or the emergency services themselves.

1.12 The programme prepared a full business case for ESN in August 2015. It was approved by sponsor bodies HM Treasury and the Cabinet Office in November 2015. This set out the following key milestones:

award of main ESN contracts complete – October 2015;

design, build and test complete – July 2017;

transition complete, ESN fully operational – January 2020; and

business case ends – March 2032.

How these key milestones evolved from the outline business case to full business case and what they currently are is discussed in Part Two.

1.13 Based on the full business case, the programme expects to spend, over the 17 years from April 2015 to March 2032, £5,207 million. Of this, £1,445 million (28%) is on running down the Airwave contract. ESN is expected to cost £3,762 million over 17 years, of which £1,182 million will be spent on building and transitioning to ESN in the five years to March 2020 (Figure 6). After March 2020 the programme expects ESN to cost £800 per device per year and therefore represent a significant saving over Airwave.

1.14 The emergency services are expected to directly contribute £354 million towards the cost of the mobile services contracts currently being provided by EE. They are also expected to incur an additional £825 million on their other areas of responsibility. In total they are therefore expected to pay £1,179 million, or around a third of the total cost of ESN.

2 We examined the procurement of Airwave in 2002. See National Audit Office, Public Private Partnerships: Airwave, Session 2001-02, HC 730, April 2002.

3 Between 2009 and 2012, 50% of Airwave was owned by an investment fund not managed by Macquarie.

4 Airwave covers more than 99% of sealed roads. A limited number of remote roads and minor tracks are not covered by Airwave.

5 Ofcom, Connected Nations 2015, December 2015, available at: http://stakeholders.ofcom.org.uk/market-data-research/market-data/infrastructure/connected-nations-2015/

6 Throughout the report we have used the term ‘per device’ but strictly the calculation is ‘per connection’ as the emergency services may have spare devices that are not connected to a network.

7 All amounts in this report are expressed in 2016-17 prices.

Part Two

Challenges with delivering ESN

2.1 This part of the report sets out the four main challenges we have identified with delivering Emergency Services Network (ESN). There are three main categories of risk associated with ESN: technical; user take-up; and commercial arrangements (Figure 7). These roughly align to the three major phases of Emergency Services Mobile Communications Programme (the programme): design, build and test; transition; and operate. There is also an overarching risk due to the ambitious nature of the timeline adopted by the programme.

Technical delivery risk

2.2 The research we commissioned from Kable looked at four emergency services networks being developed across the world, in Australia, Germany, South Korea and the United States. It concludes that ESN is the most advanced programme in the world at the moment. The programme is one of only two in the world looking to run its emergency communications with a major commercial network component. The other, South Korea, has considerably higher 4G coverage today than the EE Ltd. (EE) network on which ESN will rely.8 South Korea is also planning to use radio spectrum that is dedicated to the emergency services rather than shared with commercial users, as in ESN. Other countries are pursuing alternative solutions either fully or partly based on older terrestrial trunked radio (TETRA) technology and dedicated networks.9

2.3 The technical challenges to implementing ESN can be grouped as:

Coverage

EE’s existing 4G network needs to be increased to match Airwave Solutions Limited (Airwave). EE is currently increasing the coverage of its 4G network to meet contractual and wider operating requirements. On a comparable basis the coverage of EE’s network was 70% in July 2016 whereas Airwave’s was measured at 97% in December 2015.10 Specific solutions for London Underground and communicating with helicopters and aircraft (known as air-to-ground) need to be developed. Elements of the resilience of EE’s network need to be improved and capacity may also need to be improved in certain locations to ensure the functionality of Airwave can be matched. The work to do this is shared between EE and projects managed by the programme. The current projection is that sufficient coverage will be available by September 2017 (Figure 8).

Devices

As at July 2016, there are no devices, such as mobile phones, that will work with ESN, as ESN uses software standards that are only just coming into use. These devices are needed for the end of 2016 to allow for testing ESN. The programme is working with manufacturers to achieve this.

Integration

The ESN end-to-end solution will require more points of network integration than the solution under Airwave due to the greater number of different suppliers involved. The programme needs to ensure that all these networks work together, have sufficient capacity to meet the needs of the emergency services and are reliable enough for voice communications.

Prioritisation

For it to be possible to provide emergency communications over a commercial network, it is necessary for the emergency services to be given priority over other users. The software and standards for doing this exist but have not yet been deployed on the EE network. A network software upgrade is therefore required.

Software development

Various elements of software need to be developed for ESN to match the functionality of the Airwave service. The most significant of these are ‘push-to-talk’ and ‘group-call’ applications that will allow ESN to emulate radio-like functionality of talking to control rooms and each other. Similar software to this exists but upgrades are required to fully meet the ESN requirements.

2.4 Delivery by the programme against these technical challenges is by no means certain and while total failure seems unlikely there remains a risk that the programme will not be able to overcome these challenges for the cost or timetable proposed in the full business case or to the satisfaction of users.

User take-up risk

2.5 The programme is planning to start transitioning to ESN in September 2017. However, programme officials have agreed with representative organisations and the devolved governments in Scotland and Wales that they will not force the emergency services to transition to ESN. Emergency services can instead stay on Airwave until they agree ESN is ‘at least as good’ as the service provided by Airwave.

2.6 This may prove to be a difficult standard to agree. Current users of Airwave told us that ensuring the coverage and resilience of the EE network matches Airwave was their main concern with moving to ESN. There were indications in these discussions that users may be very demanding of ESN; expecting an almost exact match of where is covered compared with Airwave. The ESN contracts do not specify which areas of the country are to be covered, instead replicating the overall coverage standard of the Airwave contract. This means that there may be specific locations covered by the Airwave network that are not covered by the ESN network. Since payments to suppliers commence once contractual standards have been met there is a risk of payments being due before the emergency services agree that ESN is ready.

2.7 The emergency services and other users of Airwave were also concerned that ESN will not replicate all of Airwave’s functionality. It is unclear, for example, whether the current specifications for ESN will meet the security needs of counter-terrorism and covert operatives. Another concern was the use of direct device-to-device voice calling, particularly used in covert and counter-terrorism operations. Motorola Solutions’ bid states that this functionality will initially require a work-around such as using a TETRA (Airwave-compliant) device and will not be available on ESN for some time.

2.8 If even a small number of the emergency services and other users choose to delay transition then the overall programme will be delayed compared with the full business case, reducing benefits. Programme officials consider that the emergency services will be considerably worse off if transition is not completed to time and that this will be an incentive against unduly delaying the process. However, this is uncertain as the costs and benefits of delay fall beyond 2020 into a period when budgets have not been set.

Commercial arrangements risk

2.9 ESN has a complex set of commercial relationships due to its overall structure and financial risk-allocation.

Commercial structure

2.10 In line with current government policy, the programme is replacing the ‘prime-contractor’ arrangement in place with Airwave (where government contracts with Airwave, who provide some services themselves and sub-contract other elements) with multiple direct commercial relationships. The ESN commercial arrangements will therefore include at least seven frameworks, contracts or grant agreements put in place by the programme as well as a number of arrangements sourced locally (Figure 9). Some of these procurements will be run differently for ambulance trusts for whom the Ambulance Radio Programme, in the Department of Health, is undertaking activities that police and fire authorities are doing themselves.

2.11 This structure is designed to bring commercial benefits. Compared with the Airwave arrangements, it should make it easier to respond to trends in the ICT and telecommunications market, such as 5G, by having multiple suppliers on shorter contracts, each of which can be changed if a cheaper or more innovative supplier becomes available. It is also designed to reduce the risk of being locked-into a single supplier and the overall cost by having the programme bear the risk of integrating separate components. The commercial arrangements should make the achievement of these benefits easier than under Airwave, although it is too early to assess whether they can actually be realised.

2.12 The way the programme has put in place the commercial structure means that the emergency services do not have commercial levers over the full service they are receiving (Figure 10). For example, their contract with EE will give them very limited direct recourse for poor service. They will also have no contract with Motorola Solutions or the provider of the control room interface into ESN, despite the importance of these suppliers in the end-to-end service. This structure means that the operational responsibilities of the emergency services have been partly separated from the commercial responsibility held by the programme. In 2014 we looked at contract management in the Ministry of Justice and found that operational and commercial responsibility had been separated. We found that this had led to instances where operational managers felt unable to influence performance so that it met their needs.11

2.13 Programme officials consider that, in practical terms, the arrangements under ESN are similar to those under Airwave. We have observed that under Airwave the emergency services make use of a wide range of supplementary communications services for both data and voice usage. The business case for ESN assumes that those used for data stop being needed (see paragraphs 4.16 and 4.17) and there would obviously be further savings if those used for voice were also no longer needed. With this in mind, we consider that the commercial arrangements under ESN create a risk that the emergency services feel they do not have sufficient control over the service they receive and may continue to make use of supplementary services, leading to a reduction in the benefits of ESN.

Financial risk allocation

2.14 This section will briefly discuss the allocation of three of what we consider to be the most significant commercial risks of moving to ESN. In doing this we are primarily considering the arrangements under the two most significant contracts; the ones with Motorola Solutions and EE.

Technical design risk

The price for building, testing and early operation of ESN was fixed in December 2015 before detailed designs were agreed. As a result, the risk that costs need to rise due to unforeseen technical challenges has been largely passed to suppliers. Programme officials have done this as they consider that suppliers are best able to bear this risk. While this may be true if the risk materialises on a small scale, we consider that these arrangements could be detrimental to the overall commercial relationship if technical challenges prove difficult to overcome, resulting in high cost increases or long delays. A large number of programmes, including the Home Office’s own e-borders programme, have demonstrated that suppliers are often not well placed to bear technical design risk in complex ICT programmes.12 This is particularly acute for ESN where Motorola Solutions and EE are each delivering part of the overall solution and only had eight hours of discussions with each other prior to contract signature.

Volume risk

During the term of the Motorola Solutions and EE contracts 85% of the payments to suppliers are fixed irrespective of usage. This means that the risk of getting users onto ESN is largely borne by the government. In telecoms, this risk is often borne by suppliers. Programme officials told us that through the bidding process they tried to get suppliers to give a higher volume-related element in their pricing but, due to the high, ESN-specific design and build costs, no supplier was willing to take this risk. They also told us that the programme is better placed to take on volume risk.

Commercial exploitation risk

Motorola Solutions is developing new software through this programme. EE will have the option of using sites built in remote areas to increase its commercial 4G coverage.13 In pricing ESN, both suppliers will have made assumptions about how much they can earn from these opportunities as, under the contracts, the taxpayer is not entitled to any incremental revenues. As a result, commercial exploitation risk has been passed to suppliers.

2.15 Taken together, this risk allocation means that there is a higher chance of tension in the commercial relationship between the programme and suppliers as there may be significant financial gains or losses by suppliers compared to what was expected. Some, but not all, financial gains by suppliers will be shared with the programme through a gain-share mechanism. We have observed in a number of other programmes, including the Home Office’s e-borders programme, that the accumulation of commercial issues can become a problem in delivery and the programme will need to ensure that relationships are good with suppliers.

2.16 The reason the programme has allocated risks in this way is because it has structured its contracts as a service. In our experience, such a contractual structure is better suited to a situation where there is a smaller design and build component than is the case with ESN. Programme officials did consider alternatives but felt that affordability, Cabinet Office guidance and HM Treasury guidance required them to adopt the structure they did. Minutes and interview evidence suggest that programme officials and suppliers would have preferred a different risk allocation. However, there is limited documented challenge by the programme of its understanding of the guidance that it considers it had to follow. Programme officials now consider that its commercial approach has allocated risks to those best placed to manage them, with government managing volume risk while suppliers assume design and commercial exploitation risk.

Delivering to time

2.17 The outline business case in December 2013 was the first significant baselining of the target dates for delivering the programme. Figure 11 shows how these dates have evolved through to full business case, contract award and current expectations as at August 2016. Contract award was completed seven months later than planned at outline business case. However, the programme has brought forward the end date by three months, compressing the overall timeframe. In doing this it has significantly shortened the time available for transition. Programme officials consider that changes to the timeline so far have been due to factors outside their control.

2.18 The programme has shortened transition in this way due to the high cost of delay. Early in its lifecycle programme officials were very concerned about the risk of Airwave failing to provide a reasonable service at a reasonable cost during transition. This shaped much of their thinking on timing. In February 2016, Motorola Solutions bought Airwave (see paragraph 3.15). At this time, the programme negotiated a number of extensions for individual emergency services such that all contracts with Airwave now have a common end date, December 2019. The programme also negotiated a price for further extensions on either a nationwide or regional basis. This price, though, is significant: a 12-month extension of the Airwave service to cover a longer transition period could cost the taxpayer an estimated £475 million, if all emergency services nationwide chose not to use ESN.

2.19 In workshops we ran with programme staff and emergency services personnel, delivering ESN on time was consistently seen as one of the highest risks for the programme (Appendix Three). Programme staff told us, for example, that the 21 months currently allowed for designing, building and testing ESN contains no contingency.

2.20 There is a difference of opinion between the programme and the emergency services over the deliverability of the transition phase. Programme officials consider that the 27 months currently planned for transitioning onto ESN includes contingency. This plan (Figure 12) typically allows 12 months for a region to transfer but programme officials consider it should only take 10 months. They also consider that the plan can be compressed. At the moment two-thirds of regions will be transitioning at peak in June, July and October 2018 and the programme officials believe that this could be compressed so that the peak is either higher or longer, or both.

2.21 By contrast, the emergency services consider that the current plans for transitioning onto ESN are very tight and give little opportunity to learn lessons from early transition activity and limited time overall to plan. Some emergency services representatives told us that this could reduce the benefit of ESN as they would have to replace Airwave in a like-for-like manner, instead of exploring how best to use the new capabilities. More generally, we observe that the compression in time frames means that transition is getting close to being a ‘big-bang’ implementation, which has been shown to be a problem in IT-enabled programmes.

2.22 In assessing the programme we consider that the full business case provides the best benchmark as it is the latest set of targets proposed by the programme, agreed with sponsors and linked with costs and benefits. The programme is already beginning to find the full business case timeline difficult to achieve. Between August and December 2015, the programme slipped by two months (Figure 10) due to delays in the procurement process. Since contract award, detailed designs were finalised three months late, while elements of the functionality contracted to be delivered by August 2016 will now be delivered in a phased manner, with some coming as much as eight months later than planned when contracts were signed in December 2015. Overall the programme is therefore between five and ten months behind the targets it set itself in the full business case.

8 The OpenSignal availability measure of coverage, (details of which are available at: http://opensignal.com/) lists South Korea’s coverage as 97% in December 2015 compared with 60% for EE. This differs from the basis used in the contracts with Airwave and EE.

9 Kable, First Responder Solutions in the UK and Internationally, August 2016, available at: www.nao.org.uk

10 See Appendix Two for how coverage is measured for ESN purposes.

11 Comptroller and Auditor General, Transforming contract management, Session 2014-15, HC 268, National Audit Office, September 2014.

12 Comptroller and Auditor General, E-borders and successor programmes, Session 2015-16, HC 608, National Audit Office, December 2015.

13 Other operators have the right under a European state-aid agreement to also deploy their own equipment on masts paid for by the programme.

Part Three

The programme’s approach to managing delivery

3.1 This part of the report will evaluate:

the programme’s overall management and culture;

how the programme is managing specific technical, user and
commercial risks; and

oversight of the programme by the Home Office.

Programme management and culture

3.2 Our recent reports have highlighted problems in retaining senior and junior staff within high-profile programmes.14 This has not been the case for the Emergency Services Network (ESN). The senior responsible owner set the Emergency Services Mobile Communications Programme (the programme) up in February 2011 and has been in post since then. The current programme director joined the programme in 2013 before the outline business case was finalised. Turnover across the programme has risen as the  programme has grown but is still low (Figure 13).

3.3 In interviews and workshops with those outside the programme, we heard consistently positive feedback about the programme’s leadership, the energy shown by its staff and the focus on delivering as quickly and efficiently as possible. This was sometimes tempered, however, by a view that programme officials did not always listen to challenge from others about its approach. Staff on the programme have a strong record of delivering other projects. Stakeholders in other government departments were particularly complimentary about the programme’s leadership. This has helped the programme secure funding for its business case, despite the current constraints on public sector spending. During the course of preparing this study we observed how staff responded to continuous feedback we and others provided.

3.4 In a programme this complex we would expect well-developed contingency plans. The programme’s contingency measure is clear – to extend the Airwave contract. However, its contingency plans are very high-level. There is no detail on the scenarios that may occur, what early-warning indicators the programme will monitor to identify emerging problems, and how the existing sequence of development and transition could be changed to accommodate any problems. As a consequence, there is insufficient shared understanding between the programme, the emergency services and other stakeholders about contingency plans and how any delay will be funded.

Programme’s management of the technical delivery challenges

3.5 Programme officials recognise many of the technical challenges the programme face. However, in interviews, programme officials and key stakeholders consistently appeared to view these challenges as easier to overcome than we would. In a workshop we held with them, technology did not feature as one of the programme’s top three areas of risk. On the other hand, the emergency services consistently identified technology, and in particular the coverage and resilience of ESN, as their main concern (Appendix Three).

3.6 We are not convinced that the programme’s testing strategy is sufficiently robust given the complexity of the technical challenge. The programme’s testing strategy relies on Motorola Solutions Inc. (Motorola Solutions) and EE Ltd (EE) undertaking the early testing themselves, with Kellogg Brown and Root (KBR) providing assurance of the test design and implementation. This will be supplemented by field trials and regional pilots to test the service in a live environment. These trials and pilots will be undertaken by the emergency services, supported by suppliers and assured by the programme. The plan is that these events will gradually increase in scope, scale and duration. All three suppliers will receive incentive payments if testing is completed on time.

3.7 Our concerns are two fold. First, at the time of our review there was limited detail available on the timing, scope or processes for these tests and differing understandings on roles and responsibilities between suppliers and the programme. Second, we consider that on a programme this complex there would be benefit from greater independent testing or assurance of ESN and, without it, there is a higher than normal risk that aspects of functionality or user requirements may be overlooked.

3.8 The programme’s plans to test coverage were of particular concern to the emergency services. Airwave Solutions Limited (Airwave), when launching its service, was required to test-drive the network, testing the voice quality and signal strength approximately every 25 metres. It also undertook walk-testing in pedestrianised and some indoor areas and has repeated the process since. By contrast, the programme plans for EE to undertake limited drive-testing, just sufficient to assure the programme that EE’s prediction models are accurate. Some of the emergency services told us that this approach was unlikely to satisfy them and that they would undertake full drive-testing before agreeing to transition.

3.9 Technical oversight of the programme is currently insufficient. The programme has a technical steering group. Between June 2014 and June 2016 this met infrequently and did not provide a robust technical challenge function by, for example, systematically considering programme risks. The group was re-constituted in June 2016 but it is too early to judge its effectiveness. The Home Office’s chief technology officer has only attended the programme’s main board twice since the beginning of 2014. The Government Digital Service has attended the programme’s main board regularly, but representatives do not have a telecommunications specialism. Without senior independent challenge of the programme’s technical solution there is a risk that problems, or opportunities, are missed.

User engagement

3.10 The programme’s main route for engaging with the emergency services is through a team of business change leads. These are often seconded from the emergency services to the programme.

3.11 Views on engagement were mixed. The emergency services told us that engagement on user requirements in 2013 had been good, with the programme holding an extensive range of workshops to identify the features they needed from a new system. Representatives of the Scottish emergency services and fire and ambulance services in England and Wales have been generally content with engagement since then. However, police forces in England and Wales told us there was room for improvement. We spoke to a limited selection of the 307 other public sector organisations using Airwave. They told us of feeling marginalised by the programme’s focus on the emergency services. In July 2016 the programme filled a new role as head of user engagement.

3.12 Some emergency services representatives were particularly concerned about the funding and benefits for ESN. Airwave is mostly paid for by the Home Office, the Department of Health, the Scottish Government and the Welsh Government. Since early 2016 the programme has engaged with police representatives to explain how the funding and benefits will work for users. However, some emergency services continued to tell us that, despite the strong savings case for ESN, they may not be better off. This is because the savings are in future years for which funding allocations have not yet been announced. Furthermore, some police forces told us of concerns that funding allocations for transition were insufficient to meet the costs they were likely to incur.

3.13 The emergency services were particularly concerned about their capacity, and that of their suppliers, to successfully make the transition to ESN. The transition plans in place at the time of our fieldwork for this study were not very detailed. The emergency services generally accepted this, as major suppliers had not long been appointed and work on transition was only just beginning. But they did emphasise the need for this to happen quickly as they need to prepare, as well as manage existing contracts such as those for mobile data.

Programme’s management of the commercial challenges

3.14 We observed that the programme has been reasonably active since contract award in managing the potential risks of its commercial approach.

Early performance by the delivery partner, KBR, was not satisfactory. Programme officials recognised this and required changes in staff and approach. KBR has implemented these changes and shown improvement by, for example, achieving the May 2016 detailed design milestone.

Motorola Solutions has to deal with a lot of the complexity of ESN, as it is responsible for building new applications and carrying out much of the operational integration. We observed how both senior and junior members of the programme have spent significant time at Motorola Solutions’ sites.

3.15 Programme officials also agreed concessions with Motorola Solutions when Motorola purchased Airwave in February 2016. This purchase carries significant commercial risk for the programme as Motorola Solutions is now both a supplier on ESN and the main incumbent supplier. The programme considers, however, that this should help with the overall delivery of ESN and both the programme and Motorola agree that the relationship between government and Airwave is now much better. In considering Motorola’s purchase of Airwave, programme officials has agreed a flexible basis on which Airwave can be extended, a firm price for this, re-use of some of Airwave’s radio masts for ESN, and a simpler solution for working across ESN and Airwave during transition than had previously been planned. The programme also negotiated a deed of undertaking to reduce the extent to which Motorola Solutions could benefit from delays in delivering ESN and thereby exploit its position.

3.16 The purchase of Airwave, however, does introduce a potential risk around devices as Motorola Solutions is both a device manufacturer and an approver of devices for use on Airwave and ESN. The programme and Motorola Solutions consider that procurement, process and contractual controls on device approval adequately mitigate this risk.

3.17 A key element of how government can mitigate the commercial risks once ESN is operational will be through the programme’s proposed service management arrangements. The service management framework was due to be approved in May 2016. This has not yet occurred and we reviewed a draft in August 2016 and noted some omissions. Notably, there is no clarity yet on the ownership of the service within Home Office or the governance arrangements between the Home Office and the emergency services. These issues are not yet on the critical path for the programme, but early resolution would build confidence, among users that ESN will respond to their evolving needs.

How the programme is managed within the Home Office

3.18 The inherently high level of risk involved with delivering ESN should mean that the highest levels of senior oversight occur. Prior to contract awards, the programme was subject to accounting officer and investment committee review within the Home Office. At this stage the programme carried the Home Office’s highest ‘black’ risk rating. However, since December 2015 the programme’s risk rating within the Home Office has been downgraded twice to a medium ‘amber’ risk level. This meant that, by June 2016, the programme did not feature on the list of Home Office risks escalated to the management board. These downgrades were based on completing the procurement of the main contracts and the technical and commercial benefits resulting from Motorola’s purchase of Airwave. However, they also occurred at the same time as the programme was entering its critical design, build and test phase. The programme considers it has a strong approach to mitigating risk. Our view is that despite the programme’s mitigations, ESN remains an inherently high-risk programme that will require the highest levels of senior oversight throughout its lifetime.

14 Comptroller and Auditor General, E-borders and successor programmes, Session 2015-16, HC 608, National Audit Office, December 2015; Comptroller and Auditor General, Early review of the Common Agricultural Policy Delivery Programme, Session 2015-16, HC 606, National Audit Office, December 2015; and Comptroller and Auditor General, Universal Credit: progress update, Session 2014-15, HC 786, National Audit Office, November 2014.

Part Four

Why the programme adopted this approach

4.1 This part of the report will evaluate:

the strategic case for moving to the Emergency Services Network (ESN);

options considered before deciding on ESN; and

the benefits that the programme expects to result from ESN.

The strategic case for moving to ESN

4.2 The Emergency Services Mobile Communications Programme’s (the programme’s) strategic case for ESN outlined three principal drivers for change:

contracts with Airwave Solutions Limited (Airwave) were due to expire in May 2020;

current contracts with Airwave do not represent lowest cost for the taxpayer; and

the emergency services increasingly need high-speed mobile data capabilities which Airwave cannot support.

4.3 There are three main reasons why the government’s situation with Airwave made it consider change as the contracts expire.

The government had limited leverage over Airwave

The government was unable to agree with Airwave a list of assets that it would own once contracts expired, despite it having paid for a number of them in full. This limits the government’s control over the use and cost of the assets in future. There is also no like-for-like competitor for Airwave.

Airwave itself had an undiversified business model

Most of Airwave’s revenues, about 98% in 2015-16, come from selling its network to the public sector. This means it has limited opportunities to make profits from other sources. By moving to ESN, the programme hoped to attract a supplier with a more diversified business model who could offer a more competitive price.

The government had a poor commercial relationship with Airwave

In 2010, Airwave became subject to the Cabinet Office’s strategic supplier management approach. At this time, the Cabinet Office was approaching all its strategic suppliers and asking for discounts as part of the government’s austerity initiative. Airwave, however, had not yet broken even on its investment in the network – this did not occur until December 2012. The government also believed that Airwave’s owners had an unsustainable debt position which limited its ability to offer significant discounts. The government was not willing to offer extensions in return for a discount and, without this, an agreement could not be reached. These discussions led to a deteriorating commercial relationship between the government and Airwave.

4.4 Airwave does look expensive for the taxpayer. At £1,300 per device per year the Airwave service is more expensive than most mobile communication services. This is partly explained by the dedicated nature of the Airwave network, which limits the extent to which Airwave can spread its fixed costs. It is also caused by the private-financing initiative model that the government uses for buying services from Airwave. These spread the cost of designing, building, maintaining and operating the Airwave network over the 20-year term of the contract.

4.5 We are unconvinced that the programme needed to adopt ESN to get the data capabilities the emergency services need. During our field visits we observed extensive use of mobile-data services by the emergency services using existing 4G contracts. In most cases the use of mobile data today is through dedicated devices, meaning personnel face the operational inconvenience of having to carry two devices. However, some ‘dual-mode’ devices capable of running both Airwave and mobile data currently exist and programme officials are considering the need for more as a contingency during the transition to ESN.

4.6 Commentators agree that mobile data (4G) on which ESN is based may present a viable alternative to the digital-radio technologies used by Airwave. We have discussed the programme’s approach with industry commentators, Airwave and other stakeholders. All agreed that in the long term emergency services communications would move to a mobile-data solution as the security and functionality of mobile data improved. However, at the moment the standards that underpin mobile data have only recently started to look at embedding the requirements of emergency service communications and are not yet fully developed. That standards are evolving to cover these requirements is partly a response to lobbying by the UK government and other jurisdictions.

The programme’s option appraisal

4.7 The programme carried out its main option appraisal in its outline business case finalised in December 2013. This considered three main options: stay on Airwave; build a 4G network dedicated to the public sector; and the chosen, ESN, option. ESN is the most beneficial and highest-risk option considered. Based on the research from Kable we have identified a country that is currently pursuing each of these three options (Germany, USA and South Korea respectively) (Figure 14).

4.8 Other countries are planning to take a lower-risk approach than that adopted by the programme. For example, South Korea, which is the nearest comparator to ESN, is planning to use dedicated mobile spectrum for its emergency services communications rather than, as in Great Britain, sharing spectrum with commercial users. Australia is planning to use commercial 4G services for data first and move to using it for voice later, once technology is more mature. An option like that adopted in Australia was examined by the programme but not shortlisted as, compared to ESN, they considered it to be either too expensive, if commercial 4G coverage was extended into rural areas, or not sufficiently beneficial if coverage was not extended.

4.9 The programme did not consider options around the timing of the move to ESN. A slower timetable would have been more expensive than the chosen approach due to the extra cost of Airwave but would reduce the level of challenge adopted. Since the ESN option was chosen in 2013, the programme has carried out market engagement, planning and procurement activities which have given it a greater confidence in its ability to deliver ESN to time, quality and cost.

The benefits of ESN

4.10 The programme’s full business case, finalised in August 2015, anticipates that quantified and non-quantified benefits will result from switching to ESN. Valuing financial benefits can be challenging and the programme quantified benefits totalling £3,641 million, generated through a mix of cost savings, extra revenue for the taxpayer, productivity improvements and wider economic benefits (Figure 15).15

Direct cost savings

4.11 When fully operational, ESN is expected to cost £800 per device per year compared to the £1,300 per device currently paid for the Airwave service. Taking set-up costs into account (Figure 5), the programme’s business case anticipates savings of £1,153 million for the emergency services over 17 years. In addition, the business case forecasts income for the programme of £356 million from other public sector users, which will be re-distributed to the emergency services.

4.12 In calculating this saving the programme has assumed that Airwave will cost the same in the future as it has done so far, which we would not ordinarily expect to be the case. The existing contract with Airwave included the cost of designing and building the Airwave network, as well as the cost of maintaining and operating it. In the future, design and build costs will not need to be repeated, although some asset renewal costs would be anticipated, and maintenance costs will likely be lower due to technical improvements. By assuming that the contract with Airwave will continue to cost the same in the future as it has so far, the programme assumes that Airwave would make higher profits in the future.

4.13 The programme has made this assumption as a result of Airwave’s market position and the debt position of its owners (see paragraph 4.3). Programme officials believe this is reasonable as they consider Airwave had every opportunity to submit a competitive price for the government to continue on Airwave and did not. In the opinion of programme officials, Airwave’s owners, who were managed by the Macquarie Group, were assuming substantial future revenues in their accounts from Airwave until June 2015. In considering this argument we observe:

Airwave and Macquarie did make various offers to reduce their price to the government between 2010 and April 2014 but the government considered that these offered insufficient value; and

programme officials made a conscious decision not to negotiate with Airwave on a price for the period to 2032, over which the business case has been assessed.

4.14 Independent benchmarking carried out for the programme by Gartner Consulting estimates that just 40% of the current cost of Airwave is for the ongoing service; the remaining 60% finances the design, build and maintenance of the network. In the future some capital maintenance costs would be anticipated, such that the ongoing cost of the network would be higher than 40%. However, these are very unlikely to be comparable to those incurred in the initial build of the network. Therefore, our view is that savings in the business case are likely to be overly optimistic.

4.15 The savings in the business case for other public sector users of Airwave (£525 million, as shown in Figure 15) have been calculated in a similar way to those for the emergency services and are therefore subject to the same methodological limitations.

Productivity and other savings for the emergency services

4.16 The programme’s full business case assumes that ESN’s 4G capabilities will enable the emergency services to make two additional savings above the direct cost savings.

Productivity and service improvement benefits, worth £841 million, from making use of ESN’s 4G capabilities. This category includes a number of non-quantified benefits such as improved medical care resulting from data-streaming between hospitals and paramedics.

A saving of £590 million from no longer needing separate mobile-data devices as ESN provides this capability.

4.17 Our work with the emergency services has given us some reassurance that there is scope to make better use of data within the emergency services and existing data contracts that can be stopped. However, we cannot be sure that the valuations used in the programme’s business case are correct.

The way the productivity benefits have been valued, including the very limited nature of consultation with emergency services about them, mean that we cannot be assured that £841 million is the correct valuation.

For commercial and operational reasons the emergency services seem unlikely to move off existing data contracts as rapidly, or as fully, as the programme’s business case assumes.

4.18 Without major business change by the emergency services and further support to enable this there is a significant risk that the productivity and other savings will not be maximised. Supporting the achievement of these benefits is not part of the programme’s scope and the government’s plans on what additional support the emergency services may require are at a very early stage. We previously found that the Home Office’s police mobile information programme, which looked to increase the use of mobile data by police forces, had not been value for money. This took an approach of funding the technology but providing limited support for business change.16

15 All prices in this part are expressed in 2016-17 terms and no adjustment has been made for the time-value of money.

16 Comptroller and Auditor General, Mobile Technology in Policing, Session 2010–2012, HC 1765, National Audit Office, January 2012.

Appendix One

Our audit approach

1 This study examined the cross-government programme, led by the Home Office, to replace the existing emergency services mobile communications system.

We assessed:

the reasons why the current contract with Airwave needs replacing;

the Home Office’s business case for its chosen system, the Emergency Services Network (ESN); and

the risks that will be encountered in delivering ESN and the implications if those risks came to fruition.

2 Our evaluative criteria focused on four areas:

Is there an agreed understanding between the Emergency Services Mobile Communications Programme (the programme), funding organisations and user organisations on the risks they will need to manage to deliver ESN and similar risk appetites?

Are appropriate commercial arrangements and best-practice processes for managing delivery in place?

Has there been good consultation over a wide range of options before the ESN option was chosen?

Is the business case for ESN based on strong evidence and reasonable assumptions?

3 Our audit approach is summarised in Figure 16. Our evidence base is described in Appendix Two.

Appendix Two

Our evidence base

1 Our independent conclusions on the programme to deliver the Emergency Services Network (ESN) were reached following our analysis of evidence collected between March and May 2016.

2 It is too early to assess whether the Emergency Services Mobile Communications Programme (the programme) has achieved value for money, given that this is a live programme at an early stage of delivery. Instead, we use an evaluative framework designed to assess a programme in its early stages. Our key questions are outlined in Appendix One. To answer them, we carried out eight methodologies:

Contractual and non-contractual performance data

Data on the number of user organisations (paragraph 1.4 and Figure 2) was produced using programme management data supplied by Airwave Solutions Limited (Airwave) and the programme. The number of emergency services in the full business case was 107, but by June 2016 the number had fallen to 105 due to mergers by emergency services.

Data on current Airwave coverage (paragraph 1.5) and availability (paragraph 1.6) was provided to us by the programme, which collects this data on a monthly basis from Airwave as part of its performance management KPIs.

The 2016-17 cost of the Airwave service (paragraph 1.9 and Figure 4) was estimated by analysing the financial model used by the programme, summarising all the costs identified for all parties. The total cost paid by the government between 2000 and June 2016 was calculated by analysing the cost figures contained in Airwave’s published financial statements.

We estimated the overall cost of ESN (paragraph 1.13 and Figure 5) using figures from the programme’s full business case, expressed in 2016-17 prices and adjusted to exclude the impact of irrecoverable VAT and income from non-emergency service users.

Under the Airwave contract, the government retained more of the inflation risk than under the ESN contracts. In effect, for Airwave, the government contracted to pay a cost plus annual indexation adjustment whereas for ESN the price is fixed irrespective of inflation during the initial term. This means that a perfectly consistent adjustment for inflation cannot be made in Figures 6, 15 and elsewhere where prices are quoted for both the Airwave and ESN contracts. Instead, we have presented all numbers as the cost to the government in 2016-17 prices.

Coverage data (Figure 8) was provided directly by Airwave and EE Ltd (EE) to the programme. The determination of the coverage percentage is based on the Home Office methodology for ESN. This specifies a probability of extending coverage to a moving vehicle, which is different to the basis EE uses for measuring its coverage for commercial reasons. See below for more information.

Data on transition activity by region (Figure 12) was taken from the full business case and its supporting financial model. We adjusted this to take into account the delay in some contract signatures to December 2015 and the date agreed with Motorola Solutions Ltd (Motorola Solutions) for the shut-down of Airwave in February 2016.

The achievement of milestones (Figure 11) was determined from programme management documents. The reduction in transition time for the programme (paragraphs 2.17 and 2.18) was identified by comparing the outline and full business cases.

Our estimate of the maximum cost to the taxpayer of a 12-month nationwide delay in the time taken to transition to ESN of £475 million (paragraph 2.18) was based on analysing the programme’s full business case. The programme has the option to extend Airwave monthly and by region.

Turnover of programme staff (Figure 13) was calculated by dividing total programme staff by those leaving each year, using human resources management information provided by the programme for the 2013-14 to 2015-16 financial years.

Our estimate of Airwave’s revenues (paragraph 4.3) was calculated using an analysis of actual and forecast billings data (excluding VAT) generated by Airwave between 2015-16 and 2016-17.

Cost per device (paragraph 4.4) was calculated by using the number of devices, actual (Airwave) and predicted (ESN), against overall annual cost of the service taken from the full business case. Data on devices was provided by Airwave and the programme.

The estimated financial savings from implementing the ESN (paragraph 4.10 and Figure 15) were calculated by comparing cost data on ESN and Airwave taken from the programme’s full business case, expressed in 2016-17 prices. The predicted cost of Airwave included the direct cost of the Airwave contract and the cost of refreshing the current Airwave devices, the cost of additional commercial smartphone contracts and the contract management function.

Reviewing the programme’s contracts and contract change notes

3 We used the National Audit Office’s (NAO’s) commercial and contracting framework to assess the ESN procurement and its commercial management to date. This framework draws on our knowledge of commercial and contracting good practice across government. We reviewed the contracts and various supporting schedules. We also examined the programme’s evaluation and scoring of bids they received.

Reviewed documents from programme and other stakeholders

4 We applied our own framework to evaluate the programme’s project management and performance. To do this, we reviewed over 100 documents, including:

Programme organisation charts.

The strategic outline business case, outline business case and the final business case and supplementary appendices.

Programme plans, progress updates and risk registers.

Minutes and supplementary information supplied to relevant management boards, including the programme board.

External programme reviews by the Major Projects Authority.

The contracts for each of the main lots, including the contract manuals and their supporting financial models.

For stakeholder engagement, the Memorandum of Understanding with other government departments, the programme’s user engagement and communications strategy and outputs of its requirements workshops.

Interviews

5 We carried out over 60 interviews with government officials, commercial representatives, programme staff and members of the emergency services. These included:

programme staff involved in the development, procurement and management of the programme – these included:

the senior responsible owner;

the programme director; and

other representatives of the programme board.

the commercial suppliers who bid for the main ESN contracts, including those who pulled out of the competition before submitting a best and final offer;

emergency services representative organisations, including the Association of Ambulance Chief Executives, the National Police Chiefs’ Council, the Police and Crime Commissioners Treasurers’ Society and the Chief Fire Officers Association;

key stakeholders, including the British Transport Police, National Crime Agency, Ofcom, the Cabinet Office, HM Treasury and Department of Health; and

business change leads and business change and assurance managers, who are emergency services representatives responsible for liaising with the programme.

Field trips

6 We visited five emergency services: the Metropolitan Police, the London and North West Ambulance Services, and the London and North West Fire and Rescue Services. We also visited control rooms for a large-scale event.

7 At each of these trips we spoke to front-line users of the system, staff involved in local roll-out of the programme and senior managers. At each of these trips we asked for views on:

the Airwave system and expectations for its replacement;

their engagement and consultation with the programme; and

challenges and success factors for future delivery.

Workshops

8 We carried out six workshops with programme officials, representatives of user organisations from the police, fire and ambulance services, and representatives of all emergency services in Scotland and Wales (see Appendix Three for more details).

Employed consultants to analyse international comparators

9 We commissioned a UK public sector ICT consultancy, Kable, to identify how other countries deliver emergency services communications, to determine whether the proposed ESN would be the most advanced solution in the world for delivering emergency services communications. This report looked at how emergency services communications has been delivered in 30 countries, with a particular focus on the US, South Korea, Australia and Germany.

This report is available on the NAO website.

Employed consultants to analyse technical and commercial challenges the programme faces

10 We commissioned telecommunications consultancy Advanced Wireless Technology Group (AWTG) to provide expert technical analysis of the challenges in delivering the ESN.

The measurement of coverage for ESN and commercial purposes

11 EE publishes measurements of geographic coverage for commercial and regulatory purposes. These measures differ to those that the ESN programme calculates as set out in the table below.

12 Programme officials consider that the 96.16% projected for September 2017 will be sufficient to meet the obligations under the ESN contract. This is slightly lower than the 97% currently provided by Airwave due predominantly to Airwave’s wider overspill into the areas alongside roads.

13 The difference between commercial and ESN coverage data is explained by:

GB vs UK

EE commercial geographic coverage is reported for the UK, not just Great Britain.

Devices:

ESN measurement assumptions are based on coverage to vehicles using high-power devices with large antennae.

EE commercial coverage is based on normal handheld devices which have lower power and smaller antennae.

Call probability

There are different call probabilities between ESN and EE commercial coverage. Call success probability is used to calculate the radio strength and therefore coverage. The lower the call success probability, the lower the coverage level required.

Data rate

At the edge of coverage areas the data speed achievable is lower than in the centre. ESN allows for a lower data rate than EE does when calculating coverage and thus a wider coverage area.

ESN-only sites

In order to ensure a reliable ESN and EE consumer service it may be necessary for some sites to carry only ESN traffic. Such sites will not be included in measures of EE commercial coverage.

Appendix Three

Our workshops with programme
and emergency services representatives

Our method

1 For our fieldwork we were keen to capture emergency services’ perceptions of the challenges in delivering the Emergency Services Network (ESN) to see how they compared with the Emergency Services Mobile Communications Programme (the programme). To do this we ran workshops with:

programme board and other invited officials from the programme;

emergency services representatives from Wales, invited by the business change lead for Wales;

emergency services representatives from Scotland, invited by the business change lead for Scotland;

police representatives, drawn from the communications Police User Group, where police users meet regularly to give feedback on their current and future communications systems;

fire and rescue service representatives, drawn from the communications Fire Customer Group, where fire and rescue service users meet regularly to give feedback on their current and future communications systems; and

ambulance representatives, by invitation from the Ambulance Radio Programme, responsible for managing the local procurements for the ESN on behalf of all ambulance trusts.

2 These workshops asked each group about:

the current Airwave service and its proposed replacement;

consultation and engagement from the programme; and

key risks and critical success factors for delivering the ESN. We asked each group to identify and rate their highest risks in seven key areas:

a political;

b governance and accountability;

c commercial and suppliers;

d technology and infrastructure;

e delivery approach and complexity;

f capacity and capability; and

g users.

3 We also asked them to identify key success criteria that would need to be met if the ESN is to be successfully delivered.

4 All suggested risks were tallied up to produce comparable risk scores for each category and group, which we summarise below.

Results of analysis

5 We found areas where programme and emergency services representatives concurred on the risks, but also some areas where perceptions of risks, or ranking of them, differed (Figure 17).

Areas where emergency services and the programme agreed on risks

The short timescale for the project. This included limited time for learning lessons and limited time for testing to the satisfaction of the services.

Interdependencies across political boundaries, where different government departments and regions do not work together effectively.

Securing extra funding as contingency against any delays.

Integrating the range of ESN suppliers and managing them.

The programme acting as service integrator, instead of hiring a prime contractor.

Recruiting technical and commercial specialists, both locally and nationally.

Making sure emergency services use and exploit the ESN. This included the need for good training and support.

Areas where perceptions of risk differed between the programme and emergency services

Emergency services were concerned about coverage, especially in devolved regions and remote areas. They were also concerned about capacity and resilience of the ESN.

The programme was concerned about the number of technical interfaces between systems.

Emergency services expressed concerns about the clarity of future funding and how financial savings would be allocated.

Emergency services were concerned that suppliers may not have the capacity to carry out the transition in time, for example because they have a limited number of specialist suppliers who can carry out upgrades to control rooms and vehicles.

Design and Production by NAO External Relations
DP Ref: 11248-001