Left column

Middlecolumn

Article

    Forecasting for ‘Saints’

  • Posted on August 17, 2016 by

    St Helena tortoiseEver wished to be a Saint – or to visit the world’s oldest land animal: Jonathan, the 184 year-old tortoise? To help the 4,100 ‘Saints’ – the residents of the UK Overseas Territory of St Helena – to become economically and socially sustainable, the UK funded the island’s first airport. But the airport’s opening has been delayed and our investigation highlights the challenges and risks to Realising the benefits of the St Helena Airport Project. This is an unusual and fascinating example of project decision-making. It also highlights common issues about managing high risk projects, potential optimism bias in forecasting, and the challenges of realising project benefits.

    Life on a remote island excites the imagination. 1,200 miles off the African coast, accessible only by a 3-weekly Royal Mail Ship from Cape Town, St Helena couldn’t be much more remote. But its economy is shrinking and its population declined between 1976 and 2008 (although the trend has since reversed), medical services are limited, and a shortage of jobs leads people to work overseas, increasing family breakdowns. To help the self-governing island, the Department for International Development (DFID) provides an annual subsidy of about £28m. DFID funded the construction of an airport both to help St Helena become economically and socially sustainable, and to reduce this subsidy.

    St Helena airportThe airport is completed, but it hasn’t been opened to commercial flights because of the challenges of landing in the island’s strong winds. Moreover, our investigation demonstrated that the intended benefits and the UK subsidy are dependent on the projected tourist numbers and expenditure used by DFID in its business case, which are highly sensitive to the assumptions made.

    The media frequently reports on expensive and often embarrassing cost over-runs in major projects. In the case of the St Helena airport, the airport was a modest 3% more than the budget, at time we reported. We also concluded that DFID’s cost–benefit model followed good practice, with many of its underlying assumptions being evidence-based, supported by a sound rationale and calculated using robust methodologies.

    It was the project’s high risk that was the issue for the St Helena airport. DFID was well aware that the justification for the airport was heavily dependent on two areas of considerable uncertainty. One was the difficulty of estimating the impact of the airport on future tourism. The second was the extent to which the island’s government and population would be able to maximise the benefits of air access.
     


    ‘SAINTLY’ LESSONS

    Predicting without precedents

    Where there’s no past performance to project forward, nor clearly comparable examples to apply, predicting the future is an inexact science. Primary research can help. But research respondents – like forecasters – can be overly positive. Facing a shortage of comparable situations, DFID commissioned primary research, but was concerned that the predicted visitor numbers were optimistic and halved them. Nevertheless, our investigation showed that the business case for the airport is particularly sensitive to two assumptions – numbers of tourists and how much they will spend.

    Tom McDonaldThe ability of the island to attract the predicted visitor numbers is, of course, going to be highly dependent on the island’s appeal to potential tourists and how well its existing and future new attractions are promoted. St Helena has a rugged beauty and the appeal of being isolated, but it has a limited number of tourist attractions. The St Helena government’s plans for developing its tourist attractions will play a key role in driving tourism growth.

    To predict visitor numbers, DFID considered the tourism growth rate after air access was improved on four other islands: Easter Island, Galapagos Islands, the Solomon Islands and Cape Verde. Its model used a tourism growth rate similar to that of Easter Island, but this has the considerable attraction of 900 monumental statues created by its early inhabitants during the 10th-16th centuries. Only Cape Verde is reasonably comparable with St Helena in terms of tourist attractions, but it still has much more convenient air access. If Cape Verde’s rate of growth is applied to DFID’s business model, the case for the airport is not positive – casting considerable doubt on its feasibility.

    Being prepared and realistic

    Another key reason we questioned visitor numbers was, as DFID’s own business case said, “the other key area of uncertainty is the extent to which St Helena will be able to maximise the benefits that could be derived from air access.” One of the essential requirements for benefiting from the increased visitor numbers will be a considerable increase in the supply (and standard) of accommodation.

    Achieving the required growth in room numbers will require the St Helena government to make changes to its land ownership policies, its investment incentives, its tax regime and its approach to planning applications. Both achieving the intended benefits from the airport and the likelihood of attracting the needed investment depends on the forecasts being realistic, which may not be the case with the assumptions that each tourist will spend £172 per day and that tourist numbers will be spread evenly over the year.

    The lack of comparators that made it hard for DFID to predict the future will make it equally hard for possible investors in St Helena to estimate their potential return on investment. And without investment, there’ll not be the tourist facilities needed.


     
    FORECASTING CHALLENGES

    The lack of precedents made forecasting tourist numbers and expenditure for St Helena particularly difficult. However, weaknesses in forecasting are common across government. Our report Forecasting in government to achieve value for money provides a detailed review of the reasons for the weaknesses in forecasting that we identified in 71 of our reports in the four years 2010–’13. Some of the key recommendations from this report on forecasting and our wider work are as follows.

    Improve decision-makers’ understanding: Starting at the end, the point of a forecast is to inform decision-makers, such as MPs and senior government officials. So it’s key that these people understand the figures they are being given, the risks attached to them, the assumptions that have been made and how this all relates to other project information – including technical and commercial factors. Equally, senior management needs to have the knowledge to challenge experts’ forecasts. The government forecasting community can play an important role in making forecasting processes more consistent and helping people at senior levels to understand their complexity.

    Plan for the worst: Scenario planning is a key aspect of forecasting. But government tends to baulk at ‘worst case scenario’ predictions and ‘high impact, low probability’ (HILP) events are often ignored. Yet, in some ways HILP events are becoming more frequent, and the impact is spreading faster, as a result of climate change, globalisation and greater interconnection between organisations and sectors. Organisations, and government as a whole, need to plan for handling worst-case scenarios and need to build in early-warning systems – and need to be willing to hear bad news. Moreover, they must be able to communicate with the public in a way that avoids panic, but enables understanding of that this is planning for the worst case, not the expected outcome.

    Improve data and modelling: Getting comparable data was particularly difficult in the case of the St Helena airport decision, but even in many more normal situations there’s a lack of accessible data to improve the accuracy of forecasts. As Big Data becomes more readily available and shared, there should be much better opportunities to improve access to comparable data. Equally, forecasts must be based on a sound model – as set out in our Framework to review models, summarised in our recent post Beautiful models: robust and reasonable.

    Be realistic: DFID’s business case for the airport expenditure was greatly strengthened when non-monetised benefits were taken into account, such as the increased employment prospects for Saints. The process of valuing non-financial benefits is very susceptible to optimism bias. As Over-optimism in government projects says, the bias can be reduced by a culture of challenge, good data and modelling, and appreciation of the complexity of the situation before committing to a solution. Organisations and their leaders need to be willing to confront inconvenient facts (which might include strong winds around an airport), able to handle conflicting priorities, and learn from mistakes, as set out in The art of spending public money wisely. Stakeholders and their motivations need to be understood and the challenge of persuading them to change their behaviour mustn’t be underestimated. And, as always, there must be independent challenge to hold decision-makers to account.

    We all base decisions on predictions of the future. Too often what ‘sounded like a good idea at the time’ ends up being taken forward despite high risks, or implementation lacks adequate planning for the worse-case scenario. With major changes taking place across government, particularly in light of the exit from the EU, it is more important than ever that government improves data, modelling, planning, monitoring, accountability and communication to the public.

    We would love to hear your comments on the challenges of forecasting accurately. Please feel free to contact us to discuss any of the issues raised in this post.
     
     
    Tom McDonald

    About the authors:
    Tom McDonald is the Director responsible for the NAO’s overseas value for money audits, including the Department for International Development. Tom has worked at the NAO since 2001 and has focused his career on the defence, overseas, health and national security sectors. He has degrees in modern languages, international relations and management from Bristol University and Ashridge Business School.
     
     
    Michael KellMichael Kell has been the NAO’s Chief Economist since 2010 and he currently also leads VfM work on energy and environmental policy. He led the NAO’s 2013 report “Forecasting in Government to achieve Value for Money”. Prior to the NAO, Michael was Chief Economist at the Department for Communities and Local Government, and has worked as an economist at Deloitte, the International Monetary Fund, HM Treasury and the Institute for Fiscal Studies.

     
     


    Tagged:             


  • Leave a Comment

    Leave a Reply

    Your email address will not be published. Required fields are marked *

Right column

  • About the NAO blog

    Our experts share their views about issues and common challenges facing government, what public sector leaders should look out for and how organisations have addressed issues. Our posts draw together threads from across our reports, share secrets spilled in events and reveal our experts’ expectations for the future.

    We encourage comments that support the exchange of ideas for improvement, but ask that those posting are respectful of others.

  • Sign up for automatic feeds

    Sign up to receive email alerts:

  • Recent posts