Report

by the Comptroller
and Auditor General

Care Quality Commission

Capacity and capability to regulate the quality and safety of health and adult social care

HC 271 Session 2015-16 22 July 2015

Our vision is to help the nation spend wisely.

Our public audit perspective helps Parliament hold government to account and improve public services.

The National Audit Office scrutinises public spending for Parliament and is independent of government. The Comptroller and Auditor General (C&AG), Sir Amyas Morse KCB, is an Officer of the House of Commons and leads the NAO, which employs some 810 people. The C&AG certifies the accounts of all government departments and many other public sector bodies. He has statutory authority to examine and report to Parliament on whether departments and the bodies they fund have used their resources efficiently, effectively, and with economy. Our studies evaluate the value for money of public spending, nationally and locally. Our recommendations and reports on good practice help government improve public services, and our work led to audited savings of £1.15 billion in 2014.

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 20 July 2015

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Sir Amyas Morse KCB
Comptroller and Auditor General
National Audit Office

15 July 2015

This report focuses on the Commission’s progress in putting its transformation strategy in place, and its capacity to implement its new approach.

© National Audit Office 2015

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.gsi.gov.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

10765 07/15 NAO

The National Audit Office study team consisted of:
Rosie Buckley, Mark Burkett, Anjeli Chadha, David Raraty and Josh Smith, under the direction of Robert White.

This report can be found on the
National Audit Office website at
www.nao.org.uk

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

Tel: 020 7798 7400

Enquiries: www.nao.org.uk/contact-us

Website: www.nao.org.uk

Twitter: @NAOorguk

Key facts

Summary

1 The Care Quality Commission (the Commission) is the independent regulator of health and adult social care in England. Its purpose, as set out in its published strategy, is to “make sure health and social care services provide people with safe, effective, compassionate, high quality care, and to encourage them to improve”. The Commission is a non-departmental public body, sponsored by the Department of Health (the Department).

2 We concluded in 2011 that the Commission had not provided value for money. In March 2012, the Committee of Public Accounts reported that the Commission was a long way off becoming an effective regulator and that, despite evidence of failings, the Department had been slow to act. The Commission has since been working with the Department to implement significant changes, under a three-year transformation programme between 2013-14 and 2015-16.

3 This report focuses on the Commission’s progress in putting its transformation strategy in place, and its capacity to implement its new approach. Because it is too early to conclude on what impact the Commission is achieving we will report in due course on how well the Commission’s regulatory model works in practice.

4 We explain our audit approach in Appendix One and our evidence base in Appendices Two and Three. Appendix Four summarises the Commission’s progress against recommendations that we and the Committee of Public Accounts made.

Key findings

Response to criticisms and new challenges

5 The Commission has made substantial progress since public concerns were first raised in 2011, and is in the process of embedding changes to its regulatory approach. Between late 2011 and early 2013, a series of internal and external reviews recommended changes to the Commission’s capability and regulatory approach. In April 2013, the Commission published a transformation strategy for 2013-14 to 2015-16 proposing radical changes to how it regulates health and social care (paragraphs 1.4 to 1.8).

6 The Commission’s new regulatory model strengthens the way it expects to monitor and inspect hospitals, adult care providers and GPs. Most providers (91%) the Commission surveyed in 2014 said they were aware of its new approach. The survey shows that providers remain extremely cautious, however, about whether inspectors are equipped to apply the new approach in practice (paragraphs 1.9 to 1.10 and 1.17).

7 The Department has placed additional expectations on the Commission that increase risk to achieving its transformation strategy. The Commission has not yet finished implementing all of the changes in its strategy. However, the Department gave it new responsibilities, from April 2015, to oversee the financial sustainability of the largest adult social care providers. It announced in June 2015 that the Commission would also assess the financial efficiency of hospital trusts. As a result, the Commission has needed to recruit new skills for its market oversight role. It will now need to develop the expertise needed for its responsibilities to assess financial efficiency, and is still evaluating the potential impact of this work on its resource model (paragraphs 1.2 and 2.19).

Staff skills and capacity

8 The Commission does not know how accurate its staffing model is because it relies on assumptions that are still being tested on the ground. The Commission’s new regulatory model was introduced for hospitals in July 2013 and adult social care providers and GP services in October 2014. The Commission does not yet know with certainty how many providers will need to be re-inspected, and the consequent workload, because so far 91% of providers have not been inspected and rated under the new model (paragraphs 2.1 and 2.2).

9 The Commission has made progress recruiting new staff, but does not yet have enough people to do all its work. By mid April 2015, it had reached its initial target to recruit 300 inspectors by the end of April. It now plans to make job offers to 300 more inspectors by December 2015. Because of staff shortages, the Commission deferred target dates for inspecting providers. But it also has staff shortages in other parts of its business, particularly among analysts. The Commission decided that it could rely to a greater extent on analysis of mortality risk indicators undertaken by the Dr Foster Unit at Imperial College, and it has reduced the frequency with which it generates new alerts from scanning maternity outliers (paragraphs 2.4 to 2.8, and 3.19).

10 In the 2014 staff survey, 40% of staff agreed or strongly agreed they had the training and development they needed to do their job, and 38% said that the training they received was effective. The Commission established a learning and development Academy to provide role-specific training, which went live in March 2014. The Academy started providing e-learning and face to face training on new enforcement powers from January 2015. New inspectors have one week’s corporate induction and six weeks’ role-specific training before joining their teams (paragraphs 2.10, 2.13 and 2.14).

11 The Department expects the Commission to make fuller use of its authority. In the past, inspection teams did not know enough about the Commission’s enforcement powers to take effective action. The Commission has now made enforcement part of inspectors’ mandatory training. It took on new powers in April 2015 that make it easier to bring prosecutions where it finds that poor care is harming service users (paragraphs 2.12 and 2.13).

12 The Commission started overseeing the financial health of adult social care providers before having in-house expertise fully in place. From 6 April 2015, the Commission must notify relevant local authorities if it considers any of the 43 largest adult social care providers is at risk of exiting the market. The Commission’s board recognised that building new skills and capability represented a substantial risk, and the Commission is drawing on external consultancy support. When it took on its new responsibilities, senior members of the team were not in post. Two of the three people for these roles joined the Commission in May. In the interim, the Department is sharing responsibility by overseeing the largest providers. The Commission is still recruiting for the third senior post (paragraphs 2.17 to 2.21).

Knowledge and information

13 The Commission rejects many paper applications to register providers because they contain errors. In 2014-15, the Commission processed 81,840 applications to add or change registrations. Of these, 39,061 (48%) were returned, not required or withdrawn, representing wasted effort and cost for both the Commission and applicants. Over 1,700 applicants used the wrong form. After successfully registering GPs online in 2013, the Commission plans to introduce an online system for adult social care providers during 2015-16 (paragraph 3.11).

14 The Commission is using data more effectively to plan inspections, particularly for acute trusts. Sir Robert Francis’ second inquiry into failings at Mid Staffordshire NHS Foundation Trust concluded it was essential the Commission improve the way it uses information to monitor risk. The Commission has long used routinely available information to assess risk. It now makes a clearer distinction between indicators of risk (tier 1) and indicators that support inspection planning (tier 2). For acute trusts, the Commission reduced the amount of information it analyses from around 1,400 items to 115 tier 1 indicators (paragraphs 3.1 to 3.3).

15 In contrast to the national datasets available for hospitals and GP services, the Commission does not have access to routine information about adult social care good enough to monitor risk or trigger inspections. Some 13,000 adult social care providers operate services in more than 25,000 locations. However, because there are no national datasets comparable to those available for hospitals and GPs the Commission relies heavily on manual forms to collect information before inspections. It is developing an online system for providers to keep updated that it expects to implement in October 2015 (paragraphs 3.12 to 3.13 and 3.17).

16 There is a risk the public will believe a newly registered provider is complying fully with the Commission’s standards when they are not. When it receives a new application the Commission assesses systems and processes, inspects premises, and interviews applicants to judge whether they have the capacity and capability to provide a well led service which is likely to comply with regulations. Registration of a new provider cannot give the same level of assurance as inspection, because it is done before people actually use a new service. There can be more assurance when the Commission registers a change of provider. Even in these cases, however, there may be a new management team responsible for the service, or a provider may be registering to offer new services. The Commission set itself a performance indicator for the proportion of new providers needing regulatory action on first inspection, but has not published a target for what it believes is reasonable. In 2014-15, one out of three newly registered providers needed regulatory action after their first inspection (paragraph 3.10).

Accountability, leadership and governance

17 The Department and the Commission have taken appropriate, and very substantial, action in response to criticism of the Commission’s governance and leadership. The Commission’s executive team is completely different to when we reported in 2011. The Department appointed a new chair of the Commission in December 2012 and expanded the non-executive board. The Commission’s governance structures and processes are now consistent with best practice in many areas. It is not doing the amount of board development work, including periodically evaluating the board’s effectiveness, that would match best practice or the Department’s expectations (paragraphs 4.2 to 4.7).

18 The Commission published in its 2015-16 business plan a comprehensive and logically structured performance framework. This included measures of timeliness, quality and patient feedback. The Commission set a specific target for 6 of the 37 measures in the business plan. We also found that for 6 of the 37 measures there is no baseline data because the model is different to before. Until it sets specific targets or benchmarks, the Commission risks the public expecting it to be more a guarantor of quality and safety than is realistic (paragraphs 4.12 to 4.14).

19 From 2015-16, the Commission will be able to make a reasonable estimate for the full cost of its regulatory activities. The Commission has adopted a ‘top-down’ approach based on budget data to apportion costs to the different parts of its operating model based on assumptions about predicted headcount in each function. It validates the costing model annually with a retrospective ‘bottom-up’ exercise. Updating the underlying assumptions is important because the Commission’s approach apportions approximately half of its budget. The Commission’s ability to measure its own costs, and demonstrate its cost-effectiveness, will be increasingly important as it increases the proportion of its costs recovered through fees (paragraphs 4.18 to 4.19).

20 Work is still needed to manage public expectations about what the Commission can and cannot achieve. The Commission’s public awareness survey in 2014 found that just over half of respondents (55%) had heard of the Commission. This compares with 93% that had heard of Ofsted and 4% that had heard of Monitor. Its national customer service centre handles a high number of enquiries and concerns from the public which informs its intelligence about providers. However, it does not have the power to resolve individual cases. It has been improving links with ombudsmen to help direct concerns to the most relevant body, and is exploring ways to make better use of this information to assess risk (paragraphs 1.13 to 1.16 and 3.6 to 3.7).

Conclusion on value for money

21 Over the last two years and in the face of sustained criticism, the Commission has made substantial progress to change its regulatory model. It is developing a more intelligence-driven approach to regulation, relying more on data to target intervention. The Commission has designed a coherent model that sets out, in principle, connections between resources, activities, outputs and outcomes. From 2015-16, the Commission is better able to estimate how much inspections and other regulatory activities cost. So far, however, it has much more limited information for assessing efficiency or effectiveness, or measuring its overall impact on the quality of care.

22 Further challenges lie ahead to demonstrate value for money. The Commission has made progress but has a substantial challenge to recruit and train all the staff it still needs. The Commission predicts that, when at full complement, a third of staff will have been in post for less than 12 months, and existing staff have experienced significant changes. The Commission needs to build an organisational culture that gives its people the confidence, as well as the skills, to apply the regulatory model assertively, fairly and consistently. It also needs more complete data about regulated bodies, particularly in the adult social care sector, and better quantified indicators of its own performance. Managing public expectations about how far and fast it can achieve this, at the same time as it takes on new responsibilities, is a substantial demand.

Recommendations

a The Commission should reinforce and develop formal and informal mechanisms for sharing knowledge between inspectors across its three directorates. By requiring inspectors to specialise in acute care, adult social care or primary care the Commission has addressed past criticism that inspectors lacked the sector-specific skills they needed. If taken too far, however, there is a new risk that staff in the three directorates may work in too much isolation. This runs contrary to developments in other parts of the health service for more integrated care.

b The Commission should review how useful its intelligent monitoring information is once it has completed the first cycle of inspections. The Commission’s ambition is that intelligent monitoring will help it identify risk and increase its efficiency in carrying out inspections. So far, there is limited evidence that it is having this impact, partly because of limitations in the data.

c The Commission should make better use of information from service users as part of its intelligent monitoring data. It has explored partnerships with organisations such as Age UK. It also manually codes around 6,000 comments from websites such as NHS Choices each month. But so far this has not increased the amount of intelligence it can act on. As it develops online systems for real-time monitoring, particularly for adult social care, it should explore the scope to integrate more feedback from users.

d The Department of Health and the Commission should agree quantified performance measures. These should include targets for the Commission’s efficiency. For measures of the Commission’s impact on the quality and safety of services, it should use 2015-16 data to set a baseline for 2016-17, against which future changes in performance can be tracked. Few of the Commission’s published performance indicators currently have a quantified baseline or target. This makes it difficult for the Department to hold the Commission to account, and for service users to assess whether the Commission is meeting the standards they should expect. As well as strengthening public accountability, this would help address a risk that the public expect the Commission to achieve more than it is able to do.

e The Department should not add to the Commission’s responsibilities and workload without assessing the impact on its existing capability. The Commission is still in the third year of its change programme, and it is building staff numbers and skills for its existing functions. In April 2015, it took on additional responsibilities, demanding new expertise, for market oversight of adult social care providers. The Department has now asked it to build additional capability for assessing the efficiency of hospitals. There is a risk that the demands of quickly meeting successive new responsibilities will undermine progress the Commission is making to strengthen its ability to regulate care quality.

f The Commission should evaluate its board’s effectiveness each year. The Commission reviewed its committee structures in 2015, but had not carried out other reviews for the previous three years. It is good practice for organisations to evaluate their board’s effectiveness at least annually.

g Operational changes need to be supported by changes to organisational culture. Staff survey results, so far available up to 2014, show that morale and confidence in the Commission’s leadership are improving, but that there needed to be more of an embedded learning culture. It will need to test the impact of more recent initiatives particularly by analysing free-form comments in the 2015 staff survey.

Part One

Transforming the Care Quality Commission

1.1 The Care Quality Commission (the Commission) is the independent regulator of health and adult social care in England. In its published strategy and business plan it describes its purpose as to “make sure health and social care services provide people with safe, effective, compassionate, high quality care, and to encourage them to improve”.1 The Commission:

registers adult social care providers and healthcare providers, including hospitals and GPs, to provide services in 49,632 currently registered locations;

monitors, inspects and regulates services to make sure they meet fundamental standards of quality and safety;

has powers to take enforcement action where providers fail to meet those standards;

publishes findings, including performance ratings to help people choose care; and

publishes thematic reviews to help promote good practice among regulated bodies.

1.2 In April 2015, the Commission became responsible for overseeing the financial health of difficult-to-replace providers of adult social care. In April 2016 it will also start evaluating hospitals’ financial efficiency.

1.3 The Commission is a non-departmental public body, sponsored by the Department of Health (the Department). Its 2015-16 budget is £249 million, funded by grant-in-aid from the Department and fees charged to regulated bodies.2

The need for change

1.4 The Commission began operating on 1 April 2009, when it took over functions of the Healthcare Commission, Commission for Social Care Inspection and Mental Health Act Commission. In 2011 and 2012, however, the Commission faced repeated and sustained criticism of its regulatory approach, capacity and capability. Key criticisms were that the Commission:

lacked strategic direction;

had been poorly governed and led;

did not have the skills or capacity to make sound and consistent regulatory judgements;

had been slow to identify problems in providers and acknowledge the seriousness of some issues;

did not listen to whistleblowers or understand the patient’s perspective; and

had not intervened quickly or strongly enough in failing providers, or followed up to ensure issues were addressed.

1.5 High profile failures of care at Mid Staffordshire NHS Foundation Trust predate the Commission. In his second inquiry report, however, Sir Robert Francis examined the oversight, scrutiny and regulation of the NHS.3 He suggested, in February 2013, the Commission’s regulatory model needed to:

include specialist inspectors;

distinguish between standards of care that are essential and standards that are desirable; and

draw on a wider range of information to assess risk, including users’ experiences.

1.6 The Commission published its strategy for 2013 to 2016 in April 2013.4 It proposed radical changes to the way it regulates health and social care. It committed to using information and inspections in a more focused way. It also promised to use more effectively the views of people who use services, and collaborate more with its partners in the health and social care system.

1.7 This report looks at what the Commission has done to redesign how it regulates health and social care providers, and to build the staff skills and capacity it needs. We also look at the Commission’s accountability, leadership and governance. Because the Commission’s regulatory model is new, it is too early to conclude whether it has had a beneficial impact on the quality and safety of care. We propose to examine this issue in a later report.

Changes to the Commission’s regulatory model

1.8 Since April 2013, the Commission has been implementing substantial changes to the way it regulates healthcare providers, in the wake of severe criticism in 2011 and a series of subsequent reviews between December 2011 and March 2013 (Figure 1). The Commission drew on external expertise including advice on using its regulatory powers, how its previous model compared with other regulators and what use it could make of monitoring data. It consulted widely with providers and other stakeholders on its proposals, through formal consultation and other listening events. It also took on board recommendations from Professor Sir Bruce Keogh’s 2013 review into 14 hospitals with high mortality rates.5 From September 2014, the Commission started publishing provider handbooks for each different sector, with details of how it would regulate services.

1.9 Figure 2 summarises the Commission’s regulatory model. The model is built around 5 key questions to test whether care providers are meeting fundamental standards of care. The Commission tests providers at three stages: when it registers them; as it reviews performance data; and when it carries out inspections. After inspections, the Commission publishes its assessments and rates providers on a four-point scale.

1.10 In seeking to address previous criticism, the Commission’s new regulatory model is very different to its previous approach (Figure 3). The biggest changes have been to the way the Commission approaches its monitoring and inspection work. Providers responding to the Commission’s provider survey in November 2014 were cautious, however, about whether inspectors would be equipped to apply the new regulatory framework consistently, accurately and fairly in practice.

1.11 The Commission collected feedback on its early inspections of providers. It commissioned an independent evaluation of the new model for the acute and mental health sectors. The Commission also carried out an internal review of the adult social care and primary medical services sectors. It commissioned independent interviews with providers from the adult social care, primary care, independent healthcare and dental practice sectors. The Commission is considering how it will respond to the findings from these reviews.

Changes to the Commission’s organisational structure

1.12 The Commission, supported by the Department, appointed a new chief executive and completely changed its executive leadership. The leadership team includes, for the first time, three chief inspectors with responsibility for hospitals, adult social care and primary medical services. The Commission also changed its organisational structure, from one front-line directorate to three front-line directorates specialising in: acute care; adult social care; and primary care.

Communicating its new approach

1.13 When we last reported we found there was a gap between what the public and providers expect of the Commission and what it can achieve as a regulator. The Committee of Public Accounts concluded that information the Commission provided to the public on the quality of care was inadequate, and that it did not give a picture of the state of care.

1.14 The Commission now publishes a greater range of information about its strategy and performance. It has published its strategy, business plans and provider handbooks, and put videos and minutes of its public board meetings on its website. It also publishes high-level data on the number and types of enforcement actions it is taking, and the length of time it takes for providers to rectify a breach of the regulations. It has also used less formal methods, such as social media campaigns and web forums, and executive leaders have spoken at public events.

1.15 Staff responding to the Commission’s staff survey in July 2014 nonetheless believed there was still poor public awareness and understanding of the role and powers of the Commission. Others said the Commission needs to be more assertive in intervening against poor providers and defending itself against public criticism.

1.16 The Commission’s public awareness survey in 2014 found that just over half of respondents (55%) said they had heard of the Commission. This compares with 93% who had heard of Ofsted, the regulator for providers of education and skills and care services for children and young people, and 4% who had heard of Monitor, the regulator of NHS foundation trusts. When asked what the Commission ‘does’, people who had heard of the Commission most commonly said it had a role in monitoring and inspecting services (64%). Fewer mentioned its role as a regulator (19%) or that it issued standards, guidance and best practice (3%). Some 3% thought it handled complaints, although it does not have the powers to resolve complaints about providers. The Commission published a new strategy for engaging with the public in January 2015, which included commitments to carry out local public engagement and encourage greater use of social media. However, because the strategy is so recent, there is not yet good evidence of its impact.

1.17 The Commission’s provider survey from 2014 found that most providers surveyed (91%) were aware that the Commission had changed its inspection methodology. The number with a very good or fairly good understanding of the new approach to regulation and inspection varied (88% for adult social care, 83% for hospitals and 57% for primary medical services). Providers had mixed views about the value of information the Commission sent them. Some 6% of surveyed providers said they received regular updates from the Commission and they were clear about what to expect from the new approach. Some 8% of providers responding to the same question said they did not have time to read the Commission’s documents and guidance, which were difficult to understand.

1 Care Quality Commission, Business Plan April 2015 to March 2016, 27 March 2015. Available at: www.cqc.org.uk/content/business-plan-shaping-future

2 The budget shown includes £16 million to cover the cost of staff to deliver the new approach to regulation. This will only be used if required. It excludes depreciation and capital expenditure.

3 Sir Robert Francis QC, Final report of the Mid Staffordshire NHS foundation trust public inquiry, 6 February 2013. Available at: www.midstaffspublicinquiry.com/report

4 Care Quality Commission, Raising standards, putting people first – Our strategy for 2013 to 2016, April 2013. Available at: www.cqc.org.uk/public/about-us/our-performance-and-plans/our-strategy-and-business-plan

5 Professor Sir Bruce Keogh KBE, Review into the quality of care and treatment provided by 14 hospital trusts in England; overview report, 16 July 2013. Available at: www.nhs.uk/NHSEngland/bruce-keogh-review/Pages/published-reports.aspx

Part Two

Staff skills and capacity

Understanding staffing needs

2.1 At 31 March 2015, the Care Quality Commission (the Commission) employed 2,681 people. The Commission has developed a resourcing model to estimate the number of staff it needs to meet all of its commitments. The model includes assumptions, for example, for the duration of inspections, the skill mix needed and the proportion of staff time used for activities such as training. Because inspection results determine how quickly providers will be re-inspected, the model also makes assumptions about the level of risk in providers to estimate the volume of re-inspection work.

2.2 The Commission does not yet know whether the model accurately predicts the number of staff it needs in practice. This is because:

The Commission does not yet know how many providers it will need to re-inspect and the impact on workload because, by the end of March 2015, 91% of organisations had not been inspected and rated under the new regulatory model. The Commission introduced the new regulatory model for hospitals in July 2013, but did not roll it out for adult social care and primary medical services until October 2014.

Assumptions about the time taken to complete inspections are not as robust as they could be because 75% of inspectors complete their timesheets and there are concerns about the quality of data recorded.

Building staff skills and capacity

Recruiting staff

2.3 In March 2012, the Committee of Public Accounts found the Commission had consistently failed to spend its budget because of delays in filling staff vacancies. We reported that staff vacancies caused compliance activity to fall significantly during 2009-10 and 2010-11.

2.4 In 2014, the Commission found it would need to increase its workforce significantly to implement its new regulatory model, in particular because inspections are more resource intensive than before and require greater analytic capability. The Commission set a target of making 300 employment offers to inspectors by the end of April 2015, and a further 300 offers by the end of December 2015. These targets were an estimate of the number of offers the Commission thought it could make over a short period, rather than a calculation from the resourcing model of the number of inspectors needed to meet the Commission’s business plan commitments.

2.5 By mid April, the Commission had reached its first target to make 300 employment offers for inspector roles. The Commission’s risk register notes that filling remaining staff vacancies remains a key issue. At 10 April 2015, the Commission’s vacancy rate was 34% for inspectors, 36% for senior analysts and 35% for managers (Figure 4).

2.6 The Commission will also need to manage the risk that more staff leave the organisation than forecast. The Commission’s staff turnover in 2014-15 was 7.5%, which is above its target of less than 5.0%. Some 81 inspectors left the Commission in 2014-15, equivalent to a turnover for inspectors of 7.7%.

2.7 The Commission has used temporary staff, including staff on secondment from health and adult social care providers, to fill staffing gaps. It employed an average of 266 temporary staff in the year ending 31 March 2015, at a total cost of £17.2 million. The Commission has since reduced its reliance on temporary staff, in part by offering fixed term contracts to those meeting its selection criteria. It currently has 36 temporary staff, down from 207 that it had on 31 March 2015.

2.8 The Commission deferred its target dates for inspecting providers because it had underestimated how long it would take to train staff and carry out inspections under its new approach, and it did not have enough staff. New inspectors join inspection teams after seven weeks, although it takes longer for them to build experience and work at full capacity. In 2014, the Commission published its 2014-15 to 2015-16 business plan. It set out to inspect and publish ratings for all acute NHS hospitals by 31 December 2015, all adult social care providers by 29 February 2016, and all GP providers by 29 February 2016. In July 2014, the Commission’s board agreed to revise the inspection timetable for 2014-15 to include fewer inspections than originally scheduled. In March 2015, the Commission published its business plan for 2015-16, setting out later target dates for inspecting all providers. It is now committed to inspecting and publishing ratings for all acute hospital providers by April 2016, all adult social care providers by 1 October 2016, and all GPs by 1 October 2016.

Staff training

2.9 The Commission has faced a substantial challenge to train new staff and increase the skill base among existing staff. The Committee of Public Accounts previously concluded that individual inspectors did not have enough support to develop the expertise and experience they needed. It also found there was a lack of consistency in inspectors’ judgements and in the Commission’s approach to taking enforcement action.

2.10 Responses to the Commission’s 2014 staff survey showed at that point it still had some way to go to ensure staff were sufficiently trained and were confident about the new approach. Less than half of staff (40%) agreed or strongly agreed they had the training and development they needed to do their job and 38% said the training and development they received was effective (Figure 5). Staff also said there was a lack of training opportunities including support for professional qualifications, and that training needed to be more specific to their role. Some staff raised concerns about how training was designed, including a reliance on e-learning, and said that their workload meant they did not have enough time for training when they needed it.

2.11 The Commission most recently surveyed providers in November 2014. When asked how they rated inspectors’ understanding of the type of care they or their organisation provides, 69% of providers rated it as very good or good, 19% rated it as satisfactory, and 9% rated it as poor or very poor. In response to the same question asked in 2013, 73% of providers rated it very good or good, 16% satisfactory, and 10% poor or very poor. In additional comments, 10% of providers said the level of inspectors’ understanding varies both within and between inspection teams, inspections give insufficient regard to patients’ needs, or focus on judging compliance rather than service outcomes. This suggests that providers have yet to be convinced that the Commission’s inspection model will, in practice, lead to a greater focus on patient experience.

2.12 The Department told us the Commission had made little use of the enforcement powers available to it under the Health and Social Care Act 2008. It expected the Commission to make fuller use of those powers in future. The Commission also recognises that, in the past, inspection teams did not know enough about its enforcement powers and that this may have delayed action, or meant that some necessary action was not taken. In responses to the July 2014 staff survey, some staff said they were completing tasks, including enforcement, without the training and support they believed they needed.

2.13 The Commission’s enforcement powers changed in April 2015, when new fundamental standards were introduced. These were designed to make it easier for the Commission to prosecute providers that are failing to meet the required standards of care by giving an explicit description of the minimum standards that all providers must meet. Following the Care Act 2014, the Commission no longer needs to issue a warning notice before bringing a prosecution. A further consequence, however, is that staff must be sufficiently confident and capable of taking necessary enforcement action, sometimes over a short timescale. The Commission started providing e-learning and face to face training on its new enforcement powers from January 2015. In February 2015, it published an enforcement decision tree to help staff choose the regulatory action to take in different circumstances.

2.14 More fundamentally, successfully embedding the Commission’s new approach to regulation will depend on cultural changes. Over 20% of staff responding to questions about learning and development in the January 2014 staff ‘pulse’ survey said the Commission needed a more embedded culture of continuous learning. The Commission set up a new Academy, which went live in March 2014, to take responsibility for staff learning and development including role-specific training on the Commission’s new approach to inspection. It also provides new inspectors and inspection managers a week’s corporate induction and six weeks’ role-specific training. Because the Academy is so new, there is not yet good evidence of its impact. The Commission intends to repeat its staff survey in August 2015. This will show how staff members’ confidence in their skills and training has changed.

Staff morale

2.15 In our 2011 report we found that staff morale was low. The Commission’s staff surveys over the past three years show that morale has improved steadily since then and on several measures scores exceed civil service benchmarks. Despite this, there is still clear scope for further progress (Figure 6).

2.16 When asked to choose what would most improve morale, 53% of staff responding to the 2014 survey said: having the right staffing resources in their team to deliver their work (Figure 7).

New responsibilities for financial oversight

2.17 On 6 April 2015, the Commission became responsible for overseeing the financial health of adult social care providers that are so large, or provide such a high proportion of services, that the local authorities would find it difficult to replace them if they failed. The Commission must assess whether providers are financially sustainable, and inform local authorities if it believes the provider is likely to become unable to continue providing one or more service because of business failure. The scheme is designed to give local authorities early warning so they can fulfil their statutory duty to ensure continuity of care. The Care and Support (Market Oversight Criteria) Regulations 2015 set out the conditions for joining the scheme.6 These follow the Care Act 2014 requirement that the conditions must have regard to:

the amount of social care provided by a registered care provider;

the geographical concentration of a registered care provider’s business; and

the extent to which a registered care provider specialises in providing a particular type of care.

2.18 The Commission identified 43 providers, operating 3,533 locations, that met these conditions. In March 2015, it published guidance explaining how the scheme would work and how it would monitor providers’ financial position and service quality. The Commission has not set explicit thresholds for when it would investigate further or notify the relevant authority of a likely business failure.

2.19 The Commission identified in autumn 2014 that, to carry out its new responsibilities, it would need senior financial capability (for example in financial restructuring and insolvency) as well as analytic capability (for example, to make judgements on the financial sustainability of providers), both of which it lacked. It therefore started a campaign to recruit three senior posts (a director of corporate providers and market oversight and two heads of market oversight) as well as two market oversight managers.

2.20 None of the senior people recruited to these new roles had taken up post when the Commission took on its new responsibilities in April 2015. By May 2015, the Commission had appointed a director of corporate providers and market oversight and a head of market oversight, both of whom took up post in May. It was still recruiting an extra head of market oversight and two market oversight managers. The Commission’s deputy chief inspector of adult social care oversaw the set up of the scheme while the director of corporate providers and market oversight was being recruited.

2.21 For a transitional period between April and October 2015, the Department is sharing responsibility by overseeing the five largest providers of adult social care services. As the Commission did not have an in-house team, it appointed consultants to undertake financial sustainability assessments over the same six-month period.

2.22 In June 2015, the Department announced that the Commission would also start assessing hospitals’ financial efficiency. The Commission will start doing this from April 2016, when it starts its second wave of hospital inspections under the new regulatory framework.

6 The specific conditions for joining the scheme are set out in the Care and Support (Market Oversight Criteria Regulations 2015). Available at: www.legislation.gov.uk/uksi/2015/314/contents/made

Part Three

Knowledge and information

What the Commission has achieved

3.1 The inquiry led by Sir Robert Francis into failings to provide high quality care at Mid Staffordshire NHS Foundation Trust found that it was essential for the Commission to improve risk related monitoring. The inquiry report recommended that results of patient feedback, including qualitative information, should be available to all stakeholders in as near ‘real time’ as possible. The Commission has reorganised the way it uses information to support its new approach to inspection by developing a new system of ‘intelligent monitoring’.

3.2 Intelligent monitoring builds on the ‘quality and risk profiles’ the Commission previously used. Both use statistical analysis of quantitative and qualitative data to compare performance with what was expected. The main difference is that intelligent monitoring distinguishes between indicators of risk (tier 1) and indicators that support inspection planning (tier 2). These changes have had the biggest impact on regulation of acute hospitals. For acute providers the number of indicators used to identify risk decreased from approximately 1,400 measures to around 115 tier 1 indicators (Figure 8).

3.3 The Commission now gives inspectors tailored information packs to help them plan their work on site. The new packs make it easier for inspectors to focus on areas of greatest potential concern by collating evidence that was previously held in different locations and flagging any result outside what is expected. Each indicator is aligned to one of the Commission’s 5 key questions. Hospital inspectors tend to receive more support in using the evidence than those inspecting other providers. Inspection teams for acute providers have support from analysts as needed. The information packs to support GP and adult social care inspections are shorter and more standardised.

3.4 The Commission involved staff in developing its approach to intelligent monitoring, and tested it when it piloted inspections under the new framework. The Commission also set up an expert reference group, including specialists in their fields, partner organisations and think tanks, to provide critical challenge. Because there were high vacancy levels on the intelligence directorate, the Commission made extensive use of consultants to deliver the new approach. Intelligent monitoring reports were first published for acute hospitals in October 2013. These were followed, in November 2014, by reports for GP services, mental health trusts and draft reports for adult social care providers.

3.5 In November 2014, the Commission set out its strategy for intelligence based regulation. The strategy, which runs to 2016-17, sets out to:

better support staff using the Commission’s knowledge and information;

strengthen inspection briefing materials so they give inspectors more focused, targeted analysis and insight;

deliver a sustainable approach to analysing and reporting non-numerical information (including user feedback), drawn from multiple sources in real time;

invest in the right systems, software and technology to underpin this ambition; and

give staff the training and support they need to gain the most benefit from these changes.

Further steps

Responding to concerns raised by the public

3.6 The Commission is developing a new process for handling evidence provided by the public and whistleblowers in response to concerns about its ability to identify providers delivering poor care. The Commission’s National Customer Service Centre receives and classifies contacts before passing them on for follow up by inspectors. The amount of this evidence has grown in recent years (Figure 9). The Commission feeds this information into its intelligent monitoring to generate approximately 2,500 new tier 1 flags (indicating potential risk) each month in adult social care.

3.7 Work by the Commission suggests that the figures it uses to track action taken in response to communication from the public are still not robust. It is taking steps to improve information by auditing backlogs and improving training for inspectors. The latest figures suggest that one out of three safeguarding alerts is not acted on within the Commission’s two-day target.7

3.8 Each month the Commission manually codes around 6,000 comments from the public on websites like NHS Choices, Patient Opinion and its own Share Your Experience. It uses negative comments to identify risk in intelligent monitoring reports. Even at these volumes there are many providers with few or no comments, particularly for adult social care and GPs.

3.9 The Commission has piloted collecting feedback from the public through partnerships with other organisations, including Age UK and The Silver Line, although this has as yet to materially increase the volume of actionable intelligence. Increasing the Commission’s capacity to use this information will need either more resources to manually code new sources or new systems to automate the process.

Registering providers

3.10 Registering new providers is the first stage in the Commission’s regulatory model. All applicants must set out how they intend to meet the fundamental standards of care. A dedicated team of registration inspectors now deals with all applications. However, the Commission has limited information about new services before they start operating. The Commission’s business plan for 2015-16 sets one key performance indicator as the percentage of newly registered providers where regulatory action is taken on first inspection. Information for 2014-15 suggests that one out of three newly registered locations faced action after their first inspection (Figure 10).

3.11 Registration is also the first opportunity to collect the information upon which subsequent monitoring of those providers depends. This is particularly important for adult social care providers where information collected at registration is one of the few sources of consistent information. In 2014-15, the Commission processed 51,814 paper applications to add or vary, and 30,026 applications to remove or cancel, registrations for care managers or providers.8 Of the 81,840 total number of applications, 39,061 (48%) were returned, not required or withdrawn, representing wasted effort and cost for both the Commission and applicants. The Commission rejected over 1,700 applications because they used the wrong form. The Commission plans to provide an online option for adult social care registrations during in 2015-16. This follows the positive experience of registration for GPs in 2013 which was carried out online, with big improvements in error rates and processing times.

A comprehensive dataset for adult social care

3.12 With more than 25,000 locations, adult social care providers are the largest sector regulated by the Commission. The Commission’s ability to monitor risk is constrained because there are no national datasets comparable to those available for hospitals and GPs. As a result, only a few indicators exist to monitor some of the fundamental standards. Many of these come from sources that are not available for all providers. For example, we found that the pre-inspection information pack for a nursing home for older people contained data for only 11 of the 64 indicators potentially available to support the inspection.

3.13 The Commission uses a manual form, the ‘provider information return’, to collect data from adult social care providers before a scheduled inspection. The extent to which inspectors can use this information to plan their work depends on the quality and quantity of data collected. The Commission plans to move to an online system where providers can update the form throughout the year. This change is supported by providers. However, it depends on setting up online processing for registration. Until these changes are made data will not be available to help the Commission routinely monitor risk in all adult social care inspections.

More efficient processing of intelligence

3.14 Efficient data processing is central to the Commission’s ambition for a comprehensive system of surveillance. Intelligent monitoring brings together information from many different sources, updated at different intervals, which needs to be reported for thousands of providers. Any new system will only be sustainable if it makes best use of technology to create and access the data analysts need. The Commission has developed an intelligence hub to provide a single point of access to data. This will make it easier to integrate data from different sources. The Commission aims to improve the systems it uses to generate risk information and to pre-populate parts of pre-inspection data packs, freeing up analysts’ time for analysis and data interpretation.

Moving to an intelligence-driven regulatory model

3.15 The Commission is working towards a surveillance model that responds to concerns, triggers action and targets resources where risks are greatest. Evidence from acute hospitals indicates progress towards this goal. However, intelligent monitoring is not sufficient to predict inspection findings reliably.

3.16 Emerging evidence from acute hospitals suggests those with higher risk predicted by intelligent monitoring are more likely to get a worse inspection rating. However, these are early findings that may be misleading because the Commission prioritised higher risk trusts for early inspection. Of the providers that received an inspection rating of ‘inadequate’ or ‘requires improvement’, 60% had a higher risk rating from intelligent monitoring. The risk ratings are a less accurate predictor of good performance. Half the providers receiving an inspection rating of ‘good’ or ‘outstanding’ had a higher risk rating from intelligent monitoring (Figure 11).

3.17 There is limited scope for intelligent monitoring to prioritise scheduling for GP and adult social care inspections. The current information base for adult social care is not of sufficient quantity, quality or consistency to calculate overall risk ratings. Although the first intelligent monitoring reports for GPs did include risk ratings, GP stakeholders complained that the ratings could be interpreted as an incorrect judgement on quality of care. The Commission’s research also shows that the public found the risk ratings bandings less useful than inspection reports. Risk ratings were withdrawn in April 2015.The challenge for the Commission now is to develop a way of reporting that has the confidence of the sector and which the public understands.

3.18 The Commission acknowledges that the new information packs are not yet being used to direct the work inspectors do on site as well as intended. There may be scope to increase their impact but this will depend on research to show whether indicators of risk are good predictors of inspection findings. The packs provide a useful starting point for planning, and may flag up issues that need to be followed up during the inspection. The limited impact of the packs is also due in part to gaps in the information available to populate them and to inspectors’ skill in interpreting them. The induction programme for new inspectors includes a one day session on inspection planning. Training is not tailored to the issues that arise when working with different types of provider. Staff shortfalls, however, have reduced the level of analyst support available to inspection teams.

3.19 The Intelligence function has a vacancy rate of approximately 20%. In some teams half the posts are either vacant or filled by temporary staff. Capacity for development work has been reduced and some operational activities have also been re-prioritised. Since 2009, the Commission, together with the Dr Foster Unit at Imperial College, has routinely monitored death rates in acute trusts to help identify risk of harm to people using these services by looking for changes that cannot be explained confidently by random variation. When an alert is raised, the Commission follows a staged process to consider if, and how, it should be pursued. This may lead to the alert being raised with the provider and, in some cases, to action by inspectors. The Commission runs a parallel process to detect outliers for maternity indicators. Shortfalls in analyst staffing have contributed to a substantial fall in the number of alerts raised by the service. In 2014, the number of alerts raised by the Commission was 36% lower than in 2013. In April 2014, the Commission decided that it could use the data provided by the Dr Foster Unit as the basis for any additional analysis and follow up action. There are a small group of conditions that the Dr Foster Unit exclude from their analysis. The Commission identified pneumonia as the main condition it would want to analyse that is not included in the Dr Foster Unit alerts, but it has not so far had the resources to carry out this analysis. The Commission continued scanning maternity outliers, but has reduced the frequency with which it has generated new alerts.

3.20 The Commission intends to develop a new monitoring programme, and has a target to recruit 60 senior analysts and team leaders by the end of 2015. In 2014, it set up a graduate training scheme and recruited 13 people to analyst posts. However, internal capacity is unlikely to increase in the short term.

7 Figures relate to the three months January to March 2015.

8 Year to 20 April 2015.

Part Four

Accountability, leadership and governance

4.1 In March 2012, the Committee of Public Accounts voiced serious concerns about the Care Quality Commission’s (the Commission’s) governance, leadership and culture. The Committee noted that a board member, Commission staff, and representatives of the health and adult social care sectors had all criticised how the Commission was run. This reinforced the findings of a capability review of the Commission by the Department of Health (the Department), published in February 2012. The review concluded that the board needed to include a broader range of experience, and that it should provide greater challenge to the executive team.

Leadership

4.2 The Commission has completely changed its executive leadership. This now includes the chief executive, three chief inspectors, a director of strategy and intelligence and director of customer and corporate services. The Commission’s 2014 staff survey shows 70% of staff feel the leadership behaviours are consistent with the Commission’s values of integrity, teamwork, professional excellence and treating people with dignity and respect (Figure 12). Staff said they were less confident about the decisions made by the leadership and the way changes are communicated. Taken as a whole, however, scores compare well against benchmarks from the civil service people survey. The Commission is considering how to improve its communication to staff and has, for example, introduced a weekly email to staff from the chief executive.

Board capacity

4.3 The Commission’s board has nine non-executive members. The Department appointed a new chairman in December 2012 and eight new non-executive members between July 2012 and July 2014. The eight new appointments bring experience from the public and private sectors, other regulators, consumer advocacy and medical law. The chair of the board stepped down in May 2015 to take an appointment as a government minister in the House of Lords. The Department has appointed the lead non-executive member as interim chairman until it appoints a replacement.

4.4 When we last reported, the Commission’s board comprised only the non-executive members. The Department has since appointed five senior executives to the board: the chief executive, three chief inspectors and the executive director of strategy and intelligence. This meets good practice that there should be executives (as well as non-executives) on the board. This helps the board understand the organisation’s performance and improve accountability.

4.5 The executive leadership consulted the board in July 2014 on its proposal that the Commission carry out fewer inspections in 2014-15 than it had originally planned. The board agreed to the revised timetable, but challenged the executive on the way inspections would be prioritised, whether inspection teams were the right size and whether the executive were confident intelligent monitoring data would identify problems in providers without an inspection. In March 2015, the executive asked the board to agree the Commission’s business plan for 2015-16, highlighting that it had pushed back its dates for inspecting all providers. The board agreed to the revised plan.

Governance

4.6 Following an internal review in autumn 2014, the Commission redefined the executive team’s terms of reference and reconfigured the board committee structures for 2015-16. Its governance structure and processes are consistent with best practice in many areas. For example, board meetings are regularly held in public, the board is supported by a functioning audit committee chaired by an independent non-executive director, and there is an internal audit service that regularly reports to the audit committee. The Commission nonetheless recognises that it needs to do more to make sure that staff understand all its governance processes and these are being followed in practice. Allowing for all new recruits, the Commission predicts that a third of its staff will have been at the organisation for less than a year. So, while the Commission has set out a scheme of delegation, it needs to increase awareness of it among operational staff to make sure decisions are made by the right people.

4.7 The Commission could also do more board development work to match best practice and the Department’s expectations. It is good practice to evaluate the board’s effectiveness at least annually. The Commission reviewed its board and subcommittees in 2015, although it did not carry out any reviews for at least the three preceding years.

The Department of Health’s sponsorship role

4.8 The Department is ultimately responsible for the effective regulation of health and adult social care. The Commission is independent of the Department in respect of the regulatory decisions it makes, but remains accountable to the Department for its own performance and value for money. The Committee of Public Accounts previously reported that the Department had underestimated the scale of the task it had set in requiring the Commission to merge three bodies at the same time as taking on an expanded role. It also reported that, although the Commission had not dealt with problems effectively the Department had been slow to take action.

4.9 Over the last two years, the Department has been closely involved in developing the Commission’s new regulatory approach. In March 2014, it published a framework agreement, setting out the accountability arrangements between the Department and Commission.9 It agreed the Commission’s three-year strategic plan, which sets out the Commission’s priorities for its new approach to regulation, as well as its business plans and budget. It holds quarterly accountability meetings with the Commission, monthly finance meetings and other meetings as necessary at both a senior and a working level. The Commission consulted the Department on the changes the Commission made to its commitment dates for inspecting all providers using its new approach. The need for these changes suggests, however, that the Department could have examined more rigorously whether the original commitment dates were achievable. The Department set additional challenges in requiring the Commission to take on new responsibilities for the financial oversight of difficult to replace providers at the same time it was rolling out a new approach to inspection.

4.10 The Department sees the Commission as an organisation that has been in transition but is now settling into a steadier state. The Committee of Public Accounts reported in 2012 that neither the Commission nor the Department had defined what success would look like in regulating health and adult social care. The Department and the Commission have since made very significant changes to the regulatory regime. The Department intends to reduce its operational oversight of the Commission in the future. It has not yet formalised how it will hold the Commission to account for its performance under these new arrangements, or established what information (including measures of efficiency and effectiveness) it will use to monitor the Commission’s performance.

Measuring value for money

4.11 When we last reported, we found the Commission had limited information to measure the quality, outcomes, impact and cost of its regulatory activities. There were gaps in the data, with few time-related measures of performance and no quality or outcome indicators.

Understanding the quality, outcome and impact of regulation

4.12 Since our report, the Commission has updated its performance framework to include a much wider range of metrics. The Commission published in its 2015-16 business plan a framework that sets out the logical relationships between its resource use, activities and the outcomes it is seeking to achieve.10 It now includes measures of timeliness (such as the time taken to respond to safeguarding concerns), quality (such as the proportion of ratings that are challenged and upheld), and outcome (such as the proportion of providers rated as inadequate or requires improvement that improve when re-inspected). There is a clear rationale for the selection of indicators, which are linked to the operational activities of the organisation and the outcomes it is seeking to achieve.

4.13 There are, however, still gaps in the Commission’s performance information:

The Commission does not yet measure and report on the overall impact of its regulatory activities. It has commissioned advice from external consultants to help develop a methodology for doing so, and is considering how to translate this into measurable indicators.

The performance measurement framework does not include direct measures of outcomes for patients. The set of indicators includes two relating to improvement in inspection ratings.

The Commission set a specific target for no more than 6 of the 37 measures in its published business plan. We also found that for 6 of the 37 measures there is no baseline data because the model is different to before. Setting targets improves accountability and helps demonstrate whether organisations are doing well against the objectives they are trying to achieve.

4.14 There is also scope to improve the quality of data used to measure some aspects of performance. For example, the Commission does not have accurate data for tracking the progress of individual inspections, and cannot follow complaints received from whistleblowers and members of the public through to outcomes. In May 2015, it purchased software to help improve its planning and monitoring of regulatory activities.

Understanding the cost of regulation

4.15 The Commission’s costs have been increasing over recent years, historically in line with an increase in the level of regulatory activity, and more recently to fund the transformation of the Commission and its new approach to inspection (Figure 13). The Commission underspent against its budget for 2014-15 by £12.1 million (7%). This was because of a £12.1 million underspend against its staff budget as it was operating below its full complement of staff.

4.16 The Commission is funded by grant-in-aid from the Department and fees charged to regulated bodies. The Health and Social Care Act 2008 gives the Commission power to charge fees to providers once they are registered. Fees cover the cost of monitoring, inspecting and regulating services. The fees charged depend on the size and type of provider. The Commission must consult providers on proposals to change its fee structure. This must be approved by the Secretary of State for Health.

4.17 In line with HM Treasury guidance, the Commission is moving towards fuller cost recovery, which means that fees charged to providers will rise.11 The Department has not set a specific timetable for this. The amount recovered from fees charged to providers has increased in recent years from £93 million in 2012-13 and £100 million in 2013-14 to £103 million in 2014-15 (Figure 14). The Commission’s budget increased significantly between 2013-14 and 2014-15. This meant that although the fees charged to providers increased, the overall proportion of the Commission’s total budget recovered from fees in 2014-15 (46%) was lower than in 2013-14 (56%). The Commission has increased the amount recovered from fees in 2015-16 to £113 million (a 9% increase from the previous year). Providers responding to the Commission’s consultation on fees for 2015-16 raised concerns about an increase in fees at a time when their budgets and revenues are under greater pressure.

4.18 The Commission will need to understand the full cost of its regulatory activities to justify the fees charged to different sectors and demonstrate value for money. The Commission has adopted a ‘top-down’ approach to costing, based on budget data to allocate overhead costs. It validates assumptions in the model with a retrospective ‘bottom-up’ exercise.

4.19 The Commission’s top-down approach creates a cost based on the budget for four parts of its operating model: registration; inspection; enforcement; and independent voice. The costings are likely to be highly sensitive to changes in the assumptions used to share costs between the four areas. These assumptions will need to be refreshed as the Commission recruits new staff and better understands how the operating model works. Changed assumptions are likely to have a material impact because approximately half the Commission’s total budget is apportioned to the four parts of its operating model.

9 Department of Health and Care Quality Commission framework agreement, available at: www.cqc.org.uk/sites/default/files/20150706_dh_framework_agreement_2014.pdf

10 Care Quality Commission, Business Plan April 2015 to March 2016, available at: www.cqc.org.uk/content/business-plan-shaping-future

11 HM Treasury, Managing public money, July 2013.

Appendix One

Our audit approach

1 This report focuses on the Commission’s progress in transforming its business, and its capacity to implement its new approach. We examined:

whether the Commission has designed a robust approach to its regulatory activities that adequately responds to previous criticism;

whether the Commission, in practice, has the capacity to implement its new regulatory approach; and

whether the Commission measures its own performance and is effectively held to account.

2 In reviewing these criteria, we applied an analytical framework with evaluative criteria, of the characteristics we would expect to find as evidence the Commission has transformed its business and has the capacity to implement its new approach. This included assessing the extent to which the Commission has addressed previous criticism and whether it has met its own targets for building its capacity and carrying out discrete work packages.

3 Our audit approach is summarised in Figure 15. Our evidence base is described in Appendix Two.

Appendix Two

Our evidence base

1 We reached our independent conclusions on the progress the Care Quality Commission has made in transforming its business in fieldwork carried out between February 2015 and May 2015. Our audit approach is outlined in Appendix One.

2 We reviewed the Commission’s progress in transforming its business, and its capacity to implement its new approach.

3 We assessed whether the Commission had designed a robust approach to its regulatory activities that adequately responds to previous criticism:

We reviewed the Commission’s documents and interviewed officials at the Commission about its new approach to regulation. Our interviews were unstructured and were carried out with all of the Commission’s executive leadership. We also reviewed previous reports and review about the Commission, such as the Department of Health’s capability review, Sir Robert Francis’s report into the failures of care at Mid Staffordshire NHS Foundation Trust and the 2012 report by the Committee of Public Accounts.

We analysed the findings from the Commission’s survey of providers, carried out in 2014. This included a review of free-form provider comments about their understanding of the Commission’s new approach to inspection and regulation, and whether they thought the Commission’s communications had been clear. A total of 4,935 providers responded to the survey (a response rate of 11%).

We reviewed data and information the Commission uses to monitor provider compliance with fundamental standards. This included a review of the data the Commission uses to track action taken in responses to communication from the public, categorised into: significant notifications; complaints about providers; safeguarding calls; and whistleblowing enquiries. We also carried out unstructured interviews with staff at the Commission, including the Director of Intelligence, to understand how the Commission uses intelligent monitoring to monitor the risk in providers, and the Commission’s progress in moving towards an intelligence-driven regulatory model.

4 Whether the Commission, in practice, has the capacity to implement its new regulatory approach:

We analysed the Commission’s staff surveys, carried out in 2012, 2013 and 2014, to understand staff views on training and development, organisational culture and morale. A total of 2,537 staff responded to the 2014 survey (a response rate of 85%). We also carried out an analysis of free-form staff comments in the 2014 survey (although not all staff provided comments as it was optional). We benchmarked the Commission’s performance by comparing how people responding to the 2014 civil service people survey answered similar questions.

We analysed the Commission’s data on recruitment, resourcing and turnover, including by analysing whether the Commission had met its own targets.

We analysed the findings from the Commission’s survey of providers, carried out in 2014, to understand whether providers think the Commission’s staff have the relevant skills and experience. A total of 4,935 providers responded to the survey (a response rate of 11%).

We reviewed documents and carried out unstructured interviews with Commission staff about the Commission’s new responsibilities for financial oversight. Key documents included the Commission’s guidance to providers about how it will carry out its new responsibilities.

5 Whether the Commission measures its own performance and is effectively held to account:

We reviewed documents, and held unstructured interviews with Commission staff, about the Commission’s governance structure and processes. Key documents included, for example, the Commission’s corporate governance statement. We also reviewed how the Commission’s processes compared to good practice.

We analysed the Commission’s staff surveys, carried out in 2012, 2013 and 2014, to understand staff views on the Commission’s leadership and culture. We benchmarked the Commission’s performance by comparing how people responding to the 2014 civil service people survey answered similar questions.

We reviewed the Commission’s published performance management framework, and held unstructured interviews with Commission staff, to understand how it measures and reports on its performance.

We analysed the Commission’s data on its budget, spending and fee income, including by comparing changes over time.

We reviewed data on the cost of the Commission’s regulatory activities. We also carried out an unstructured interview with the Commission’s finance director, to understand how costs are measured.

Appendix Three

The inspections we observed

1 We observed inspections at three separate care providers: a hospital trust; a nursing home; and a GP practice. We did this to help us understand how the Commission carries out inspections. This is not a representative sample, and we cannot extrapolate from three inspections to draw any overall conclusions about how well the Commission carries out inspections. We are, however, able to comment on what we saw and the insights we gained from our discussions with inspectors. In this appendix we draw together common themes emerging from the three inspections we shadowed.

The inspected bodies

2 We attended three inspections:

A hospital trust: a multi-site general and acute hospital trust with 2 accident and emergency departments and over 1,100 beds. The members of the full inspection team were grouped into sub teams allocated to, for example, accident and emergency, surgery, medical care, critical care and other services. On the first day, the inspection chair, lead inspector and analyst presented an overview of the trust. The trust’s board of directors then presented an assessment of the organisation’s successes and areas in need of improvement. During a week-long process, inspection teams assessed the care provided to patients. In some cases they returned unannounced during the evening to look again at areas they had seen during the day. The inspection examined how well the trust’s strategy and vision was communicated and the systems in place for managing risks to patient care. It also included very detailed activity, such as looking for out of date items in resuscitation trollies and drug fridges. When in clinical areas, the inspectors, identifiable by their Commission name badges, approached patients, their relatives, hospital staff and ambulance personnel to ask them for their views about the care being provided.

A care home: an independent family-run nursing home for older people with 30 residents that has provided services for a number of years. Residents’ care is funded mainly through local authority placements. Inspectors arrived with no advance notice and spent a day talking to staff and people living in the home, as well as reviewing files and observing how care was provided.

A medium-sized general practice with 4 doctors serving approximately 6,000 patients. The practice undertakes minor procedures, provides nurse clinics and teaches qualified doctors in training. Inspectors met staff, patients waiting in reception and members of the practice’s patient participation group. Inspectors examined the interior and exterior of the building, including the GP consulting rooms, the treatment area and its contents. They also considered the ease of access to the practice, policies, patient information leaflets, notices and how appointments are made. This was an announced inspection.

Preparing for inspections

3 Before the inspections, the Commission’s central teams prepared information packs, and inspectors collected additional local information. The lead inspector took time to prepare other members of the inspection team and led a briefing to bring them up to speed on key issues and challenges including any concerns about performance. Areas in need of particular attention were identified from pre-inspection reports and information given to the Commission by members of the public, staff or other organisations.

4 Data are much more plentiful in the hospital sector than in general practice and social care. There were mixed views from hospital inspection teams as to the helpfulness of the pre-inspection data packs in terms of how these influenced inspection activities. In some cases the Commission may decide team sizes in advance because there are known issues in particular clinical areas. Otherwise inspectors proceed based on the standard five questions.

How inspections were carried out

5 The hospital inspection team included specialist clinicians, experts by experience, full time inspectors and support staff. Support was available from an analyst during the inspection. The inspection teams worked flexibly, because the total number of people available was lower than had been sought.

6 The inspection team for the care home was made up of 2 inspectors plus a specialist in pressure care. The general practice team also had 3 members: a lead inspector; a practising GP; and a practice manager.

7 In all three inspections we saw that questions and checking were linked directly to the Commission’s five key questions designed to test whether services are safe, effective, caring, responsive and well-led. Inspectors made concerted efforts to talk to service users and, where appropriate, witness the care they received. Inspectors formed their views by using different sources of information: for example, using interviews and checking files.

8 In face-to-face interviews and in focus groups, inspectors encouraged participants to be open and forthcoming about any concerns they had about patient care, the resources available to them and how those responsible for managing care responded to their concerns.

Feedback to inspected bodies

9 At the end of the on-site inspection, teams reviewed their findings and evidence. They discussed good practice and areas in need of attention. In the case of the hospital inspection, the separate clinical sub-teams provided preliminary ratings for the 5 key questions (safety, effectiveness, caring, responsiveness and being well-led), indicating whether these were inadequate, needing improvement, good or outstanding.

10 At all three locations, the inspection teams considered whether there were concerns that needed the immediate attention of the inspected body. The teams prepared their findings and lead inspectors provided feedback to the organisation’s management during the closing session.

Inspectors’ feedback

11 Inspectors told us that the inspection days are just one step in preparing an inspection report and issuing a rating. They referred to gathering pre-inspection information, planning for and assigning CQC personnel to specific inspections, collecting documents from the inspected body, and preparing a draft report and scrutiny within the Commission’s internal quality assurance process before it is shared with the inspected body for comment.

12 We were also told that the redesigned inspection approach is better than the previous system in that it allows a degree of flexibility and judgement when evaluating actions being taken to improve services. They said this was preferable to simply marking the body compliant or non-compliant with the standard. However, the new process was increasing the workload and lengthening the time between the inspection and issuing of a draft report.

Appendix Four

Summary of progress against Committee of Public Accounts’ recommendations

Design and Production by NAO Communications
DP Ref: 10765-001