Evaluating innovation in children’s social care

Department for Education

REPORT

by the Comptroller
and Auditor General

Session 2022-23
24 June 2022
HC 70

The National Audit Office (NAO) scrutinises public spending for Parliament and is independent of government and the civil service. We help Parliament hold government to account and we use our insights to help people who manage and govern public bodies improve public services.

The Comptroller and Auditor General (C&AG), Gareth Davies, is an Officer of the House of Commons and leads the NAO. We audit the financial accounts of departments and other public bodies. We also examine and report on the value for money of how public money has been spent.

In 2020, the NAO’s work led to a positive financial impact through reduced costs, improved service delivery, or other benefits to citizens, of £926 million.

We are the UK’s independent public spending watchdog.

We support Parliament in holding government to account and we help improve public services through our high-quality audits.

Evaluating innovation in children’s social care

Department for Education

Report by the Comptroller and Auditor General

Ordered by the House of Commons
to be printed on 22 June 2022

This report has been prepared under Section 6 of the National Audit Act 1983 for presentation to the House of Commons in accordance with Section 9 of the Act

Gareth Davies
Comptroller and Auditor General
National Audit Office

16 June 2022

This report examines the Department for Education’s approach to funding innovative practices in children’s social care, through the Children’s Social Care Innovation Programme and related interventions. It sets out how evaluation informed policy development and funding.

The material featured in this document is subject to National Audit Office (NAO) copyright. The material may be copied or reproduced for non-commercial purposes only, namely reproduction for research, private study or for limited internal circulation within an organisation for the purpose of review.

Copying for non-commercial purposes is subject to the material being accompanied by a sufficient acknowledgement, reproduced accurately, and not being used in a misleading context. To reproduce NAO copyright material for any other use, you must contact copyright@nao.org.uk. Please tell us who you are, the organisation you represent (if any) and how and why you wish to use our material. Please include your full contact details: name, address, telephone number and email.

Please note that the material featured in this document may not be reproduced for commercial gain without the NAO’s express and direct permission and that the NAO reserves its right to pursue copyright infringement proceedings against individuals or companies who reproduce material for commercial gain without our permission.

Links to external websites were valid at the time of publication of this report. The National Audit Office is not responsible for the future validity of the links.

011776 06/22 NAO

This report can be found on the National Audit Office website at www.nao.org.uk

If you need a version of this report in an alternative format for accessibility reasons, or any of the figures in a different format, contact the NAO at enquiries@nao.org.uk

The National Audit Office study team consisted of:

Chris Battersby, Andrew Denney, Richard Douglas and Mark Parrett, under the direction of Ashley McDougall.

For further information about the National Audit Office please contact:

National Audit Office
Press Office
157–197 Buckingham Palace Road
Victoria
London
SW1W 9SP

020 7798 7400

www.nao.org.uk

@NAOorguk

What this report is about

1 The quality of children’s social care matters: where it is not adequate, children in need of help or protection may be exposed to neglect, abuse or harm. Since 2014 the Department for Education (the Department) has been funding a range of innovation projects intended to help local authorities improve both outcomes for children and value for money by developing new ways of working.

2 In total, since 2014, the Department has provided or committed some £333 million in support of innovation projects, their evaluations and linked programmes. At the core of the support for innovation was a commitment to evaluating the impact of local projects and using the results of evaluations to support more effective and efficient ways of delivering children’s social care services.

3 In 2021 we received correspondence from a member of Parliament, drawing our attention to a local project, funded through the Children’s Social Care Innovation Programme (the Innovation Programme), which had received a positive evaluation. We decided that there was a good case for a report setting out the strengths of the Department’s overall approach to funding and evaluating innovation projects. Looking at the Department’s approach, through setting out the facts on a series of interventions of this kind, allows us to explore the wider value of evaluation to a department’s work and draw out lessons for other departments.

4 The Department has already subjected its innovation projects to academic evaluation, and What Works for Children’s Social Care (fully operational since 2019) is reviewing the ongoing projects. We have designed this report as a case study of a department’s approach to using evaluation within a policy development cycle, covering how it:

designs policy interventions with evaluation in mind;

assures the quality of its evaluations; and

uses the results of evaluation to inform the development of further interventions.

5 This report does not seek to examine the value for money of the Department’s innovation projects. What it provides is a factual account of the Department’s oversight and funding of innovation projects, and how it has used evaluation in its policy development process. Our focus is on the role of the Department, given its policy responsibility for children’s services.

6 Our report covers:

Part One: development of the Department’s funding of innovation in children’s social care since 2014;

Part Two: use of evaluation in the Innovation Programme between 2014 and 2020;

Part Three: building on learning from the Innovation Programme; and

Part Four: overview of evaluation and next steps.

7 We conducted our fieldwork between January and March 2022. Our methodology comprised document review, review and analysis of financial and performance data, and interviews with officials at the Department and at What Works for Children’s Social Care. Further details are set out in Appendix One.

Summary

Key findings

The Children’s Social Care Innovation Programme

8 The Department for Education (the Department) launched the Children’s Social Care Innovation Programme (the Innovation Programme) in 2014 with the aim of improving outcomes for children and value for money. In March 2014, the Department reported on the challenges of achieving innovation in children’s social care and found unexplained variations across local authorities in the outcomes, quality, value and pace of improvement for children in need. The Department intended the Innovation Programme to:

create stronger incentives and mechanisms for innovation, experimentation and replication of successful new approaches;

achieve better life chances for children receiving help from the social care system; and

deliver better value for money across children’s social care (paragraphs 1.5 and 2.2).

9 The Department committed £212.8 million to the Innovation Programme between 2014-15 and 2019-20, funding a range of sector-led projects that were designed by local authorities and the providers they work with. The Department invited competitive bids from local authorities and other providers and contracted a delivery partner to assist bidders in developing their ideas. The Innovation Programme funded 94 projects with £199.3 million, including such schemes as:

an integrated mental health, education and families service to support children in or on the edge of care;

intensive targeted support for young people experiencing multiple placements to secure more stable placements and a positive transition to adulthood; and

relationship-based support for women who have experienced removal of their children into care to avoid further removal of children.

The Department also spent a further £13.5 million, on top of the £199.3 million in project funding, on support for, and evaluation of, Innovation Programme projects (paragraphs 2.5 to 2.7, Figure 2 and Figures 3 to 5).

10 The Department awarded three rounds of funding in total, having decided to extend the Innovation Programme in 2016:

Round One (2014-15 to 2015-16). The Department funded 57 projects focusing on areas including rethinking children’s social work and support for adolescents in or on the edge of care.

Round Two (April 2016 to March 2020). The Department funded 24 projects (including eight previously funded in Round One), aimed at building on the innovations found in Round One.

Round Three (November 2016 to March 2020). The Department funded 21 projects in further priority areas of residential care, alternative delivery models, targeted support and social investment models (paragraphs 2.5 to 2.7).

The role of evaluation in the Innovation Programme

11 From the outset the Department made evaluation and dissemination of learning a central part of the Innovation Programme. The Department made it a condition of receiving Innovation Programme funding that projects would be subject to external evaluation and appointed a range of research teams to carry out these evaluations. In the absence of a What Works Centre (which was first established in 2017, and not fully operational until 2019), the Department contracted two academic research partners (one for Round One and another for Rounds Two and Three) to oversee individual evaluations and publish overall evaluations of each round of funding. The Department’s delivery partner for the Innovation Programme shared emerging learning from local projects through conferences and a website (paragraphs 2.8, 2.9 and 4.10).

12 The Department strengthened its approach to evaluation in later rounds of innovation funding. For Rounds Two and Three of the Innovation Programme the Department developed a new evaluation strategy, which assessed the robustness of evaluation plans and encouraged harmonisation of research questions and measurement of outcomes. It also sought to build on practices that had already been developed by inviting applications from Round One projects looking to undertake wider testing of their innovations (paragraph 2.9).

13 Most evaluations of individual projects noted positive results and some reported cost savings. By September 2020, the Department had published around 100 evaluation reports covering individual projects. A number of these reported positive outcomes. In particular:

Round One. The Department’s overall evaluation of Round One found that 42 of the 56 completed projects reported improvements in the quality of services at the end of their delivery periods. Of these, 24 projects reported quantitative improvements such as reduced numbers of children in care or identified as in need, and reunifications with birth families. On value for money, it found that 21 projects had reported cost savings; and

Rounds Two and Three. The Department’s overall evaluation of Rounds Two and Three found that of 31 Innovation Programme evaluations assessing impact, and or outcomes, the majority (21) demonstrated some positive impact or positive changes (paragraph 2.10).

14 The Department was transparent in publishing the evaluation findings from across the Innovation Programme. The Department published around 100 evaluation reports, for projects representing 90% of the Innovation Programme’s value. This contrasts favourably with the findings in our 2021 report Evaluating government spending that only 36% by spend of the projects in the Government Major Projects Portfolio had impact evaluation plans in place (paragraphs 2.10 and 2.11).

15 The Department carried out its own quality review of local evaluations of funded projects. During the Innovation Programme the Department had a process in place to review and approve higher-value and sensitive project evaluation plans and to oversee the progress and quality of evaluations. It contracted with a single research partner in Round One (the Rees Centre), and another for Rounds Two and Three (Opcit Research), to help support the individual project evaluations and to produce an overall evaluation of the entire round. In each round the Department operated an evaluation steering group or advisory board. Comprising departmental officials, these groups liaised with evaluation partners on the progress of project evaluations, critically reviewing a selection of higher-value or more sensitive project evaluation plans for approval (paragraphs 2.8 and 2.11 to 2.13).

16 The Department commissioned and published independent overall assessments of Round One, and of Rounds Two and Three together, which were carried out by its research partners. These gave critical assessments, highlighting a number of significant challenges and common limitations of the evaluations carried out on individual projects. The independent reviewers noted that project evaluations were affected by brief timescales for measuring impacts, small sample sizes, a lack of genuine comparison groups and a lack of high-quality data. They also concluded that variations in project and evaluation design meant it was impossible to compare projects’ relative costs and benefits. The Department and What Works for Children’s Social Care point to practical challenges to implementing robust evaluation during the Innovation Programme, including the lack of a culture of evaluation within the sector. They suggest that the experience of the Innovation Programme has nevertheless helped to build stronger evaluation into its successor programmes (paragraphs 2.12 to 2.17).

17 The Department has not monitored the longer-term outcomes from projects in the Innovation Programme after their funding came to an end or assessed the extent to which the reported good practices have spread across the sector. The Department required Innovation Programme projects to demonstrate how they would sustain benefits beyond the period of funding. In general, however, it has not revisited projects since the completion of the Innovation Programme to track how many have been maintained by local authorities in the absence of grant funding, and whether any benefits delivered during the programme have been maintained. Equally, while the Department and its Innovation Programme delivery partner have promoted learning from innovation projects (a function which What Works for Children’s Social Care has now begun to take on), the Department has not sought to assess the wider influence of learning from projects on other local authorities which it did not fund (paragraph 2.18).

Building on learning from the Innovation Programme in subsequent schemes

18 Following the Innovation Programme, the Department has continued to fund children’s social care innovation projects and evaluations with a further £120.2 million. Successor programmes have included:

Strengthening Families, Protecting Children (Strengthening Families): a programme testing the wider adoption of three ‘whole system’ Innovation Programme projects across 17 further local authorities over five years from 2019-20. This scheme specifically focused on testing these innovations in authorities rated by Ofsted as ‘requiring improvement’ and with high and rising demand. The Department is providing £77.4 million in project funding, £1.6 million for evaluation partners and £2.5 million for Strengthening Families delivery partner support; and

Supporting Families: Investing in Practice (Supporting Families): a programme testing the wider adoption of three ‘targeted’ projects from the Innovation Programme across more than 40 further local authority areas over two years from 2019-20. The Department is providing £16.1 million in project funding and £1.6 million for support from evaluation partners.

In April 2020 the Department also provided £13.7 million through an Innovation Programme Continuation Fund to allow 17 other Innovation Programme projects to continue work and publish learning. In July 2021 it further provided £7.3 million through its wider COVID-19 Regional Recovery Fund to 27 projects supporting local authorities to explore adoption of some 17 Innovation Programme models (paragraphs 3.2 to 3.6 and Figure 2).

19 The Department used learning from the Innovation Programme in selecting projects for the two successor programmes. The Department made decisions about which Innovation Programme projects to support through Strengthening Families and Supporting Families in 2018-19. By that point it had published evaluations available for all Round One projects, although none for Rounds Two and Three projects. The Department took into account a range of factors when selecting the most promising projects to support through these programmes:

Strengthening Families: The Department considered the candidate projects’ impact, as reported in evaluations, and the quality of those evaluations. It also looked at whether projects displayed seven common elements of good practice that the Department and the Innovation Programme delivery partner had identified during the first round of the programme. Documents we saw suggest the Department considered a shortlist of five Round One projects.

Supporting Families: the Department and What Works for Children’s Social Care considered the strength of evidence and impact from each project’s evaluation, as well as the sector’s appetite for the change, the capacity of the original innovating organisations to support roll-out and the project’s fit with government policy and evaluation priorities. Seven Round One projects were shortlisted for this assessment (paragraphs 3.7 and 3.8).

20 The Department has also used learning from the Innovation Programme to design more sophisticated evaluations of projects in the successor programmes. The Department wanted the evaluation of Strengthening Families to be as robust as possible to provide an evidence base sufficient to say with greater certainty whether the projects work, and in what conditions. It therefore decided to roll out and evaluate Supporting Families in partnership with What Works for Children’s Social Care and meet the Centre’s standards of evidence. Examples of the increased sophistication of the evaluation are:

the Department implementing each of the three projects in the Strengthening Families programme in waves to allow for a ‘stepped wedge’ randomised control trial and an implementation and process evaluation. Under this approach, the start dates for local authorities beginning their projects will be staggered, allowing for comparisons in outcomes between early and later starters. Programme funding will end by March 2024, with evaluations led by What Works for Children’s Social Care reporting in 2026 and 2027. The Department also let a contract for a new delivery partner for Strengthening Families to provide project support, including spreading learning between programme participants and out to the wider sector; and

in the Supporting Families programme, the Department is using a randomised control trial in one project and other types of comparison groups in the remaining two. The Department expects projects to continue until June 2022, with evaluations reporting between October 2022 and May 2023. What Works for Children’s Social Care is overseeing the evaluation and has outsourced the work to three independent providers (paragraphs 3.10 to 3.12 and Figure 2).

Concluding remarks

21 The Department’s Innovation Programme was a positive step in its aim of funding new ways of working and evidence-based practice, with a commitment to evaluation at its heart. Evaluation can identify where local authorities can make the greatest difference for children and their families. The Department showed good practice in embedding evaluation into the projects it funded from an early stage and in publishing academic assessments, which were often critical of the quality of these evaluations. While the Department has faced practical challenges to securing robust evaluations of its interventions, working with What Works for Children’s Social Care it has been able to improve the sophistication of its evaluations over time and demonstrate its long-term commitment to evaluation. It has also been able to draw on the evidence from the Innovation Programme to select a number of more promising projects for wider roll-out within the sector.

22 Evaluating innovation and building that into decision-making is fundamental to good spending of public money. The Department’s approach provides an example of using evaluation within a cycle of policy development which will be of wider value for other departments to study. We encourage the Department to continue, expand and share its approach to evidence-informed decision-making. Where the Department may need to go further is in demonstrating how its funding of innovation in select local authorities will lead to:

the building of greater local evaluative capacity;

better use of public money;

wider dissemination of good practice across the sector; and

ultimately, improved outcomes for children in and around the care system.

Part One

Innovation in children’s social care

1.1 This part of the report sets out the background to the Department for Education’s (the Department’s) promotion of innovation in children’s social care.

The children’s services system

1.2 The Department has overall policy responsibility for children’s services. One of its four priority outcomes is to support the most disadvantaged and vulnerable children and young people through high-quality local services so that no one is left behind.1 Local authorities in England have statutory responsibility for protecting the welfare of children and delivering children’s social care. The Department is responsible for the legal and policy frameworks within which local authorities operate. The Department is accountable to Parliament for ensuring that children’s services, which local authorities are responsible for delivering, are of adequate quality to protect and support children. To achieve this, it issues statutory guidance, which clearly sets out the duty for local authorities and their partners, such as the police and health services, to work together to protect children. The Department is also responsible for:

providing information to support benchmarking (of practice and costs) and local accountability;

agreeing the framework against which Ofsted inspects local authority services; and

intervening in a failing local authority when Ofsted judges services to be ‘Inadequate’ because of widespread or serious failures that leave children being harmed, or at risk of harm.

1.3 Within central government, the Department for Levelling Up, Housing & Communities has overall responsibility for local authority funding, including the funding available to provide children’s services. Local authorities’ children’s services are inspected by Ofsted. Local authorities are responsible for ensuring that funding for children’s social care is spent with regularity and propriety and achieves value for money. In 2019-20, local authorities spent some £9.2 billion on children’s social care.2 The Department provides some funding through separate grants – in 2020-21, it spent a total of £235 million on children’s services.3

Innovation in children’s social care

1.4 The Munro Review of Child Protection recommended in 2011 that the government should remove constraints to local innovation and professional judgement that are created by prescribing or endorsing particular approaches.4

1.5 The Department announced the Children’s Social Care Innovation Programme (the Innovation Programme) in 2013 with plans to provide £100 million for projects from 2014-15 to 2015-16. In March 2014, in Children’s Social Care Innovation Programme – The challenge, the Department reported on the challenges of achieving innovation in children’s social care and found unexplained variations across local authorities in the outcomes, quality, value and pace of improvement for children in need.5 It identified eight main areas for significant and sustained improvement in service delivery, with a strong emphasis on developing evidence-based practices and processes.6

1.6 The Department launched two further rounds of Innovation Programme funding in 2016. In July 2016, in Putting Children First, the Department set out in more detail its strategy for achieving its vision through reform of people and leadership, practice and systems, and governance and accountability.7 This strategy included expansion of the Innovation Programme to “test new approaches to tackling the most important and difficult practice questions facing the children’s social care system”.

1.7 Subsequent to the Innovation Programme the Department has introduced successor programmes which seek to build on its work. In October 2018 the Department announced the Strengthening Families, Protecting Children programme, while in May 2019 it announced the Supporting Families: Investing in Practice programme. A timeline showing the development of the Department’s sponsorship of innovation in children’s social care since 2013 is at Figure 1, with funding in Figure 2.

1.8 When the Department started its Innovation Programme there was no What Works Centre for children’s social care to draw on for expertise in practice and evaluation. The Department announced the creation of a new What Works Centre focusing on children’s social care in 2016, stating that its purpose was to test the strength of evaluated evidence, disseminate key findings and work with the sector to develop innovative approaches.8 What Works for Children’s Social Care was established in 2017 and became fully functional in 2019.9 The Department appointed the Spring Consortium as a delivery partner for the Innovation Programme. The consortium consisted of Deloitte, Mutual Ventures and Innovation Unit with a cost of £12.4 million between 2014 and 2018.

1 Department for Education, DfE Outcome Delivery Plan: 2021 to 2022, July 2021.

2 National Audit Office, Financial sustainability of local authorities visualisation: update, July 2021.

3 Department for Education, Consolidated annual report and accounts for the year ended 31 March 2021, HC 907, December 2021.

4 Department for Education, The Munro Review of Child Protection: Final Report: A child-centred system, Cm 8062, May 2011.

5 Department for Education, Children’s Social Care Innovation Programme – The challenge, March 2014.

6 The Department highlighted four areas in particular where increased use of evidence could lead to improved outcomes: ‘Greater social worker capabilities and capacity’, ‘More targeted early intervention’, ‘More support for evidence-based care planning’ and ‘Better care for adolescents’. Department for Education, Children’s Social Care Innovation Programme – The challenge, March 2014, p. 8.

7 Department for Education, Putting children first: Delivering our vision for excellent children’s social care, July 2016.

8 See footnote 7, p. 41.

9 What Works Network, The What Works Network: Five Years On, January 2018.

Part Two

Evaluation of work funded through the Children’s Social Care Innovation Programme

2.1 This part of the report describes the Department for Education’s (the Department’s) implementation and evaluation of the Children’s Social Care Innovation Programme (the Innovation Programme).

Innovation Programme objectives

2.2 The Department intended the Innovation Programme to provide tailored and substantial support for innovation and re-design of service delivery with the aim of achieving, in five years:

stronger incentives and mechanisms for innovation, experimentation and replication of successful new approaches;

better life chances for children receiving help from the social care system; and

better value for money across children’s social care.10

2.3 The Department selected projects for funding on the basis of six criteria – that projects should:

significantly improve outcomes for children and young people in need of social care support;

improve value for money;

be successfully delivered;

be sustained and mainstreamed;

have an impact at a bigger scale; and

be innovative.11

2.4 One of the Department’s design principles for the Innovation Programme was that it should: “Foster bottom up, sector-led innovation, drawing on a strong base of evidence and evaluation.”12 The Department told us that, through the Innovation Programme, it wanted to build a culture of evaluation in children’s social care and encourage local authorities to take an active role in the development of evaluation.

Innovation Programme funding

Round One (2014 to 2016)

2.5 The first round of the Innovation Programme invited applications on two themes:

rethinking children’s social work; and

rethinking support for adolescents in or on the edge of care.

Between July 2014 and October 2015 the Department awarded 57 grants totalling £100.8 million, with completion expected by March 2016. Examples of projects in Round One are at Figure 3. In March 2016 a further £6.8 million was awarded through a competitive Transition Fund to extend work on a number of Round One projects judged to have the potential to further evaluate their impact over a longer period and allow other projects that had experienced set-up delays in Round One to complete the data collection needed for their initial evaluations.

Rounds Two and Three 2016–2020

2.6 In April 2016 the Department launched Round Two of the Innovation Programme. As well as bids to further develop projects already supported in Round One, it funded new bids focused on its latest priorities in rethinking children’s social care and transitions to adulthood for young people in the children’s social care system (see examples at Figure 4). The Department awarded £34.3 million to eight existing Round One projects and £29.6 million to 16 new projects, with expected completion by March 2020.

2.7 In November 2016 the Department announced Round Three of the Innovation Programme. The Department invited bids for new projects in its priority areas of residential care, alternative delivery models, targeted support and social investment models (see examples at Figure 5). The Department awarded £27.8 million to 21 projects with expected completion by March 2020.

Evaluation of the Children’s Social Care Innovation Programme

Governance of evaluation

2.8 The Department made evaluation central to its design of the Innovation Programme from the outset, with all projects having to agree to evaluation as a condition for their funding. During the Innovation Programme the Department had a process in place to review and approve higher-value and sensitive project evaluation plans and to oversee the progress and quality of evaluations:

Project evaluation teams. The Department invited tenders from research teams to be placed on a framework contract, from which they would be selected to carry out evaluations of individual projects. There were 22 evaluation teams in Round One and 17 evaluation teams in Round Two. The Department considered that inviting tenders from multiple teams helped to build a research market for children’s social care, increasing the capacity for evaluation within the sector.

Programme evaluation partners. The Department contracted with a single research partner for Round One (the Rees Centre, appointed in September 2014), and another for Rounds Two and Three (Opcit Research, appointed in April 2017), to help support the individual project evaluations and to produce an overall evaluation of the entire round. Evaluation partners provided guidance to project evaluation teams regarding the form of evaluation they should pursue. For example, evaluation teams in Round Two were asked to assess project impacts (using quasi-experimental designs where possible), processes and costs, using a ‘theory of change’ approach to explain how and why any impact or outcomes were achieved.13 The Department spent £1.1 million on these contracts.

Departmental evaluation teams. In each round the Department operated an evaluation steering group or advisory board. Comprising departmental officials (with the addition of officials from What Works for Children’s Social Care in Round Two), these groups critically reviewed a selection of higher-value or more sensitive project evaluation plans for approval, challenging them to meet established measures of good practice (such as the use of comparison groups). In Round One the individual project teams funded the research teams that were evaluating their work. In Round Two the Department both funded and managed evaluation teams itself, to ensure control of spend and consistency in measuring outcomes.

2.9 The Department developed its approach to evaluation as the Innovation Programme progressed. In Round One, drawing on the advice of its research partner, the Department operated a relatively non-prescriptive set of principles for assessing evaluation plans (for instance, that the aims of the proposed evaluation were clearly linked to the programme’s objectives and as robust as possible). It also sought to build on practices that had already been developed by inviting applications from Round One projects looking to undertake wider testing of their innovations. For Round Two the Department developed an explicit evaluation strategy to guide its assessment of evaluation plans. Key features of this strategy included that:

evaluation plans should be graded against the Early Intervention Foundation categories, a recognised five-point scale to assess the likely robustness of the eventual evaluations in advance;14

evaluation teams would be encouraged to harmonise research questions, approaches to measuring outcomes and forms of cost-benefit analysis; and

evaluations should aim to increase both sample sizes and use of comparison groups compared with Round One.

At the same time, the Department recognised that there should be a bespoke, bottom-up element to individual evaluation plans.

Findings of project evaluations

2.10 A number of the individual project evaluations reported positive outcomes. The Department’s two overall reports on project evaluations highlighted:

Round One: Forty-two of the 56 completed projects reported improvements in the quality of services at the end of the period (ranging from 10 to 18 months) over which projects were assessed.15 There were 24 projects which reported quantitative improvements against metrics such as numbers of children in care, children identified as in need, and reunifications with birth families. Twenty-one projects had reported cost savings, although four of these were projections of future savings rather than savings already achieved. Some 32 projects did not report on value for money; and

Rounds Two and Three: of the 31 Innovation Programme evaluations assessing impact or outcomes, the majority (21) demonstrated some positive impact or positive changes.16 Three Innovation Programme project evaluations produced evidence of statistically significant positive impact on intended outcomes. Four evaluations produced quantitative evidence of some positive outcomes, although without assessment of statistical significance.

Transparency of evaluation

2.11 The Department carried out evaluations on a large proportion of the projects funded by the Innovation Programme and was transparent in publishing the findings. By September 2020 the Department had published around 100 evaluation reports covering projects representing 90% of the Innovation Programme’s value. This contrasts favourably with the findings in our 2021 report Evaluating government spending that only 36% by spend of the projects in the Government Major Projects Portfolio had impact evaluation plans in place.17 The Department also published two reports evaluating the overall programme both in terms of the impacts achieved and the quality of evaluations undertaken. A November 2017 report assessed evaluations of the 56 projects completed in Round One, and a November 2020 report covered 47 evaluations including 38 of the 45 projects funded in Rounds Two and Three.

The Department’s assessment of project evaluations

2.12 During the Innovation Programme the Department had a process in place to review and approve higher-value and sensitive project evaluation plans and to oversee the progress and quality of evaluations. The Department commissioned the Rees Centre, Oxford University, to support and report on the quality of local evaluations for Round One, and Opcit Research to provide the same role for Rounds Two and Three together. These reports identified a number of significant challenges and common limitations of the evaluations carried out on individual projects in the Innovation Programme.

On the design of evaluations

2.13 There are three different types of evaluation: process, impact and economic (also known as value-for-money evaluation). Process evaluations examine activities and implementation. Impact evaluations focus on the impact of an intervention and identify the change in outcomes directly attributable to an intervention, as well as its impact on different groups. Economic evaluations compare the benefits and costs of an intervention and assess whether an intervention was a good use of resources.18 The overarching reports commissioned by the Department found:

for Round One, against the Early Intervention Foundation’s five-point scale measuring the quality of evaluation design, no evaluation plan was rated in the highest category, while almost all (49 out of 56) were in the bottom two;19

for Rounds Two and Three Innovation Programme projects in total:

nine evaluations produced ‘stronger’ comparative analyses, including measures of statistical significance;

six produced ‘moderate’ comparative analyses, not including any measure of statistical significance;

one produced comparative analyses that included measures of statistical significance but could not rule out selection bias and therefore had compromised validity;

15 evaluations assessed changes in outcomes among target populations but did not include comparative analyses; and

eight evaluations did not report on quantitative outcomes or impact.20

On the cost of evaluations

2.14 The Department only reported on the cost of evaluations for Round One. The majority (41) of evaluations cost less than 8% of total project funding, compared with 15%–20% in other major government-funded programmes. The report considered that the resources allocated to Round One evaluations were low for achieving robust results, but concluded that some Round One evaluations provided good value for money. For the review of Rounds Two and Three the report did not comment on the cost of the evaluations.

On the quality of evaluations completed

2.15 The Department reported similar limitations for all rounds of the Innovation Programme:

The evaluation periods were brief and did not allow realistic timescales for project activities and intended outcomes.

Project evaluations were affected by small sample sizes, a lack of genuine comparison groups and a lack of high-quality data.

There was insufficient commonality on either what data to collect or how to define the indicators.

Variations in project and evaluation design meant it was impossible to compare projects’ relative costs and benefits.

Several later evaluations were also affected by the COVID-19 pandemic.

2.16 The overall evaluation for Rounds Two and Three also reported on eight follow-up evaluations of Round One projects that had not received further funding. In general, the review found few signs of substantial or sustained positive impacts, with most projects showing mixed results. The review was unable to account for these results, in part owing to the light-touch nature of the follow-up studies.

2.17 In July 2021, What Works for Children’s Social Care published a review of the individual evaluations of Round Two projects, with the aim of summarising key lessons.21 It highlighted a “serious need for more research”, particularly more impact evaluations. It found that the less robust a project’s impact evaluation, the more likely it was to find a positive impact for the intervention. It concluded that “many of those positive findings are likely to be misleading or even false”. What Works for Children’s Social Care found that only around half of projects (26) had carried out a cost-benefit analysis, of which 13 reported a positive return. It noted that the absence of any cost-benefit analysis in many cases would make it difficult for local authorities to make investment decisions on future proposals for innovation projects.

2.18 The Department has not monitored the longer-term outcomes from Innovation Programme projects after their funding has come to an end, nor assessed the extent to which good practice has spread more widely across the sector. The Department required Innovation Programme projects to demonstrate how they would sustain benefits beyond the period for which they would be funded. In general, however, it has not revisited projects since the completion of the Innovation Programme to track how many have been maintained by local authorities in the absence of grant funding, nor whether any benefits delivered during the Innovation Programme have been maintained. Equally, while the Department, the Innovation Programme delivery partner and What Works for Children’s Social Care have promoted learning from innovation projects, the Department has not sought to assess the wider influence of learning from these projects on other local authorities which it did not fund.

10 Department for Education, Overview report: Department for Education Children’s Social Care Innovation Programme, April 2014.

11 Spring Consortium, Children’s Social Care Innovation Programme Interim Learning Report, January 2016.

12 See footnote 10, p. 8.

13 The ‘theory of change’ approach endorsed for Round Two evaluations aims to establish a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context. Department for Education, Children’s Social Care Innovation Programme Round 2 Final Report, September 2020, pp. 5, 10.

14 The Early Intervention Foundation categories were developed in association with the What Works Centres to assess interventions in terms of their impact and cost. The categories are broadly similar to the established Maryland Scale, and rate evaluations on a five-point scale ranging from 1 (evaluations based on simple cross-sectional correlations) to 5 (randomised control trials).

15 Department for Education, Children’s Social Care Innovation Programme: Final evaluation report, November 2017.

16 Department for Education, Children’s Social Care Innovation Programme Round 2 Final Report, September 2020. The full report included 38 evaluations assessing impact and outcomes, 25 of which demonstrated positive impact. The report includes evaluation of additional projects outside the scope of Rounds Two and Three of the Innovation Programme as covered in our report.

17 A December 2019 review found that projects worth 36.1% of planned expenditure in the Government Major Projects Portfolio had some form of impact evaluation plans in place (with 8.1% having robust impact evaluation plans, and 28.0% where impacts could be evaluated more robustly), while projects representing 63.9% of planned expenditure had no impact evaluation plans in place. Comptroller and Auditor General, Evaluating government spending, Session 2021-22, HC 860, National Audit Office, December 2021, Figure 2.

18 Comptroller and Auditor General, Evaluating government spending, Session 2021-22, HC 860, National Audit Office, December 2021.

19 J Sebba, et al (Rees Centre), Children’s Social Care Innovation Programme: Final evaluation report, Evaluation report 58, Department for Education, November 2017, p. 76.

20 A FitzSimons and K McCracken (Opcit Research), Children’s Social Care Innovation Programme: Round 2 final report, Department for Education, September 2020, pp. 20–21. Our analysis excludes evaluation of eight innovation projects which are outside the scope of our focus on Innovation Programme projects.

21 What Works for Children’s Social Care, Learning from innovation: Summaries of the Children’s Social Care Innovation Programme round 2 evaluations, July 2021, pp. 11–12.

Part Three

Building on learning from the Children’s Social Care Innovation Programme

3.1 In autumn 2018 the Department for Education (the Department) announced a new programme to further support three projects from the Children’s Social Care Innovation Programme (the Innovation Programme), which it considered from the evidence to be the most promising. By summer 2019 the Department had launched a supplementary scheme to broaden testing of a further three innovations. This part of the report sets out how learning from the Innovation Programme was used in the design of these successor programmes.

Two successor programmes

Strengthening Families, Protecting Children

3.2 The Department launched Strengthening Families, Protecting Children (Strengthening Families) in October 2018. It planned to invest up to £84 million over five years, from 2019-20 to 2023-24, to expand three projects that it judged to be successful in the Innovation Programme across 17 further local authorities.22 To date, it has committed £77.4 million to the projects with a further £4.1 million provided for evaluation and support. The innovation projects supported are:

Family Safeguarding is a redesign of children’s safeguarding teams. It brings together the professionals working with a family as one multidisciplinary team and aims to keep children safely within their families by identifying and meeting need. Hertfordshire County Council implemented the model with funding from Round One and in Round Two it was introduced in four more local authorities. Strengthening Families expands the model to six more local authorities. Project funding totals £41.8 million across Strengthening Families and earlier Innovation Programme funding rounds;

No Wrong Door is an integrated mental health, education and families service designed by North Yorkshire County Council for young people aged 12 to 25 who are either in care or on the edge of care or have recently moved to supported or independent accommodation. Project funding totals £26.6 million across Strengthening Families and earlier Innovation Programme funding rounds. Strengthening Families expands the model to five more local authorities; and

Family Valued looks to spread restorative practice across children’s services and the social work service. A key element is the expansion of the family group conferencing service, including for families experiencing domestic violence. Project funding totals £32.3 million across Strengthening Families and earlier Innovation Programme funding rounds. Started by Leeds City Council, and expanded in Round One, Strengthening Families expands the model to six more local authorities.

3.3 The Department intended to focus the testing of the three projects on local authorities that were seeing high and rising demand for children’s social care, enabling them to improve their practice and work more effectively with vulnerable children and their families. The Department developed metrics to assess demand and restricted the scheme to authorities rated as ‘Requires Improvement’ by Ofsted. The Department also let a contract for a new delivery partner for Strengthening Families to provide project support, including spreading learning between programme participants and out to the wider sector.

Supporting Families: Investing in Practice

3.4 The Department launched the Supporting Families: Investing in Practice (Supporting Families) programme in May 2019 for three projects with evidence of impact through the Innovation Programme but where the evidence was not yet strong enough to conclude with certainty that they work and in what conditions. The Department’s aim was to test the projects further in a wider group of local authorities between 2019-20 and 2020-21.23 The innovation projects supported are:

Mockingbird family model, led by The Fostering Network, is structured around providing foster carers with the kind of support and relationships an extended family provides. It aims to nurture relationships between children, young people and foster families, and to support them to build a resilient and caring community. Foster carers are grouped around a ‘hub’ which, led by a hub home carer and liaison worker, offers peer support, guidance, short breaks and sleepovers. Project funding totals £7.8 million across Supporting Families and earlier Innovation Programme funding rounds. Supporting Families is extending the model to seven new sites involving 10 local authorities;

Family Group Conferences are a way of bringing a family together to resolve family problems. It is being trialled with families to whom a letter of intent to initiate care proceedings has been issued, with the intention of increasing safe placement and reducing the duration of care proceedings. Project funding totals £6.5 million across Supporting Families and earlier Innovation Programme funding rounds. Round One funded a pilot in two local authorities. Supporting Families extends this to 21 local authorities. The project is led by the charity Daybreak; and

Family Drug and Alcohol Courts are an alternative to standard care proceedings in court for children who are at risk of significant harm as a result of parental substance misuse. Family Drug and Alcohol Courts adopt a problem-solving approach to respond to the problems that led to care proceedings. Set up in London in 2008 as a pilot, it has since been adopted by other local authorities. Project funding totals £11.9 million across Supporting Families and earlier Innovation Programme funding rounds. The project is led by the Centre for Justice Innovation.

3.5 In contrast to Strengthening Families, the Department invited expressions of interest from any local authority wishing to work with the scheme projects, running a competitive selection process in which candidates were assessed by the Department, What Works for Children’s Social Care and the original innovators of the three projects. The Department funded 58 authorities to take forward the three models in paragraph 3.4.

3.6 In addition to the Strengthening Families and Supporting Families programmes, the Department has provided further support for Innovation Programme projects beyond the initial £212.8 million:

In April 2020 the Department provided £13.7 million through an Innovation Programme Continuation Fund to allow 17 other Innovation Programme projects to continue work and publish learning.

In July 2021 the Department further provided £7.3 million through its COVID-19 Regional Recovery Fund to 27 projects to help local authorities explore adoption of Innovation Programme models.

Use of learning in project selection

3.7 At the time the Department selected projects for the successor programmes it had a range of evaluation evidence from the Innovation Programme including:

an Interim Learning Report from the Innovation Programme delivery partner (January 2016); and

evaluations for all Round One projects (November 2017).

3.8 When assessing candidate projects for Strengthening Families the Department considered their impact, as reported in evaluations, and the quality of the evaluations themselves, using a recognised academic framework. The Department considered whether the candidate projects displayed seven common elements the Department and the Innovation Programme delivery partner had identified as good practice in Round One. The Department could not show us evidence of a formal selection process but its documents suggest it considered a shortlist of five projects, which it narrowed down to three. For Supporting Families, internal documents show that the Department considered a shortlist of seven projects. To select three, it considered strength of evidence and impact, sector appetite for candidate projects, the capacity of the innovator to support roll-out and the projects’ fit with departmental policy priorities.

Developing improved evaluation

3.9 The Department developed measures for the outcomes it expected to see across the Strengthening Families programme. The Department’s business case stated the programme’s overall rationale as: “Supporting struggling local authorities to manage demand for care and to reduce costs.” Using information on the costs and savings reported from the Innovation Programme, it suggested that the Strengthening Families projects could pay for themselves through cost savings in six years. Based on the objectives in its business case, the Department expects that, by the end of the programme in 2023-24, participating local authorities will achieve a minimum average reduction of 10% in:

the looked-after children rate per 10,000 children;

the number of cases that progress to either child protection plans or care proceedings; and

spend on looked-after children.

The Department did not develop metrics of this kind for the Supporting Families programme.

3.10 The Department intended that both schemes would use more sophisticated evaluations than used in the Innovation Programme. The Department planned for the Strengthening Families evaluation to be as robust as possible through meeting the standards of evidence of What Works for Children’s Social Care, including those around sample size, use of reliable measures and comparability of groups. All three projects are using a ‘stepped wedge’ cluster randomised control trial, rolling out delivery across participating local authorities sequentially between 2019 and 2024. This approach allows for a comparison of results between cohorts in local authorities that have started the new interventions and those that have not, as well as a before-and-after comparison in each local area. The staggered implementation also helps the three ‘innovator’ local authorities to provide support to each cohort in turn, while minimising the impact on their own work.

3.11 Evaluation of Strengthening Families is being delivered by What Works for Children’s Social Care. It will monitor the reduction in demand for children’s services and will report on the cost-benefit ratio for each project as part of final evaluation reports, scheduled for publication in 2026 (Family Safeguarding and No Wrong Door) and 2027 (Family Valued). To date, it has published pilot evaluations of the initial roll-out in the first ‘trailblazer’ local authorities to trial each project. These reports focused on whether it was feasible to roll out projects that had been successful in the original ‘innovator’ local authorities more widely, in each case finding that the projects were being largely implemented as planned.24 The Department is monitoring progress, asking participating local authorities to provide monitoring data every six months, and has a tool to allow local authorities to report cost savings in a consistent way. It is too early for the Department to have collected sufficient data to indicate the impact of the programme to date.

3.12 The Department also intended to use randomised control trials on the Supporting Families programme, although it had to change the approach where it encountered practical and ethical barriers. The Department and What Works for Children’s Social Care developed a randomised control trial for the Family Group Conferencing project, which randomly allocates families willing to participate into either trial intervention or standard intervention groups. The Department rejected a randomised control trial for the Family Drug and Alcohol Courts project, however, following judicial concerns that randomisation of families in care proceedings could be subject to legal challenge. Plans for a randomised control trial for the Mockingbird project were also abandoned because of concerns about the practicalities and ethics of randomising children into foster placements. The Department and What Works for Children’s Social Care selected alternative approaches which would still allow for the interventions to be assessed against comparison groups. What Works for Children’s Social Care is overseeing the work on these evaluations, which it has contracted out to external providers. The programme evaluations are due for publication between October 2022 and May 2023, with interim reports updating the Department on progress of evaluation work having been received in 2021. The Department also monitors:

formal quarterly reporting covering service caseloads and outcomes across projects;

a range of monthly and weekly trackers covering key project milestones; and

other support for Innovation Programme projects.

22 The Department initially planned to support 18 local authorities but one subsequently withdrew.

23 The Department halted the project for six months from April 2020 due to the COVID-19 pandemic, with delivery subsequently expected to complete by June 2022.

24 What Works Centre for Children’s Social Care, Strengthening Families, Protecting Children: Pilot evaluation reports, June 2021, September 2021, November 2021.

Part Four

Overview of evaluation and next steps

4.1 It is government policy that all policies, programmes and projects should be subject to proportionate evaluation.25 In particular, in areas that are innovative or breaking new ground, there is a need for evidence to illustrate whether an intervention is working as intended. Evaluation should inform thinking throughout the policy cycle before, during and after implementation. This part of the report considers the themes in use of evaluation that can be drawn out of the innovation programmes covered in this report.

Strategy for innovation

4.2 Having a strategy can help ensure that the most pressing evaluation gaps are prioritised, planned and resourced. In December 2021, we reported that 10 of the 16 departments we surveyed, including the Department for Education (the Department), did not have an evaluation strategy that spans their whole department. However, the Department told us that it plans to develop one for publication in late 2022.26

Barriers to innovation in children’s social care

4.3 In our 2021 report Evaluating government spending, we identified a number of barriers to evaluation.27 These included lack of political engagement with evaluations, capacity concerns and a lack of incentives for departments to produce and use evaluation evidence, together with few adverse consequences for not doing so. More practically, barriers included a lack of the people and skills to undertake evaluation, limited access to administrative data and time limits on spending that forced departments to proceed with delivering policies without planning for evaluation.

4.4 What Works for Children’s Social Care, in setting its research priorities in 2020, stated that the “evidence base for ‘what works’ in children’s social care is sparse”.28 The Department aims to build a culture of evaluation in the sector. The Department published evaluations for more than 90% (by value) of the projects in the Children’s Social Care Innovation Programme (the Innovation Programme). As discussed in Part Two, while the majority of these evaluations were not sufficiently robust (for example, due to small sample sizes) the number of evaluations that were commissioned and published is noteworthy.

4.5 The Department and What Works for Children’s Social Care highlighted barriers to achieving robust evaluation in this sector, including:

the capacity and expertise to undertake evaluation in a sector where evaluation is not widespread and where there is a lack of familiarity with evaluation techniques;

perceived ethical challenges of withholding services from some families for the purposes of comparison with a new intervention, based on the assumption that the intervention being evaluated is positive. There are also some instances where evaluators cannot withhold a service to which a child and their family are entitled; and

obtaining comparator data from local authorities that are not receiving project funding.

4.6 Both the Department and What Works for Children’s Social Care told us that the emphasis placed on evaluation through the Innovation Programme has helped to build capacity and understanding to allow for more robust evaluations in the sector. The Department has been able to increase its ambition for project evaluations in more recent interventions through the Strengthening Families, Protecting Children and Supporting Families: Investing in Practice programmes. As discussed in Part Three, the Department is undertaking impact evaluations through randomised control trials and, where this is not appropriate, using quasi-experimental design to generate comparators.

Long-term outcomes and good practice

4.7 It is government policy that evaluation should inform thinking before, during and after implementation and that policies and programmes that involve a series of related sub-programmes must be subject to monitoring and evaluation during and after implementation.29

4.8 The Department required Innovation Programme projects to demonstrate how they would sustain benefits beyond the period for which they would be funded, and assessed whether they were on track to deliver such sustainable benefits while their funding was still ongoing. It has not, however, monitored or evaluated projects since the completion of the Innovation Programme to establish how far any benefits delivered during the Innovation Programme have been maintained and grown after grant funding stopped. It has also not assessed the wider spread and impact of projects, for example, in local authorities which the Department did not fund.

4.9 In 2019, in its inquiry into Transforming children’s services, the Committee of Public Accounts concluded that while local authorities share good practice through programmes established by the Department, it was not yet apparent whether this is effectively spreading high-quality and well-evidenced children’s social care practice.30 The Committee of Public Accounts recommended that the Department should set out what action it has taken to encourage the take-up of good practice in children’s social care across local authorities and how it will assess take-up.

4.10 The Department appointed a delivery partner for the Innovation Programme from 2014 to 2020, which provided access to good practice through a website and by running events for innovators. Since March 2020, the Department has provided evaluation and other evidence from the Innovation Programme through its own gov.uk website. In 2020, the Department let a new contract for a delivery partner for the Strengthening Families, Protecting Children programme. This delivery partner disseminates learning through a regular newsletter and online journal and, together with the Department, is planning a two-day online learning event in summer 2022. More broadly, the Department began funding what would become What Works for Children’s Social Care in 2017 to generate, collate and make accessible the best evidence for practitioners, policy-makers and practice leaders. The Department has also provided funding through ‘Partners in Practice’, a peer support programme between local authorities. While the Department and its delivery partners have promoted learning and good practice emerging from innovation projects, the Department has not assessed the extent to which good practice is adopted.

Future developments

4.11 The Department told us that it is not yet able to set out future plans for innovation in children’s social care until the existing programmes have completed. In January 2021, the Department announced an independent review of the children’s social care system to make sure children and young people get the support they need. In its report, The Case for Change, the review found:

“The Innovation Programme and subsequent pots of funding for scaling and spreading good practice have had an important role in recent years of supporting stronger practice and giving exposure to and evaluating different ways of working […] there is a limit to this kind of scale-and-spread strategy in the absence of changes to the most fundamental drivers and forces in the system.”

The Department told us that it has shared evidence from its innovation programmes with the independent review, published in May 2022.31

25 HM Treasury, Magenta Book: Central Government guidance on evaluation, March 2020.

26 Comptroller and Auditor General, Evaluating government spending, Session 2021-22, HC 860, National Audit Office, December 2021.

27 See footnote 26.

28 What Works for Children’s Social Care, What matters for what works? Setting research priorities for what works for children’s social care, April 2020.

29 HM Treasury, Magenta Book: Central Government guidance on evaluation, March 2020; HM Treasury, Green Book: Central Government guidance on appraisal and evaluation, 2020.

30 HC Committee of Public Accounts, Transforming children’s services, Eighty-eighth Report of Session 2017–2019, HC 1741, March 2019.

31 The independent review of children’s social care, The Case for Change, October 2021; The independent review of children’s social care, Final report, May 2022.

Appendix One

Our audit approach

Scope

1 This report examined how the Department for Education (the Department) has used evaluation and other evidence in its innovation programmes for children’s social care. It was prompted by a request to evaluate a specific project. The following programmes were within the scope of the report:

Children’s Social Care Innovation Programme.

Strengthening Families, Protecting Children.

Supporting Families: Investing in Practice.

Innovation Programme Transition Fund and Continuation Fund.

COVID-19 Regional Recovery and Building Back Better Fund (Strand 3: scale and spread of innovation projects).

We carried out the fieldwork between January and March 2022.

2 The report covers:

background to the innovation programmes and progress to date; and

use of evaluation and other evidence in investment decisions.

Methods

3 In examining these issues, we drew on a variety of evidence sources.

4 We interviewed staff from the Department. The people we interviewed included those responsible for policy related to the innovation programmes, for operation and financial management of the programmes on a day-to-day basis and for programme evaluation.

5 We reviewed published and unpublished information from the Department. These documents included material relating to:

business cases and briefing papers;

minutes and papers associated with the governance of both the programmes and their evaluations;

evaluation reports for the programmes and component projects; and

documents associated with the selections of projects and bidders for funding.

6 We analysed management information from the Department. The information included data showing:

the funding provided to projects, how much has been spent and the funding remaining to be spent.

7 We interviewed staff from What Works for Children’s Social Care. We used this interview to understand its role in overseeing and evaluating the innovations programmes within scope.

Design and Production by NAO Communications Team
DP Ref: 011776-001