Other types of monitoring
Process
What is this?
In certain programmes, you may be concerned
about how a provider carries out the work you fund (the
process) as well as what it achieves in doing so (the
outcomes)[1]. To this end, you may
wish to monitor the provider’s adherence to the agreed process.
Advice
For example, if you are funding a TSO to work
with young offenders in the community, you may express the
objectives as an outcome (such as reduction in re-offending).
However, in certain circumstances, you may also wish to specify
which interventions the TSO should use in working with the young
people. This could be because there is solid research
evidence that a certain methodology is effective and/or because it
is known to be safe. You would usually include the
requirement to include this methodology in the financial
agreement.
If you require assurance that a specified
methodology is being used, you can use a number of different
sources. Not all involve monitoring. For example, you
may place some reliance on the fact that the staff have appropriate
professional qualifications and membership. Or the TSO may
subscribe to a licensing arrangement for the methodology
concerned. Alternatively, the service may be subject to
external inspections (by an inspectorate such as Ofsted or HM
Inspectorate of Probation). You can take assurance from their
reports. Similarly, you can study the methodologies used by
the provider if you visit its premises.
But, if you decide that, despite taking the
above steps, you still want to monitor the provider’s
adherence to a specified methodology, you and the provider will
have to agree what you are going to monitor. To do this, you
need to select a small number of the important steps in the
process. You will then need to express these ‘standards’ in
Smart terms (in the same way as the objectives – see above).
You will then need to agree with the provider that it will supply
you with data on its adherence to these standards.
This kind of monitoring – of process – should
be avoided where possible. It is burdensome on the provider –
and thus costly (for the funder, ultimately). It deters the
TSO from innovation – because it is tied to a single, specified
methodology. And it undermines the professional confidence of
the TSO and its staff. You should introduce this kind of
monitoring, therefore, only if risk analysis shows it to be a
requirement. This is more likely for particularly vulnerable
beneficiaries or dangerous processes.
Policy evaluation
What is this?
Funders have to evaluate their
policies[2]. To support an
evaluation, they need information about the programme and the
environment it operates in – often over time, so they can detect
and explore the impact of the programme. Funders may gather
some of this information from the providers of the programme that
is being evaluated. For providers, good monitoring and
reporting help them to showcase the work they are doing as well as
learn and develop.
Advice
The impacts of a policy can be economic,
environmental and social and may require a range of evaluation
tools. Some of these tools may be more appropriate than
others when evaluating policies with third sector involvement, for
example Social Return On Investment (SROI)[3].
Funders should work with their evaluation
experts (in-house analysts, for example) to:
- Develop their plan for evaluating a
programme
- Draw up a list of the information they will need for this
evaluation
- Work out how this information is going to be
collected.
If you are going to use a provider as the
source of some information for evaluation, be aware of the
additional workload this will place on a provider. You will
need to pay the provider for collecting it on your behalf.
An example
of how to monitor in practice for policy evaluation is set out
below.
Policy
development
What is this?
Funders have to keep their policies up to
date. To support such policy development and maintenance,
they need information about the programme and the environment in
operates in – often over time, so they can detect and explore
changes in the environment. Funders may gather some of this
information from the providers of the programme.
Advice
Funders should:
- Draw up a list of the information they will
need for this policy development
- Work out how this information is going to be
collected.
If you are going to use a provider as the
source of some information for policy development, be aware of the
additional workload this will place on a provider. You will
need to pay the provider for collecting it on your behalf.
An example
of how to monitor in practice for policy development is set out
below.
Parliamentary
accountability
What is this?
Information that the funder may need to give
Parliament about the programme.
Advice
Ministers and Accounting Officers are
accountable to Parliament and, ultimately, to the public for their
programmes. This means funders have to be prepared to answer
reasonable questions from Parliament or the public[4]. For example, it may be
reasonable to answer a question from a Member of Parliament about
how many people have benefited from a programme broken down by
simple geography and ethnic origin. But a funder would not
normally be expected to give a lot of information about
beneficiaries that is not directly relevant to the objectives and
management of the programme.
Funders need to make sensible judgements about
what information they will collect for accountability. You
should be able to answer many questions by drawing on information
collected for other purposes. This means the question can be
answered without imposing any burden of collecting additional
information. You may also be able to draw on information the
provider collects for its own purposes.
However, in certain cases, you may need to
collect information specifically for accountability. To
prepare for this, you may collect routinely information that may be
needed in the future. But you should not use accountability
as an excuse to collect a lot of information that is not really
needed.
An example
of how to monitor in practice for accountability is set out
below.
Example 4: combining
monitoring for accountability, evaluation and policy
The last three types of monitoring –
accountability, evaluation and policy – are different in nature
from the others. Those are concerned with the effective
management of the programme. The last three are concerned
with the funder’s wider responsibilities. In particular:
- Accountability monitoring is designed to help
a funder account for its work to Parliament and/or the public
- Evaluation monitoring is designed to help the
funder learn lessons from the programme in action that will be
helpful improve the funder’s work in the future
- Policy monitoring is designed to give the
funder information about the policy environment that will help it
to develop better policy.
Funders need to recognise that requiring
providers to supply these types of information is a requirement on
top of the need for effective programme management. You must
agree the requirement for these types of information with providers
beforehand. You should compare monitoring - as a method of
collecting information for policy work - with other methods (such
as surveys) before deciding to add to the monitoring requirement in
this respect.
Example
You are responsible for a programme to check that old people’s
homes are secure against burglaries. You have decided to
procure the service on a regional basis. In three of the
regions, third sector organisations win the work.
Before drawing up the invitation to tender for the procurement
process, you should consider what information you are likely to
need for:
- Accountability
- Evaluation
- Policy.
Once you have a list of these items of
information, you should decide how you are going to collect
them. In broad terms, you have three choices:
- Your own organisation may
collect the information
- Fund a research company to collect them for
you
- Fund your providers to collect them for
you.
You can appraise these options on value for
money grounds. A mix of these is also possible. Once
you have decided, any requirement for providers to collect this
information for you needs to be included in the invitation to
tender or apply for a grant or grant-in-aid. The funder
should then include this in the price it quotes you in its
proposal.
Notes
- Outcomes are the real-life
improvements brought about by the programme.
Outcomes result from the outputs of the
programme. Outputs are the product of the provider.
Outputs result from processes. Processes are the
activity of the provider. Processes result from inputs;
inputs are the resources put in the programme
- Government Social Research Unit,
Guidance Notes on Policy Evaluation (aka the ‘Magenta
Book’), HM Treasury, 2007
- Social Return on Investment (SROI)
is an approach to measurement and value that can be used across the
public, private and third sectors. Developed from
cost-benefit analysis and social accounting, SROI uses economic
valuation to make visible a far greater range of social,
environmental and economic costs and benefits than conventional
analyses. The Office of the Third Sector is developing
guidance on SROI.
- Cabinet Office,
Ministers’ Correspondence with Members of
Parliament, Cabinet Office