The failures in monitoring and evaluation of schemes, that I started touching on last week, can often be sourced back to before the scheme has been defined. The aims and objectives of the scheme are usually easily determined (although whether or not they are determined well or in line with policy is another thing entirely). But what is often not thought about in any great detail is the logic behind the scheme, and how the scheme will result in the impacts that are anticipated.

There is plenty of guidance on how to come up with a good scheme logic map. This simply sets out the steps that are taken from the aims of the scheme, to the inputs the scheme provides, to the outputs that result from the scheme. By considering each in turn, it is possible to identify what the outcomes of the scheme will be, and its anticipated impacts.

For monitoring and evaluation of schemes, this is important in two ways. First, it helps us to define what impacts and changes we are likely to see, and the appropriate indicator that will determine whether the impact has not just happened, but whether there is the early signs of it happening. Sometimes, indicators don’t tell us that the change has happened, it just tells us that something may be happening, that is worthwhile exploring further.

The second way it is important is that it helps to define how we should be monitoring. Many data collection methods can give you a number for an indicator (e.g. traffic counts), but good monitoring and evaluation shows whether or not your underlying scheme logic is sound.

This is where much in the way of monitoring and evaluation fails. It focusses too much on the outcome, and not on demonstrating the scheme logic. This is just as important, as it both explains what has happened and whether the intervention has achieved through the scheme logic being true or not.

To demonstrate this, lets take a simple example. You are looking to install a Low Traffic Neighbourhood as a way of reducing the amount of traffic through a particular neighbourhood. You determine that by re-rerouting traffic and closing roads, this will make the route less attractive to through traffic, and accordingly traffic levels within the area will reduce (I know the logic behind LTNs is usually more complex than this, but I am keeping this simple for brevity).

Using an approach focussing on outcomes, you would determine through traffic surveys within the neighbourhood whether or not you have achieved your outcome. If you have, then your logic must be true in that case. If not, then either you have implemented your scheme wrong, or your scheme logic is wrong. Which of these is true then becomes a value judgement of the officers (or members) who delivered the scheme. And not one that is evidence based.

Lets then think about a monitoring regime that would test the scheme logic. In addition to monitoring the traffic levels on the roads in the area, you may also consider methods that monitor the routes that traffic takes – maybe using the Google Journey Time API, or also monitor levels on alternative routes. With this, you are now testing whether your scheme logic is true, as well as understanding the impact of the scheme.

But doing so is often a challenge. How can we easily make the link between the challenge, scheme aims, the objectives, and the indicators of change, so that we can start to think about how what we measure will also test the scheme logic? Luckily, that is something that I have been working on a template for, that can be applied for any scheme. And here it is.

And here is what it looks like in a worked example.

This is a basic scheme logic that can be applied quickly to any level of scheme. By thinking through the scheme challenges and aims in a logical manner, and in order from challenge to KPI, the result is not just robust KPIs, but KPIs based upon an understanding of the whole scheme logic, and therefore being ones that test the logic of the scheme.

This is intended as a basic tool to overcome a basic problem, and to encourage practitioners to develop metrics and monitoring that tests the logic of the scheme, and not just understand its outcomes. Feel free to take this and use this as part of your own schemes, and let me know how you get on with it.