When designing research, and particularly randomised controlled trials, we ask ourselves many questions. Who should be in the trial? Why might the intervention work, or not work? What is the intervention, really? Where will it be located? It is much less common to think about when.
And yet, the question of when to do a piece of research is just as critical as these other questions. There are those who want to rush headlong into things – to get research into the field, and to learn as much as possible as fast as possible – I am certainly guilty of this. There are others, who are much more cautious – wanting to have nailed down, and manualised, each element of an intervention before it is subjected to the bright light of an impact evaluation.
Everything, as they say, in moderation.
When we move too quickly into an impact evaluation, we run some familiar risks. The intervention may not be stable yet – so it’s not clear what we’re actually evaluating. It may not yet have been optimised – even a straightforward idea can be implemented in myriad ways, and if we ask developers to make decisions too quickly, they might pick an option that doesn’t help. Having longer to develop interventions and try a few ideas before an impact evaluation makes it more likely that it will succeed. Next, an intervention might need time to ‘bed-in – a complex change to a system is unlikely to be fully adopted, with all the wrinkles ironed out, within the first week. Evaluating too soon would mean concluding that an intervention doesn’t work.
What about being too cautious with research?
There are also risks inherent to this approach. First, and most obviously, a large proportion of all interventions don’t have the desired effects, or the effects are smaller than anticipated. As long as we continue to run them without testing their impact, we cannot learn this, and risk doing a disservice to the people the intervention is designed to serve. Second, interventions could get stuck in a rut – failing to innovate without data to support that innovation or spur new ideas; or they could innovate in unhelpful ways.
One option that I hadn’t previously considered is that over time interventions could become inevaluable, as they either become too prevalent or too diffuse.
This was a challenge that we faced in our evaluation of Signs of Safety, published last month. Our evaluation was of the nine local authorities who rolled out Signs of Safety as a part of the English Innovation Programme. For various reasons, we used a quasi-experimental impact evaluation approach to try and work out how much better local authorities using Signs of Safety were doing, compared to how they’d have been doing if they hadn’t. The evaluation report finds modest impacts at best, with no robust, statistically significant positive impacts detected, and some evidence of a reduction in the use of Kinship Care by local authorities with Signs of Safety.
One of the biggest challenges of this evaluation was, ironically, just how successful Signs of Safety had been. According to a 2017 survey by Mary Baginsky at King’s College London, over one third of English local authorities are already using Signs of Safety exclusively and one third are using parts of it, with more still making use of some similar practice model. This makes it hard to find a counterfactual group – the people or places that we compare with those in the intervention group – that is good enough for a reliable evaluation.
Our evaluation of Signs of Safety overcame these challenges, just about, and we were aided by information on the quality of delivery provided by the delivery organisation; but for other whole system changes, it is already too late. Reclaiming Social Work, the model of practice developed in Hackney, is effectively inevaluable, both because nowhere is ‘doing it’, and because it has in some senses entered the water supply of social care, and everywhere is ‘doing it to some extent’.
If we are to move to a more evidence-based, and particularly impact evidence-based system, we need to calibrate the timings of our impact evaluations more carefully, and be mindful not to go too soon – or too late.