Today, the What Works Centre published its first research protocols, which can be found on our research projects page. This is an important milestone for us, as a part of our commitment to starting as we mean to go on, and we thought it might be useful to talk about why.

It might seem strange to publish a detailed description of research we haven’t done yet, but this is an important part of the centre’s process. By describing our research in as much detail as possible before it starts, we can keep the sector informed about what we’re doing, as well as other researchers who can better scrutinise what we’re up to. If, at the end of the project, we find something positive, this description will also help other practitioners and researchers to make use of the findings.

Just as importantly, though, publishing our research questions, and how we’re going to try and answer them, before the study starts, keeps us honest. By nailing our colours to the mast in this way, we’re not able to change our minds at the end in a way that tells a better story.

If this seems like a strange thing to guard against, it shouldn’t. Researchers face a challenge that’s sometimes called the “garden of forking paths” – there are a number of decisions we can take about how to conduct our analysis, each of which might given a slightly different answer. The way that the statistical tests we use are applied, we have a 5% false positive rate – which means that 5% of the time, we’ll “find” a result that isn’t there. This is a pretty high standard of proof, and lets us run research projects that are both robust and practical.

However, if we run 20 tests on the same sample, we’d expect to find some evidence that something “works” even if there’s no actual effect there. This is a problem, and one which is exacerbated by a lack of transparency – because without pre-registration, researchers could run 20 tests, and only present the results of the ones that confirmed their hypotheses. Pre-registration prevents this from happening, and is why we also make an independent record of what we’re going to test on the Open Science Framework.

Pre-registration isn’t intended as a strait-jacket – it doesn’t prevent researchers from conducting more exploratory analyses, and learning more from the rich complexity of the data – but it does mean that we have to be clear, from the start, about what we’re trying to learn, and what our main hypotheses are. Hopefully, what it also means is that our research will be more widely replicable, more transparent, and more useful to practitioners who put their trust in it.