August 2021
Summary
Social workers make potentially life-changing decisions every day. These range from, choosing whether to accept a referral, undertaking a child protection investigation, pursuing care proceedings, and making judgements about the likelihood of future significant harm.
But people are very poor at predicting the future. In part, these difficulties are caused by our susceptibility to cognitive bias (systematic errors in thinking that typically occur when we interpret information). A recent review of the literature found that there are currently no well-evidenced interventions to help mitigate the effects of cognitive bias in social work. In other fields, such as politics and economics, various interventions have been developed and found to significantly improve prediction of outcomes (forecasting abilities).
This study examines the impact of a Checklist intervention through a randomised controlled trial, exploring the differences it made in relation to forecasting abilities and confirmation bias.
Objectives
The Checklist Intervention aimed to help social workers avoid some of the negative effects of confirmation bias and improve forecasting accuracy using a low-cost and easy to administer tool.
The trials aimed to evaluate the effectiveness of the Checklist interventions on the difference it made in relation to forecasting abilities and a direct measure of confirmation bias.
Specifically, it aimed to answer the following questions:
- What is the impact of the checklist intervention on forecasting accuracy amongst social workers?
- What is the impact of the checklist intervention on confirmation bias amongst social workers?
- Is there a relationship between social workers’ forecasting accuracy and their level of confirmation bias?
- Is there a relationship between social workers’ forecasting accuracy and i) age-group, ii) gender, and iii) length of post-qualifying experience?
How we went about it
This study involved a randomised controlled trial (RCT), where a total of 87 social workers and one student social worker took part in an online survey. Each participant read two baseline case studies and answered four questions for each case study about the likelihood of different outcomes. Participants were then randomly allocated to the control group or the intervention group. Participants in the control group were asked to complete two endline case studies and the Wason Selection Task before accessing the checklist intervention. Participants in the intervention group were asked to complete the checklist intervention first, before completing the two Endline case studies, and the Wason Selection Task.
Key findings
To assess whether the Checklist intervention had an impact on social workers forecasting abilities, multiple regression was used on Endline Brier scores. There was a post-intervention reduction in Brier scores for participants in the intervention group (indicating improved forecasting accuracy). However, this difference between the intervention and control groups was small and not statistically significant. Thus, the checklist intervention did not have a significant impact on forecasting accuracy. Multiple regression was also used to determine the impact of the checklist intervention on confirmation bias. Findings indicated the checklist intervention did not have a significant impact on confirmation bias. The personal and professional characteristics of the participants were not associated with forecasting accuracy or confirmation bias.
Notably, Checklist interventions are typically designed to be used within actual decision-making settings and are evaluated accordingly. In this study, because of the Covid-19 pandemic, we had to design an intervention that could be used online. This limits our ability to generalise from these findings, to what might happen if the same (or a similar) checklist were used in real-life decision-making environments. This is because any similar tools would most likely be used differently in practice, for example in discussion with a supervisor in a supervision session. Other limitations of the study include the lack of a pre-intervention measure of confirmation bias, and the smaller-than-intended sample size, lowering the likelihood of detecting significant effects (or non-effects) of the intervention.
Implications
Overall, we found no evidence that the checklist intervention made a significant difference in relation to forecasting accuracy or confirmation bias. These results, especially when considered alongside those of a previous study, suggest that interventions to improve forecasting accuracy (or mitigate cognitive bias) in social work would need to be much more in-depth than the relatively brief online interventions we have tested here and previously. For example, through longer training courses.