Risk of Bias in Non-Randomised Studies

Crowd of people walking along the pedestrian area. 700 pixels

A key component of a systematic review is the development of conclusions that take into consideration the quality of the synthesised data or evidence. This requires studies to be assessed for Risk of Bias (sometimes referred to as ‘study quality’). Tools to assess bias in Randomised Controlled Trials typically focus on the adequacy and concealment of the random allocation of the intervention. However, Non-Randomised Studies (NRS) are often used in public health reviews and include several additional sources of potential bias. 

Cochrane has recently developed a tool to assess Risk of Bias in NRS of interventions (ROBINS-i, previously ACROBAT-NRSi). This tool has been developed using mainly clinical examples. We have piloted the ROBINS-i tool to assess how easily this new tool can be applied to complex healthy public policy intervention studies using a group of housing improvement studies.

The aim of this work was to help establish ease of use in applying the tool beyond the clinical realm, and use lack of consensus across those experienced in critical appraisal of NRS to identify and articulate any emergent issues in face validity and interpretation of the tool.

Following this work we concluded that while the ROBINS-i tool is a conceptually rigorous tool it is difficult to apply to natural experiments of community based public health interventions. In particular we found it difficult to determine the Effect of Interest (Intention to Treat or per protocol) where the timing of the “intervention status” was unclear.

A paper reporting the findings of this work is available here.

MRC/CSO Social and Public Health Sciences Unit logo 800 wide