ROB-ME: a tool for assessing risk of bias due to missing evidence in systematic reviews with meta-analysis
BMJ 2023; 383 doi: https://doi.org/10.1136/bmj-2023-076754 (Published 20 November 2023) Cite this as: BMJ 2023;383:e076754Linked editorial
Will the ROB-ME checklist prevent omission bias in meta-analyses?
- Matthew J Page
, senior research fellow1, - Jonathan A C Sterne, professor2 3 4,
- Isabelle Boutron, professor5,
- Asbjørn Hróbjartsson, professor6 7,
- Jamie J Kirkham, professor8,
- Tianjing Li, associate professor9,
- Andreas Lundh, associate professor6 7 10,
- Evan Mayo-Wilson, associate professor11,
- Joanne E McKenzie, professor1,
- Lesley A Stewart, professor12,
- Alex J Sutton, professor13,
- Lisa Bero, professor14,
- Adam G Dunn, professor15,
- Kerry Dwan, senior lecturer16,
- Roy G Elbers, senior researcher17,
- Raju Kanukula, doctoral student1,
- Joerg J Meerpohl, professor18 19,
- Erick H Turner, professor20 21,
- Julian P T Higgins, professor2 3
- 1Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC 3004, Australia
- 2Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
- 3National Institute for Health and Care Research Bristol Biomedical Research Centre, Bristol, UK
- 4Health Data Research UK South-West, Bristol, UK
- 5Université de Paris Cité and Université Sorbonne Paris Nord, Inserm, INRAE, Paris, France
- 6Centre for Evidence-Based Medicine Odense and Cochrane Denmark, Department of Clinical Research, University of Southern Denmark, Odense, Denmark
- 7Open Patient data Exploratory Network, Odense University Hospital, Odense, Denmark
- 8Centre for Biostatistics, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
- 9Department of Ophthalmology, School of Medicine, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
- 10Department of Respiratory Medicine and Infectious Diseases, Copenhagen University Hospital, Copenhagen, Denmark
- 11Department of Epidemiology, UNC Gillings School of Global Public Health, Chapel Hill, NC, USA
- 12Centre for Reviews and Dissemination, University of York, York, UK
- 13Department of Population Health Sciences, University of Leicester, Leicester, UK
- 14Center for Bioethics and Humanities, University of Colorado, Aurora, CO, USA
- 15Biomedical Informatics and Digital Health, School of Medical Sciences, University of Sydney, Sydney, NSW, Australia
- 16Liverpool School of Tropical Medicine, Liverpool, UK
- 17Department of General Practice, Intellectual Disability Medicine, Erasmus MC, University Medical Centre Rotterdam, Rotterdam, Netherlands
- 18Institute for Evidence in Medicine, Medical Centre and Faculty of Medicine, University of Freiburg, Freiburg, Germany
- 19Cochrane Germany, Cochrane Germany Foundation, Freiburg, Germany
- 20Department of Psychiatry, Oregon Health and Science University, Portland, OR, USA
- 21Behavioral Health and Neurosciences Division, Veterans Affairs Portland Health Care System, Portland, OR, USA
- Correspondence to: M Page matthew.page{at}monash.edu
- Accepted 13 September 2023
A key feature of systematic reviews of quantitative research is the attempt to identify all studies that meet the review inclusion criteria and to include relevant data from all such studies in meta-analyses. This goal is compromised when reporting of primary studies is influenced by the P value, magnitude, or direction of study results.1 These factors might influence whether a study is published at all (selective non-publication of studies or publication bias),23 the speed at which a study report is published (time lag bias),4 or type of journal (indexed or not) in which a study report is published (location bias),5 each of which can lead to studies missing from meta-analyses. The P value, magnitude, or direction of the study results might also influence whether, or how completely, particular results are reported (selective non-reporting of study results or outcome reporting bias),6 leading to results missing from meta-analyses even when the study has been identified. The term “reporting bias” has often been used to describe such selective dissemination of evidence, but here we use the term “non-reporting bias” to emphasise the non-availability of evidence.7
We present some examples of non-reporting bias to explain the concepts above. Suppose that after conducting a randomised trial comparing glucocorticoid injection with placebo for shoulder pain, investigators find higher shoulder strength in participants receiving glucocorticoid injections, but find no difference in pain intensity and function between groups. The investigators might never submit a study report for publication, because the observed results conflict with their prior hypotheses about the benefits of glucocorticoid injection, or because they assume that journal editors will not be interested in publishing the paper. This example demonstrates selective non-publication of studies (ie, publication bias). Alternatively, suppose that a study report is published but results for pain intensity and function are omitted entirely or presented incompletely; for example, a statement that pain and function scores were “not different between groups,” without summary statistics, effect estimates, or measures of precision. This example demonstrates selective non-reporting of study results (ie, outcome reporting bias).
A meta-analysis result will be biased when the available evidence (from studies or results) differs systematically from the missing evidence. However, existing methods have limited ability to determine which meta-analyses are at risk of bias due to missing evidence.89 For example, several tools have been developed to help users assess whether selective non-reporting of study results has occurred, but not its impact on a meta-analysis. Furthermore, current methods tend to focus on only one type of non-reporting bias (only selective non-publication of studies, or only selective non-reporting of study results). This limitation of existing methods leaves users to decide their own approach for combining the risks of each type of bias into an overall judgment of risk of bias in the meta-analysis result; likely leading to inconsistency in judgments. In this paper, we describe the ROB-ME (risk of bias due to missing evidence) tool, the first structured approach for assessing the risk of bias that arises when entire studies, or particular results within studies, are missing from a meta-analysis because of the P value, magnitude, or direction of the study results.
Summary points
The reporting of primary studies or results might be influenced by the P value, magnitude, or direction of the study results; this influence can lead to bias in a meta-analysis, because the available evidence (from studies or results) differs systematically from the missing evidence
Existing methods mostly help users assess whether selective non-publication of studies or selective non-reporting of study results has occurred, but not its impact on a meta-analysis, which leaves users to decide their own approach for combining the risks of each type of bias into an overall judgment of risk of bias in a meta-analysis result
The ROB-ME (risk of bias due to missing evidence) tool is a structured approach for assessing the risk of bias that arises when entire studies, or particular results within studies, are missing from a meta-analysis because of the P value, magnitude, or direction of the study results
The tool consists of three preliminary steps: select and define which meta-analyses will be assessed, determine which studies meeting the inclusion criteria for the meta-analyses have missing results, and consider the potential for missing studies across the review; these steps inform an assessment of risk of bias due to missing evidence in a particular meta-analysis result
The tool is anticipated to help authors and users of systematic reviews identify meta-analyses at high risk of bias and interpret results appropriately
Development of the ROB-ME tool
We followed the framework for developing risk of bias tools recommended by Whiting et al.10 A core group (MJP, JACS, and JPTH) coordinated development of the tool, which included assembling the team of collaborators, preparing meeting materials, and leading the drafting and revising of the tool. Preliminary work to inform development of the tool included a cross sectional study of how selective non-reporting of study results was assessed in Cochrane reviews,11 a systematic review of scales, checklists and domain based tools for assessing risk of non-reporting biases in studies and meta-analyses of studies,8 and a non-systematic review of the empirical evidence of non-reporting biases.5
Informed by the preliminary work, the core group developed an initial proposal for a new tool for assessing risk of bias due to missing evidence in a meta-analysis result and presented it at a development meeting in April 2017. Seventeen contributors with expertise in the empirical evidence of non-reporting biases, graphical and statistical approaches to assess non-reporting biases, and methods for identifying and accessing trial protocols, trials register entries, and information submitted to regulators (eg, clinical study reports) attended the meeting. Through a series of presentations and facilitated discussion sessions, meeting participants agreed on the scope, structure, and content to be addressed by the new tool and identified topics for further consideration.
Following the development meeting, the lead author (MJP) prepared initial drafts of the ROB-ME tool and discussed them with meeting participants via video conferences in August and November 2017. The core group developed the tool, the assessment framework underpinning it and accompanying guidance further between 2018 and 2020 while drafting a chapter on the topic for the 2019 edition of the Cochrane Handbook for Systematic Reviews of Interventions.7 Additional edits were made in response to feedback received on a draft of the tool presented at the 2018 Cochrane Colloquium12 and a draft was sent to all coauthors of this paper. A preliminary version of the tool (template and detailed guidance) was uploaded to https://www.riskofbias.info/ and presented in a webinar in October 2020, at which systematic reviewers were invited to provide feedback via a template form seeking views on each component of the tool. Twelve systematic reviewers subsequently provided written feedback via the template form or by commenting directly on the tool template; six had piloted the tool on a meta-analysis they were conducting before providing feedback. All systematic reviewers viewed the tool favourably, although some edits to the instructions and wording of questions (but not to the structure of the tool) were suggested. This written feedback, along with verbal feedback received from attendees of seven webinars delivered throughout 2021 and 2022 was considered by the core team, who revised the tool by amending the wording to improve clarity. The final version was sent to all coauthors for approval.
The ROB-ME tool
The full ROB-ME tool is available at https://www.riskofbias.info/. Four worked examples of applying the tool are provided in appendices 1-4 of the supplement.
Terminology
Throughout this article, we use the phrase “study outcome” to refer to an outcome measurement collected on, or by, participants in a study; a measurement could be continuous or non-continuous (eg, a binary event or rate). We use the phrase “study result” to describe the combination of a point estimate and a measure of its precision (or the summary statistics required to calculate these) for a particular study outcome.713 An example of a study outcome might be the pain at the end of eight weeks of treatment, measured using a 100 point visual analogue scale. A corresponding study result for this outcome might be an estimated difference in mean pain scores between intervention groups with 95% confidence interval.
Scope of the tool
The ROB-ME tool is designed for authors or users of systematic reviews to assess risk of bias due to missing evidence in a pairwise meta-analysis of the effect of interventions. It can be applied regardless of the number and types of studies with results available for inclusion in the meta-analysis, including in cases where only one of the studies identified has results available. The tool is not designed for assessing risk of bias due to missing evidence in a network meta-analysis; a tool for this purpose (ROB-MEN) is available and described elsewhere.14
The ROB-ME tool is not intended to examine a related source of bias, which we describe as the risk of bias in selection of the reported result. This bias arises when an available study result was selected for reporting by the study authors from among multiple measurements or analyses, on the basis of the P value, magnitude, or direction of these multiple results.15 For example, study investigators might measure pain using two measurement instruments, yet report results only for the instrument that yielded a significant effect estimate. In this circumstance, a study result is available for inclusion in a meta-analysis of pain scores, although the study result is at high risk of bias because of the way it was selected for reporting. Risk of bias in selection of the reported result is considered in tools designed to assess risk of bias in a study result (such as the RoB 2 tool for assessing risk of bias in randomised trials15 and the ROBINS-I tool for assessing risk of bias in non-randomised studies of interventions16). By contrast, ROB-ME is used to assess risk of bias in a meta-analysis result and looks at the risk of bias arising from omission of results from one or more studies from the meta-analysis. ROB-ME is therefore not designed to replace the assessment of risk of bias in selection of the reported result in a study, and so users assessing that source of bias should continue to apply RoB 2 or ROBINS-I as appropriate, in addition to ROB-ME. Table 1 specifies which risk-of-bias tool to use when confronted with different reporting scenarios.
Tools needed to assess risk of bias in different reporting scenarios. Note that not all scenarios represent a high risk of bias
How to conduct ROB-ME assessments
Application of the ROB-ME tool to a systematic review consists of four steps (fig 1). Firstly, select and define which meta-analyses will be assessed for risk of bias due to missing evidence. Secondly, determine which studies meeting the inclusion criteria for the meta-analyses have missing results and thus cannot contribute to the meta-analyses. Thirdly, consider the potential for missing studies across the systematic review. And finally, assess risk of bias due to missing evidence in each meta-analysis.
Summary of the process of assessing risk of bias due to missing evidence in meta-analyses
We recommend that the tool is completed by at least two users independently, with any discrepancies resolved via discussion, or adjudication from another user. We recommend that step 1 is conducted at the protocol stage, step 2 during data collection or when assessing risk of bias in the results of included studies, and steps 3 and 4 after generating meta-analysis results, but before assessing certainty in the body of evidence (eg, via GRADE (grading of recommendations assessment, development and evaluation)17).
Planning the risk-of-bias assessment
Assessing risk of bias due to missing evidence in all meta-analyses in a systematic review might not be possible (eg, when resources are limited). In such cases, users should select which meta-analyses to assess for risk of bias based on which outcomes are most important for decision making (step 1). Such outcomes tend to be those selected for inclusion in summary of findings tables.18 Ideally, users should prespecify the meta-analyses they intend to assess for risk of bias and indicate the population, intervention, comparator, and outcome (PICO) for each meta-analysis (ie, define the question that each meta-analysis aims to answer).719 Users should also seek to define fully the types of studies and results that are eligible for inclusion in each meta-analysis to be assessed for risk of bias: doing so helps clarify which results are missing.
Evidence of selective non-reporting of results should then be gathered for each study that is eligible for inclusion in the meta-analyses selected for ROB-ME assessments (step 2). Users should start by assembling available sources of information about each eligible study. These sources might include the trials register entry (eg, at ClinicalTrials.gov), study protocol, statistical analysis plan, reports of results of the study (eg, journal article, clinical study report), or information obtained directly from the study authors or sponsor (eg, data files supplied). If study plans are available (eg, in a trials register entry,20 study protocol, or statistical analysis plan), then details of prespecified outcomes should be compared with other sources of information about a study, such as a journal article presenting the main findings, to identify any outcomes without results available. Users might find it helpful to construct a matrix for each eligible study that lists all outcomes described in the study plans and records whether results were available for each. If no study plans are available, users can crosscheck the methods and results sections against one another to identify any outcomes with no results reported, or results reported incompletely (ie, without both a point estimate and measure of precision, or means of deriving these values).
Once users have identified that a study does not have a result available for inclusion in a meta-analysis, they should consider why the result is unavailable. Possible reasons include: the outcome was not measured at all by the study investigators; the outcome was measured but not analysed by study investigators (eg, due to a substantial amount of missing data); the outcome was measured and analysed but the result did not support the investigators’ hypothesis (eg, a non-significant result was observed in a superiority trial); or another reason that might or might not be related to the nature of the result (box 1). Building on the ORBIT (outcome reporting bias in trials) approach,6 we recommend that users of ROB-ME record in a matrix whether results were available for each study meeting the inclusion criteria for the meta-analyses, which lists each study in rows and each meta-analysis to be assessed for risk of bias in columns (see examples of completed matrices in the supplement). When completing the matrix, assessments of availability of results should be based on all the sources of information about a study obtained by users. Therefore, if harms results were not reported in a journal article but were provided by study authors on request, then ROB-ME users should specify in the matrix that they have a study result available for inclusion in the meta-analysis.
Possible scenarios in which study results are missing (adapted from Kirkham et al6)
Example scenarios where it is reasonable to suspect, given a lack of explanation from the study investigators, that a result is missing because of the P value, magnitude, or direction of the result
Study authors report in the methods section, trials register entry, protocol, or elsewhere that they measured (or intended to measure) the outcome of interest, but results are missing for the outcome
All results for an outcome were non-significant and are reported incompletely (eg, described only as “results were not significant,” means were reported with no measure of precision, or change scores within the experimental group were reported but no data for the comparison group were presented), whereas results for other outcomes that were significant are reported completely
Results are missing for one of two outcomes that tend to be measured together (eg, results are available for cause specific mortality and are favourable to the experimental intervention, yet results for all cause mortality, which must have been assessed given that cause specific mortality was also assessed, are missing)
Study authors prespecified that they would report results separately for different outcomes (eg, myocardial infarction, stroke, hypertension) yet instead report results for a composite outcome (eg, cardiovascular events) that happens to be significant and favourable to the experimental intervention
Summary statistics (number of events, or mean scores) are available only globally across all groups (eg, study authors state that 10 of 100 participants in the study experienced adverse events, but do not report the number of events by intervention group)
A result is expected to have been generated for an outcome but it is not available and there is notable concern about the conflicts of interest of primary study investigators or funders involved in the analysis or reporting, which have likely influenced them to withhold results that are unfavourable to the intervention (an assessment using TACIT (tool for addressing conflicts of interest in trials)21 for the study should facilitate this judgment).
Example scenarios where it is reasonable to assume that a result is missing for a reason unrelated to the P value, magnitude, or direction of the result
It is clear that the outcome of interest was not measured in the study based on examination of the study protocol or statistical analysis plan or correspondence with the authors and sponsors
It can be assumed that the outcome of interest was not measured in the study because the instrument or equipment needed to measure the outcome was not available at the time or location where the study was conducted
The outcome of interest was measured but data were not analysed at all owing to a reason unrelated to the nature of the results (eg, the measurement instrument had a fault, funding for the research team discontinued, study staff changed jobs)
Study authors state that results for all outcomes measured appear in an appendix, but the appendix has been removed (or was not uploaded) by mistake
Study authors report results in a format unsuitable for inclusion in a meta-analysis for a reason unrelated to the P value, magnitude, or direction of the result (eg, study investigators report median and (interquartile) range for a continuous outcome because the data were skewed)
Users of ROB-ME should then consider whether it is possible that some eligible studies, in addition to those considered in step 2, were not identified because of the P value, magnitude, or direction of their results (step 3). We anticipate this scenario will be the case for most systematic reviews, apart from those for which an inception cohort was defined (eg, only prospectively registered studies or studies identified for a prospective meta-analysis were eligible for inclusion in the review). In such reviews, all such studies are identified before their results became known.
Assessing a specific meta-analysis result
In step 4, users of the ROB-ME tool answer eight signalling questions, which seek to elicit information about what happened or was observed (box 2). Answers to the signalling questions are informed by the material collated in steps 1 to 3. Step 4 should be completed for each meta-analysis assessed for risk of bias.
Summary of information assessed by the ROB-ME tool
Assessment of non-reporting bias within studies (known unknowns)
Of the studies identified, whether:
Any studies had no result available for inclusion in the meta-analysis, likely because of the P value, magnitude, or direction of the result generated
(If applicable) a notable change to the summary effect estimate would have been likely if the omitted results had been included
There were any studies for which it was unclear that an eligible result was generated
(If applicable) a notable change to the summary effect estimate would have been likely if the potentially omitted results had been included
Assessment of non-reporting bias across studies (unknown unknowns)
Whether:
Circumstances indicate potential for some eligible studies not being identified because of the P value, magnitude or direction of the results generated
(If applicable) it is likely that studies not identified had results that were eligible for inclusion in the meta-analysis
(If applicable) the pattern of observed study results suggest that the meta-analysis is likely to be missing results that were systematically different (in terms of P value, magnitude, or direction) from those observed
(If applicable) sensitivity analyses suggest that the summary effect estimate was biased owing to missing results
ROB-ME=tool for assessing risk of bias due to missing evidence. For the precise wording of signalling questions and guidance for answering each one, see the full risk-of-bias tool at https://www.riskofbias.info/.
The response options for the signalling questions are: yes; probably yes; probably no; no; no information; and not applicable. To maximise simplicity and clarity, the questions are phrased such that a response of “yes” indicates higher risk of bias and “no” indicates lower risk of bias. Responses of “yes” and “probably yes” have the same implications for risk of bias as do responses of “no” and “probably no.” The definitive versions (“yes” and “no”) would typically be selected when firm evidence is available in relation to the signalling question, whereas the “probably” versions would typically be selected when firm evidence is lacking and some judgment has been made. Guidance on how to answer each signalling question is provided in the tool available at https://www.riskofbias.info/.
Signalling questions relating to the assessment of non-reporting bias within studies
The first four signalling questions ask users to consider the extent of missing results in the studies identified for the meta-analysis being assessed for risk of bias (ie, known unknowns). If they determine that one or more of the studies is missing from the meta-analysis because of selective non-reporting of study results, users should consider whether the amount of missing evidence matters. This means whether inclusion of the omitted results would likely lead to a notable change in the summary (combined) effect estimate, given the likely weight and direction of effect in studies omitted from the meta-analysis (if known), meta-analysis model used (such as fixed effect or random effects), and extent of heterogeneity observed. To clarify the potential for bias in the meta-analysis result, users could generate a forest plot displaying studies with results, along with information about studies known to be missing from the meta-analysis due to selective non-reporting of their results (see fig 2 for an example).
Example forest plot showing results missing from a meta-analysis of the effect of preoperative exercise training compared with usual care on postoperative complications (data taken from Gravier et al22)
Signalling questions relating to the assessment of non-reporting bias across studies
The remaining four signalling questions ask users to consider the risk that the meta-analysis result is biased because additional studies or study results, beyond those already identified, are missing (ie, unknown unknowns). Factors to consider include whether missing studies are likely to have had eligible results because the outcome is typically measured in all studies on the topic; whether the pattern of results included in the meta-analysis reveals a tendency for studies with particular results, such as those with P>0.05, to be missing (as observed through graphical methods such as contour enhanced funnel plots23); and whether the findings of appropriate sensitivity analyses, which might include restricting the meta-analysis to the largest studies or using selection models242526 or regression based adjustment methods,2728 suggest that a meta-analysis result is not robust to plausible assumptions about the extent and nature of missing evidence.
Risk-of-bias judgment
ROB-ME operates in the same way as the RoB 215 and ROBINS-I16 tools, in which responses to signalling questions provide the basis for a judgment about the risk of bias in the specific meta-analysis result being assessed. The tool includes an algorithm that maps responses to signalling questions onto a proposed risk-of-bias judgment (see appendices 5 and 6 in the supplement). Possible risk-of-bias judgments are:
Low risk of bias: the meta-analysis result is unlikely to be biased due to missing evidence
High risk of bias: the meta-analysis result is likely to be biased due to missing evidence
Some concerns: uncertainties about the extent or potential impact of missing evidence exist that preclude a judgment of low or high risk of bias
Although ROB-ME considers only the summary effect estimate, we recognise that suppression of results could affect other statistics, such as the estimate of heterogeneity, and in turn the width of the confidence interval for the summary effect estimate.
Presentations of risk-of-bias assessments
Users of ROB-ME should present risk-of-bias judgments in the main systematic review report (eg, in a table or within the forest plot), along with a brief free text justification for each judgment. In addition, we encourage reporting of the completed results matrix (step 2) and answers to all questions in steps 3 and 4 (with supporting text, where applicable) as supplementary material. Only consensus judgments and answers, rather than judgments and answers from individual users, should be presented.
Discussion
Inadequate consideration of risk of bias due to missing evidence could lead to ineffective or harmful treatments being recommended but, despite the implementation of various initiatives to resolve the problem, selective non-publication of studies and selective non-reporting of study results persists.23 For this reason, systematic reviewers should routinely assess the possibility that these issues have biased the results of meta-analyses they have conducted. We developed the ROB-ME tool to help reviewers undertake these assessments. The risk-of-bias judgments drawn from ROB-ME should help distinguish stronger from weaker synthesised evidence and influence the certainty of conclusions drawn from a systematic review (potentially as part of a GRADE assessment 17).
The ROB-ME tool includes several innovations in the assessment of non-reporting biases. In the original Cochrane tool for assessing risk of bias in randomised trials,29 users were prompted to judge the risk of selective reporting bias at the study level, based on whether any results in the study were selectively reported. In reviews adopting this approach, many studies have been judged at high risk of selective reporting bias11; however, the corresponding risk of bias in meta-analyses affected by selective non-reporting of study results is infrequently acknowledged, because no guidance on how to reach such a judgment was provided. ROB-ME explicitly deals with this gap, directing assessments at the level of the meta-analysis result and outlining what factors need to be considered to determine whether the amount of evidence known or assumed to be missing matters. Furthermore, to our knowledge, ROB-ME is the first tool to help users reach an overall judgment about risk of bias in a meta-analysis result arising from both missing studies and missing results in studies.
Of the various components of ROB-ME, we anticipate that assessment of non-reporting bias within studies will be the most resource intensive, yet also the most valuable. This value arises because the impact of selective non-reporting of results in a set of studies known to be missing from a meta-analysis can be quantified more easily than the impact of selective non-publication of an unknown number of studies. Furthermore, if systematic reviewers suspect that a meta-analysis result is biased because results were missing selectively from a large proportion of the studies identified, then the assessment of non-reporting bias across studies is unlikely to change their judgment (other than increasing their certainty that the meta-analysis result is at high risk of bias). This role of the assessment of non-reporting bias within studies does not imply that the assessment across studies should be considered an optional component of the tool. Despite their well known limitations,130 methods originally developed for an assessment across studies (such as funnel plots, tests for funnel plot asymmetry, and sensitivity analyses) are useful when the assessment of selective non-reporting within studies is limited (eg, when detailed study plans are unavailable for most studies included in the review).
The ROB-ME tool was designed to complement tools such as RoB 215 and ROBINS-I16 for assessing risk of bias in study results. These tools enable assessment of bias in selection of the reported result, a domain of bias related to, but not dealt with by, ROB-ME. As a rule of thumb, ROB-ME should be used when systematic reviewers do not have a result from a study to include in a particular meta-analysis, whereas RoB 2 or ROBINS-I should be used to assess whether a result that is available for inclusion might have been cherry picked31 from among multiple measures or analyses. Assessments of risk of bias due to missing evidence and risk of bias in selection of the reported result will likely be informed by the same sources of information about studies (eg, study protocols, register entries), so we advise systematic reviewers to apply step 2 of ROB-ME (completion of the results matrix) in parallel with RoB 2 or ROBINS-I.
The ROB-ME tool is most suitable for assessing meta-analyses of evidence from randomised trials. We believe that it can also be used to assess meta-analyses of non-randomised studies of interventions (eg, cohort studies, interrupted time series studies), but some components of the tool will not apply to such studies. For example, the applicability of tests for funnel plot asymmetry in the context of meta-analyses of non-randomised studies of interventions is unclear,30 so these should not be used in the assessment of non-reporting bias across studies. Furthermore, analyses of non-randomised studies are less likely than randomised trials to be registered or have a publicly accessible protocol, so comparison of prespecified with reported outcomes will usually not be possible. However, assessment of selective non-reporting of study results can still be undertaken for such studies, for example, by comparing the methods and results sections of a study report.
Plans for future development of the ROB-ME tool include further refining to ensure that it is suitable for assessing types of pairwise synthesis other than meta-analysis that yield a point estimate of an intervention effect (such as calculation of the median effect across studies when meta-analysis is not possible or appropriate).3233 We also plan to develop an interactive online version of the tool to facilitate use and prepare a bank of worked examples for educational purposes. We will also explore how ROB-ME judgments should feed into the GRADE framework for assessing certainty in the body of evidence. We welcome feedback from users of ROB-ME and any subsequent updates to the tool will be uploaded to https://www.riskofbias.info/.
Conclusion
The ROB-ME tool deals with gaps in existing approaches for assessing risk of non-reporting biases. We hope that the tool will be useful to authors and users of systematic reviews, by helping to identify meta-analyses at high risk of bias and facilitating appropriate interpretation of results.
Acknowledgments
We thank Sarah Dawson, Kay Dickersin, Carl Heneghan, Toby Lasserson, Hannah Rothstein, and James Thomas for their participation at the development meeting and comments on early drafts of the ROB-ME tool; Robert Boyle, Mahnuma Estee, Alix Hall, Kylie Hunter, Alison Su-Hsun Liu, David Marshall, Paul McCann, Riaz Qureshi, Jelena Savović, Rik van Eekelen, Rui Wang, and Rebecca Wilson for providing feedback on a preliminary version of the tool; and Phi-Yen Nguyen for providing feedback on a more advanced version of the tool.
Footnotes
Contributors: All authors declare to meet the ICMJE conditions for authorship. MJP, JACS, and JPTH conceived the project and oversaw the project. MJP led the research to inform the tool and prepared all materials for the development meeting. MJP, JACS, IB, AH, TL, LAS, AJS, KD, JJM, EHT, and JPTH attended and contributed to the development meeting. RGE took notes at the development meeting. MJP, JACS, and JPTH wrote the first draft of the ROB-ME tool. All authors contributed to the development of the ROB-ME tool and to writing associated guidance. MJP led the drafting of this manuscript. All authors reviewed and commented on drafts of the manuscript. All authors approved the final version of the manuscript. MJP is the guarantor of this work. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: MJP was supported by a National Health and Medical Research Council (NHMRC) Early Career Fellowship (1088535) and an Australian Research Council Discovery Early Career Researcher Award (DE200101618) during the conduct of this research. JACS is supported by the NIHR Bristol Biomedical Research Centre and Health Data Research UK South-West and is a member of the MRC Integrative Epidemiology Unit at the University of Bristol. TL was supported by grant UG1 EY020522 from the National Eye Institute, National Institutes of Health. JEM is supported by an Australian NHMRC Investigator Grant (GNT2009612). JPTH is a National Institute for Health and Care Research (NIHR) senior investigator (NF-SI-0617-10145), is supported by the NIHR Bristol Biomedical Research Centre and NIHR Applied Research Collaboration West (ARC West) at University Hospitals Bristol NHS Foundation Trust, and is a member of the MRC Integrative Epidemiology Unit at the University of Bristol. The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.
Competing interests: All authors have completed the ICMJE disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: no support from any organisation for the submitted work; JJK is a statistical editor for The BMJ, for which he receives remuneration; EM-W has received grants/contracts from the Patient Centered Outcomes Research Institute, National Institutes of Health, Arnold Ventures, and Robert Wood Johnson Foundation, has received consulting fees for editorial work from Origin Editorial, payment/honorariums from the American Public Health Association and National Institutes of Health, and has an unpaid leadership role in the American Psychological Association; LB has received grants/contracts from the state of Colorado, Cochrane, and the US Environmental Protection Agency, has received consulting fees for work as a conflict of interest advisor from Health Canada, has received support for travel from the National Academy of Science Engineering and Medicine, Cochrane and the World Congress on Research Integrity, and has an unpaid chair role for the National Academy of Science Engineering and Medicine committees; EHT has received an honorarium for a lecture on publication bias from the University of British Columbia, Canada; all other authors declare having no competing interests.
Patient and public involvement: Patients and members of the public were not involved in this methodological research. Our motivation for developing the ROB-ME tool arose from our concerns as people who interact with the healthcare system that bias due to missing evidence in meta-analyses can lead to ineffective or harmful treatments being delivered to patients. We plan to disseminate the research widely, including to community participants in evidence synthesis organisations, as we believe increased awareness about non-reporting biases and its consequences can help minimise the problem.
Provenance and peer review: Not commissioned; externally peer reviewed.