Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2014 Apr 1:14:45.
doi: 10.1186/1471-2288-14-45.

Newcastle-Ottawa Scale: comparing reviewers' to authors' assessments

Carson Ka-Lok Lo, Dominik Mertz, Mark Loeb  1
Affiliations
Comparative Study

Newcastle-Ottawa Scale: comparing reviewers' to authors' assessments

Carson Ka-Lok Lo et al. BMC Med Res Methodol. .

Abstract

Background: Lack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The same might be true for observational studies. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza.

Methods: Cohort studies included in the systematic review and published between 2008-2011 were included. The corresponding or first authors completed a survey covering all NOS items. Results were compared with the NOS assessment applied by reviewers of the systematic review. Inter-rater reliability was calculated using kappa (K) statistics.

Results: Authors of 65/182 (36%) studies completed the survey. The overall NOS score was significantly higher (p < 0.001) in the reviewers' assessment (median = 6; interquartile range [IQR] 6-6) compared with those by authors (median = 5, IQR 4-6). Inter-rater reliability by item ranged from slight (K = 0.15, 95% confidence interval [CI] = -0.19, 0.48) to poor (K = -0.06, 95% CI = -0.22, 0.10). Reliability for the overall score was poor (K = -0.004, 95% CI = -0.11, 0.11).

Conclusions: Differences in assessment and low agreement between reviewers and authors suggest the need to contact authors for information not published in studies when applying the NOS in systematic reviews.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Distribution of total scores for the Newcastle-Ottawa Scale (NOS) for cohort studies. Reviewers (left) and authors (right) evaluated for risk of bias for cohort studies (n = 65).
Figure 2
Figure 2
Differences in Newcastle-Ottawa Scale (NOS) total score between reviewers and authors. The total score assigned for each cohort study by reviewers was subtracted with the total score assigned by authors.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. Higgins JPT, Green S, editor. Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011] [ http://www.cochrane-handbook.org]
    1. Wells G, Shea B, O’Connell D, Peterson J, Welch V, Losos M, Tugwell P. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. 2013. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.
    1. Hartling L, Milne A, Hamm MP, Vandermeer B, Ansari M, Tsertsvadze A, Dryden DM. Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. J Clin Epidemiol. 2013;66:982–993. doi: 10.1016/j.jclinepi.2013.03.003. - DOI - PubMed
    1. Oremus M, Oremus C, Hall GBC, McKinnon MC. ECT & Cognition Systematic Review Team. Inter-rater and test–retest reliability of quality assessments by novice student raters using the Jadad and Newcastle–Ottawa Scales. BMJ Open. 2012;2:e001368. - PMC - PubMed
    1. Devereaux PJ, Choi PTL, El-Dika S, Bhandari M, Montori VM, Schünemann HJ, Garg AX, Busse JW, Heels-Ansdell D, Ghali WA, Manns BJ, Guyatt GH. An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. J Clin Epidemiol. 2004;57:1232–1236. doi: 10.1016/j.jclinepi.2004.03.017. - DOI - PubMed

Publication types

LinkOut - more resources