Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Multicenter Study
. 2021 Feb 1;78(2):195-209.
doi: 10.1001/jamapsychiatry.2020.3604.

Multimodal Machine Learning Workflows for Prediction of Psychosis in Patients With Clinical High-Risk Syndromes and Recent-Onset Depression

Nikolaos Koutsouleris  1   2   3 , Dominic B Dwyer  1 , Franziska Degenhardt  4   5 , Carlo Maj  6 , Maria Fernanda Urquijo-Castro  1 , Rachele Sanfelici  1   7 , David Popovic  1   8 , Oemer Oeztuerk  1   8 , Shalaila S Haas  9 , Johanna Weiske  1 , Anne Ruef  1 , Lana Kambeitz-Ilankovic  10 , Linda A Antonucci  11 , Susanne Neufang  12 , Christian Schmidt-Kraepelin  12 , Stephan Ruhrmann  10 , Nora Penzel  10 , Joseph Kambeitz  10 , Theresa K Haidl  10 , Marlene Rosen  10 , Katharine Chisholm  13 , Anita Riecher-Rössler  14 , Laura Egloff  14 , André Schmidt  14 , Christina Andreou  14 , Jarmo Hietala  15 , Timo Schirmer  16 , Georg Romer  17 , Petra Walger  18 , Maurizia Franscini  19 , Nina Traber-Walker  19 , Benno G Schimmelmann  20   21 , Rahel Flückiger  21 , Chantal Michel  21 , Wulf Rössler  22 , Oleg Borisov  6 , Peter M Krawitz  6 , Karsten Heekeren  22   23 , Roman Buechler  22   24 , Christos Pantelis  25 , Peter Falkai  1   2 , Raimo K R Salokangas  15 , Rebekka Lencer  26   27 , Alessandro Bertolino  28 , Stefan Borgwardt  14   27 , Markus Noethen  4 , Paolo Brambilla  29   30 , Stephen J Wood  31   32 , Rachel Upthegrove  13 , Frauke Schultze-Lutter  12   33 , Anastasia Theodoridou  22 , Eva Meisenzahl  12 ; PRONIA Consortium
Affiliations
Multicenter Study

Multimodal Machine Learning Workflows for Prediction of Psychosis in Patients With Clinical High-Risk Syndromes and Recent-Onset Depression

Nikolaos Koutsouleris et al. JAMA Psychiatry. .

Abstract

Importance: Diverse models have been developed to predict psychosis in patients with clinical high-risk (CHR) states. Whether prediction can be improved by efficiently combining clinical and biological models and by broadening the risk spectrum to young patients with depressive syndromes remains unclear.

Objectives: To evaluate whether psychosis transition can be predicted in patients with CHR or recent-onset depression (ROD) using multimodal machine learning that optimally integrates clinical and neurocognitive data, structural magnetic resonance imaging (sMRI), and polygenic risk scores (PRS) for schizophrenia; to assess models' geographic generalizability; to test and integrate clinicians' predictions; and to maximize clinical utility by building a sequential prognostic system.

Design, setting, and participants: This multisite, longitudinal prognostic study performed in 7 academic early recognition services in 5 European countries followed up patients with CHR syndromes or ROD and healthy volunteers. The referred sample of 167 patients with CHR syndromes and 167 with ROD was recruited from February 1, 2014, to May 31, 2017, of whom 26 (23 with CHR syndromes and 3 with ROD) developed psychosis. Patients with 18-month follow-up (n = 246) were used for model training and leave-one-site-out cross-validation. The remaining 88 patients with nontransition served as the validation of model specificity. Three hundred thirty-four healthy volunteers provided a normative sample for prognostic signature evaluation. Three independent Swiss projects contributed a further 45 cases with psychosis transition and 600 with nontransition for the external validation of clinical-neurocognitive, sMRI-based, and combined models. Data were analyzed from January 1, 2019, to March 31, 2020.

Main outcomes and measures: Accuracy and generalizability of prognostic systems.

Results: A total of 668 individuals (334 patients and 334 controls) were included in the analysis (mean [SD] age, 25.1 [5.8] years; 354 [53.0%] female and 314 [47.0%] male). Clinicians attained a balanced accuracy of 73.2% by effectively ruling out (specificity, 84.9%) but ineffectively ruling in (sensitivity, 61.5%) psychosis transition. In contrast, algorithms showed high sensitivity (76.0%-88.0%) but low specificity (53.5%-66.8%). A cybernetic risk calculator combining all algorithmic and human components predicted psychosis with a balanced accuracy of 85.5% (sensitivity, 84.6%; specificity, 86.4%). In comparison, an optimal prognostic workflow produced a balanced accuracy of 85.9% (sensitivity, 84.6%; specificity, 87.3%) at a much lower diagnostic burden by sequentially integrating clinical-neurocognitive, expert-based, PRS-based, and sMRI-based risk estimates as needed for the given patient. Findings were supported by good external validation results.

Conclusions and relevance: These findings suggest that psychosis transition can be predicted in a broader risk spectrum by sequentially integrating algorithms' and clinicians' risk estimates. For clinical translation, the proposed workflow should undergo large-scale international validation.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Koutsouleris reported receiving grants from the European Union (EU) during the conduct of the study and having a patent to US20160192889A1 issued. Ms Sanfelici reported receiving personal fees from H. Lundbeck A/S outside the submitted work. Dr Ruhrmann reported receiving grants from the European Commission during the conduct of the study. Dr Riecher-Rössler reported receiving grants from the EU during the conduct of the study. Dr Andreou reported receiving nonfinancial support from Sunovion Pharmaceuticals, Inc, and H. Lundbeck A/S outside the submitted work. Dr Hietala reported receiving personal fees from Orion Company, Ltd, Otsuka Pharmaceutical Co, Ltd, and H. Lundbeck A/S and European College of Neuropsychopharmacology Congress participation support from Takeda Pharmaceutical Company Limited during the conduct of the study. Dr Schirmer reported receiving personal fees from GE Healthcare GmbH outside the submitted work. Dr Romer reported receiving grants from the EU during the conduct of the study. Dr Schimmelmann reported receiving personal fees from Shire Deutschland GmbH outside the submitted work. Dr Flückiger reported receiving grants from the Swiss National Foundation during the conduct of the study. Dr Michel reported receiving grants from the Swiss National Foundation during the conduct of the study. Dr Rössler reported receiving grants from The Zurich Program for Sustainable Development of Mental Health Services (for Zurich Early Recognition Program [ZInEP]) and support by a private donation. Dr Heekeren reported receiving grants from The Zurich Program for Sustainable Development of Mental Health Services during the conduct of the study. Dr Pantelis reported receiving grants from Australian National Health and the Medical Research Council during the conduct of the study and personal fees from H. Lundbeck A/S and Australia Pty Ltd outside the submitted work. Dr Noethen reported receiving personal fees from the Lundbeck Foundation, Robert-Bosch-Stiftung GmbH, HMG Systems Engineering GmbH, Shire Deutschland GmbH, and Life & Brain GmbH outside the submitted work and having a patent to Means and Methods for Establishing a Clinical Prognosis of Diseases Associated With the Formation of Aggregates of Aß1-42 issued. Dr Upthegrove reported receiving personal fees from Sunovion Pharmaceuticals, Inc, outside the submitted work. Dr Meisenzahl reported having a patent to US20160192889A1 licensed. No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Predictive Signatures Underlying the Clinical-Neurocognitive Models
The reliability of predictive pattern elements was evaluated using cross-validation ratio mapping (A). In addition, the significance of predictive features used by the clinical-neurocognitive model was assessed by means of sign-based consistency mapping (B). Both visualization methods are detailed in the eMethods in the Supplement. CTQ indicates Childhood Trauma Questionnaire; DANVA, Diagnostic Analysis of Non-Verbal Accuracy; DSST, Digit-Symbol Substitution Test; FDR, false discovery rate; PVF, Phonetic Verbal Fluency; ROCF, Rey-Osterreith Figure; SIPS, Structured Interview for Psychosis-Risk Syndromes; SOPT, Self-Ordered Pointing Task; SVF, Semantic Verbal Fluency; TMT, Trail Making Test; and WAIS, Wechsler Adult Intelligence Scale.
Figure 2.
Figure 2.. Predictive Signatures Underlying the Polygenic Risk Score (PRS)–Based and Cybernetic Risk Calculator Models
The reliability of predictive pattern elements was evaluated using cross-validation ratio (CVR) mapping (A). In addition, the significance of predictive features used by the PRS-based model was assessed by means of sign-based consistency mapping (B) (as described in the eMethods in the Supplement). The cybernetic model combines all algorithmic and human components (C). FDR indicates false discovery rate; and sMRI, structural magnetic resonance imaging.
Figure 3.
Figure 3.. Statistical Comparison of Prognostic Models
Cohorts include patients with follow-up of 18 months or longer (PRONIA plus 18M), the complete PRONIA cohort, and the Zurich Early Recognition Program (ZInEP). Data points indicate median. The Quade test was used to compare the models’ median balanced accuracy (BAC) computed across the cross-validation cycle (CV2) test data partitions. The BAC measures obtained for the ZInEP cohort (C) were produced by applying the condensed clinical-neurocognitive (Clin-NC), structural magnetic resonance imaging (sMRI)–based, and respective stacked risk calculators of the complete PRONIA sample (B) to this external sample (eFigure 2 in the Supplement). Post hoc comparisons were performed using the t distribution approximation described by Heckert and Filliben. P values were corrected for multiple comparisons using the false discovery rate (FDR). The upper graphs represent the median BAC for each risk calculator in analyses A, B, and C along with the lower and upper quartiles of the BAC distributions (whiskers of the error bars). The lower figures show the logarithmized, FDR-corrected P matrix for the pairwise post hoc classifier comparisons. For an in-depth analysis of the prognostic sequence included in the risk classifier comparison, see eFigures 14 and 15 in the Supplement. The cybernetic risk calculator analyzed the combined predictions of raters, Clin-NC, polygenic risk score (PRS)–based, and sMRI-based risk calculators; the stacked risk calculator, the combined predictions of Clin-NC, PRS-based, and sMRI-based risk calculators. aIndicates risk calculator encompassing the condensed Clin-NCs and sMRI-based models and specifically trained to externally validate the effect of stacking on prognostic performance in the ZInEP cohort.
See this image and copyright information in PMC

Similar articles

See all similar articles

Cited by

See all "Cited by" articles

References

    1. McGorry PD, Mei C. Ultra-high-risk paradigm: lessons learnt and new directions. Evid Based Ment Health. 2018;21(4):131-133. doi:10.1136/ebmental-2018-300061 - DOI - PubMed
    1. Fusar-Poli P, Salazar de Pablo G, Correll CU, et al. . Prevention of psychosis: advances in detection, prognosis, and intervention. JAMA Psychiatry. 2020;77(7):755-765. doi:10.1001/jamapsychiatry.2019.4779 - DOI - PubMed
    1. Fusar-Poli P, Rutigliano G, Stahl D, et al. . Development and validation of a clinically based risk calculator for the transdiagnostic prediction of psychosis. JAMA Psychiatry. 2017;74(5):493-500. doi:10.1001/jamapsychiatry.2017.0284 - DOI - PMC - PubMed
    1. Schultze-Lutter F, Michel C, Schmidt SJ, et al. EPA guidance on the early detection of clinical high risk states of psychoses. Eur Psychiatry. 2015;30(3):405-416. doi:10.1016/j.eurpsy.2015.01.010 - DOI - PubMed
    1. Carrión RE, McLaughlin D, Goldberg TE, et al. . Prediction of functional outcome in individuals at clinical high risk for psychosis. JAMA Psychiatry. 2013;70(11):1133-1142. doi:10.1001/jamapsychiatry.2013.1909 - DOI - PMC - PubMed

Publication types

MeSH terms