. 2024 Jan 11;14(1):1084.

doi: 10.1038/s41598-023-47934-8.

Multi-site benchmark classification of major depressive disorder using machine learning on cortical and subcortical measures

Vladimir Belov¹, Tracy Erwin-Grabner¹, Moji Aghajani^{2

3}, Andre Aleman⁴, Alyssa R Amod⁵, Zeynep Basgoze⁶, Francesco Benedetti⁷, Bianca Besteher⁸, Robin Bülow⁹, Christopher R K Ching¹⁰, Colm G Connolly¹¹, Kathryn Cullen⁶, Christopher G Davey¹², Danai Dima^{13

14}, Annemiek Dols², Jennifer W Evans¹⁵, Cynthia H Y Fu^{16

17}, Ali Saffet Gonul¹⁸, Ian H Gotlib¹⁹, Hans J Grabe²⁰, Nynke Groenewold⁵, J Paul Hamilton^{21

22}, Ben J Harrison¹², Tiffany C Ho^{23

24}, Benson Mwangi^{25

26}, Natalia Jaworska²⁷, Neda Jahanshad¹⁰, Bonnie Klimes-Dougan²⁸, Sheri-Michelle Koopowitz⁵, Thomas Lancaster^{29

30}, Meng Li⁸, David E J Linden^{29

30

31

32}, Frank P MacMaster³³, David M A Mehler^{29

30

34}, Elisa Melloni⁷, Bryon A Mueller⁶, Amar Ojha^{35

36}, Mardien L Oudega², Brenda W J H Penninx², Sara Poletti⁷, Edith Pomarol-Clotet³⁷, Maria J Portella³⁸, Elena Pozzi^{39

40}, Liesbeth Reneman⁴¹, Matthew D Sacchet⁴², Philipp G Sämann⁴³, Anouk Schrantee⁴¹, Kang Sim^{44

45

46}, Jair C Soares²⁶, Dan J Stein⁴⁷, Sophia I Thomopoulos¹⁰, Aslihan Uyar-Demir¹⁸, Nic J A van der Wee⁴⁸, Steven J A van der Werff^{48

49}, Henry Völzke⁵⁰, Sarah Whittle⁵¹, Katharina Wittfeld^{20

52}, Margaret J Wright^{53

54}, Mon-Ju Wu^{25

26}, Tony T Yang²³, Carlos Zarate⁵⁵, Dick J Veltman², Lianne Schmaal^{39

40}, Paul M Thompson¹⁰, Roberto Goya-Maldonado⁵⁶; ENIGMA Major Depressive Disorder working group

Affiliations

¹ Laboratory of Systems Neuroscience and Imaging in Psychiatry (SNIP-Lab), Department of Psychiatry and Psychotherapy, University Medical Center Göttingen (UMG), Georg-August University, Von-Siebold-Str. 5, 37075, Göttingen, Germany.
² Department of Psychiatry, Amsterdam UMC, Amsterdam Neuroscience, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
³ Institute of Education and Child Studies, Section Forensic Family and Youth Care, Leiden University, Leiden, The Netherlands.
⁴ Department of Biomedical Sciences of Cells and Systems, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.
⁵ Department of Psychiatry and Mental Health, University of Cape Town, Cape Town, South Africa.
⁶ Department of Psychiatry and Behavioral Science, University of Minnesota Medical School, Minneapolis, MN, USA.
⁷ Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Milan, Italy.
⁸ Department of Psychiatry and Psychotherapy, Jena University Hospital, Jena, Germany.
⁹ Institute for Radiology and Neuroradiology, University Medicine Greifswald, Greifswald, Germany.
¹⁰ Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, CA, USA.
¹¹ Department of Biomedical Sciences, Florida State University, Tallahassee, FL, USA.
¹² Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne, Parkville, VIC, Australia.
¹³ Department of Psychology, School of Arts and Social Sciences, City, University of London, London, UK.
¹⁴ Department of Neuroimaging, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK.
¹⁵ Experimental Therapeutics and Pathophysiology Branch, National Institute for Mental Health, National Institutes of Health, Bethesda, MD, USA.
¹⁶ School of Psychology, University of East London, London, UK.
¹⁷ Centre for Affective Disorders, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK.
¹⁸ SoCAT Lab, Department of Psychiatry, School of Medicine, Ege University, Izmir, Turkey.
¹⁹ Department of Psychology, Stanford University, Stanford, CA, USA.
²⁰ Department of Psychiatry and Psychotherapy, University Medicine Greifswald, Greifswald, Germany.
²¹ Center for Social and Affective Neuroscience, Department of Biomedical and Clinical Sciences, Linköping University, Linköping, Sweden.
²² Center for Medical Imaging and Visualization, Linköping University, Linköping, Sweden.
²³ Department of Psychiatry and Behavioral Sciences, Division of Child and Adolescent Psychiatry, Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, USA.
²⁴ Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA.
²⁵ Louis A. Faillace, MD, Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at Houston, Houston, TX, USA.
²⁶ Center Of Excellence On Mood Disorders, Louis A. Faillace, MD, Department of Psychiatry and Behavioral Sciences at McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, USA.
²⁷ Department of Psychiatry, McGill University, Montreal, QC, Canada.
²⁸ Department of Psychology, University of Minnesota, Minneapolis, MN, USA.
²⁹ Cardiff University Brain Research Imaging Center, Cardiff University, Cardiff, UK.
³⁰ MRC Center for Neuropsychiatric Genetics and Genomics, Cardiff University, Cardiff, UK.
³¹ Division of Psychological Medicine and Clinical Neurosciences, Cardiff University, Cardiff, UK.
³² School of Mental Health and Neuroscience, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands.
³³ Departments of Psychiatry and Pediatrics, University of Calgary, Calgary, AB, Canada.
³⁴ Department of Psychiatry, Psychotherapy and Psychosomatics, Medical School, RWTH Aachen University, Aachen, Germany.
³⁵ Center for Neuroscience, University of Pittsburgh, Pittsburgh, PA, USA.
³⁶ Center for Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA.
³⁷ FIDMAG Germanes Hospitalàries Research Foundation, Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM), Barcelona, Catalonia, Spain.
³⁸ Sant Pau Mental Health Research Group, Institut de Recerca de L'Hospital de La Santa Creu I Sant Pau, Barcelona, Catalonia, Spain.
³⁹ Centre for Youth Mental Health, The University of Melbourne, Parkville, VIC, Australia.
⁴⁰ Orygen, Parkville, VIC, Australia.
⁴¹ Department of Radiology and Nuclear Medicine, Amsterdam University Medical Centers, Amsterdam, The Netherlands.
⁴² Meditation Research Program, Department of Psychiatry, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
⁴³ Max Planck Institute of Psychiatry, Munich, Germany.
⁴⁴ West Region, Institute of Mental Health, Singapore, Singapore.
⁴⁵ Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
⁴⁶ Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore.
⁴⁷ SA MRC Research Unit on Risk and Resilience in Mental Disorders, Department of Psychiatry and Neuroscience Institute, University of Cape Town, Cape Town, South Africa.
⁴⁸ Leiden Institute for Brain and Cognition, Leiden University Medical Center, Leiden, The Netherlands.
⁴⁹ Department of Psychiatry, Leiden University Medical Center, Leiden, The Netherlands.
⁵⁰ Institute for Community Medicine, University Medicine Greifswald, Greifswald, Germany.
⁵¹ Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne and Melbourne Health, Melbourne, VIC, Australia.
⁵² German Center for Neurodegenerative Diseases (DZNE), Site Rostock/ Greifswald, Greifswald, Germany.
⁵³ Queensland Brain Institute, The University of Queensland, Brisbane, QLD, Australia.
⁵⁴ Centre for Advanced Imaging, The University of Queensland, Brisbane, QLD, Australia.
⁵⁵ Section on the Neurobiology and Treatment of Mood Disorders, National Institute of Mental Health, Bethesda, MD, USA.
⁵⁶ Laboratory of Systems Neuroscience and Imaging in Psychiatry (SNIP-Lab), Department of Psychiatry and Psychotherapy, University Medical Center Göttingen (UMG), Georg-August University, Von-Siebold-Str. 5, 37075, Göttingen, Germany. roberto.goya@med.uni-goettingen.de.

PMID: 38212349
PMCID: PMC10784593
DOI: 10.1038/s41598-023-47934-8

Multi-site benchmark classification of major depressive disorder using machine learning on cortical and subcortical measures

Vladimir Belov et al. Sci Rep. 2024.

. 2024 Jan 11;14(1):1084.

doi: 10.1038/s41598-023-47934-8.

Authors

Affiliations

¹ Laboratory of Systems Neuroscience and Imaging in Psychiatry (SNIP-Lab), Department of Psychiatry and Psychotherapy, University Medical Center Göttingen (UMG), Georg-August University, Von-Siebold-Str. 5, 37075, Göttingen, Germany.
² Department of Psychiatry, Amsterdam UMC, Amsterdam Neuroscience, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam, The Netherlands.
³ Institute of Education and Child Studies, Section Forensic Family and Youth Care, Leiden University, Leiden, The Netherlands.
⁴ Department of Biomedical Sciences of Cells and Systems, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands.
⁵ Department of Psychiatry and Mental Health, University of Cape Town, Cape Town, South Africa.
⁶ Department of Psychiatry and Behavioral Science, University of Minnesota Medical School, Minneapolis, MN, USA.
⁷ Division of Neuroscience, IRCCS San Raffaele Scientific Institute, Milan, Italy.
⁸ Department of Psychiatry and Psychotherapy, Jena University Hospital, Jena, Germany.
⁹ Institute for Radiology and Neuroradiology, University Medicine Greifswald, Greifswald, Germany.
¹⁰ Imaging Genetics Center, Mark and Mary Stevens Neuroimaging and Informatics Institute, Keck School of Medicine, University of Southern California, Marina del Rey, CA, USA.
¹¹ Department of Biomedical Sciences, Florida State University, Tallahassee, FL, USA.
¹² Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne, Parkville, VIC, Australia.
¹³ Department of Psychology, School of Arts and Social Sciences, City, University of London, London, UK.
¹⁴ Department of Neuroimaging, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK.
¹⁵ Experimental Therapeutics and Pathophysiology Branch, National Institute for Mental Health, National Institutes of Health, Bethesda, MD, USA.
¹⁶ School of Psychology, University of East London, London, UK.
¹⁷ Centre for Affective Disorders, Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, UK.
¹⁸ SoCAT Lab, Department of Psychiatry, School of Medicine, Ege University, Izmir, Turkey.
¹⁹ Department of Psychology, Stanford University, Stanford, CA, USA.
²⁰ Department of Psychiatry and Psychotherapy, University Medicine Greifswald, Greifswald, Germany.
²¹ Center for Social and Affective Neuroscience, Department of Biomedical and Clinical Sciences, Linköping University, Linköping, Sweden.
²² Center for Medical Imaging and Visualization, Linköping University, Linköping, Sweden.
²³ Department of Psychiatry and Behavioral Sciences, Division of Child and Adolescent Psychiatry, Weill Institute for Neurosciences, University of California, San Francisco, San Francisco, CA, USA.
²⁴ Department of Psychology, University of California, Los Angeles, Los Angeles, CA, USA.
²⁵ Louis A. Faillace, MD, Department of Psychiatry and Behavioral Sciences, The University of Texas Health Science Center at Houston, Houston, TX, USA.
²⁶ Center Of Excellence On Mood Disorders, Louis A. Faillace, MD, Department of Psychiatry and Behavioral Sciences at McGovern Medical School, The University of Texas Health Science Center at Houston, Houston, TX, USA.
²⁷ Department of Psychiatry, McGill University, Montreal, QC, Canada.
²⁸ Department of Psychology, University of Minnesota, Minneapolis, MN, USA.
²⁹ Cardiff University Brain Research Imaging Center, Cardiff University, Cardiff, UK.
³⁰ MRC Center for Neuropsychiatric Genetics and Genomics, Cardiff University, Cardiff, UK.
³¹ Division of Psychological Medicine and Clinical Neurosciences, Cardiff University, Cardiff, UK.
³² School of Mental Health and Neuroscience, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, The Netherlands.
³³ Departments of Psychiatry and Pediatrics, University of Calgary, Calgary, AB, Canada.
³⁴ Department of Psychiatry, Psychotherapy and Psychosomatics, Medical School, RWTH Aachen University, Aachen, Germany.
³⁵ Center for Neuroscience, University of Pittsburgh, Pittsburgh, PA, USA.
³⁶ Center for Neural Basis of Cognition, University of Pittsburgh, Pittsburgh, PA, USA.
³⁷ FIDMAG Germanes Hospitalàries Research Foundation, Centro de Investigación Biomédica en Red de Salud Mental (CIBERSAM), Barcelona, Catalonia, Spain.
³⁸ Sant Pau Mental Health Research Group, Institut de Recerca de L'Hospital de La Santa Creu I Sant Pau, Barcelona, Catalonia, Spain.
³⁹ Centre for Youth Mental Health, The University of Melbourne, Parkville, VIC, Australia.
⁴⁰ Orygen, Parkville, VIC, Australia.
⁴¹ Department of Radiology and Nuclear Medicine, Amsterdam University Medical Centers, Amsterdam, The Netherlands.
⁴² Meditation Research Program, Department of Psychiatry, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
⁴³ Max Planck Institute of Psychiatry, Munich, Germany.
⁴⁴ West Region, Institute of Mental Health, Singapore, Singapore.
⁴⁵ Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore.
⁴⁶ Lee Kong Chian School of Medicine, Nanyang Technological University, Singapore, Singapore.
⁴⁷ SA MRC Research Unit on Risk and Resilience in Mental Disorders, Department of Psychiatry and Neuroscience Institute, University of Cape Town, Cape Town, South Africa.
⁴⁸ Leiden Institute for Brain and Cognition, Leiden University Medical Center, Leiden, The Netherlands.
⁴⁹ Department of Psychiatry, Leiden University Medical Center, Leiden, The Netherlands.
⁵⁰ Institute for Community Medicine, University Medicine Greifswald, Greifswald, Germany.
⁵¹ Melbourne Neuropsychiatry Centre, Department of Psychiatry, The University of Melbourne and Melbourne Health, Melbourne, VIC, Australia.
⁵² German Center for Neurodegenerative Diseases (DZNE), Site Rostock/ Greifswald, Greifswald, Germany.
⁵³ Queensland Brain Institute, The University of Queensland, Brisbane, QLD, Australia.
⁵⁴ Centre for Advanced Imaging, The University of Queensland, Brisbane, QLD, Australia.
⁵⁵ Section on the Neurobiology and Treatment of Mood Disorders, National Institute of Mental Health, Bethesda, MD, USA.
⁵⁶ Laboratory of Systems Neuroscience and Imaging in Psychiatry (SNIP-Lab), Department of Psychiatry and Psychotherapy, University Medical Center Göttingen (UMG), Georg-August University, Von-Siebold-Str. 5, 37075, Göttingen, Germany. roberto.goya@med.uni-goettingen.de.

PMID: 38212349
PMCID: PMC10784593
DOI: 10.1038/s41598-023-47934-8

Abstract

Machine learning (ML) techniques have gained popularity in the neuroimaging field due to their potential for classifying neuropsychiatric disorders. However, the diagnostic predictive power of the existing algorithms has been limited by small sample sizes, lack of representativeness, data leakage, and/or overfitting. Here, we overcome these limitations with the largest multi-site sample size to date (N = 5365) to provide a generalizable ML classification benchmark of major depressive disorder (MDD) using shallow linear and non-linear models. Leveraging brain measures from standardized ENIGMA analysis pipelines in FreeSurfer, we were able to classify MDD versus healthy controls (HC) with a balanced accuracy of around 62%. But after harmonizing the data, e.g., using ComBat, the balanced accuracy dropped to approximately 52%. Accuracy results close to random chance levels were also observed in stratified groups according to age of onset, antidepressant use, number of episodes and sex. Future studies incorporating higher dimensional brain imaging/phenotype features, and/or using more advanced machine and deep learning methods may yield more encouraging prospects.

PubMed Disclaimer

Conflict of interest statement

PMT and NJ received a research grant from Biogen, Inc., for research unrelated to this manuscript. HJG has received travel grants and speakers honoraria from Fresenius Medical Care, Neuraxpharm, Servier and Janssen Cilag as well as research funding from Fresenius Medical Care unrelated to this manuscript. JCS has served as a consultant for Pfizer,Sunovion, Sanofi, Johnson & Johnson, Livanova, and Boehringer Ingelheim. The remaining authors declare no conflict of interest.

Figures

**Figure 1**
Feature weights of support vector machines (SVM) with the linear kernel. To assess the decision-making of SVM to differentiate subjects with major depressive disorder (MDD) from healthy controls (HC), we investigate the importance of the structural brain features by looking at the corresponding feature weights for the regional cortical surface areas, cortical thicknesses and subcortical volumes. The horizontal bars indicate the 95% confidence interval calculated using percentile method via bootstrapping.

**Figure 2**
The most informative features for classification including regional cortical surface areas, thicknesses and subcortical volumes, trained on the whole data set without and with ComBat harmonization. Increased and decreased feature weight values in the major depressive disorder (MDD) group are represented by red and blue colormap, respectively.

**Figure 3**
Detailed analysis pipeline. Initial data from all cohorts is split into training and test sets according to splitting strategies (Splitting by Age/Sex and Splitting by Site) after removing subjects with more than 75% missing data and data imputation steps. The corresponding training folds are then residualized directly to remove ICV, age and sex related effects and fed to the classification algorithms. In case of harmonization by ComBat, the residualization step takes place after the harmonization step is conducted. If training folds were harmonized by ComBat, the test fold was harmonized as well by using ComBat estimates from the training folds. Next, the test fold was residualized by using estimates obtained from the training folds. We estimated classification performance on the residualized test fold. This routine was performed iteratively for each combination of training and test folds.

See this image and copyright information in PMC

Cited by

Systematic Review: White Matter Microstructural Organization in Adolescents With Depression.
Radoeva PD, Milev VT, Hunt JI, Legere CH, Deoni SCL, Sheinkopf SJ, Mazefsky CA, Philip NS, Dickstein DP. Radoeva PD, et al. JAACAP Open. 2023 Dec;1(4):233-245. doi: 10.1016/j.jaacop.2023.08.006. Epub 2023 Sep 5. JAACAP Open. 2023. PMID: 38576601 Free PMC article.

References

1. Kessler RC, Bromet EJ. The epidemiology of depression across cultures. Annu. Rev. Public Health. 2013;34:119–138. doi: 10.1146/annurev-publhealth-031912-114409. - DOI - PMC - PubMed
1. Cho Y, et al. Factors associated with quality of life in patients with depression: A nationwide population-based study. PLOS ONE. 2019;14:e0219455. doi: 10.1371/journal.pone.0219455. - DOI - PMC - PubMed
1. Cai, H. et al. Prevalence of suicidality in major depressive disorder: A systematic review and meta-analysis of comparative studies. Front. Psychiatry12, (2021). - PMC - PubMed
1. Cleare AFSWCDKMBMLPAJ. A multidimensional tool to quantify treatment resistance in depression: The Maudsley staging method. J. Clin. Psychiatry. 2009;70:12363. - PubMed
1. Han LKM, et al. Brain aging in major depressive disorder: Results from the ENIGMA major depressive disorder working group. Mol. Psychiatry. 2020 doi: 10.1038/s41380-020-0754-0. - DOI - PMC - PubMed

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Multi-site benchmark classification of major depressive disorder using machine learning on cortical and subcortical measures

Affiliations

Multi-site benchmark classification of major depressive disorder using machine learning on cortical and subcortical measures

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

MeSH terms

Grants and funding

LinkOut - more resources

Full Text Sources

Medical

Miscellaneous

Abstract

Conflict of interest statement

Figures

Similar articles

Cited by

References

MeSH terms

Related information

Grants and funding

LinkOut - more resources

Full Text Sources

Medical

Miscellaneous