Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study
- PMID: 38051574
- PMCID: PMC10731551
- DOI: 10.2196/46718
Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study
Abstract
Background: Reproductive health conditions such as endometriosis, uterine fibroids, and polycystic ovary syndrome (PCOS) affect a large proportion of women and people who menstruate worldwide. Prevalence estimates for these conditions range from 5% to 40% of women of reproductive age. Long diagnostic delays, up to 12 years, are common and contribute to health complications and increased health care costs. Symptom checker apps provide users with information and tools to better understand their symptoms and thus have the potential to reduce the time to diagnosis for reproductive health conditions.
Objective: This study aimed to evaluate the agreement between clinicians and 3 symptom checkers (developed by Flo Health UK Limited) in assessing symptoms of endometriosis, uterine fibroids, and PCOS using vignettes. We also aimed to present a robust example of vignette case creation, review, and classification in the context of predeployment testing and validation of digital health symptom checker tools.
Methods: Independent general practitioners were recruited to create clinical case vignettes of simulated users for the purpose of testing each condition symptom checker; vignettes created for each condition contained a mixture of condition-positive and condition-negative outcomes. A second panel of general practitioners then reviewed, approved, and modified (if necessary) each vignette. A third group of general practitioners reviewed each vignette case and designated a final classification. Vignettes were then entered into the symptom checkers by a fourth, different group of general practitioners. The outcomes of each symptom checker were then compared with the final classification of each vignette to produce accuracy metrics including percent agreement, sensitivity, specificity, positive predictive value, and negative predictive value.
Results: A total of 24 cases were created per condition. Overall, exact matches between the vignette general practitioner classification and the symptom checker outcome were 83% (n=20) for endometriosis, 83% (n=20) for uterine fibroids, and 88% (n=21) for PCOS. For each symptom checker, sensitivity was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, and 100% for PCOS; specificity was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 75% for PCOS; positive predictive value was reported as 81.8% for endometriosis, 84.6% for uterine fibroids, 80% for PCOS; and negative predictive value was reported as 84.6% for endometriosis, 81.8% for uterine fibroids, and 100% for PCOS.
Conclusions: The single-condition symptom checkers have high levels of agreement with general practitioner classification for endometriosis, uterine fibroids, and PCOS. Given long delays in diagnosis for many reproductive health conditions, which lead to increased medical costs and potential health complications for individuals and health care providers, innovative health apps and symptom checkers hold the potential to improve care pathways.
Keywords: accuracy; chatbot; clinical vignettes; digital health; digital health tool; eHealth apps; endometriosis; gynecology; mHealth; mobile health; mobile health app; mobile phone; polycystic ovary syndrome; symptom checker; symptom checkers; uterine; uterine fibroids; uterus; vignettes; women's health.
©Kimberly Peven, Aidan P Wickham, Octavia Wilks, Yusuf C Kaplan, Andrei Marhol, Saddif Ahmed, Ryan Bamford, Adam C Cunningham, Carley Prentice, András Meczner, Matthew Fenech, Stephen Gilbert, Anna Klepchukova, Sonia Ponzo, Liudmila Zhaunova. Originally published in JMIR mHealth and uHealth (https://mhealth.jmir.org), 05.12.2023.
Conflict of interest statement
Conflicts of Interest: KP, APW, OW, YCK, A Marhoi, SA, ACC, AK, SP, and LZ were employees at Flo Health, Inc and have stock ownership in the company. RB, CP, A Meczner, MF, and SG are paid consultants for Flo Health, Inc. SG declares no nonfinancial interests but the following competing financial interests: he has or has had consulting relationships with Una Health GmbH, Lindus Health Ltd, Flo Health UK Limited, Thymia Ltd, and Ada Health GmbH and holds share options in Ada Health GmbH. MF declares no nonfinancial interests but the following competing financial interests: he has a consulting relationship with Flo Health UK Limited and holds share options in Una Health GmbH. A Meczner declares no nonfinancial interests but the following competing financial interests: he is an employee and shareholder at Healthily or Your.MD.
Figures
Similar articles
-
Evaluating the Diagnostic Performance of Symptom Checkers: Clinical Vignette Study.JMIR AI. 2024 Apr 29;3:e46875. doi: 10.2196/46875. JMIR AI. 2024. PMID: 38875676 Free PMC article.
-
How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective.Digit Health. 2023 Aug 21;9:20552076231194929. doi: 10.1177/20552076231194929. eCollection 2023 Jan-Dec. Digit Health. 2023. PMID: 37614591 Free PMC article.
-
Triage and Diagnostic Accuracy of Online Symptom Checkers: Systematic Review.J Med Internet Res. 2023 Jun 2;25:e43803. doi: 10.2196/43803. J Med Internet Res. 2023. PMID: 37266983 Free PMC article. Review.
-
The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review.NPJ Digit Med. 2022 Aug 17;5(1):118. doi: 10.1038/s41746-022-00667-w. NPJ Digit Med. 2022. PMID: 35977992 Free PMC article. Review.
-
Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation.J Med Internet Res. 2022 May 10;24(5):e31810. doi: 10.2196/31810. J Med Internet Res. 2022. PMID: 35536633 Free PMC article.
References
-
- Azziz R, Carmina E, Chen Z, Dunaif A, Laven JSE, Legro RS, Lizneva D, Natterson-Horowtiz B, Teede HJ, Yildiz BO. Polycystic ovary syndrome. Nat Rev Dis Primers. 2016 Aug 11;2:16057. doi: 10.1038/nrdp.2016.57. https://www.nature.com/articles/nrdp201657 nrdp201657 - DOI - PubMed
-
- Bozdag G, Mumusoglu S, Zengin D, Karabulut E, Yildiz BO. The prevalence and phenotypic features of polycystic ovary syndrome: a systematic review and meta-analysis. Hum Reprod. 2016;31(12):2841–2855. doi: 10.1093/humrep/dew218. https://academic.oup.com/humrep/article/31/12/2841/2730240?login=false dew218 - DOI - PubMed
-
- Carmina E, Azziz R. Diagnosis, phenotype, and prevalence of polycystic ovary syndrome. Fertil Steril. 2006;86(Suppl 1):S7–S8. doi: 10.1016/j.fertnstert.2006.03.012. https://linkinghub.elsevier.com/retrieve/pii/S0015-0282(06)00722-9 S0015-0282(06)00722-9 - DOI - PubMed
-
- Deswal R, Narwal V, Dang A, Pundir CS. The prevalence of polycystic ovary syndrome: a brief systematic review. J Hum Reprod Sci. 2020;13(4):261–271. doi: 10.4103/jhrs.JHRS_95_18. https://europepmc.org/abstract/MED/33627974 JHRS-13-261 - DOI - PMC - PubMed
-
- Ellis K, Munro D, Clarke J. Endometriosis is undervalued: a call to action. Front Glob Womens Health. 2022;3:902371. doi: 10.3389/fgwh.2022.902371. https://europepmc.org/abstract/MED/35620300 - DOI - PMC - PubMed
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources
Medical