You are here

  • Home
  • Archive
  • Volume 14, Issue 6
  • Cerina: cognitive–behavioural therapy-based mobile application for managing GAD symptoms among Ulster University Students in Northern Ireland – a protocol for a pilot feasibility randomised controlled trial

Article Text

Article menu
Mental health
Protocol
Cerina: cognitive–behavioural therapy-based mobile application for managing GAD symptoms among Ulster University Students in Northern Ireland – a protocol for a pilot feasibility randomised controlled trial
  1. Ozlem Eylem-van Bergeijk1,
  2. Siobhan Poulter2,
  3. Kurtis Ashcroft2,
  4. Tony Robinson2,
  5. Prasannajeet Mane3,
  6. Momotaj Islam3,
  7. Joan Condell4,
  8. Gerard Leavey5
  1. 1 Division of Population Health, Health Services Research and Primary Care, University of Manchester, Manchester, UK
  2. 2 School of Computing, Engineering & Intelligent Systems, Ulster University, Coleraine, UK
  3. 3 Cerina Health, London, UK
  4. 4 Centre for Personalised Medicine, Ulster University Faculty of Life and Health Sciences, Derry, UK
  5. 5 Department of Psychology, Ulster University, Derry, UK
  1. Correspondence to Dr Ozlem Eylem-van Bergeijk; ozlem.eylem-vanbergeijk{at}manchester.ac.uk

Abstract

Introduction University students are one of the most vulnerable populations for anxiety disorders worldwide. In Northern Ireland, anxiety disorders appear to be more common among the university student population due to the population demographics across the region. Despite the need, these students show less inclination to access the widely available on-campus well-being services and other external professional services. Digital cognitive–behavioural therapy (CBT) aims to bridge this gap between the need for psychological help and access to it. However, challenges such as limited reach, low adoption, implementation barriers and poor long-term maintenance are mainstay issues resulting in reduced uptake of digital CBT. As a result, the potential impact of digital CBT is currently restricted. The proposed intervention ‘Cerina’ is a scalable CBT-based mobile app with an interactive user interface that can be implemented in university settings if found to be feasible and effective.

Methods and analysis The study is a single-blind pilot feasibility randomised controlled trial aiming to test the feasibility and preliminary effects of Cerina in reducing Generalised Anxiety Disorder (GAD) symptoms. Participants are 90 Ulster University students aged 18 and above with self-reported GAD symptoms. They will be allocated to two conditions: treatment (ie, access to Cerina for 6 weeks) and a wait-list control group (ie, optional on-campus well-being services for 6 weeks). Participants in the wait-list will access Cerina 6 weeks after their randomisation and participants in both conditions will be assessed at baseline, at 3 (mid-assessment) and 6 weeks (postassessment). The primary outcome is the feasibility of Cerina (ie, adherence to the intervention, its usability and the potential to deliver a full trial in the future). The secondary outcomes include generalised anxiety, depression, worry and quality of life. Additionally, participants in both conditions will be invited to semistructured interviews for process evaluation.

Ethics and dissemination Ethical approval for the study has been granted by the Ulster University Research Ethics Committee (ID: FCPSY-22-084). The results of the study will be disseminated through publications in scientific articles and presentations at relevant conferences and/or public events.

Trial registration number NCT06146530.

  • clinical trial
  • mental health
  • anxiety disorders
  • randomized controlled trial
  • public health
http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

https://doi.org/10.1136/bmjopen-2023-083554

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    STRENGTHS AND LIMITATIONS OF THIS STUDY

    • Low-threshold cognitive–behavioural therapy-based mobile application, easily accessible to the university student population.

    • Partnership between Ulster University, Ulster Student Well-being team and a mental health start-up to increase the reach and access of the target population and evaluate the intervention.

    • The study is designed as a feasibility trial and is not powered to detect statistically significant effects of the intervention.

    • The study results might not generalise beyond the inclusion and exclusion criteria of the current study.

    • The wait-list control groups are known to overestimate the effects of the intervention compared with treatment as usual.

    Introduction

    It is estimated that 264 million adults around the globe have anxiety.1 Of these adults, 179 million are female (63%) and 105 million are male (37%).2 Anxiety disorders are very costly and are one of the leading contributors to the global health-related burden.3 According to the data on anxiety disorders obtained from the WHO Global Burden of Disease Study in 2019, the relative risk of the incidence of anxiety disorders and the disability-adjusted life-years increased with age in adolescence and middle age and then decreased among people of both sexes.4 The most common anxiety disorder is Generalised Anxiety Disorder (GAD) affecting almost 30%–40% of the global population.1 It is estimated that the prevalence of GAD varies widely across countries, with lifetime prevalence highest in high-income countries (5.0% (0.1%)), lower in middle-income countries (2.8% (0.1%)) and lowest in low-income countries (1.6% (0.1%)).2 Lifetime comorbidity is high (81.9% (0.7%)), particularly with mood (63.0% (0.9%)) and other anxiety (51.7% (0.9%)) disorders. Treatment is sought by approximately half of the affected individuals (49.2% (1.2%)) in high-income countries.3 This means that GAD is untreated and unrecognised.5

    University students are one of the vulnerable populations for common mental health conditions (CMDs) in general and anxiety disorders specifically.6–8 According to the WHO, one out of three students was diagnosed with at least one CMDs, the most prevalent anxiety disorder in 2018.6 During the COVID-19 pandemic, anxiety disorders alone affected about one in three students.9–11 The existing longitudinal studies do not consistently support the increased risk for GAD among students in the long term.12–14 However, they indicate a potential subgroup of students such as those who report more financial uncertainty or a history of CMDs, who could be more at risk of severe GAD and comorbid CMDs and problems.13 From the life course perspective,15 university years correspond to emerging adulthood that is characterised by a transition period between adolescence and adulthood.16 On the one hand, ongoing life transitions and developmental alterations in this period can offer opportunities for self-exploration and growth.17 While on the other hand, such life transitions can exacerbate uncertainty, ambivalent feelings and risky behaviours.15 Additionally, international students and students from ethnic and sexual minority backgrounds may experience additional stressors related to culture shock, discrimination and lack of social and cultural capital.18 These stressors can interact with pre-existing vulnerabilities and well-known stressors such as academic pressure, financial uncertainties due to large amounts of study loans and can lead to heightened stress which is central to the onset of physical and psychological conditions.19 20

    In Northern Ireland (NI), mental health problems and anxiety disorders are especially prevalent among university students due to the population demographics across the region in addition to the above-mentioned factors common to the global university student population.9 21 22 According to the data collected at Ulster University as part of the WHO World Mental Health International College Student Project (WMH-ICS), receipt of treatment was low (37.8%) (N=392), particularly among males and those with life time mental health condition with no suicidal thoughts and behaviours (STBs).23 Males were less inclined to seek external professional services. However, interestingly they were less likely than females to rate embarrassment (OR=0.60) and/or worry about being treated differently (OR=0.63) as reasons for not seeking help.23 Additionally, those with a lifetime mental health condition reported lower perceived need and/or readiness to change, and they rated being unsure about where to go as a more important barrier (OR=1.80) compared with those with STBs and without a lifetime mental health condition.23 Moreover, a recent study conducted at Ulster University based on WMH-ICS data indicated that 22.6% (N=739) of students reported GAD symptoms, and students older than 21, those from sexual minority (Lesbian, Gay, Bisexual, Transgender, Queer: LGBTQ+) and low SES backgrounds were less likely to access available psychological help.9 These results highlight vulnerability in line with the literature on mental health problems and the barriers to help-seeking among university student populations.24 25 Therefore, early interventions addressing the need for psychological care and treatment of diverse student populations are crucial even if they do not have suicidal thoughts.23

    Cognitive–behavioural therapy (CBT) and pharmacotherapy are recommended evidence-based first-line interventions for GAD.26 CBT is equally effective as pharmacotherapy at the short term but has larger overall treatment effects and fewer side effects than pharmacotherapy at the longer term (6–12 months follow-up) (g=0.34).27 Despite the effectiveness and the favourable benefits of CBT over medication, there are recognised shortcomings in the current treatment of GAD resulting in a lengthy delay in the help-seeking process.6 Face-to-face CBT may not be accessible to everyone in need due to social (eg, perceived stigma), organisational (eg, waiting times and service availability) or financial reasons (eg, in healthcare contexts where one must pay for their treatments).28 29

    Digital CBT including CBT-based smartphone applications and web-based interventions may help overcome the above-mentioned barriers. They allow anonymity and accessibility. Also, they can reduce the costs associated with the provision of human resources by providing low-threshold access to evidence-based treatment among those who are in need.30 The existing literature point to mixed results about the effectiveness of these interventions in reducing GAD symptoms among student populations. For instance, a randomised controlled trial (RCT) investigating the effectiveness of a guided, individually tailored web-based CBT programme, ICare Prevent in the Netherlands, indicated no evidence of a difference between the intervention and treatment as usual in anxiety and any of the examined outcomes.31 However, other RCTs investigating the same intervention reported promising results in terms of the feasibility, acceptability and potential effectiveness of ICare Prevent among students with anxiety and/or depression in Germany32 33 South Africa34 and Indonesia.35 Further support for the feasibility and potential effectiveness of digital CBT interventions among students has been provided by other RCTs.36 37

    Despite the obvious benefits of digital CBT, engaging with the intended users can be challenging. The percentage of those who complete the sessions ranges from 10% to 76% and drop-out ranges from 30% to 40%.38–41 Furthermore, an open feasibility trial by Lattie et al 42 indicated that only a relatively small portion of a sample of university students (26.5%; 31/117) downloaded one or more of the CBT-based apps tested in the study, and approximately 24% (28/117) participants used these apps only once.42 Recent systematic reviews pointed out that two intervention methods show promise in mitigating the problem of low engagement with digital interventions.43 44 One approach aims to improve engagement through personalised feedback, customised notifications and reminders.43 An alternative approach is aimed at changing users’ attitudes or behaviours by incorporating game-like strategies such as avatars (ie, virtual therapists) or chatbots initiating conversations with users about specific content and/or exercises.43 44

    Overall, there are mixed results for the effectiveness of digital CBT interventions, and their impact is restricted by the issues related to limited reach, low adoption, implementation barriers and poor long-term maintenance among the intended users.45 Nevertheless, there is a need to test further approaches to address generalised anxiety symptoms among university student populations. Furthermore, there is a significant scope to improve the adherence to digital CBT-based interventions based on the above-mentioned intervention approaches utilising emerging technologies. In light of this, Cerina, a mental health start-up based in the UK, has developed a CBT-based mobile application for treating GAD symptoms which aims to address the unmet need through an interactive user interface (UI) incorporating game-like strategies such as a chatbot, and engagement features such as customised notifications and pop-up reminders. The aims of the proposed study are twofold: (1) to learn about the feasibility (ie, adherence to the intervention, its usability and potential to deliver a full trial in the future) of Cerina among a university student population and (2) to test the preliminary effects of Cerina in reducing GAD symptoms compared with a wait-list control group among Ulster University students presenting with self-reported GAD symptoms.

    Methods

    Setting

    The study will be carried out in partnership between Cerina, a company developing CBT-based mobile applications for treating psychological disorders, and Ulster University as part of the NWE INTERREG IT4Anxiety project. The overarching aim of the IT4Anxiety project is to support the implementation of innovative solutions through start-ups to reduce anxiety. Ulster University has approximately 30 000 students enrolled in undergraduate, postgraduate and e-learning programmes across its four main campuses: Belfast, Coleraine, Derry Londonderry (Magee) and Jordanstown in NI.

    Study design

    The study is a single-blind pilot feasibility RCT with two conditions: treatment and a wait-list control group.

    Eligible participants who have completed the baseline questionnaire will be randomised either into the treatment group in which they have access to the Cerina application for 6 weeks or into the wait-list control group in which they have access to the on-campus well-being services until the intervention group completes Cerina. They will then receive access to Cerina. Please see table 1 for the study design and figure 1 for the participant flow through trial.

    View this table:
    Table 1

    Study design

    Figure 1
    Figure 1

    Participant flow through trial.

    All participants will be asked to fill in online self-report measures of GAD, depression, worry, and work and social adjustment at baseline (t0), at 3 weeks (t1) and at 6 weeks (t2) (see table 2). Additionally, semistructured interviews will be conducted with the participants to explore which elements of the intervention facilitated or hindered participants’ progress during the study and how the intervention can be optimised to increase its acceptability, relevance and user-friendliness in real-life settings. Intervention group participants will be invited to the semistructured interviews right after they complete their follow-up assessments at week 6. Wait-list control group participants will also be invited to the interviews after they access Cerina and complete their 6-week follow-up assessments.

    View this table:
    Table 2

    Measures collected and the time point of data collection for both groups

    Randomisation, treatment allocation and blinding

    The randomisation scheme will be derived using random allocation software by the Cerina technical lead who is not involved in the study. Randomisation will take place in blocks of 6 and will take place in a 1:1 ratio. Allocation will be concealed from the researchers involved in the study. It is not possible to mask participants due to the wait-list control group and the nature of the intervention. All participants will be informed about the condition they are assigned to.

    Participants

    Participants will be adult students attending Ulster University who have reported anxiety symptoms based on the GAD Scale-7 (GAD7) questionnaire.

    Inclusion criteria

    • Having mild to severe self-reported anxiety symptoms. Those who score between 5 and 19 on the GAD7 questionnaire will be accepted as eligible.

    • 18 years of age and older.

    • Enrolled as a student at Ulster University (ie, having a student ID number).

    • Fluency in English.

    • Provision of an informed consent.

    • Have a smartphone (ie, Android device or iPhone).

    • Have an internet connection.

    Exclusion criteria

    • Have minimal anxiety symptoms as defined by a score of 5 and below on the GAD7 questionnaire.

    • Scoring 19 and above on the GAD7 questionnaire.

    • Have self-reported suicidal thoughts based on their scores on the Patient Health Questionnaire-9 (PHQ9).

    • Have recently (within the last 6 weeks) started taking psychotropic medication.

    • Not consenting that their contact details (name, surname, email address) will be shared with the Student Well-being team (see the ethical and safety considerations).

    • Receiving psychological treatment is not part of the exclusion criteria.

    Sample size

    The sample size of the study is 90 participants (N=45 in treatment, N=45 in wait-list control) in total. As this is a pilot RCT study and the objective is mostly to test the feasibility of the trial procedure, and intervention uptake and evaluate the acceptability of the intervention towards building capacity for a larger test population, no formal sample size calculation was performed.46 47 The audit of the sample sizes for the pilot and feasibility RCTs indicated that the median sample size per arm across all the types of studies was 30.48 Browne also recommended that 30 participants per condition are needed to estimate a parameter.49 Taking a possible dropout rate into account, the aim is to include 50% more participants than initially intended in both arms.

    Procedure

    Recruitment will take place at the Ulster University campuses from April 2023 to April 2024.

    Information about the study will be provided on the Cerina website (https://www.cerina.co/) and on the Ulster University Student Well-being team’s website where potential participants can register by filling out the online registration form. Those who do will receive an email with an information sheet, consent form and a link to the baseline questionnaire. Those who are eligible (based on the baseline questionnaire) and who give consent will receive another email explaining which group they have been allocated to. Those who are in the treatment group will receive further instructions on how to download the Cerina app and technical assistance will be provided if needed. Those who are allocated to the wait-list control group will receive an email informing them about their group allocation and their access to the services offered by the Student Well-being team voluntarily while they are waiting to access Cerina.

    Participants will be recruited from the Ulster University student population through several channels as described below.

    The research team will liaise with the Student Well-being team to promote the study within their networks of students through word of mouth and employing banners and/or social media posts on Cerina’s LinkedIn and Instagram accounts, the Ulster University Student Well-being Facebook page and the Ulster University Instagram account. All these communications (ie, emails, banners and social media posts) are prepared by the research team and reviewed by the Student Well-being team and the Ulster University Research Ethics Committee in terms of understandability and appropriateness of the content and the language. They will be circulated within the student email lists and/or posted on relevant social media pages.

    The research team will collaborate with the Student Well-being team and student networks to organise online and/or face-to-face workshops to introduce the study and answer interested students’ questions.

    Finally, the Ulster University Students’ Union will be approached in person and/or using emails to circulate information about the study within their networks.

    Data collection procedures

    The consent forms can be signed either digitally through DocuSign or on paper. If the paper version is signed, the participants will need to email the picture and/or the scanned copy of the consent form to the research team.

    After providing informed consent, participants are informed about their group status via email. All self-report assessments will be administered via email at baseline, 3 weeks and 6 weeks follow-ups except for the System Usability Scale (SUS). SUS will only be assessed at 3 weeks and 6 weeks for intervention group participants. Wait-list group participants will complete SUS at 3 weeks and 6 weeks after their access to Cerina. Qualtrics software is used to generate all self-report assessments including GAD7, PHQ9, Penn State Worry Questionnaire (PSWQ), Work Social Adjustment Scale (WSAS) and SUS. It will take approximately 10–15 min to complete the self-report assessments.

    Intervention group participants will be invited to the semistructured interviews once they complete their 6-week follow-up assessments. Wait-list group participants will be invited to these interviews after accessing Cerina, and once they complete their 6-week follow-up assessments. The data collection for the semistructured interviews will last approximately 40 min, will be remote (online via the Microsoft Teams account of Ulster University) and will be facilitated by the members of the research team. The interviews will be digitally (audio and video) recorded and transcribed verbatim based on participants’ consent. Participants will be able to opt out of the video recording or they could choose to complete an online post-assessment feedback questionnaire (or neither) if they wish to give more in-depth feedback on the usability aspects and the clinical content of Cerina. The online postassessment feedback questionnaire is generated through Qualtrics software (same as other self-report questionnaires used in the study) and includes the same questions as the topic guide. The response format is open-ended to give participants more space to express themselves.

    Patient public involvement

    We have used the NIHR INCLUDE framework50 to ensure the project considers equality, diversity and inclusivity in delivering its objectives. During the product development phase, we ran a series of workshops and interviews with key stakeholders (aged 18–34, N=4) including a product expert, researcher (ie, with expertise in digital interventions), policy expert (ie, with expertise in policies regarding data protection, safety and implementation) and a university student between October 2022 and April 2023. The group highlighted the key priorities to facilitate engagement with the intended users such as customised notifications, chatbots and flexibility to revisit the completed sessions and exercises whenever they want. Their feedback and the usability and/or data protection-related concerns were discussed with Cerina’s technical team and the clinical safety officer. As a result, decisions were made regarding the UI and the engagement features. This codesign process involved identifying intended users’ needs and preferences based on the literature and the previous users’ feedback on Cerina, defining intervention features to address the user needs and preferences, designing content and visual features and testing prototypes in accordance with the recommended guidelines.51 Design and intervention features were incorporated into the final intervention as described below. Outcomes from the codesign workshops and interviews were disseminated to the group via email. The potential burden of the intervention and possible bugs, glitches and/or typos were also assessed by the group and amendments were made accordingly. The group will not be involved in the recruitment and/or conduct of the study.

    The study will be conducted in partnership with Ulster University. Additionally, the research team has a good working relationship with the Ulster University Student Well-being Team. During the design stage of the study, the Ulster University Student Well-being team was engaged extensively. They reviewed the recruitment materials and they have been involved in the key decisions regarding the signposting and risk assessment procedures. The research team will work closely with the Ulster University Student Well-being team during the recruitment process to ensure that students will go through a further assessment with advisors and they will be signposted to the available services whenever needed. Additionally, the partnership with Ulster University will also ensure the reach and access to Ulster University students from across different campuses.

    Finally, all the involved parties (Ulster University, Ulster University Student Well-being team, IT4Anxiety assessment and validation team) will be involved in reviewing the recruitment and signposting processes and dissemination of the findings.

    Intervention

    The intervention is unguided, therefore, it does not provide any professional support related to the therapeutic content of the sessions.52 If participants experience any difficulty with a particular therapeutic technique on the app and/or a technical problem, they are advised to contact the Cerina team at support@cerina.co. If the users experience a distressing situation unrelated to Cerina, they are advised to contact their general practitioner and/or a healthcare professional. Also, there is a self-care page available on the app including links to external resources for further help such as National Health Services helpline, mental health charities, support groups and Samaritans.

    Cerina consists of seven sessions of CBT for the treatment of GAD. Each session contains a range of information and tasks/exercises to help the user understand the condition of GAD, the treatment approach and how it will apply to them. The intervention is based on an evidence-based treatment protocol, hence the sessions will flow from one to the other and the user will complete the sessions in a progressive direction. Participants are advised to complete the first six sessions on a weekly basis. However, they can repeat a session before going on to the next session. The last session is designed as a review of progress session rather than a full therapy session including exercises and homework assignments. Once all sessions have been completed, participants can go back over any of the sessions. There are anxiety management exercises, which the user can go to whenever they wish. There are also a therapy reflection journal and self-care resources including further anxiety management techniques, resources and podcasts that the user will have access to whenever they want.

    A summary of the content of each session is as follows:

    1. Learning about GAD: The first session provides the user with psychoeducation as an introduction to GAD as a mental health condition and will ask them to consider whether they recognise the features in themselves.

    2. Dealing with worry—using the worry tree: The aim of this session is to provide the user with psychoeducation on the types of worries. The user starts conceptualising their worries and making decisions about how to manage different types of worries through the technique called ‘worry tree’.

    3. Managing worry and anxiety: In this session, the user learns about the link between worry and anxiety and they gain a deeper understanding of the vicious cycle (ie, particular situations trigger worry thoughts and they lead to physiological and psychological reactions and noticing those anxious symptoms lead to more worrying). The worry management techniques such as worry postponement, and attention focus training, are also introduced to help the user break this vicious cycle.

    4. Beliefs about worry: This session introduces core beliefs about worrying which perpetuate the worry thoughts. The user learns about the advantages and disadvantages of their beliefs and how helpful they are in their life.

    5. Managing uncertainty: This session helps the user manage uncertainty through problem-solving and imaginal exposure techniques. The user learns to deal with their worries and develops an action plan in a step-by-step manner through problem-solving techniques. The latter helps them to spend less time and energy on their worries gradually through imaginal exposure.

    6. Testing beliefs: This session aims to guide the user to test their beliefs through behavioural experiments. They also learn to do a cost–benefit analysis of worrying to help them order their thinking and be more objective.

    7. Review and therapy blueprint: This session helps the user to review their goals and what they have learnt during the therapy, and to think about how they will take this learning forward to maintain their progress in continuing to manage their GAD symptoms in the future.

    Design features identified in the patient and public involvement phase were included in the intervention to promote user engagement. These include (1) vibrant and simple illustrations to present the case studies in each sessions; (2) customised notification settings; (3) a separate self-care page including podcasts and available resources for further help if needed; (4) regular check-ins to rate mood; (5) embedding a safety mechanism within the user’s conversations with the chatbot. For instance, if a user talks about death and suicidal thoughts, the chatbot provides contact details of the available crisis lines and services for further help and (6) providing more flexibility regarding the navigation. For instance, the users are able to start where they are left off once they log in to the app, and they can go back and review their therapy journey whenever they want.

    Wait-list control condition

    Participants in the wait-list control condition will have access to the on-campus well-being services offered by the Student Well-being team should they wish to use the services. The Student Well-being team on the Belfast, Coleraine and Magee campuses provides free and confidential support and guidance to students with a broad range of issues, concerns and challenges, helping them to successfully engage in their studies. Student Well-being Assistants provide an initial assessment to determine a student’s primary need and then Well-being Advisers, Student Mental Health Advisers and Accessibility Advisers are available to provide a variety of solution-focused interventions based on that need. The Accessibility Advisers also help students with disabilities including diagnosed mental health conditions to get additional support and access funded disability support through Disabled Students’ Allowance. Each campus team is led by an experienced Student Well-being Manager to support the advisers in the management of risk and lead to an efficient and robust response if a Clinical Incident occurs. Additionally, therapeutic counselling support is also available free to students through the external counselling provider via a dedicated 24/7 counselling helpline. The Student Well-being Team works closely with the counselling provider to monitor student engagement and promote the service as appropriate (for further information, please see https://www.ulster.ac.uk/wellbeing/health-and-wellbeing).

    Materials

    Primary outcome measure

    1. Feasibility is defined as adherence to the intervention, its usability and potential to deliver a full trial in the future.53 54

    Adherence is defined based Staudt’s model of adherence55 and it includes behavioural and attitudinal adherence.55 56 Behavioural adherence refers to the participant’s performance on the tasks that are necessary to implement an intervention in daily life such as homework completion.55 56 Attitudinal adherence refers to the emotional investment and commitment to the intervention.55 56 To measure behavioural adherence to Cerina, we will collect data on the number (%) of the login attempts of the participants per week and per month, the number (%) of intervention group participants with at least one session attended, and the number (%) of intervention group participants with desired number of sessions (N=6) completed. Furthermore, to measure attitudinal adherence, we will collect data on the number (%) of the therapeutic content completed by the participants. We will collect data on whether participants complete the core therapeutic content (ie, have entries in their worry diary in session 1 and reflect on their worry diary in session 2), some therapeutic content (ie, psychoeducational part) or no therapeutic content at all (ie, few taps, invalid entries).

    The usability of Cerina will be measured with the SUS57 and with semistructured interviews. An example item: ‘I think I would use Cerina frequently’. The SUS is composed of 10 statements that are scored on a 5-point scale of the extent of agreement (score 0–100). The reliability is good (Cronbach’s alpha 0.91).57 Interventions with scores of 70 and above are accepted as highly usable57 and scores between 50 and 70 indicate acceptable usability of an intervention. Interventions with scores of 50 and below are subject to concerns about their usability by the target population and should be investigated further.57

    In order to investigate the potential to deliver a full scale RCT in the future, we will assess the response rates to the self-report questionnaires, examine the recruitment procedures and participants’ willingness to be randomised to a wait-list control group.53

    Secondary outcome measures

    1. The GAD-758 is a seven-item self-report scale that identifies and measures the severity of GAD. An example item for GAD7: ‘Worrying too much about different things’. Scores range from 0 to 21, with a cut-off score of 5 distinguishing between clinical and non-clinical populations. The scale has good psychometric properties.58

    2. The PSWQ-Past Week (PW) is a 15-item inventory assessing both the weekly status of pathological worry and treatment-related changes of worry during the treatment.59 An example item: ‘If I didn’t have enough time to do everything, I didn’t worry about it’. Each item is scored on a 7-point rating scale, ranging from never 0 (never) to 6 (almost always). The total score ranges from 0 to 90 with a high score indicating more worrying. PSWQ-PW shows good reliability and convergent validity.59

    3. The PHQ-942 is a nine-item self-report scale that measures depression symptoms. An example item: ‘Little interest or pleasure in doing things’. Scores range from 0 to 27, with a score of 10 and above considered to be a clinically significant level of depression. The PHQ-9 has good reliability and validity.60

    4. The WSAS61 is a five-item self-report measure that assesses functional impairment. An example item: ‘Because of my mental health, my ability to work is impaired’. Scores range from 0 to 40. The scale assesses the impact on work, home, social and private activities, and personal and family relationships. A score of 20 and above is considered to indicate severe functional impairment, scores between 10 and 20 suggest severe but functional impairment and scores of 10 and less are considered subclinical. The scale has good reliability and validity.61

    Process evaluation

    Semistructured interviews will be conducted with the participants who complete the intervention and consented to be interviewed. The topic guide for the interview was developed by the research team based on the Consolidated Framework for Implementation Research (CFIR).62 The core topics included participants’ overall experience and perceptions on the usability of the UI of Cerina, existing pathways and barriers to using the app in daily life, the clinical utility of the application, the contexts in which the implementation of the clinical content takes place and the processes of intervention delivery (please see online supplemental file for the topic guide). We will recruit until data saturation,63 with an estimated sample size of 20 participants. Descriptions of the context of each interview will be documented in field notes to support analysis.

    Supplemental material

    Draft interview schedules have been developed and amended by the research team based on the CFIR framework and feedback from the participants from the previous studies. Additionally, they have been reviewed by the Ulster Student Well-being team. The topic guide focuses on:

    • The usability of the intervention (ie, potential barriers and facilitators while using the Cerina app in daily life).

    • Whether the participants relate to the UI and/or content of the intervention.

    • How the clinical content of the intervention help participants understand and manage their anxiety symptoms and which specific exercises help and/or do not help them while they are using the intervention.

    Data analysis plan

    The recruitment and consent rates will be carefully monitored according to the Consolidated Standards of Reporting Trials guidelines. The group allocation procedure will be monitored by the technical lead of Cerina (independent researcher). The response rates to the study questionnaires and the adherence rates will be monitored by the corresponding author.

    The process evaluation will be investigated through semistructured interviews. These interviews will be audiorecorded and transcribed. The transcripts will be analysed thematically and independently by two researchers (one experienced qualitative researcher and one researcher who is involved in the data collection) in NVivo software. The deductive coding will be informed by the CFIR framework.62 Additionally, there will be an inductive coding. Coders will meet each other regularly and the wider study team to discuss codes and begin by identifying and indexing themes relevant to an interview.64 Also, they will explore relationships between descriptive themes.63 The identified themes will be interrogated about each account, as a means of understanding a particular case; compared across cases by highlighting potential similarities and differences between respondents; and, finally, related to those characteristics of the respondent that could be reasonably justified as an explanation which mediated experience.65

    The preliminary effects of Cerina in reducing elevated levels of anxiety and worrying among students will be tested through the linear mixed models in RStudio V.3.6.1.66 We will compare reductions in primary/secondary outcomes between and within groups across two time points in the intention-to-treat sample, we will use linear mixed models in RStudio V.3.6.1.66 This method allows the number of observations to vary between participants and handles missing outcome data. The mixed model uses a longitudinal data structure that includes both fixed and random effects. Time (categorical), group (treatment vs wait-list control), and interactions between group and time will be included as fixed effects in mixed models together with a random intercept and random time effect. Differences in least-squares mean (intervention effects) at each time point with 95% CIs will be derived. Cohen’s d for the effect of the intervention will be estimated by calculating the difference between estimated means (corrected for baseline) divided by raw pooled SD. Effect sizes of 0.8 are accepted as large, effect sizes of 0.5 are moderate and effect sizes of 0.2 are small.67 A two-sided p<0.05 indicates statistical significance. These analyses will be done based on the intention-to-treat sample and missing data will be handled through multiple imputations.

    The reliable change index will also be used to evaluate whether participants have reliable and clinically significant change scores from baseline to postassessment.68

    Ethics and dissemination

    Ethical approval for the study has been granted by the Ulster University Research Ethics Committee (ID: FCPSY-22-084). Cerina includes information on crisis lines in the content of the application. Participants can go to a page where further information on self-care resources and help-seeking are provided within the application. Additionally, receiving psychological treatment is not exclusion criteria and all participants are signposted to the services offered by the Student Well-being team, their GP and/or additional resources (eg, crisis lines, Lifeline) and are encouraged to use these services regularly through online communication throughout their participation. Eligible participants also need to agree on sharing their student ID numbers with the Student Well-being team. This is necessary for two main reasons: (1) to signpost participants to the Student Well-being when needed and (2) to be able to monitor the use of additional psychological services among all participants throughout the study.

    Interested participants who are currently on psychotropic medication or presenting minimal and/or severe anxiety symptoms (as measured by the GAD7) and/or reporting suicidal thoughts during the baseline assessment will receive an email about their results and will be informed that Cerina is not designed to meet their needs and they will be signposted to the Student Well-being team for further help. In that email, the crisis lines and resources for further help will be provided.

    Another potential risk is that some participants might get worse while they are using Cerina. The existing studies testing CBT-based mobile applications in treating GAD symptoms indicate that this is a rare risk.19 47 We will monitor this risk based on participants’ responses to the online questionnaires. We will also monitor their answers to the regular check-in questions asking about their mood whenever they log in to the app. If participants’ scores are getting worse and/or their check-in questions indicate that they might be feeling low lately, they will receive a notification on the app directing them to the self-care page, anxiety management exercises and/or the resources pages for further help. They will also receive an option as a notification, to be referred to the Student Well-being team for further risk assessment. If they choose this option, the research team will notify the Student Well-being team to schedule a meeting with them for risk assessment and further help.

    The results of the study will be disseminated through publications in scientific journals, presentations at relevant congresses, conferences and/or public events.

    Findings to date

    As of 31 March 2024, we have recruited 158 (N=79 intervention; N=79 wait-list control group) participants. In the intervention group, a total of 29 participants completed their 3-week assessments, and a total of 47 participants completed their 6-week postassessments. In the wait-list control group, a total of 43 participants completed their 3-week assessments, and a total of 56 participants completed their 6-week assessments. Additionally, we have conducted 4 interviews with the participants and a total of 16 participants opted out of the interviews but completed postassessment feedback questionnaires.

    Discussion

    The current paper describes a study protocol of a pilot feasibility RCT with two main objectives: (1) to learn about the feasibility (ie, adherence to the intervention, its usability and potential to deliver a full trial in the future) of delivering Cerina among a student population and (2) to test the preliminary effects of Cerina in reducing GAD symptoms compared with a wait-list control group among Ulster University students presenting with self-reported GAD symptoms.

    One of the strengths of this study is that it proposes to test a low-threshold CBT-based mobile application among Ulster University student population who have low receipt of on-campus and external psychological support services despite their vulnerability to mental health problems.23 Digital CBT-based interventions aim to provide easy access, anonymity, reduced costs and stigma attached to help-seeking.30 However, there are mixed results on the effectiveness of these interventions among student populations.31–37 Additionally, recent studies point to low engagement with CBT-based mobile applications in particular among student populations despite the assumed benefits.42 Ciharova et al 69 investigated the reasons for drop-out from the ICare Prevent intervention among university students in the Netherlands.69 There were several person-related and intervention-related reasons for drop-out most commonly, lengthy sessions colliding with students’ busy academic schedules, needs for different kinds of help and/or absence of personal contact.69 Although Cerina is an unguided intervention, we expect that the design features such as customised notification settings, flexibility in navigation and the chatbot function will address some of the key person-related and intervention-related drop-out reasons identified by Ciharova et al.69 Therefore, this study will provide an opportunity to contribute to the growing evidence base on the role of emerging technology-based engagement features, and their impact on the effectiveness of digital CBT-based interventions.

    Another strength of this study is that it is based on a partnership and a collaboration between Ulster University, Ulster University Student Well-being team and Cerina, a mental health start-up. All these parties have been involved in making key decisions about the recruitment strategies, risk assessment and signposting procedures as well as dissemination plans. A recent study by Lattie et al 42 highlighted the digital overload among university students during and postpandemic and the ineffectiveness of email and social media communication strategies during the recruitment process. Additionally, they note the disruption caused by the pandemic had a negative impact on the communications between the research team and the student well-being staff as the staff members were overburdened by competing priorities.42 In the current study, we expect that the established partnership and the utilisation of various recruitment channels will help reach and access the intended population, and such challenges specific to the pandemic context will not pose problems.

    The limitations of the current study must also be acknowledged. First of all, the study is designed as a feasibility trial and it is not powered to detect statistically significant effects of the intervention.46 47 Second, although the study participants will be recruited from across the four campuses of Ulster University, the results might not generalise beyond the inclusion and exclusion criteria of the current study. Third, the wait-list control group has been chosen as a comparison group in the current study to offer the possibility of completing the intervention to a vulnerable population who might not have access to any form of psychological help otherwise. It is known that wait-list control groups motivate people to participate in the control group and the trial and lead to high expectations of the effects of the intervention.70 It is also known that high expectations result in better outcomes of interventions. Therefore, the wait-list control group may considerably overestimate the effects of an intervention in the current study.70

    To conclude, this study has several strengths which will contribute to the literature about digital CBT-based interventions for treating generalised anxiety symptoms among university students. These are established partnerships with the relevant stakeholders strengthening the reach and access to the intended population, engagement features which are incorporated into the Cerina intervention based on the codesign principles and low-intensity CBT-based mobile application targeting Ulster University students.

    Ethics statements

    Patient consent for publication

    Acknowledgments

    The authors would like to acknowledge Omkar Kadam (Cerina Head of Innovation), Priyadharshini Rajaram (Cerina Technical Product Manager) and Charlotte Monnickendam (Cerina Head of Partnerships) for their work and dedication to product development and innovation. They also wish to thank the Cerina Board of Directors Andrew Vallance Owen, Anoop Aggarwal and rest of the advisors for their continuous support for Cerina’s evidence-based approach. Finally, the authors would like to thank the stakeholder group who is involved in the product development.

    References

    1. ↵
      1. World Health Organisation
      . Anxiety disorders, Available: https://www.who.int/news-room/fact-sheets/detail/anxiety-disorders
    2. ↵
      2. Ruscio AM ,
      3. Hallion LS ,
      4. Lim CCW , et al
      . Cross-sectional comparison of the epidemiology of DSM-5 generalized anxiety disorder across the globe. JAMA Psychiatry 2017;74:465–75. doi:10.1001/jamapsychiatry.2017.0056
      OpenUrl
    3. ↵
      2. Santomauro DF ,
      3. Mantilla Herrera AM ,
      4. Shadid J
      . Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. Lancet 2021;398:1700–12. doi:10.1016/S0140-6736(21)02143-7
      OpenUrlCrossRefPubMed
    4. ↵
      2. Cao H ,
      3. Wu Y ,
      4. Yin H , et al
      . Global trends in the incidence of anxiety disorders from 1990 to 2019: Joinpoint and age-period-cohort analysis study. JMIR Public Health Surveill 2024;10:e49609. doi:10.2196/49609
    5. ↵
      2. Szuhany KL ,
      3. Simon NM
      . Anxiety disorders: A review. JAMA 2022;328:2431–45. doi:10.1001/jama.2022.22744
      OpenUrl
    6. ↵
      2. Alonso J ,
      3. Liu Z ,
      4. Evans-Lacko S , et al
      . Treatment gap for anxiety disorders is global: results of the world mental health surveys in 21 countries. Depress Anxiety 2018;35:195–208. doi:10.1002/da.22711
      OpenUrlCrossRefPubMed
    7. ↵
      2. Auerbach R ,
      3. Mortier P ,
      4. Bruffaerts R , et al
      . Student project: prevalence and distribution of mental disorders. J Abnorm Psychol 2019;127:623–38. doi:10.1037/abn0000362
      OpenUrl
    8. ↵
      2. Xiao H ,
      3. Carney DM ,
      4. Youn SJ , et al
      . Are we in crisis? National mental health and treatment trends in college counseling centers. Psychol Serv 2017;14:407–15. doi:10.1037/ser0000130
      OpenUrl
    9. ↵
      2. McLafferty M ,
      3. Lapsley CR ,
      4. Ennis E , et al
      . Mental health, behavioural problems and treatment seeking among students commencing University in Northern Ireland. PLoS One 2017;12:e0188785. doi:10.1371/journal.pone.0188785
    10. ↵
      2. Halperin SJ ,
      3. Henderson MN ,
      4. Prenner S , et al
      . Prevalence of anxiety and depression among medical students during the COVID-19 pandemic: A cross-sectional study. J Med Educ Curric Dev 2021;8:2382120521991150. doi:10.1177/2382120521991150
    11. ↵
      2. Huang Y ,
      3. Zhao N
      . Generalized anxiety disorder, depressive symptoms and sleep quality during COVID-19 outbreak in China: a web-based cross-sectional survey. Psychiatry Res 2020;288:112954. doi:10.1016/j.psychres.2020.112954
      OpenUrlCrossRefPubMed
    12. ↵
      2. Voltmer E ,
      3. Köslich-Strumann S ,
      4. Walther A , et al
      . The impact of the COVID-19 pandemic on stress, mental health and coping behavior in German university students – a longitudinal study before and after the onset of the pandemic. BMC Public Health 2021;21:1385. doi:10.1186/s12889-021-11295-6
    13. ↵
      2. Saraswathi I ,
      3. Saikarthik J ,
      4. Senthil Kumar K , et al
      . Impact of COVID-19 outbreak on the mental health status of undergraduate medical students in a COVID-19 treating medical college: A prospective longitudinal study. PeerJ 2020;8:e10164. doi:10.7717/peerj.10164
    14. ↵
      2. Reuter PR ,
      3. Forster BL ,
      4. Kruger BJ
      . A longitudinal study of the impact of COVID-19 restrictions on students’ health behavior, mental health and emotional well-being. PeerJ 2021;9:e12528. doi:10.7717/peerj.12528
    15. ↵
      2. Jones NL ,
      3. Gilman SE ,
      4. Cheng TL , et al
      . Life course approaches to the causes of health disparities. Am J Public Health 2019;109:S48–55. doi:10.2105/AJPH.2018.304738
      OpenUrlCrossRefPubMed
    16. ↵
      2. Schulenberg JE ,
      3. Sameroff AJ ,
      4. Cicchetti D
      . The transition to adulthood as a critical juncture in the course of psychopathology and mental health. Dev Psychopathol 2004;16:799–806. doi:10.1017/s0954579404040015
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      2. Arnett JJ
      . College students as emerging adults: the developmental implications of the college context. Emerg Adulthood 2016;4:219–22. doi:10.1177/2167696815587422
      OpenUrl
    18. ↵
      2. Pei J ,
      3. Amanvermez Y ,
      4. Vigo D , et al
      . Sociodemographic correlates of mental health treatment seeking among college students: A systematic review and meta-analysis. Psychiatr Serv 2024;75:556–69. doi:10.1176/appi.ps.20230414
      OpenUrl
    19. ↵
      2. Acharya L ,
      3. Jin L ,
      4. Collins W
      . College life is stressful today–emerging stressors and depressive symptoms in college students. J Am Coll Health 2018;66:655–64. doi:10.1080/07448481.2018.1451869
      OpenUrl
    20. ↵
      2. de Girolamo G ,
      3. Dagani J ,
      4. Purcell R , et al
      . Age of onset of mental disorders and use of mental health services: needs, opportunities and obstacles. Epidemiol Psychiatr Sci 2012;21:47–57. doi:10.1017/s2045796011000746
      OpenUrlCrossRefPubMed
    21. ↵
      2. Lagdon S ,
      3. Ross J ,
      4. Robinson M , et al
      . Assessing the mediating role of social support in childhood Maltreatment and psychopathology among college students in Northern Ireland. J Interpers Violence 2021;36:2112–2136NP. doi:10.1177/0886260518755489
      OpenUrl
    22. ↵
      2. Travers Á ,
      3. Armour C ,
      4. Hansen M , et al
      . Lesbian, gay or Bisexual identity as a risk factor for trauma and mental health problems in northern Irish students and the protective role of social support. Eur J Psychotraumatol 2020;11. doi:10.1080/20008198.2019.1708144
    23. ↵
      2. Ennis E ,
      3. McLafferty M ,
      4. Murray E , et al
      . Readiness to change and barriers to treatment seeking in college students with a mental disorder. J Affect Disord 2019;252:428–34. doi:10.1016/j.jad.2019.04.062
      OpenUrl
    24. ↵
      2. Ebert DD ,
      3. Mortier P ,
      4. Kaehlke F , et al
      . Barriers of mental health treatment utilization among first-year college students: first cross-national results from the WHO world mental health International college student initiative. Int J Methods Psychiatr Res 2019;28:e1782. doi:10.1002/mpr.1782
    25. ↵
      2. Dunbar MS ,
      3. Sontag-Padilla L ,
      4. Ramchand R , et al
      . Mental health service utilization among Lesbian, gay, Bisexual, and questioning or queer college students. J Adolesc Health 2017;61:294–301. doi:10.1016/j.jadohealth.2017.03.008
      OpenUrlCrossRefPubMed
    26. ↵
      1. NICE
      . Generalised anxiety disorder and panic disorder in adults (CG133);(April 2007):1–928, 2020. Available: www.nice.org.uk/guidance/cg113
    27. ↵
      2. Cuijpers P ,
      3. Miguel C ,
      4. Harrer M , et al
      . Cognitive behavior therapy vs. control conditions, other Psychotherapies, Pharmacotherapies and combined treatment for depression: a comprehensive meta-analysis including 409 trials with 52,702 patients. World Psychiatry 2023;22:105–15. doi:10.1002/wps.21069
      OpenUrl
    28. ↵
      2. Cuijpers P ,
      3. Eylem O ,
      4. Karyotaki E , et al
      . Psychotherapy for depression and anxiety in Low- and middle-income countries. In: Global Mental Health and Psychotherapy: Adapting Psychotherapy for Low- and Middle-Income Countries. Elsevier Inc, 2019: 173–92. Available: http://dx.doi.org/10.1016/B978-0-12-814932-4.00008-2
    29. ↵
      2. Kelsey C ,
      3. Martin Mhatre V ,
      4. Ho J-A , et al
      . NIH public access. Bone 2012;23:1–7.
      OpenUrlCrossRef
    30. ↵
      2. Linardon J ,
      3. Fuller-Tyszkiewicz M
      . Attrition and adherence in Smartphone-delivered interventions for mental health problems: A systematic and meta-analytic review. J Consult Clin Psychol 2020;88:1–13. doi:10.1037/ccp0000459
      OpenUrlCrossRefPubMed
    31. ↵
      2. Karyotaki E ,
      3. Klein AM ,
      4. Ciharova M , et al
      . Guided Internet-based Transdiagnostic individually tailored cognitive behavioral therapy for symptoms of depression and/or anxiety in college students: A randomized controlled trial. Behav Res Ther 2022;150:S0005-7967(21)00227-8. doi:10.1016/j.brat.2021.104028
    32. ↵
      2. Weisel KK ,
      3. Zarski A-C ,
      4. Berger T , et al
      . Efficacy and cost-effectiveness of guided and unguided Internet- and mobile-based indicated Transdiagnostic prevention of depression and anxiety (Icare prevent): A three-armed randomized controlled trial in four European countries. Internet Interventions 2019;16:52–64. doi:10.1016/j.invent.2018.04.002
      OpenUrl
    33. ↵
      2. Weisel KK ,
      3. Zarski A-C ,
      4. Berger T , et al
      . User experience and effects of an individually tailored Transdiagnostic Internet-based and mobile-supported intervention for anxiety disorders: mixed-methods study. J Med Internet Res 2020;22:e16450. doi:10.2196/16450
    34. ↵
      2. Gericke F ,
      3. Ebert DD ,
      4. Breet E , et al
      . A qualitative study of University students’ experience of Internet-based CBT for depression. Couns and Psychother Res 2021;21:792–804. doi:10.1002/capr.12465
      OpenUrl
    35. ↵
      2. Rahmadiana M ,
      3. Karyotaki E ,
      4. Schulte M , et al
      . Transdiagnostic Internet intervention for Indonesian university students with depression and anxiety: evaluation of feasibility and acceptability. JMIR Ment Health 2021;8:e20036. doi:10.2196/20036
    36. ↵
      2. Lee RA ,
      3. Jung ME
      . Evaluation of an Mhealth App (Destressify) on university students-mental health: pilot trial. JMIR Ment Health 2018;5:e2. doi:10.2196/mental.8324
    37. ↵
      2. Becker TD ,
      3. Torous JB
      . Recent developments in Digital mental health interventions for college and university students. Curr Treat Options Psych 2019;6:210–20. doi:10.1007/s40501-019-00178-8
      OpenUrl
    38. ↵
      2. Carl JR ,
      3. Miller CB ,
      4. Henry AL , et al
      . Efficacy of Digital cognitive behavioral therapy for moderate-to-severe symptoms of generalized anxiety disorder: A randomized controlled trial. Depress Anxiety 2020;37:1168–78. doi:10.1002/da.23079
      OpenUrl
    39. ↵
      2. Miller CB ,
      3. Gu J ,
      4. Henry AL , et al
      . Feasibility and efficacy of a Digital CBT intervention for symptoms of generalized anxiety disorder: A randomized multiple-baseline study. J Behav Ther Exp Psychiatry 2021;70:S0005-7916(19)30269-1. doi:10.1016/j.jbtep.2020.101609
    40. ↵
      2. Andersson E ,
      3. Enander J ,
      4. Andrén P , et al
      . Internet-based cognitive behaviour therapy for obsessive-compulsive disorder: A randomized controlled trial. Psychol Med 2012;42:2193–203. doi:10.1017/S0033291712000244
      OpenUrlCrossRefPubMed
    41. ↵
      2. Paxling B ,
      3. Almlöv J ,
      4. Dahlin M , et al
      . Guided Internet-delivered cognitive behavior therapy for generalized anxiety disorder: A randomized controlled trial. Cogn Behav Ther 2011;40:159–73. doi:10.1080/16506073.2011.576699
      OpenUrlPubMed
    42. ↵
      2. Lattie EG ,
      3. Cohen KA ,
      4. Hersch E , et al
      . Uptake and effectiveness of a self-guided mobile App platform for college student mental health. Internet Interv 2022;27:100493. doi:10.1016/j.invent.2021.100493
      OpenUrl
    43. ↵
      2. Borghouts J ,
      3. Eikey E ,
      4. Mark G , et al
      . Barriers to and Facilitators of user engagement with Digital mental health interventions: systematic review. J Med Internet Res 2021;23:e24387. doi:10.2196/24387
    44. ↵
      2. Gan DZQ ,
      3. McGillivray L ,
      4. Larsen ME , et al
      . Technology-supported strategies for promoting user engagement with Digital mental health interventions: A systematic review. Digit Health 2022;8:20552076221098268. doi:10.1177/20552076221098268
      OpenUrl
    45. ↵
      2. Ramos G ,
      3. Hernandez-Ramos R ,
      4. Taylor M , et al
      . State of the science: using Digital mental health interventions to extend the impact of psychological services. Behavior Therapy 2024. doi:10.1016/j.beth.2024.04.004
    46. ↵
      2. Cooper CL ,
      3. Whitehead A ,
      4. Pottrill E , et al
      . Are pilot trials useful for predicting Randomisation and attrition rates in definitive studies: A review of publicly funded trials. Clin Trials 2018;15:189–96. doi:10.1177/1740774517752113
      OpenUrlPubMed
    47. ↵
      2. Lancaster GA ,
      3. Thabane L
      . Guidelines for reporting non-randomised pilot and feasibility studies. Pilot Feasibility Stud 2019;5:114. doi:10.1186/s40814-019-0499-1
    48. ↵
      2. Billingham SAM ,
      3. Whitehead AL ,
      4. Julious SA
      . An audit of sample sizes for pilot and feasibility trials being undertaken in the United Kingdom registered in the United Kingdom clinical research network database. BMC Med Res Methodol 2013;13:2–7:104. doi:10.1186/1471-2288-13-104
      OpenUrlCrossRefPubMed
    49. ↵
      2. Browne RH
      . On the use of a pilot sample for sample size determination. Stat Med 1995;14:1933–40. doi:10.1002/sim.4780141709
      OpenUrlCrossRefPubMed
    50. ↵
      1. National Institute for Health and Care Research
      . Improving inclusion of under-served groups in clinical research: guidance from INCLUDE project. n.d. Available: https://www.nihr.ac.uk/documents/improving-inclusion-of-under-served-groups-in-clinical-research-guidance-from-include-project/25435
    51. ↵
      2. Li SH ,
      3. Achilles MR ,
      4. Spanos S , et al
      . A cognitive behavioural therapy Smartphone App for adolescent depression and anxiety: Co-design of Clearlyme. tCBT 2022;15. doi:10.1017/S1754470X22000095
    52. ↵
      2. Andersson G ,
      3. Carlbring P ,
      4. Titov N , et al
      . Internet interventions for adults with anxiety and mood disorders: A narrative umbrella review of recent meta-analyses. Can J Psychiatry 2019;64:465–70. doi:10.1177/0706743719839381
      OpenUrlPubMed
    53. ↵
      2. Eylem O ,
      3. van Straten A ,
      4. de Wit L , et al
      . Reducing suicidal Ideation among Turkish migrants in the Netherlands and in the UK: the feasibility of a randomised controlled trial of a guided online intervention. Pilot Feasibility Stud 2021;7:30. doi:10.1186/s40814-021-00772-9
    54. ↵
      2. Arshad U Farhat-Ul-Ain
      3. Gauntlett J , et al
      . A systematic review of the evidence supporting Mobile- and Internet-based psychological interventions for self-harm. Suicide Life Threat Behav 2020;50:151–79. doi:10.1111/sltb.12583
      OpenUrlPubMed
    55. ↵
      2. Staudt M
      . Treatment engagement with Caregivers of at-risk children: gaps in research and conceptualization. J Child Fam Stud 2007;16:183–96. doi:10.1007/s10826-006-9077-2
      OpenUrl
    56. ↵
      2. Hanano M ,
      3. Rith-Najarian L ,
      4. Boyd M , et al
      . Measuring adherence within a self-guided online intervention for depression and anxiety: secondary analyses of a randomized controlled trial. JMIR Ment Health 2022;9:e30754. doi:10.2196/30754
    57. ↵
      2. Bangor A ,
      3. Kortum PT ,
      4. Miller JT
      . An empirical evaluation of the system usability scale. Int J Human-Computer Interact 2008;24:574–94. doi:10.1080/10447310802205776
      OpenUrl
    58. ↵
      2. Spitzer RL ,
      3. Kroenke K ,
      4. Williams JBW
      . Validation and utility of a self-report version of PRIME-MD. Prim Care Companion J Clin Psychiatry 2000;2:31.
    59. ↵
      2. Stöber J ,
      3. Bittencourt J
      . Weekly assessment of worry: an adaptation of the Penn state worry questionnaire for monitoring changes during treatment. Behav Res Ther 1998;36:645–56. doi:10.1016/s0005-7967(98)00031-x
      OpenUrlCrossRefPubMedWeb of Science
    60. ↵
      2. Spitzer RL ,
      3. Kroenke K ,
      4. Williams JBW
      . Validation and utility of a self-report 563 version of PRIME-MD. Prim Care Companion J Clin Psychiatry 2000;2:31.
    61. ↵
      2. Mundt JC ,
      3. Marks IM ,
      4. Shear MK , et al
      . The work and social adjustment scale: A simple measure of impairment in functioning. Br J Psychiatry 2002;180:461–4. doi:10.1192/bjp.180.5.461
      OpenUrlAbstract/FREE Full Text
    62. ↵
      2. Birken SA ,
      3. Powell BJ ,
      4. Presseau J , et al
      . Combined use of the Consolidated framework for implementation research (CFIR) and the theoretical domains framework (TDF): A systematic review. Implement Sci 2017;12:2. doi:10.1186/s13012-016-0534-z
    63. ↵
      2. Hennink M ,
      3. Kaiser BN
      . Sample sizes for saturation in qualitative research: A systematic review of empirical tests. Soc Sci Med 2022;292:114523. doi:10.1016/j.socscimed.2021.114523
      OpenUrlCrossRefPubMed
    64. ↵
      2. Braun V ,
      3. Clarke V
      . Using thematic analysis in psychology. Qualit Res Psychol 2006;3:77–101. doi:10.1191/1478088706qp063oa
      OpenUrlCrossRef
    65. ↵
      2. Braun V ,
      3. Clarke V ,
      4. Rance N
      . How to use thematic analysis with interview data (process research). In: Moller NP , Vossler A , eds. The Counselling & Psychotherapy Research Handbook. Sage, 2014.
    66. ↵
      1. R Core Team
      . R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Available: https://www.R-project.org
    67. ↵
      2. Cohen J
      . Statistical Power Analysis for the Behavioral Sciences. Routledge, 2013. Available: https://www.taylorfrancis.com/books/9781134742707
    68. ↵
      2. Jacobson NS ,
      3. Truax P
      . Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12–9. doi:10.1037//0022-006x.59.1.12
      OpenUrlCrossRefPubMedWeb of Science
    69. ↵
      2. Ciharova M ,
      3. Cuijpers P ,
      4. Amanvermez Y , et al
      . Use of Tailoring features and reasons for dropout in a guided Internet-based Transdiagnostic individually-tailored cognitive behavioral therapy for symptoms of depression and/or anxiety in college students. Internet Interv 2023;34:100646. doi:10.1016/j.invent.2023.100646
    70. ↵
      2. Tønning ML ,
      3. Kessing LV ,
      4. Bardram JE , et al
      . Methodological challenges in randomized controlled trials on Smartphone-based treatment in psychiatry: systematic review. J Med Internet Res 2019;21:e15362. doi:10.2196/15362

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Contributors OE-vB had the idea for the study, and OE-vB and GL designed the study. SP, TR, PM and JC contributed to the conception of the work. MI wrote the treatment protocol for the intervention. KA and TR contributed to the literature search. All authors read and contributed to the revisions and the final paper.

    • Funding This work was conducted as part of the NWE INTERREG IT4Anxiety project in partnership between Cerina Health and Ulster University. The project is supported by the European Commission. The sponsor was not involved in writing up the manuscript and/or in the decision to submit it for the publication.

    • Competing interests None declared.

    • Patient and public involvement Patients and/or the public were involved in the design, or conduct, or reporting, or dissemination plans of this research. Refer to the Methods section for further details.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.