Skip to main content

Review of cognitive behavioural therapy mobile apps using a reference architecture embedded in the patient-provider relationship



Mobile health apps (mHealth apps) are increasing in popularity and utility for the management of many chronic diseases. Although the current reimbursement structure for mHealth apps is lagging behind the rapidly improving functionality, more clinicians will begin to recommend these apps as they prove their clinical worth. Payors such as the government or private insurance companies will start to reimburse for the use of these technologies, especially if they add value to patients by providing timely support, a more streamlined patient experience, and greater patient convenience. Payors are likely to see benefits for providers, as these apps could help increase productivity between in-office encounters without having to resort to expensive in-person visits when patients are having trouble managing their disease.

Key findings

To guide and perhaps speed up adoption of mHealth apps by patients and providers, analysis and evaluation of existing apps needs to be carried out and more feedback must be provided to app developers. In this paper, an evaluation of 35 mHealth apps claiming to provide cognitive behavioural therapy was conducted to assess the quality of the patient-provider relationship and evidence-based practices embedded in these apps. The mean score across the apps was 4.9 out of 20 functional criteria all of which were identified as important to the patient-provider relationship. The median score was 5 out of these 20 functional criteria.


Overall, the apps reviewed were mostly stand-alone apps that do not enhance the patient-provider relationship, improve patient accountability or help providers support patients more effectively between visits. Large improvements in patient experience and provider productivity can be made through enhanced integration of mHealth apps into the healthcare system.


Cognitive behavioral therapy (CBT) is an evidence-based therapeutic approach used to treat psychological distress and a variety of mental disorders [1]. This therapy aims to modify maladaptive cognitions that lead to distress and problematic behaviours, thereby reducing negative symptoms and improving functioning [2]. CBT has been shown to produce large effect size improvements for treatment of mental health disorders, such as anxiety and depression [3]. CBT can be paired with pharmaceutical treatments to improve outcomes and has proven to be more effective than antidepressants when used for the treatment of depression in adults [3].

One method of CBT delivery that has been proven effective is internet-based CBT (iCBT), which has led to symptom reduction in both small and large effect sizes [4]. In this treatment method, a licensed therapist supports patients through online messaging platforms, e-mail, or web pages and provides them with exercises and behavioural intervention programs [5, 6]. iCBT has been identified as a plausible alternative to traditional CBT for patients with depression; helping to improve patient outcomes [7]. Randomized controlled trials have also shown that therapist-assisted iCBT is comparable to face-to-face CBT [8, 9], even when considering regarding development of strong patient-provider relationship [10].

With this shift towards alternative delivery methods for mental health therapies, an increasing number of mobile health (mHealth) apps in the mobile marketplace have emerged claiming to provide CBT. In contrast with iCBT, mHealth CBT apps tend to be self-guided and it is unknown if these apps effectively implement the evidence-based principles of CBT [11,12,13,14]. Additionally, there is little evidence demonstrating that these CBT apps can be recommended for unsupervised self-management [15]. The small existing evidence base is further exacerbated by the fast pace of technology relative to the pace of research and evaluation of mHealth apps [16]. Further research is required to better understand the mHealth CBT apps marketplace, particularly related to the effect on patient-provider relationships [17]. In addition, while research demonstrates patient interest in using mHealth apps for self-management, clinician interaction and health system integration of app has been identified as an important factor for patient confidence and ultimate behaviour change [18].

The purpose of this paper is to apply an mHealth app evaluation framework to CBT mHealth apps, to better understand the current marketplace for CBT mHealth apps, focusing primarily on the presence of functionalities to support patient-provider relationships. Specifically, this paper will focus on apps targeted towards adults with depression and/or anxiety.

Framework development

An evaluation framework was developed to evaluate the quality of the patient-provider relationship in CBT mHealth apps based on a reference architecture for health app design [19], (see Table 1). The evaluation framework is composed of 20 measures aimed at measuring the evidence-based support of CBT mHealth apps and their ability to enhance the patient-provider relationship. These 20 measures were based on properties from Chindalo et al. reference architecture which distinguishes features such as explicitly identifying patient’s diagnosis, enabling interoperability with EMRs, identifying and tracking process and proxy metrics for diseases as well as identifying and tracking important outcome measures [19]. These concepts fit with Albrecht et al. framework which provides details on evidence-based criteria that should be considered when evaluating mobile applications [20]. The framework also identifies features that are based on the patient engagement framework created by Balouchi et al. which focuses on functionalities of mobile apps that enhance the patient-provider relationship [21]. The rationale for the methodology is to provide a perspective on the experience of general users and clinicians when identifying mHealth apps for the purpose of CBT.

Table 1 Ranking of functionalities

The final list of measures was developed with an experienced clinician (KK) and took into account the information needed to provide high quality clinical care to a patient requiring CBT. The measures developed were customized for the treatment of mental health disorders, such as depression and anxiety; diseases that respond to CBT. Although some of the measures can be used for evaluating other disease types, the set of measures developed for CBT are only appropriate for mental health and related disorders.


50 CBT mHealth apps were identified from the Apple iTunes and Google Play app stores using the search terms “Cognitive Behavioural Therapy” or “CBT.” The rationale for the use of the health app design reference architecture over other popular frameworks used for mHealth App reviews is described previously [19].

Each app was downloaded and screened independently against 20 functional measures by two reviewers. Each measure was scored on a binary scale (0,1). Apps received a score of 1 if they had at least one attribute of that measure. To generate an evaluation score for each app, the sum of the binary measures was taken. The agreement between scores was determined after a blind independent review. The agreement between scores was completed by examining the number of scores the reviewers agreed on divided by the total number of functions in the framework. The mean evaluation score was calculated and used for analysis.

Before screening initiation, a calibration exercise was carried out with five randomly selected mHealth apps, which were evaluated by six reviewers. The calibration allowed the areas of discrepancies in interpretation of measures to be surfaced and addressed and improved the standardization of the approach. All reviewers were trained in the standardized method, and each of the 50 apps was evaluated by two independent reviewers.

Reviewers gave their ratings and included descriptions justifying their decision for each measure. After evaluations were complete, all data were collated into a single spreadsheet. Before data analysis, 15 apps identified by reviewers were excluded as they did not claim to provide CBT and offered other functions unrelated to the patient-provider relationship. Reviewers downloaded the app and scored them using the standardized method. Each app was independently and blindly screened against the evaluation criteria. For each of the measures, the higher score between the two reviewers was accepted, and final scores were generated for each app. The complete list of apps downloaded can be found in Appendix 1.


The mean evaluation score across the 35 apps was 4.9 out of 20 functional criteria. The median score was 5. The two highest apps met 11 out of 20 functional criteria. The lowest app met 2 out of 20 functional criteria (see Fig. 1).

Fig. 1
figure 1

Distribution of app evaluation scores

Overall, the apps scored well on functions including education and recommendations, user interface, and behaviour tracking functional criteria (See Table 1). Primarily, these criteria were met through the provision of education about CBT techniques and how they may reduce patient symptoms. Apps generally scored poorly on criteria including physiological measurement, patient health information collection, lab results, medication or comorbidities as well as health system integration and utilization; all of which may be important for management of patient with mental health disorders.


While recent literature suggests the potential of mHealth apps to improve care accessibility and decrease depression levels in users, findings from this research suggest that the current marketplace for mHealth apps is limited in its ability to provide benefits for the patient-provider relationship [12, 13]. Overall, our research found that the mHealth apps in the marketplace primarily act only as symptom trackers or educational resources with little integration into the larger healthcare system (see Fig. 2).

Fig. 2
figure 2

App evaluation scores after download

While apps overall did not score high on the evaluation framework, particularly in regards to healthcare integration, it should be noted that apps that perform only one core function may still provide some benefit to users. For instance, one empirical study reported the use of CBT-based depression apps are especially useful when they provide mood prediction; demonstrating the potential benefits of apps containing this feature alone [22]. Since our criteria were used to evaluate overall prevalence of functions as well as market gaps and opportunities, effectiveness of the individual functions was not taken into consideration.

Overall, by not providing healthcare integration, the apps under review did not provide opportunities for ensuring patient accountability and presented very little opportunity for use by healthcare providers. In addition, this lack of integration with providers and the healthcare system as a whole may limit the effectiveness of these apps in supporting sustained behaviour change [18]. It has been argued that mHealth apps should not be designed for healthcare provider use, and instead their main purpose is for patient empowerment outside the provider-patient relationship, suggesting their utility despite lack of integration. For instance, recent studies have found that mHealth apps may be useful and effective when used for self-monitoring and providing support for patients who are interested in self-treatment [23]. Therefore, apps that scored low on our evaluation criteria may present utility for highly motivated patients who are self-starters. Additional areas of improvement identified for the apps include more meaningful use of data collected, a stronger evidence base, and the ability to send notifications.

Identified limitations of the study are as follows: (1) the research team was not able to establish how often apps were used, or by which populations; (2) no patient representatives were included in the creation of the evaluation framework nor the reviewing of the individual apps. In future iterations, the inclusion of patients would improve the quality of data collected. These limitations can inform future research to collect data on the users of these apps to draw more insights on how often the apps were used and the types of users and their likelihood to have better patient outcomes.


Overall, there is a lack of evidence-based information and integration that enhance the patient-provider relationship in the CBT mobile app marketplace. Many apps only perform one function, mainly for patient engagement, and lack the functionality necessary to help patients adhere to their treatment within the larger health system. App developers should take note of the importance of evidence-based functionalities to improve patient outcomes which would encourage insurers and payers to begin to reimburse for the use of these technologies. Integration and connectivity with clinicians may facilitate improved app desirability and performance.



cognitive behavioral therapy


internet-based cognitive behavioral therapy


patient reported experience measures


patient reported outcome measures


  1. Berking M, Whitley B. Overview of current treatments that enhance emotion regulation skills. Affect Regul Training. 2014.

    Article  Google Scholar 

  2. Hofmann SG, Asnaani A, Vonk IJJ, Sawyer AT, Fang A. The efficacy of cognitive behavioral therapy: a review of meta-analyses. Cogn Ther Res. 2012;26(5):427–40.

    Article  Google Scholar 

  3. Gaudiano B. Cognitive-behavioral therapies: achievements and challenges. Evid Based Mental Health. 2008;11(1):5–7.

    Article  Google Scholar 

  4. Williams AD, O’Moore K, Mason E, Andrews G. The effectiveness of internet cognitive behaviour therapy (iCBT) for social anxiety disorder across two routine practice pathways. Internet Interven. 2014;1(4):225–9.

    Article  Google Scholar 

  5. Andersson G. Using the internet to provide cognitive behaviour therapy. Behav Res Ther. 2009;47(3):175–80.

    Article  Google Scholar 

  6. Ritterband LM, Andersson G, Christensen HM, Carlbring P, Cuijpers P. Directions for the international society for research on internet interventions (ISRII). Journal of Medical Internet Research. 2006;8(3):e23.

    Article  Google Scholar 

  7. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry. 2017;74(4):351–9.

    Article  Google Scholar 

  8. Cuijpers P, Donker T, van Straten A, Li J, Andersson G. Is guided self-help as effective as face-to-face psychotherapy for depression and anxiety disorders? A systematic review and meta-analysis of comparative outcome studies. Psychol Med. 2010;40:1943–57.

    Article  Google Scholar 

  9. Wagner B, Horn AB, Maercker A. Internet-based versus face-to-face cognitive-behavioral intervention for depression: a randomized controlled non-inferiority trial. J Affect Disord. 2013.

    Article  Google Scholar 

  10. Klein B, Austin D, Pier C, Kiropoulos L, Shandley K, et al. Internet-based treatment for panic disorder: does frequency of therapist contact make a difference? Cogn Behav Ther. 2009;38:100–13.

    Article  Google Scholar 

  11. Aguilera A, Muench F. There’s an app for that: information technology applications for cognitive behavioral practitioners. Assoc Adv Behav Ther J. 2012;35(4):65.

    Google Scholar 

  12. Donker T, Petrie K, Proudfoot J, Clarke J, Birch M-R, Christensen H. Smartphones for smarter delivery of mental health programs: a systematic review. J Med Internet Res. 2013;15(11):e247.

    Article  Google Scholar 

  13. Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. J Med Internet Res. 2015;3(1):e20.

    Google Scholar 

  14. Price M, Yuen EK, Goetter EM, Herbert JD, Forman EM, Acierno R, Ruggiero KJ. mHealth: a mechanism to deliver more accessible, more effective mental health care. Clin Psychol Psychother. 2014;21(5):427–36.

    Article  Google Scholar 

  15. Watts S, Mackenzie A, Thomas C, Griskaitis A, Mewton L, Williams A, Andrews G. CBT for depression: a pilot RCT comparing mobile phone vs. computer. BMC Psychiatry. 2013;13(1):49.

    Article  Google Scholar 

  16. Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med. 2011;1(1):53–71.

    Article  Google Scholar 

  17. Philpott D, Guergachi A, Keshavjee K. Design and validation of a platform to evaluate mHealth apps. Studies Health Technol Inf. 2017;235:3–7.

    Google Scholar 

  18. Miyamoto SW, Henderson S, Young HM, Pande A, Han JJ. Tracking health data is not enough: a qualitative exploration of the role of healthcare partnerships and mHealth technology to promote physical activity and to sustain behavior change. J Med Internet Res. 2016;4(1):e133.

    Google Scholar 

  19. Chindalo P, Karim A, Brahmbhatt R, Saha N, Keshavjee K. Health apps by design: a reference architecture for mobile engagement. IJHCR. 2016;7(2):34–43.

    Google Scholar 

  20. Albrecht UV. Transparency of health-apps for trust and decision making. J Med Internet Res. 2013;15(12):e277.

    Article  Google Scholar 

  21. Balouchi S, Keshavjee K, Zbib A, Vassanji K, Toor J. Creating a supportive environment for self-management in healthcare via patient electronic tools. Soc Media Mobile Technol Healthc. 2014;2014:109.

    Article  Google Scholar 

  22. Burns MN, Begale M, Duffecy J, Gergle D, Karr CJ, Giangrande E, Mohr DC. Harnessing context sensing to develop a mobile intervention for depression. J Med Internet Res. 2011;13(3):e55.

    Article  Google Scholar 

  23. Anderson K, Burford O, Emmerton L. Mobile health apps to facilitate self-care: a qualitative study of user experiences. PLoS ONE. 2016;11(5):e0156164.

    Article  Google Scholar 

Download references

Authors’ contributions

AL and KK developed and executed the study strategy. AL, KM, LK, AL2 and CM reviewed the apps. AL analyzed data collected for the review. AL and KM drafted the paper. CM, KM and LK revised the paper. All authors read and approved the final manuscript.


Not applicable.

Competing interests

The authors declare that they have no competing interests.

Availability of data and materials

The datasets developed and analyzed during the current study are available from the corresponding author on reasonable request.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Not applicable.


No external funding was obtained for this research.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Karim Keshavjee.

Appendix 1

Appendix 1

See Table 2.

Table 2 Apps downloaded for analysis

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lan, A., Lee, A., Munroe, K. et al. Review of cognitive behavioural therapy mobile apps using a reference architecture embedded in the patient-provider relationship. BioMed Eng OnLine 17, 183 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: