Skip to main content
Download PDF

Evaluation of HIV/AIDS-related mobile health applications content using an evidence-based content rating tool

BMC Medical Informatics and Decision Making volume 21, Article number: 135 (2021) Cite this article

Abstract

Background

Despite the increasing number of mobile health applications, the validity of their content is understudied. The objective of this study was to rate the content of HIV/AIDS-related mobile applications and to determine the extent to which evidence-based medicine is being incorporated into their content using a new tool called the Evidence-based content rating tool of mobile health applications (EBCRT-mHealth).

Methods

All available HIV/AIDS-related applications in Iran from Cafe Bazaar and Google Play Store were evaluated. This study was first conducted in 2018, then after almost two years in 2021 was done again. In this study, researchers developed the EBCRT-mHealth tool to rate the content of applications based on the evidence-based medicine pyramid. Its reliability was calculated (α = 0.78), and five specialists confirmed its validity. Two reviewers independently reviewed all HIV/AIDS applications directly downloaded and installed from the Google Play Store and Cafe Bazaar.

Results

Out of 980 retrieved applications, in 2018, 85, and in 2021, 78 applications were included in the study. Only in 17 (28%) out of the 60 in 2018, and 25 (51%) in 2021 Google Play store applications the source of content information was mentioned. All Cafe Bazaar mobile applications mentioned the source of information. The mean rating of all application content in 2018 was 2.38 (SD = 0.74), and in 2021 was 2.90 (SD = 1.35) out of 5. The mean rating of the content of Cafe Bazaar applications in 2018 was 2.10 (SD = 0.49), and in 2021 was 1.94 (SD = 0.29). The mean content rating of Google Play store applications in 2018 was 2.50 (SD = 0.80) and in 2021 was 3.86 (SD = 1.18).

Conclusion

After two years, the rating of the content of HIV/AIDS-related applications available in Iran that existed in Cafe Bazaar decreased from "poor" to "inappropriate". Also, the content score of the Google Play Store applications increased from "poor" to "good". It is critical to ensure the credibility of the sources used in developing their content and removing applications with inappropriate and unreliable content from the App Stores. Also, mobile health application developers should use the highest quality information in their applications.

Peer Review reports

Background

Providing education to HIV/AIDS patients and others using mobile applications installed on smartphones is one of the most accessible and up-to-date ways which maintains patient confidentiality and reduces social stigma. Mobile applications have many benefits for the management and control of HIV/AIDS including collecting and communicating patient data to care providers, providing educational messages to patients and their families, and meeting the information needs of people to prevent HIV/AIDS [1, 2]. There are 38 million and 61 thousand people living with HIV/AIDS in the world and Iran, respectively in 2019 [3, 4].

Providing information and education to patients and community is one of the most important and effective ways to control HIV/AIDS [5]. In addition to HIV/AIDS prevention, some studies evaluate the application of mobile health in self-care and self-management of patients [6,7,8]. Moreover, some studies reported on the use of the mobile application in increasing the awareness of the community and care coordination to control HIV/AIDS [9,10,11]. In the past few decades, evidence-based medicine has played an important role in converting HIV/AIDS from a deadly disease to a chronic disease [12, 13]. Studies [14,15,16] have shown that people with and at risk of HIV/AIDS are interested in using mobile health (mHealth) applications. Given the impact of mobile application content on the decision making of users and health care providers, and since many people with different levels of knowledge access these apps [17, 18], it is critical to use credible sources and information for the development of the content of mHealth applications [18].

Several tools have been used to evaluate mobile applications so far [19,20,21,22,23]. To our knowledge, none of these tools have been specifically developed to rate the content of applications and to evaluate the extent of scientific evidence used in the content of mHealth applications. Moreover, although some studies [24,25,26,27] evaluated HIV/AIDS-related mobile applications, none of these studies have focused on the rating of the content of mobile health applications based on evidence-based medicine. The objective of this study was to rate the content of HIV/AIDS-related mobile applications and to determine the extent to which evidence-based medicine is being incorporated into their content using a new tool called Evidence-based content rating tool of mobile health applications (EBCRT-mHealth).

Material and methods

This article is part of a larger ongoing study regarding the evaluation of HIV/AIDS-related applications in various terms including their features and content. The purpose of the larger study is to develop tools for the evaluation and rating of the mobile health applications and compare this tools with other developed tools such as mobile apps rating scale (MARS). In the current study only the use of evidence-based medicine in the development of the content of applications is studied. This study was initially conducted from 10 June 2018 to 16 August 2018. After almost two years, due to the non-publication of its results, from 10 January to 5 March 2021, this study is repeated. The study population consisted of all HIV/AIDS-related mobile applications available at Google Play store [28] and Cafe Bazaar [29]. Google Play store is the largest mobile application store in the world [30]. Cafe Bazaar is also the biggest Iranian mobile applications store that has the largest number of Persian mobile applications [31]. We included all HIV/AIDS-related mobile applications in the study. In this study, all included HIV/AIDS mobile applications in the Google Play Store were in English and available around the globe. All applications in the Cafe Bazaar were in Persian and targeted Iranian users or Persian language users, and none of them were bilingual.

Google Play Store and Cafe Bazaar were searched for mobile HIV/AIDS-related applications using the keywords "HIV", "AIDS", "HIV/AIDS", "Human Immunodeficiency Virus", "Acquired Immunodeficiency Syndrome", and their equivalents in Persian. The retrieved applications were installed on the Samsung Galaxy C7000 running Android. If the application was not installed on this smartphone, or there was a difficulty, we checked it on another smartphone (Samsung Galaxy A70). The inclusion criteria for selecting applications were: 1- the possibility of installation on Android OS, 2- in Persian or English languages, 3- available in Iran and 4- the focus of the application on HIV/AIDS.

Two reviewers with health information technology background independently evaluated the mobile applications. These reviewers were trained in the evaluation of health information systems before evaluation. Moreover, prior to the evaluation, both reviewers had received training on how to rate and analyze the content of the applications. After the installation of the applications on smartphones, one of the reviewers evaluated the applications. Afterward, all data stored on the app were removed and the other evaluator performed the evaluation. Whenever there was a difference between the two reviewers' scores, the third evaluator (supervisor) resolved the discrepancy through discussion. All evaluation data were collected on paper forms.

EBCRT-mHealth tool

Data were collected using EBCRT-mHealth tool (Evidence-based content rating tool of mobile health applications). This tool was developed by researchers in this study to determine the type of content and rate the information content of HIV/AIDS mobile applications. The EBCRT-mHealth tool is available in Additional file 1. This tool had three sections. The first section included general questions about application specifications. The second section was about the type of sources used to develop different parts of the content based on the hierarchy of evidence pyramid. This part was developed based on the review of previous studies [2, 32,33,34] concerning the pyramid of evidence. Additionally, we added other sources to the hierarchy of evidence pyramid based on the opinion of two Medical Informatics, two Health Information Management, and one Infectious Disease specialist (Fig. 1).

Fig. 1
figure 1

Evidence-based medicine pyramid used in this study

Full size image

The third section consisted of 16 questions for rating the applications based on the type of content, including prevention, treatment, education, self-care, medication, symptoms and signs, general information about the illness, health and wellness issues, and other topics. In order to select a content rating method, the rating methods of the previous tools [19,20,21,22,23, 35,36,37] were first reviewed, and then the content rating method was determined according to the opinion of two medical informatics specialists.

Based on this tool, if a type of content was not available in the application, a zero was assigned by the reviewers. For the available content, a score of 1–5 (inappropriate to excellent) was assigned to the mobile application. If the source of the content was mentioned in an application, that content was considered as evidence-based. The accuracy and validity of the content of applications can be verified if their references and sources are mentioned in the applications.

Figure 1 illustrates how the content of the applications is rated through the evidence-based medicine pyramid. The scoring of the applications content in this study was according to Fig. 1. Based on the evidence-based medicine pyramid, the following rating was devised: a score of 4.5–5 was assigned to guidelines, a score of 3–4.10 to different type of studies from meta-analysis to case studies and specialized websites, a score of 2–2.60 to laboratory research, books, textbooks, and experts opinion, and a score of 1–1.60 to news websites, Wikipedia or blogs and irrelevant sources. The details of these scores for each resource type are stated in Fig. 1. To determine these scores on Fig. 1 we used the division 1–5 (1- inappropriate, 2- poor, 3- acceptable, 4- good, and 5- excellent) and the importance of evidence-based on the hierarchy of evidence. The reviewers also checked whether the content of the applications was consistent with the content of the source used in the development of the applications. To determine the source of information, we considered all items that indicate the source of the content of the application, including links and hyperlinks, the mentioned sources at the end of the content, and the list of sources mentioned separately in the reference section inside the application. If a mobile application's content is based on a specific source, like a national guideline or several national guidelines are available in that mobile application. We assigned the national guideline score (score = 4.8) to that mobile application. To verify an application be a guideline or not to be, we checked the website registered in the Google Play Store, search on the internet, and details inside of that application. If the application used several sources, like a reference list. We calculated the sum of all the sources' scores, and then the average score was considered. No score was assigned to the application if its content did not match the content of a related source. If the source was not available to determine the consistency and accuracy of the content, the application did not receive a score. We used the mean to calculate each applications' average score and the total score of the applications.

The validity of the EBCRT-mHealth tool was confirmed by two Medical Informatics specialists and two Health Information Management specialists, and one Infectious Disease specialist physician. First, an initial draft of the tool is made and then sent to all experts. After applying their comments, it was sent to them again. This cycle continued until all experts approved the validity of the tool. After the validation of this tool, to ensure our developed tool's reliability before using it to all HIV/AIDS applications. We tested our tools on the first 20 Google Play Store applications, which were included in this study based on the inclusion criteria, were evaluated using this tool [19]. Then the reliability of the EBCRT-mHealth was assessed using Cronbach's alpha (α = 0.78). These 20 applications were not excluded from the study. After ensuring the reliability of this tool, we applied it to evaluate other applications. Moreover, to evaluate to what extent mobile app developers respect the confidentiality of their users and their users can use the application secretly, we checked whether information representing HIV/AIDS are used in the title and the logo of each application.

Statistical analysis method

In this study, descriptive statistics, including mean and standard deviation were used to calculate the application rate. The EBCRT-mHealth score of zero was not used to calculate the mean scores. To rate the mean scores of the EBCRT-mHealth tool, the scores 1–2 was considered as "inappropriate", 2–3 as "poor", 3–4 as "acceptable", 4–5 as "good", and 5 as "excellent". The Kolmogorov–Smirnov normalization test did not confirm the normality of the variables. Therefore, the Spearman correlation test was used to examine the relationship between the EBCRT-mHealth tool score with the number of HIV/AIDS application downloads and the apps rate in the App Stores. Two way mixed inter-correlation coefficient (ICC) was used to calculate internal validity and agreement between reviewers [38]. Descriptive data were analyzed using Microsoft Excel version 2016 and inferential statistics using SPSS version 24.

Results

Searching for apps in the Google Play Store and Cafe Bazaar took place on June 10 and 11, 2018, and on 13 and 14, January 2021. In 2018: out of 980 applications, 85 (60 from Google Play store and 25 from Cafe Bazaar) were included in this study. All included google play store applications are free of charge. Four of 25 (16%) Cafe Bazaar applications are paid apps. Figure 2 illustrates the process of selecting applications for inclusion in the study in 2018. A number of applications were excluded due to the issues such as the difficulty of downloading application content (n = 9), inability to login to access the content (n = 12), and inability to install the application (one of the Cafe Bazaar applications).

Fig. 2
figure 2

The process of selecting applications for inclusion in the study in 2018

Full size image

In 2021, the total number of included mobile applications in both app stores was 78 applications. In Cafe Bazaar, seven new applications have been added, and three applications have been removed from the Cafe Bazaar. The total number of applications included in this study from the Cafe Bazaar were 29 mobile applications. From Google Play Store, 14 applications have been added, and 25 have been removed. The total number of the included applications from the Google Play Store in 2021 was 49 mobile applications.

Descriptive characteristics of included HIV/AIDS mobile applications in 2018 and 2021 are shown in Table 1.

Table 1 Descriptive characteristics of included mobile applications in 2018 and 2021
Full size table

Nine of the 60 (15%) apps of the Google Play Store, and none of the Cafe Bazaar apps, designed in such way that users can document and collect their personal and health data inside the application. However, our inspection revealed that none of these application allowed user to post their information on social media. Eighteen out of 85 (21%) applications allowed users to share the content of the application (such as: information about HIV/AIDS prevention, etc.) to social media.

Forty-one out of 60 (68%) Google Play Store applications contained the word HIV or AIDS in the title of the applications, and the red ribbon symbol of HIV/AIDS was used in the logo of 29 (48%) applications. Thirty applications (50%) not only contained the HIV/AIDS symbol, but also in their titles, it was clear that the application is related to HIV/AIDS. All Cafe Bazaar applications had these conditions.

In 2018, given that in 17 (28%) of the 60, and in 2021, 25 (51%) out of 49 Google Play store applications, the sources of information were listed, the content of these 17 applications was reviewed and rated. Moreover, all Cafe Bazaar applications (n = 25 and n = 29) in two years were reviewed, as they all had the sources that were used in developing their content.

Results concerning the review of application content based on the hierarchy of evidence pyramid

In this section, we just reviewed the applications that had mentioned the information acquisition sources. The results of the content type and information sources of Google Play store applications are shown in Table 2. In 2018, of 17 Google Play Store applications, the most commonly content type was “Therapeutic” as same as in 2021. The most commonly used source was “National guideline” in 53% (n = 9) of applications, also in 2021 with 40% (n = 10) out of 25. In 2018 and 2021, among the Google play store applications, none of them reported that they used the evidences of the lower level of the hierarchy of evidences such as Laboratory research, Books, Expert opinions, News and Non-specialized website, Wikipedia, and Weblogs.

Table 2 The results of the content type and information sources of Google Play store applications
Full size table

The results related to the content type and information sources used in the development of cafe Bazaar applications are shown in Table 3. In 2018, the content types of most Cafe Bazaar applications were "general information on disease" and "prevention", and the most frequently used source in these applications was "The mentioned source unrelated to HIV/AIDS", each in 32% (n = 8) of the applications. In 2021, the most common content type was “Prevention”, and the most commonly used source was “The mentioned source unrelated to HIV/AIDS”. In 2018, among the Cafe Bazaar applications, in one application, the source was a national guideline, and in another application, it was a book. In 2021 these applications did not exist. None of the Cafe Bazaar applications reported that they used the evidences of the higher level of the hierarchy of evidences, such as different types of guidelines.

Table 3 The results related to the content type and information sources used in the development of cafe Bazaar applications
Full size table

Results of rating the content type of applications

In this section, we just reviewed the applications that had mentioned the information acquisition sources. The mean rating of all application content in 2018 was 2.38 (SD = 0.74), and in 2021 was 2.90 (SD = 1.35) out of 5. The mean content rating of Cafe Bazaar applications in 2018 was 2.10 (SD = 0.49) and in 2021 was 1.94 (SD = 0.29), and the mean content rating of Google Play store applications in 2018 was 2.50 (SD = 0.80) and in 2021 was 3.86 (SD = 1.18) out of 5.

In 2018, three (17%) out of the 17 Google Play store applications, including Aids drug database (HIV Drugs), EACS, and HIV in practice rated above 4 (good).

In 2021, two (8%) out of the 25 Google Play store applications rated 5 (Excellent), including “WHO HTS Info”, and “WHO HIV Tx‏”. Also, 11 (44%) applications were rated above 4 (good).

In 2018, of the 25 Cafe Bazaar applications, three (12%) rated 3 (acceptable), nine (36%) 2 (poor), and 13 (52%) 1 (inappropriate). Neither of the HIV/AIDS applications evaluated in this study from the Cafe Bazaar was given the score 5 out of 5 by both reviewers.

According to the rating results of 2021 with using the EBCRT-mHealth tool, The top three HIV/AIDS Cafe Bazaar applications are: Agahibakhshi AIDS (Score = 3.1), Rahnamye AIDS va Bimarihaye Amizeshi (Score = 3.0), Zang khatar AIDS (Score = 2.58). Also, the top three Google Play Store applications are: WHO HIV Tx‏ (Score = 5), WHO HTS Info (Score = 5), EACS‏ (Score = 4.9), and ClinicalInfo HIV/AIDS Guidelines‏ (Score = 4.9).

The results of rating the type of application content in 2018 and 2021 are shown in Table 4. In 2018, The highest rates of content type in Cafe Bazaar applications was related to the content "identifying health and social service centers" (2.63 ± 1.22), and the lowest related to "information on conferences and events" (1.33 ± 0.47). Applications in the Cafe Bazaar also lacked information on “management of alcohol and drug use", "abbreviations and dictionary", and "diagnostic aid testing". The highest rates of the content type in Google Play applications were related to "therapeutic information" (3.0 ± 1.5) also in 2021 were (4.38 ± 1.08) the lowest was related to “management of alcohol and drug use" (1.27 ± 0.62), in 2021 were lack of information.

Table 4 The results of rating the type of applications content
Full size table

There was a significant relationship between the number of HIV/AIDS application downloads in the App Store and the EBCRT-mHealth tool score (P-value < 0.001, r = 0.599). In other words, with the increase of the number of downloads, the EBCRT-mHealth tool score increases too. Also, there was no significant relationship between app rates in the App Store with EBCRT-mHealth tool score (P-value = 0.146). The agreement between the two reviewers for the total score of the EBCRT-mHealth tool was ICC = 0.78 (CI95% = 0.67–0.86).

Discussion

The objective of this study was to evaluate the content of HIV/AIDS-related mobile applications and the extent to which evidence-based knowledge was applied to their content. The results showed that the content of HIV/AIDS-related mobile applications available in Iran in 2018 was rated as "poor", and the content of few applications was "acceptable". Therefore, it is not easy to trust the content of most applications available in Iran, especially Cafe Bazaar applications.

In 2021, after almost two years, the Google Play Store HIV/AIDS mobile applications score increased from "poor" to "acceptable". But the score of Cafe Bazaar HIV/AIDS applications decreased from "poor" to "inappropriate". One of the reasons may be a large number of guidelines available in English compared to Persian. Another reason may be the policies for reviewing these applications and researching on the Google Play Store applications [39,40,41]. In Cafe Bazaar, there are no mobile application review policies, and very few studies have been done on these applications [42]. Mentioning the content acquisition source in mobile applications can help users assess content validity; also, users can refer to that source for more information [43, 44]. Given the important role of evidence-based education in the past few decades in transforming HIV/AIDS from a rapidly fatal disease to a chronic disease [5, 12, 13], using inappropriate and inaccurate information in these applications' content may adversely affect the treatment process.

The results also showed that the sources used in the Cafe Bazaar applications were at the lower of the evidence pyramid, and the sources used in the Google Play applications were at higher levels of the evidence pyramid. There was less than one score difference in 2018 and almost two score differences in 2021 between the total content rate of the Google Play Store and the Cafe Bazaar applications. A specific type of content in some Google Play store applications received a score of higher than 4, but due to the presence of other poor content with a low score, the overall rate of the application was low. The score of the content of most Cafe Bazaar applications was low. The source of information acquisition in most HIV/AIDS-related applications in Cafe Bazaar was unrelated. A review of the sources mentioned in the applications also showed that the application content was not found in the referenced sources. The results of the study by Robustillo Cortés et al. [26] showed that the quality of educational applications for HIV/AIDS positive patients is poor.

The organizational affiliation of more than half of the HIV/AIDS-related applications was unknown. The results of the study by Rosa et al. [25] also showed that more than half of the applications are not supported by health care organizations.

Checking the logo and title of the reviewed application revealed that only almost half of the applications did not appear to be on the HIV/AIDS topic. To prevent the sense of stigma in people that use these apps, and in order to preserve the confidentiality of these patients, it is suggested that uncustomary logos and names be used in the apps developed for these kinds of patients.

Mobile applications can be evaluated from different aspects and methods [18]. In the previously developed tools, such as Mobile Application Rating Scale (MARS) [19], Health On the Net (HON) [20], and DISCERN Instrument [21], along with other aspects, have questions about the evidence-based medicine and rated the evidence of the applications subjectively, without providing the basis for rating the evidences. In this study, we proposed an evidence-based pyramid that could be the basis for rating the evidence of the mobile applications. Also, we rated the applications according to this tool.

The most important advantage of using the content rating method developed in this study is the possibility of comparing applications based on content ratings using the hierarchy of evidence. Using the five-point ratings to rate the content of the applications enables the comparison of the results with the rating based on other tools [19, 21] that used the five-point rating method, as well as the five-star rating methods. Currently, users select a mobile application based on its popularity and the number of downloads, regardless of the quality of its information and content [45]. The evaluation method used in this study can be used to rate the content of mobile applications, along with the star rating method used by App Stores. We used this method to rate the content of HIV/AIDS-related applications; future studies can use this method to rate the content of other mHealth applications.

Limitations

The present study had two limitations. First, this study was performed in Iran. Since all IOS and some Android applications are not available in Iran due to sanctions and regional restrictions of Google Play, the results may not be generalizable to other HIV/AIDS-related applications available in other countries. The researcher tried to access IOS mobile applications but was unable to access them in Iranian territory. The results of a previous study [27] showed that more than 60 percent of the HIV/AIDS mobile applications are available on both platforms. Also, another study [41] that evaluated 11 HIV pre-exposure prophylaxis mobile applications showed that all of these mobile applications are available in Android. Nevertheless, in terms of the number of HIV/AIDS-related applications, this study included the highest number of evaluated applications compared to previous studies [24,25,26,27]. Second, since it is not possible to pay for the Google Play Store in Iran, only the free applications were evaluated in this study. According to our searches in Google Play Store, all HIV/AIDS available mobile applications in Iran are free of charge. None of these applications have been excluded from the study due to impossible payment. The paid apps of Cafe Bazaar also entered to the study and were evaluated. A previous study [27] also reported that all HIV/AIDS-related applications of Google Play and iTunes were available free of charge.

In this study, the content evaluation was done based on the content of the applications at the time of downloading. Given that it is possible to update the applications in stores, we recommend developers to cite the source of the content in future updates of their applications and use reliable and evidence-based sources for developing content in order to increase the content rating of these applications.

Conclusion

This study developed the EBCRT-mHealth tool and rated the HIV/AIDS mobile applications based on the evidence used in that mobile application content in 2018 and 2021. The results of this study showed, after almost two years, the rating of the content of HIV/AIDS-related applications available in Iran that existed in Cafe Bazaar decreased from "poor" to "inappropriate". Also, the Google Play Store applications' content score increased from "poor" to "good". It is critical to ensure the credibility of the sources used in developing their content and removing applications with inappropriate and unreliable content from the App Stores. Mobile health application developers can use the hierarchy of evidence pyramid developed in this study to ensure and demonstrate that they are using the highest quality information. Also, they can use this tool to evaluate their applications before sharing them in App Stores. The mobile applications researchers can use this tool to evaluate the applications in terms of evidence-based content and trustworthiness.

Availability of data and materials

Not applicable.

Abbreviations

EBCRT-mHealth:

Evidence-based content rating tool of mobile health applications

MARS:

Mobile apps rating scale

HIV:

Human immunodeficiency virus

AIDS:

Acquired immune deficiency syndrome

ICC:

Internal correlation coefficient

SD:

Standard deviation

WHO:

World Health Organization

HON:

Health on the net

CI:

Confidence interval

References

  1. Schnall R, Rojas M, Travers J, Brown W 3rd, Bakken S. Use of design science for informing the development of a mobile app for persons living with HIV. AMIA Ann Symp Proc. 2014;2014:1037–45.

    Google Scholar 

  2. Catalani C. mHealth for HIV treatment and prevention: a systematic review of the literature. Open AIDS J. 2013;7:17–41.

    Article  Google Scholar 

  3. WHO. Global Health Observatory (GHO) data|HIV/AIDS. World Health Organization website. 2020. http://www.who.int/gho/hiv/en/. Accessed 4 Aug 2020.

  4. UNAIDS. Islamic Republic of Iran|UNAIDS. the Joint United Nations Programme on HIV/AIDS website. 2019. https://www.unaids.org/en/regionscountries/countries/islamicrepublicofiran. Accessed 4 Aug 2020.

  5. Sadeghi R, Khanjani N. Impact of educational intervention based on theory of planned behavior (TPB) on the AIDS-preventive behavior among health volunteers. Iran J Health Educ Health Promot. 2015;3:23–31.

    Google Scholar 

  6. Rana AI, van den Berg JJ, Lamy E, Beckwith CG. Using a mobile health intervention to support HIV treatment adherence and retention among patients at risk for disengaging with care. AIDS Patient Care STDS. 2016;30:178–84.

    Article  Google Scholar 

  7. Ramanathan N, Swendeman D, Comulada WS, Estrin D, Rotheram-Borus MJ. Identifying preferences for mobile health applications for self-monitoring and self-management: focus group findings from HIV-positive persons and young mothers. Int J Med Inform. 2013;82:e38-46.

    Article  Google Scholar 

  8. Laurence C, Wispelwey E, Flickinger TE, Grabowski M, Waldman AL, Plews-Ogan E, et al. Development of positivelinks: a mobile phone app to promote linkage and retention in care for people with HIV. J Med Internet Res. 2019;21.

  9. Karanja S, Mbuagbaw L, Ritvo P, Law J, Kyobutungi C, Reid G, et al. A workshop report on HIV mHealth synergy and strategy meeting to review emerging evidence-based mHealth interventions and develop a framework for scale-up of these interventions. Pan Afr Med J. 2011;10:37.

    PubMed  PubMed Central  Google Scholar 

  10. Westergaard RP, Genz A, Panico K, Surkan PJ, Keruly J, Hutton HE, et al. Acceptability of a mobile health intervention to enhance HIV care coordination for patients with substance use disorders. Addict Sci Clin Pract. 2017;12:11.

    Article  Google Scholar 

  11. Evans C, Turner K, Suggs LS, Occa A, Juma A, Blake H. Developing a mHealth intervention to promote uptake of HIV testing among African communities in the conditions: a qualitative study. BMC Public Health. 2016;16:656.

    Article  CAS  Google Scholar 

  12. Horwitz RI, Hayes-Conroy A, Caricchio R, Singer BH. From evidence based medicine to medicine based evidence. Am J Med. 2017;130:1246–50.

    Article  Google Scholar 

  13. Horwitz RI, Singer BH. Why evidence-based medicine failed in patient care and medicine-based evidence will succeed. J Clin Epidemiol. 2017;84:14–7.

    Article  Google Scholar 

  14. Shrestha R, Karki P, Copenhaver M. Interest in use of mHealth technology in HIV prevention and associated factors among high-risk drug users enrolled in methadone maintenance program. AIDS Care 2017;:1–5.

  15. Shrestha R, Huedo-Medina TB, Altice FL, Krishnan A, Copenhaver M. Examining the acceptability of mHealth technology in HIV prevention among high-risk drug users in treatment. AIDS Behav. 2016;21:3100–10.

    Article  Google Scholar 

  16. Chang LW, Njie-Carr V, Kalenge S, Kelly JF, Bollinger RC, Alamo-Talisuna S. Perceptions and acceptability of mHealth interventions for improving patient care at a community-based HIV/AIDS clinic in Uganda: a mixed methods study. AIDS Care. 2013;25:874–80.

    Article  Google Scholar 

  17. Peiris D, Miranda JJ, Mohr DC. Going beyond killer apps: building a better mHealth evidence base. BMJ Glob Health. 2018;33:e000676.

    Article  Google Scholar 

  18. Nouri R, R Niakan Kalhori S, Ghazisaeedi M, Marchand G, Yasini M. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc 2018;25:1089–98.

  19. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth uHealth. 2015;3:e27.

    Article  Google Scholar 

  20. HONcode: Principles for Quality and Trustworthy Health Information. http://www.hon.ch/HONcode/Patients/Conduct.html. Accessed 30 Dec 2020.

  21. The DISCERN Instrument. http://www.discern.org.uk/discern_instrument.php. Accessed 14 Oct 2017.

  22. Chyjek K, Farag S, Chen KT. Rating pregnancy wheel applications using the applications scoring system. Obstet Gynecol. 2015;125:1478–83.

    Article  Google Scholar 

  23. Jin M, Kim J. Development and evaluation of an evaluation tool for healthcare smartphone applications. Telemed e-Health. 2015;21:831–7.

    Article  Google Scholar 

  24. Muessig KE, Pike EC, Legrand S, Hightow-weidman LB. Mobile phone applications for the care and prevention of HIV and other sexually transmitted diseases: a review. J Med Internet Res. 2013;15:e1.

    Article  Google Scholar 

  25. Cantudo Cuenca MR, Cantudo Cuenca MD, Morillo VR. Availability and medical professional involvement in mobile healthcare applications related to pathophysiology and pharmacotherapy of HIV/AIDS. Eur J Hosp Pharm. 2013;20:356–61.

    Article  Google Scholar 

  26. Robustillo Cortés M de las A, Cantudo Cuenca MR, Morillo Verdugo R, Calvo Cidoncha E. High quantity but limited quality in healthcare applications intended for HIV-infected patients. Telemed e-Health 2014;20:729–35.

  27. Schnall R, Mosley JP, Iribarren SJ, Bakken S, Carballo-Dieguez A, Brown Iii W, et al. Comparison of a user-centered design, self-management app to existing mHealth apps for persons living with HIV. JMIR mHealth uHealth. 2015;3:e91.

    Article  Google Scholar 

  28. Android Apps on Google Play. https://play.google.com/store. Accessed 2 Nov 2018.

  29. Cafe Bazar Online Persian App Store. https://cafebazaar.ir/. Accessed 2 Nov 2018.

  30. Statista. App stores: number of apps in leading app stores 2019|Statista. https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/. Accessed 4 Dec 2019.

  31. SaeedI MG, Kalhori SRN, Nouria R, Yasini M, Nouri R, Yasini M. Persian mHealth apps: a cross sectional study based on use case classification. Stud Health Technol Inform. 2016;228:230–4.

    PubMed  Google Scholar 

  32. Maki KC, Slavin JL, Rains TM, Kris-etherton PM. Limitations of observational evidence : implications for evidence-based dietary recommendations 1, 2. Adv Nutr. 2014;5:7–15.

    Article  CAS  Google Scholar 

  33. Petticrew M, Roberts H. Evidence, Hierarchies, and typologies: horses for courses. J Epidemiol Community Health. 2003;57:527–9.

    Article  CAS  Google Scholar 

  34. Powell AC, Torous J, Chan S, Raynor GS, Shwarts E, Shanahan M, et al. Interrater reliability of mhealth app rating measures: analysis of top depression and smoking cessation apps. JMIR mHealth uHealth. 2016;4:e15.

    Article  Google Scholar 

  35. Wyatt JC, Thimbleby H, Rastall P, Hoogewerf J, Wooldridge D, Williams J. What makes a good clinical app? Introducing the RCP health informatics unit checklist. Clin Med J R Coll Phys Lond. 2015;15:519–21.

    Google Scholar 

  36. Aitken M, Gauntlett C. Patient apps for improved healthcare: from novelty to mainstream. IMS Institute for Healthcare Informatics. 2013.

  37. Choi YK, Demiris G, Lin SY, Iribarren SJ, Landis CA, Thompson HJ, et al. Smartphone applications to support sleep self-management: review and evaluation. J Clin Sleep Med. 2018;14:1783–90.

    Article  Google Scholar 

  38. Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8:23–34.

    Article  Google Scholar 

  39. Siddique AB, Krebs M, Alvarez S, Greenspan I, Patel A, Kinsolving J, et al. Mobile apps for the care management of chronic kidney and end-stage renal diseases: systematic search in app stores and evaluation. JMIR mHealth uHealth. 2019;7:e12604.

    Article  Google Scholar 

  40. Brown HM, Bucher T, Collins CE, Rollo ME. A review of pregnancy apps freely available in the Google Play Store. Health Promot J Aust. 2019;31:340–2.

    Article  Google Scholar 

  41. Sharpe JD, Kamara MT. A systematic evaluation of mobile apps to improve the uptake of and adherence to HIV pre-exposure prophylaxis. Sex Health. 2018;15:587–94.

    Article  Google Scholar 

  42. Abbasi R, Nabovati E, Raeesi A, Ostadmohammadi F. Investigating the quality of persian mobile applications related to patients with chronic diseases. J Health Biomed Informat. 2020;7:273–81.

    Google Scholar 

  43. Birrell L, Deen H, Champion KE, Newton NC, Stapinski LA, Kay-Lambkin F, et al. A mobile app to provide evidence-based information about crystal methamphetamine (Ice) to the community (cracks in the ice): co-design and beta testing. JMIR mHealth uHealth. 2018;6:e11107–e11107.

    Article  Google Scholar 

  44. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental health smartphone apps: review and evidence-based recommendations for future developments. JMIR Mental Health. 2016;3:e7–e7.

    Article  Google Scholar 

  45. Kuehnhausen M, Frost VS. Trusting Smartphone Apps? To install or not to install, that is the question. In: IEEE international multi-disciplinary conference on cognitive methods in situation awareness and decision support (CogSIMA). 2013. p. 30–7.

Download references

Acknowledgements

Authors would like to thank Fatemeh Tabatabaei for contribution in the evaluation of the apps. They would also like to thank Dr. Mohammad Reza Safaei for consultation about HIV/AIDS disease.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Department of Health Information Sciences, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Iran

    Ahmad Raeesi

  2. Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran

    Ahmad Raeesi

  3. Department of Health Information Sciences, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Iran

    Reza Khajouei

  4. HIV/STI Surveillance Research Center, WHO Collaborating Center for HIV Surveillance, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Haft-Bagh Highway, Kerman, Iran

    Leila Ahmadian

Authors
  1. Ahmad Raeesi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reza Khajouei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leila Ahmadian
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LA, RK, and AR designed the study. LA and RK supervised the project. AR contributed to the app's content evaluation. AR and LA analyzed and interpreted the data. LA and AR wrote the final manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Leila Ahmadian.

Ethics declarations

Ethics approval and consent to participate

In this study, no applications have been downloaded illegally, and all applications were downloaded directly from the Google Play Store and Café Bazaar. This study was approved by the Research Ethics Committee of Kerman University of Medical Sciences (Approval ID: IR.KMU.REC.1397.246). We confirm that all methods were performed in accordance with the relevant guidelines and regulations. Moreover, as we did not include the individuals as the participants to the study, there was no need to have informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1

. The EBCRT-mHealth tool.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Raeesi, A., Khajouei, R. & Ahmadian, L. Evaluation of HIV/AIDS-related mobile health applications content using an evidence-based content rating tool. BMC Med Inform Decis Mak 21, 135 (2021). https://doi.org/10.1186/s12911-021-01498-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12911-021-01498-7

Keywords

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us