Skip to main content
Download PDF

Evaluating and rating HIV/AIDS mobile apps using the feature-based application rating method and mobile app rating scale

BMC Medical Informatics and Decision Making volume 22, Article number: 281 (2022) Cite this article

Abstract

Background

The purpose of this study was to evaluate HIV/AIDS mobile applications using the Mobile App Rating Scale (MARS) and rate the features of these applications using the new tool called the Feature-based Application Rating Method (FARM).

Methods

In this study, all available HIV/AIDS apps in Iran from Cafe Bazaar and Google Play Store due to inclusion criteria were studied. The evaluation of the quality of applications was done using the MARS tool and the FARM tool. The FARM, which was developed in this study, was applied to rank the features of the applications.

Results

In this study, 79 applications were included. The mean score of all apps using both tools was 3.58 (SD = 0.95) out of 5. The overall mean quality score based on the MARS was 3.14 (SD = 0.84), and the mean score of features based on FARM was 3.81 (SD = 1.23). This study showed a higher than moderate correlation between the scores assigned to apps based on the MARS and FARM tools (r > 0.4).

Conclusions

The HIV/AIDS mobile applications available in Iran had the "acceptable" ranking. Also, our study results showed that to evaluate mobile apps, using a single tool may not provide good insight to evaluators about the assessed apps. However, using more than one tool may provide more details about the evaluated apps. To improve the quality of mobile health apps and help users select the most desirable app, we suggested using tools like FARM for ranking apps based on the features of each app in the app stores.

Peer Review reports

Background

The high penetration rate of mobile phones worldwide makes mobile applications one of the fastest-growing technologies [1]. Approximately 70% of mobile phones use the Android operating system [2]. During the second quarter of 2022, almost 3.5 million Android apps were available in the Google Play Store [3]. Of these, 54,603 are mobile health apps [4]. Mobile health apps have many uses in preventing and treating chronic diseases like HIV/AIDS [5,6,7,8]. HIV/AIDS is one of the most serious socio-economic threats to public health due to its chronic nature, the possibility of its prevalence among individuals, and the absence of a cure for this disease [9]. Approximately 38.4 million and 53 thousand people were living with HIV/AIDS in the world and Iran, respectively [10, 11].

Mobile applications related to HIV/AIDS can help communicate with care providers and reduce hospital care. These applications can be used to provide self-care and help patients achieve compliance with antiretroviral therapy. Moreover, these mobile applications can be used to send alerts and reminders, collect data, provide real-time audio and video communication, deliver educational information, and provide requested information to the community to prevent the disease and control its transmission to others [7, 12, 13]. The number of these apps and their features change over time [3, 14]. Despite the high number of mobile health apps and their features, the quality and validity of many apps are unknown [15, 16]. Therefore, there is a need for tools to continuously review and evaluate these apps to determine their quality and validity.

Several tools have been used to evaluate mobile apps [17,18,19,20,21,22]. These tools evaluate mobile apps based on quality and trustworthy health information [18, 19], some objective and subjective components [20], functionality scoring [21], and usability [22]. One of these tools is the Mobile Application Rating Scale (MARS), which evaluates and rates mobile apps in terms of qualitative, objective, and subjective aspects that were developed in the previous study by Stoyanov et al. [17]. The MARS tool consists of 23 questions in four objective dimensions (A_D), including Engagement (5 questions), Functionality (4 questions), Aesthetics (3 questions), Information Quality (7 questions), and a Subjective dimension (E) (4 questions) [17, 23]. The MARS tool is a comprehensive and reliable tool that is widely used to rate the quality of mobile health apps like Epilepsy, COVID-19, self-management mobile health apps, Spine disorders, and Alzheimer's disease [24,25,26,27,28]. This tool has been widely translated and validated into other languages, including French [29], Italian [30], Korean [16], Spanish [31], Japanese [32], Arabic [33], and German [34].

Although the MARS tool can evaluate various aspects of a mobile application, it has limitations in assessing the mobile application's features [35, 36]. In an app, a feature is typically an essential function or a service provided by the app for users [14]. Features may be desirable or undesirable to users. If the existence of a feature is positive and useful for the users, that is a desirable or positive feature. If the existence of a feature is negative for users, such as advertisements, or its presence is irritating for users, such as the presence of corrupted and misleading links, that is an undesirable or negative feature [37,38,39]. Some previous studies [35, 40,41,42,43,44] that used the MARS tool to evaluate the apps also reviewed the existence of features without reviewing the quality of each feature separately. The only tool that was developed to rate the mobile app features is the IQVIA functionality score (previously known as the IMS functionality score) [26]. This tool is based on seven functionality criteria and four functional subcategories detailed in the report of the IQVIA Institute for Healthcare Informatics [21, 45]. In some previous studies [26, 43, 45], this tool was used along with the MARS tool to rate mobile health app features. The IQVIA functionality score focuses on the availability of the 11 previously determined functionalities, and finally, each mobile app gives a score between 0 and 11. The MARS functionality score is an overall score on a five-point Likert scale that is measured based on the quality of performance, navigation, ease of use, and gestural design of that app [17, 21, 45]. The IQVIA evaluates the availability of functionality of each feature without considering quality, and the functionality section of the MARS tool evaluates the overall quality of functionality of each app. The functionality score range of IQVIA differs from the functionality score of MARS, so their scores are not comparable. The IQVIA feature lists are predetermined and not flexible for each app. Also, in these two tools, undesirable features are not considered.

Due to the high number of mobile apps in app stores, it is difficult for users to find their desired applications [46]. Currently, users choose a mobile application based on the application's popularity, star rating, comments on app stores, and the number of downloads, regardless of the quality of the application [47]. Identifying and rating the quality of mobile apps and their features can help users find and select an app based on the features they need [14, 37]. Mobile apps may have different features compared to each other. The previously developed evaluation tools [17,18,19, 21, 48, 49] have not adequately addressed the evaluation of each mobile app feature. When users are faced with an abundance of similar apps with many functionalities or features, they tend to choose apps with their desired features [14]. Therefore, it is necessary to develop a tool that has flexibility based on the availability of each feature on a certain app to evaluate and rate mobile apps.

Some mobile apps related to HIV/AIDS exist in app stores [50]. The literature review showed that few studies [50,51,52,53] evaluated the quality of HIV/AIDS-related mobile applications, and none of these studies had rated these applications' features. Despite the important role of mobile apps in HIV/AIDS prevention and treatment, no study has been conducted to review HIV/AIDS mobile apps in Iran. Therefore, the purpose of this study was to (1) evaluate the quality of HIV/AIDS-related mobile applications in the Google Play and Café Bazaar stores available in Iran using the Mobile Application Rating Scale (MARS); and (2) to evaluate and rate that applications' features (desirable and undesirable) using the new tool called the Feature-Based Application Rating Method (FARM).

Methods

This article is the second part of a two-part series regarding evaluating HIV/AIDS-related applications in various terms, including their features and content [15]. This study was a cross-sectional descriptive-analytical study carried out on HIV/AIDS mobile applications in the Persian or English language available in Iran from May 6, 2021, to September 23, 2021. All HIV/AIDS-related mobile apps in the following two app stores were evaluated: The Google Play Store, the world's largest app store of mobile apps [3], and Café Bazaar, the most prominent Iranian app store for Persian mobile applications [54]. More than 3500 Android mobile applications in the Café Bazaar are related to the health and medical fields [55]. Given the small population size, all HIV/AIDS-related applications were included in the study.

Mobile apps are searched using the keywords "HIV", "AIDS", "Human Immunodeficiency Virus", "Acquired Immunodeficiency Syndrome", and the Persian keywords with similar meanings in the Google Play Store and Café Bazaar store. Then, the downloaded apps were installed on the Android smartphone (SAMSUNG Galaxy A51). The inclusion criteria to enter into this study are: (1) the mobile application can be installed on the Android operating system (Android 11.0); (2) it is written in Persian or English; (3) it is available in Iran, and (4) the focus of the mobile app is on HIV/AIDS.

Two evaluators with a background in health information technology independently evaluated all mobile apps using the MARS and FARM tools. Before evaluation, evaluators watched the MARS training video [56] and were trained about using the FARM tool. They are not the HIV/AIDS mobile app's real users. These two evaluators passed the related courses on the evaluation of mobile health apps. The first evaluator performed the evaluation after downloading and installing the included apps on the smartphone. When the evaluation was completed by the first evaluator, the stored data during the evaluation of the first evaluator inside of the apps was deleted, and then the second evaluator started the evaluation of the apps. Both evaluators first evaluated the HIV/AIDS apps using the MARS tool and then used the FARM. The evaluation sessions were limited to a maximum of 45 min per each session. We also asked the evaluators to spend sufficient time to gather the required information before assigning the scores. The evaluation was done when the evaluators were mentally prepared to perform the evaluation, and if the evaluators were tired, the evaluation stopped and continued to another time when the evaluators had sufficient mental preparation. The differences between the scores of the two evaluators were resolved by discussion between them. If the differences were not resolved through discussion between them, we used the third evaluator (supervisor) to resolve the discrepancy. The collected data was recorded on a paper form and then entered into a Microsoft Excel spreadsheet and SPSS for analysis.

In this study, all apps were downloaded directly from the Google Play Store and Cafe Bazaar. Our searches in the Google Play Store showed that all HIV/AIDS available mobile apps in Iran are free of charge. Also, the previous study [50] stated that all HIV/AIDS-related apps are available free of charge. In this study, the paid apps of Cafe Bazaar were included and evaluated.

Data collection tools

The MARS and FARM tools were used to rate the mobile apps. This study was conducted to apply the FARM tool to evaluate the apps based on their features. In this study, we used the MARS tool to compare the results of the FARM with a previously developed tool and also to get a better view of the existing HIV/AIDS apps. Since the MARS and other previously developed mobile app evaluation tools [17,18,19, 21, 48, 49] have not adequately addressed the evaluation of mobile app features, to address this issue in this study, a tool called the Feature-Based App Rating Method (FARM) was developed to rate mobile apps based on their features. The FARM evaluates and rates mobile apps based on both the availability and quality of each feature. The items of this tool are not predetermined; they are flexible based on each app's features.

To develop the FARM tool, all HIV/AIDS-related mobile apps in the Google Play Store and the Café Bazaar that were included in this study were reviewed, and all of their features were extracted, and a list of these features was prepared. This list was considered as the desirable features for the FARM. The list of undesirable features was also prepared based on a previous study [38] and the opinions of four experts who had checked the validity of the FARM. The FARM is available in Additional file 1. In total, 33 desirable features and nine undesirable features were identified and added to the FARM. To determine a ranking method for the features of the mobile apps and create a ranking method in line with previously developed tools. The ranking methods of the previously developed tools [17,18,19,20, 45, 48, 57] were reviewed. In the FARM, we used the standard 5-point Likert ranking method to compare apps with other tools that used the 5-point ranking method to rank the apps (such as star rating and the MARS tool).

Two medical informatics specialists confirmed the ranking method used in the FARM tool. A score of zero was assigned to the app to rank the applications based on the FARM tool for the absence of each desirable feature. If the app contained a desirable feature, the evaluators checked its functionalities and assigned a score of one to five (1-inappropriate, 2-poor, 3-acceptable, 4-good, and 5-excellent) to that feature based on the extent to which the feature met its expected function. Moreover, to rank an undesirable feature, a score between one (the undesirable feature is very annoying) and five (the absence of the undesirable feature) was assigned to that feature.

The validity of MARS and FARM was confirmed by two Medical Informatics specialists and two Health Information Management specialists. To investigate the reliability of the MARS and FARM tools, the first 20 mobile apps retrieved from the Google Play Store were evaluated using these tools, and Cronbach's alpha was calculated for each tool. The internal reliability of the MARS tool was 0.94 for all questions. The internal reliability of the MARS dimensions was between 0.63 and 0.93. The internal reliability of the FARM tool was 0.85 for the desirable features and 0.76 for undesirable features.

Data analysis

This study used descriptive statistics, including mean and standard deviation, to calculate the app ratings. To calculate the mean scores, the zero scores of the FARM tool and the N/A score of the MARS tool were not considered. The mean scores for the MARS and FARM tools were classified as the scores between 1 to 2 being considered as "inappropriate", 2 to 3 as "poor", 3 to 4 as "acceptable", 4 to 5 as "good", and 5 as "excellent". The Kolmogorov–Smirnov normalization test did not confirm the normality of variables related to the MARS and FARM tools. Therefore, the Spearman correlation test was used to examine the relationship between the dimensions of the MARS and FARM tools. The internal validity and consistency of the evaluators were calculated using the two-way mixed internal correlation coefficient (ICC) [58]. Microsoft Excel version 2019 was used to analyze the descriptive data, and SPSS version 24 was used to analyze the analytical statistics.

Results

A total of 971 apps were retrieved from the two app stores, of which 79 apps based on the inclusion criteria were included in the study. Of these, 29 apps were in the Café Bazaar and 50 were in the Google Play Store. All HIV/AIDS apps available in the Google Play Store are free. Five of 29 (17%) Café Bazaar applications are paid apps, and 14 (48%) are in-app purchases. The process utilized to identify the apps is shown in Fig. 1.

Fig. 1
figure 1

Flow chart of the selection process for inclusion of the Apps

Full size image

The mean rating score of apps in stores was 4.37 (SD = 0.60). The organizational affiliations of 19 apps (38%) in the Google Play Store were unknown; one app (2%) was commercial, 14 apps (28%) were governmental, non-governmental organizations developed ten apps (20%), and the affiliations of 6 apps (12%) were universities. The organizational affiliations of 13 apps (45%) in the Café Bazaar were unknown; thirteen apps (45%) were commercial; three apps (7%) were non-governmental organizations, and the university developed one app (3%).

The results of the evaluation of the mobile app

The results of evaluating HIV/AIDS-related apps using the MARS and FARM tools are shown in Table 1. The average score of all apps using both tools was 3.53 (SD = 0.68) out of 5. As Table 1 shows, the overall mean score for Google Play Store apps was 3.74 (SD = 0.68), and for Café Bazaar apps it was 3.15 (SD = 0.52). Among all the dimensions of both tools, "undesirable features" was the highest-scoring domain (4.67 ± 0.42). The lowest score was related to "Engagement" (2.85 ± 0.93).

Table 1 The results of the evaluation of HIV/AIDS apps based on MARS and FARM tools
Full size table

Fourteen apps (28%) out of 50 apps in the Google Play Store give a score above 4: "Every Dose, Every Day" (4.76 ± 0.29), "HIV Oral PrEP Implementation Tool" (4.75 ± 0.40), "inPractice HIV" (4.71 ± 0.30), "HIV Client Treatment Preparedness" (4.70 ± 0.24), "HIV Care Tools" (4.65 ± 0.30), "WHO HIV Tx" (4.59 ± 0.38), "Liverpool HIV iChart" (4.55 ± 0.27), "YourPrEP" (4.53 ± 0.34), "WHO HTS Info"‏ (4.51 ± 0.39), "HIV-HCV Drug Therapy Guide" (4.51 ± 0.39), "ClinicalInfo HIV/AIDS Guidelines" (4.45 ± 0.40), "life4me + " (4.24 ± 0.53), "EACS" (4.08 ± 0.79), and "HIV-Rx DDI Check" (4.02 ± 0.66). Also, two apps (7%) out of 29 Cafe Bazaar apps give scores above 4: "Pishgiriye Pas az Tamas (PEP)" (4.49 ± 0.44), and "Agahi Bakshi AIDS" (4.18 ± 0.60). The lowest score among all apps was taken by the Cafe Bazaar app named "AIDS va Darman" (2.02 ± 1.31). Among all of the Google Play Store HIV/AIDS rated apps, 14 apps (28%) scored above 4, 23 apps (46%) scored between 3 and 4, and 13 apps (26%) scored between 2 and 3. Among all of the Cafe Bazaar HIV/AIDS rated apps, two apps (7%) scored above 4, nine apps (31%) scored 3 to 4, and 18 apps (62%) scored between 2 and 3.

The MARS tool results

The MARS total mean score for all apps was 3.13 ± 0.83. The highest rank was related to the Functionality dimension (4.00 ± 0.59), and the lowest was related to the Subjective dimension (2.50 ± 1.26) (Table 1). Fifteen apps (19%) scored above 4 out of 5. The highest three scored apps were "HIV Oral PrEP Implementation Tool" (4.82 ± 0.66), "inPractice HIV" (4.74 ± 0.45), and "Every Dose, Every Day" (4.71 ± 0.56). Also, 23 apps (29%) scored 3, 38 apps (48%) scored 2, and only three apps (4%) scored 1. The lowest three scored apps were "HIV AIDS Awareness" (1.77 ± 0.92), "AIDS va Darman" (1.80 ± 1.06), and "AIDS va Moghabele ba an" (1.86 ± 0.96).

The results of ranking mobile app features using the FARM tool

The results of the ranking of HIV/AIDS mobile app features using the FARM tool are shown in Table 1. The FARM mean score of all apps was (3.92 ± 0.69) out of 5. The apps retrieved from Cafe Bazaar did not have 21 (63%) of the 33 evaluated features in this study. All the Cafe Bazaar apps had a textual content. The features "app description inside of the app" (3.50 ± 1.29), the "search functionality" (3.16 ± 0.71), and the "Bookmark feature" (3.12 ± 0.83), respectively, scored the highest rank.

The ranking of the Cafe Bazaar HIV/AIDS apps based on undesirable features showed that features such as "being a free app but requiring payment for basic features" (3.86 ± 1.16), "advertising" (4.00 ± 0.80), and "the existence of unrelated information" (4.28 ± 0.84) had the lowest score for Café Bazaar apps. The features "Difficulties to login into the app" (5.0 ± 0), "Inactive and misleading buttons" (4.97 ± 0.19), and "stopping the app after execution" (4.97 ± 0.19) had the highest ranking.

In Google Play Store apps, evaluators assigned the highest scores to the features "Collect medication data" (4.37 ± 0.51), "medication management and medication reminder" (4.28 ± 1.49), and "documentation and presentation of the disease progression" (4.00 ± 1.73). The lowest scores were assigned to desirable features such as "direct interaction and visual contact" (1.14 ± 0.35), "direct audio contact" (1.38 ± 0.49), and "communication with people with similar conditions" (1.40 ± 0.49).

According to the ranking results of the Google Play Store apps based on the absence of undesirable features, the lowest scores were assigned to "corrupted and misleading links" (4.50 ± 0.91), "inactive and misleading buttons" (4.60 ± 0.78), and "taking a long time to load the content of the app after executing the program" (4.66 ± 0.89). The highest ratings in this regard were related to "being a free app but requiring payment for basic features" (4.88 ± 0.44), "difficulties in logging into the app" (4.84 ± 0.51), and "advertising" (4.82 ± 0.48).

The agreement rate between the two evaluators regarding the app's rating based on FARM and MARS tools is shown in Table 2. The agreement rate between the two evaluators for the overall MARS score, calculated using the ICC, was 0.947 (CI 95% = 0.919–0.965). The agreement between the two evaluators for the overall FARM score was 0.882 (CI 95% = 0.819–0.922).

Table 2 The agreement rate between the two evaluators regarding the rating based on FARM and MARS
Full size table

The relationship between the dimensions of the MARS tool and the FARM tool was calculated using the Spearman correlation test, and the results are shown in Table 3. The lowest correlation was found between the score of the Subjective dimension and the functionality score of the MARS tool (r = 0.450). The highest correlation was found between the score of the Subjective dimension and the Information quality score of the MARS tool (r = 0.832).

Table 3 The correlation between the FARM dimensions and The MARS dimensions
Full size table

Discussion

The results of this study showed that there is a higher than the moderate correlation between the scores assigned to apps based on the MARS and FARM tools. Therefore, the high MARS score somewhat indicates the existence of desirable and the absence of undesirable features in the app. So, evaluating mobile apps by using a single tool may not provide good insight about the evaluated apps. However, using more than one tool provides more details about the evaluated apps.

According to the results of this study, the HIV/AIDS mobile applications available in Iran had an "acceptable" ranking. However, the overall ranking score of the MARS tool and desired features for Cafe Bazaar apps scored "poor". In previous studies [27, 44, 59,60,61,62] conducted on health applications using the MARS tool, evaluated apps were ranked as "acceptable". Moreover, the results of the study done by Young et al. [63] showed that the overall quality of the apps for men who have sex with men (MSM) in China is "acceptable". In this study, we evaluated all HIV/AIDS mobile applications available in Iran, including mobile applications related to HIV pre-exposure prophylaxis. Sharpe et al. [64], in 2018, only evaluated 11 mobile apps for HIV pre-exposure prophylaxis using the MARS tool. Just one mobile app (PreP4U) reviewed in the study by Sharpe et al. [64] was available in Iran in 2021 and was also reviewed in our study. The results of the average ratings of this mobile app were almost the same as in our study.

The FARM does not check the quality of information. In this study, the quality of information is checked with the "information quality" section of the MARS tool. The Google Play Store apps were ranked "acceptable" in terms of "information quality", but the Cafe Bazaar apps were rated "poor". The first published part of our study rated the HIV/AIDS mobile apps based on the evidence showed that the Cafe Bazaar apps were rated as "inappropriate" and the Google Play Store applications were rated as "good" [15]. Due to the importance of the information content of an app [65], it is necessary to evaluate and, if possible, eliminate apps with inappropriate information from app stores. The results of our study confirm the results of the Robustillo Cortés et al. [53] study conducted in 2013. Their study indicates that the quality of the evaluated apps on HIV is limited, and only one app (inPractice HIV) is categorized in Class A. This app, in our study, was ranked as "good". According to the results of our study, of all the Café-Bazaar apps, only one has been written by health organizations. Also, the affiliation of more than half of the apps was not determined, and they may not be reliable. This is in line with the results of a study by Rosa et al. [52], which showed that more than half of the apps were not written by health professionals.

In this study, the mean FARM score for all desirable features was low. Most apps were ranked with a low score in terms of their desirable features. However, the high number of features considered in the FARM tool may affect the total desirable score results. Also, the Functionality dimension of the MARS tool had the highest score among all other dimensions of the MARS tool. According to the Functionality dimension, all apps ranked higher than the "acceptable" score, but most ranked as "inadequate" in terms of desirable features. In most previous studies using the MARS tool [26, 28, 30, 60, 66], the functionality score was the highest compared to the scores of the other dimensions of the MARS tool. Therefore, ranking apps with the Functionality dimension of the MARS tool and its general questions cannot conclude that the apps contain desirable features. In the study of Schnall et al. [50], only nine features of HIV/AIDS-related mobile apps were evaluated, and the results had not been compared with other valid tools, nor had the evaluated features been ranked. Our study confirmed the results of the Schnall et al. study [50], which showed that a small number of HIV/AIDS-related apps have the desired functionalities.

In this study, the best features of the HIV/AIDS mobile apps were collecting medication data, medication management and medication reminders, and documentation and presentation of the disease progression. A previous study [67] showed that the most interesting features for HIV/AIDS patients' needs are the reminders/alerts feature, collecting lab data and lab results tracking, and notes about health status. These features were better designed than other features, but they need to be given more attention and designed better.

Based on our results, the lowest undesirable feature score of the Cafe Bazaar apps was assigned to the "being a free app, but requiring payment for basic features" and "advertising" features. The Cafe Bazaar apps did not have a good status in terms of the existence of undesirable features. Although an app may have rich content, it's very annoying for users to face a lot of advertising and unwanted features.

The FARM evaluates and rates mobile apps based on the availability and quality of each feature. The items of the FARM are flexible based on each app's features. In this study, we included all the features available at the time of the study. The features of the apps may change with each update, and apps may contain different features compared to each other. Feature lists also differ from one disease to another. The list of FARM features may change over time with app updates and with a decrease or increase in the number of included apps. The FARM has flexibility and can be used to rank mobile apps to help users choose the app they want.

Currently, most of the features of mobile apps in the App Store are unknown to users, a few apps have mentioned the features in the app description section, but the quality of these features is unknown. So, the results of evaluated HIV/AIDS mobile apps in this study can be helpful for these app users to choose their desired apps. The desirable and undesirable features extracted in the FARM can help mobile app developers to develop new applications for HIV/AIDS patients. Furthermore, the tools used in this study to rate the desirable and undesirable features of mobile apps can be used by researchers to evaluate mobile apps in future works. Further research is required to investigate the implications of the FARM tool.

For future work, we recommend mobile app evaluators use and test the FARM tool to evaluate other mobile health apps. For studies that use the FARM, it is suggested to first prepare a list of the features of the mobile apps that they want to evaluate, then rank the quality of each feature of a mobile app using the FARM, and finally, the mean of these scores is the mobile app's overall score. The FARM can be used for just one app. Mobile app developers can use the FARM to rate their mobile apps. It is also suggested to divide the features into sub-features, first assign a score to the sub-features, and then calculate the feature score based on the average score of the features. Moreover, we recommend using quantitative methods like quantitative usability methods for each feature instead of qualitative methods. Also, for qualitative evaluations, use more than two evaluators or real users.

Strengths and limitations of the study

Using the five-choice ranking in the developed tool in this study for rating app features is one of the strengths of this study because this made the results comparable with the results of the evaluations done with the other tools with the same ranking [17, 19]. Moreover, in this study, we ranked the desirable and undesirable features available in the apps and compared the rankings with the MARS tool.

This study has several limitations. First, this study was conducted on mobile applications available in Iran. Since not all applications are available in Iran due to the sanctions against the country and regional restrictions, this study was conducted in a country with limited access to mobile apps, and therefore the results may not be generalizable to other HIV/AIDS-related mobile apps available in other countries. However, in terms of the number of evaluated HIV/AIDS apps, this study has the highest number of apps being assessed compared to the previous studies [50,51,52,53].

Second, in this study, mobile apps running on the IOS operating system were not evaluated due to sanctions against the country and the removal of Iranian apps from this store. To deal with this issue, the researcher wrote to Apple Company to access the mobile apps for conducting this research but failed to obtain the necessary permission. According to the findings of a previous study [50], more than 60% of HIV/AIDS mobile apps are available on the IOS and Android platforms.

Third, another limitation of this study was using two evaluators to rate 79 mobile apps with many features using FARM and MARS. We used two evaluators because most of the previous studies that used the MARS tool for evaluating mobile health apps used two evaluators [16, 17, 29,30,31,32,33, 36, 42, 44, 45, 64]. The evaluation process may be affected by evaluator bias, fatigue bias, experience bias, and familiarity bias with using two evaluators. To reduce the above mentioned biases in this study, the evaluators were trained about biases before conducting the evaluation. The evaluation was performed when the evaluators were mentally prepared to complete the evaluation, and if the evaluators were tired, the evaluation was stopped and continued until another time when the evaluators had sufficient mental preparation.

Fourth, according to similar studies [15,16,17,18,19,20, 26, 35], we applied a subjective approach to evaluate HIV/AIDS mobile apps. Given to limitations of subjective evaluation methods, using quantitative methods is recommended for future studies.

Conclusions

In this study, we used the MARS tool and a new tool called the FARM to evaluate desirable and undesirable features of HIV/AIDS mobile apps. This study showed that the rank of HIV/AIDS-related available apps in Iran is "acceptable". According to the results of the MARS tool and desirable features, Cafe Bazaar apps were ranked as "poor" and lacked a third of the desirable features. The developers of the Cafe Bazaar apps should add some features based on users’ needs to their apps.

The FARM can determine the desirable and undesirable features of mobile apps and the quality of those features and then rank mobile apps based on their features. Our study results also showed that using a single tool to evaluate mobile apps may not provide good insight to evaluators about the assessed apps. However, using more than one tool provides more details about the evaluated apps. The FARM is a new tool. Therefore, further studies are needed to test the FARM on mobile health apps in different health domains.

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Abbreviations

FARM:

Feature-based Application Rating Method

MARS:

Mobile Apps Rating Scale

HIV:

Human Immunodeficiency Virus

AIDS:

Acquired Immune Deficiency Syndrome

ICC:

Internal Correlation Coefficient

SD:

Standard Deviation

WHO:

World Health Organization

CI:

Confidence Interval

IOS:

IPhone Operating System

References

  1. Ericsson Mobility Report- June 2022 report edition. Ericsson. 2022. https://www.ericsson.com/en/reports-and-papers/mobility-report/reports/june-2022. Accessed 7 Sep 2022.

  2. Mobile operating systems’ market share worldwide from January 2012 to August 2022 | Statista. Statista. 2022. https://www.statista.com/statistics/272698/global-market-share-held-by-mobile-operating-systems-since-2009/. Accessed 7 Sep 2022.

  3. Number of apps available in leading app stores as of 2nd quarter 2022 | Statista. Statista. 2022. https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/. Accessed 7 Sep 2022.

  4. Number of mHealth apps available in the Google Play Store from 1st quarter 2015 to 2nd quarter 2022. Statista. 2022. https://www.statista.com/statistics/779919/health-apps-available-google-play-worldwide/. Accessed 6 Sep 2022.

  5. Peng Y, Wang H, Fang Q, Xie L, Shu L, Sun W, et al. Effectiveness of mobile applications on medication adherence in adults with chronic diseases: a systematic review and meta-analysis. J Manag Care Spec Pharm. 2020;26:550–61.

    PubMed  Google Scholar 

  6. Moses JC, Adibi S, Shariful Islam SM, Wickramasinghe N, Nguyen L. Application of smartphone technologies in disease monitoring: a systematic review. Healthc (Basel, Switzerland). 2021;9:889.

    Google Scholar 

  7. Catalani C. mHealth for HIV treatment and prevention: a systematic review of the literature. Open AIDS J. 2013;7:17–41.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Abbasi R, Nabovati E, Raeesi A, Ostadmohammadi F. Investigating the quality of persian mobile applications related to patients with chronic diseases. J Heal Biomed Inform. 2020;7:273–81.

    Google Scholar 

  9. Zadsar M, Pourfathollah AA, Rasouli M, Karimi G. Trends in sero-epidemiology of human immunodeficiency virus in voluntary blood donations in Iran, 2008–2013. Arch Iran Med. 2017;20:135–40.

    PubMed  Google Scholar 

  10. Islamic Republic of Iran | UNAIDS. UNAIDS. 2022. https://www.unaids.org/en/regionscountries/countries/islamicrepublicofiran. Accessed 7 Sep 2022.

  11. Global Statistics | HIV.gov. HIV.gov. 2022. https://www.hiv.gov/hiv-basics/overview/data-and-trends/global-statistics. Accessed 7 Sep 2022.

  12. Schnall R, Rojas M, Travers J, Brown W, Bakken S, Bakken S, et al. Use of Design science for informing the development of a mobile app for persons living with HIV. AMIA .Annual Symp proceedingsAMIA Symp. 2014;2014:1037–45.

  13. Devi BR, Syed-Abdul S, Kumar A, Iqbal U, Nguyen P-AA, Li Y-CC, et al. mHealth: an updated systematic review with a focus on HIV/AIDS and tuberculosis long term management using mobile phones. Comput Methods Prog Biomed. 2015;122:257–65.

    Article  Google Scholar 

  14. Jiang H, Zhang J, Li X, Ren Z, Lo D, Wu X, et al. Recommending new features from mobile app descriptions. ACM Trans Softw Eng Methodol. 2019;28:1–29.

    Article  Google Scholar 

  15. Raeesi A, Khajouei R, Ahmadian L. Evaluation of HIV/AIDS-related mobile health applications content using an evidence-based content rating tool. BMC Med Inform Decis Mak. 2021;21:135.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Hee Ko KK, Kim SK, Lee Y, Lee JY, Stoyanov SR. Validation of a Korean version of mobile app rating scale (MARS) for apps targeting disease management. Health Inform J. 2022;28:1–15.

    Article  Google Scholar 

  17. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth. 2015;3: e27.

    Article  PubMed  PubMed Central  Google Scholar 

  18. HONcode: Principles for Quality and Trustworthy Health Information. http://www.hon.ch/HONcode/Patients/Conduct.html. Accessed 30 Dec 2020.

  19. DISCERN - The DISCERN Instrument. http://www.discern.org.uk/discern_instrument.php. Accessed 7 Sep 2022.

  20. Chyjek K, Farag S, Chen KT. Rating pregnancy wheel applications using the applications scoring system. Obstet Gynecol. 2015;125:1478–83.

    Article  PubMed  Google Scholar 

  21. Aitken M, Gauntlett C. Patient apps for improved healthcare: from novelty to mainstream. IMS Institute for Healthcare Informatics. 2013; October:60. http://ignacioriesgo.es/wp-content/uploads/2014/03/iihi_patient_apps_report_editora_39_2_1.pdf.

  22. Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mhealth app usability questionnaire (MAUQ): development and validation study. JMIR mHealth uHealth. 2019;7:1–15.

    Article  CAS  Google Scholar 

  23. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile Application Rating Scale (MARS). 2015. https://jmir.org/api/download?alt_name=mhealth_v3i1e27_app2.pdf&filename=86161240dc0e5f7e86306b19d1bcd1a8.pdf.

  24. Carvalho C, Prando BC, Dantas LO, da Serrão PRMS. Mobile health technologies for the management of spine disorders: a systematic review of mHealth applications in Brazil. Musculoskelet Sci Pract. 2022;60:102562.

    Article  PubMed  Google Scholar 

  25. Alwakeel L, Lano K. Functional and technical aspects of self-management mHealth apps: systematic app search and literature review. JMIR Hum Fact. 2022;9(2):e29767.

    Article  Google Scholar 

  26. Schmeelk S, Davis A, Li Q, Shippey C, Utah M, Myers A, et al. Monitoring symptoms of COVID-19: review of mobile apps. JMIR Mhealth Uhealth. 2022;10: e36065.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Escoffery C, McGee R, Bidwell J, Sims C, Thropp EK, Frazier C, et al. A review of mobile apps for epilepsy self-management. Epilepsy Behav. 2018;81:62–9.

    Article  PubMed  Google Scholar 

  28. Choi SK, Yelton B, Ezeanya VK, Kannaley K, Friedman DB. Review of the content and quality of mobile applications about Alzheimer’s disease and related dementias. J Appl Gerontol. 2018;39:601–8.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Saliasi I, Martinon P, Darlington E, Smentek C, Tardivo D, Bourgeois D, et al. Promoting health via mHealth applications using a french version of the mobile app rating scale: adaptation and validation study. JMIR mHealth uHealth. 2021;9:e30480.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Domnich A, Arata L, Amicizia D, Signori A, Patrick B, Stoyanov S, et al. Development and validation of the Italian version of the Mobile Application Rating Scale and its generalisability to apps targeting primary prevention. BMC Med Inform Decis Mak. 2016;16:83.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Martin Payo R, Fernandez Álvarez MM, Blanco Díaz M, Cuesta Izquierdo M, Stoyanov SR, Llaneza SE. Spanish adaptation and validation of the mobile application rating scale questionnaire. Int J Med Inform. 2019;129:95–9.

    Article  PubMed  CAS  Google Scholar 

  32. Yamamoto K, Ito M, Sakata M, Koizumi S, Hashisako M, Sato M, et al. Japanese version of the mobile app rating scale (MARS): development and validation. JMIR mHealth uHealth. 2022;10:e33725.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Bardus M, Awada N, Ghandour LA, Fares EJ, Gherbal T, Al-Zanati T, et al. The Arabic version of the mobile app rating scale: development and validation study. JMIR mHealth uHealth. 2020;8:e16956. https://doi.org/10.2196/16956.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Messner EM, Terhorst Y, Barke A, Baumeister H, Stoyanov S, Hides L, et al. The German version of the mobile app rating scale (MARS-G): development and validation study. JMIR mHealth uHealth. 2020;8:e14479.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Armas R, Montenegro C, Larco A, Yanez C. Identifying key quality features of mhealth applications: unsupervised feature selection approach: MARS case study. Intell Sustain Syst Lect Notes Networks Syst. 2022;333:13–21.

  36. Santo K, Richtering SS, Chalmers J, Thiagalingam A, Chow CK, Redfern J. Mobile phone apps to improve medication adherence: a systematic stepwise process to identify high-quality apps. JMIR mHealth uHealth. 2016;4: e132.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Wu H, Deng W, Niu X, Nie C. Identifying key features from app user reviews. In: 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 2021. p. 922–32.

  38. Mangone ER, Lebrun V, Muessig KE. Mobile phone apps for the prevention of unintended pregnancy: a systematic review and content analysis. JMIR mHealth uHealth. 2016;4: e6.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Luiz W, Viegas F, Alencar R, Mourão F, Salles T, Carvalho D, et al. A feature-oriented sentiment rating for mobile app reviews. In: Proceedings of the 2018 World Wide Web Conference. Republic and Canton of Geneva, CHE: International World Wide Web Conferences Steering Committee; 2018. p. 1909–1918.

  40. Knitza J, Tascilar K, Messner E-M, Meyer M, Vossen D, Pulla A, et al. German mobile apps in rheumatology: review and analysis using the mobile application rating scale (MARS). JMIR mHealth uHealth. 2019;7: e14991.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Sereda M, Smith S, Newton K, Stockdale D. Mobile apps for management of tinnitus: users’ survey, quality assessment, and content analysis. JMIR mHealth uHealth. 2019;7: e10353.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Talwar D, Yeh Y-L, Chen W-J, Chen L-S. Characteristics and quality of genetics and genomics mobile apps: a systematic review. Eur J Hum Genet. 2019;27:833–40.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Diaz-Skeete YM, McQuaid D, Akinosun AS, Ekerete I, Carragher N, Carragher L. Analysis of apps with a medication list functionality for older adults with heart failure using the mobile app rating scale and the IMS Institute for healthcare informatics functionality score: evaluation study. JMIR mHealth uHealth. 2021;9: e30674.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Kim BY, Sharafoddini A, Tran N, Wen EY, Lee J. Consumer mobile apps for potential drug-drug interaction check: systematic review and content analysis using the mobile app rating scale (MARS). JMIR mHealth uHealth. 2018;6: e74.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Choi YK, Demiris G, Lin SY, Iribarren SJ, Landis CA, Thompson HJ, et al. Smartphone applications to support sleep self-management: review and evaluation. J Clin Sleep Med. 2018;14:1783–90.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Cummings E, Borycki EM, Roehrer E. Issues and considerations for healthcare consumers using mobile applications. Stud Heal Technol Inf. 2013;183:227–31.

    Google Scholar 

  47. Kuehnhausen M, Frost VS. Trusting smartphone apps? To install or not to install, that is the question. In: 2013 IEEE International multi-disciplinary conference on cognitive methods in situation awareness and decision support (CogSIMA). 2013. p. 30–7.

  48. Jin M, Kim J. Development and evaluation of an evaluation tool for healthcare smartphone applications. Telemed e-Health. 2015;21:831–7.

    Article  Google Scholar 

  49. Nouri R, Kalhori SRN, Ghazisaeedi M, Marchand G, Yasini M. Criteria for assessing the quality of mHealth apps: a systematic review. J Am Med Inform Assoc. 2018;25:1089–98.

    Article  PubMed  PubMed Central  Google Scholar 

  50. Schnall R, Mosley JP, Iribarren SJ, Bakken S, Carballo-Dieguez A, Brown Iii W, et al. Comparison of a user-centered design, self-management app to existing mhealth apps for persons living with HIV. JMIR Mhealth Uhealth. 2015;3: e91.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Muessig KE, Pike EC, Legrand S, Hightow-weidman LB. Mobile phone applications for the care and prevention of HIV and other sexually transmitted diseases: a review. J Med Internet Res. 2013;15: e1.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Cantudo Cuenca MR, Cantudo Cuenca MD, Morillo VR. Availability and medical professional involvement in mobile healthcare applications related to pathophysiology and pharmacotherapy of HIV/AIDS. Eur J Hosp Pharm. 2013;20:356–61.

    Article  Google Scholar 

  53. Robustillo Cortés M de las A, Cantudo Cuenca MR, Morillo Verdugo R, Calvo Cidoncha E. High quantity but limited quality in healthcare applications intended for HIV-infected patients. Telemed e-Health. 2014;20:729–35.

  54. Cafe Bazaar Online Persian App Store. https://cafebazaar.ir/app. Accessed 7 Sep 2022.

  55. SaeedI MG, Kalhori SRN, Nouria R, Yasini M, Nouri R, Yasini M. Persian mHealth apps: a cross sectional study based on use case classification. Stud Health Technol Inform. 2016;228:230–4.

    PubMed  Google Scholar 

  56. Stoyanov S. MARS Training Video. YouTube. 2016. https://www.youtube.com/watch?v=25vBwJQIOcE. Accessed 7 Sep 2022.

  57. Wyatt JC, Thimbleby H, Rastall P, Hoogewerf J, Wooldridge D, Williams J. What makes a good clinical app? Introducing the RCP health informatics unit checklist. Clin Med J R Coll Phys London. 2015;15:519–21.

    Google Scholar 

  58. Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol. 2012;8:23–34.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Chapman C, Elizabeth Champion K, Psych B, Birrell L, Deen H, Brierley M-E, et al. Smartphone apps about crystal methamphetamine (“Ice”): systematic search in app stores and assessment of composition and quality. JMIR mHealth uHealth. 2018;6: e10442.

    Article  PubMed  PubMed Central  Google Scholar 

  60. Salazar A, de Sola H, Failde I, Moral-Munoz JA. Measuring the quality of mobile apps for the management of pain: systematic search and evaluation using the mobile app rating scale. JMIR mHealth uHealth. 2018;6: e10718.

    Article  PubMed  PubMed Central  Google Scholar 

  61. Moral-Munoz JA, Esteban-Moreno B, Herrera-Viedma E, Cobo MJ, Pérez IJ. Smartphone applications to perform body balance assessment: a standardized review. J Med Syst. 2018;42:119.

    Article  PubMed  Google Scholar 

  62. Mani M, Kavanagh DJ, Hides L, Stoyanov SR. Review and Evaluation of Mindfulness-based iPhone Apps. JMIR mHealth uHealth. 2015;3: e82.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Yang G, Long J, Luo D, Xiao S, Kaminga AC. The characteristics and quality of mobile phone apps targeted at men who have sex with men in China: A window of opportunity for health information dissemination? JMIR mHealth uHealth. 2019;7: e12573.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Sharpe JD, Kamara MT. A systematic evaluation of mobile apps to improve the uptake of and adherence to HIV pre-exposure prophylaxis. Sex Health. 2018;15:587–94.

    Article  PubMed  Google Scholar 

  65. Courtenay T, Baraitser P. Improving online clinical sexual and reproductive health information to support self-care: a realist review. Digit Heal. 2022;8:20552076221084464.

    Google Scholar 

  66. Bahadori S, Wainwright TW, Ahmed OH. Smartphone apps for total hip replacement and total knee replacement surgery patients: a systematic review. Disabil Rehabil. 2018;42(7):983–8. https://doi.org/10.1080/09638288.2018.1514661.

    Article  PubMed  Google Scholar 

  67. Schnall R, Bakken S, Rojas M, Travers J, Carballo-Dieguez A. mHealth technology as a persuasive tool for treatment, care and management of persons living with HIV. AIDS Behav. 2015;19:81–9.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We would like to thank Fatemeh Tabatabaei for her contribution to the app's evaluation. Also, we would like to thank Dr. Mohammad Reza Safaei for his consultation about HIV/AIDS disease. The authors would like to thank Felix Holl, a researcher working at the Neu-Ulm University of Applied Sciences, Germany, for his valuable comments and insights on the manuscript.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Student Research Committee, Mashhad University of Medical Sciences, Mashhad, Iran

    Ahmad Raeesi

  2. Department of Health Information Sciences, Kerman University of Medical Sciences, Kerman, Iran

    Ahmad Raeesi

  3. Department of Health Information Sciences, Faculty of Management and Medical Information Sciences, Kerman University of Medical Sciences, Kerman, Iran

    Reza Khajouei

  4. HIV/STI Surveillance Research Center, WHO Collaborating Center for HIV Surveillance, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Haft-Bagh Highway, Kerman, Iran

    Leila Ahmadian

Authors
  1. Ahmad Raeesi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reza Khajouei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leila Ahmadian
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LA, RK, and AR designed the study. LA and RK supervised the project. AR contributed to the app's evaluation. AR and LA analyzed and interpreted the data. LA and AR wrote the final manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Leila Ahmadian.

Ethics declarations

Ethics approval and consent to participate

In this study, all apps were downloaded directly from the Google Play Store and Cafe Bazaar, and no apps were downloaded illegally. This study was approved by the Research Ethics Committee of the Kerman University of Medical Sciences (I.D. approved: IR.KMU.REC.1397.246). We confirm that all methods were performed in accordance with the relevant guidelines and regulations. We did not include individuals as study participants, so there was no requirement for informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

 The Feature-Based Application Rating Method (the FARM) Tool.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Raeesi, A., Khajouei, R. & Ahmadian, L. Evaluating and rating HIV/AIDS mobile apps using the feature-based application rating method and mobile app rating scale. BMC Med Inform Decis Mak 22, 281 (2022). https://doi.org/10.1186/s12911-022-02029-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12911-022-02029-8

Keywords

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us