Skip to main content
Download PDF

Evaluation of usability and user feedback to guide telepharmacy application development in Indonesia: a mixed-methods study

BMC Medical Informatics and Decision Making volume 24, Article number: 130 (2024) Cite this article

Abstract

Background

In Indonesia, the adoption of telepharmacy was propelled by the COVID-19 pandemic, prompting the need for a user-friendly application to support both the general population and pharmacists in accessing healthcare services. Therefore, this study aimed to evaluate usability and user feedback of a pioneering telepharmacy application known as Tanya Obat (translating to “Ask about Medications”) in Indonesia, from the perspectives of the general population and pharmacists.

Methods

A mixed-methods sequential study was conducted with the early-stage Tanya Obat application in Bandung City. Participants, including the general population and pharmacists, were instructed to use the application for a week. Questionnaires for the general population and pharmacists were distributed from March to May and February to June 2023, respectively. The System Usability Scale questionnaire was adopted to describe usability of the developed application. Further exploration of the quantitative results required collecting open-ended feedback to assess the impressions of the participants, difficulties encountered, and desired features for enhanced user-friendliness. The collected statements were summarized and clustered using thematic analysis. Subsequently, the association between the characteristics of participants and perceived usability was determined with the Chi-square test.

Result

A total of 176 participants, comprising 100 individuals from the general population and 76 pharmacists, engaged in this study. In terms of usability, the questionnaire showed that Tanya Obat application was on the borderline of acceptability, with mean scores of 63.4 and 64.1 from the general population and pharmacists, respectively. Additionally, open-ended feedback targeted at achieving a more compelling user experience was categorized into two themes, including concerns regarding the functionality of certain features and recommendations for improved visual aesthetics and bug fixes. No significant associations were observed between the characteristics of participants and perceived usability (p-value > 0.05).

Conclusion

The results showed that the perceived usability of Tanya Obat developed for telepharmacy was below average. Therefore, feature optimizations should be performed to facilitate usability of this application in Indonesia.

Peer Review reports

Background

The development of telepharmacy, with technology incorporation into pharmaceutical care services, was propelled by the COVID-19 pandemic. Telepharmacy provides remote counseling, medication information, online purchases, adverse effects monitoring, and therapy tracking through digital platforms [1, 2]. Moreover, it creates a virtual channel for pharmaceutical care, connecting pharmacists and patients remotely to facilitate easy health evaluation [3]. Other benefits include reducing direct interaction between healthcare professionals and patients, improving pharmaceutical service quality, and minimizing medication errors and adverse effects through various channels such as application, text messaging, video, and voice calls [2]. Telepharmacy improves cost-effectiveness, healthcare access, and after-hours availability, as well as reduces travel time to healthcare facilities [4, 5], breaking geographical barriers and enhancing healthcare accessibility [6, 7]. However, implementing this system in clinical practice may be challenging due to legal considerations, operational costs, and patient trust-building [8, 9].

Recent guidelines in Indonesia mainly address general aspects of delivering telehealth services during the COVID-19 pandemic, particularly focusing on definitions and procedures [10]. Specific regulations for telemedicine are necessary to ensure compliance with ethics and other regulations [11]. Additionally, evaluation of usability and applicability is crucial for optimal implementation and adoption in real-world settings. Gaining insights into usability from the perspectives of the general population and pharmacists during the design and testing stages can enhance application effectiveness, which is essential for successful implementation and scalability [12,13,14]. Despite the rapid growth of telepharmacy in the country [15,16,17,18], only a few available application is tested [19,20,21], leading to varied usability experiences based on features and limitations encountered [19,20,21]. Many types of Indonesian telepharmacy application lack appropriate study designs and methods to address engagement issues faced by users and pharmacists. Considering the interconnection between program usability and the engagement of users, usability studies are essential for understanding and improving engagement [22]. Usability is defined as a user interface characteristic that facilitates application adoption, effectiveness, efficiency, and satisfaction in a targeted environment [23]. In telepharmacy implementation, poor usability can hinder technology acceptance [24]. Therefore, the products, systems, processes, and procedures constituting telepharmacy must be designed and implemented to be usable, useful [25], accessible, and user-friendly for both healthcare providers and patients [26]. Engaging these users in usability testing can help to address specific needs and preferences, promoting successful technology acceptance and adoption [27].

An Indonesian telepharmacy application known as Tanya Obat, translating to “Ask about Medication”, is developed through design-based study [28] by a team of pharmacists and academicians. This provides a comprehensive ecosystem connecting pharmacies, pharmacists, and technicians with the general population. Furthermore, the application comprises features such as locating nearby pharmacies, online medication purchasing, consultation services, medication use information, and educational resources including webinars, e-modules, and coaching clinics for pharmacists and technicians. Tanya Obat differs from other types of application in Indonesia by offering a dedicated ecosystem for pharmacists, including online medication consultation and educational opportunities to enhance competency in delivering health services. Therefore, this study aimed to evaluate Tanya Obat usability and obtain user feedback on the application at early development stage in Bandung City from the perspectives of the general population and pharmacists. The association between the characteristics of participants and perceived usability will be explored.

Method

Study design

A mixed-methods sequential study design was used to assess usability and obtain user feedback on Tanya Obat application at the early development stage before launching. Furthermore, questionnaires for the general population and pharmacists were distributed from March to May and February to June 2023, respectively. Approval with contract number 670/UN6.KEP/EC/2022 was received from the Research Ethics Commission of Universitas Padjadjaran, Indonesia, and all recruited participants provided written informed consent.

Study population and settings

Participants recruited in Bandung City comprised untrained and first-time Tanya Obat users from the general population, irrespective of being health system users, as well as pharmacists registered in the application or not. One week was provided for Tanya Obat usage without supervision to permit full exploration of the application, and recruiting was performed online through a convenience sampling method. Inclusion criteria for the general population consisted of individuals with (i) age above 18 years, (ii) ability to use a smartphone, (iii) lack of prior application experience, and (iv) willingness to participate. Meanwhile, the inclusion criteria for pharmacists were (i) current employment at a Community Health Center (CHC), hospital, or pharmacy, (ii) ability to use a smartphone, (iii) no prior experience with the tested application, and (iv) willingness to participate. All participants received an explanation regarding the study stages, application installation, and features.

Tanya Obat application

Tanya Obat application was developed by a team of pharmacists and pharmacy academicians in collaboration with software engineers using Dart (https://dart.dev/) programming language. As shown in Fig. 1, the application offers five main features including:

Fig. 1
figure 1

Tanya Obat Application Features

Full size image
  1. 1.

    Finding Nearby Pharmacies

This feature allows users to conveniently search for the price sites of nearby pharmacies and view available medication lists, saving time spent on locating the physical buildings.

  1. 2.

    Medication Purchasing

The medication purchasing feature extends the reach of pharmacies to patients through a digital platform. Users can freely browse categories of non-prescription and prescription medications which can be ordered with the delivery service available on the application.

  1. 3.

    Ask a Pharmacist

This flagship feature enables digital consultation with pharmacists regarding self-medication, regular medications currently in use, and other concerns about consumption and side effects. Additionally, it offers pharmacists the opportunity to gain consultation experience, which can subsequently be converted into a credit unit for pursuing professional competency certification. Recruiting for this feature was performed by distributing information through webinars, scientific events, social media and ads, personal and group chat application, as well as direct visits to pharmacies. Users will be charged IDR 20,000 or USD 1.5 for 15-minute consultations based on regulations established by the Indonesian Pharmacists Association in 2019.

  1. 4.

    Health Article

Tanya Obat application allows both the general population and pharmacists to access accurate medication information in a clear and easily understandable manner. Moreover, the authors of this study were accredited pharmacists, pharmacy graduates, and students with competencies and excellent knowledge in writing health articles. Each published article was successfully curated and reviewed by professionals and an editorial team.

  1. 5.

    Sustainable Educational Development for Pharmacists

This feature can only be accessed by pharmacists, serving as a channel for enhancing educational knowledge and professional competency through webinars, e-modules, and coaching clinics.

Each feature of Tanya Obat application is available for both the general population and pharmacists. However, “Ask a Pharmacist” has slightly different user interfaces tailored for the respective group of participants. This specific feature shows a list of online pharmacists for the general population. Meanwhile, the inability of accounts designed for pharmacists to request consultations as patients, is complemented by the provision of access to current and completed consultation history.

Measurements

Perceived usability was evaluated based on a standardized quantitative System Usability Scale (SUS). Meanwhile, additional feedback from users engaged in this study was assessed through open-ended qualitative questions.

Usability measurement

Usability, often defined as user interface characteristic that facilitates application adoption, effectiveness, efficiency, and satisfaction in a targeted environment [23], was assessed with the Indonesian version of the standardized SUS questionnaire [29, 30]. This evaluation was conducted by identifying potential user interface, functionality, and design issues based on the perspective of untrained, first-time users from the general population and pharmacists. Moreover, SUS questionnaire comprised 10 questions rated on a 5-point Likert scale ranging from [1] “Strongly Disagree”, “Disagree”, “Neutral”, “Agree”, to “Strongly Agree” [5]. Participants responded based on the subjective assessment of usability, as presented in Table S1 Supplementary data. Each question contributed a score, with different scoring conversions for odd [1, 3, 5, 7, 9] and even [2, 4, 6, 8, 10] questions, reflecting positive and negative responses, respectively. Summation of all the points yielded a maximum score of 40 which generated a scale ranging from 0 to 100 through multiplication by 2.5. Higher scores suggested favorable user perceptions of the application, while lower scores signified low usability [31]. A previous study showed a significant association between SUS scores reported as continuous or dichotomous data and outcomes [32]. Reporting as dichotomous data may be more practical in real-world scenarios due to the ease of interpretation [33]. Therefore, in this study, a SUS score exceeding 68 points was considered above average in perceived usability [34], calculated based on a published formula [33].

The questionnaire was tested for construct validity and internal consistency using 30 participants who were not included in the main analyses. Construct validity was assessed with the Pearson Product Moment correlation by correlating the score of each question with the total score [35]. The questionnaire was deemed valid when the correlation coefficient (r-value) exceeded the critical value [28]. The values for each question were > 0.361, signifying correlation with the respective dimension and affirming this construct as a measurement instrument. Additionally, Cronbach’s Alpha coefficient yielded a score of 0.721, meeting the criterion established for internal consistency, namely a value > 0.60 [36].

Additional user feedback

After completing SUS questionnaire, participants provided additional feedback based on open-ended questions to obtain impressions following one week of Tanya Obat usage. Three questions were used to assess the perceptions of participants about the application (e.g.,), potential confusion or difficulties encountered (e.g.,), and desired features for enhanced user-friendliness. These included “What was your first impression of this application?”, “What would you say was the most challenging aspect of using this application?”, and “What are your recommendations to improve the application?”. Subsequently, a summary of interpretations made by the study team was sent to all participants for verification.

Demographic characteristics

The survey collected sociodemographic information about the participants, including factors such as gender (male, female), age, highest education level (junior high school, senior high school, bachelor’s degree, registered pharmacists, master’s degree, doctorate), practice settings, and years of practice (for pharmacists). Additionally, participants provided information on subjective experiences with smartphone application (amateur, beginner, skilled, highly skilled), daily duration of smartphone usage in hours (< 1, 1−2, 3−4, and > 4), and internet accessibility at home (available, unavailable).

Sample size calculation

The sample size was calculated using Slovin’s formula [37] due to a lack of prior knowledge about outcome distributions. To achieve a 95% confidence interval (CI) and a margin error of 0.15 with a statistical power of 80%, a minimum sample size of 100 for the general population and 64 pharmacists was required.

Data collection

The link to both the application and survey was disseminated through a digital leaflet across social media, as well as online personal and group chat platforms. This application was available for download on the Google Play store, and data were collected using the Qualtrics platform. To prevent duplicate entries, participants were restricted to filling out the questionnaire only once from a single email address. For pharmacists, a convenient sample was formally or personally recruited through the Indonesian Pharmacists Association network. The estimated time for questionnaire completion was 10 min, while participants were initially reminded that the study aimed to assess user-friendliness of the application and not the ability to use the device correctly. Following one week of exploring the application, participants were immediately instructed to complete the questionnaire to avoid bias introduced by repeated usage and becoming overly familiar with the application. Additionally, screenshots of the features found were requested as evidence of exploring the application.

Data analysis

Basic demographic characteristics were examined using descriptive statistics, where mean SUS scores were analyzed for each subgroup of the general population and pharmacists. Survey quantitative data were tabulated and presented as frequencies, then a Chi-square test was used to determine the association between the characteristics of participants and perceived usability. Open-ended questions were summarized and clustered with thematic analysis, following steps including familiarization with data, initial code generation, theme development, theme review, theme definition, and conduction of independent qualitative analysis by JAS and DQA using NVivo software version 12. Any disagreements among JAS and DQA were resolved through discussion with a third study team member (SDA). To ensure content analysis reliability, continuous discussion and negotiation regarding the content of keywords, broader concepts, and units of meaning were performed among the team.

Result

Quantitative results

Demographic characteristics

A total of 176 participants (100 individuals from the general population and 76 pharmacists) engaged in this study. Those from the general population were recruited from 26 districts in Bandung City, majorly comprising 18−25 years old (59%) females (71%) with high school qualifications (71%). Furthermore, most participants were ‘skilled’ at using smartphone application (56%), with over four hours of daily smartphone usage (65%), and found to use Wi-Fi at home (72%), as shown in Table 1.

Table 1 The characteristics of participants
Full size table

Subsequently, a total of 76 responses were obtained from pharmacists in Bandung City, Indonesia. These majorly included females (73.7%), working in community pharmacies (79.0%), with over six years of job experience (38.2%), considered ‘skilled’ in using mobile application (64.5%), possessing access to the internet (89%), and known to use the smartphone for more than four hours daily (63.2%).

Associations between the characteristics of participants and perceived usability

The characteristics of the general population and pharmacists were presented in Tables 2 and 3, respectively. This study identified no significant correlations between the characteristics and perceived application usability (p-value > 0.05).

Table 2 Associations between the characteristics of the general population and perceived usability (N = 100)
Full size table
Table 3 Associations between the characteristics of pharmacists and perceived usability (N = 76)
Full size table

Usability results

According to the calculation matrix in Table S2 Supplementary data, the average SUS score of the general population was 63.4, representing a below-average perceived usability. A significant variation in scores was observed, ranging from a high of 87.5 to a low of 30. The average SUS score of pharmacists was 64.1 (Table S3, Supplementary data), which was also below average, with a significant variation in scores ranging from 95 to 42.5.

Qualitative results

Two themes, including concerns and recommendations, were identified from the responses provided to the open-ended questions. These were further divided into subthemes comprising feature functionality for concerns as well as visual enhancements, feature improvements, system improvements, and other parameters for recommendations, as presented in Table 4 along with examples of supporting quotes.

Table 4 General population concerns and recommendations for Tanya Obat application (N = 100)
Full size table

User feedback: the general population

Concerns identified for Tanya Obat were features such as consulting a pharmacist, searching for online medication purchases, and locating nearby pharmacies. Users reported that the application was not functioning optimally and required assistance using features including purchasing medication, redeeming prescribed medicines, and consulting a pharmacist. Additionally, inconsistencies in color schemes and delays or errors when accessing features within the application were noted (Table 4).

Potential enhancements for the application included creating more visually appealing shows in terms of color combinations, layout, and font styles. Additionally, it was suggested to ensure that features function as intended, providing relevant descriptions for each feature, and enhancing filters. System-related recommendations comprised addressing bugs or errors such as loading failures, adding a user guide, improving maps, and enhancing search speed. Other suggestions included introducing Tanya Obat on iOS devices and promoting the application among a wider demographic (Table 4).

User feedback: pharmacists

Concerns observed in using Tanya Obat application were related to medication purchase, address configuration, and specific hurdles in the registration of pharmacists. Another concern was identified with the “consult a pharmacist” feature, where the need for more assistance was reported (Table 5).

Table 5 Concerns and recommendations of pharmacists for Tanya Obat application (N = 76)
Full size table

Several suggestions obtained from feedback in open-ended questions were majorly related to features. Regarding the medication purchase feature, the requirement for expanded stock availability and the incorporation of new functionalities was reiterated. Furthermore, pharmacists stated the need for an online Continuing Professional Development (CPD) feature, which would be incorporated by the Indonesian Pharmacists Association as the issuer of competency certificates. This suggestion was provided to support consistent education through seminars, articles, and learning modules, along with the integration of counseling history or medical records of patients that can be accessed by pharmacists in Indonesia and claimed as credit points for CPD. The feature for registration of pharmacists should be enhanced, while modification of the color palette was recommended to ensure increased visibility and maintain satisfactory show and font choices.

Discussion

Principal observations

In this study, usability and user feedback of the innovative Tanya Obat application were evaluated. Perceived usability was categorized as below average, with average scores of 63.4 and 64.1 for the general population and pharmacists, respectively. The identified concerns primarily included the functionality of some features, while chances for improvement were observed in the areas of visual, features, and system enhancements. These concerns and recommendations slightly differed between the general population and pharmacists.

From the general population perspective, the below-average score obtained might be attributed to disparities in digital literacy, defined as the ability to acknowledge and use information from various sources presented through computers [38]. A national survey conducted among 10,000 Indonesians showed a digital literacy index of 3.49 in 2021 and 3.54 on a scale of 5 in 2022 [39,40,41], and despite the improvement, digital literacy in the country remained at a moderate level. Previous results showed that the general population and pharmacy students in Indonesia had a positive perception and were willing to use telepharmacy services [42, 43], providing an opportunity for successful implementation of the application.

Several areas for improvement were identified to enhance and adapt the future application to the understanding and capabilities of users, thereby leading to a more usable and useful system [25 ]. Moreover, primarily existing problems included complications using the features, attributed to unfamiliarity with technology, lack of confidence in using electronic devices, or fear of committing mistakes [39]. A survey reported that 69% of 10,000 Indonesians have not accessed health services through digital platforms [39]. Complications while running the application can be reduced by simplifying the log-in process, reducing required tasks, and showing fewer buttons on the screen [44,45,46,47,48,49,50,51,52,53,54,55]. Selecting the appropriate design, wording, and development language in a user-centered and participatory design process is crucial and may have an important impact on engagement [56]. These improvements tend to increase satisfaction, which plays a vital role in the implementation and continual use of the application [57, 58]. Therefore, developers must prioritize maximizing the application performance to be more user-friendly [44, 59,60,61,62,63,64].

The similar marginally acceptable score obtained from the feedback of pharmacists signified that more work was needed for usability improvement, as these professionals were expected to be more exposed to telepharmacy application than the general population. This unsatisfying score could be partly explained by the varying levels of readiness to use telepharmacy application [65]. Pharmacists were forced to implement telehealth services without adequate readiness assessment, particularly in a setting without well-established telehealth services before the COVID-19 pandemic, such as Indonesia [66] Furthermore, the main concerns that require adjustment to enhance application usability and reduce errors include expanding the network of available pharmacies and pharmacists with specialties comprising traditional medication and cosmetics. Specialization among pharmacists has various important benefits such as higher adherence and persistence, better clinical outcomes, monetary benefits for both patients and the healthcare system, and higher patient satisfaction [67]. Therefore, expanding the network of the pharmacy sector to include traditional medication specialists and pharmacist cosmetologists will significantly improve the application.

Recommendations of pharmacists regarding giving details about all medication were consistent with a previous study that reported an increment of adherence among patients after providing simple and brief written medication information [68]. Additionally, offering live interaction during counseling led to care quality improvement. Specific concerns need to be addressed, such as ensuring properly configured camera and audio settings, appropriate light quality, stable internet connection, and readiness to assist patients unfamiliar with the technology [69]. Furthermore, pharmacists recommended interactive video learning for application enhancement. This was consistent with another study that showed effectiveness in interactive video [70, 71], with an 89.7% reported increase in learning outcomes [70]. Interactive virtual content had significant effectiveness compared to the online class group method [71]. Another recommendation was related to visualization enhancement, playing an important role in attracting user attention [44] with a tendency to influence perceived usability even when no differences existed in functionality offered [72]. Users preferred simplicity, showing more graphics than crowded text, and consistent style using combinations of colors [45,46,47, 73,74,75]. Consequently, telepharmacy application usability would be optimized with the results of this study before proceeding to evaluate effectiveness because the standards of health professionals need to be met [76].

The characteristics of the general population and pharmacists were found to have no significant association with perceived usability. These results showed that the application could be used for all age groups irrespective of educational background, familiarity with smartphone operation, and internet accessibility level [77]. However, certain studies reported inconsistent results on the association between sociodemographic factors and perceived usability among the general population and pharmacists, respectively [77,78,79]. Significant associations were previously shown between demographic-related variables and usability [80] from an investigation conducted among participants with years of mobile application experience [80]. The less experienced participants using mobile health application in Indonesia and their subjectivity might lead to under or overestimating the experience had with smartphones, which could clarify the non-significant associations observed in this study [81]. Moreover, some unmeasured factors tended to be associated with perceived usability, such as readiness [81], experience in using mobile health application [80], digital literacy [38], and health literacy [82, 83].

Implications and future directions

The mixed-methods evaluation conducted in this study and the results represent the first step in optimizing the development and evaluation of Tanya Obat application. A previous investigation assessed satisfaction with telehealth application using SUS method and showed improvement from 71.8 to 82.5 following enhancements based on the first round of testing [84]. Therefore, SUS score tends to increase when adequate improvements are provided to Tanya Obat application. These results support the importance of incorporating usability studies as part of the digital health intervention design process [85].

Continuous explorations to obtain post-refinement data with a larger sample size from a multicenter study in different provinces in Indonesia are essential to provide more accurate representative results. Additional investigations are required to evaluate the impact of Tanya Obat application, particularly among patients with chronic diseases who regularly consume medications, as usability may vary between different clinical groups. This can help identify context-related issues in the future, such as patient adherence, access to healthcare, and satisfaction.

Strengths and limitations

The strengths of this study include the recruitment of two different subjects, namely the general population and pharmacists, representing potential and relevant end users. Furthermore, the quantitative and qualitative data provided by real-world users are crucial in ensuring the addition of obtained views in the application development to improve usability. The inclusion of real-world participants showed important usability problems and unidentified solutions during the development or expert panel review stages. However, one main limitation of convenience sampling is the risk of bias due to the lack of random selection. Certain groups within the population may be overrepresented or underrepresented due to the inability to control the questionnaire link distribution. This skew might lead to the production of results not accurately representing the entire population. For example, the majority of participants were aged below 50 years and had a moderate to high educational level, both of which were factors increasing the tendency of technological proficiency and willingness to adopt new technology [86, 87]. In future investigations, combination of multiple methods, such as random, stratified, or systematic sampling, can improve the quality and representativeness of the convenience sample, which tends to produce more accurate and reliable results, thereby enhancing generalizability. Furthermore, the subjectivity of respondents may be resulting in under or overestimating their experience in using smart-phones which may lead to the non-significant association with the usability score. There was a challenge in ensuring full exploration of all Tanya Obat application features, and the reliance on screenshots from feedback provided by participants without direct verification constituted potential limitations. Future investigations need to conduct a real-time observation or the recording of screen activity as a more objective measurement [88]. The method of this study was limited to user-based usability evaluation, focusing solely on the satisfaction aspect, and not incorporating the think-aloud method [89]. Additionally, expert usability evaluation through heuristic tests [90] were not performed, generating a less comprehensive scope for the results obtained. Despite these limitations, Tanya Obat application usability was improved before launching by making changes based on the concerns and recommendations of the target user groups.

Conclusion

In conclusion, the results showed that the developed Tanya Obat application was below average in perceived usability. Therefore, specific feature optimizations should be performed, particularly in terms of visual appeal, features, and system functionality, to improve potential acceptance and usability, facilitating successful adoption in Indonesia.

Data availability

The datasets used and/or analyzed are available from the corresponding authors upon reasonable request.

References

  1. Ahmed NJ, Almalki ZS, Alsawadi AH, Alturki AA, Bakarman AH, Almuaddi AM, et al. Knowledge, perceptions, and readiness of Telepharmacy among Hospital pharmacists in Saudi Arabia. Healthcare. 2023;11(8):1087. https://doi.org/10.3390/healthcare11081087

    Article  PubMed  PubMed Central  Google Scholar 

  2. Farid AF, Firdausy AZ, Sulaiman AM, Simangunsong DE, Sulistyani FE. Efektivitas Penggunaan Layanan Telefarmasi Di Era Pandemi COVID-19 Dari Perspektif Masyarakat. Jurnal Farmasi Komunitas. 2022;9(2):152–7. https://doi.org/10.20473/jfk.v9i2.32924

    Article  Google Scholar 

  3. Ridho A, Alfian SD, van Boven JFM, Levita J, Yalcin EA, Le L, et al. Digital Health Technologies to improve medication adherence and treatment outcomes in patients with tuberculosis: systematic review of Randomized controlled trials. J Med Internet Res. 2022;24(2):e33062. https://doi.org/10.2196/33062

    Article  PubMed  PubMed Central  Google Scholar 

  4. Baldoni S, Amenta F, Ricci G. Telepharmacy services: Present Status and Future perspectives: a review. Med (B Aires). 2019;55(7):327. https://doi.org/10.3390/medicina55070327

    Article  Google Scholar 

  5. Ameri A, Salmanizadeh F, Keshvardoost S, Bahaadinbeigy K. Investigating pharmacists’ views on Telepharmacy: prioritizing Key relationships, barriers, and benefits. J Pharm Technol. 2020;36(5):171–8. https://doi.org/10.1177/8755122520931442

    Article  PubMed  PubMed Central  Google Scholar 

  6. Alfian SD, Insani WN, Puspitasari IM, Wawruch M, Abdulah R. Effectiveness and Process Evaluation of Using Digital Health Technologies in Pharmaceutical Care in Low- and Middle-Income Countries: A Systematic Review of Quantitative and Qualitative Studies. Telemed J E Health. 2023;29(9):1289–303. https://doi.org/10.1089/tmj.2022.0406

    Article  PubMed  Google Scholar 

  7. Atmojo JT, Sudaryanto WT, Widiyanto A, Ernawati E, Arradini D, Telemedicine. Cost effectiveness, and patients satisfaction: a systematic review. J Health Policy Manage. 2020;5(2):103–7. https://doi.org/10.26911/thejhpm.2020.05.02.02

    Article  Google Scholar 

  8. Poudel A, Nissen L. Telepharmacy: a pharmacist’s perspective on the clinical benefits and challenges. Integr Pharm Res Pract. 2016;5:75–82. https://doi.org/10.2147/IPRP.S101685

  9. Elbeddini A, Yeats A. Pharmacist intervention amid the coronavirus disease 2019 (COVID-19) pandemic: From direct patient care to telemedicine. J Pharm Policy Pract. 2020;13(1):1–4. https://doi.org/10.1186/s40545-020-00229-z.

  10. Menteri Kesehatan Republik Indonesia. Pedoman, Pelayanan Kesehatan Melalui Telemedicine Pada Masa Pandemi Corona. Virus Disease 2019 (COVID-19) [Internet]. 2021 [cited 2022 Oct 4]. https://jdih.kemkes.go.id/.

  11. Sulistiyono A, Budiyanti RT, Sriatmi. A regulatory framework for telemedicine in Indonesia. Eubios Journal of Asian and International Bioethics [Internet]. 2019 [cited 2023 Nov 17];29(4). https://philpapers.org/rec/SULARF-3.

  12. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A’Court C et al. Beyond Adoption: A New Framework for Theorizing and Evaluating Nonadoption, Abandonment, and Challenges to the Scale-Up, Spread, and Sustainability of Health and Care Technologies. J Med Internet Res. 2017 N;19(11). https://doi.org/10.2196/jmir.8775

  13. Jones RB, Ashurst EJ, Trappes-Lomax T. Searching for a sustainable process of service user and health professional online discussions to facilitate the implementation of e-health. Health Informatics J. 2016;22(4):948–61. https://doi.org/10.1177/1460458215599024

  14. Zapata BC, Fernández-Alemán JL, Idri A, Toval A. Empirical studies on usability of mHealth apps: a systematic literature review. J Med Syst. 2015;39(2):1–19. https://doi.org/10.1007/s10916-014-0182-2

  15. Fitrina Andiani A, Taruna B, Putra1 W, Khoiri A. The Future of Telemedicine in Indonesia During Covid-19 Pandemic Era: Literature Review. Health Technology Assessment in Action. 2022;6(2). https://doi.org/10.18502/htaa.v6i2.12198

  16. Komalasari R. Telemedicine in Pandemic Times in Indonesia: Healthcare Professional’s Perspective. https://services.igi-global.com/resolvedoi/resolve.aspx?doi=104018/978-1-6684-5499-2.ch008 [Internet]. 1AD Jan 1 [cited 2023 Nov 17];138–53. https://www.igi-global.com/chapter/telemedicine-in-pandemic-times-in-indonesia/314113.

  17. Antarsih NR, Setyawati SP, Ningsih S, Deprizon, Sulaiman E, Pujiastuti N. Telehealth Business Potential in Indonesia. Proceedings of the International Conference on Social, Economics, Business, and Education (ICSEBE 2021). 2022;205:73–8. https://doi.org/10.2991/aebmr.k.220107.015

  18. Dentons. - The Rise of Telemedicine in Indonesia [Internet]. [cited 2023 Oct 31]. https://www.dentons.com/en/insights/articles/2020/july/20/the-rise-of-telemedicine-in-indonesia.

  19. Novrianda D, Herini ES, Haryanti F, Supriyadi E, Lazuardi L. Chemo assist for children mobile health application to manage chemotherapy-related symptoms in acute leukemia in Indonesia: a user-centered design approach. BMC Pediatr. 2023;23(1). https://doi.org/10.1186/s12887-023-04076-0

  20. Rahayu SR, Zainafree I, Merzistya ANA, Cahyati WH, Farida E, Wandastuti AD et al. Development of the SIKRIBO Mobile Health Application for Active Tuberculosis Case Detection in Semarang, Indonesia. Healthc Inform Res. 2022;28(4):297–306. https://doi.org/10.4258/hir.2022.28.4.297

  21. Fitria N, Idrus L, Putri AR, Sari YO. The usability testing of the integrated electronic healthcare services for diabetes mellitus patients during the pandemic in Indonesia. Digit Health. 2023;9. https://doi.org/10.1177/20552076231173227

  22. Nitsch M, Dimopoulos CN, Flaschberger E, Saffran K, Kruger JF, Garlock L et al. A Guided Online and Mobile Self-Help Program for Individuals With Eating Disorders: An Iterative Engagement and Usability Study. J Med Internet Res. 2016;18(1). https://doi.org/10.2196/jmir.4972

  23. IEC 62366-1. 2015(en), Medical devices — Part 1: Application of usability engineering to medical devices [Internet]. [cited 2023 Oct 30]. https://www.iso.org/obp/ui/#iso:std:iec:62366:-1:ed-1:v1:en.

  24. Scholtz B, Mahmud I, Ramayah T. Does usability matter? An analysis of the impact of usability on technology acceptance in ERP settings. Interdisciplinary Journal of Information, Knowledge, and Management. 2016;11:309–330. https://doi.org/10.28945/3591

  25. Kayser L, Kushniruk A, Osborne RH, Norgaard O, Turner P. Enhancing the effectiveness of consumer-focused health information technology systems through ehealth literacy: a framework for understanding users’ needs. JMIR Hum Factors. 2015;2(1). https://doi.org/10.2196/humanfactors.3696

  26. Senjam SS, Manna S, Bascaran C. Smartphones-based assistive technology: accessibility features and apps for people with visual impairment, and its usage, challenges, and usability testing. Clinical Optometry. 2021;13:311–22. https://doi.org/10.2147/OPTO.S336361

  27. Sripathi V, Sandru V. Effective usability testing-knowledge of user centered design is a key requirement. Int J Emerg Technol Adv Eng [Internet]. 2013;3(1):627–35. Available from: www.ijetae.com.

    Google Scholar 

  28. Jayatilleke BG, Ranawaka GR, Wijesekera C, Kumarasinha MCB. Development of mobile application through design-based research. Asian Association Open Universities J. 2019;13(2):145–68. https://doi.org/10.1108/AAOUJ-02-2018-0013

    Article  Google Scholar 

  29. Nugroho HNIA, Ferdiana PI. R. Pengujian Usability Website Menggunakan System Usability Scale. JURNAL IPTEKKOM: Jurnal Ilmu Pengetahuan & Teknologi Informasi. 2015;17(1):31.

  30. Brooke JSUS. A quick and dirty usability scale. Usability Evaluation Ind. 2020;207–12.

  31. Bangor A, Staff T, Kortum P, Miller J, Staff T. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud. 2009;4(3):114–23.

    Google Scholar 

  32. Bloom BM, Pott J, Thomas S, Gaunt DR, Hughes TC. Usability of electronic health record systems in UK EDs. Emerg Med J. 2021;38(6):410–415 https://doi.org/10.1136/emermed-2020-210401

  33. Sauro J, Lewis JR. Quantifying the User Experience. Quantifying the User Experience [Internet]. 2012 [cited 2023 Nov 20]; https://doi.org/10.1016/C2010-0-65192-3

  34. Sauro J. A practical guide to measuring usability. Should you use 5 or 7 point scales. Denver; 2010. 2–13 p.

  35. e Silva JL, de Sousa Mata M, Câmara SMA. do Céu Clara Costa Í, de Medeiros KS, Cobucci RN, Validity and reliability of the lederman Prenatal Self-Evaluation Questionnaire (PSEQ) in Brazil. BMC Pregnancy Childbirth. 2021;21(1):481 https://doi.org/10.1186/s12884-021-03959-3

  36. Firdaus MM. Metodologi Penelitian Kuantitatif: Dilengkapi Analisis Regresi IBM SPSS Statistics Version 26.0. Riau: CV. Dotplus; 2021.

    Google Scholar 

  37. Almeda JV, Capistrano TG, Sarte GM. Elementary statistics. Diliman Quezon City: University of the Philippines; 2010.

    Google Scholar 

  38. Spante M, Hashemi SS, Lundin M, Algers A. Digital competence and digital literacy in higher education research: systematic review of concept use. Cogent Educ. 2018;5(1):1519143. https://doi.org/10.1080/2331186X.2018.1519143

    Article  Google Scholar 

  39. Kominfo. Status Literasi Digital di Indonesia 2022 [Internet]. 2022 [cited 2023 Oct 19]. https://web.kominfo.go.id/sites/default/files/ReportSurveiStatusLiterasiDigitalIndonesia2022.pdf.

  40. Nurhayati-Wolff H. Digital literacy index in Indonesia from 2020 to 2022, by type [Internet]. 2023 [cited 2023 Oct 19]. https://www.statista.com/statistics/1337349/indonesia-digital-literacy-index-by-type/#statisticContainer.

  41. Harmoko DD. Digital literacy as a solution to improve the quality of Indonesia’s Human resources. Res Dev J Educ. 2021;7(2):413. https://doi.org/10.30998/rdje.v7i2.10569

    Article  Google Scholar 

  42. Alfian SD, Khoiry QA, Andhika A, Pratama M, Pradipta IS, Kristina SA, Zairina E et al. Knowledge, perception, and willingness to provide telepharmacy services among pharmacy students: a multicenter cross-sectional study in Indonesia. BMC Medical Education 2023;23(1):1–9. https://doi.org/10.1186/s12909-023-04790-4.

  43. Tjiptoatmadja NN, Alfian SD. Knowledge, Perception, and Willingness to Use Telepharmacy Among the General Population in Indonesia. Front Public Health. 2022;10. https://doi.org/10.3389/fpubh.2022.825554

  44. Wei Y, Zheng P, Deng H, Wang X, Li X, Fu H. Design features for improving Mobile Health intervention user Engagement: systematic review and thematic analysis. J Med Internet Res. 2020;22(12):e21687. https://doi.org/10.2196/21687

    Article  PubMed  PubMed Central  Google Scholar 

  45. Perski O, Blandford A, Ubhi HK, West R, Michie S. Smokers’ and drinkers’ choice of smartphone applications and expectations of engagement: a think aloud and interview study. BMC Med Inf Decis Mak. 2017;17(1):25. https://doi.org/10.1186/s12911-017-0422-8

    Article  Google Scholar 

  46. Lazard AJ, Pikowski J, Horrell L, Ross JC, Noar SM, Sutfin EL. Adolescents’ and young adults’ aesthetics and Functionality preferences for Online Tobacco Education. J Cancer Educ. 2020;35(2):373–9. https://doi.org/10.1007/s13187-019-1475-4

    Article  PubMed  Google Scholar 

  47. Ledel Solem IK, Varsi C, Eide H, Kristjansdottir OB, Mirkovic J, Børøsund E, et al. Patients’ needs and requirements for eHealth Pain Management interventions: qualitative study. J Med Internet Res. 2019;21(4):e13205. https://doi.org/10.2196/13205

    Article  PubMed  PubMed Central  Google Scholar 

  48. Milward J, Deluca P, Drummond C, Watson R, Dunne J, Kimergård A. Usability testing of the BRANCH Smartphone App designed to reduce Harmful drinking in young adults. JMIR Mhealth Uhealth. 2017;5(8):e109. https://doi.org/10.2196/mhealth.7836

    Article  PubMed  PubMed Central  Google Scholar 

  49. Rabin C, Bock B. Desired features of Smartphone Applications promoting physical activity. Telemedicine e-Health. 2011;17(10):801–3. https://doi.org/10.1089/tmj.2011.0055

    Article  PubMed  Google Scholar 

  50. Coyne I, Prizeman G, Sheehan A, Malone H, While AE. An e-health intervention to support the transition of young people with long-term illnesses to adult healthcare services: design and early use. Patient Educ Couns. 2016;99(9):1496–504. https://doi.org/10.1016/j.pec.2016.06.005

    Article  CAS  PubMed  Google Scholar 

  51. Peng W, Yuan S, Holtz BE. Exploring the Challenges and Opportunities of Health Mobile Apps for individuals with type 2 diabetes living in Rural communities. Telemedicine e-Health. 2016;22(9):733–8. https://doi.org/10.1089/tmj.2015.0180

    Article  PubMed  Google Scholar 

  52. Gkatzidou V, Hone K, Sutcliffe L, Gibbs J, Sadiq ST, Szczepura A, et al. User interface design for mobile-based sexual health interventions for young people: design recommendations from a qualitative study on an online Chlamydia clinical care pathway. BMC Med Inf Decis Mak. 2015;15(1):72. https://doi.org/10.1186/s12911-015-0197-8

    Article  Google Scholar 

  53. Nathalie Lyzwinski L, Caffery L, Bambling M, Edirippulige S. University Students’ perspectives on mindfulness and mHealth: a qualitative exploratory study. Am J Health Educ. 2018;49(6):341–53. https://doi.org/10.1080/19325037.2018.1502701

    Article  Google Scholar 

  54. Phillips SM, Courneya KS, Welch WA, Gavin KL, Cottrell A, Nielsen A, et al. Breast cancer survivors’ preferences for mHealth physical activity interventions: findings from a mixed methods study. J Cancer Surviv. 2019;13(2):292–305. https://doi.org/10.1007/s11764-019-00751-3

    Article  PubMed  PubMed Central  Google Scholar 

  55. Herbeć A, Perski O, Shahab L, West R. Smokers’ views on Personal Carbon Monoxide Monitors, Associated Apps, and their use: an interview and think-Aloud Study. Int J Environ Res Public Health. 2018;15(2):288. https://doi.org/10.3390/ijerph15020288

    Article  PubMed  PubMed Central  Google Scholar 

  56. Ludden GDS, Van Rompay TJL, Kelders SM, Van Gemert-Pijnen JEWC. How to increase reach and adherence of web-based interventions: A design research viewpoint. J Med Internet Res. 2015;17(7):e4201. https://doi.org/10.2196/jmir.4201

  57. Assael H. Consumers behavior. 6th ed. Ohio: Southwestern College Publishing.; 1995.

    Google Scholar 

  58. Dulkhatif HAT, Warso MM, Pengaruh Kualitas Pelayanan, Kepuasan Pelanggan Dan Lokasi, Terhadap Loyalitas Pelanggan Pada Penyedia Jasa Internet Study Pt Noken Mulia Tama Semarang. J Manag. 2016;2(2):1–34.

    Google Scholar 

  59. Partridge SR, McGeechan K, Hebden L, Balestracci K, Wong AT, Denney-Wilson E, et al. Effectiveness of a mHealth Lifestyle Program With Telephone support (TXT2BFiT) to prevent Unhealthy Weight Gain in Young adults: Randomized Controlled Trial. JMIR Mhealth Uhealth. 2015;3(2):e66. https://doi.org/10.2196/mhealth.4530

    Article  PubMed  PubMed Central  Google Scholar 

  60. Recio-Rodriguez J, Agudo Conde C, Calvo-Aponte M, Gonzalez-Viejo N, Fernandez-Alonso C, Mendizabal-Gallastegui N, et al. The effectiveness of a Smartphone application on modifying the intakes of Macro and micronutrients in Primary Care: a Randomized Controlled Trial. The EVIDENT II study. Nutrients. 2018;10(10):1473. https://doi.org/10.3390/nu10101473

    Article  PubMed  PubMed Central  Google Scholar 

  61. Nguyen Thanh V, Guignard R, Lancrenon S, Bertrand C, Delva C, Berlin I, et al. Effectiveness of a fully automated internet-based Smoking Cessation Program: a randomized controlled trial (STAMP). Nicotine Tob Res. 2019;21(2):163–72. https://doi.org/10.1093/ntr/nty016

    Article  PubMed  Google Scholar 

  62. Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al. The effectiveness of Mobile-Health Technology-Based Health Behaviour Change or Disease Management Interventions for Health Care consumers: a systematic review. PLoS Med. 2013;10(1):e1001362. https://doi.org/10.1371/journal.pmed.1001362

    Article  PubMed  PubMed Central  Google Scholar 

  63. Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, et al. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an International Consensus for the reporting of Behavior Change interventions. Ann Behav Med. 2013;46(1):81–95. https://doi.org/10.1007/s12160-013-9486-6

    Article  PubMed  Google Scholar 

  64. Schwarzer R. Modeling Health Behavior Change: how to predict and modify the Adoption and Maintenance of Health Behaviors. Appl Psychol. 2008;57(1):1–29. https://doi.org/10.1111/j.1464-0597.2007.00325.x

    Article  Google Scholar 

  65. Farha RA, Gharaibeh L, Alzoubi KH, Alhamad H. Exploring Community Pharmacists’ Perception and Readiness Toward Telepharmacy Implementation in Jordan: A Cross-Sectional Study. Telemed J E Health.. 2023; 30(3): 816-824 https://doi.org/10.1089/tmj.2023.0264

  66. Elawady A, Khalil A, Assaf O, Toure S, Cassidy C. Telemedicine during COVID-19: a survey of Health Care professionals’ perceptions. Monaldi Arch Chest Dis. 2020;90(4):576–81. https://doi.org/10.4081/monaldi.2020.1528

    Article  Google Scholar 

  67. Zuckerman AD, Whelchel K, Kozlicki M, Simonyan AR, Donovan JL, Gazda NP, et al. Health-system specialty pharmacy role and outcomes: a review of current literature. Am J Health-System Pharm. 2022;79(21):1906–18. https://doi.org/10.1093/ajhp/zxac212

    Article  Google Scholar 

  68. Grime J, Blenkinsopp A, Raynor DK, Pollock K, Knapp P. The role and value of written information for patients about individual medicines: a systematic review. Health Expect. 2007;10(3):286–98. https://doi.org/10.1111/j.1369-7625.2007.00454.x

    Article  PubMed  PubMed Central  Google Scholar 

  69. Barnett N, Jubray B. Remote consultations: how pharmacy teams can practise them successfully. Pharm J. 2020.

  70. Sholikhah R, Krisnawati M, Sudiyono. Effectiveness of the Use of Interactive Video Learning Media in Fashion Technology courses. Adv Social Sci Educ Humanit Res. 2019;379:172–6. https://doi.org/10.2991/assehr.k.191217.029

    Article  Google Scholar 

  71. Vakilian A, Ranjbar E, Hassanipour M, Ahmadinia H, Hasani H. The effectiveness of virtual interactive video in comparison with online classroom in the stroke topic of theoretical neurology in COVID-19 pandemic. J Educ Health Promot. 2022;11(1):219. https://doi.org/10.4103/jehp.jehp_1297_21

    Article  PubMed  PubMed Central  Google Scholar 

  72. Tractinsky N. Aesthetics and Apparent Usability: Empirically Assessing Cultural and Methodological Issues. Proceedings of the ACM SIGCHI Conference on Human factors in computing systems. https://doi.org/10.1145/258549.258626

  73. Crane D, Garnett C, Brown J, West R, Michie S. Factors influencing usability of a smartphone app to reduce excessive alcohol consumption: think aloud and interview studies. Front Public Health. 2017;5. https://doi.org/10.3389/fpubh.2017.00039

  74. Su MC, Chen WC, Liu CY, Jou HJ, Hsiao YC, Tsao LI. The design requirements for an E-Health Management platform: addressing the needs of adolescent girls at high risk of metabolic syndrome. Hu Li Za Zhi. 2015;62(5):51–60. https://doi.org/10.6224/jn62.5.51

    Article  PubMed  Google Scholar 

  75. Peters D, Deady M, Glozier N, Harvey S, Calvo RA. Worker Preferences for a Mental Health App within male-dominated industries: participatory study. JMIR Ment Health. 2018;5(2):e30. https://doi.org/10.2196/mental.8999

    Article  PubMed  PubMed Central  Google Scholar 

  76. Johnson CM, Johnson TR, Zhang J. A user-centered framework for redesigning health care interfaces. J Biomed Inform. 2005;38(1):75–87. https://doi.org/10.1016/j.jbi.2004.11.005

  77. Rezaee R, Asadi S, Yazdani A, Rezvani A, Kazeroon AM. Development, usability and quality evaluation of the resilient mobile application for women with breast cancer. Health Sci Rep. 2022;5(4). https://doi.org/10.1002/hsr2.708

  78. Padrini-Andrade L, Balda R, de Areco CX, Bandiera-Paiva KCN, Nunes P, Marba M, Evaluation Of Usability Of A Neonatal Health Information System According To The User’S Perception. STM,. Revista Paulista de Pediatria. 2019;37(1):90–6. https://doi.org/10.1590/1984-0462/;2019;37;1;00019

  79. Fitria N, Idrus L, Putri AR, Sari YO. The usability testing of the integrated electronic healthcare services for diabetes mellitus patients during the pandemic in Indonesia. Digit Health. 2023;9:205520762311732. https://doi.org/10.1177/20552076231173227

    Article  Google Scholar 

  80. Mkpojiogu EOC, Hashim NL, Adamu R. Observed Demographic Differentials in User Perceived Satisfaction on the Usability of Mobile Banking Applications. Knowledge Management International Conference (KMICe). 2016;263–8.

  81. Handayani PW, Indriani R, Pinem AA. Mobile health readiness factors: from the perspectives of mobile health users in Indonesia. Inf Med Unlocked. 2021;24:100590. https://doi.org/10.1016/j.imu.2021.100590

    Article  Google Scholar 

  82. Schillinger D. Association of Health Literacy with Diabetes Outcomes. JAMA. 2002;288(4):475. https://doi.org/10.1001/jama.288.4.475

    Article  PubMed  Google Scholar 

  83. Soemitro DH, Analisis Tingkat Health Literacy Dan Pengetahuan Pasien Hipertensi, Di Puskesmas Kabupaten Malang. Calyptra: Jurnal Ilmiah Mahasiswa Universitas Surabaya. 2014;3(1):1–13.

  84. Immanuel SS, Usability Testing Pada Aplikasi Klikdokter Mobile. Berdasarkan ISO 9241-11. Universitas Diponegoro; 2023.

  85. Horvath KJ, Ecklund AM, Hunt SL, Nelson TF, Toomey TL. Developing Internet-based health interventions: a guide for public health researchers and practitioners. J Med Internet Res. 2015;17(1):e28. https://doi.org/10.2196/jmir.3770

  86. Yap YY, Tan SH, Choon SW. Elderly’s intention to use technologies: A systematic literature review. Heliyon. 2022;8(1):e08765 https://doi.org/10.1055/s-0040-1714693

  87. Rochmawati E, Kamilah F, Iskandar AC. Acceptance of e-health technology among older people: A qualitative study. Nurs Health Sci. 2022;24(2):437–46. https://doi.org/10.1111/nhs.12939

  88. Richter Lagha R, Burningham Z, Sauer BC, Leng J, Peters C, Huynh T et al. Usability Testing a Potentially Inappropriate Medication Dashboard: A Core Component of the Dashboard Development Process. Appl Clin Inform. 2020;11(4):528–34. https://doi.org/10.1055/s-0040-1714693

  89. Cho H, Powell D, Pichon A, Kuhns LM, Garofalo R, Schnall R. Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation. Int J Med Inf. 2019;129:366–73. https://doi.org/10.1016/j.ijmedinf.2019.07.010

    Article  Google Scholar 

  90. Wahyuningrum T, Kartiko C, Wardhana AC. Exploring e-Commerce Usability by Heuristic Evaluation as a Compelement of System Usability Scale. In: 2020 International Conference on Advancement in Data Science, E-learning and Information Systems (ICADEIS). 2020. pp. 1–5.

Download references

Acknowledgements

The authors are grateful to all respondents for their efforts and contributions.

Funding

Open access funding provided by University of Padjadjaran. Financial assistance was received through a grant-in-aid from The Ministry of Education, Culture, Research, and Technology of Indonesia. This funding body did not play any role in the design, writing, and publication of this study.

Open access funding provided by University of Padjadjaran

Author information

Authors and Affiliations

  1. Department of Pharmacology and Clinical Pharmacy, Faculty of Pharmacy, Universitas Padjadjaran, Jatinangor, Indonesia

    Sofa D. Alfian, Jihan A. Sania, Dzulfah Q. Aini, Qisty A. Khoiry, Yudisia Ausi, Neily Zakiyah, Irma M. Puspitasari, Auliya A. Suwantika & Rizky Abdulah

  2. Drug Utilization and Pharmacoepidemiology Research Group, Center of Excellence for Pharmaceutical Care Innovation, Universitas Padjadjaran, Jatinangor, Indonesia

    Sofa D. Alfian, Meliana Griselda, Yudisia Ausi, Neily Zakiyah, Irma M. Puspitasari, Auliya A. Suwantika & Rizky Abdulah

  3. Center for Health Technology Assessment, Universitas Padjadjaran, Jatinangor, Indonesia

    Sofa D. Alfian & Auliya A. Suwantika

  4. Dienggo Kreasi Nusantara Company, Jakarta, Indonesia

    Mariska Mahfud & Saktian Aji

  5. Department of Nursing, School of Health Sciences, Cyprus University of Technology, Limassol, Cyprus

    Angelos P. Kassianos

Authors
  1. Sofa D. Alfian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jihan A. Sania
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dzulfah Q. Aini
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Qisty A. Khoiry
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Meliana Griselda
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yudisia Ausi
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Neily Zakiyah
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Irma M. Puspitasari
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Auliya A. Suwantika
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Mariska Mahfud
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Saktian Aji
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Rizky Abdulah
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Angelos P. Kassianos
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SDA, QAK, YA, NZ, IMP, AAS, MM, SA, and RA were involved in protocol development; SDA and QAK were responsible for gaining ethical approval; SDA, QAK, JAS, DQA, MG, and APK were involved in patient recruitment and data analysis. SDA wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version.

Corresponding author

Correspondence to Sofa D. Alfian.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Research Ethics Commission of Universitas Padjadjaran, Indonesia (number 670/UN6.KEP/EC/2022) and all participants provided written informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Alfian, S.D., Sania, J.A., Aini, D.Q. et al. Evaluation of usability and user feedback to guide telepharmacy application development in Indonesia: a mixed-methods study. BMC Med Inform Decis Mak 24, 130 (2024). https://doi.org/10.1186/s12911-024-02494-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12911-024-02494-3

Keywords

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us