Skip to main content
Download PDF

The most used questionnaires for evaluating telemedicine services

BMC Medical Informatics and Decision Making volume 21, Article number: 36 (2021) Cite this article

Abstract

Background

Questionnaires are commonly used tools in telemedicine services that can help to evaluate different aspects. Selecting the ideal questionnaire for this purpose may be challenging for researchers. This study aims to review which questionnaires are used to evaluate telemedicine services in the studies, which are most common, and what aspects of telemedicine evaluation do they capture.

Methods

The PubMed database was searched in August 2020 to retrieve articles. Data extracted from the final list of articles included author/year of publication, journal of publication, type of evaluation, and evaluation questionnaire. Data were analyzed using descriptive statistics.

Results

Fifty-three articles were included in this study. The questionnaire was used for evaluating the satisfaction (49%), usability (34%), acceptance (11.5%), and implementation (2%) of telemedicine services. Among telemedicine specific questionnaires, Telehealth Usability Questionnaire (TUQ) (19%), Telemedicine Satisfaction Questionnaire (TSQ) (13%), and Service User Technology Acceptability Questionnaire (SUTAQ) (5.5%), were respectively most frequently used in the collected articles. Other most used questionnaires generally used for evaluating the users’ satisfaction, usability, and acceptance of technology were Client Satisfaction Questionnaire (CSQ) (5.5%), Questionnaire for User Interaction Satisfaction (QUIS) (5.5%), System Usability Scale (SUS) (5.5%), Patient Satisfaction Questionnaire (PSQ) (5.5%), and Technology Acceptance Model (TAM) (3.5%) respectively.

Conclusion

Employing specifically designed questionnaires or designing a new questionnaire with fewer questions and more comprehensiveness in terms of the issues studied provides a better evaluation. Attention to user needs, end-user acceptance, and implementation processes, along with users' satisfaction and usability evaluation, may optimize telemedicine efforts in the future.

Peer Review reports

Background

Telemedicine provides healthcare services when patients and healthcare providers are at different locations using Information and Communication Technologies (ICT). The core purpose of telemedicine is to improve the health of individuals and communities by exchanging useful information for various goals, such as preventing diseases, for diagnosis, monitoring, providing treatment, educating healthcare providers, and for conducting research [1, 2].

It is essential to identify the limitations, find approaches to overcome them, and create a reasonable structure to implement and use a telemedicine service successfully. Therefore, precise evaluations of telemedicine services are required [3]. Evaluation is the use of systematic and logical methods to assess the attributes of different aspects of the project, including its design, implementation, operation, and outcome [4]. The telemedicine evaluation process should be done independently and systematically on various features as the feasibility of the project, acceptance by participants, availability of service, technical capabilities of participants, clinical outcomes, user satisfaction, quality, and the cost–benefit of the offered service [3].

There are several methods to evaluate different aspects of a telemedicine project. In order to choose the right evaluation method, several issues should be considered: (1) Considering project goals; (2) determining the required amount of budget, energy, and time for evaluation; (3) selecting the appropriate method based on the predetermined criteria and metrics; (4) using understandable and easy evaluation methods for users; (5) being completely aware of the evaluation method used in the study; and (6) using validated methods for the evaluation process [5].

In the field of telemedicine, the most commonly used tools for evaluating user satisfaction are questionnaires and interviews [6]; and in order to assess the usability of telemedicine systems, usually questionnaires, interviews, observations, self-descriptive, and logging methods are used [7]. The clinical outcomes of telemedicine services are evaluated by means of biometric measurements, quality of life, and disease-specific tools that are all generally questionnaires [8].

Researchers use a variety of questionnaires for various purposes. Some of them are valid questionnaires specified for telemedicine whereas some others are more general. Due to the great variety of questionnaires, it is difficult for researchers to choose the best tool to evaluate their telemedicine services. Therefore, this review aims at listing the most commonly used questionnaires for evaluating telemedicine, which would in turn help researchers to select the most appropriate questionnaire, based on their objectives, to evaluate telemedicine services.

Methods

Database and date

This is a scoping review. PubMed database was searched in August 2020 to retrieve articles without date limitation. Approximately, 80–90% of studies conducted in Telemedicine were accessible on PubMed, which sufficed for our purpose; therefore, no other database was used [9].

Search strategy

The following combinations of keywords were used to find relevant articles in the Title/Abstract search field: (telemedicine) AND (evaluation OR assessment) AND (questionnaire).

Inclusion criteria

Original observational and interventional research articles in which a valid and referenced questionnaire was used to evaluate telemedicine services and systems were included in this study.

Exclusion criteria

Articles were excluded if they were review articles, non-English language, without full text, not specifically addressing telemedicine, did not have communication between patients and healthcare providers in mHealth, evaluated clinical outcomes, evaluated acceptance of telemedicine prior to implementation, did not include details about the questionnaires used, questionnaires that have not been validated, or combined questionnaires.

Article selection

First, all retrieved articles were assessed based on title and abstract by one researcher (SH). Next, the same researcher reviewed the full-text of the selected articles. When necessary, the second researcher (K.B), being a telemedicine professional, provided consult. Manual searching was also performed to find additional articles that had used specific evaluation questionnaires to investigate Telemedicine. Finally, a list of all included articles was prepared.

Data extraction

The following information was extracted from included articles: author, year of publication, journal of publication, evaluation type, and evaluation questionnaire.

Data analysis

Descriptive statistics (frequency and frequency percent) was used to analyze data.

Results

PubMed database found 214 articles. After removing review articles and those written in a language other than English, 208 articles remained whose titles and abstracts were screened. After excluding 95 articles, the full-text of the 113 remaining articles were reviewed. Finally, 53 articles were included in this study (Fig. 1), from which the required data was extracted (Table 1).

Fig. 1
figure 1

The process of PRISMA in finding and including articles

Full size image
Table 1 Information of included articles
Full size table

Journal of publication

Nine articles were published in the Telemedicine and e-health Journal (17%), six in the Journal of Medical Internet Research (11.5%), four in the International Journal of Telerehabilitation (7.5%), four in the Journal of Telemedicine and Telecare (7.5%), and the remaining articles were published in other journals (n = 30, 56.5%).

Evaluation type

A questionnaire was used in 26 articles (49%) to evaluate patients’ or physicians’ satisfaction; in 18 articles to evaluate system usability (34%), in 6 articles to evaluate acceptance (11.5%), and in one article to evaluate the implementation process (2%). In 2 articles a questionnaire was used to evaluate the usability and the acceptance or satisfaction of individuals (3.5%).

Evaluation questionnaire

The final list of articles showed that the Telehealth Usability Questionnaire (TUQ) (n = 10, 19%) [10,11,12,13,14,15,16,17,18,19] and Telemedicine Satisfaction Questionnaire (TSQ) (n = 7, 13%) [20,21,22,23,24,25,26] were the most commonly used. Then, the Service User Technology Acceptability Questionnaire (SUTAQ) (n = 3, 5.5%) [27,28,29], the Client Satisfaction Questionnaire (CSQ) (n = 3, 5.5%) [30,31,32], the Questionnaire for User Interaction Satisfaction (QUIS) (n = 3, 5.5%) [33,34,35], the System Usability Scale (SUS) questionnaire (n = 3, 5.5%) [36,37,38], the Patient Satisfaction Questionnaire (PSQ) (n = 3, 5.5%) [22, 39, 40], the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) (n = 2, 3.5%) [41, 42], the Patient Assessment of Communication during Telemedicine (PACT) questionnaire (n = 2, 3.5%) [43, 44], the Post Study System Usability Questionnaire (PSSUQ) (n = 2, 3.5%) [19, 45] and the Technology Acceptance Model (TAM) (n = 2, 3.5%) [46, 47] were the most used questionnaires, respectively. The rest of the articles had used other questionnaires (n = 18, 34%) [14, 19, 46, 48,49,50,51,52,53,54,55,56,57,58,59,60,61,62] and 4 articles had used more than one questionnaire.

The most used questionnaires

Out of 59 used questionnaires in the studies, the most frequently (more than two times) used were TUQ, TSQ, SUTAQ, CSQ, QUIS, SUS, PSQ, TSUQ, PACT, PSSUQ, and TAM, which have been shown in Fig. 2.

Fig. 2
figure 2

The frequency of the most used questionnaires for evaluating telemedicine services

Full size image

Some information about these questionnaires are presented below:

Parmanto et al. in 2016, formally introduced TUQ. This questionnaire evaluates the usability of telehealth services. It has 21 items that are based on 6 criteria including usefulness (3 items), ease of use and learnability (3 items), interface quality (4 items), interaction quality (4 items), reliability (3 items), and satisfaction and future use (4 items) [18]. The TUQ was designed using existing telehealth and general usability questionnaires and includes all usability aspects.

The TSQ was developed by Yip et al. in 2003 for evaluating patient satisfaction in using telemedicine. This questionnaire has 14 items with no categories [26].

Hirani et al. presented SUTAQ in 2017, as a tool to judge the acceptability of telehealth services by investigating user opinions. This questionnaire consists of 22 items that is categorized in six sectors: benefits (9 items), privacy (4 items), personal care skill (3 items), substitution (3 items), and satisfaction (3 items) [29].

Bakken et al. in 2006, designed TSUQ. This questionnaire evaluates the satisfaction and usefulness criteria of video consultation and telephone monitoring. This instrument has 26 items in two satisfaction (21 items) and usefulness (5 items) sections [42].

PACT questionnaire was introduced by Agha et al. in 2009. This questionnaire measures the satisfaction of patients and physicians of their communication by asking 33 questions. These questions are asked about the quality of physician and patient communication (16 questions), clinical skills (9 questions), interpersonal skills (6 questions), and comfortable of visit (2 questions) sectors [44].

Attkisson et al. in 1982, firstly introduced CSQ. This questionnaire evaluates the general users’ satisfaction with different services. The original version of CSQ has 18 items; however, the questionnaire has other versions in which there are only 3 or 8 items [65].

QUIS was developed by Chin et al. in 1988. It is a usability testing instrument that has been designed to measure the satisfaction of users about their interaction with the computer interface. QUIS consists of 27 items in five sections. The first section measures the overall satisfaction, and the four others measure user satisfaction regarding screen, terminology and information, learning, and system capabilities aspects [69].

The SUS questionnaire was created by Brooke et al. in 1986 for testing the usability of electronic systems. This tool has ten items with no specific categories [78].

The PSQ was firstly designed by Ware et al. in 1983 for measuring the satisfaction of patients with using medical cares. The original version of this questionnaire has 80 items measured in seven categories, such as overall satisfaction, technical quality, interpersonal mode, communication, financial features, time spent with physician, availability, and convenience [84]. A shorter version of this questionnaire also exists which consists of 18 items [67].

PSSUQ was developed by the International Business Machines Corporation (IBM) to evaluate users’ satisfaction at the end of the study. This questionnaire has three versions. The last version consists of 16 questions with seven Likert scales. These questions are arranged is such way to evaluate matters in the three following sections: system usefulness, information quality, and interface quality [75, 83].

TAM is a model that shows how users use and accept a technology. A questionnaire that has been developed based on this model was presented by Davis in 1989. This questionnaire investigates 12 items that are generally concerned with two matters: the perceived usefulness and perceived ease of use sections with 12 items [77].

Discussion

The TUQ, TSQ, SUTAQ were the three most common telemedicine-specific questionnaires used in the retrieved articles, respectively. The four most used general questionnaires related to satisfaction and usability evaluation were the CSQ, QUIS, SUS, and PSQ.

In the below sections it will be discussed why each questionnaire is used more frequently. To do this we chose the most three used questionnaires which are designed specific for telemedicine and four general questionnaires that are used in telemedicine evaluation as well.

TUQ is usable for evaluating different types of telehealth systems such as sole target videoconferencing systems, computer and mobile-based systems, and collecting the opinions of both patients and physicians [18]. As TUQ is considerably comprehensive when comparing with other questionnaires such as QUIS and SUS, it is most frequently used for evaluating the usability of telemedicine systems. SUS questionnaire, like QUIS, is a general usability evaluation tool that is also used to evaluate telemedicine systems; yet, unlike QUIS it has no segmentation and it examines fewer items [78]. Also, ever since TUQ has been introduced, SUS has been seldom used.

The TSQ is a preliminary tool for measuring patient satisfaction with telemedicine, and it is used quite frequently. This might be due to the fact that TSQ covers various satisfaction factors such as the quality of care, quality of virtual visits, interpersonal interactions, and also it has fewer number of items [26].

Although TSUQ was also introduced many years ago, it has rarely been used for studies conducted in telemedicine, which may be due to two reasons: (1) it has been designed specifically for telemedicine services provided to diabetes patients, and (2) it investigates more items than other questionnaires especially, TSQ [42].

The CSQ was a generic most used questionnaire for evaluating users’ satisfaction with the telemedicine services. This may be due to the fact that CSQ measures the quality of diverse attributes such as the physical environment, procedure, assistance staff, type of service, treatment staff, amount or length of service, service quality, outcome, and general satisfaction with few items [65]. PSQ is a general questionnaire as well and it is designed to evaluate patient satisfaction. However, with presenting the TSQ, evaluators preferred to use TSQ as a specific questionnaire in assessing the users’ satisfaction of a telemedicine service. Evaluators may use different tools depending on their purpose of investigation, and the only thing that matters is the validity of the used tool.

The SUTAQ is the only questionnaire that specifically designed to gather patients’ opinions about the acceptability of the telehealth systems [29]. While there are various models for assessing technology acceptance, this instrument is used more in this field since it has been specifically designed for evaluating telehealth acceptance.

In terms of evaluation types for telemedicine services, users’ satisfaction, usability, acceptance, and implementation process are non-clinical aspects that have been evaluated by use of validated questionnaires. Approximately, a questionnaire had been used on half of the collected articles for evaluating the satisfaction of telemedicine users; and approximately in a third of the articles, the usability of telemedicine systems was evaluated.

Telemedicine acceptance [27,28,29, 46, 47, 49, 62] and its implementation process [50] were evaluated in only a few articles. A review study investigating the evaluation methods for telemedicine services in hospitals showed that telemedicine users' satisfaction more frequently evaluated than clinical and economical aspects, and the most commonly used method to evaluate satisfaction was questionnaire. Similar to our result, this study also showed that the development and implementation process of telemedicine had gained less attention [6].

Evaluating the implementation process could be an essential stage for the successful usage of telemedicine services due to showing obstacles and facilitators of implementation [50]. Also, considering the needs of the users and planning the process based on these needs may affect the successful implementation of telemedicine services and increase the rate of their acceptance. Therefore, we recommend that researchers pay more attention to this aspect in the evaluation of telemedicine services. Another study that reviewed the usability evaluation methods of eHealth services for patients who had HIV revealed that questionnaire was the most employed method for evaluation [89].

Based on the researchers’ knowledge, this is the first review study that has identified validated and the most used questionnaires in evaluating telemedicine services. Nevertheless, this study has its own limitations. Only PubMed database was used for searching and retrieving articles. In the search strategy, we used telemedicine as a MeSH term and evaluation and assessment keywords that may use in different grammatical style. Moreover, we restricted our search to the Title/Abstract field. These issues may cause some articles missed from our study. For this reason, we conducted a manual search using the name of telemedicine specific questionnaires in the PubMed database and added additional articles.

Conclusion

Many questionnaires were used to assess telemedicine services. Some of them were specifically designed to evaluate telemedicine services while others were more general. As the results of this study showed, telemedicine service evaluators should use questionnaires specifically designed for telemedicine to assess its various aspects. However, if an evaluator wants to design a questionnaire for evaluating a telemedicine service, it is better to pay attention to goal-based design, the number of questions, and comprehensiveness in terms of the issues studied.

Users' satisfaction with telemedicine services and the usability of the system have been two of the most frequently investigated issues in telemedicine when comparing with other existing issues in the field. Attention to user needs, end-user acceptance, and implementation processes may optimize telemedicine efforts in the future.

Availability of data and materials

Not applicable.

Abbreviations

ICT:

Information and communication technologies

TUQ:

Telehealth usability questionnaire

TeSS:

Telehealth satisfaction scale

USE:

Usefulness, satisfaction, and ease of use

TSQ:

Telemedicine satisfaction questionnaire

CSQ:

Client satisfaction questionnaire

FPAS:

Florida patient acceptance survey

TSUQ:

Telemedicine satisfaction and usefulness questionnaire

PSQ:

Patient satisfaction questionnaire

QUIS:

Questionnaire for user interaction satisfaction

PACT:

Patient assessment of communication during telemedicine

D-QUEST:

Dutch version of Quebec user evaluation of satisfaction with assistive technology

SUTAQ:

Service user technology acceptability questionnaire

WEQ:

Web evaluation questionnaire

CSUQ:

Computer system usability questionnaire

TAM:

Technology acceptance model

SUS:

System usability scale

Health-ITUES:

Health-information technology usability survey

SADL:

Satisfaction with amplification in daily life

PSSUQ:

Post study system usability questionnaire

ASQ:

The after-scenario questionnaire

TMPQ:

Telemedicine perception questionnaire

References

  1. WHO. Telemedicine, Telehealth, and Health Information Technology, 2006 [cited 2020 August 14]. https://www.who.int/goe/policies/countries/usa_support_tele.pdf?ua=1.

  2. WHO. Telemedicine 2010 [cited 2020 August 14]. https://www.who.int/goe/publications/goe_telemedicine_2010.pdf.

  3. Pan American Health Organization. Defining evaluation indicators for telemedicine as a tool for reducing health inequities: Study and results of a community of practice 2016 [cited 2020 August 14]. https://iris.paho.org/handle/10665.2/28562.

  4. Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Informat. 2004;73(6):479–91.

    Article  Google Scholar 

  5. McNair JB. Handbook of evaluation methods for health informatics. Amsterdam: Elsevier; 2006.

    Google Scholar 

  6. AlDossary S, Martin-Khan MG, Bradford NK, Smith ACJI. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Informat. 2017;97:171–94.

    Article  Google Scholar 

  7. Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems—a literature survey. Int J Med Inf. 2016;93:57–69.

    Article  CAS  Google Scholar 

  8. Kidholm K, Clemensen J, Caffery LJ, Smith AC. The Model for Assessment of Telemedicine (MAST): a scoping review of empirical studies. J Telemed Telecare. 2017;23(9):803–13.

    Article  PubMed  Google Scholar 

  9. Bahaadinbeigy K, Yogesan K, Wootton R. MEDLINE versus EMBASE and CINAHL for telemedicine searches. Telemed J E-health. 2010;16(8):916–9.

    Article  PubMed  Google Scholar 

  10. Layfield E, Triantafillou V, Prasad A, Deng J, Shanti RM, Newman JG et al. Telemedicine for head and neck ambulatory visits during COVID‐19: evaluating usability and patient satisfaction. Head Neck. 2020:1–9.

  11. Mostafa PI, Hegazy AA. Dermatological consultations in the COVID-19 era: Is teledermatology the key to social distancing? An Egyptian experience. J Dermatol Treat. 2020:1–6.

  12. Whitehouse CR, Long JA, Maloney LM, Daniels K, Horowitz DA, Bowles KH. Feasibility of diabetes self-management telehealth education for older adults during transitions in care. Res Gerontol Nurs. 2020;13(3):138–45.

    Article  PubMed  Google Scholar 

  13. Vaughan EM, Naik AD, Lewis CM, Foreyt JP, Samson SL, Hyman DJ. Telemedicine training and support for community health workers: improving knowledge of diabetes. Telemed J E-health. 2020;26(2):244–50.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Cheng O, Law N-H, Tulk J, Hunter M. Utilization of telemedicine in addressing musculoskeletal care gap in long-term care patients. JAAOS Global Res Rev. 2020;4(4).

  15. Serwe KM. The provider’s experience of delivering an education-based wellness program via telehealth. Int J Telerehabil. 2018;10(2):73.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Serwe KM, Hersch GI, Pancheri KJ. Feasibility of using telehealth to deliver the “Powerful Tools for Caregivers” program. Int J Telerehabil. 2017;9(1):15.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Yu DX, Parmanto B, Dicianno BE, Watzlaf VJ, Seelman KD. Accessibility needs and challenges of a mHealth system for patients with dexterity impairments. Disabil Rehabil Assist Technol. 2017;12(1):56–64.

    Article  PubMed  Google Scholar 

  18. Parmanto B, Lewis AN Jr, Graham KM, Bertolet MH. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil. 2016;8(1):3.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Schutte J, Gales S, Filippone A, Saptono A, Parmanto B, McCue M. Evaluation of a telerehabilitation system for community-based rehabilitation. Int J Telerehabil. 2012;4(1):15.

    PubMed  PubMed Central  Google Scholar 

  20. Lin CH, Tseng WP, Wu JL, Tay J, Cheng MT, Ong HN, et al. A double triage and telemedicine protocol to optimize infection control in an emergency department in Taiwan during the COVID-19 pandemic: retrospective feasibility study. J Med Internet Res. 2020;22(6):e20586.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Talal AH, McLeod A, Andrews P, Nieves-McGrath H, Chen Y, Reynolds A, et al. Patient reaction to telemedicine for clinical management of hepatitis C virus integrated into an opioid treatment program. Telemed J E-Health. 2019;25(9):791–801.

    Article  PubMed  Google Scholar 

  22. Le LB, Rahal HK, Viramontes MR, Meneses KG, Dong TS, Saab S. Patient satisfaction and healthcare utilization using telemedicine in liver transplant recipients. Dig Dis Sci. 2019;64(5):1150–7.

    Article  PubMed  Google Scholar 

  23. Losiouk E, Lanzola G, Del Favero S, Boscari F, Messori M, Rabbone I, et al. Parental evaluation of a telemonitoring service for children with type 1 diabetes. J Telemed Telecare. 2018;24(3):230–7.

    Article  CAS  PubMed  Google Scholar 

  24. Segura-Sampedro JJ, Rivero-Belenchón I, Pino-Díaz V, Rodríguez Sánchez MC, Pareja-Ciuró F, Padillo-Ruiz J, et al. Feasibility and safety of surgical wound remote follow-up by smart phone in appendectomy: a pilot study. Ann Med Surgery. 2012;2017(21):58–62.

    Google Scholar 

  25. Dechêne L, Tousignant M, Boissy P, Macoir J, Héroux S, Hamel M, et al. Simulated in-home teletreatment for anomia. Int J Telerehabil. 2011;3(2):3.

    Article  PubMed  PubMed Central  Google Scholar 

  26. Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the telemedicine satisfaction questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare. 2003;9(1):46–50.

    Article  CAS  PubMed  Google Scholar 

  27. Torbjørnsen A, Småstuen MC, Jenum AK, Årsand E, Ribu L. The service user technology acceptability questionnaire: psychometric evaluation of the norwegian version. JMIR Human Factors. 2018;5(4):e10255.

    Article  PubMed  PubMed Central  Google Scholar 

  28. Dario C, Luisotto E, Dal Pozzo E, Mancin S, Aletras V, Newman S, et al. Assessment of patients’ perception of telemedicine services using the service user technology acceptability questionnaire. Int J Integrated Care. 2016;16(2):13.

    Article  Google Scholar 

  29. Hirani SP, Rixon L, Beynon M, Cartwright M, Cleanthous S, Selva A, et al. Quantifying beliefs regarding telehealth: development of the whole systems demonstrator service user technology acceptability questionnaire. J Telemed Telecare. 2017;23(4):460–9.

    Article  PubMed  Google Scholar 

  30. Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Videoconferences and treatment of depression: satisfaction score correlated with number of sessions attended but not with age. Telemed J E-Health. 2020;26(7):898–904.

    Article  PubMed  Google Scholar 

  31. Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Disagreement in satisfaction between patients and providers in the use of videoconferences by depressed adults. Telemed J E-health. 2020;26(5):614–20.

    Article  PubMed  Google Scholar 

  32. Fields BG, Behari PP, McCloskey S, True G, Richardson D, Thomasson A, et al. Remote ambulatory management of veterans with obstructive sleep apnea. Sleep. 2016;39(3):501–9.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Safdari R, Langarizadeh M, Ramezani A, Khodaveisi T, Nejad AF. Development of a store-and-forward telescreening system of diabetic retinopathy: lessons learned from Iran. J Diabetes Metabolic Disorders. 2018;17(1):31–6.

    Article  Google Scholar 

  34. Hosseini F, Ayatollahi H, Salehi SH, Jafar K. Teleburn: designing a telemedicine application to improve burn treatment. Open Med Informat J. 2018;12:33–41.

    Article  Google Scholar 

  35. Alanzi T, Istepanian R, Philip N. Design and usability evaluation of social mobile diabetes management system in the Gulf Region. JMIR Res Protocols. 2016;5(3):e93.

    Article  Google Scholar 

  36. Smaradottir B, Gerdes M, Martinez S, Fensli R. The EU-project United4Health: user-centred design of an information system for a Norwegian telemedicine service. J Telemed Telecare. 2016;22(7):422–9.

    Article  PubMed  Google Scholar 

  37. Ligons FM, Mello-Thoms C, Handler SM, Romagnoli KM, Hochheiser H. Assessing the impact of cognitive impairment on the usability of an electronic medication delivery unit in an assisted living population. Int J Med Informat. 2014;83(11):841–8.

    Article  Google Scholar 

  38. Lacerda TC, von Wangenheim CG, von Wangenheim A, Giuliano I. Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context. J Biomed Inform. 2014;52:222–30.

    Article  PubMed  Google Scholar 

  39. McFarland LV, Raugi GJ, Reiber GE. Primary care provider and imaging technician satisfaction with a teledermatology project in rural Veterans Health Administration clinics. Telemed J E-health. 2013;19(11):815–25.

    Article  PubMed  Google Scholar 

  40. Wallace P, Haines A, Harrison R, Barber JA, Thompson S, Roberts J, et al. Design and performance of a multi-centre randomised controlled trial and economic evaluation of joint tele-consultations [ISRCTN54264250]. BMC Family Practice. 2002;3:1.

    Article  PubMed  PubMed Central  Google Scholar 

  41. McGloin H, O’Connell D, Glacken M, Mc Sharry P, Healy D, Winters-O’Donnell L, et al. Patient empowerment using electronic telemonitoring with telephone support in the transition to insulin therapy in adults with type 2 diabetes: observational, pre-post, mixed methods study. J Med Internet Res. 2020;22(5):e16161.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Bakken S, Grullon-Figueroa L, Izquierdo R, Lee N-J, Morin P, Palmas W, et al. Development, validation, and use of english and spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006;13(6):660.

    Article  PubMed  PubMed Central  Google Scholar 

  43. Hatton J, Chandra R, Lucius D, Ciuchta E. Patient satisfaction of pharmacist-provided care via clinical video teleconferencing. J Pharm Pract. 2018;31(5):429–33.

    Article  PubMed  Google Scholar 

  44. Agha Z, Schapira RM, Laud PW, McNutt G, Roter DL. Patient satisfaction with physician-patient communication during telemedicine. Telemed J E-health. 2009;15(9):830–9.

    Article  PubMed  Google Scholar 

  45. Kwon S, Kim J, Kang S, Lee Y, Baek H, Park K. CardioGuard: a brassiere-based reliable ECG monitoring sensor system for supporting daily smartphone healthcare applications. Telemed J E-Health. 2014;20(12):1093–102.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Agnisarman SO, Chalil Madathil K, Smith K, Ashok A, Welch B, McElligott JT. Lessons learned from the usability assessment of home-based telemedicine systems. Appl Ergonom. 2017;58:424–34.

    Article  Google Scholar 

  47. Parra C, Jódar-Sánchez F, Jiménez-Hernández MD, Vigil E, Palomino-García A, Moniche-Álvarez F, et al. Development, implementation, and evaluation of a telemedicine service for the treatment of acute stroke patients: TeleStroke. Interact J Med Res. 2012;1(2):e15.

    Article  PubMed  PubMed Central  Google Scholar 

  48. Mehra S, van den Helder J, Visser B, Engelbert RH, Weijs PJ, Kröse BJ. Evaluation of a blended physical activity intervention for older adults: mixed methods study. J Med Internet Res. 2020;22(7):e16380.

    Article  PubMed  PubMed Central  Google Scholar 

  49. Leppert F, Siebermair J, Wesemann U, Martens E, Sattler SM, Scholz S, et al. The INFluence of remote monitoring on anxiety/depression, quality of life, and device acceptance in ICD patients: a prospective, randomized, controlled, single-center trial. Clin Res Cardiol. 2020.

  50. van der Meij E, Huirne JA, Ten Cate AD, Stockmann HB, Scholten PC, Davids PH, et al. A perioperative ehealth program to enhance postoperative recovery after abdominal surgery: process evaluation of a randomized controlled trial. J Med Internet Res. 2018;20(1):e1.

    Article  PubMed  PubMed Central  Google Scholar 

  51. Host BK, Turner AW, Muir J. Real-time teleophthalmology video consultation: an analysis of patient satisfaction in rural Western Australia. Clin Exper Optometry. 2018;101(1):129–34.

    Article  Google Scholar 

  52. D’Hooghe M, Van Gassen G, Kos D, Bouquiaux O, Cambron M, Decoo D, et al. Improving fatigue in multiple sclerosis by smartphone-supported energy management: the MS TeleCoach feasibility study. Multiple Sclerosis Related Disorders. 2018;22:90–6.

    Article  PubMed  Google Scholar 

  53. Ammenwerth E, Modre-Osprian R, Fetz B, Gstrein S, Krestan S, Dörler J, et al. HerzMobil, an integrated and collaborative telemonitoring-based disease management program for patients with heart failure: a feasibility study paving the way to routine care. JMIR Cardio. 2018;2(1):e11.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Oliveira AC, Mattos S, Coimbra M. Development and assessment of an e-learning course on pediatric cardiology basics. JMIR Med Educ. 2017;3(1):e10.

    Article  PubMed  PubMed Central  Google Scholar 

  55. Roberts S, Spain B, Hicks C, London J, Tay S. Telemedicine in the Northern Territory: an assessment of patient perceptions in the preoperative anaesthetic clinic. Aust J Rural Health. 2015;23(3):136–41.

    Article  PubMed  Google Scholar 

  56. Poulsen KA, Millen CM, Lakshman UI, Buttner PG, Roberts LJ. Satisfaction with rural rheumatology telemedicine service. Int J Rheumatic Diseases. 2015;18(3):304–14.

    Article  Google Scholar 

  57. Vélez O, Okyere PB, Kanter AS, Bakken S. A usability study of a mobile health application for rural Ghanaian midwives. J Midwifery Women’s Health. 2014;59(2):184–91.

    Article  Google Scholar 

  58. Penteado SP, Bento RF, Battistella LR, Silva SM, Sooful P. Use of the satisfaction with amplification in daily life questionnaire to assess patient satisfaction following remote hearing aid adjustments (telefitting). JMIR Med Informat. 2014;2(2):e18.

    Article  Google Scholar 

  59. Scalvini S, Tridico C, Glisenti F, Giordano A, Pirini S, Peduzzi P, et al. The SUMMA project: a feasibility study on telemedicine in selected Italian areas. Telemed J E-health. 2009;15(3):261–9.

    Article  PubMed  Google Scholar 

  60. Kim HM, Lowery JC, Hamill JB, Wilkins EG. Patient attitudes toward a web-based system for monitoring chronic wounds. Telemed J E-Health. 2004;10(Suppl 2):S26-34.

    Article  Google Scholar 

  61. Eminovic N, Wyatt JC, Tarpey AM, Murray G, Ingrams GJ. First evaluation of the NHS direct online clinical enquiry service: a nurse-led web chat triage service for the public. J Med Internet Res. 2004;6(2):e17.

    Article  PubMed  PubMed Central  Google Scholar 

  62. van Rosmalen-Nooijens K, Lo Fo Wong S, Prins J, Lagro-Janssen T. Young people, adult worries: randomized controlled trial and feasibility study of the internet-based self-support method “feel the vibe” for adolescents and young adults exposed to family violence. J Med Internet Res. 2017;19(6):e204.

    Article  PubMed  PubMed Central  Google Scholar 

  63. Morgan DG, Kosteniuk J, Stewart N, O’Connell ME, Karunanayake C, Beever R. The telehealth satisfaction scale: reliability, validity, and satisfaction with telehealth in a rural memory clinic population. Telemed J E-health. 2014;20(11):997–1003.

    Article  PubMed  Google Scholar 

  64. Lund AM. Measuring usability with the USE questionnaire. Usability Interface. 2001;8(2):3–6.

    Google Scholar 

  65. Attkisson CC, Zwick R. The client satisfaction questionnaire: psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plan. 1982;5(3):233–7.

    Article  CAS  Google Scholar 

  66. Burns JL, Serber ER, Keim S, Sears SF, et al. Measuring patient acceptance of implantable cardiac device therapy: initial psychometric investigation of the Florida Patient Acceptance Survey. J Cardiovasc Electrophysiol. 2005;16(4):384–90.

    Article  PubMed  Google Scholar 

  67. Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). RAND; 1994.

  68. Linnan L, Steckler A. Process evaluation for public health interventions and research. 2002.

  69. Chin J, Diehl V, Norman KL. Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1988:213–8.

  70. Fatehi F, Martin-Khan M, Smith AC, Russell AW, Gray LCJDT. Therapeutics Patient satisfaction with video teleconsultation in a virtual diabetes outreach clinic. Diabetes Technol Ther. 2015;17(1):43–8.

    Article  PubMed  Google Scholar 

  71. Wessel R. KH, De Witte L. D-Quest meetinstrument voor de tevredenheid over een hulpmiddel verstrekking. Hoensbroek. 2000.

  72. Delone WH, McLean ERJJ. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst. 2003;19(4):9–30.

    Article  Google Scholar 

  73. Elling S, Lentz L, De Jong M, editors. Website evaluation questionnaire: development of a research-based tool for evaluating informational websites. In: International conference on electronic government; 2007. Springer.

  74. Wang Y-SJI. Management. Assessment of learner satisfaction with asynchronous electronic learning systems. Inf Manag. 2003;41(1):75–86.

    Article  Google Scholar 

  75. Lewis JR, JIJoHCI. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78.

    Article  Google Scholar 

  76. Hart SG, Staveland LE. Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol. 1988;52:139–83.

    Article  Google Scholar 

  77. Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13(3):319–40.

    Article  Google Scholar 

  78. Brooke J. SUS: A “Quick and Dirty” Usability Scale. Milton Park: Taylor & Francis Group; 1986.

    Google Scholar 

  79. Loane M, Bloomer S, Corbett R, Eedy D, Gore H, Mathews C, et al. Patient satisfaction with realtime teledermatology in Northern Ireland. J Telemed Telecare. 1998;4(1):36–40.

    Article  CAS  PubMed  Google Scholar 

  80. Sabesan S, Simcox K, Marr I. Medical oncology clinics through videoconferencing: an acceptable telehealth model for rural patients and health workers. Internal Med J. 2012;42(7):780–5.

    Article  CAS  Google Scholar 

  81. Yen P-Y, Wantland D, Bakken S, editors. Development of a customizable health IT usability evaluation scale. AMIA Annual Symposium Proceedings; 2010: American Medical Informatics Association.

  82. Cox RM, Alexander GC. Measuring satisfaction with amplification in daily life: the SADL scale. Ear Hear. 1999;20(4):306–20.

    Article  CAS  PubMed  Google Scholar 

  83. Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. Amsterdam: Elsevier; 2012.

    Google Scholar 

  84. Ware JE Jr, Snyder MK, Wright WR, Davies AR. Defining and measuring patient satisfaction with medical care. Evaluat Program Plan. 1983;6(3–4):247–63.

    Article  Google Scholar 

  85. Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78.

    Article  Google Scholar 

  86. Bertini G. Il learning audit nella sanità. Guida ai processi di miglioramento autogestiti: FrancoAngeli; 2003.

  87. Brick JE, Bashshur RL, Brick JF, D’alessandri RMJTJ, . Public knowledge, perception, and expressed choice of telemedicine in rural West Virginia. Telemed J. 1997;3(2):159–71.

    Article  CAS  PubMed  Google Scholar 

  88. Demiris G, et al. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare. 2000;6(5):278–84.

    Article  CAS  PubMed  Google Scholar 

  89. Davis R, Gardner J, Schnall R. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Current HIV/AIDS Reports. 2020:1–16.

Download references

Acknowledgements

We would like to express our gratitude to the Institute for Futures Studies in Health of Kerman University of Medical Sciences for providing the research environment.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Authors and Affiliations

  1. Medical Informatics Research Center, Institute for Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran

    Sadrieh Hajesmaeel-Gohari

  2. Gastroenterology and Hepatology Research Center, Institute of Basic and Clinical Physiology Sciences, Kerman University of Medical Sciences, Kerman, Iran

    Kambiz Bahaadinbeigy

Authors
  1. Sadrieh Hajesmaeel-Gohari
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kambiz Bahaadinbeigy
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

SH and KB both contributed in designing the study. The selection and evaluation of the articles and data extraction were done by SH, and validated by KB. All authors participated in drafting the manuscript and approving the final version of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Kambiz Bahaadinbeigy.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent to publish

Not applicable.

Competing interests

The Authors declare that there is no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hajesmaeel-Gohari, S., Bahaadinbeigy, K. The most used questionnaires for evaluating telemedicine services. BMC Med Inform Decis Mak 21, 36 (2021). https://doi.org/10.1186/s12911-021-01407-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12911-021-01407-y

Keywords

  • Telemedicine
  • Evaluation
  • Questionnaire

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us