- Research article
- Open Access
- Published:
The most used questionnaires for evaluating telemedicine services
BMC Medical Informatics and Decision Making volume 21, Article number: 36 (2021)
Abstract
Background
Questionnaires are commonly used tools in telemedicine services that can help to evaluate different aspects. Selecting the ideal questionnaire for this purpose may be challenging for researchers. This study aims to review which questionnaires are used to evaluate telemedicine services in the studies, which are most common, and what aspects of telemedicine evaluation do they capture.
Methods
The PubMed database was searched in August 2020 to retrieve articles. Data extracted from the final list of articles included author/year of publication, journal of publication, type of evaluation, and evaluation questionnaire. Data were analyzed using descriptive statistics.
Results
Fifty-three articles were included in this study. The questionnaire was used for evaluating the satisfaction (49%), usability (34%), acceptance (11.5%), and implementation (2%) of telemedicine services. Among telemedicine specific questionnaires, Telehealth Usability Questionnaire (TUQ) (19%), Telemedicine Satisfaction Questionnaire (TSQ) (13%), and Service User Technology Acceptability Questionnaire (SUTAQ) (5.5%), were respectively most frequently used in the collected articles. Other most used questionnaires generally used for evaluating the users’ satisfaction, usability, and acceptance of technology were Client Satisfaction Questionnaire (CSQ) (5.5%), Questionnaire for User Interaction Satisfaction (QUIS) (5.5%), System Usability Scale (SUS) (5.5%), Patient Satisfaction Questionnaire (PSQ) (5.5%), and Technology Acceptance Model (TAM) (3.5%) respectively.
Conclusion
Employing specifically designed questionnaires or designing a new questionnaire with fewer questions and more comprehensiveness in terms of the issues studied provides a better evaluation. Attention to user needs, end-user acceptance, and implementation processes, along with users' satisfaction and usability evaluation, may optimize telemedicine efforts in the future.
Background
Telemedicine provides healthcare services when patients and healthcare providers are at different locations using Information and Communication Technologies (ICT). The core purpose of telemedicine is to improve the health of individuals and communities by exchanging useful information for various goals, such as preventing diseases, for diagnosis, monitoring, providing treatment, educating healthcare providers, and for conducting research [1, 2].
It is essential to identify the limitations, find approaches to overcome them, and create a reasonable structure to implement and use a telemedicine service successfully. Therefore, precise evaluations of telemedicine services are required [3]. Evaluation is the use of systematic and logical methods to assess the attributes of different aspects of the project, including its design, implementation, operation, and outcome [4]. The telemedicine evaluation process should be done independently and systematically on various features as the feasibility of the project, acceptance by participants, availability of service, technical capabilities of participants, clinical outcomes, user satisfaction, quality, and the cost–benefit of the offered service [3].
There are several methods to evaluate different aspects of a telemedicine project. In order to choose the right evaluation method, several issues should be considered: (1) Considering project goals; (2) determining the required amount of budget, energy, and time for evaluation; (3) selecting the appropriate method based on the predetermined criteria and metrics; (4) using understandable and easy evaluation methods for users; (5) being completely aware of the evaluation method used in the study; and (6) using validated methods for the evaluation process [5].
In the field of telemedicine, the most commonly used tools for evaluating user satisfaction are questionnaires and interviews [6]; and in order to assess the usability of telemedicine systems, usually questionnaires, interviews, observations, self-descriptive, and logging methods are used [7]. The clinical outcomes of telemedicine services are evaluated by means of biometric measurements, quality of life, and disease-specific tools that are all generally questionnaires [8].
Researchers use a variety of questionnaires for various purposes. Some of them are valid questionnaires specified for telemedicine whereas some others are more general. Due to the great variety of questionnaires, it is difficult for researchers to choose the best tool to evaluate their telemedicine services. Therefore, this review aims at listing the most commonly used questionnaires for evaluating telemedicine, which would in turn help researchers to select the most appropriate questionnaire, based on their objectives, to evaluate telemedicine services.
Methods
Database and date
This is a scoping review. PubMed database was searched in August 2020 to retrieve articles without date limitation. Approximately, 80–90% of studies conducted in Telemedicine were accessible on PubMed, which sufficed for our purpose; therefore, no other database was used [9].
Search strategy
The following combinations of keywords were used to find relevant articles in the Title/Abstract search field: (telemedicine) AND (evaluation OR assessment) AND (questionnaire).
Inclusion criteria
Original observational and interventional research articles in which a valid and referenced questionnaire was used to evaluate telemedicine services and systems were included in this study.
Exclusion criteria
Articles were excluded if they were review articles, non-English language, without full text, not specifically addressing telemedicine, did not have communication between patients and healthcare providers in mHealth, evaluated clinical outcomes, evaluated acceptance of telemedicine prior to implementation, did not include details about the questionnaires used, questionnaires that have not been validated, or combined questionnaires.
Article selection
First, all retrieved articles were assessed based on title and abstract by one researcher (SH). Next, the same researcher reviewed the full-text of the selected articles. When necessary, the second researcher (K.B), being a telemedicine professional, provided consult. Manual searching was also performed to find additional articles that had used specific evaluation questionnaires to investigate Telemedicine. Finally, a list of all included articles was prepared.
Data extraction
The following information was extracted from included articles: author, year of publication, journal of publication, evaluation type, and evaluation questionnaire.
Data analysis
Descriptive statistics (frequency and frequency percent) was used to analyze data.
Results
PubMed database found 214 articles. After removing review articles and those written in a language other than English, 208 articles remained whose titles and abstracts were screened. After excluding 95 articles, the full-text of the 113 remaining articles were reviewed. Finally, 53 articles were included in this study (Fig. 1), from which the required data was extracted (Table 1).
Journal of publication
Nine articles were published in the Telemedicine and e-health Journal (17%), six in the Journal of Medical Internet Research (11.5%), four in the International Journal of Telerehabilitation (7.5%), four in the Journal of Telemedicine and Telecare (7.5%), and the remaining articles were published in other journals (n = 30, 56.5%).
Evaluation type
A questionnaire was used in 26 articles (49%) to evaluate patients’ or physicians’ satisfaction; in 18 articles to evaluate system usability (34%), in 6 articles to evaluate acceptance (11.5%), and in one article to evaluate the implementation process (2%). In 2 articles a questionnaire was used to evaluate the usability and the acceptance or satisfaction of individuals (3.5%).
Evaluation questionnaire
The final list of articles showed that the Telehealth Usability Questionnaire (TUQ) (n = 10, 19%) [10,11,12,13,14,15,16,17,18,19] and Telemedicine Satisfaction Questionnaire (TSQ) (n = 7, 13%) [20,21,22,23,24,25,26] were the most commonly used. Then, the Service User Technology Acceptability Questionnaire (SUTAQ) (n = 3, 5.5%) [27,28,29], the Client Satisfaction Questionnaire (CSQ) (n = 3, 5.5%) [30,31,32], the Questionnaire for User Interaction Satisfaction (QUIS) (n = 3, 5.5%) [33,34,35], the System Usability Scale (SUS) questionnaire (n = 3, 5.5%) [36,37,38], the Patient Satisfaction Questionnaire (PSQ) (n = 3, 5.5%) [22, 39, 40], the Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) (n = 2, 3.5%) [41, 42], the Patient Assessment of Communication during Telemedicine (PACT) questionnaire (n = 2, 3.5%) [43, 44], the Post Study System Usability Questionnaire (PSSUQ) (n = 2, 3.5%) [19, 45] and the Technology Acceptance Model (TAM) (n = 2, 3.5%) [46, 47] were the most used questionnaires, respectively. The rest of the articles had used other questionnaires (n = 18, 34%) [14, 19, 46, 48,49,50,51,52,53,54,55,56,57,58,59,60,61,62] and 4 articles had used more than one questionnaire.
The most used questionnaires
Out of 59 used questionnaires in the studies, the most frequently (more than two times) used were TUQ, TSQ, SUTAQ, CSQ, QUIS, SUS, PSQ, TSUQ, PACT, PSSUQ, and TAM, which have been shown in Fig. 2.
Some information about these questionnaires are presented below:
Parmanto et al. in 2016, formally introduced TUQ. This questionnaire evaluates the usability of telehealth services. It has 21 items that are based on 6 criteria including usefulness (3 items), ease of use and learnability (3 items), interface quality (4 items), interaction quality (4 items), reliability (3 items), and satisfaction and future use (4 items) [18]. The TUQ was designed using existing telehealth and general usability questionnaires and includes all usability aspects.
The TSQ was developed by Yip et al. in 2003 for evaluating patient satisfaction in using telemedicine. This questionnaire has 14 items with no categories [26].
Hirani et al. presented SUTAQ in 2017, as a tool to judge the acceptability of telehealth services by investigating user opinions. This questionnaire consists of 22 items that is categorized in six sectors: benefits (9 items), privacy (4 items), personal care skill (3 items), substitution (3 items), and satisfaction (3 items) [29].
Bakken et al. in 2006, designed TSUQ. This questionnaire evaluates the satisfaction and usefulness criteria of video consultation and telephone monitoring. This instrument has 26 items in two satisfaction (21 items) and usefulness (5 items) sections [42].
PACT questionnaire was introduced by Agha et al. in 2009. This questionnaire measures the satisfaction of patients and physicians of their communication by asking 33 questions. These questions are asked about the quality of physician and patient communication (16 questions), clinical skills (9 questions), interpersonal skills (6 questions), and comfortable of visit (2 questions) sectors [44].
Attkisson et al. in 1982, firstly introduced CSQ. This questionnaire evaluates the general users’ satisfaction with different services. The original version of CSQ has 18 items; however, the questionnaire has other versions in which there are only 3 or 8 items [65].
QUIS was developed by Chin et al. in 1988. It is a usability testing instrument that has been designed to measure the satisfaction of users about their interaction with the computer interface. QUIS consists of 27 items in five sections. The first section measures the overall satisfaction, and the four others measure user satisfaction regarding screen, terminology and information, learning, and system capabilities aspects [69].
The SUS questionnaire was created by Brooke et al. in 1986 for testing the usability of electronic systems. This tool has ten items with no specific categories [78].
The PSQ was firstly designed by Ware et al. in 1983 for measuring the satisfaction of patients with using medical cares. The original version of this questionnaire has 80 items measured in seven categories, such as overall satisfaction, technical quality, interpersonal mode, communication, financial features, time spent with physician, availability, and convenience [84]. A shorter version of this questionnaire also exists which consists of 18 items [67].
PSSUQ was developed by the International Business Machines Corporation (IBM) to evaluate users’ satisfaction at the end of the study. This questionnaire has three versions. The last version consists of 16 questions with seven Likert scales. These questions are arranged is such way to evaluate matters in the three following sections: system usefulness, information quality, and interface quality [75, 83].
TAM is a model that shows how users use and accept a technology. A questionnaire that has been developed based on this model was presented by Davis in 1989. This questionnaire investigates 12 items that are generally concerned with two matters: the perceived usefulness and perceived ease of use sections with 12 items [77].
Discussion
The TUQ, TSQ, SUTAQ were the three most common telemedicine-specific questionnaires used in the retrieved articles, respectively. The four most used general questionnaires related to satisfaction and usability evaluation were the CSQ, QUIS, SUS, and PSQ.
In the below sections it will be discussed why each questionnaire is used more frequently. To do this we chose the most three used questionnaires which are designed specific for telemedicine and four general questionnaires that are used in telemedicine evaluation as well.
TUQ is usable for evaluating different types of telehealth systems such as sole target videoconferencing systems, computer and mobile-based systems, and collecting the opinions of both patients and physicians [18]. As TUQ is considerably comprehensive when comparing with other questionnaires such as QUIS and SUS, it is most frequently used for evaluating the usability of telemedicine systems. SUS questionnaire, like QUIS, is a general usability evaluation tool that is also used to evaluate telemedicine systems; yet, unlike QUIS it has no segmentation and it examines fewer items [78]. Also, ever since TUQ has been introduced, SUS has been seldom used.
The TSQ is a preliminary tool for measuring patient satisfaction with telemedicine, and it is used quite frequently. This might be due to the fact that TSQ covers various satisfaction factors such as the quality of care, quality of virtual visits, interpersonal interactions, and also it has fewer number of items [26].
Although TSUQ was also introduced many years ago, it has rarely been used for studies conducted in telemedicine, which may be due to two reasons: (1) it has been designed specifically for telemedicine services provided to diabetes patients, and (2) it investigates more items than other questionnaires especially, TSQ [42].
The CSQ was a generic most used questionnaire for evaluating users’ satisfaction with the telemedicine services. This may be due to the fact that CSQ measures the quality of diverse attributes such as the physical environment, procedure, assistance staff, type of service, treatment staff, amount or length of service, service quality, outcome, and general satisfaction with few items [65]. PSQ is a general questionnaire as well and it is designed to evaluate patient satisfaction. However, with presenting the TSQ, evaluators preferred to use TSQ as a specific questionnaire in assessing the users’ satisfaction of a telemedicine service. Evaluators may use different tools depending on their purpose of investigation, and the only thing that matters is the validity of the used tool.
The SUTAQ is the only questionnaire that specifically designed to gather patients’ opinions about the acceptability of the telehealth systems [29]. While there are various models for assessing technology acceptance, this instrument is used more in this field since it has been specifically designed for evaluating telehealth acceptance.
In terms of evaluation types for telemedicine services, users’ satisfaction, usability, acceptance, and implementation process are non-clinical aspects that have been evaluated by use of validated questionnaires. Approximately, a questionnaire had been used on half of the collected articles for evaluating the satisfaction of telemedicine users; and approximately in a third of the articles, the usability of telemedicine systems was evaluated.
Telemedicine acceptance [27,28,29, 46, 47, 49, 62] and its implementation process [50] were evaluated in only a few articles. A review study investigating the evaluation methods for telemedicine services in hospitals showed that telemedicine users' satisfaction more frequently evaluated than clinical and economical aspects, and the most commonly used method to evaluate satisfaction was questionnaire. Similar to our result, this study also showed that the development and implementation process of telemedicine had gained less attention [6].
Evaluating the implementation process could be an essential stage for the successful usage of telemedicine services due to showing obstacles and facilitators of implementation [50]. Also, considering the needs of the users and planning the process based on these needs may affect the successful implementation of telemedicine services and increase the rate of their acceptance. Therefore, we recommend that researchers pay more attention to this aspect in the evaluation of telemedicine services. Another study that reviewed the usability evaluation methods of eHealth services for patients who had HIV revealed that questionnaire was the most employed method for evaluation [89].
Based on the researchers’ knowledge, this is the first review study that has identified validated and the most used questionnaires in evaluating telemedicine services. Nevertheless, this study has its own limitations. Only PubMed database was used for searching and retrieving articles. In the search strategy, we used telemedicine as a MeSH term and evaluation and assessment keywords that may use in different grammatical style. Moreover, we restricted our search to the Title/Abstract field. These issues may cause some articles missed from our study. For this reason, we conducted a manual search using the name of telemedicine specific questionnaires in the PubMed database and added additional articles.
Conclusion
Many questionnaires were used to assess telemedicine services. Some of them were specifically designed to evaluate telemedicine services while others were more general. As the results of this study showed, telemedicine service evaluators should use questionnaires specifically designed for telemedicine to assess its various aspects. However, if an evaluator wants to design a questionnaire for evaluating a telemedicine service, it is better to pay attention to goal-based design, the number of questions, and comprehensiveness in terms of the issues studied.
Users' satisfaction with telemedicine services and the usability of the system have been two of the most frequently investigated issues in telemedicine when comparing with other existing issues in the field. Attention to user needs, end-user acceptance, and implementation processes may optimize telemedicine efforts in the future.
Availability of data and materials
Not applicable.
Abbreviations
- ICT:
-
Information and communication technologies
- TUQ:
-
Telehealth usability questionnaire
- TeSS:
-
Telehealth satisfaction scale
- USE:
-
Usefulness, satisfaction, and ease of use
- TSQ:
-
Telemedicine satisfaction questionnaire
- CSQ:
-
Client satisfaction questionnaire
- FPAS:
-
Florida patient acceptance survey
- TSUQ:
-
Telemedicine satisfaction and usefulness questionnaire
- PSQ:
-
Patient satisfaction questionnaire
- QUIS:
-
Questionnaire for user interaction satisfaction
- PACT:
-
Patient assessment of communication during telemedicine
- D-QUEST:
-
Dutch version of Quebec user evaluation of satisfaction with assistive technology
- SUTAQ:
-
Service user technology acceptability questionnaire
- WEQ:
-
Web evaluation questionnaire
- CSUQ:
-
Computer system usability questionnaire
- TAM:
-
Technology acceptance model
- SUS:
-
System usability scale
- Health-ITUES:
-
Health-information technology usability survey
- SADL:
-
Satisfaction with amplification in daily life
- PSSUQ:
-
Post study system usability questionnaire
- ASQ:
-
The after-scenario questionnaire
- TMPQ:
-
Telemedicine perception questionnaire
References
WHO. Telemedicine, Telehealth, and Health Information Technology, 2006 [cited 2020 August 14]. https://www.who.int/goe/policies/countries/usa_support_tele.pdf?ua=1.
WHO. Telemedicine 2010 [cited 2020 August 14]. https://www.who.int/goe/publications/goe_telemedicine_2010.pdf.
Pan American Health Organization. Defining evaluation indicators for telemedicine as a tool for reducing health inequities: Study and results of a community of practice 2016 [cited 2020 August 14]. https://iris.paho.org/handle/10665.2/28562.
Ammenwerth E, Brender J, Nykänen P, Prokosch H-U, Rigby M, Talmon J. Visions and strategies to improve evaluation of health information systems: reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Informat. 2004;73(6):479–91.
McNair JB. Handbook of evaluation methods for health informatics. Amsterdam: Elsevier; 2006.
AlDossary S, Martin-Khan MG, Bradford NK, Smith ACJI. A systematic review of the methodologies used to evaluate telemedicine service initiatives in hospital facilities. Int J Med Informat. 2017;97:171–94.
Klaassen B, van Beijnum BJ, Hermens HJ. Usability in telemedicine systems—a literature survey. Int J Med Inf. 2016;93:57–69.
Kidholm K, Clemensen J, Caffery LJ, Smith AC. The Model for Assessment of Telemedicine (MAST): a scoping review of empirical studies. J Telemed Telecare. 2017;23(9):803–13.
Bahaadinbeigy K, Yogesan K, Wootton R. MEDLINE versus EMBASE and CINAHL for telemedicine searches. Telemed J E-health. 2010;16(8):916–9.
Layfield E, Triantafillou V, Prasad A, Deng J, Shanti RM, Newman JG et al. Telemedicine for head and neck ambulatory visits during COVID‐19: evaluating usability and patient satisfaction. Head Neck. 2020:1–9.
Mostafa PI, Hegazy AA. Dermatological consultations in the COVID-19 era: Is teledermatology the key to social distancing? An Egyptian experience. J Dermatol Treat. 2020:1–6.
Whitehouse CR, Long JA, Maloney LM, Daniels K, Horowitz DA, Bowles KH. Feasibility of diabetes self-management telehealth education for older adults during transitions in care. Res Gerontol Nurs. 2020;13(3):138–45.
Vaughan EM, Naik AD, Lewis CM, Foreyt JP, Samson SL, Hyman DJ. Telemedicine training and support for community health workers: improving knowledge of diabetes. Telemed J E-health. 2020;26(2):244–50.
Cheng O, Law N-H, Tulk J, Hunter M. Utilization of telemedicine in addressing musculoskeletal care gap in long-term care patients. JAAOS Global Res Rev. 2020;4(4).
Serwe KM. The provider’s experience of delivering an education-based wellness program via telehealth. Int J Telerehabil. 2018;10(2):73.
Serwe KM, Hersch GI, Pancheri KJ. Feasibility of using telehealth to deliver the “Powerful Tools for Caregivers” program. Int J Telerehabil. 2017;9(1):15.
Yu DX, Parmanto B, Dicianno BE, Watzlaf VJ, Seelman KD. Accessibility needs and challenges of a mHealth system for patients with dexterity impairments. Disabil Rehabil Assist Technol. 2017;12(1):56–64.
Parmanto B, Lewis AN Jr, Graham KM, Bertolet MH. Development of the telehealth usability questionnaire (TUQ). Int J Telerehabil. 2016;8(1):3.
Schutte J, Gales S, Filippone A, Saptono A, Parmanto B, McCue M. Evaluation of a telerehabilitation system for community-based rehabilitation. Int J Telerehabil. 2012;4(1):15.
Lin CH, Tseng WP, Wu JL, Tay J, Cheng MT, Ong HN, et al. A double triage and telemedicine protocol to optimize infection control in an emergency department in Taiwan during the COVID-19 pandemic: retrospective feasibility study. J Med Internet Res. 2020;22(6):e20586.
Talal AH, McLeod A, Andrews P, Nieves-McGrath H, Chen Y, Reynolds A, et al. Patient reaction to telemedicine for clinical management of hepatitis C virus integrated into an opioid treatment program. Telemed J E-Health. 2019;25(9):791–801.
Le LB, Rahal HK, Viramontes MR, Meneses KG, Dong TS, Saab S. Patient satisfaction and healthcare utilization using telemedicine in liver transplant recipients. Dig Dis Sci. 2019;64(5):1150–7.
Losiouk E, Lanzola G, Del Favero S, Boscari F, Messori M, Rabbone I, et al. Parental evaluation of a telemonitoring service for children with type 1 diabetes. J Telemed Telecare. 2018;24(3):230–7.
Segura-Sampedro JJ, Rivero-Belenchón I, Pino-Díaz V, Rodríguez Sánchez MC, Pareja-Ciuró F, Padillo-Ruiz J, et al. Feasibility and safety of surgical wound remote follow-up by smart phone in appendectomy: a pilot study. Ann Med Surgery. 2012;2017(21):58–62.
Dechêne L, Tousignant M, Boissy P, Macoir J, Héroux S, Hamel M, et al. Simulated in-home teletreatment for anomia. Int J Telerehabil. 2011;3(2):3.
Yip MP, Chang AM, Chan J, MacKenzie AE. Development of the telemedicine satisfaction questionnaire to evaluate patient satisfaction with telemedicine: a preliminary study. J Telemed Telecare. 2003;9(1):46–50.
Torbjørnsen A, Småstuen MC, Jenum AK, Årsand E, Ribu L. The service user technology acceptability questionnaire: psychometric evaluation of the norwegian version. JMIR Human Factors. 2018;5(4):e10255.
Dario C, Luisotto E, Dal Pozzo E, Mancin S, Aletras V, Newman S, et al. Assessment of patients’ perception of telemedicine services using the service user technology acceptability questionnaire. Int J Integrated Care. 2016;16(2):13.
Hirani SP, Rixon L, Beynon M, Cartwright M, Cleanthous S, Selva A, et al. Quantifying beliefs regarding telehealth: development of the whole systems demonstrator service user technology acceptability questionnaire. J Telemed Telecare. 2017;23(4):460–9.
Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Videoconferences and treatment of depression: satisfaction score correlated with number of sessions attended but not with age. Telemed J E-Health. 2020;26(7):898–904.
Christensen LF, Gildberg FA, Sibbersen C, Skjoeth MM, Nielsen CT, Hansen JP. Disagreement in satisfaction between patients and providers in the use of videoconferences by depressed adults. Telemed J E-health. 2020;26(5):614–20.
Fields BG, Behari PP, McCloskey S, True G, Richardson D, Thomasson A, et al. Remote ambulatory management of veterans with obstructive sleep apnea. Sleep. 2016;39(3):501–9.
Safdari R, Langarizadeh M, Ramezani A, Khodaveisi T, Nejad AF. Development of a store-and-forward telescreening system of diabetic retinopathy: lessons learned from Iran. J Diabetes Metabolic Disorders. 2018;17(1):31–6.
Hosseini F, Ayatollahi H, Salehi SH, Jafar K. Teleburn: designing a telemedicine application to improve burn treatment. Open Med Informat J. 2018;12:33–41.
Alanzi T, Istepanian R, Philip N. Design and usability evaluation of social mobile diabetes management system in the Gulf Region. JMIR Res Protocols. 2016;5(3):e93.
Smaradottir B, Gerdes M, Martinez S, Fensli R. The EU-project United4Health: user-centred design of an information system for a Norwegian telemedicine service. J Telemed Telecare. 2016;22(7):422–9.
Ligons FM, Mello-Thoms C, Handler SM, Romagnoli KM, Hochheiser H. Assessing the impact of cognitive impairment on the usability of an electronic medication delivery unit in an assisted living population. Int J Med Informat. 2014;83(11):841–8.
Lacerda TC, von Wangenheim CG, von Wangenheim A, Giuliano I. Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context. J Biomed Inform. 2014;52:222–30.
McFarland LV, Raugi GJ, Reiber GE. Primary care provider and imaging technician satisfaction with a teledermatology project in rural Veterans Health Administration clinics. Telemed J E-health. 2013;19(11):815–25.
Wallace P, Haines A, Harrison R, Barber JA, Thompson S, Roberts J, et al. Design and performance of a multi-centre randomised controlled trial and economic evaluation of joint tele-consultations [ISRCTN54264250]. BMC Family Practice. 2002;3:1.
McGloin H, O’Connell D, Glacken M, Mc Sharry P, Healy D, Winters-O’Donnell L, et al. Patient empowerment using electronic telemonitoring with telephone support in the transition to insulin therapy in adults with type 2 diabetes: observational, pre-post, mixed methods study. J Med Internet Res. 2020;22(5):e16161.
Bakken S, Grullon-Figueroa L, Izquierdo R, Lee N-J, Morin P, Palmas W, et al. Development, validation, and use of english and spanish versions of the telemedicine satisfaction and usefulness questionnaire. J Am Med Inform Assoc. 2006;13(6):660.
Hatton J, Chandra R, Lucius D, Ciuchta E. Patient satisfaction of pharmacist-provided care via clinical video teleconferencing. J Pharm Pract. 2018;31(5):429–33.
Agha Z, Schapira RM, Laud PW, McNutt G, Roter DL. Patient satisfaction with physician-patient communication during telemedicine. Telemed J E-health. 2009;15(9):830–9.
Kwon S, Kim J, Kang S, Lee Y, Baek H, Park K. CardioGuard: a brassiere-based reliable ECG monitoring sensor system for supporting daily smartphone healthcare applications. Telemed J E-Health. 2014;20(12):1093–102.
Agnisarman SO, Chalil Madathil K, Smith K, Ashok A, Welch B, McElligott JT. Lessons learned from the usability assessment of home-based telemedicine systems. Appl Ergonom. 2017;58:424–34.
Parra C, Jódar-Sánchez F, Jiménez-Hernández MD, Vigil E, Palomino-García A, Moniche-Álvarez F, et al. Development, implementation, and evaluation of a telemedicine service for the treatment of acute stroke patients: TeleStroke. Interact J Med Res. 2012;1(2):e15.
Mehra S, van den Helder J, Visser B, Engelbert RH, Weijs PJ, Kröse BJ. Evaluation of a blended physical activity intervention for older adults: mixed methods study. J Med Internet Res. 2020;22(7):e16380.
Leppert F, Siebermair J, Wesemann U, Martens E, Sattler SM, Scholz S, et al. The INFluence of remote monitoring on anxiety/depression, quality of life, and device acceptance in ICD patients: a prospective, randomized, controlled, single-center trial. Clin Res Cardiol. 2020.
van der Meij E, Huirne JA, Ten Cate AD, Stockmann HB, Scholten PC, Davids PH, et al. A perioperative ehealth program to enhance postoperative recovery after abdominal surgery: process evaluation of a randomized controlled trial. J Med Internet Res. 2018;20(1):e1.
Host BK, Turner AW, Muir J. Real-time teleophthalmology video consultation: an analysis of patient satisfaction in rural Western Australia. Clin Exper Optometry. 2018;101(1):129–34.
D’Hooghe M, Van Gassen G, Kos D, Bouquiaux O, Cambron M, Decoo D, et al. Improving fatigue in multiple sclerosis by smartphone-supported energy management: the MS TeleCoach feasibility study. Multiple Sclerosis Related Disorders. 2018;22:90–6.
Ammenwerth E, Modre-Osprian R, Fetz B, Gstrein S, Krestan S, Dörler J, et al. HerzMobil, an integrated and collaborative telemonitoring-based disease management program for patients with heart failure: a feasibility study paving the way to routine care. JMIR Cardio. 2018;2(1):e11.
Oliveira AC, Mattos S, Coimbra M. Development and assessment of an e-learning course on pediatric cardiology basics. JMIR Med Educ. 2017;3(1):e10.
Roberts S, Spain B, Hicks C, London J, Tay S. Telemedicine in the Northern Territory: an assessment of patient perceptions in the preoperative anaesthetic clinic. Aust J Rural Health. 2015;23(3):136–41.
Poulsen KA, Millen CM, Lakshman UI, Buttner PG, Roberts LJ. Satisfaction with rural rheumatology telemedicine service. Int J Rheumatic Diseases. 2015;18(3):304–14.
Vélez O, Okyere PB, Kanter AS, Bakken S. A usability study of a mobile health application for rural Ghanaian midwives. J Midwifery Women’s Health. 2014;59(2):184–91.
Penteado SP, Bento RF, Battistella LR, Silva SM, Sooful P. Use of the satisfaction with amplification in daily life questionnaire to assess patient satisfaction following remote hearing aid adjustments (telefitting). JMIR Med Informat. 2014;2(2):e18.
Scalvini S, Tridico C, Glisenti F, Giordano A, Pirini S, Peduzzi P, et al. The SUMMA project: a feasibility study on telemedicine in selected Italian areas. Telemed J E-health. 2009;15(3):261–9.
Kim HM, Lowery JC, Hamill JB, Wilkins EG. Patient attitudes toward a web-based system for monitoring chronic wounds. Telemed J E-Health. 2004;10(Suppl 2):S26-34.
Eminovic N, Wyatt JC, Tarpey AM, Murray G, Ingrams GJ. First evaluation of the NHS direct online clinical enquiry service: a nurse-led web chat triage service for the public. J Med Internet Res. 2004;6(2):e17.
van Rosmalen-Nooijens K, Lo Fo Wong S, Prins J, Lagro-Janssen T. Young people, adult worries: randomized controlled trial and feasibility study of the internet-based self-support method “feel the vibe” for adolescents and young adults exposed to family violence. J Med Internet Res. 2017;19(6):e204.
Morgan DG, Kosteniuk J, Stewart N, O’Connell ME, Karunanayake C, Beever R. The telehealth satisfaction scale: reliability, validity, and satisfaction with telehealth in a rural memory clinic population. Telemed J E-health. 2014;20(11):997–1003.
Lund AM. Measuring usability with the USE questionnaire. Usability Interface. 2001;8(2):3–6.
Attkisson CC, Zwick R. The client satisfaction questionnaire: psychometric properties and correlations with service utilization and psychotherapy outcome. Eval Program Plan. 1982;5(3):233–7.
Burns JL, Serber ER, Keim S, Sears SF, et al. Measuring patient acceptance of implantable cardiac device therapy: initial psychometric investigation of the Florida Patient Acceptance Survey. J Cardiovasc Electrophysiol. 2005;16(4):384–90.
Marshall GN, Hays RD. The patient satisfaction questionnaire short-form (PSQ-18). RAND; 1994.
Linnan L, Steckler A. Process evaluation for public health interventions and research. 2002.
Chin J, Diehl V, Norman KL. Development of an Instrument Measuring User Satisfaction of the Human-Computer Interface. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1988:213–8.
Fatehi F, Martin-Khan M, Smith AC, Russell AW, Gray LCJDT. Therapeutics Patient satisfaction with video teleconsultation in a virtual diabetes outreach clinic. Diabetes Technol Ther. 2015;17(1):43–8.
Wessel R. KH, De Witte L. D-Quest meetinstrument voor de tevredenheid over een hulpmiddel verstrekking. Hoensbroek. 2000.
Delone WH, McLean ERJJ. The DeLone and McLean model of information systems success: a ten-year update. J Manag Inf Syst. 2003;19(4):9–30.
Elling S, Lentz L, De Jong M, editors. Website evaluation questionnaire: development of a research-based tool for evaluating informational websites. In: International conference on electronic government; 2007. Springer.
Wang Y-SJI. Management. Assessment of learner satisfaction with asynchronous electronic learning systems. Inf Manag. 2003;41(1):75–86.
Lewis JR, JIJoHCI. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78.
Hart SG, Staveland LE. Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol. 1988;52:139–83.
Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 1989;13(3):319–40.
Brooke J. SUS: A “Quick and Dirty” Usability Scale. Milton Park: Taylor & Francis Group; 1986.
Loane M, Bloomer S, Corbett R, Eedy D, Gore H, Mathews C, et al. Patient satisfaction with realtime teledermatology in Northern Ireland. J Telemed Telecare. 1998;4(1):36–40.
Sabesan S, Simcox K, Marr I. Medical oncology clinics through videoconferencing: an acceptable telehealth model for rural patients and health workers. Internal Med J. 2012;42(7):780–5.
Yen P-Y, Wantland D, Bakken S, editors. Development of a customizable health IT usability evaluation scale. AMIA Annual Symposium Proceedings; 2010: American Medical Informatics Association.
Cox RM, Alexander GC. Measuring satisfaction with amplification in daily life: the SADL scale. Ear Hear. 1999;20(4):306–20.
Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for User Research. Amsterdam: Elsevier; 2012.
Ware JE Jr, Snyder MK, Wright WR, Davies AR. Defining and measuring patient satisfaction with medical care. Evaluat Program Plan. 1983;6(3–4):247–63.
Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Human Comput Interact. 1995;7(1):57–78.
Bertini G. Il learning audit nella sanità. Guida ai processi di miglioramento autogestiti: FrancoAngeli; 2003.
Brick JE, Bashshur RL, Brick JF, D’alessandri RMJTJ, . Public knowledge, perception, and expressed choice of telemedicine in rural West Virginia. Telemed J. 1997;3(2):159–71.
Demiris G, et al. A questionnaire for the assessment of patients’ impressions of the risks and benefits of home telecare. J Telemed Telecare. 2000;6(5):278–84.
Davis R, Gardner J, Schnall R. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Current HIV/AIDS Reports. 2020:1–16.
Acknowledgements
We would like to express our gratitude to the Institute for Futures Studies in Health of Kerman University of Medical Sciences for providing the research environment.
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Contributions
SH and KB both contributed in designing the study. The selection and evaluation of the articles and data extraction were done by SH, and validated by KB. All authors participated in drafting the manuscript and approving the final version of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent to publish
Not applicable.
Competing interests
The Authors declare that there is no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Hajesmaeel-Gohari, S., Bahaadinbeigy, K. The most used questionnaires for evaluating telemedicine services. BMC Med Inform Decis Mak 21, 36 (2021). https://doi.org/10.1186/s12911-021-01407-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s12911-021-01407-y
Keywords
- Telemedicine
- Evaluation
- Questionnaire