- Research article
- Open Access
- Open Peer Review
Formative evaluation of a telemedicine model for delivering clinical neurophysiology services part II: The referring clinician and patient perspective
BMC Medical Informatics and Decision Makingvolume 10, Article number: 49 (2010)
Feedback from service users will provide insight into opportunities for improvement so that performance can be optimised. In the context of a formative evaluation referring clinician and patient satisfaction with a teleneurophysiology service was examined during a 20 week pilot period.
Questionnaire surveys of referring clinicians and patients were conducted.
Fifteen (58%) clinicians responded to the first part of a postal survey which examined their satisfaction with traditional clinical neurophysiology services. Nine (35%) responded to a second part which assessed their experience with the teleneurophysiology service. Teleneurophysiology improved satisfaction with waiting times, availability of results and impact on patient management. There was unanimous support from the clinicians for the permanent development of a teleneurophysiology service, although 2 cautioned this could delay establishing a neurology service in their region.
Eighty-two percent (116/142) of patients responded to a survey of their satisfaction with teleneurophysiology. This was compared to a previous report of 322 patients' experience with traditional CN services in Ireland. Waiting times for appointment were shorter for the former group who supported the telemedicine model recognising that it reduced the travel burden and need for overnight journeys. The two groups were equally anxious about the investigation although the teleneurophysiology patients received more prior information.
This study illustrates that teleneurophysiology is an acceptable model of service delivery for its primary customers. Their feedback is important in informing appropriate design and governance of such innovative models of health service provision.
Clinical neurophysiology (CN) involves the recording and assessment of bioelectric signals originating in the nervous system to evaluate its integrity. Traditional CN requires the patient to travel to a specialist central department to have these bioelectric signals recorded. With teleneurophysiology, data acquisition takes place at a satellite centre from where the data is communicated to a central department for analysis . To rephrase Jarvis and Stanberry , teleneurophysiology is the point-to-point communication of bioelectric signal data from one location to another for the purpose of interpretation or consultation. Fundamentally, the data rather than the patient does the travelling. Teleneurophysiology can improve access to CN services and expert opinion for patients and referring clinicians at remote under-served sites [3, 4]. In the context of burgeoning demand this model of service delivery has the potential to refocus limited resources and increase efficiencies [5, 6].
To test this potential, a teleneurophysiology service providing routine EEG investigation was established and assessed . A formative evaluation was conducted to examine the teleneurophysiology model in terms of its utility, technical performance and stakeholder satisfaction (clinical neurophysiology service providers, patients and referring clinicians). Formative or constructive evaluation is conducted the early stages of an implementation to assess its operational aspects and identify any changes to the service that are required to stabilise or optimise performance. This study reports on the teleneurophysiology service performance from the perspective of the patient and referring clinician. Technical and health service provider aspects of the evaluation are reported in an accompanying paper.
The primary customers of neurophysiology services are patients and referring clinicians. The referring clinician requires the expert CN opinion to guide the management of their patient. This requirement makes them, as well as their patients, users of the service. As with any newly introduced service, the consumer feedback is an essential component of its formative evaluation. Satisfaction surveys help assess performance of a service, further the understanding of customer needs and expectations, and expose opportunities for improvement. They are considered an important component of quality assurance programmes by bringing customer preferences into the quality assessment process [7–9]. The purpose of this study was to survey the satisfaction of referring clinicians and patients with the teleneurophsyiology model.
Over a twenty week period, during which 40 clinical sessions were conducted, a teleneurophysiology service which provided an EEG service was piloted and evaluated. A quasi-experimental one-group pretest-posttest design was employed to assess customer satisfaction with the teleneurophysiology model [10, 11]. The medical research ethics committees of both Beaumont Hospital and Sligo General Hospital reviewed and approved the study.
Context of the study
The context of this study is fully described a related paper . Beaumont Hospital on the east coast of Ireland provided the host expert CN department while the satellite centre was set up 130 miles away in the northwest region at Sligo General Hospital (SGH). Prior to the pilot only traditional CN services, based at either Dublin or Galway (85 miles from SGH), were available to the clinicians and patients of the northwest of Ireland. The consequent geographical inequities of this have previously been described .
The study participants included medical consultants in practice in the northwest of Ireland who referred patients to the teleneurophysiology service at SGH. Referred patients who attended for EEG recording at SGH were also invited to take part.
Referring clinical satisfaction
Referring clinician satisfaction data was collected by means of a detailed postal questionnaire delivered in two parts. Part one of the of the questionnaire was administered to collect pre-teleneurophysiology intervention observations while part two aimed to assess post-intervention observations. In this regard, the clinician satisfaction survey employed a pretest/posttest study design.
On receipt of a referral to the teleneurophysiology service, part one of the questionnaire together with information and consent form was sent by post to the referring clinician. Therefore, part one was essentially a survey of satisfaction with traditional clinical neurophysiology services and included questions on the profile of their clinical practice, their requirements for clinical neurophysiology, their perception of the impact of clinical neurophysiology investigation on patient management, and their perception of traditional clinical neurophysiology services in Ireland.
Part two of the questionnaire was sent to referring clinicians at the end of the twenty week pilot phase. This second part questioned the clinicians on aspects of the teleneurophysiology service including its impact on patient management, service quality and their perspective on establishing a permanent teleneurophysiology service for the region. Both parts of the questionnaire included spaces for respondents to add any additional comments and they were invited to continue these on a separate sheet if necessary. Referring clinicians were provided with pre-addressed stamped envelopes to return the questionnaires.
Patient satisfaction data was collected immediately after the patient's teleneurophysiology EEG recording at SGH. Patients, or their escort, were given information about the satisfaction survey and if happy to participate were asked to complete a consent form and a questionnaire before leaving the EEG centre. Questions regarding the teleneurophysiology appointment, waiting time for appointment, satisfaction with prior information about the EEG and arrangements made to get to the teleneurophysiology centre were asked. Patients were asked to add any additional comments and if necessary continue these on a separate sheet. It was intended that any such comments would provide further insight into the patients' experience. This data represents the posttest (after the teleneurophysiology intervention) observation. Pretest observation data was provided from a previously reported study of patient satisfaction with traditional CN services in Ireland .
Data from the satisfaction questionnaires were tabulated and from this totals and proportions or percentages in different categories were established. Pearson's chi-square test was used to assess the difference in proportions between parts 1 and 2 (pre and post intervention) of the clinician survey.
Responses from the survey of patient satisfaction with the teleneurophysiology model (post-intervention observations) were compared with data from a previously reported survey of satisfaction with the traditional model of CN service delivery. In that previous study 322 patients who attended 6 different CN centres in Dublin responded to a satisfaction questionnaire. These pre-intervention observations included data for both paediatric and adult patients, for all CN investigation modalities (EEG, EMG, NCS, and EP) and for patients resident in a region with and without a local CN service . As the pre and post-intervention data were not equivalent, the pre-intervention data represent a control period rather than a control group and provided a baseline for a preliminary comparison of the two models of service delivery. Pearson's chi-square test was used to compare proportions between the pre and post-intervention data.
Additional comments made by survey participants were also reviewed to further reveal service users, patients and their referring clinicians, perception of the teleneurophysiology model.
Over the 20 week teleneurophysiology pilot period, 40 separate clinic sessions were conducted during which 142 patients (74 female, 68 male) had an EEG investigation at SGH. Further details of the utilisation of the teleneurophysiology service are documented in an accompanying paper .
Twenty-six different consultant clinicians practicing in the northwest region of Ireland referred patients to the teleneurophysiology service during the pilot phase. Of these 15 (58%) responded to part one of the survey while 9 (35%) responded to part two. Not all survey questions were answered by each of the respondents. The range of different medical specialties that availed of the teleneurophysiology service is illustrated in the accompanying paper . Tables 1 and 2 summarise responses to parts one and two of the survey respectively.
In part one of the survey, 100% (12/12) of the responding clinicians considered that CN was relevant to their medical practice with 64% (9/14) indicating that they were more likely to avail of EEG services than other neurophysiology investigation modalities. Prior to the availability of the teleneurophysiology service clinicians referred their patients to either Dublin (130 miles from Sligo) or Galway (85 miles from Sligo) for investigation. All respondents (13/13) perceived that waiting time for traditional CN appointments was greater than 1 month with seven of these (54%) suggesting that patients wait longer than 3 months to be seen. All respondents agreed that the lack of a local service impacted negatively on patient management and indicated that this deficit either sometimes (77%) or always (15%) influenced their decision to refer patients for CN. Similarly, in the absence of a local service 67% (8/12) respondents reported that they would refer patients to other clinical specialities in preference to sending them for CN with 62% (8/13) indicating that they would routinely send patients for MRI or CT before referring for EEG. Seventy-five percent (9/12) of the referring clinicians believe that more than 50% of previous referrals were helpful in patient management, although 64% (7/11) said neurophysiology reports were often not available at the patient's follow-up appointment (table 1). Comments provided by respondents help to further contextualise these results (Additional file 1).
The second part of the survey documented the referring clinicians experience with the teleneurophysiology pilot service (table 2). Compared to the traditional services: the perceived waiting time for appointment was significantly reduced as 78% (7/9) reported that their patients received a teleneurophysiology appointment within 1 month of referral (p < .001); the proportion of respondents who believed that more than 50% of investigations were helpful in patient management rose significantly (p < .05) from 75% to 100% (8/8); and EEG reports were more promptly (p < .01) available (from 36% to 100%). One hundred percent (9/9) of the referring clinicians would welcome a permanent teleneurophysiology service in the northwest region. However, 25% (2/8) indicated that there may be negative implications with the introduction of such a service. Annotations included by respondents indicated that they had concerns that the establishment of a permanent service might delay the appointment of a consultant neurologist to the region. Additional comments provided by the referring clinicians further elucidate these responses (Additional file 2).
Eighty-two percent (116) of the 142 patients who had an EEG investigation at the SGH teleneurophysiology centre returned completed questionnaires. Not all survey questions were answered by each of the respondents (table 3). Twenty-five percent (24/97) of the teleneurophysiology patients who responded reported having had a previous EEG investigation. Information about where those previous EEGs were conducted was provided by 16 respondents. Of these 14 had travelled to Dublin, 1 had gone to Galway and 1 had gone to both Galway and Dublin on separate occasions. Forty-seven percent (37/78) said that they took time off either school, college or work to attend the teleneurophysiology centre.
A previous examination of traditional CN services in Ireland reported the satisfaction of 322 patients and provides a baseline for interpreting this patient survey . The teleneurophysiology pilot resulted in 80% (73/91) of patients being seen within 1 month of referral whereas 50% (157/313) of patients were seen within this time frame with traditional CN services. Similar percentages of patients attending traditional and teleneurophysiology services reported that the reason for investigation had been explained to them by their doctor (80% - 241/305 and 83% - 76/92 respectively). Likewise, equal proportions of patients attending either model of service delivery reported being anxious about the test (43% - 130/305 in traditional service and 46% - 76/92 in teleneurophysiology service). However, a notably larger proportion of teleneurophysiology patients (73% - 73/99 compared to 26% - 81/308) reported receiving written information about the test procedure in advance and knowing how long the test would take (86% - 83/97 against 35% - 109/308). Compared to traditional CN services more patients were accompanied to the teleneurophysiology centre (82% - 67/82 versus 65% - 184/285). In 52% (33/63) of the teleneurophysiology cases, the accompanying person also took time of either school or work to escort the patient. Thirty-eight (53%) respondents said that they had an appointment to return to the doctor who sent them to the teleneurophysiology centre compared to 70% (191/273) in the traditional CN service study. While 4% (12/292) of attendances at traditional CN services have been reported to necessitate an overnight journey, none of the teleneurophysiology EEG appointments required this. Comments provided by respondents further describe the patients' perspective on the teleneurophysiology service (Additional file 3).
When introducing a new or changed service it is appropriate to understand the needs, and expectations of the people who will use the service . This study illustrates both patient and referring clinician readiness for a telemedicine model of CN service delivery. Feedback from patients was positive indicating that teleneurophysiology helps meet their needs with minimal travel involved and a reduced waiting time for the one-hour test. Referring clinicians consider teleneurophysiology to be beneficial in the clinical management of patients with improved access to an informative test. This user acceptance of the model together with improved access to service and its cost effectiveness, reported in an accompanying manuscript , demonstrate potential for improved quality, safety and efficiency with the introduction of teleneurophysiology.
Part one of the referring clinician survey illustrated a dissatisfaction with the availability of traditional CN services and highlighted opportunities for improvement by the introduction of a teleneurophysiology service. The indication was that they are discouraged from referring patients as delays in getting an appointment can often make referral irrelevant in terms of managing the patient's condition. Where tests are carried out there are often delays in receiving the report. In addition, patients may be reluctant to travel for investigation. These findings echo those of a previously reported survey of referring clinician needs, expectations and satisfaction with CN services in Ireland . The second part of the survey showed that teleneurophysiology improved referring clinician satisfaction in terms of waiting time and impact on patient management. There was absolute support for continuing the teleneurophysiology service.
The main observations from the patient survey were that teleneurophysiology can reduce geographical inequities by extending CN services to under-served sites. The proportions of teleneurophysiology and traditional CN patients who reported being anxious about the investigation were similar despite the former group receiving more prior information. Teleneurophysiology eased the travel burden on patients and their families and eliminated a need for overnight accommodation to facilitate CN investigation. It also resulted in reduced waiting times for appointment for EEG investigation compared to traditional CN.
Study motivation is the main difference between this and previously reported evaluations of teleneurophysiology [14–16]. The focus of evaluation of a health informatics system will depend on where it is in its life-cycle . Identification of potential solutions motivates the exploration phase while technical feasibility and user acceptance of a particular solution is assessed during technical development phase. During the adaptation phase the system is evaluated within controlled conditions to examine how well it works in practice and to identify any adjustments required to optimise its performance. In the expansion phase the optimised system is extended to more users and more applications, and summative or outcome evaluation becomes the focus . Previously reported teleneurophysiology studies were motivated by demonstrating technical feasibility and user acceptance [14–16]. By comparison, this study was an adaptation phase evaluation to determine which aspects of the teleneurophysiology system work well and which parts may need improvement. In this regard the concept of system covers technical structure and process features.
In Ireland the need for more clinical neuroscience resources has been acknowledged . Teleneurophysiology can enhance efficiency and effectiveness of limited resources. It is technically achievable, acceptable to service providers [5, 14–16], referring clinicians and patients and the unit cost per investigation is comparable in both the traditional and teleneurophysiology mode of service delivery . It is well suited to the delivery of a routine EEG service, although some of the more complex CN procedures may still require patients to attend in person at a traditional centre. With teleneurophysiology, patients, clinicians and healthcare managers benefit from having local, fast access to important investigations for people with neurological symptoms. Although, the tele-based service has comparable costs to a traditional CN department, the advantage to the patients including: more patients receive an expert neurophysiology opinion; earlier diagnosis; early more rational and safer treatment; provision of consistent CN services to geographically diverse areas; and reduction in waiting lists [19, 20], as well as more equitable access to services from a referring clinician perspective are the arguments for implementing the described teleneurophysiology service.
As previously acknowledged, this study does not prove causality between the telemedicine model and customer satisfaction . For example, the pre and post-intervention patient groups in the patient satisfaction survey were not equivalent in a number of respects including age profile, CN investigation modality referred for, CN centre attended and region of residence (e.g. availability of local CN). Furthermore, due to unmet demand , waiting times for traditional CN services were considerably longer than for the teleneurophysiology service. The reduced waiting time may have positively biased the satisfaction of the post-intervention group. Without controlling these variables it cannot be claimed that the teleneurophysiology model is solely responsible for the level of patient satisfaction.
Twenty-five percent (24/97) of the teleneurophysiology patients had previously had an EEG at another centre. It may have been possible for this group to act as their own controls by asking them additional questions about their prior EEG investigation experience. These were not included as it was not known in advance the proportion of patients that were likely to have had a previous EEG. Also, the time span between their EEG investigations may have biased results. Another possible limitation of the patient questionnaire was the lack of questions regarding satisfaction with their clinician. However, the patient questionnaire was designed to specifically examine patient acceptance of the teleneurophysiology model.
The low response rate to parts one and two of the referring clinician questionnaire together with incomplete item response indicates self-selection and may have caused bias in the survey response. For example, those who chose not to participate may have been either more or less satisfied with the traditional CN and teleneurophysiology service models. Therefore, reasons other than the teleneurophysiology service may have influenced the findings of the clinician survey.
The pre-intervention measurements in the surveys represent data from a control period rather than data from a control group . In this regard the study contributes to a formative evaluation that can inform further development of a teleneurophysiology service. Definitive verification of the model and its sustainability will require prospective, randomised controlled studies.
In a globalised world international teleneurophysiology, where recordings made in one country are reviewed and interpreted by overseas clinicians, is a realistic concept [2, 6]. Where shortages in health service resources exist, there may be opportunities to outsource for services from other countries. As the model crosses over traditional organisational boundaries and potentially international borders, professional, clinical, ethical and legal implications of teleneurophysiology need to be fully understood so that appropriate governance of such services can be put in place [2, 21]. Essential to this is engagement with stakeholders, including service users, whose feedback will inform the delivery of a safe, efficient and acceptable service.
Connolly S, Fitzsimons M: Telemedicine and Clinical Neurophysiology. Teleneurology. Edited by: Wootton R, Patterson V. 2005, The Royal Society of Medicine Press Ltd, 95-104.
Jarvis L, Stanberry B: Teleradiology: threat or opportunity?. Clin Radiol. 2005, 60 (8): 840-5. 10.1016/j.crad.2005.04.001.
Raza T, Joshi M, Schapira RM, Agha Z: Pulmonary telemedicine--a model to access the subspecialist services in underserved rural areas. Int J Med Inform. 2009, 78 (1): 53-9. 10.1016/j.ijmedinf.2008.07.010.
Ronan L, Murphy K, Browne G, Connolly S, McMenamin J, Lynch B, Delanty N, Fitzsimons M: Needs analysis for tele-neurophysiology in the Irish North-Western Health Board. Ir Med J. 2004, 97 (2): 46-9.
Breen P, Murphy K, Browne G, Molloy F, Reid V, Doherty C, Delanty N, Connolly S, Fitzsimons M: Formative evaluation of a telemedicine model for delivering clinical neurophysiology services part I: Utility, technical performance and service provider perspective. BMC Medical Informatics and Decision Making. 2010, 10: 48-10.1186/1472-6947-10-48.
Patterson V, Wootton R: How can teleneurology improve patient care?. Nat Clin Pract Neurol. 2006, 2 (7): 346-7. 10.1038/ncpneuro0219.
Oja PI, Kouri TT, Pakarinen AJ: From customer satisfaction survey to corrective actions in laboratory services in a university hospital. Int J Qual Health Care. 2006, 18 (6): 422-8. 10.1093/intqhc/mzl050.
Jones BA, Bekeris LG, Nakhleh RE, Walsh MK, Valenstein PN: College of American Pathologists Physician satisfaction with clinical laboratory services: a College of American Pathologists Q-probes study of 138 institutions. Arch Pathol Lab Med. 2009, 133 (1): 38-4.
Zarbo RJ: Determining customer satisfaction in anatomic pathology. Arch Pathol Lab Med. 2006, 130 (5): 645-9.
Harris AD, McGregor JC, Perencevich EN, Furuno JP, Zhu J, Peterson DE, Finkelstein J: The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc. 2006, 13 (1): 16-23. 10.1197/jamia.M1749.
Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M: STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform. 2009, 78 (1): 1-9. 10.1016/j.ijmedinf.2008.09.002.
Fitzsimons M, Ronan L, Murphy K, Browne G, Connolly S, McMenamin J, Delanty N: Customer needs, expectations, and satisfaction with clinical neurophysiology services in Ireland: a case for tele-neurophysiology development. Ir Med J. 2004, 97 (7): 208-11.
Patterson V, Donaghy C, Loizou L: Email triage for new neurological outpatient referrals: what the customers think. J Neurol Neurosurg Psychiatry. 2006, 77 (11): 1295-6. 10.1136/jnnp.2006.092098.
Loula P, Rauhala E, Erkinjuntti M, Räty E, Hirvonen K, Häkkinen V: Distributed clinical neurophysiology. J Telemed Telecare. 1997, 3 (2): 89-95. 10.1258/1357633971930922.
Lasierra N, Alesanco A, Campos C, Caudevilla E, Fernandez J, Garcia J: Experience of a real-time tele-EEG service. Conf Proc IEEE Eng Med Biol Soc. 2009, 1: 5211-5214.
Holder D, Cameron J, Binnie C: Tele-EEG in epilepsy: review and initial experience with software to enable EEG review over a telephone link. Seizure. 2003, 12: 85-91. 10.1016/S1059131102002297.
Brender Y: Handbook of Evaluation Methods for Health Informatics. 2006, Elsevier Academic Press, London
Comhairle na nOspidéal: Report of the committee to review neurology and neurophysiology services. 2003, (last accessed 24-02-2010), [http://www.nai.ie]
Patterson V, Bingham E: Telemedicine for epilepsy: a useful contribution. Epilepsia. 2005, 46 (5): 614-5. 10.1111/j.1528-1167.2005.05605.x.
Patterson V: Teleneurology. J Telemed Telecare. 2005, 11 (2): 55-9. 10.1258/1357633053499840.
Schwamm LH, Audebert HJ, Amarenco P, Chumbler NR, Frankel MR, George MG, Gorelick PB, Horton KB, Kaste M, Lackland DT, Levine SR, Meyer BC, Meyers PM, Patterson V, Stranne SK, White CJ, American Heart Association Stroke Council, Council on Epidemiology and Prevention, Interdisciplinary Council on Peripheral Vascular Disease, Council on Cardiovascular Radiology and Intervention: Recommendations for the implementation of telemedicine within stroke systems of care: a policy statement from the American Heart Association. Stroke. 2009, 40 (7): 2635-60. 10.1161/STROKEAHA.109.192361.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/10/49/prepub
This project was funded by the Irish Health Services Executive Health. We are grateful for the support and enthusiasm of the following individuals: Alan Moran, Attracta Pender, Domhnall McLoughlin, Patrick Rooney, Marcella Lee, Maresa McGee, Paula Hickey, Ken Mulpeter, Peter Connolly, David Kelly, Mike McCrohan, Barry McKenna.
The authors declare that they have no competing interests.
PB, KM, GB, FM and MF were involved in the conception and design of the study, the collection of data, its analysis and interpretation. The process of the study and interpretation of data were further reviewed and monitored in detailed discussion with VR, CD, ND and SC. The manuscript was prepared for publication by PB and MF with critical review from the remaining authors. All authors approved the final version.