Usability evaluation of an integrated electronic medication management system implemented in an oncology setting using the unified theory of acceptance and use of technology

Abstract

Background

Medication management processes in an Oncology setting are complex and difficult to examine in isolation from interrelated processes and contextual factors. This qualitative study aims to evaluate the usability of an Electronic Medication Management System (EMMS) implemented in a specialised oncology unit using the Unified Theory of Acceptance and Use of Technology (UTAUT) framework.

Methods

The study was conducted in a 12-bed outpatient Oncology unit of a major teaching hospital 6 months following implementation of a commercial EMMS. In-depth semi-structured interviews were conducted with doctors, nurses and pharmacists using the system to assess usability. The UTAUT framework was used to analyse the results, which facilitated evaluation of interrelated aspects and provided a structured summary of user experience and usability factors.

Results

Direct cross-comparison between user groups illustrated that doctors and pharmacists were generally satisfied with the facilitating conditions (hardware and training), but had divergent perceptions of performance (automation, standardised protocols and communication and documented) and effort (mental and temporal demand) expectancy. In counterpoint, nurses were generally satisfied across all constructs. Prior experience using an alternative EMMS influenced performance and effort expectancy and was related to early dissatisfaction with the EMMS. Furthermore, whilst not originally designed for the healthcare setting, the flexibility of the UTAUT allowed for translation to the hospital environment.

Conclusion

Nurses demonstrated overall satisfaction with the EMMS, whilst doctors and pharmacists perceived usability problems, particularly related to restricted automaticity and system complexity, which hindered perceived EMMS success. The study demonstrates the feasibility and utility of the UTAUT framework to evaluate usability of an EMMS for multiple user groups in the Oncology setting.

Background

Electronic Medication Management Systems (EMMS) that support cancer care must consider key areas of practice that differentiate oncology from other medical specialties. Result flowsheets, the need for multidisciplinary workflow documentation, integration of laboratory and imaging reporting, and dealing with time-dependent, patient-specific chemotherapy dosing as well as supportive medications are some of these unique demands [1, 2]. Particularly demanding is the ordering of complex chemotherapy regimens, documentation, and workflow management functionalities such as multiple authorisations and checks of cytotoxic chemotherapy orders by an oncologist and verified by pharmacists and nurses [1, 2].

Studies of EMMS implementations in the oncology setting have found that critical success factors include; the design and usability of the EMMS [3], standardization of chemotherapy protocols [3,4,5,6], seamless integration with other health information systems and user workflows [2,3,4,5, 7,8,9,10,11], effective training and support [4, 12], support from leadership [4, 12, 13], collaborative project management [13], and effective ongoing maintenance and support [9, 13, 14].

Usability can be defined as the extent to which a product can be used by specific users to achieve specific goals, with effectiveness, efficiency, and satisfaction in a specific context of use [15]. The usability of a technology is determined not only by its user–computer interactions, but also by the degree to which it can be successfully integrated to perform tasks in the intended work environment [15]. Previous studies comparing the user satisfaction of doctors and nurses found different degrees of user acceptance [16,17,18,19], however most research on usability has been limited to one type of user through quantitative research.. Usability is evaluated through the interaction of user, system, and task in a specified setting. Several theories that measure individual and organizational acceptance and success have been designed and validated [20].

The Unified Theory of Acceptance and Use of Technology (UTAUT) is widely used to evaluate usability in the healthcare system [20]. However, no study has applied the UTAUT framework to explore usability in the Oncology EMMS setting. Furthermore, previous studies exploring Oncology EMMS have predominately focused on usability from a single user perspective in isolation, such as prescribers [21], EMMS providers [22], user requirements before implementation [23] or its impact on medication safety [2, 24].

In light of the complexity of the medication process and the difficultly of examining it in isolation from other interrelated processes and contextual factors, this study aims to evaluate the usability of an EMMS in the Oncology setting from the perspective of nurses, doctors and pharmacists using the UTAUT model.

Method

Sample and setting

This study was undertaken in a 12-bed outpatient Oncology unit in a major teaching hospital in Sydney, Australia. The outpatient oncology unit is attached to a 750-bed teaching hospital treating a range of adult solid organ tumours and haematology malignancies. A commercial EMMS (Cerner Millennium) was implemented in the Oncology unit in June-2018. The EMMS is used for chemotherapy and antineoplastic prescribing and administration. The EMMS is part of a single, integrated Electronic Medical Record (EMR) which contains most other aspects of patient care, including pathology, imaging, tests orders and electronic clinical documentation.

The EMMS introduced oncology-specific order sets to ensure that potentially related orders are available in one place for the convenience of the prescriber (Fig. 1). The order sets are designed to comply with best-practice clinical guidelines that are widely used in Australia [25]. Throughout the evaluation, approximately 70% of cancer treatment regimens were managed electronically, whilst the remaining 30% were paper based.

Fig. 1

Sample Electronic Medication Management System Oncology specific order sets

The EMMS allows prescribing, pharmacy verification, and documentation of drug administration and medication reconciliation. All orders are subjected to series of system checks including drug allergies and interactions. Following prescribing, orders appear on a summary view of medications prescribed, as well an administration screen called the Medication Administration Record (MAR). Once orders have been prescribed, the medication protocol auto-populates as a sequence on its calculated due date. Once verified by a pharmacist, it requires activation by the nurse prior to administration.

Participant recruitment began in December 2018, after 6 months had elapsed following implementation, and concluded in May 2019. Delaying evaluation by 6 months after EMMS implementation was to minimise the potential effect of initial user resistance and to allow adoption of new processes.

Interview guide

Semi-structured interviews were conducted, guided by the constructs of the UTUAT that consisted of 16 question items (Additional file 1), allowing participants to express their attitudes towards the constructs. UTAUT integrates eight theoretical perspectives on technology adoption and provides a comprehensive view of the factors related to users’ adoption behaviour [26]. The main UTAUT constructs are [27]:

• Performance expectancy (PE): “The degree to which an individual believes that using the system will help him or her attain gains in job performance”.

• Effort expectancy (EE): “The degree of ease associated with the use of the system”.

• Social influence (SI): “The degree to which an individual perceives that important others believe he or she should use the new system’.

• Facilitating conditions (FC): “The degree to which an individual believes that an organizational and technical infrastructure exists to support use of the system’.

• Behavioural intention (BI): “The willingness of respondents to use the system.”

The model was chosen as it examines the presence of each construct in a “real world” environment, allowing assessment of an individual’s intention to use a specific system, thus allowing for the identification of the key influences on acceptance in any given context [26].

Participants and recruitment

Purposive, maximum-diversity sampling was used to recruit nurses, doctors and pharmacists that care for oncology patients. They were expected to have worked in the oncology unit for at least 3-months prior to and during the implementation of the EMMS. For nursing staff, recruitment material was distributed by the ward’s allocated education nurse. Doctors and pharmacists were recruited via email from their heads of department. Recruitment ceased once data saturation for each of the user groups had been achieved. The same researcher (a pharmacist research student) interviewed all user groups. The researcher had not worked closely with interviewees previously or in the Oncology unit. The interviewees were provided with an explanation of the purpose of the study. It was also explained that despite the interviewer also being an employee of the hospital, their role in this study was purely an investigator [28]. The interviewee’s role in the study was also described to them as well as the opportunity to withdraw at any stage of the study [28].

Data analysis

Inductive and deductive methods [29] were used in the data analyses. Interviews were transcribed verbatim and a process of inductive and iterative transcript analysis methods [30] were applied to identify new or emerging themes. Data collection and inductive thematic analysis were iterative, allowing themes in the data to be explored in depth and contradictory data to be investigated. Inductively identified codes were independently developed by two of the researchers and the differences were resolved through discussion between the coders, assisted using NVivo 11 (QSR International Pty Ltd., released 2015, Version 1.0.1.1). Codes were grouped into overarching themes by the research team and then deductively mapped to the constructs under the UTAUT [31]. The UTAUT framework helped to guide the research; facilitate comparison of findings between user groups and relate these findings to the existing literature. In addition, an ongoing review of the literature on the topic of EMMS implementation enabled us to validate, compare, and extend our findings [32].

Results

Twenty-seven participants took part in the interviews, representing approximately 79% of all eligible staff. The mean length of interviews was 45 min. The demographic characteristics of the participants are presented in Table 1. Most of the participants (n = 23, 85%) were female and included doctors (n = 10), pharmacists (n = 7) and nurses (n = 10). Interviews generated extensive data relating to usability and acceptability of the EMMS. The themes in these categories were mapped to the UTAUT framework (Table 2). The perception of usability related to the various elements of the UTAUT have been described in detail below.

Table 1 Demographic of interview respondent – EMMS = Electronic Medication Management System

Table 2 Interview themes mapped to the Unified Theory of Acceptance and Use of Technology (UTAUT) framework. EMMS = Electronic Medication Management System

EMMS factors

EMMS factors impacted on the perception of Performance and Effort Expectancy across all user-groups. System factors such as introduced automation, standardized protocols and communication and the downstream impact on mental and temporal demand, had individualised impacts on perceived system usability.

Performance expectancy

Overall, all user groups felt that improved automation and standardization introduced by the EMMS led to the improvement of their overall performance. However, doctors and pharmacists expected a greater impact on their performance. They were disappointed by the limited degree of automation, influenced by pre-conceived ideas as well as experience with other systems.

Automation and medication safety

Reduced ‘mental energy’ introduced by the EMMS was described by a senior doctor involved in transcribing medication charts. Doctor 1 (Table 3) expressed satisfaction with the automated calculation of Body Surface Area and Area Under the Curve. However, doctors and pharmacists believed the safeguards within the EMMS aren’t adequate to prevent errors or inappropriate prescribing, as described by pharmacist 4 (Table 3).

Table 3 User perceptions of elements of performance expectancy extracted from nurses, doctors and pharmacists’ interviews

Standardizing protocols and dosing

Both nurses and doctors appreciated the impact of standardized protocols on their performance. The standardized layout of medication charts was perceived to improve medication safety and nurse’s performance. As they described that there was less chance of chart misinterpretations (nurse 3, Table 3) and a user-friendly layout that supports medication administration (nurse 5, Table 3). Similarly, doctors appreciated that ‘the protocols written in there are set in there … and are established protocols’ (doctor 3, Table 3). The benefits introduced by an automated dose variance report allowed management to oversee unusual prescribing, as described by doctor 3 (Table 3). Collectively, standardization and the ability to track dose variance were perceived as beneficial in improving overall performance.

Communication & documentation

Doctors and pharmacists felt that communication and documentation improved to a degree. Pharmacists appreciated the clarity of communication regarding dose reductions as described by pharmacist 1 (Table 3). The system also highlighted original gaps in their workflows that previously existed, indicated by doctor 2 (Table 3). However, there were concerns about the format of the treatment plan documents. The templates were not being used optimally, as described by pharmacist 2 (Table 3). From the nursing perceptive, it greatly improved communication between the multidisciplinary teams and amongst each other, as identified by nurse 2 (Table 3).

Effort expectancy

Effort levels for the various user groups drew on various dimensions of effort such as mental demand and temporal demand. Users described varying impacts of the EMMS on these elements of effort and are summarised in Table 4. Doctors expressed satisfaction with the integrated order sets and the reduced time spent tracking previously prescribed regimens, as explained by doctors 3 and 4 (Table 4). However, a lack of system flexibility and complexity had a negative impact on both the mental and temporal demand. Pharmacists expressed dissatisfaction across both mental and temporal demand, outlining the increased steps required to perform simple tasks, and an increase in administrative and staff support tasks. Nurses however, described less effort required to perform their role more effectively and efficiently.

Table 4 User perceptions of elements of effort expectancy extracted from nurses, doctors and pharmacists’ interviews

Organizational factors

Social influence

Amongst the nurses and pharmacists, there was a hierarchical influence of senior users on the juniors. Across all nurses there was the perception that all staff supported the EMMS (nurse 3, Table 5). On the other hand, junior pharmacists were feeling the negative influences of their seniors, highlighted by pharmacist 1, Table 5. Senior doctors were positively influencing their juniors (doctor 3, Table 5) despite some seniors, such as doctor 1 indicating ‘I’d prefer another system that I’ve used before’. Highlighting that the senior doctors were not allowing their individual perceptions to influence their junior doctors.

Table 5 User perceptions under the theme’s- organisational factors (social influence and facilitating condition) and individual factors (behavioural intention and Expectations based on experience related to performance and effort expectancy) from nurses, doctors and pharmacists’ interviews

Facilitating condition

Training

Nurses expressed appreciation for the initial training and ongoing ‘elbow support’ that was provided by the health informatics team during EMMS roll-out. Doctors and pharmacists expressed varying levels of confidence and competence in using the EMMS. Difficulties were associated with the inability to comfortably use the system when dealing with complex regimens, such as being unable to ‘amend or interpret dose adjustments’. Reasons for this are described by doctor 5, Table 5. Similarly, concerns were raised by pharmacists that they weren’t provided with an adequate level of training to give them the confidence to use the EMMS (pharmacist 3, Table 5).

Hardware

Doctors and Pharmacists were generally satisfied with the hardware being used to support the EMMS. On the other hand, all nurses expressed concern for the ergonomics of the mobile trolleys that the computers were being supported on, as described by nurse 5 (Table 5).

Individual factors

Expectations based on experience for performance and effort expectancy

This study showed that previous experience with alternate EMMS platforms at different hospitals, influenced user’s pre-defined expectations for this EMMS. Previous experience was found to influence expectations of the system related to performance and effort expectancy. Previous experience meant that users expected a degree of automation and safeguards not achieved by the current EMMS, as described by pharmacist 2 (Table 5). Expectations were associated with concerns raised by both pharmacists and doctors related to the increased reliance on a clinician memory rather than automation. There were expectations that a system would prevent ‘duplicated prescribed medications’, and include reminders such as ‘standardized vitamin b12 every 3 cycles’ as well as ‘cycle 1 dose reductions carrying over to cycle 2’, rather than putting greater reliance on prescriber’s memory as outlined by the doctors 4 and 5 (Table 5): On the other hand, nurses felt that a patient’s previous experience at other hospitals, positively influenced their perception, as described by nurse 5 (Table 5).

Behavioural intention

Overall, all user groups were enthusiastic about continuing to use the system in the short-term. For nurses this was attributed to the benefits they’re experiencing across all elements of the UTAUT model, illustrated by nurse 4 (Table 5). For doctors and pharmacists, the major draw point was that the system was integrated with the remainder of the hospital’s EMMS, illustrated by doctor 2 (Table 5). However, in the long-term doctors and pharmacists believed it to be only sustainable if the EMMS was ‘optimised’, illustrated by pharmacist 1 (Table 5), based on their expectations around automation and safety of the EMMS.

Discussion

This study determined the usability of a hospital-wide EMMS implemented in a specialized Oncology unit. As multidisciplinary teams are involved in cancer-care, the three key user-groups and their inter-connected yet unique requirements were identified [23]. It illustrated that doctors and pharmacists were satisfied with the EMMS if it provided desirable utility to their practice, and nurses when the EMMS was easy to use in the nursing processes [33]. Three key themes emerged throughout the study; EMMS factors, organizational factors and individual factors. The UTAUT framework facilitated the evaluation of interrelated aspects and provided a structured summary of usability user experience factors. It allowed for cross comparisons of user groups and illustrated the key role that ‘previous experience’ plays in influencing these constructs.

This study identified the need for multi-disciplinary usability studies as the EMMS impacts individual user groups in unique ways. Outcomes of user’s views illustrated that tasks related to effort expectancy varied between the user groups. Like previous studies it was identified that aspects of the EMMS layout, configuration and output quality can reduce the mental energy required of searching for important information and the time taken to achieve this [34]. This is highlighted in the contrast in acceptance between nurses with doctors and pharmacists’. It illustrated that doctors and pharmacists EMMS dissatisfaction was related to the system not providing desirable utility to their specific practice, unlike nurses who identified benefits to their work practices [33]. Similarly, our study highlighted that managerial satisfaction was due to the benefits of automated auditing [7], thus reducing the effort expectancy, which has previously been found to be a key driver for uptake and support [21, 34].

The UTAUT framework facilitated a systematic comparison between user-groups, allowing for evaluation of interrelated aspects. The UTAUT highlights the importance of contextual analysis in developing strategies for technology implementation within organisations [27]. Despite the ability of the existing models to predict intention and usage, current theoretical perspectives on individual acceptance are notably weak demonstrating the need for qualitative investigations [27]. The interrelated nature of facilitating condition and its impact on user groups demonstrated this. A lack of understanding of system functionality often results in independent troubleshooting by users, skipping steps or entering information in the EMMS differently to overcome system barriers [2]. By doctors not initially appreciating the source of the problem when prescribing, in a strongly multidisciplinary unit, it had downstream consequences for other user’s (pharmacists and nurses) workflow. The findings are consistent with those of previous studies that illustrate the role of facilitating conditions in user acceptance [35, 36]. Considerations should be made when system redesign isn’t possible, to incorporate lessons learned from the troubleshooting incidents into the training of staff to improve usability across all groups [37].

User’s previous experience with other EMMS resulted in pre-defined expectations of the EMMS, [27]. The difference in expectations between user groups who had had previous experience with an Oncology EMMS on a different platform at other hospitals (doctors and pharmacists) and those who didn’t (nurses), showed a difference in user acceptance. Palm et al. also demonstrated that confirmation of expectations was strongly associated with doctors’ and nurses’ satisfaction [33]. Previous studies have found that the main required system expectations were the possibility of searching data, security and confidentiality protocols, the availability of data analytic tools and collecting data about different types of cancer [23]. Achieving these expectations can potentially be achieved by the application of user centred design which has been shown to be more efficient and more usable than when not initially considered in the design [38]. However, user-centred design can also be limited by the nature of an integrated system. Whilst integrated systems facilitate the availability of information in one place, they also limit the customisability thus impeding innovation and potentially impacting on the usability in specialized settings [39, 40]. Interestingly, the UTAUT theorises ‘experience’ as a moderating influencer on facilitating condition, social influence and effort expectancy [27]. Our findings illustrated that experiences with a ‘best-of-breed’ system influenced performance and effort expectations of the integrated system.

The finding that performance expectancy can impact on the intention to use the new EMMS aligns with previous studies that claim performance expectancy is an important determinant of doctors’ intention to use a new technology [32]. The greatest benefit and also dis-benefit on performance expectancy for doctors was introduced by pre-defined order sets. Our findings are consistent with other studies that found a key facilitator for user-acceptance were order sentences that increased prescriber efficiency [21, 34] and evidence-based practices [6, 21, 41]. However, we also identified oncology specific issues related to protocol-mandated care. A major limitation in flexibility was found when doctors attempted to order outside the order sets [2] and when adjusting doses or frequencies. The perception of the system was that it lacked the flexibility required in the oncology setting, which was paradoxically often a symptom of systems attempting to improve safety [34]. Like previous studies [6], automaticity contributed to a negative user experience to a degree [2], as users (doctors and pharmacists) felt reduced efficacy was being facilitated by a greater number of mouse clicks [21] and effort were required to prescribe or review medications outside of standardised order sets.

Limitations of this study are that it was conducted in a single-centre site and was conducted 6 months following EMMS implementation, therefore initial user resistance issues that may have arisen during the shakedown phase had not been overcome or ironed out. It is currently unknown if these issues would remain after more time using the system. Evidence suggests that it may take up to 2 years post-implementation until the unit returns to complete stability as EMMS optimization can be an iterative process [42], and up to 4 years before there is a return on investment [34]. Furthermore, the full EMMS had not been implemented at the time of the study, with only 70% of the protocols integrated into the EMMS creating a hybridized environment. We also did not directly measure safety or efficiency quantitatively, and perceptions around these domains can differ substantially from objective assessments.

Conclusion

Our research has identified that different user groups had different usability needs from an EMMS implemented in the Oncology setting. Nurses demonstrated overall satisfaction with the EMMS, whilst doctors and pharmacists indicated that changes were required to better meet their needs. The greatest usability problems were related to restricted automaticity and system complexity, which hindered user uptake and EMMS success. Doctors and pharmacists indicated short-term system acceptance however questioned the long-term sustainability if usability issues weren’t addressed. The study demonstrates the feasibility and utility of the UTAUT framework to evaluate usability of an EMMS for multiple user groups in the Oncology setting.