- Research article
- Open Access
Supporting deprescribing in hospitalised patients: formative usability testing of a computerised decision support tool
BMC Medical Informatics and Decision Making volume 21, Article number: 116 (2021)
Despite growing evidence that deprescribing can improve clinical outcomes, quality of life and reduce the likelihood of adverse drug events, the practice is not widespread, particularly in hospital settings. Clinical risk assessment tools, like the Drug Burden Index (DBI), can help prioritise patients for medication review and prioritise medications to deprescribe, but are not integrated within routine care. The aim of this study was to conduct formative usability testing of a computerised decision support (CDS) tool, based on DBI, to identify modifications required to the tool prior to trialling in practice.
Our CDS tool comprised a DBI MPage in the electronic medical record (clinical workspace) that facilitated review of a patient’s DBI and medication list, access to deprescribing resources, and the ability to deprescribe. Two rounds of scenario-based formative usability testing with think-aloud protocol were used. Seventeen end-users participated in the testing, including junior and senior doctors, and pharmacists.
Participants expressed positive views about the DBI CDS tool but testing revealed a number of clear areas for improvement. These primarily related to terminology used (i.e. what is a DBI and how is it calculated?), and consistency of functionality and display. A key finding was that users wanted the CDS tool to look and function in a similar way to other decision support tools in the electronic medical record. Modifications were made to the CDS tool in response to user feedback.
Usability testing proved extremely useful for identifying components of our CDS tool that were confusing, difficult to locate or to understand. We recommend usability testing be adopted prior to implementation of any digital health intervention. We hope our revised CDS tool equips clinicians with the knowledge and confidence to consider discontinuation of inappropriate medications in routine care of hospitalised patients. In the next phase of our project, we plan to pilot test the tool in practice to evaluate its uptake and effectiveness in supporting deprescribing in routine hospital care.
Deprescribing is the process of discontinuing a medication, supervised by clinicians, when the potential benefits of taking that medication no longer outweigh the potential harms to the patient [1, 2]. Despite the feasibility of deprescribing and the growing evidence that deprescribing can improve clinical outcomes, quality of life and reduce the likelihood of adverse drug events [1, 3, 4], the practice is not widespread, particularly in hospital settings.
Limited work has specifically focused on understanding the complexities surrounding deprescribing in hospitals, although this setting provides a safe context to trial deprescribing, with frequent access to healthcare professionals and close monitoring of patients. A recent study comprising interviews with geriatricians and pharmacists across four UK hospitals, identified a number of barriers to deprescribing in this context . These included barriers related to professional roles (i.e. perceived scope of practice, lack of confidence), to the inpatient environment (i.e. limited information, acute nature of admission), and to attitudes and beliefs (e.g. attitudes towards medications, beliefs about risks) . These barriers are consistent with those identified in a study conducted across six Australian hospitals, where many healthcare professionals did not consider deprescribing to be their responsibility, felt insecure and uncertain about deprescribing, and reported that it did not fit with the timeframe and workflow of an acute admission .
Clinical risk assessment tools can help prioritise patients for medication review and prioritise medications to deprescribe. One well validated and evidence-based tool is the Drug Burden Index (DBI). The DBI is a measure of a patient’s total exposure to medications with anticholinergic and sedative properties . International pharmaco-epidemiological studies show that high DBI is associated with adverse outcomes in older people such as poor physical function, falls, frailty and death [8, 9]. In pre-clinical studies, high DBI has been shown to cause functional impairment and frailty, which is reversible on deprescribing . DBI calculated without software has been used as an intervention  and an outcome  in clinical deprescribing studies. Stand-alone, web-based software to calculate and report on DBI has been used in clinical research as a risk assessment tool in the community [13, 14] and hospital setting . Thus, easy access to a patient’s DBI during a hospital admission could assist in identifying at-risk patients in whom deprescribing may be appropriate. Studies have shown that when clinicians are presented with a patient’s DBI (i.e. a paper-based DBI report), they view this as useful [13, 16] and it affects deprescribing recommendations. However, no research to date has investigated the impact of a patient’s DBI integrated into the electronic medical record on deprescribing or patient outcomes.
With the rapid uptake of health information technology, new ways of supporting clinicians to deprescribe have now emerged in the hospital setting. Two recent literature reviews highlight the potential positive impact computerised decision support (CDS) can have on inappropriate medication use [17, 18]. There is also some evidence to suggest that CDS is cost effective [19,20,21], but further studies examining cost outcomes are needed. What these reviews do show is that poor user acceptance of CDS is a significant barrier to uptake and effectiveness of the tools, and authors have recommended that future work focus on improving the design of CDS, ensuring better integration into workflow [17, 18]. The recent international SENATOR trial of the STOPP/START prescribing criteria in older inpatients found poor uptake of the recommendations made by the CDS , despite good uptake of, and outcomes from, recommendations delivered by clinical staff in previous studies . That is, only 15% of CDS recommendations were followed by clinicians in the SENATOR trial , compared to 84% when recommendations were delivered by clinical researchers in another randomized controlled trial .
There is now little doubt that designing technology that meets users’ needs and preferences, via a process of co-design with users, is necessary for successful implementation and uptake of health information technology . This current study represented a final step in a co-design process with doctors, nurses and pharmacists to develop a CDS tool, based on DBI, to facilitate deprescribing in hospitalised patients. The aim of this study was to conduct formative usability testing of our CDS tool to identify well-designed aspects and modifications required before pilot testing the tool in practice.
Two rounds of scenario-based formative usability testing with think-aloud protocol were used. Usability testing, in particular, the think-aloud method, has been identified as one of the most effective ways to identify usability problems with digital health tools . Think-aloud requires users to continuously verbalise their thoughts as they interact with a tool or system, allowing participants’ thought processes, feedback and emotional responses to be captured in real-time . This technique has been used to evaluate CDS tools in other contexts [27,28,29].
Round 1 testing occurred in July 2019 and Round 2 in September 2019.
Our CDS tool comprised multiple components within the electronic medical record, informed by a large number of focus groups and interviews with end-users , including review of paper prototypes. Central to our CDS tool is the concept of a DBI. We used high DBI (DBI > 1) as a trigger to prompt medication review and consideration of deprescribing, prioritising review of medications with anticholinergic and sedative effects. DBI is a clinical risk assessment tool that can be calculated from the information in a patient’s electronic prescription, without need for additional clinical information. In this paper, we report results related to only one component of our CDS tool, a DBI MPage. The MPage, or clinical workspace, is the primary work area for reviewing a patient’s DBI score (current and past) and medications in the electronic medical record. The DBI MPage is one of a number of pages available within the electronic medical record (i.e. it is not a separate system or external application). As shown in Fig. 1, it includes (1) a graph showing the trend in the patient’s DBI over the course of the current admission; clicking on a point on the graph displays the DBI score and the patient’s active medications on that day, (2) the patient’s total DBI score, (3) a list of the patient’s medications, each with a DBI score, (4) hyperlinks to deprescribing guides and consumer information leaflets, (5) buttons beside each medication which direct the user to the orders page for deprescribing, and (6) a help function.
In total, 17 individuals participated in the formative usability testing, six in Round 1 and 11 in Round 2. They comprised three senior doctors (consultants), five registrars, two junior doctors (1–3 years post-graduation) and seven pharmacists. All worked at the same teaching hospital in Sydney, Australia. To recruit participants, managers of the departments of aged care, general medicine and pharmacy were contacted by email and were asked to nominate participants, who were then contacted by a member of the research team and invited to take part.
Ten scenarios were developed by a Professor of geriatric pharmacology (SH) and pharmacist researcher (MD), with the assistance of a Human Factors researcher (MB) and two eHealth experts (MSB and SA). Scenarios were designed to test all functionalities (i.e. use all parts of the DBI MPage) and possible outcomes of the tool. An outcome common to doctors and pharmacists was review of medications. For doctors, outcomes also included decision to deprescribe vs. not to deprescribe and initiating deprescribing vs not deprescribing in hospital or on discharge. For pharmacists, outcomes included providing advice to deprescribe vs. providing advice to not deprescribe. Example scenarios are presented in the Additional file 1: Appendix.
Data collection and analysis
Testing was completed in a classroom on the campus of the hospital where the participants worked. At the beginning of the test session, each participant was briefed on the concept of a DBI, and on the purpose of the usability testing. Participants were then instructed to complete the scenarios while verbalising all thoughts as they interacted with the CDS tool. Two testers were present during each session, a pharmacist and an eHealth expert. One tester worked through scenarios with participants, prompting users to think aloud if they remained silent. The second tester acted as an observer. Following completion of scenarios, users participated in a brief semi-structured interview to explore their views of the CDS tool, including the component names, locations, navigation and utility. All test sessions, including semi-structured interviews, were audio-recorded and transcribed verbatim. The observer also took hand-written notes while observing scenarios.
All transcripts were reviewed and analysed independently by two Human Factors researchers (MB and WYZ), and observation notes were consulted if clarification was needed. Following independent analysis, the two researchers came together to discuss themes and reach a consensus on modifications or enhancements required for each CDS component based on user performance and feedback. These were presented to the wider research team, including members responsible for CDS redesign, to determine if modifications were feasible.
Ethics approval was obtained from Northern Sydney Local Health District Human Research Ethics Committee (LNR/17/HAWKE/138). All participants gave informed written consent to take part and to be audio-recorded.
Usability issues identified during Round 1 testing
Participants expressed positive views about several components of the DBI MPage. For example, colour to indicate risk level was viewed positively: “It’s obvious. So me, not being familiar with the numbers, it was easy for me to work out those probably are bad” (P2), as was the complete medication list “It’s nice just having a list of everyone’s medications because I find that…often a lot of our geriatric patients have a few, a lot of scrolling to do in order to get through all their medication. So, this is quite nice in the sense that it just gives you a list, which is very straightforward. And it just goes through what medications you would consider stopping”. (P7).
However, participants also made a number of suggestions to improve the design of the page, primarily related to more clearly conveying the meaning of a DBI score and the DBI graph. Table 1 lists the usability issues identified, illustrative quotes, and recommended changes. The recommended changes in Table 1 were implemented prior to Round 2 testing.
Usability issues identified during Round 2 testing
Consistent with Round 1 testing, during Round 2, users continued to express positive views about the DBI MPage. For example, a participant said: “It feels quite well-integrated into the current EMR (electronic medical record) system” (P6). The deprescribing guides that were available via a hyperlink on the page were particularly valued. For example, a user said “I’ll definitely use this. This looks amazing. I think that’s very useful” (P5). Despite these positive statements, we identified a number of additional usability issues during Round 2 testing, as shown in Table 2. These largely related to consistency. Users wanted information and functionality on the DBI MPage to be consistent with other pages in the electronic medical record. Following Round 2 testing, additional modifications were made to the DBI MPage in response to the user feedback. An updated MPage appears in Fig. 2.
Formative usability testing revealed a number of clear areas for improvement of our CDS tool, primarily related to terminology used, and consistency of functionality and display. Users were unfamiliar with the concept of a DBI score and suggested the inclusion of a definition and clear explanation of the DBI in the help section of the tool. Additional information was also required to allow accurate interpretation of the DBI graph. Furthermore, users expressed a preference for the CDS tool to look and function in a similar way to other decision support tools in the electronic medical record.
These findings are not unexpected, as they reflect two well-known usability heuristics: match between system and the real world and consistency and standards . Systems should be designed to speak the user’s language, including words, phrases and concepts familiar to users. Systems should also be consistent, with all actions, functions and words meaning the same thing when used on multiple screens or locations. These principles are fundamental to good user-centred design . Our results highlight the importance of consistency in both visual display and functionality when designing CDS tools in an electronic medical record.
Consistency is one of the greatest contributors to usability, with research showing that users will be faster to learn and use a new tool or system if it includes elements they are familiar with, or elements they frequently encounter while using other tools or systems . Although consistency in design appears to be at odds with innovation, usability testing is an ideal way to determine whether a new display or feature, one that is inconsistent with other elements, will be understood and used by end-users . In this study, we attempted to include a novel feature on our DBI graph: clicking on a point on the graph displayed medication data below the graph corresponding to that point in time. Users were confused by this feature, were not able to use it unprompted, and expressed a preference for having the graph function in a similar way to other graphs in the electronic system. Formative usability testing thus allowed us to determine this feature, in its current form, is unlikely to be used by clinicians. We recommend this approach for testing of any new CDS feature or function.
General impressions of the CDS tool were positive and we hope that our revisions to the tool based on this testing will result in a tool that is used and integrated into current clinician work processes. We appreciate that the CDS tool is unlikely to address all the barriers we identified in our preliminary work . However, if used as intended, the tool directly addresses key barriers, namely fitting with the timeframe and workflow of an acute admission, and uncertainty and insecurity about deprescribing. If all design elements are used, the CDS tool will (1) make it possible to quickly identify patients with a high DBI who could be candidates for deprescribing, (2) facilitate review of a patient’s medications over the course of their admission, (3) provide access to guidance on whether and how to deprescribe, (4) provide clinicians with relevant information to pass onto consumers, and (5) allow easy navigation to the page in the electronic medical record where deprescribing is possible.
Participants were naïve not only to the CDS tool, but to the concept of a DBI, and this impacted their perception of the intervention. Real-life implementation of the CDS tool is likely to be accompanied by an education module on reviewing polypharmacy in hospitalized patients (https://www.heti.nsw.gov.au/education-and-training/my-health-learning; course code: 185346268), which includes information on DBI. Formative testing allowed us to identify a number of usability problems with the CDS tool, but represents only one component of usability evaluation, and we recommend additional testing be undertaken to assess efficiency, effectiveness, and satisfaction following integration of the CDS tool into clinical workflow.
To ensure our CDS tool is used, usable and useful, we undertook usability testing and identified a number of ways to improve the layout, content and components of the tool. This approach proved extremely informative and we recommend formative usability testing be undertaken prior to implementation of all digital health interventions. Key lessons include ensuring CDS tools include visual displays and functionalities (i.e. a look and feel) consistent with other components in the eMR, and incorporating terminology known to end-users. We hope our revised CDS tool will address a key barrier to deprescribing by equipping clinicians with the knowledge and confidence to consider deprescribing in hospitalised patients within their limited timeframes and existing workflows. In the next phase of our project, we plan to test our CDS tool in practice in order to evaluate its uptake and effectiveness in hospital geriatric medicine and internal medicine settings.
Availability of data and materials
The datasets used and analysed during the current study are available from the corresponding author on reasonable request.
Electronic medical record
Drug Burden Index
Computerised decision support
Pro re nata
Scott IA, Hilmer SN, Reeve E, Potter K, Le Couteur D, Rigby D, et al. Reducing inappropriate polypharmacy: the process of deprescribing. JAMA Intern Med. 2015;175(5):827–34.
Reeve E, Gnjidic D, Long J, Hilmer S. A systematic review of the emerging definition of ‘deprescribing’ with network analysis: implications for future research and clinical practice. Br J Clin Pharmacol. 2015;80(6):1254–68.
Reeve E, Thompson W, Farrell B. Deprescribing: a narrative review of the evidence and practical recommendations for recognizing opportunities and taking action. Eur J Intern Med. 2017;38:3–11.
Page AT, Clifford RM, Potter K, Schwartz D, Etherton-Beer CD. The feasibility and effect of deprescribing in older adults on mortality and health: a systematic review and meta-analysis. Br J Clin Pharmacol. 2016;82(3):583–623.
Scott S, Twigg MJ, Clark A, Farrow C, May H, Patel M, et al. Development of a hospital deprescribing implementation framework: a focus group study with geriatricians and pharmacists. Age Ageing. 2019;49:102–10.
Baysari MT, Duong M, Zheng WY, Nguyen A, Lo S, Ng B, et al. Delivering the right information to the right person at the right time to facilitate deprescribing in hospital: a mixed methods multisite study to inform decision support design in Australia. BMJ Open. 2019;9(9):e030950.
Hilmer SN, Mager DE, Simonsick EM, Cao Y, Ling SM, Windham BG, et al. A drug burden index to define the functional burden of medications in older people. Arch Intern Med. 2007;167(8):781–7.
Kouladjian L, Gnjidic D, Chen TF, Mangoni AA, Hilmer SN. Drug Burden Index in older adults: theoretical and practical issues. Clin Interv Aging. 2014;9:1503–15.
Wouters H, van der Meer H, Taxis K. Quantification of anticholinergic and sedative drug load with the Drug Burden Index: a review of outcomes and methodological quality of studies. Eur J Clin Pharmacol. 2017;73(3):257–66.
Mach J, Gemikonakli G, Logan C, Vander Wyk B, Allore H, Ekambareshwar S, et al. Chronic polypharmacy with increasing Drug Burden Index (DBI) exacerbates frailty and impairs physical function, with effects attenuated by deprescribing, in aged mice. J Gerontol A Biol Sci Med Sci. 2020. https://doi.org/10.1093/gerona/glaa060.
Ailabouni N, Mangin D, Nishtala PS. DEFEAT-polypharmacy: deprescribing anticholinergic and sedative medicines feasibility trial in residential aged care facilities. Int J Clin Pharm. 2019;41(1):167–78.
van der Meer HG, Wouters H, Pont LG, Taxis K. Reducing the anticholinergic and sedative load in older patients on polypharmacy by pharmacist-led medication review: a randomised controlled trial. BMJ Open. 2018;8(7):e019042.
Kouladjian O’Donnell L, Gnjidic D, Chen TF, Hilmer SN. Integration of an electronic Drug Burden Index risk assessment tool into home medicines reviews: deprescribing anticholinergic and sedative medications. Ther Adv Drug Saf. 2019;10:2042098619832471.
Kouladjian O’Donnell L, Sawan M, Reeve E, Gnjidic D, Chen TF, Kelly PJ, et al. Implementation of the goal-directed medication review electronic decision support system (G-MEDSS)© into home medicines review: a protocol for a cluster-randomised clinical trial in older adults. BMC Geriatr. 2020;20(1):51.
Dearing ME, Bowles S, Isenor J, Kits O, Kouladjian O’Donnell L, Neville H, et al. Pharmacist-led intervention to improve medication use in older inpatients using the Drug Burden Index: a study protocol for a before/after intervention with a retrospective control group and multiple case analysis. BMJ Open. 2020;10(2):e035656.
Kouladjian L, Gnjidic D, Reeve E, Chen TF, Hilmer SN. Health care practitioners’ perspectives on deprescribing anticholinergic and sedative medications in older adults. Ann Pharmacother. 2016;50(8):625–36.
Scott IA, Pillans PI, Barras M, Morris C. Using EMR-enabled computerized decision support systems to reduce prescribing of potentially inappropriate medications: a narrative review. Ther Adv Drug Saf. 2018;9(9):559–73.
Dalton K, O’Brien G, O’Mahony D, Byrne S. Computerised interventions designed to reduce potentially inappropriate prescribing in hospitalised older adults: a systematic review and meta-analysis. Age Ageing. 2018;47(5):670–8.
Murphy EV. Clinical decision support: effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J Biol Med. 2014;87(2):187–97.
Vermeulen KM, van Doormaal JE, Zaal RJ, Mol PG, Lenderink AW, Haaijer-Ruskamp FM, et al. Cost-effectiveness of an electronic medication ordering system (CPOE/CDSS) in hospitalized patients. Int J Med Inform. 2014;83(8):572–80.
Lewkowicz D, Wohlbrandt A, Boettinger E. Economic impact of clinical decision support interventions based on electronic health records. BMC Health Serv Res. 2020;20(1):871.
O’Mahony D, Gudmundsson A, Soiza RL, Petrovic M, Jose Cruz-Jentoft A, Cherubini A, et al. Prevention of adverse drug reactions in hospitalized older patients with multi-morbidity and polypharmacy: the SENATOR* randomized controlled clinical trial. Age Ageing. 2020;49(4):605–14.
O’Connor MN, O’Sullivan D, Gallagher PF, Eustace J, Byrne S, O’Mahony D. Prevention of hospital-acquired adverse drug reactions in older people using screening tool of older persons’ prescriptions and screening tool to alert to right treatment criteria: a cluster randomized controlled trial. J Am Geriatr Soc. 2016;64(8):1558–66.
Dobson J. Co-production helps ensure that new technology succeeds. BMJ. 2019;366:l4833.
Broekhuis M, van Velsen L, Hermens H. Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. Int J Med Inform. 2019;128:24–31.
Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004;37(1):56–76.
Richardson S, Mishuris R, O’Connell A, Feldstein D, Hess R, Smith P, et al. “Think aloud” and “Near live” usability testing of two complex clinical decision support tools. Int J Med Inform. 2017;106:1–8.
Rose AF, Schnipper JL, Park ER, Poon EG, Li Q, Middleton B. Using qualitative studies to improve the usability of an EMR. J Biomed Inform. 2005;38(1):51–60.
Kilsdonk E, Peute LW, Riezebos RJ, Kremer LC, Jaspers MWM. From an expert-driven paper guideline to a user-centred decision support system: a usability comparison study. Artif Intell Med. 2013;59(1):5–13.
Nielson J, Mack R, editors. Usability inspection methods. New York: Wiley; 1994.
Budiu R. The power law of learning: Consistency vs. innovation in user interfaces 2016. https://www.nngroup.com/articles/power-law-learning/.
Nielsen J. Do interface standards stifle design creativity? 1999. https://www.nngroup.com/articles/do-interface-standards-stifle-design-creativity/.
We would like to acknowledge Ms Jenny Crane, A/Prof Susan Ogle and Dr. Mamta Bagia for their assistance with participant recruitment.
This study was funded by a NSW Health Translational Research Grant #274. Some members of NSW Health are co-authors, and assisted with data collection, analysis, and interpretation of data, and in writing the manuscript.
Ethics approval and consent to participate
Ethics approval was obtained from Northern Sydney Local Health District Human Research Ethics Committee (LNR/17/HAWKE/138). All participants gave informed written consent to researchers to take part and to be audio-recorded. All methods were carried out in accordance with relevant guidelines and regulations.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Baysari, M.T., Duong, M.H., Hooper, P. et al. Supporting deprescribing in hospitalised patients: formative usability testing of a computerised decision support tool. BMC Med Inform Decis Mak 21, 116 (2021). https://doi.org/10.1186/s12911-021-01484-z
- Computerised decision support