Discrepancy between perceptions and acceptance of clinical decision support Systems: implementation of artificial intelligence for vancomycin dosing
BMC Medical Informatics and Decision Making volume 23, Article number: 157 (2023)
Artificial intelligence (AI) tools are more effective if accepted by clinicians. We developed an AI-based clinical decision support system (CDSS) to facilitate vancomycin dosing. This qualitative study assesses clinicians' perceptions regarding CDSS implementation.
Thirteen semi-structured interviews were conducted with critical care pharmacists, at Mayo Clinic (Rochester, MN), from March through April 2020. Eight clinical cases were discussed with each pharmacist (N = 104). Following initial responses, we revealed the CDSS recommendations to assess participants' reactions and feedback. Interviews were audio-recorded, transcribed, and summarized.
The participants reported considerable time and effort invested daily in individualizing vancomycin therapy for hospitalized patients. Most pharmacists agreed that such a CDSS could favorably affect (N = 8, 62%) or enhance (9, 69%) their ability to make vancomycin dosing decisions. In case-based evaluations, pharmacists' empiric doses differed from the CDSS recommendation in most cases (88/104, 85%). Following revealing the CDSS recommendations, we noted 78% (69/88) discrepant doses. In discrepant cases, pharmacists indicated they would not alter their recommendations. The reasons for declining the CDSS recommendation were general distrust of CDSS, lack of dynamic evaluation and in-depth analysis, inability to integrate all clinical data, and lack of a risk index.
While pharmacists acknowledged enthusiasm about the advantages of AI-based models to improve drug dosing, they were reluctant to integrate the tool into clinical practice. Additional research is necessary to determine the optimal approach to implementing CDSS at the point of care acceptable to clinicians and effective at improving patient outcomes.
Contributions to the literature
The authors have implemented several educational modules, including quality improvement (PMID 24988421)
We also developed and implemented acute kidney injury electronic surveillance and prediction (PMID 26070247; 31054606)
Artificial intelligence (AI) is the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions . In simple terms, AI is defined as “machines that mimic cognitive functions similar to the human mind, such as learning and problem solving” . It has many potential applications in the medical field, including predicting and diagnosing diseases, clinical decision support, medical education, and drug discovery [3,4,5,6]. Investigations regarding the AI's role in individualizing drug dosing are limited, but it could enrich the practice with its higher precision and timeliness. Clinical decision support systems (CDSS) hold the promise of helping clinicians make better and more personalized treatment decisions, streamlining workflow, improving outcomes, and reducing healthcare expenditures [7,8,9,10].
Vancomycin is widely used to treat serious Gram-positive infections, including methicillin-resistant Staphylococcus aureus (MRSA). The vancomycin therapeutic window is narrow [11, 12]. Overdosing could lead to vancomycin-associated acute kidney injury, and underdosing could lead to antibiotic ineffectiveness and resistance emergence [13,14,15]. Especially in ICU patients, the complex and dynamic pathophysiology leads to significant inter- and intraindividual variability in vancomycin pharmacokinetics [16, 17]. Studies have found that nearly 2/3 of patients in ICU do not meet the initial target blood concentration of vancomycin . Therefore, there is a need to evolve from a one-size-fits-all dosing approach to tailored regimens designed to improve antimicrobial efficacy and safety . With the advances in AI, attempts have been made to develop dose guidance or plasma concentration prediction models for vancomycin both in and outside the ICU [20, 21]. While these efforts are necessary, AI is only effective if it can be implemented into the workflow at the point of care. Clinical outcomes benefits and technical integration considerations need to be evaluated parallel with the end-user experience. In any clinical scenario, if a very high performing AI-based CDSS is implemented to inform drug dosing, but clinicians are unwilling or unable to adjust drug dosing according to AI recommendations, there could be a potential risk of harm, waste, or confusion.
In the case of vancomycin, we developed an AI-based CDSS tool that uses a comprehensive analysis of patient-specific data to suggest an appropriate regimen for vancomycin (dose and interval) to optimize drug-level target achievement. In this study, we qualitatively assessed pharmacist perceptions and attitudes toward implementing such a CDSS tool using case-based scenarios to evaluate the feasibility and acceptability of implementation.
Setting and participants
The study was conducted at Mayo Clinic in Rochester, MN, from March through April 2020 and reported according to the Standards for Reporting Qualitative Research . This study was deemed exempt by the institutional review board of Mayo Clinic, Rochester, MN (#19–010472). Mayo Clinic is a large academic medical center with 215 intensive care unit beds. Published institutionally endorsed vancomycin dosing recommendations adapted from national guidelines [23, 24] were in place throughout the study. These broadly included weight-based vancomycin doses and intervals informed by the estimated creatinine clearance (eCLcr) based on the Cockcroft-Gault equation. In addition, the care team tailored therapeutic plans according to other patient-specific factors (i.e., the severity of illness, urine output)- consistent with routine clinical practice. The institutional policy provides broad authority for pharmacists to independently select and monitor vancomycin regimens in hospitalized patients. Clinical pharmacists are available seven days weekly from 07:00 to 22:30 in the ICU nursing units and overnight by consultation.
We selected clinical pharmacists involved with hospitalized patients treated with vancomycin via purposive sampling. This study focused on critically ill patients, given their high variability in vancomycin pharmacokinetics and the robust availability of granular patient data in ICUs. Individuals were recruited from different specialty areas that provide care to critically ill patients, including decentralized clinical pharmacists in the ICU and those consulted overnight. Participants were also sampled to attain diversity in levels of expertise and training, from relatively new to the practice to advanced training in individualizing pharmacotherapy.
Interviews were conducted for two months. Two weeks before the interview, a study team member (XL) emailed each eligible pharmacist a recruitment letter describing the reasons for doing the study. Those who provided informed consent to participate were scheduled for a 60-min semi-structured interview, in person, over the phone, or virtually. No one else was present besides the participants and researchers, and no compensation was provided for participation.
Semi-structured interviews were informed by an a priori developed interview guide created for the study (Supplementary Appendix 1). The interview script was designed to assess the pharmacists' demographics, attitudes toward CDSS implementation and acceptance of its recommendations, and reasons for reluctance when presented with clinical cases. There were three primary sections of the interview script, 1) capturing participants' demographics (age, gender), specialty, experience level, and current approach to dosing and monitoring vancomycin, 2) focusing on participants' impressions regarding implementing a CDSS tool to inform vancomycin dosing, and 3) involvement in eight cases inspired by the real-world experience of patients treated with vancomycin. Pharmacists were asked to recommend a dosing scheme before revealing the CDSS recommendations. We then asked about their willingness to accept the CDSS-recommended dosing regimen. The interview concluded with a discussion about factors that would enhance the acceptability of CDSS recommendations in practice. Interviews were audio-recorded with the permission of each participant, transcribed verbatim, and de-identified. The transcripts of the interviews were then subjected to content analysis for themes. Finally, the summary of each interview was returned to the participant for correction.
The primary objective was to identify factors influencing agreement with and adherence to the CDSS recommendations for vancomycin. Secondary objectives included assessing critical care pharmacists' impressions towards CDSS, interactions with CDSS, acceptance of its recommendations, and reasons for not following the dosing regimen recommended by CDSS.
Interview transcripts were analyzed using the thematic analysis method described by Braun and Clarke . The following steps were performed in the analysis, 1) transcripts were repeatedly read to ensure familiarisation with the data, 2) XL and XS produced initial codes, 3) emerging themes and subthemes were subsequently generated based on significant patterns in the codes, 4) themes and subthemes were continuously reviewed and refined using the constant comparison approach before being included in the final write-up, and 5) the final themes and subthemes were discussed with all members of the research team to ensure that a consensus was reached.
We recruited 15 pharmacists, 2 of whom dropped out due to loss to follow-up, and finally, 13 of whom completed the interviews (Table 1). The median interview time was 42 min (34 to 52 min). Participants were 31% female, with an average age of 35 (ranging from 27 to 50) years. The majority of participants were critical care pharmacists (N = 8). In addition, five interviewees were overnight pharmacists responsible for consultation about critically ill patients. Their levels of work experience, training, and CDSS understanding varied.
Vancomycin dosing work burden
For patients newly started on vancomycin, most pharmacists (N = 10, 77%) admitted the need for 0–5 time reviews of the vancomycin regimen, with 23% (N = 3) of pharmacists indicating the need for 6–10 time reviews of the vancomycin regimen. Eight (62%) pharmacists indicated they would need up to 10 min when determining the vancomycin dose for a new patient. Five (38%) pharmacists mentioned needing 11 to 20 min to complete the vancomycin dosing regimen (Table 2).
For patients on stable maintenance vancomycin doses, all pharmacists indicated that they require 0–5 time reviews of the vancomycin regimen per patient, with 62% (8/13) needing 5 to 10 min per review, and one (8%) needing 11 to 15 min. After reviewing the dosing regimen, 62% (8/13) of pharmacists mentioned changing the vancomycin dosing regimen in 70–80% of patients. To determine the appropriate dosing regimen, 69% (N = 9) of pharmacists primarily used the Cockcroft-Gault formula (Table 2).
Attitudes toward vancomycin CDSS in general
Most pharmacists (N = 8, 62%) agreed or strongly agreed that an AI-based CDSS tool could affect dosing decisions for vancomycin. In addition, most (N = 9, 69%) agreed or strongly agreed that CDSS could enhance their ability to make vancomycin dosing decisions, and they would use it in their work. However, 69% (N = 9) were neutral on whether their performance would be superior if CDSS was routinely used (Fig. 1).
Acceptability of vancomycin CDSS
Each of the 13 pharmacists provided a dosing scheme for eight ICU cases who received vancomycin therapy. The total number of cases discussed with all pharmacists was 104. In 88 (85%) cases, the pharmacists suggested a different dosing regimen from the CDSS recommendation (40 cases with lower dosage suggestions and 48 cases with higher dosage suggestions). However, when the CDSS recommendations were revealed, most pharmacist decisions (69/88, 78%) did not adjust the dosing regimen. This was despite generally a positive attitude toward CDSS with comments like "a great project, and it would be really helpful and useful." Others endorsed how CDSS could prompt them to double-check their decision, which could help them make more informed and thoughtful decisions. Still, few were enthusiastic about modifying their previously recommended doses. The provided rationales for their decisions included a distrust and lack of understanding of the justification for the CDSS recommendations, concerns related to the lack of dynamic evaluation or in-depth analysis, incomplete integration of all clinical data, and lack of availability of risk index along with the recommendation (Tables 3 and 4). In the few discrepant cases where the pharmacist was willing to change their doses (19 out of 88, 22%), the primary motivation was the proximity of their dose selection with the CDSS recommendation.
In the final interview question, participants discussed potential methods to enhance compliance with the CDSS dosage recommendations for each patient (Table 5). These included integrating CDSS into electronic health records (EHR), transparency regarding the recommended doses' rationale, including a holistic view of the patient's clinical state, and risk index.
This qualitative study is the first in-depth assessment of pharmacists' attitudes toward using an AI-based CDSS tool to enhance vancomycin dosing for hospitalized and critically ill patients. While the clinical providers expressed significant interest in the models, most did not accept changing their dose calculation results when their dose calculation differed from the model's recommended dose.
Despite vancomycin's vital role in treating MRSA infections, a consensus has not been reached on the optimal dosing regimens and pharmacokinetic/pharmacodynamic goals in critically ill patients . Given those critically ill patients in ICUs have varying levels of organ function, therapeutic drug monitoring should be considered to achieve pharmacokinetic goals [27, 28]. Pharmacists spend considerable time reviewing and changing the dosing scheme of vancomycin for patients daily . Our study showed that different pharmacists use different formulae to prescribe vancomycin, resulting in variability in dosing schemes, even though they all consider age, weight, GFR, medical history, and fluid status in their calculations. Additionally, Flannery et al. conducted a recent survey of vancomycin dosing practices among critical care pharmacists . The authors revealed that while pharmacists largely adhere to the 2009 vancomycin dosing guidelines by the American Society of Health-System Pharmacists, the Infectious Diseases Society of America, and the Society of Infectious Diseases Pharmacists [23, 31], they often deviate from these guidelines regarding loading doses, dosing based on patients' weight, and systematically monitoring patients for nephrotoxicity. Developing a vancomycin dosing regimen for critically ill ICU patients is a complicated task involving several variables and often pharmacist judgment calls.
AI is rapidly advancing the field of medicine, which has become a subject of great interest and intense debate. Significant developments based on real-time prediction of adverse events (patient deterioration, acute kidney injury, etc.), personalized drug dosing (antibiotics, etc.), and virtual scribes to help manage patient notes are being implemented . These technologies promise to improve decision-making processes, create more personalized approaches, boost workflow, and reduce healthcare expenditures [7,8,9, 33, 34]. In the ICU, applications of AI include a collection of data analytics and modeling techniques aimed at generating knowledge from data . We developed an AI-based CDSS tool to address the inconsistent dosing of vancomycin regimen and decrease the incidence of vancomycin-associated nephrotoxicity while maintaining its clinical efficacy.Footnote 1
There are several technological challenges to overcome in building these complex models and turning them into bedside tools, including but not limited to preparing curated real-time databases, handling interoperability issues, and managing incomplete and spurious data artifacts. Medicine is a high-risk environment where deploying inaccurate or poorly calibrated algorithms would be unacceptable . The successful deployment of AI requires the trust and buy-in of all stakeholders . Clinicians are primarily interested in scrutinizing the accuracy of the recommended dosing regimen, the rationale, and the specificity of the recommendation for each patient vs. recommendations given for the entire population . Additionally, as noted by our study participants, the user interface will play a role in integrating CDSS into their workflow. In CDSS development, understanding the attitudes and perceptions of those who will be the primary users is critical. Understanding their hesitancies in incorporating AI into their workflow will inform future iterations of AI for higher clinical compliance and clinician acceptance.
Our interviews with pharmacists who provided vancomycin dosing regimens for patients showed positive attitudes toward the AI as a second opinion and a resource that offers reasons for double-checking their decisions. Still, many pharmacists held distrust, leading to low compliance with the recommendations. The primary reason for low compliance was the black-box nature of CDSS recommendations. In addition, the complexity of some AI algorithms, their lack of transparency, and a widespread lack of prospective validation may lead to lower trust and enhanced concerns . Our study provides valuable insights into improvements that would potentially increase the compliance of pharmacists with the vancomycin-dosing CDSS recommendations. In addition, our findings could be extrapolated to other areas, such as renally eliminated and nephrotoxic drugs.
This study had several limitations. First, our results may not be transferrable considering the small number of participating pharmacists from a single large academic medical center. Second, some interview questions could lead the clinicians and result in biased responses. Third, while the clinicians were informed about the very good overall performance of the AI tool, detailed information regarding the model's features was not shared with them. This notion could have impacted their trust in the model's output, particularly if they disagreed with the model’s recommended dosage.
While pharmacists acknowledge the advantages of AI, they prefer AI as a supportive tool rather than a decision-maker. AI may have a role in improving their workflow and instilling more support into their practice, but a lack of trust in the AI recommendations could potentially hinder improving compliance. Therefore, to enhance clinicians' compliance with the CDSS tools regarding drug dosing, we suggest the integration of AI into electronic health records and workflow and to improve its transparency regarding its features and performance accuracy. Change management strategies to transform culture would be the next steps toward higher compliance with these digital health solutions in critical care settings.
Availability of data and materials
The script and supplementary materials provide a copy of the verbal consent script and summarize interview data.
The publication regarding the performance of this model is under review.
Clinical decision support system
Chronic Kidney Disease Epidemiology Collaboration
Estimated glomerular filtration rate
Electronic health record
Intensive care unit
Modification of Diet in Renal Disease
Methicillin-resistant Staphylococcus aureus
Every 8 h
Vancomycin associated Acute Kidney Injury
Liu J, Kong X, Xia F, Bai X, Wang L, Qing Q, Lee I. Artificial Intelligence in the 21st Century, vol. 6. IEEE Access. 2018.
Mintz Y, Brodie R. Introduction to artificial intelligence in medicine. Minim Invasive Ther Allied Technol. 2019;28(2):73–81.
Komorowski M. Artificial intelligence in intensive care: are we there yet? Intensive Care Med. 2019;45(9):1298–300.
Miller DD, Brown EW. Artificial Intelligence in Medical Practice: The Question to the Answer? Am J Med. 2018;131(2):129–33.
Patel VL, Shortliffe EH, Stefanelli M, Szolovits P, Berthold MR, Bellazzi R, Abu-Hanna A. The coming of age of artificial intelligence in medicine. Artif Intell Med. 2009;46(1):5–17.
Gutierrez G. Artificial Intelligence in the Intensive Care Unit. Crit Care. 2020;24(1):101.
Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019;25(1):44–56.
Ghassemi M, Celi LA, Stone DJ. State of the art review: the data revolution in critical care. Crit Care. 2015;19(1):118.
Ince C. Intensive care medicine in 2050: the ICU in vivo. Intensive Care Med. 2017;43(11):1700–2.
Bleher H, Braun M. Diffused responsibility: attributions of responsibility in the use of AI-driven clinical decision support systems. AI Ethics. 2022;2(4):747–61.
Patel S, Preuss CV, Bernice F. Vancomycin. In: StatPearls. Treasure Island (FL): StatPearls Publishing; 2023.
Moellering RC Jr. Vancomycin: a 50-year reassessment. Clin Infect Dis. 2006;42(Suppl 1):S3-4.
van Hal SJ, Paterson DL, Lodise TP. Systematic review and meta-analysis of vancomycin-induced nephrotoxicity associated with dosing schedules that maintain troughs between 15 and 20 milligrams per liter. Antimicrob Agents Chemother. 2013;57(2):734–44.
Hanrahan TP, Harlow G, Hutchinson J, Dulhunty JM, Lipman J, Whitehouse T, Roberts JA. Vancomycin-associated nephrotoxicity in the critically ill: a retrospective multivariate regression analysis*. Crit Care Med. 2014;42(12):2527–36.
Cong Y, Yang S, Rao X. Vancomycin resistant Staphylococcus aureus infections: A review of case updating and clinical features. J Adv Res. 2020;21:169–76.
Jeurissen A, Sluyts I, Rutsaert R. A higher dose of vancomycin in continuous infusion is needed in critically ill patients. Int J Antimicrob Agents. 2011;37(1):75–7.
Roberts JA, Lipman J, Blot S, Rello J. Better outcomes through continuous infusion of time-dependent antibiotics to critically ill patients? Curr Opin Crit Care. 2008;14(4):390–6.
Hirai K, Ishii H, Shimoshikiryo T, Shimomura T, Tsuji D, Inoue K, Kadoiri T, Itoh K. Augmented Renal Clearance in Patients With Febrile Neutropenia is Associated With Increased Risk for Subtherapeutic Concentrations of Vancomycin. Ther Drug Monit. 2016;38(6):706–10.
De Corte T, Elbers P, De Waele J. The future of antimicrobial dosing in the ICU: an opportunity for data science. Intensive Care Med. 2021;47(12):1481–3.
Imai S, Takekuma Y, Miyai T, Sugawara M. A New Algorithm Optimized for Initial Dose Settings of Vancomycin Using Machine Learning. Biol Pharm Bull. 2020;43(1):188–93.
Huang X, Yu Z, Wei X, Shi J, Wang Y, Wang Z, Chen J, Bu S, Li L, Gao F, et al. Prediction of vancomycin dose on high-dimensional data using machine learning techniques. Expert Rev Clin Pharmacol. 2021;14(6):761–71.
O’Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med. 2014;89(9):1245–51.
Rybak MJ, Le J, Lodise TP, Levine DP, Bradley JS, Liu C, Mueller BA, Pai MP, Wong-Beringer A, Rotschafer JC, et al. Therapeutic monitoring of vancomycin for serious methicillin-resistant Staphylococcus aureus infections: A revised consensus guideline and review by the American Society of Health-System Pharmacists, the Infectious Diseases Society of America, the Pediatric Infectious Diseases Society, and the Society of Infectious Diseases Pharmacists. Am J Health Syst Pharm. 2020;77(11):835–64.
EL Wilson J. Mayo Clinic Antimicrobial Therapy: Quick Guide. 2nd ed. New York, NY: Oxford University Press Inc; 2012.
Virginia Braun VC. Using thematic analysis in psychology. Qual Res Psychol [Internet]. 2006;2006(3):77–101.
Giuliano C, Haase KK, Hall R. Use of vancomycin pharmacokinetic-pharmacodynamic properties in the treatment of MRSA infections. Expert Rev Anti Infect Ther. 2010;8(1):95–106.
van der Veen A, Somers A, Vanhaesebrouck S, Ter Heine R, Brüggemann R, Allegaert K, De Cock P. Variation in vancomycin dosing and therapeutic drug monitoring practices in neonatal intensive care units. Int J Clin Pharm. 2022;44(2):564–9.
Mabilat C, Gros MF, Nicolau D, Mouton JW, Textoris J, Roberts JA, Cotta MO, van Belkum A, Caniaux I. Diagnostic and medical needs for therapeutic drug monitoring of antibiotics. Eur J Clin Microbiol Infect Dis. 2020;39(5):791–7.
Gagnon DJ, Roberts R, Sylvia L. Implementation of the systems approach to improve a pharmacist-managed vancomycin dosing service. Am J Health Syst Pharm. 2014;71(23):2080–4.
Flannery AH, Hammond DA, Oyler DR, Li C, Wong A, Smith AP, Yeo QM, Chaney W, Pfaff CE, Plewa-Rusiecki AM, et al. Vancomycin Dosing Practices among Critical Care Pharmacists: A Survey of Society of Critical Care Medicine Pharmacists. Infect Dis (Auckl). 2020;13:1178633720952078.
Rybak MJ, Lomaestro BM, Rotschafer JC, Moellering RC, Craig WA, Billeter M, Dalovisio JR, Levine DP. Vancomycin therapeutic guidelines: a summary of consensus recommendations from the infectious diseases Society of America, the American Society of Health-System Pharmacists, and the Society of Infectious Diseases Pharmacists. Clin Infect Dis. 2009;49(3):325–7.
Price WN 2nd, Gerke S, Cohen IG. Potential Liability for Physicians Using Artificial Intelligence. JAMA. 2019;322(18):1765–6.
Bailly S, Meyfroidt G, Timsit JF. What’s new in ICU in 2050: big data and machine learning. Intensive Care Med. 2018;44(9):1524–7.
Beam AL, Kohane IS. Translating Artificial Intelligence Into Clinical Care. JAMA. 2016;316(22):2368–9.
Gottesman O, Johansson F, Komorowski M, Faisal A, Sontag D, Doshi-Velez F, Celi LA. Guidelines for reinforcement learning in healthcare. Nat Med. 2019;25(1):16–8.
Zhang Y, Liao QV, Bellamy RKE. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. Barcelona: Association for Computing Machinery; 2020. p. 295–305.
This work was supported by Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, MN, USA, grant number J011972. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript.
Ethics approval and consent to participate
This study was deemed exempt by the institutional review board of Mayo Clinic, Rochester, MN (#19–010472). All participants provided verbal consent to contribute to the interview. The Mayo Clinic, Institutional Review Board, agreed to obtain verbal instead of oral contrast as the study carried minimal risk based on the assessment of the IRB. The verbal consent script (provided as supplementary material) was read to the participants, and they only entered the interviews when they agreed with the content of the verbal consent. Therefore, pharmacists who rejected the verbal consent (all pharmacists who were approached agreed to contribute) did not participate in the study.
Consent for publication
None of the authors had any conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Liu, X., Barreto, E.F., Dong, Y. et al. Discrepancy between perceptions and acceptance of clinical decision support Systems: implementation of artificial intelligence for vancomycin dosing. BMC Med Inform Decis Mak 23, 157 (2023). https://doi.org/10.1186/s12911-023-02254-9
- Artificial intelligence
- Qualitative study
- Implementation science
- Acute kidney injury
- Drug dosing