Skip to main content

End-users feedback and perceptions associated with the implementation of a clinical-rule based Check of Medication Appropriateness service



To support appropriate prescribing hospital-wide, the ‘Check of Medication Appropriateness’ (CMA) service was implemented at the University Hospitals Leuven. The CMA concerns a clinical rule based and pharmacist-led medication review service. The aim of this study was to explore both physicians’ and pharmacists’ feedback on the optimised CMA service to further improve the service.


An anonymous e-questionnaire was sent to all physicians active in the University Hospitals Leuven (n = 1631) and to all clinical pharmacists performing the CMA service (n = 16). Feedback was collected using multiple choice questions. During a 5-month period, physicians were also contacted in case of non-acceptance of recommendations to investigate barriers affecting implementation. Thematic analysis was performed and additional acceptance after telephone contact within 24 h was registered.


A total of 119 physicians (7.3%) and 16 pharmacists (100%) completed the e-questionnaire. The overall service was assessed as clinically relevant to highly relevant by 77.7% of physicians. The main reasons for non-acceptance of recommendations were related to workload, work environment and time constraints. About two thirds (66.3%) of initially not-accepted recommendations were accepted after phone contact. A nearly full consensus was reached among pharmacists (15/16) on the centralised CMA being complementary to current clinical pharmacy activities. Two major barriers were reported by pharmacists: (1) too limited time allocation and (2) a large number of irrelevant alerts.


The CMA was perceived as clinically relevant by the majority of end-users. Acceptance rate of pharmaceutical recommendations was further increased by calling the physician. Increasing the specificity of clinical rules in the future is imperative.

Peer Review reports


Inappropriate prescribing is common and may lead to adverse drug events, drug-related hospital (re)admissions and increased healthcare costs [1]. Several interventions have already been identified to promote appropriate prescribing and avoid the associated iatrogenic burden [2]. Software support is used increasingly to manage the large amounts of patient data, and to optimise medication safety. In this regard, major emphasis has been placed on the use of clinical decision support systems (CDSSs) [3].

CDSSs have already proven their efficacy in improving prescribing practices [3]. For instance, a Dutch interrupted time series study showed that a CDSS reduced the incidence of medication errors significantly [4]. In an RCT involving 1278 patients with chronic kidney disease, subjects were randomized to prescribers who had or did not have access to a CDSS tool. Prescribing orders were appropriately changed in 17 versus 5.7% of the time in the intervention arm and control arm, respectively (OR 1.89, 95% CI 1.45–2.47) [5]. However, alert-based CDSSs have also frequently demonstrated inconclusive effects due to low uptake and limited engagement by clinicians [3, 6]. A recent systematic review revealed that most often reported barriers for CDSS adoptation according to physicians are related to a lack of usefulness and relevance of information, and ease of use and efficiency of the system [7]. To better impact prescribing, more specific and clinically relevant approaches have been recommended. Redirecting alerts to clinical pharmacists—prior to alerting prescribers—can reduce alert fatigue and increase downstream uptake. Clinical pharmacists may be more willing to use computerised tools to optimise medication safety [8, 9].

Hence, in 2016, we designed a centralised pharmacy-led alert service to reduce potentially inappropriate prescriptions (PIPs) in inpatients at the University Hospitals Leuven (UZ Leuven): the Check of Medication Appropriateness (CMA). The CMA has been fully implemented in daily practice and currently comprises both a clinical rule-based screening for PIPs, drawing from the available information in the electronic health record (EHR), as well as a medication review performed by clinical pharmacists. Clinical rules are derived from guideline recommendations and then validated by a local multidisciplinary expert panel [10, 11]. In a first stand-alone prototype of the CMA, EHRs were screened once daily and generated alerts were listed in a non-integrated Microsoft Access database. Preliminary results showed that during an 18-month study period, 39,481 clinical rule alerts were reviewed by pharmacists for which 3205 (8.1%) recommendations were formulated. An acceptance rate by physicians of 69.4% was obtained [11].

Optimisation of the service then led to the development of the current CMA software, which is fully integrated into the EHR. This provides the opportunity to directly gather and combine information from all the databases in the EHR (e.g. medication data, laboratory values, demographics, microbiology reports, etc.). The rule-based screening now runs continuously on real-time EHR-extracted patient data and is able to discern time-dependent changes, such as those of biochemical values. Clinical rules were further reformulated and fine-tuned with the inclusion of more alert criteria [10, 12, 13]. Since January 2019, a wider range of clinical rules were implemented, specifically focusing on analgesic prescribing (e.g. screening for opioid-induced constipation) [12], antimicrobial stewardship (e.g. screening for treatment omission of Clostridioides difficile-associated diarrhoea) [13], intravenous (IV) to oral switch therapy [14], anticoagulant therapy (e.g. screening for inappropriate dosing of non-vitamin K antagonist oral anticoagulants) [15] and stroke prevention in atrial fibrillation patients [16]. From January 2019 to December 2020, 82,456 clinical rule alerts were reviewed and 12,688 (15.4%) recommendations were provided with an acceptance rate of 77.7%.



Despite the obvious increase in the recommendation rate and increase in acceptance rate, there is still ample opportunity left to further improve the performance of the service. Therefore, the goal of this study was to obtain insights in end-users perceptions and satisfaction of the overall CMA service and to identify barriers affecting physicians’ acceptance of pharmacist recommendations.

Study design and setting

A mixed-methods cross-sectional study applying survey and interview methodologies was conducted at UZ Leuven, a 1950-bed tertiary care hospital in Belgium. The CMA is a hospital-wide, centralised clinical service, provided to all non-critically ill inpatients [11]. Funding is currently provided for a 0.6 full time equivalent (FTE) service. The CMA is performed each afternoon from Monday until Saturday by a trained clinical pharmacist. The pharmacist performs a medication review of selected drug therapies or indications, based on clinical rules which provide the pharmacist with an overview of clinical rules alerts of patients that are most at risk for predefined PIPs. Each clinical rule alert is provided with a validated flowchart (an example is provided in Additional file 1: Fig. S1) to support clinical pharmacists when performing medication reviews. For clinical relevant alerts, the pharmacist adds a note directly in the EHR addressing the treating physician based on predefined, but adaptable, recommendations. Acceptance of recommendations is assessed by reviewing the EHR.

Data collection

The data for this study was collected in three phases. First, an anonymous e-survey was conducted to evaluate physicians’ feedback and satisfaction. The e-questionnaire was developed using Google Forms and sent via e-mail to all staff physicians and physicians in training who were active in UZ Leuven (n = 1631). The questionnaire was sent on June 24th and terminated on July 26th 2020. A reminder was sent after two weeks. The content of the survey was developed by two clinical pharmacists (CQ, IS) (Additional file 2: Table S1). Six multiple choice questions were drafted, with the aim of gathering information on: (1) knowledge of the service; (2) relevance of the overall service; (3) relevance of the recommendations for IV to oral switch; (4) relevance of the content of the clinical rules; (5) preferred communication method to provide the pharmacist recommendations and 6) the quality of the recommendations. Questions 2, 3 and 6 were scored on a 4-point Likert scale (ranging from very irrelevant to very relevant for questions 2 and 3; and from inadequate to very good for question 6). Each multiple choice question included an option to provide additional remarks.

Second, during a 5-month period (January 2021–May 2021), physicians were also contacted by phone in case of non-acceptance of pharmacist recommendations within 48 h after the CMA intervention. Physicians were questioned if the recommendation was noticed and if so, about the predominant reason and specific clinical arguments driving their decision making on the recommendation. Notes were taken during these interviews. Furthermore, for each non-accepted recommendation after 48 h, additional acceptance after telephone contact within 24 h was registered. Calling physicians and monitoring the acceptance rate was performed by a clinical pharmacist (CQ).

Third, pharmacists’ satisfaction and perceptions were also evaluated by an e-survey developed via Google Forms. The e-questionnaire was sent via e-mail on June 8th 2021 to all staff clinical pharmacists who routinely performed the CMA up to that point (n = 16). Following themes of the structured questions were defined by two clinical pharmacists (CQ, IS) (Additional file 2: Table S2): satisfaction with the service; feeling competent to perform the service; usability of the integrated worklist; usability, clarity and completeness of the flowcharts; usability of the predefined recommendations; clinical relevance of the decision rules; time allocation and overall organisation of clinical pharmacy activities in UZ Leuven. Pharmacists had the choice to complete the questionnaire anonymously or by name. Answers were collected using multiple choice questions with free-text areas for additional remarks. Pharmacists’ feedback was scored using a 4-point Likert-scale for each question (Additional file 2: Table S2).


Quantitative and qualitative approaches were combined for data analysis. Survey data were retrieved from Google Forms. Descriptive analyses were used to summarise the results of the multiple choice questions of both e-surveys. Categorical data were presented as counts and percentages. Thematic analysis based on an inductive approach was applied for the free-text remarks on the multiple choice questions.

Textual qualitative data gathered by telephone contact with physicians were also analysed using inductive thematic analysis. Answers were grouped and recoded independently by the primary researcher (CQ) and a senior clinical pharmacist (IS) until consensus was reached for the interpretation of all answers. Additional acceptance within 24 h after the phone contact was assessed by reviewing the EHR. Acceptance was defined as a modification of therapy (stop, start or dose correction) or a further follow-up of clinical and/or laboratory parameters.


Prescribers’ perceptions: e-survey

Answers were retrieved from 119 (n = 1631; 7.3%) physicians, comprising 33 staff physicians and 86 physicians in training. In total, 33 different medical disciplines were represented with most of the physicians active on the general internal medicine ward (n = 15; 12.6%). One hundred physicians (84.0%) (i.e. 25/33 (75.8%) staff physicians; 75/86 (87.2%) physicians in training) indicated to be familiar with the CMA service. The following five multiple choice questions (relevance of the overall service; relevance of the recommendations for IV to oral switch; relevance of the content of the clinical rules; preferred communication method to provide the pharmacist recommendations and the quality of the recommendations) were answered by 103, 105, 116, 117 and 102 physicians, respectively.

The overall CMA service and specifically the recommendations for IV to oral switch were assessed relevant to very relevant by 77.7% (80/103) and 66.7% (70/105) of the physicians, respectively. The distribution according to physician’s level of experience is shown in Fig. 1. A main remark, mentioned by eight physicians, was that recommendations based on guidelines do not always take into account the particular clinical and medical context of the patient. Alerts for IV to oral switch therapy were rather considered ‘annoying’ by five physicians. Twenty three participants stated that there is usually a good (clinical/medical) reason to continue IV therapy. One physician, who rated the IV to oral switch service as not relevant, further indicated that such alerts are rather redundant, and that the switch may even take place fully automatically without the physician's permission.

Fig. 1
figure 1

Relevance of a the overall CMA service (n = 103) and b recommendations for IV to oral switch (n = 105), according to the physician’s level of experience. CMA: check of medication appropriateness; IV: intravenous

The clinical topic which was most frequently selected as clinically relevant was the follow-up of drug-drug interactions (92/116; 79%), followed by dose recommendations in patients with abnormal kidney function (87/116; 75%) and dose recommendations for antimicrobial agents (86/116; 74.1%). Five physicians assessed the CMA as not relevant for all the proposed CMA topics (Fig. 2). Suggestions to further expand the CMA service in the near future comprised the inclusion of more clinical rules focusing on the paediatric population (n = 2), recommendations based on the STOPP/START criteria [17] (n = 1), follow-up of patients who receive total parenteral nutrition (n = 1), follow-up of pharmacological management of acute delirium [1] and recommendations on the management of arterial hypertension (n = 1).

Fig. 2
figure 2

Relevance of selected topics focused on in the CMA according to the physicians (n = 116). CMA: check of medication appropriateness; DDIs: drug-drug interactions; IV: intravenous; TDM: therapeutic drug monitoring

Electronic notes were considered the preferred way of providing the recommendations by 59% (69/117) of physicians (Fig. 3) (i.e. 19/32 (59.4%) staff physicians; 50/85 (58.8%) physicians in training). Other proposals for communication included: electronic notes supplemented with telephone contact in case of severe or urgent risks (n = 17), alerts generated directly in the computerised physician order entry upon prescribing (n = 1), electronic notes with the option to remove (n = 1) and direct contact at the ward (n = 1). The quality of the pharmacist recommendations was found to be good to very good according to 91.2% (93/102) of the physicians. Figure 4 shows the response according to the physician’s level of experience.

Fig. 3
figure 3

Preferred communication method to provide the pharmacist recommendations according to physicians (n = 117)

Fig. 4
figure 4

Quality of provided pharmacists recommendations according to the physician’s level of experience (n = 102)

Prescribers’ perceptions: non-accepted recommendations

During a 5-month period, treating physicians were called to discuss non-acceptance of 209 PIPs. This was limited to contact with the physicians in training as they are almost exclusively responsible for prescribing. Most phone calls were performed for persistent PIPs identified in patients admitted at the abdominal surgery ward (11.0%) and pulmonology ward (9.1%). Four barriers were predominant in influencing acceptance (Fig. 5): (1) workload, work environment and time constraints (n = 132); (2) presence of an additional clinical factor that was considered relevant (n = 30; Table 1); (3) low clinical relevance of recommendations (n = 16) and (4) reluctance towards altering prescriptions initiated by another physician (n = 11). Reasons related to the hospital environment mainly included simply not noticing the recommendation in the EHR, lack of time, new patients and forgetting to adjust therapy.

Fig. 5
figure 5

Reasons for not accepting the pharmacist recommendation (n = 209). CG: Cockcroft-Gault; CKD-EPI: Chronic Kidney Disease Epidemiology Collaboration; ECG: electrocardiogram; IV: intravenous; SmPC: summary of product characteristics; UZ Leuven: University Hospitals Leuven

Table 1 Clinical reasons for non-acceptance of pharmacist recommendations mentioned by treating physicians

Follow-up after the phone contact revealed an additional acceptance of the CMA recommendations in 66.3% (124/187) of the persistent PIPs. The distribution of acceptance after telephone contact according to the different physician’s reason for initial non-adherence is shown in Fig. 6.

Fig. 6
figure 6

Acceptance of the recommendations by physicians after initial non-acceptance and additional telephone contact (n = 187). CG: Cockcroft-Gault; CKD-EPI: Chronic Kidney Disease Epidemiology Collaboration; ECG: electrocardiogram; IV: intravenous; SmPC: summary of product characteristics; UZ Leuven: University Hospitals Leuven

Phone calls were made for 23 non-accepted IV to oral switch recommendations. There was no clear reason for non-acceptance in 16 (69.6%) cases. In four patients there was a clinical reason that justified, according to the physician, continuation of IV paracetamol therapy (Table 1). In two patients the IV therapy was continued because they preferred to allow nurses to administer paracetamol IV overnight without disturbing the patient. For one patient, the physician in training stated that this was done specifically at the request of the supervisor, without additional information. However, additional telephone contact resulted eventually in IV to oral switch in 86.4% (19/22) of the initially non-accepted recommendations.

Pharmacists’ perceptions: e-survey

All 16 staff clinical pharmacists completed the e-questionnaire (100%) by name. Satisfaction with the CMA service was very high: 62.5% of pharmacists were very satisfied, 31.3% were satisfied. One pharmacist (6.3%) was dissatisfied due to the high rate of irrelevant alerts.

All pharmacists indicated to feel competent when performing the CMA (37.5% very competent; 62.5% competent). Certain clinical rules were considered more complex than others, depending on the specialty of the individual pharmacist (n = 5).

Usability of the integrated CMA worklist in the EHR was overall perceived very user-friendly (75% very user-friendly; 18.8% user-friendly; 6.3% not user-friendly). All pharmacists agreed on the usability of the flowcharts (62.5% very user-friendly; 37.5% user-friendly), clarity of the flowcharts (81.3% very clear; 18.8% clear) and usability of the predefined recommendations (56.3% very user-friendly; 43.8% user-friendly). Six pharmacists stated that they usually customise the predefined recommendations based on patient-specific data to provide more individualised messages to the prescriber.

Almost all pharmacists agreed on the completeness of the flowcharts (56.3% very complete; 37.5% complete, 6.3% incomplete). The most important remark was that not all clinical questions and nuances or all clinical exceptions can be captured in a structured flowchart that is applicable for each individual patient. Consequently, in order to competently perform the CMA, pharmacists emphasised the importance of ward-based training in clinical pharmacy (n = 6).

The majority of pharmacists rated the clinical rules as clinically relevant (18.8% very relevant; 62.5% relevant; 18.8% irrelevant). Some pharmacists scored this item as less relevant because they questioned the clinical relevance of some specific rules (i.e. screening for opioid-induced constipation (n = 6), QTc prolongation (n = 3), opioid-induced nausea (n = 1), incorrect dosing of paracetamol (n = 1), overridden very severe drug-drug interactions (n = 1), intake of rivaroxaban 15 mg/20 mg without food (n = 1) and IV to oral switch therapy (n = 1)).

None of the pharmacists agreed that the time commitment of the 0.6 FTE mandate is sufficient (62.5% completely disagree; 37.5% not agree). Statements on the limited time allocation were further supplemented with remarks on the high rate of clinically irrelevant alerts (n = 4).

With the exception of a single respondent, all clinical pharmacists agreed that the centralised CMA service is complementary to the current bedside clinical pharmacy activities (37.5% completely agree; 56.3% agree; 6.3% not agree).


In order to evaluate and further improve the CMA service, feedback was gathered from both physicians and pharmacists. The overall service was scored positively by all respondents. Evaluation of non-accepted recommendations revealed the main reasons for non-adherence and highlighted the added value of contacting physicians by phone.

The majority of physicians assessed the overall CMA service as clinically relevant, which translated into a similar acceptance rate of pharmacists recommendations in daily practice. Furthermore, the quality of the pharmacotherapeutic recommendations, provided as an electronic note directly in the patient’s EHR, was considered as good to very good by more than 90% of physicians. Conversely, eight (6.7%) physicians specifically referred to the suboptimal wording of the recommendations, which were considered as being insufficiently tailored to the individual patient. The risk of alerts being irrelevant for the specific clinical patient context was already highlighted in other studies evaluating the uptake of alerts [18,19,20]. In the CMA, predefined recommendations are formulated for each clinical rule. Yet, before sending an alert to the physician, the recommendation can be adapted by adding patient-specific data to generate more individualised messages, and subsequently to improve downstream uptake. Efforts should be continued to further improve this.

The perception of the relevance of recommendations for IV to oral switch was found different according to the physician’s level of experience (Fig. 1). Twenty-three of the 35 physicians who rated the switch recommendations as irrelevant, stated that there is usually a (clinical/medical) reason to continue IV therapy. Unfortunately, the study of the non-accepted switch recommendations could only identify three specific clinical reasons. Two of these, i.e. the degree of pain and fever, are not included in the clinical rule or medication review. The development of the IV to oral switch clinical rules was however based on a definite set of 13 switch criteria, obtained by literature search and validated by a multidisciplinary expert panel [14]. Specific clinical criteria such as pain, temperature, blood leucocyte count, C-reactive protein, respiratory rate, heart rate and blood pressure were all excluded from the decision algorithm for IV to oral switch. Furthermore, we already showed the effectiveness of the switch recommendations in an interrupted time series analysis. The implementation of IV to oral switch clinical rules in the CMA showed a relative reduction of 79% (p < 0.0001) in residual potentially inappropriate IV prescriptions [14].

Likewise, less than 40% of physicians agreed on the relevance for follow-up of pain therapy and also fewer physicians agreed on the relevance of topics related to monitoring antimicrobial therapy. This is surprising, as this is in clear contrast to the results observed in our interrupted time series analyses evaluating the impact of clinical rules focusing on analgesic prescribing and antimicrobial stewardship. In these studies, we showed that the CMA approach reduced the number of pain-related and antimicrobial stewardship-related residual PIPs in a significant and sustained manner by 66% [12] and 86.7% [13], respectively.

In the CMA, pharmaceutical recommendations are mainly communicated to physicians via an electronic note in the EHR. For (very) severe or urgent risks, the physician is also called by phone. Based on the survey, this approach appeared to be the preferred way according to physicians. Previous studies already showed however that verbally communicated pharmaceutical recommendations are more likely to be accepted [21]. Despite the already high acceptance rate of 77.7%, we also showed an additional acceptance of 66.3% after contacting the physician by phone for initially non-accepted recommendations. This means that the overall acceptance rate could be increased from 77.7 to 92.5%. Oral communication allows the pharmacist on the one hand to alert the physician for a message that may have been overlooked or forgotten and, on the other hand, to provide more (clinical) information about the content and relevance of the advice or to gain insights in the clinical reasoning behind the non-acceptance. In the future, it is therefore recommended to systematically call the physician in case of non-adherence to improve the downstream uptake of alerts. This is further corroborated by the high percentage of non-acceptance purely as a result of workload, work environment and time constraints. Overall, four key barriers [(1) the hospital environment; (2) presence of an additional clinical factor that was considered relevant; (3) low clinical relevance of recommendations and (4) reluctance towards altering prescriptions initiated by another physician] were predominant in influencing prescribers’ acceptance of recommendations. A previous investigation by Dalton et al. already highlighted the same factors affecting prescriber implementation of SENATOR recommendations. The authors also concluded that prescribing advice should be combined with an additional face-to-face interaction with expert physicians or pharmacists to be effective [20].

The presence of an additional clinical reason considered as relevant by the physician was the second most common reason for not accepting the recommendations. This indicates that physicians may consider other factors beyond those included in the clinical rule and medication review. Consequently, some clinical rules may need to be revised, for instance by including thrombopenia and body weight when considering dose recommendations for anticoagulants and antibiotics, respectively (Table 1). On the other hand, not all of these clinical arguments were justified or based on evidence (e.g. the presence of arteriovenous malformation for dose reductions of anticoagulants). In addition, these clinical arguments might also be underreported in the EHR or even be completely missed during the review (e.g. presence of bleeding, difficulties with oral intake). This rather further supports the idea of systematically discussing the pharmaceutical recommendation with the prescriber by phone in case of non-acceptance.

Pharmacists agreed on the usefulness of the standardised flowcharts as these provide a practical guide for medication review. Importantly, since not every clinical nuance can be included in a flowchart, minimal clinical skills remain mandatory. This should be further addressed through education and training sessions. Until now, only an introductory session is provided [11]. Regularly recurring educational sessions with a focus on new rules, changes in clinical rules and complex patient cases could add value.

All pharmacists concluded that 0.6 FTE, which is currently foreseen to run the service, should be expanded in the future. Especially given the need to provide more individualised recommendations and the need to systematically call the physician in case of non-acceptance. Furthermore, a major obstacle cited by the pharmacists is the large number of false positive and thus irrelevant alerts. This problem is mainly a result of technical limitations (e.g. repeated alerts, therapy was already adjusted or stopped) and the lack of structural digitalisation of patients’ data in the EHR (e.g. health diagnoses were only mentioned in unstructured free text fields). Our ratio of recommendations (12,688 recommendations/82,456 clinical rule alerts; 15.4%) is however similar or even higher compared with results obtained in previous studies investigating pharmacy-focused alerts [22,23,24,25]. Furthermore, numerous studies have already described that patient problem lists in EHRs are often incomplete or out of date, making CDSSs less effective [26, 27]. As the pharmacists had dedicated time to perform the CMA, as opposed to alerts interrupting the many tasks of prescribing physicians, the risk of alert fatigue was expected to be lower. Although, increasing specificity of clinical rules in the future is imperative. This depends on further structural digitalisation of patients’ data in the EHR and technical operationalisation. With regard to the latter, machine learning-based methods might be a valuable option to analyse unstructured clinical data in the EHR [28, 29].

The main limitation of our study is that the surveys and interviews were not subjected to a content validation prior to sending to participants. Also, the construct and analysis was not grounded on a theoretical framework. To further assess possible additional determinants and/or challenges affecting adoptation of CMA recommendations in clinical practice, it might be recommendable to apply a theoretical adoptation model (e.g. Technology Acceptance Model [30], the Clinical Adoptation Meta-Model [31] or Human, Organization and Technology-fit (HOT-fit) model [7, 32] for analysis). Furthermore, only a sample of physicians responded, based on voluntary and anonymous participation, so self-selection bias is not excluded. This also limits the generalisability of the findings and may possibly explain the observed differences with the results of the ITS analyses. Next, only one reminder was sent to the physicians; participation rates could have been higher with one additional reminder and an extended evaluation period. Lastly, there was a year between the survey of physicians and the survey of pharmacists due to practical considerations. As a result, pharmacists have appraised a more recent version of the service. However, this may not be of great importance as very few adjustments were implemented during that year.


In summary, the overall CMA service was perceived as clinically relevant by the majority of physicians. Acceptance rate of recommendations was further increased by calling the physician. Satisfaction among pharmacists was also very high. However, the current time allocation of a 0.6 FTE mandate for a large teaching hospital is too limited. Further expansion of the mandate and improving the specificity of clinical rules is therefore urgently needed. Furthermore, we need to re-evaluate and possibly adjust some clinical rules in the future.

Availability of data and materials

All data generated or analysed during this current study are included in this published article.



Clinical decision support system




Chronic Kidney Disease Epidemiology Collaboration


Check of Medication Appropriateness


Creatinine clearance


Drug-drug interaction




Electronic health record


Full time equivalent




Low molecular weight heparin


Potentially inappropriate prescription


Software ENgine for the Assessment and optimisation of drug and non-drug Therapy in Older peRsons


Summary of product characteristics


Screening Tool of Older People’s Prescriptions/Screening Tool to Alert to Right Treatment


Therapeutic drug monitoring

UZ Leuven:

University Hospitals Leuven


  1. Avery AJ, Coleman JJ. Tackling potentially inappropriate prescribing. BMJ. 2018;363(November):1–2.

    Google Scholar 

  2. Guchelaar HJ, Colen HBB, Kalmeijer MD, Hudson PTW, Teepe-Twiss IM. Medication errors hospital pharmacist perspective. Drugs. 2005;65(13):1735–46.

    Article  Google Scholar 

  3. Scott IA, Pillans PI, Barras M, Morris C. Using EMR-enabled computerized decision support systems to reduce prescribing of potentially inappropriate medications: a narrative review. Ther Adv Drug Saf. 2018;12(9):559–73.

    Article  Google Scholar 

  4. van Doormaal JE, van den Bemt PMLA, Zaal RJ, Egberts ACG, Lenderink BW, Kosterink JGW, et al. The influence that electronic prescribing has on medication errors and preventable adverse drug events: an interrupted time-series study. J Am Med Inform Assoc. 2009;16(6):816–25.

    Article  Google Scholar 

  5. Awdishu L, Coates CR, Lyddane A, Tran K, Daniels CE, Lee J, et al. The impact of real-time alerting on appropriate prescribing in kidney disease: a cluster randomized controlled trial. J Am Med Inform Assoc. 2016;23(3):609–16.

    Article  Google Scholar 

  6. Ibáñez-Garcia S, Rodriguez-Gonzalez C, Escudero-Vilaplana V, Martin-Barbero ML, Marzal-Alfaro B, De La Rosa-Triviño JL, et al. Development and evaluation of a clinical decision support system to improve medication safety. Appl Clin Inform. 2019;10(3):513–20.

    Article  Google Scholar 

  7. Westerbeek L, Ploegmakers KJ, de Bruijn GJ, Linn AJ, van Weert JCM, Daams JG, et al. Barriers and facilitators influencing medication-related CDSS acceptance according to clinicians: a systematic review. Int J Med Inform. 2021;152(February):104506.

    Article  Google Scholar 

  8. Robertson J, Walkom E, Pearson S-A, Hains I, Williamsone M, Newby D. The impact of pharmacy computerised clinical decision support on prescribing, clinical and patient outcomes: a systematic review of the literature. Int J Pharm Pract. 2010;18(2):69–87.

    PubMed  Google Scholar 

  9. Verdoorn S, Kwint HF, Hoogland P, Gussekloo J, Bouvy ML. Drug-related problems identified during medication review before and after the introduction of a clinical decision support system. J Clin Pharm Ther. 2018;43(2):224–31.

    Article  CAS  Google Scholar 

  10. Quintens C, Van de Sijpe G, Van der Linden L, Spriet I. Computerized prescribing support still needs a human touch. Age Ageing. 2021;50(6):e3–4.

    Article  Google Scholar 

  11. Quintens C, De Rijdt T, Van Nieuwenhuyse T, Simoens S, Peetermans WE, Van Den Bosch B, et al. Development and implementation of “check of Medication Appropriateness” (CMA): advanced pharmacotherapy-related clinical rules to support medication surveillance. BMC Med Inform Decis Mak. 2019;19(1):1–10.

    Article  Google Scholar 

  12. Quintens C, De Coster J, Van der Linden L, Morlion B, Nijns E, Van den Bosch B, et al. Impact of Check of Medication Appropriateness (CMA) in optimizing analgesic prescribing: an interrupted time series analysis. Eur J Pain. 2021;25(3):704–13.

    Article  Google Scholar 

  13. Quintens C, Peetermans WE, Lagrou K, Declercq P, Schuermans A, Debaveye Y, et al. The effectiveness of Check of Medication Appropriateness for antimicrobial stewardship: an interrupted time series analysis. J Antimicrob Chemother. 2021;dkab364:1–9.

    Google Scholar 

  14. Quintens C, Coenen M, Declercq P, Casteels M, Peetermans WE, Spriet I. From basic to advanced computerised intravenous to oral switch for paracetamol and antibiotics: an interrupted time series analysis. BMJ Open. 2022;12(4):e053010.

    Article  Google Scholar 

  15. Quintens C, Verhamme P, Vanassche T, Vandenbriele C, Van den Bosch B, Peetermans WE, et al. Improving appropriate use of anticoagulants in hospitalised patients: a pharmacist-led Check of Medication Appropriateness intervention. Br J Clin Pharmacol. 2021;2021:1–10.

    Google Scholar 

  16. Quintens C, Van der Linden L, Meeusen K, Nijns E, Willems R, Spriet I. Development and validation of a screening tool to evaluate and warrant anticoagulation treatment prior to discharge in inpatients with atrial fibrillation (STEWARxD-AF). Int J Med Inform. 2021;154(April):104555.

    Article  Google Scholar 

  17. O’mahony D, O’sullivan D, Byrne S, O’connor MN, Ryan C, Gallagher P. STOPP/START criteria for potentially inappropriate prescribing in older people: version 2. Age Ageing. 2015;44(2):213–8.

    Article  Google Scholar 

  18. Catho G, Centemero NS, Catho H, Ranzani A, Balmelli C, Landelle C, et al. Factors determining the adherence to antimicrobial guidelines and the adoption of computerised decision support systems by physicians: a qualitative study in three European hospitals. Int J Med Inform. 2020;141(January):104233.

    Article  Google Scholar 

  19. Tolley CL, Slight SP, Husband AK, Watson N, Bates DW. Improving medication-related clinical decision support. Am J Health Pharm. 2018;75(4):239–46.

    Article  Google Scholar 

  20. Dalton K, O’Mahony D, Cullinan S, Byrne S. Factors affecting prescriber implementation of computer-generated medication recommendations in the SENATOR trial: a qualitative study. Drugs Aging. 2020;37(9):703–13.

    Article  Google Scholar 

  21. Bedouch P, Tessier A, Baudrant M, Labarere J, Foroni L, Calop J, et al. Computerized physician order entry system combined with on-ward pharmacist: analysis of pharmacists’ interventions. J Eval Clin Pract. 2012;18(4):911–8.

    Article  Google Scholar 

  22. Silverman JB, Stapinski CD, Huber C, Ghandi TK, Churchill WW. Computer-based system for preventing adverse drug events. Am J Heal Pharm. 2004;61(15):1599–603.

    Article  Google Scholar 

  23. Jha AK, Laguette J, Seger A, Bates DW. Can surveillance systems identify and avert adverse drug events? A prospective evaluation of a commercial application. J Am Med Inform Assoc. 2008;15(5):647–53.

    Article  Google Scholar 

  24. Rommers MK, Zwaveling J, Guchelaar HJ, Teepe-Twiss IM. Evaluation of rule effectiveness and positive predictive value of clinical rules in a Dutch clinical decision support system in daily hospital pharmacy practice. Artif Intell Med. 2013;59(1):15–21.

    Article  Google Scholar 

  25. de Wit HAJM, Mestres Gonzalvo C, Cardenas J, Derijks HJ, Janknegt R, van der Kuy PHM, et al. Evaluation of clinical rules in a standalone pharmacy based clinical decision support system for hospitalized and nursing home patients. Int J Med Inform. 2015;84(6):396–405.

    Article  Google Scholar 

  26. Wright A, Pang J, Feblowitz JC, Maloney FL, Wilcox AR, McLoughlin KS, et al. Improving completeness of electronic problem lists through clinical decision support: a randomized, controlled trial. J Am Med Inform Assoc. 2012;19(4):555–61.

    Article  Google Scholar 

  27. Wright A, McCoy AB, Hickman TTT, Hilaire DS, Borbolla D, Bowes WA, et al. Problem list completeness in electronic health records: a multi-site study and assessment of success factors. Int J Med Inform. 2015;84(10):784–90.

    Article  Google Scholar 

  28. Beam AL, Kohane IS. Big data and machine learning in health care. JAMA. 2018;319(13):1317–8.

    Article  Google Scholar 

  29. Corny J, Rajkumar A, Martin O, Dode X, Lajonchère JP, Billuart O, et al. A machine learning-based clinical decision support system to identify prescriptions with a high risk of medication error. J Am Med Inform Assoc. 2020;27(11):1688–94.

    Article  Google Scholar 

  30. Rahimi B, Nadri H, Afshar HL, Timpka T. A systematic review of the technology acceptance model in health informatics. Appl Clin Inform. 2018;9(3):604–34.

    Article  Google Scholar 

  31. Price M, Lau F. The clinical adoption meta-model: a temporal meta-model describing the clinical adoption of health information systems. BMC Med Inform Decis Mak. 2014;14:43.

    Article  Google Scholar 

  32. Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for Health Information Systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform. 2008;77(6):386–98.

    Article  Google Scholar 

Download references


We are very grateful for the dedicated work of the IT department in the development of the CMA service.


This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. Funding was provided by Clinical Research Fund of the University Hospitals Leuven.

Author information

Authors and Affiliations



CQ, WEP and IS substantially contributed to conception of the study. CQ and IS substantially contributed to the study design and results interpretation. The manuscript was drafted by CQ. PD, LVDL and IS critically revised the manuscript for important intellectual content and approved it. WEP and BVDB commented on the manuscript and approved the final version for publication. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Charlotte Quintens.

Ethics declarations

Ethics approval and consent to participate

The study was approved by the institutional and/or national research committee ‘Ethics Committee Research University Hospitals Leuven, Belgium’ (Approval Number S61615). Participation was voluntary. Analysed data were entirely anonymised and included no information that could be used to identify specific participants, and they were guaranteed to no unethical use of the information. Therefore, no consent to participate was required for this study according to the institutional and/or national research committee ‘Ethics Committee Research University Hospitals Leuven, Belgium’. All methods were performed in accordance with the relevant guidelines and the institutional regulations.

Consent for publication

Not applicable.

Competing interests

IS and LVDL are supported by the Clinical Research Fund of the University Hospitals Leuven. The other authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1. Figure S1.

Example of a flowchart for the clinical rule ‘Screening for vancomycin lock therapy’.

Additional file 2. Table S1.

Physicians e-survey. Table S2. Pharmacists e-survey.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Quintens, C., Peetermans, W.E., Van der Linden, L. et al. End-users feedback and perceptions associated with the implementation of a clinical-rule based Check of Medication Appropriateness service. BMC Med Inform Decis Mak 22, 177 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: