Physicians' Perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems
BMC Medical Informatics and Decision Making volume 12, Article number: 111 (2012)
One possible approach towards avoiding alert overload and alert fatigue in Computerized Physician Order Entry (CPOE) systems is to tailor their drug safety alerts to the context of the clinical situation. Our objective was to identify the perceptions of physicians on the usefulness of clinical context information for prioritizing and presenting drug safety alerts.
We performed a questionnaire survey, inquiring CPOE-using physicians from four hospitals in four European countries to estimate the usefulness of 20 possible context factors.
The 223 participants identified the ‘ severity of the effect’ and the ‘clinical status of the patient’ as the most useful context factors. Further important factors are the ‘complexity of the case’ and the ‘risk factors of the patient’.
Our findings confirm the results of a prior, comparable survey inquiring CPOE researchers. Further research should focus on implementing these context factors in CPOE systems and on subsequently evaluating their impact.
Medication errors and adverse drug events
According to estimates from the World Health Organization and the European Commission, approximately 10% of all patients in developed countries are harmed by errors or adverse events during their hospitalization care [1, 2]. Medication-related events are among the most common adverse events , of which, according to the Institute of Medicine, at least 25% are preventable . Moreover, the costs associated with additional hospitalization are high and the economic benefits of improving patient safety are likewise compelling . Therefore, actions to ensure medication safety have become major global public health issues [1, 5].
The Council of Europe defines a medication error as “any preventable event that may cause or lead to inappropriate medication use or patient harm while the medication is in the control of the health care professional, patient or consumer” and an adverse drug event (ADE) as “any injury occurring during the patient’s drug therapy and resulting either from appropriate care, or from unsuitable or suboptimal care”. ADEs related to a medication error are considered preventable .
CPOE systems and alert fatigue
Of the preventable ADEs, 56%-71% occur during drug prescription [7, 8]. Computerized Physician Order Entry (CPOE) systems have the potential to reduce medication error rates as well as ADEs [4, 9–12]. CPOE systems can be combined with computerized decision support (CDS). Kuperman et al. distinguish between basic (e.g. offers drug-drug interaction checks) and advanced decision support (e.g. offers drug-disease contraindication checking) . CPOE systems with advanced CDS have proven to be more effective in decreasing medication errors compared to CPOE systems with basic or without CDS .
However, CPOE systems with CDS tend to produce a high number of drug safety alerts and often suffer from poor signal-to-noise ratios [14–18]. This burden of unspecific alerts for the physicians is called alert overload and can consequently lead to alert fatigue. Alert fatigue is described by van der Sijs as “the mental state that is the result of alerts consuming too much time and mental energy, which can cause relevant alerts to be unjustifiably overridden along with clinically irrelevant ones”. Furthermore, a systematic review from 2006 showed that in 49%-96% of the cases, CPOE/CDS alerts are overridden by the clinicians, which can indirectly impair patient safety .
Contextualization of CPOE alerts
One possible approach to reducing alert fatigue is to filter irrelevant and non-urgent alerts and, furthermore, tailor the alerts to a specific patient . A similar approach was pursued by the European PSIP (Patient Safety through Intelligent Procedures in Medication) project, which aimed to contextualize the delivery of alert information based on antecedent semi-automatic ADE detection using data and semantic mining techniques .
In the course of the PSIP project, Riedmann, Jung et al. propounded a concept of a CPOE system that prioritizes and presents drug safety alerts depending on the clinical context. The underlying concept is based on the definition of context provided by Dey: “Context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves”. Consequently, a context-aware system “uses context to provide relevant information and/or services to the user, where relevancy depends on the user’s task”. Figure 1 schematically illustrates the concept of a context-aware CPOE system that uses information about the context of the clinical situation in order to prioritize alerts and to offer a tiered alert presentation to the user (figure published originally in ).
According to this concept, Riedmann, Jung et al. compiled a set of 20 potential context factors on the basis of a literature search and of qualitative expert interviews. The context factors were then hierarchically structured into three main axes: 1.) Characteristics of the organizational unit; 2.) Characteristics of the patient or case; 3.) Characteristics of the alert (see Figure 2, published originally in ). For definitions and examples of each context factor, see Additional file 1.
To determine which of these context factors were useful to support this concept of context-aware CPOE systems, Riedmann et al. conducted a Delphi study with international CPOE researchers . The obtained results reflect the opinions of publishing CPOE researchers; more than 50% of them held a university perspective and only a quarter of the participants were practicing physicians .
The objective of the present study was to investigate the viewpoints of the clinical CPOE users in a survey, as the results of the Delphi study may not reflect their opinions.
At the moment, most physicians have to deal with often annoying CPOE alerts and mostly do not have experience with contextualized alert presentation. Therefore, it may be difficult for them to foresee which approaches may be useful and which not. Answers in a survey may thus not reflect actual behavior in the future. However, surveys are an important and efficient way to obtain a first understanding of what physicians consider useful to reduce alert overload.
What are the perceptions of physicians on the usefulness of clinical context information for prioritizing and presenting drug safety alerts?
The study was carried out with a convenience sample of European hospitals in Amsterdam, Copenhagen, Denain and Sofia (see Table 1). All hospitals implemented CPOE/CDS systems for prescribing drugs. In Copenhagen, Denain and Amsterdam, commercial CPOE products were currently in use, while the hospital in Sofia developed its own CPOE system together with a commercial vendor. All CPOE systems, except for the CPOE system in Denain, provide automatic alerts to the physicians; this means that alerts are automatically triggered and presented to the user during drug prescription. Physicians in Denain are only alerted optionally if they explicitly ask the system for advice (e.g. by clicking a “check prescription” button). According to the classification provided by Kuperman , all CPOE systems possess basic CDS functionalities. None of the CPOE systems offered advanced CDS functionalities. Table 2 shows the details of the CPOE systems in use.
We performed a one-group cross-sectional quantitative questionnaire survey. We chose an exploratory and observational design to generate new insights into the issue of CPOE alert contextualization from a physician’s point of view.
The study design was presented to the ethics committee at UMIT. A formal approval of the design was not considered necessary by the committee.
To meet the requirements of the local study managers, the questionnaires were either available paper-based or online. Copenhagen, Denain and Sofia wanted to use the paper-based version, whereas Amsterdam opted for the electronic questionnaire. In all hospitals except for Copenhagen, full-sample surveys were conducted. In Copenhagen, due to organizational reasons, we drew a convenience sample of participants. As all three participating hospitals from Copenhagen had been using the same CPOE system for approximately the same time period, the data was aggregated to one single dataset. Table 3 provides further information about the sampling.
At all the hospitals, the physicians were informed beforehand about the objectives of the survey personally or via e-mail. Local coordinators then took charge of the organizational issues (i.e. recruiting and reminding the physicians as well as distributing and collecting the questionnaires). No medical students were addressed. At all the hospitals except for Amsterdam, the physicians were recruited and reminded personally; in Amsterdam this was done electronically via e-mail.
The questionnaire was constructed and then pre-tested with seven hospital doctors from different specialties. It was then translated completely into Bulgarian and French and partly into Danish and Dutch. The questionnaire was then again pre-tested at each hospital with two or three doctors. The survey was conducted between 2010 and 2011.
Methods for data acquisition and measurement
The paper-based and the electronic questionnaire (using LimeSurvey http://www.limesurvey.org) comprised the following parts:
Selecting and ranking of useful context factors
All 20 potential context factors were presented to the physicians. They were asked to mark with a cross those factors which they considered useful to filter irrelevant alerts. The order of the statements was randomized to avoid an unintentional ‘serial position effect’, meaning that the first factors in the list are judged differently than the factors listed further down .
The physicians were then asked to have another look at those factors marked with a cross, to select the five most useful factors and to rank them according to their usefulness.
A double-staged rating process was developed in cooperation with a psychologist responsible for survey management at the University Hospital of Innsbruck:
We also asked the physicians to amend as free-text further factors that they would consider useful for filtering alerts.
We asked the physicians to provide demographic data about their age, sex, professional role, years of work experience and years of experience with CPOE systems.
Both parts of the questionnaire are shown in the additional file 1.
Methods for data analysis
We calculated the frequencies for the number of times each factor was marked as ‘useful’ with a cross and generated a heat map.
Furthermore, we calculated a ranking of the most important context factors. We only took those questionnaires into account in which the physicians had correctly followed the ranking instructions. A factor received between 1 and 5 points based on the rank assigned by each physician (i.e. rank 1 (first most useful) = 5 points; rank 5 (fifth most useful) = 1 point). Points were summed up for every factor per hospital and normalized according to the number of physicians per hospital to account for differences in the sample sizes . To determine the most useful factors, we performed a hierarchical clustering over all hospitals using SPSS v16 (method: average linkage between groups UPGMA - Unweighted Pair Group Method with Arithmetic Mean; distance measure: squared Euclidian distance). To present the outcome, we used a dendrogram.
The answers to the free-text question that were meant to determine further context factors were analyzed by means of quantitative content analysis with inductive category development according to Mayring  using the software tool MaxQDA 2007. Two researchers independently classified the comments as to whether they were already covered by our set of 20 context factors. All comments that suggested new ideas for further factors were discussed in the research team.
A total of 1,380 questionnaires were distributed. 223 completed questionnaires were returned. The return rate was between 7.5% in Amsterdam and 58.5% in Sofia (Table 4). Across all hospitals, the average physician was between 30–39 years old and had worked around 14 years as a doctor. He/she currently fulfilled a professional role at the medium level and had around 4 years of experience with CPOE systems (Table 5).
The heat map in Figure 3 presents the perceived usefulness of the context factors, i.e. the number of physicians who found a context factor would be useful for prioritizing and filtering irrelevant alerts. We added a column for the results from the previous Delphi study, where 69 CPOE researchers were asked about the same matter .
The dendrogram in Figure 4 shows the results of the cluster analysis to determine the most useful context factors for prioritizing and presenting drug alert information as perceived by the physicians. The red and green bars indicate the two clusters that are the most distinct from each other. The green cluster contains the elements with the highest ranking values.
From the 22 free-text comments provided by 17 physicians, no new ideas could be derived for additional factors to be taken into account for the contextualization of CPOE alerts. Three suggestions were already covered by our set of context factors, whereas 19 comments addressed other issues, such as complaints about the current CPOE system at the hospital.
Answer to the study question
Four factors were perceived as useful for filtering irrelevant alerts by a majority of all the participating physicians. According to the clustering of the ranking values, the most useful context factors for prioritizing alerts are ‘ severity of the effect’ and ‘clinical status of the patient’. Further important factors are ‘complexity of the case’ and ‘risk factors of the patient’ (see Figure 4).
Strengths and weaknesses of the study
The survey was carried out at four European hospitals. In general, the response rates were quite high (43%-59%). The return rate in Amsterdam was significantly lower, which may be due to the fact that the physicians were addressed electronically and not personally as it was done at the other hospitals. In Copenhagen, a potential recruitment bias because of the convenience sampling is possible. Due to organizational reasons, no efforts could be taken in order to ensure homogeneous groups between the hospitals. Due to the sampling strategy and the voluntary nature of this survey, the participants cannot be seen as fully representative for all hospital physicians. It seems that especially more experienced physicians participated (> 14 years of experience; professional role at the medium level). These more experienced physicians should provide sufficient practical knowledge to judge a CPOE system. We see this as a strength of this survey.
The translation of the questionnaire was carried out by non-professional translators who were familiar with the field. A multi-stage process including back-translation was not conducted, but might have ameliorated the quality of translation.
The physicians were presented a list of 20 factors. To avoid ‘serial position effects’, the list of context factors was randomized in two different ways in the paper questionnaire and randomized differently for every physician in the electronic questionnaires.
In Denain and Sofia, the physicians generally considered more factors useful (around 10 useful factors/physician on average) than the ones in Amsterdam and Copenhagen (around 6.5 useful factors/physician on average). This can be seen in the heat map in Figure 3, where the columns of Denain and Sofia appear to be ‘greener’. Furthermore, around 40% of the Danish physicians who returned a completed questionnaire did not assign ranks to the context factors in the second step. This was especially the case at Herlev Hospital, which accounted for around 90% of these cases. In general, hospitals at which the physicians were personally recruited and motivated had higher return and completion rates.
Another potential source of bias is the way in which the factors were explained in the questionnaires. Each factor was described with a short definition and a prime example. The assessment of the factors might have been influenced by the given examples (‘example bias’). This effect was anticipated but could not be obviated as the provision of examples is required to achieve a common understanding of a factor.
Results in relation to other studies
The underlying context factors are based on a literature review and expert interviews. Therefore, each factor has already been discussed in the literature (for details, see ). In addition, the Delphi study by Riedmann et al.  provided answers to the question as to which factors might be most beneficial in contextualizing alerts from the CPOE researchers’ point of view.
The hospital physicians and the CPOE researchers agreed on the same two top context factors ‘severity of the effect’ and ‘clinical status of the patient’. Both factors likewise are strongly supported by the literature, but controversy exists. The factor ‘clinical status of the patient’ is considered important by many authors, such as Sittig et al.  and Bates et al. , who think that it is important to take more clinical parameters about the development of a patient’s condition (e.g. renal function) into account. However, there is some controversy about the importance of the ‘severity of the effect’ as a context factor: Some authors, such as Kuperman et al. , Paterno et al.  or Weingart et al. , endorse the relevancy of this factor, whereas others discussed this factor more controversially. Van der Sijs et al.  warn that presenting alerts with different levels of severity increases the override rates compared to alerts without a tiered presentation. Furthermore, the article concluded that the seriousness of an alert, as perceived subjectively by the physicians, might deviate from established objective seriousness indices such as the Dutch seriousness index . Magnus et al.  warned that physicians might only stick to high severity alerts and override all low severity alerts. It seems that the calculation algorithm of the severity index is crucial.
The factor ‘risk factors of the patient’ was rated as the fourth most useful by the researchers and also among the top factors in this study, but it is only seldom mentioned in the literature.
The physicians perceived the factor ‘complexity of the case’ as more beneficial than the researchers (researcher-rank 12). This factor has good, positive support in the literature as well - for example, Sittig et al. stated that clinicians “were more willing to accept clinical decision support when the patient … had multiple medications or chronic conditions”.
Other factors that were top-rated by the researchers (#3 ‘probability of occurrence’ and #5: ‘strength of evidence’) were not perceived as beneficial by the physicians to the same extent. The reason could be the more objective, scientific view of the researchers, while the physicians are more concerned with the individual patient and his or her situation.
The context factor ‘professional experience of the user’ was rated lowest in the researcher survey. The factor is mentioned quite frequently in the literature and is often used as a prime example to describe the concept of contextualizing drug safety alerts (e.g. senior physicians receive fewer alerts than junior physicians). Riedmann et al.  assumed that this outcome might have been related with the theoretical research background of the researchers and that practicing clinicians might hold different views on the relevance of this factor. However, the physicians in this study also found this factor to be of very low importance.
Other factors that were rated low by the physicians were the ‘workload’ and the ‘ADE rate of the department/hospital’. Both factors are innovative ways for addressing the issue of alert filtering. However, for that reason, physicians might not have been directly impressed by their potential impact in reducing alert overload yet. Furthermore, exact ADE rates are not easy to track. Manual strategies are either not exhaustive, too time-consuming or not reproducible, and automatic detection is still an area of research [21, 33–35].
Meaning and generalizability of the study
To our knowledge, this is the first study that systematically addresses the opinions of European hospital physicians regarding the contextualization of alerts in CPOE systems. The opinions on the usefulness of the context factors for the prioritization of CPOE alerts seem to be quite comparable between the participating hospitals (cf. Figure 3), even when the physicians are working in different organizations and technical settings.
Several CPOE systems already classify their alerts using ‘traffic lights’ (e.g. highlighting potentially dangerous alerts in red color), but mostly, this classification is solely based on the expected severity of the effect and not on any other context information.
The results of our study show that clinical physicians, as well as CPOE researchers , also think that CPOE systems should make use of a broader concept of context and take into account more data in order to optimize alert logistics. Additional information about the patients’ clinical status (e.g. development of certain lab values over time) or other risk factors (e.g. allergies) are seen as useful to contextualize alerts. Furthermore, the complexity of the case (e.g. the number of prescribed drugs) could be used to improve drug interaction alerts, which are often unspecific and, thus, of particular annoyance to physicians [13, 28].
Unanswered and New questions
Our findings can be seen as a first, user-centered viewpoint on the issue of ‘contextualizing CPOE systems’. These results only reflect the opinions of clinical physicians and have not yet been empirically verified by implementing contextualized alerting within CPOE systems.
Furthermore, our results do not reveal how the context factors can be implemented into CPOE systems. Within the mentioned PSIP project, some ideas have already been developed and partly implemented .
Except for the factor ‘severity of the effect’, all top context factors identified in our survey refer to patient-related data. Implementing these factors in CPOE/CDS systems would at first require:
Access to relevant data on the patient and case in a highly structured form (including diagnosis, drugs, lab values, allergies, medical procedures and orders). This data could come from different clinical information systems and possibly even from other institutions.
Access to a knowledge base that not only contains drugs, their interactions and contraindications, but also information on the interdependencies with diagnoses, allergies, lab values, planned procedures, etc.
A presentation module of the CPOE that is able to display alerts differently depending on their prioritization value.
Secondly, if these requirements are met, it must be defined as to when and how the available context information is taken into account. Marcilly et al.  proposed an approach to classify situations related to drug prescription and tried to identify the “right moment” of alerting (for example, do not alert if a drug-related lab value is normal and sufficiently monitored).
Further work is essential to putting our still theoretical approach into practice. In reality, more than one context factor might trigger in a specific clinical situation. Hence, it might be necessary to combine certain context factors as proposed in . It is unclear which factors could be combined and which combinations prove to be most beneficial for reducing alert overload and ADE rates. Furthermore, if taking more than one factor into account, one will have to deal with multiple, possibly contradictory prioritization values. It then must be specified as to how to compile a single prioritization value. One must also bear in mind that our factors do not represent orthogonal factors, i.e. they interdepend (e.g. the severity of the effect is highly determined by the clinical status of the patient). These are issues still to be addressed by research.
In this study, 223 physicians from hospitals in four European countries judged the following factors as most useful for prioritizing alerts: ‘severity of the effect’ and ‘clinical status of the patient’. Further important factors are ‘complexity of the case’ and ‘risk factors of the patient’. These findings complement the results of a similar CPOE researcher survey . Some identified differences may be explained by the theoretical, more research-oriented viewpoint of the researchers and the practical, more clinical-oriented viewpoint of the physicians.
Our findings might contribute to the ongoing research tackling medication safety. Our survey provided further insights into how to develop more effective alerting strategies. Further research should also involve CPOE developers and vendors in order to learn how to successfully implement these context factors into the next generation of context-aware CPOE and CDS systems. The impact of these systems should then be evaluated in experimental studies and in clinical practice.
Adverse Drug Event
Academic Medical Center
Computerized Decision Support
Computerized Physician Order Entry
Patient Safety through Intelligent Procedures in Medication
University for Health Sciences, Medical Informatics and Technology
Unweighted Pair Group Method with Arithmetic Mean
University Specialized Hospital for Active Treatment of Endocrinology.
WHO: 10 facts on patient safety.http://www.who.int/features/factfiles/patient_safety/patient_safety_facts/en/index.html,
TNS Opinion & Social: Patient safety and quality of healthcare. Special Eurobarometer. 2010, 327:http://ec.europa.eu/public_opinion/archives/ebs/ebs_327_en.pdf,
European Commission: Communication from the commission to the European parliament and the council on patient safety, including the prevention and control of healthcare-associated infections. 2008,http://ec.europa.eu/health/ph_systems/docs/patient_com2008_en.pdf,
Institute of Medicine: Preventing medication errors: quality chasm series. 2006, Washington DC: National Academy Press
White SV: Patient safety issues. Patient safety: principles and practice. Edited by: Byers JF, White SV. 2004, New York: Springer
Committee of Experts on Management of Safety and Quality in Health Care (SP-SQS) - Expert Group on Safe Medication Practices: Glossary of terms related to patient and medication safety.http://www.who.int/patientsafety/highlights/COE_patient_and_medication_safety_gl.pdf,
Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, Laffel G, Sweitzer BJ, Shea BF, Hallisey R: Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE prevention study group. JAMA. 1995, 274: 29-34. 10.1001/jama.1995.03530010043033.
Benkirane RR, Abouqal R, Haimeur CC, Ech Cherif El Kettani SS S, Azzouzi AA, Mdaghri Alaoui AA: Incidence of adverse drug events and medication errors in intensive care units: a prospective multicenter study. J Patient Saf. 2009, 5 (1): 16-22. 10.1097/PTS.0b013e3181990d51.
Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U: The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc. 2008, 15: 585-600. 10.1197/jamia.M2667.
Kaushal R, Shojania KG, Bates DW: Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med. 2003, 163: 1409-1416. 10.1001/archinte.163.12.1409.
Hug BL, Witkowski DJ, Sox CM, Keohane CA, Seger DL, Yoon C, Matheny ME, Bates DW: Adverse drug event rates in six community hospitals and the potential impact of computerized physician order entry for prevention. J Gen Intern Med. 2010, 25: 31-38. 10.1007/s11606-009-1141-3.
van Doormaal JE, van den Bemt PM, Zaal RJ, Egberts AC, Lenderink BW, Kosterink JG, Haaijer-Ruskamp FM, Mol PG: The influence that electronic prescribing has on medication errors and preventable adverse drug events: an interrupted time-series study. J Am Med Inform Assoc. 2009, 16: 816-825. 10.1197/jamia.M3099.
Kuperman GJ, Bobb A, Payne TH, Avery AJ, Gandhi TK, Burns G, Classen DC, Bates DW: Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007, 14: 29-40.
Glassman PA, Simon B, Belperio P, Lanto A: Improving recognition of drug interactions: benefits and barriers to using automated drug alerts. Med Care. 2002, 40: 1161-1171. 10.1097/00005650-200212000-00004.
Taylor LK, Tamblyn R: Reasons for physician non-adherence to electronic drug alerts. Stud Health Technol Inform. 2004, 107: 1101-1105.
Magnus D, Rodgers S, Avery AJ: GPs' views on computerized drug interaction alerts: questionnaire survey. J Clin Pharm Ther. 2002, 27: 377-382. 10.1046/j.1365-2710.2002.00434.x.
Feldstein A, Simon SR, Schneider J, Krall M, Laferriere D, Smith DH, Sittig DF, Soumerai SB: How to design computerized alerts to safe prescribing practices. Jt Comm J Qual Saf. 2004, 30: 602-613.
Khajouei R, Jaspers MW: The impact of CPOE medication systems' design aspects on usability, workflow and medication orders: a systematic review. Methods Inf Med. 2010, 49: 3-19.
van der Sijs H: Drug safety alerting in computerized physician order entry: unraveling and counteracting alert fatigue. Doctoral thesis. 2009, Rotterdam: Erasmus University
van der Sijs H, Aarts J, Vulto A, Berg M: Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006, 13: 138-147. 10.1197/jamia.M1809.
Beuscart R, McNair P, Brender J: Patient safety through intelligent procedures in medication: the PSIP project. Stud Health Technol Inform. 2009, 148: 6-13.
Dey AK: Understanding and using context. Personal Ubiquitous Computing. 2001, 5: 4-7. 10.1007/s007790170019.
Riedmann D, Jung M, Hackl WO, Stuhlinger W, van der Sijs H, Ammenwerth E: Development of a context model to prioritize drug safety alerts in CPOE systems. BMC Med Inform Decis Mak. 2011, 11: 35-10.1186/1472-6947-11-35.
Riedmann D, Jung M, Hackl WO, Ammenwerth E: How to improve the delivery of medication alerts within computerized physician order entry systems: an international Delphi study. J Am Med Inform Assoc. 2011
Ebbingshaus H: On memory: a contribution to experimental psychology. 1913, New York: Teachers College, Columbia University
Bortz J, Döring N: Forschungsmethoden und evaluation für human- und sozialwissenschaftler. 2006, Heidelberg: Springer, 4
Mayring P: Qualitative content analysis. Forum Qualitative Sozialforschung/Forum Qualitative Social Research. 2000, 1 (2): Art. 20-Available from http://nbn-resolving.de/urn:nbn:de:0114-fqs0002204
Sittig DF, Krall MA, Dykstra RH, Russell A, Chin HL: A survey of factors affecting clinician acceptance of clinical decision support. BMC Med Inform Decis Mak. 2006, 6: 6-10.1186/1472-6947-6-6.
Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, Spurr C, Khorasani R, Tanasijevic M, Middleton B: Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003, 10: 523-530. 10.1197/jamia.M1370.
Paterno MD, Maviglia SM, Gorman PN, Seger DL, Yoshida E, Seger AC, Bates DW, Gandhi TK: Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc. 2009, 16: 40-46. 10.1197/jamia.M2808.
Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS: Physicians' decisions to override computerized drug alerts in primary care. Arch Intern Med. 2003, 163: 2625-2631. 10.1001/archinte.163.21.2625.
van der Sijs H, Aarts J, van Gelder T, Berg M, Vulto A: Turning off frequently overridden drug alerts: limited opportunities for doing it safely. J Am Med Inform Assoc. 2008, 15: 439-448. 10.1197/jamia.M2311.
Carenini M, The Remine Consortium: Patient safety through intelligent procedures in medication: the PSIP project. Stud Health Technol Inform. 2009, 148: 6-13.
Trifiro G, Fourrier-Reglat A, Sturkenboom MC, Diaz Acedo C, Van Der Lei J: The EU-ADR project: preliminary results and perspective. Stud Health Technol Inform. 2009, 148: 43-49.
Lovis C, Douglas T, Pasche E, Ruch P, Colaert D, Stroetmann K: DebugIT: building a European distributed clinical data mining network to foster the fight against microbial diseases. Stud Health Technol Inform. 2009, 148: 50-59.
Marcilly R, Leroy N, Luyckx M, Pelayo S, Riccioli C, Beuscart-Zephir MC: Medication related computerized decision support system (CDSS): make it a clinicians' partner!. Stud Health Technol Inform. 2011, 166: 84-94.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/12/111/prepub
We gratefully acknowledge the help of the following persons who translated the questionnaires and made valuable contributions to the organization of the surveys at the local hospitals: Maurice Langemeijer (Amsterdam); Jens Barholdy, Kaspar Cort Madsen, Flemming Steen Nielsen, Lars Nygård, Jesper Vilandt, Morten Wøjdemann (Copenhagen); Philippe Lecocq (Denain); Krassimira Nechkova, Dimitar Tcharaktchiev (Sofia). We would also like to thank all those who volunteered in pre-testing the questionnaires. Furthermore, our grateful thanks go to Josef Ascher (Innsbruck), who assisted us in developing this measurement instrument.
The research leading to these results received funding from the European Community’s Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 216130.
The authors declare that they have no competing interests.
DR, MJ, WOH and EA developed the underlying context model. MJ, DR, WOH, AH and EA conceptualized and designed the study. MJ organized the survey and designed the questionnaires. LF, KL and MWJ translated the questionnaires and organized the conduction of the surveys in their respective hospitals. MJ performed the analysis and drafted the manuscript. All authors contributed to the manuscript, critically revised the drafts and approved the final manuscript.
Electronic supplementary material
About this article
Cite this article
Jung, M., Riedmann, D., Hackl, W.O. et al. Physicians' Perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems. BMC Med Inform Decis Mak 12, 111 (2012). https://doi.org/10.1186/1472-6947-12-111