This article has Open Peer Review reports available.
“Reducing unnecessary testing in a CPOE system through implementation of a targeted CDS intervention”
© Levick et al.; licensee BioMed Central Ltd. 2013
Received: 9 June 2012
Accepted: 13 March 2013
Published: 8 April 2013
We describe and evaluate the development and use of a Clinical Decision Support (CDS) intervention; an alert, in response to an identified medical error of overuse of a diagnostic laboratory test in a Computerized Physician Order Entry (CPOE) system. CPOE with embedded CDS has been shown to improve quality of care and reduce medical errors. CPOE can also improve resource utilization through more appropriate use of laboratory tests and diagnostic studies. Observational studies are necessary in order to understand how these technologies can be successfully employed by healthcare providers.
The error was identified by the Test Utilization Committee (TUC) in September, 2008 when they noticed critical care patients were being tested daily, and sometimes twice daily, for B-Type Natriuretic Peptide (BNP). Repeat and/or serial BNP testing is inappropriate for guiding the management of heart failure and may be clinically misleading. The CDS intervention consists of an expert rule that searches the system for a BNP lab value on the patient. If there is a value and the value is within the current hospital stay, an advisory is displayed to the ordering clinician. In order to isolate the impact of this intervention on unnecessary BNP testing we applied multiple regression analysis to the sample of 41,306 patient admissions with at least one BNP test at LVHN between January, 2008 and September, 2011.
Our regression results suggest the CDS intervention reduced BNP orders by 21% relative to the mean. The financial impact of the rule was also significant. Multiplying by the direct supply cost of $28.04 per test, the intervention saved approximately $92,000 per year.
The use of alerts has great positive potential to improve care, but should be used judiciously and in the appropriate environment. While these savings may not be generalizable to other interventions, the experience at LVHN suggests that appropriately designed and carefully implemented CDS interventions can have a substantial impact on the efficiency of care provision.
Computerized Physician Order Entry (CPOE) and embedded Clinical Decision Support (CDS) have been shown to improve quality of care and reduce medical errors [1–3]. CPOE and CDS can also improve resource utilization through more appropriate use of laboratory tests and diagnostic studies. While a few Random-Controlled Trials (RCTs) have addressed CDS and CPOE efficacy [2, 4, 5], there are inherent limitations on the external validity of such studies. As a result, observational studies are also necessary in order to understand how these technologies can be successfully employed by healthcare providers. However, documentation of provider-initiated interventions is often limited, and as a result, more studies of implementation at the point of care are needed to understand and measure CDS and CPOE effectiveness [6, 7].
We describe the development and use of a CDS intervention in a CPOE system; an alert, in response to an administratively identified medical error of overuse of a diagnostic laboratory test. We discuss the complexity of effective CPOE implementation and demonstrate the benefits achieved by this quality improvement initiative through a reduction in inappropriate testing and lower medical care costs [2, 7].
The medical error motivating this intervention was identified by the Test Utilization Committee (TUC) at the Lehigh Valley Health Network (LVHN), in Allentown, PA on September, 2008. The TUC noticed critical care patients were being tested daily, and sometimes twice daily, for B-Type Natriuretic Peptide (BNP). BNP is secreted by the heart in response to changes in pressure that occur in patients with congestive heart failure (CHF). It is used as a marker to gauge the change in severity of CHF over time. Repeat and/or serial BNP testing is inappropriate for guiding the management of heart failure and may in fact be clinically misleading . Given that additional BNP tests during an acute exacerbation of CHF provide no relevant clinical information, it was initially unclear why so many tests were being ordered. This overuse did not likely result in adverse clinical events , but it did significantly increase treatment costs.
All CDS interventions in response to identified errors involve critical success factors such as a culture of acceptance of both technology and embedded ‘cognitive forcing strategies” (rules and alerts, standardized order sets, etc.) . This report describes how one healthcare system designed a targeted CDS intervention to reduce unnecessary testing (and associated costs) in a CPOE system, and evaluates its effectiveness. This intervention was part of a larger initiative to create an “advanced”  CDS environment capacity at LVHN. Advanced CDS development moves beyond basic CPOE functions to respond to identified medical errors, evidence-based information changes, potential unintended consequences and continuous quality improvement (CQI) over time, benefiting from a supportive socio-technical infrastructure .
This study emerges from a growing body of literature that seeks to rigorously evaluate applications of Health IT in daily practice. A range of studies have focused on a sociotechnical approach in order to evaluate the multiple factors that contribute to effectiveness of these new tools [12–14]. Factors that have contributed the use and ability to evaluate decision support technologies include greater emphasis by providers on knowledge-management, and an increasing quantity of patient-specific data from genetic testing and electronic health records . The enhanced ability of providers to analyze clinical practice patterns has also contributed to studies seeking to define what constitutes “appropriate testing” .
Among decision support tools, alerts and rules have received a great deal of attention in the literature due to the significant potential to improve care. CPOE systems have been shown to reduce medical errors and overall medical costs [1–3, 17]. Several other studies and recent review articles [2, 3] generally support the potential of CPOE and CDS to improve care, but note that most of the evidence related to effectiveness of these tools is from academic hospitals rather than community hospitals and or primary care facilities. Alerts and rules have also received a great deal of attention in the literature due to identified barriers to their acceptance and implementation [14, 18–21], and unintended consequences [14, 22–25]. For example, this CDS intervention, and the development of the CPOE system in general, was designed to minimize “alert fatigue” that may occur when clinicians are overwhelmed by the number of alerts .
CPOE development and implementation
Lehigh Valley Health Network (LVHN) is the largest health care provider in the Lehigh Valley region of Pennsylvania with: three not-for-profit hospitals; nine health centers in four counties; primary and specialty care practices throughout the region; pharmacy, imaging and lab services; and preferred provider services. In fiscal year 2012, LVHN had over 54,000 acute admissions and 173,000 emergency department visits. The primary service area of Lehigh, Northampton and Carbon counties has a combined population of more than 800,000, making it Pennsylvania’s third most populated region. Minorities comprise more than 54% of all Allentown residents (the region’s largest city), almost 43% of whom are Hispanic or Latino.
LVHN’s Information Technology (IT) Department was an early adopter of CPOE and CDS. The network has been named one of the 100 Most Wired and 25 Most Wireless hospitals by Hospitals and Health Networks magazine. IT applications at LVHN are implemented through a socio-technical infrastructure that facilitates input from providers and includes an evaluative component pre and post implementation. Examples of this infrastructure include the Test Utilization Council (TUC) that identified over-testing for BNP, and enterprise wide Clinical Decision Support Committee, safety committees, and technology assessment groups. These groups and committees maintain an ongoing sensitivity to work flow issues in the implementation of CDS, including just-in-time evidence-based decision support (clinical knowledge) . The discussion below demonstrates an adaptation to overcome a socio-technical barrier to effective usage of this technology—intrusion into the physician work flow.
LVHN began implementation of CPOE in 2001; by 2006, all beds were converted and CPOE use was mandatory. Concurrent with the implementation of CPOE was the installation of a closed loop medication administration system including embedded clinical decision support, automated pharmacy robot dispensing, online medication administration documentation and bar code medication administration. At LVHN, the implementation of this closed loop medication administration system is estimated to have reduced potential harmful medication errors by 80%. a
The initial approach to CDS at LVHN was very deliberate. The implementation team was acutely aware of the impact that CPOE and CDS interventions had on physician workflow as well as their potential for generating alert fatigue. Consequently, interventions that would “add clicks” (such as alerts and rules) and standardized order sets were added judiciously during the early stages of the CPOE rollout—the “basic stage” . As the physicians acclimated to the system, they became more receptive to an increasing number of CDS interventions. Moving physicians along this “continuum of intrusion”  was considered an important success factor by all stakeholders for this CDS intervention.
Identification of BNP over-testing as a medical error
In September 2008, the Senior Clinical Pathologist noticed critical care patients were being tested daily, and sometimes twice daily, for B-type natriuretic Peptide (BNP). The issue was brought to the TUC and prioritized as a high volume (and high cost) test requiring further investigation and intervention. Research revealed that repeat BNP testing is usually not indicated during a hospital stay . The TUC recommended modifying LVHN’s existing CPOE system to employ an expert rule to alert the ordering clinician that a BNP had been performed during the current admission and that repeat testing would add no value to the clinical decision making process.
CDS alert development
Multivariate statistical analysis
Descriptive statistics for analysis variables (N = 41,306)
(SD = 1.364)
Age ≤ 40
40 < Age ≤ 60
60 < Age ≤ 80
Age > 80
DCG/HCC risk score
(SD = 2.271)
Private managed care
Medicare managed care
Medicaid managed care
Monthly time trend
(SD = 0.496)
Post intervention period
We include a variety of controls in our models to account for changes over time in the characteristics of patients that may influence the propensity of physicians to order additional tests. These include indicator variables for patient age (age ≤ 40; 40 < age ≤ 60; 60 < age ≤ 80; age > 80), gender, race (white; black; Hispanic; other race), admission type (emergency; elective; urgent), and insurance type (private traditional plan; Medicare; Medicaid; self-pay; private managed care plan; Medicare managed care; Medicaid managed care). In addition, we include indicators for quartile of the inpatient cost risk distribution, where risk is based on patients’ Diagnostic Cost Group/Hierarchical Condition Category (DCG/HCC) risk scores. b
Our intervention indicator is equal to 1 for all patient admissions during or after June, 2009, the month that the CPOE alert was put and place, and is equal to 0 before that date. The mean, minimum, and maximum of the intervention indicator and all of the control variables are listed in Table 1.
Change in BNP orders associated with one unit change in selected variables from ZTNB regression
40 < Age ≤ 60
60 < Age ≤ 80
Age > 80
1st risk score quartile
2nd risk score quartile
3rd risk score quartile
Private managed care
Medicare managed care
Medicaid managed care
Post intervention period
Time trend variables
Monthly trend squared
In the first specification in column 1 we include a single monthly time trend variable to capture aggregate trends in BNP orders net of the CDS intervention. The estimates from this specification suggest that the intervention reduced the number of BNP orders by 0.6 per inpatient admission, on average. However, when we include additional variables to more flexibility control for aggregate trends over time, this estimate is reduced by over 20%. In addition to a linear monthly time trend, the specification in column 2 also contains the square of the time trend, while the specification in column 3 includes separate indicator variables for each month of the sample. Among these models, we prefer the third and most flexible specification, which indicates that the intervention reduced the number of BNP tests by 0.4 per admission. This effect is precisely estimated, and represents a reduction in BNP orders of 21% relative to the mean.
The financial impact of the rule was also significant. In the three quarters prior to the implementation of the intervention there were 11,209 BNP tests ordered at LVHN. Based on our statistical model, we project that the intervention reduced BNP orders by 21%, or 2,454 tests over three quarters. Multiplying by the direct supply cost of $28.04 per test, this suggests the intervention saved approximately $92,000 per year. These savings are based on supply cost only, and do not account for the increased capacity for the laboratory to perform other tests. As a result, the full annual long run benefits of the intervention are greater than our estimate of supply cost savings, after the short term costs of implementing the intervention are recovered.
While there are few observational studies that evaluate the effectiveness of similar CDS interventions, Bates et al . found using a randomized controlled trial that computerized reminders about redundant laboratory tests significantly reduced the final number of ordered tests. There is also a paucity of research on the impact of CDS and CPOE on costs. A comprehensive 2010 research review of the impact of CDS on test ordering and appropriate utilization in the UK was not able to identify any specific studies on reduced costs from interventions that reduced overuse of testing . Nonetheless, the importance of studies that determine the impact of CDS and CPOE on cost is well recognized [2, 4, 7, 11, 28] A recent meta analysis of the relatively small number of studies currently available that do address the cost consequences of CPOE and CDS, found a significant amount of variability in the level of cost savings; ranging from $6,000 to $84,194 .
Much of the previous literature on CPOE addresses the unintended consequences that result from the implementation of new technologies, the most serious of which is e-iatrogenesis. Because there is no evidence that this over-testing error caused harm to patients, it is not an example of e-iatrogenesis. Furthermore, it is not clear whether over-testing for BNP was in fact an unintended consequence of CPOE. [22–25] In that case it may or may not be classified as a “new kind of error”  or an error made because of the existence of CPOE. LVHN front line staff concluded that the over-testing occurred because either the ordering physician did not understand the time-period of the reliability of the test, or the physician was unable to find previous orders/ results in the electronic record (or did not take the time to search the system). If the former hypothesis is accurate then the experience of LVHN demonstrates the benefit of CPOE in uncovering previously unrecognized instances of inefficient care, and the ability of complementary CDS technologies to increase the efficiency of care provision and lower costs. In contrast, if the latter hypothesis is correct, LVHN’s experience demonstrates how CDS can be used to correct an unintended consequence of CPOE .In both cases the CDS intervention led to a positive outcome, but the two underlying reasons for the over-use error have very different implications for the efficacy of CPOE. This highlights the need for future research studies that use a mix of qualitative and quantitative methods to fully characterize and evaluate the use of new technologies [11–13, 15]. Irrespective of the source of the over-use error, LVHN’s method of identifying and correcting the error can be replicated by other health care networks with similar IT systems and socio-technical infrastructures.
In the specific case of CDS, the use of alerts has great potential to improve care, but should be used judiciously and in the appropriate environment [11, 14, 20, 26, 30]. The intervention implemented at LVHN to reduce unnecessary testing was effective because: 1) It alerted physicians that further BNP testing is potentially misleading; 2) It addressed an information failure associated with the EMR (previous episode of care results were not visible once results were posted) and; 3) The intervention was implemented and evaluated in the context of an advanced  CPOE system facilitating this cycle of implementation and evaluation. Although we cannot determine how each of these factors independently contributed to the success of the intervention, our statistical analysis indicates that they collectively resulted in the reduction of BNP testing by 21%; saving LVHN over $92,000 per year. While these savings may not be generalizable to other interventions, the experience at LVHN suggests that appropriately designed and carefully implemented CDS interventions can have a substantial impact on the efficiency of care provision.
aLehigh Valley Health Network, Internal Report, 2011
bDCG/HCC risk scores are derived from data on patient age, sex, and physician-reported diagnosis codes (ICD-9-CM). They have been validated as a proper measure of risk adjustment in the inpatient setting (Ash et al., 2003; Petersen, Pietz, Woodard & Byrne, 2005) and are also used by CMS to risk adjust Medicare payments to private insurers under Part C (Pope et al., 2004; Pope et al., 2000).
Bala Carver, M.D., Department of Pathology and Laboratory Services, Lehigh Valley Health Network; for providing support from Laboratory Services and ideas for initial design. Geoffrey Correll, BS, MT (ASCP). Department of Pathology and Laboratory Services, Lehigh Valley Health Network; for providing the data regarding test utilization. Lizhong Peng and Jeffrey McFetridge, Lehigh University, for valuable research assistance.
- Bates DW, Leape LL, Cullen DJ, Laird N, Petersen LA, Teich JM, Burdick E, Hickey M, Kleefield S, Shea B, Vander Vliet M, Seger DL: Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. J Am Med Inform Assoc. 1998, 280: 1311-316. 10.1001/jama.280.15.1311.View ArticleGoogle Scholar
- Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, Wing L, Kendrick AS, Sanders GD, Loback D: Effect of Clinical Decision-Support Systems: A systematic Review. Ann Intern Med. 2012, [Epub ahead of print]Google Scholar
- Bates DW, Kuperman GJ, Rittenburg E, Teich JM, Fiskio J, Ma’luf N, Onderdonk A, Wybenga D, Winkelman J, Brennan TA, Komaroff AL, Tanasijevic M: A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med. 1999, 106: 144-150. 10.1016/S0002-9343(98)00410-0.View ArticlePubMedGoogle Scholar
- Main C, Moxham T, Wyatt JC, Kay J, Anderson R, Stein K: Computerised decision support systems in order communication for diagnostic, screening or monitoring test ordering: systematic reviews of the effects and cost-effsets and cost-effectiveness of systems. Helth Technol Assess. 2010, 14 (48): 1-227.Google Scholar
- Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, Jacques A, Dawes M, Abrahamowicz M, Perreault R, Winslade N, Poissant L, Pinsonneault A: A Randomized Trial of the Effectiveness of On-demand versus Computer-triggered Drug Decision Support in Primary Care. J Am Med Inform Assoc. 2008, 15: 430-438. 10.1197/jamia.M2606.View ArticlePubMedPubMed CentralGoogle Scholar
- Velasco F: Clinical Decision Support and Meaningful Use. Adv Health Inform Exec. 2010, 20: 15-20.Google Scholar
- Lehmannm H, Ohno-Machado L: Evaluation of informatics systems: beyond RCTs and beyond the hospital. Am Heart J. 2006, 152 (5): 828-34. 10.1016/j.ahj.2006.08.021.View ArticleGoogle Scholar
- Wu AH: Serial testing of B-type natriuretic peptide and NTpro-BNP for monitoring therapy of heart failure: the role of biologic variation in the interpretation of results. Am Heart J. 2006, 152 (5): 828-34. 10.1016/j.ahj.2006.08.021.View ArticlePubMedGoogle Scholar
- Bates DW, Boyle DL, Vander Vliet MB, Schneider J, Leape L: Relationship between medication errors and adverse drug events. Gen Int Med. 1995, 10: 199-205. 10.1007/BF02600255.View ArticleGoogle Scholar
- Croskerry P: Cognitive Forcing Strategies in Clinical Decisionmaking. Ann Emerg Med. 2003, 41 (1): 110-120. 10.1067/mem.2003.22.View ArticlePubMedGoogle Scholar
- Kuperman GJ, Bobb A, Payne TH, Avery AJ, Ghandhi TK, Burns G, Classen DC, Bates DW: Medication-related clinical decision support in computerized provider order entry systems: a review. J Am Med Inform Assoc. 2007, 14 (1): 29-40.View ArticlePubMedPubMed CentralGoogle Scholar
- Westbrook JI, Braithwaite J, Georgiou A, Ampt A, Creswick N, Coiera E, Idema R: Multimethod evaluation of information and communication technologies in health in the context of wicket problems and sociotechnical theory. J Am Med Inform Assoc. 2007, 14 (6): 746-755. 10.1197/jamia.M2462.View ArticlePubMedPubMed CentralGoogle Scholar
- Georgiou A, Westbrook JI, Braithwaite J: An empirically-derived approach for investigating Health Information Technology: the Elementally entangled Organisational Communication (EEOC) framework. BMC Med Inform Decis Mak. 2012, 12: 68-10.1186/1472-6947-12-68.View ArticlePubMedPubMed CentralGoogle Scholar
- Harrison MI, Koppel R, Bar-Lev S: Unintended Consequences of Information Technologies in Health Care – An Interactive Sociotechnical Analysis. JAMIA. 2007, 14: 542-549.PubMedPubMed CentralGoogle Scholar
- Wang XS, Nayda L, Dettinger R: Infrastructure for a clinical-decision-intelligence system. IBM Syst J. 2007, 46 (1): 151-169.View ArticleGoogle Scholar
- van Walraven C, Naylor CD: Do we know what inappropriate laboratory utilization is? A systematic review of laboratory clinical audits. JAMA. 1998, 280 (6): 550-558. 10.1001/jama.280.6.550.View ArticlePubMedGoogle Scholar
- Galanter WL, Didomenico RJ, Polikaitis A: A trial of automated decision support alerts for contraindicicated medications using computerized physician order entry. J Am Med Inform Assoc. 2005, 12 (3): 269-274. 10.1197/jamia.M1727.View ArticlePubMedPubMed CentralGoogle Scholar
- Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS: Physicians’ Decisions to Override Computerized Drug Alerts in Primary Care. Arch Intern Med. 2003, 163 (21): 2625-2631. 10.1001/archinte.163.21.2625.View ArticlePubMedGoogle Scholar
- Payne TH, Nichol WP, Hoey P, Savarino J: Characteristics and Override Rates of Order Checks in a Practitioner Order Entry System. Proc AMIA Annu Fall Symp. 2002, 602-606.Google Scholar
- Wright A, Sittig DF, Ash JS, Bates DW, Feblowitz J, Fraser G, Maviglia SM, McMullen C, Nichol WP, Pang JE, Starmer J, Middleton B: Governance for clinical decision support: case studies and recommended practices from leading institutions. JAMIA. 2011, 18: 187-194.PubMedPubMed CentralGoogle Scholar
- Osheroff JA, Teich JM, Levick D, Saldana L, Velasco FT, Sittig DR, Rogers KM, Jenders RA: Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. 2012, HIMSS Publishing, 2Google Scholar
- Ash JS, Sittig DF, Dykstra R, Campbel E, Guappone K: The Unintended Consequences of Computerized Provider Order Entry: Finding from a Mixed Methods Exploration. Int J Med Inform. 2009, 78 (Suppl 1): S69-S76.View ArticlePubMedGoogle Scholar
- Ash JS, Sittig DF, Poon EG, Guappone D, Campbell E, Dykstra RH: The Extent and Importance of Unintended Consequences Related to Computerized Provider Order Entry. JAMIA. 2007, 14 (4): 415-423.PubMedPubMed CentralGoogle Scholar
- Campbell EM, Sittig DF, Ash JS, Guappone DP, Dykstra RH: Types of Unintended Consequences Related to Computerized Provider Order Entry. JAMIA. 2006, 13: 547-556.PubMedPubMed CentralGoogle Scholar
- Koppel R, Metlayy JP, Cohen A, Abaluck B, Localio AR, Kimmel SE, Strom BL: Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors. JAMA. 2005, 293 (10): 1197-1203. 10.1001/jama.293.10.1197.View ArticlePubMedGoogle Scholar
- Byers J: CMIO Summit: Levick says DCS must be clear for success. Health Imaging. in pressGoogle Scholar
- Cameron A, Trivedi P: Microeconometrics: Methods and Applications. 2005, New York: Cambridge University PressView ArticleGoogle Scholar
- Qaseem A, Alguire P, Dallas P, Feinberg L, Fitzgerald F, Horwitch C, Humphrey L, LeBlond R, Moyer D, Wiese J, Weinberger S: Appropriate Use of Screening and Diagnostic Tests to Foster High-Value, Cost-Conscious Care. Ann Intern Med. 2012, 156 (2): 147-149. 10.7326/0003-4819-156-2-201201170-00011.View ArticlePubMedGoogle Scholar
- Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux R, Samsa G, Hasselblad V, Williams JW, Wing L, Musty M, Kendrick AS: Enabling Health Care Decisionmaking Through Clinical Decision Support and Knowledge Management. Evidence Report No. 203. (Prepared by the Duke Evidence-based Practice Center under Contract No. 290-2007-10066-I.) AHRQ Publication No. 12-E001-EF. 2012, Rockville, MD: Agency for Healthcare Research and QualityGoogle Scholar
- Schedlebauer A, Prasad V, Mulvaney C, Phansalkar S, Stanton W, Bates DW, Avery AJ: What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior?. JAMIA. 2009, 16: 531-538.Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/13/43/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.