- Research article
- Open Access
- Open Peer Review
Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department
BMC Medical Informatics and Decision Making volume 14, Article number: 105 (2014)
Early recognition of severe sepsis and septic shock is challenging. The aim of this study was to determine the diagnostic accuracy of an electronic alert system in detecting severe sepsis or septic shock among emergency department (ED) patients.
An electronic sepsis alert system was developed as a part of a quality-improvement project for severe sepsis and septic shock. The system screened all adult ED patients for a combination of systemic inflammatory response syndrome and organ dysfunction criteria (hypotension, hypoxemia or lactic acidosis). This study included all patients older than 14 years who presented to the ED of a tertiary care academic medical center from Oct. 1, 2012 to Jan. 31, 2013. As a comparator, emergency medicine physicians or the critical care physician identified the patients with severe sepsis or septic shock.
In the ED, vital signs were manually entered into the hospital electronic heath record every hour in the critical care area and every two hours in other areas. We also calculated the time from the alert to the intensive care unit (ICU) referral.
Of the 49,838 patients who presented to the ED, 222 (0.4%) were identified to have severe sepsis or septic shock. The electronic sepsis alert had a sensitivity of 93.18% (95% CI, 88.78% - 96.00%), specificity of 98.44 (95% CI, 98.33% – 98.55%), positive predictive value of 20.98% (95% CI, 18.50% – 23.70%) and negative predictive value of 99.97% (95% CI, 99.95% – 99.98%) for severe sepsis and septic shock. The alert preceded ICU referral by a median of 4.02 hours (Q1 - Q3: 1.25–8.55).
Our study shows that electronic sepsis alert tool has high sensitivity and specificity in recognizing severe sepsis and septic shock, which may improve early recognition and management.
Severe sepsis and septic shock are responsible for significant morbidity and mortality. In the United States, sepsis mortality reportedly occurs in 65.5 per 100,000 persons . In Europe, the in-hospital mortality rate from sepsis is estimated to be 24.1% . Notably, the incidence of sepsis has dramatically increased in recent years, with the sepsis rate per 10,000 admissions doubling between 2000 and 2008 . Moreover, in-hospital deaths are reported to be eight times greater for patients hospitalized for sepsis compared to those hospitalized with other diagnoses (17% and 2%, respectively) . Sepsis-related mortality is also extremely costly to the healthcare system, with an estimated $14.6 billion in 2008 in USA . In Europe, 25% of sepsis patients are admitted to the intensive care unit (ICU) through the emergency department (ED) .
Compliance with evidence-based guidelines for severe sepsis and septic shock management has been shown to be low, which has been attributed to factors such as delayed recognition . Thus, a number of international campaigns (e.g., Surviving Sepsis Campaign, World Sepsis Day) have been launched to raise awareness, improve the care of patients with severe sepsis and septic shock, and emphasize early identification and intervention, which have been shown to reduce mortality ,. Furthermore, the 2012 Surviving Sepsis Campaign guidelines recommended routine screening for severe sepsis to allow earlier implementation of therapy and stated that the “key to achieving a reduction in mortality from severe sepsis is not just standardized evidence-based treatment, but equally important, the early recognition of sepsis” .
A significant challenge in recognizing severe sepsis early is the complexity of the sepsis presentation, which makes it significantly more challenging to identify patients compared to many other time-critical conditions that are treated in the ED, such as ST-segment elevation myocardial infarction or acute ischemic stroke. Another challenge is ED crowding, which has been linked to the decreased likelihood of adherence to guideline-concordant care. Studies have also demonstrated a relationship between ED crowding and delayed antibiotic administration . A promising measure in improving the efficacy of the health care services provided in ED during crowdedness periods . Such systems should be time efficient and easy to use in order to maintain their impact even during periods of ED crowding.
Different screening tools using different combinations of severe sepsis and septic shock criteria have been studied in ED patients, with sensitivities ranging from 14 to 69% and specificities ranging from 35 to 99% -.
To be effective, a screening tool should meet a number of important criteria . It must cause little or no patient morbidity, be affordable and easily available, identify the conditions for which treatment exists and be more effective when applied early in the disease course . Sepsis is a prime example of a disease in which a screening tool could significantly impact ED management. The objective of this study was to determine the diagnostic accuracy of an electronic alert system in detecting severe sepsis or septic shock in (ED) patients.
This single-center study of an electronic sepsis alert tool was conducted at a Joint Commission International (JCI)-accredited tertiary care academic medical center. This initiative was a part of a hospital-wide quality-improvement project that addressed the care of patients with severe sepsis and septic shock. The study was performed over a 4-month period from Oct. 1, 2012 to Jan. 31, 2013. The hospital has 900 beds and receives approximately 200,000 ED visits each year. ED admissions represents 46% of the total hospital admissions per year.
In the ED, vital signs were manually entered into the hospital information system (QuadraMed Computerized Patient Record System, Reston, VA, USA) hourly in the critical care area of the ED (patient triage levels one or two based on the Canadian Triage And Acuity Scale [CTAS]) and every 2 hours in other areas in the ED (patient CTAS triage levels three and four) . The ED had a stat laboratory with a turnaround time for white blood cell and lactate tests of approximately 1 hour. Patients presenting with severe sepsis and septic shock to the ED were first seen by the ED staff and then treated according to their clinical conditions. The patients were then were referred to the ICU team or to other services, as necessary.
Sepsis screening tool development
To develop a sepsis-screening tool, we first began a multidisciplinary sepsis-working group to develop a severe sepsis and septic shock alert for patients older than 14 years of age. The group used multiple “Plan, Do, Study, Act” (PDSA) cycles to test different combinations and determine appropriate detection parameters . After multiple PDSA cycles in the development (testing) domain of our electronic health record (EHR) system, the combination shown in Sepsis-screening tool alert parameters appeared to have the best ability to screen for severe sepsis and septic shock in an ED setting. This tool works as follows.
The screening tool automatically scans certain clinical and laboratory parameters, as well as the physician orders for fluid bolus or oxygen therapy (List 1).
If certain conditions are met (Sepsis-screening tool alert parameters), the system generates a “severe sepsis and septic shock” alert, and the test is considered to be positive.
This alert goes to the “nurse work list”.
If the criteria are not met, the test is considered to be negative (Figure 1).
The nurse responds to the alert and notifies a physician using a paging system, as instructed in the alert message.
To avoid multiple activations on the same patient, the alert is deactivated as follows:
for 48 hours if the patient has suspected severe sepsis and septic shock,
for 24 hours if the patient does not have severe sepsis or septic shock, and
indefinitely if the code status precludes intensive care management of sepsis.
Alerts do not occur during deactivation time.
List 1: Sepsis-screening tool alert parameters
Two of the systemic inflammatory response syndrome criteria
Temperature >38°C or <36°C
Pulse >90 beats per minute
Respiratory rate >20 breaths per minute
White blood cell count >12,000 or <4,000 ml
One organ dysfunction
Systolic blood pressure <90 to 86 mm Hg with intravenous fluids or <86 mm Hg regardless of fluids
Blood oxygen saturation <90% to 85% with supplemental oxygen or <85% without oxygen
Lactate >2 mmol/L
Two of the above organ dysfunction criteria
The study included a prospective consecutive series of all adult patients presenting to the ED from Oct. 1, 2012 to Jan. 31, 2013. We considered the clinical assessment of the emergency and ICU physicians to be the reference standard. ED and ICU physicians assessed patients for the presence of severe sepsis or septic shock using the standard diagnostic criteria . The assessment was independent from the electronic sepsis alert system. The study excluded patients younger than 14 years of age, which is the cutoff in our hospital for adult patients. The study was approved by the National Guard Health Affairs (NGHA) Institutional Review Board (IRB) and no consent was required.
We calculated the test characteristics of the electronic sepsis alert system, including the sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio, and negative likelihood the ratio with corresponding 95% confidence intervals (CIs). For patients’ who were referred to ICU, we collected additional demographic and clinical data. In addition, we calculated the time from alert activation to the ICU referral and values were expressed as medians and quartiles. We used the SPSS 20.0 software package (SPSS Inc., Chicago, IL, USA) for statistical analysis.
In the 4-month study period, 49,838 patients presented to the ED, of whom 220 were identified by the ED or ICU physicians to have severe sepsis or septic shock. Table 1 shows the true positive and true negative test results. Table 2 shows the demographic and clinical characteristics of patients admitted to ICU.
Table 3 shows the test characteristics of the electronic sepsis alert. For recognizing severe sepsis and septic shock, the test had a sensitivity of 93% (95% CI = 89%–96%), specificity of 98% (95% CI = 98.3%–98.5%), positive predictive value of 20% (95% CI = 18%–23%), and negative predictive value of 99.9% (95% CI = 99.95%–99.98%). The positive likelihood ratio was 59.88 (95% CI, 55.36–64.78), and the negative likelihood ratio was 0.069 (95% CI, 0.429–0.11). The electronic sepsis alert preceded ICU referral by a median of 4.02 hours (Q1–Q3, 1.25–8.55 hours).
In this study, we described the test characteristics of an electronic screening tool for severe sepsis and septic shock in the ED. The electronic sepsis alert had high sensitivity and specificity and negative predictive value. The low positive predictive value probably resulted from the detection criteria used in the alert system, which did not include clinical information, such as presenting symptoms or confirmatory laboratory tests. In addition, we demonstrated that our electronic sepsis alert tool preceded ICU referral for severe sepsis or septic shock. This finding is important and would facilitate time-sensitive sepsis management and early sepsis care beginning in the ED.
Various tools to screen for sepsis and severe sepsis have been previously evaluated in the ED setting. In a prospective observational study by Meurer et al., an electronic alert was sent to the care team if two or more systemic inflammatory response syndrome (SIRS) criteria were detected in patients older than 70 years. Their system had a sensitivity of 14% and a specificity of 98% for detecting an infection . Nelson et al. used an automated messaging system that alerted the care team if a patient presented to the ED with two or more SIRS criteria in addition to two systolic blood pressure readings of <90 mm Hg. Their system had a sensitivity of 64%, specificity of 99%, positive predictive value of 54%, and negative predictive value of 99% . Jaimes et al. also found that for patients presenting to the ED, the presence of two or more SIRS criteria had a sensitivity to detect infection of 69%, specificity of 35%, positive predictive value of 90%, and negative predictive value of 12% .
The main difference between our study and previous studies is the criteria for activating the alert system. We used a combination of clinical and laboratory parameters and a physician order of fluid or oxygen therapy. We used an electronic alert rather than a paper-based system and applied this system in an ED setting. Our results demonstrated that the electronic alert preceded ICU referral by a median of 4.02 hours (Q1–Q3, 1.25–8.55 hours). The low prevalence found in our study was because the alert system screened all patients who presented to the ED, not only the high-risk patients or those who presented to the ED with infection. Alert systems should focus on preventing active failures and individual error-producing conditions . Therefore, achieving a high specificity of 98.4% was important. To avoid “alert fatigue and overriding”, the alert should be followed by directing the bedside nurse to ask the physician to respond to it .
Our results must be interpreted based on their strengths and limitations. We demonstrated good diagnostic properties in an ED with a high number of visits. One of the limitations of this study is that it was conducted at a single academic medical center. EHR systems vary widely among institutions, therefore it is not possible to comment specifically on the ease with which our sepsis recognition strategy might be used in other institutions. Another limitation is that we described only the characteristics of patients who were admitted to the ICU. Another factor is related to the limitation of the alert design: our alert system scans the most recent vital signs for sepsis criteria; hence, the trigger threshold is decreased because all criteria must be aligned at the same time.
The high specificity and sensitivity and negative predictive values of the alert system are promising. We have demonstrated a feasible approach that allows us to recognize patients with severe sepsis with high sensitivity and specificity. An electronic alert preceding ICU referral could lead to earlier sepsis management and minimize delays in recognizing sepsis.
Intensive care unit
Canadian Triage and Acuity Scale
Plan, Do, Study, Act
Institutional review board
National Guard Health Affairs
Systemic inflammatory response syndrome
Electronic health records
Wang HE, Devereaux RS, Yealy DM, Safford MM, Howard G: National variation in United States sepsis mortality: a descriptive study. Int J Health Geogr. 2010, 9: 9-10.1186/1476-072X-9-9.
Vincent JL, Sakr Y, Sprung CL, Ranieri VM, Reinhart K, Gerlach H, Moreno R, Carlet J, Le Gall JR, Payen D: Sepsis in European intensive care units: results of the SOAP study. Crit Care Med. 2006, 34 (2): 344-353. 10.1097/01.CCM.0000194725.48928.3A.
Hall MJ, Williams SN, DeFrances CJ, Golosinskiy A: Inpatient care for septicemia or sepsis: a challenge for patients and hospitals. NCHS Data Brief. 2011, 62: 1-8.
Bullard MJ, Unger B, Spence J, Grafstein E: Revisions to the Canadian Emergency Department Triage and Acuity Scale (CTAS) adult guidelines. CJEM. 2008, 10 (2): 136-151.
Rivers E, Nguyen B, Havstad S, Ressler J, Muzzin A, Knoblich B, Peterson E, Tomlanovich M: Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001, 345 (19): 1368-1377. 10.1056/NEJMoa010307.
Dellinger RP, Levy MM, Rhodes A, Annane D, Gerlach H, Opal SM, Sevransky JE, Sprung CL, Douglas IS, Jaeschke R, Osborn TM, Nunnally ME, Townsend SR, Reinhart K, Kleinpell RM, Angus DC, Deutschman CS, Machado FR, Rubenfeld GD, Webb SA, Beale RJ, Vincent JL, Moreno R: Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012. Crit Care Med. 2013, 41 (2): 580-637. 10.1097/CCM.0b013e31827e83af.
Surviving Sepsis Campaign: Statement from SSC Leadership on Time Zero in the Emergency Department. , [http://www.survivingsepsis.org/SiteCollectionDocuments/Time-Zero.pdf]
Fee C, Weber EJ, Maak CA, Bacchetti P: Effect of emergency department crowding on time to antibiotics in patients admitted with community-acquired pneumonia. Ann Emerg Med. 2007, 50 (5): 501-509. 10.1016/j.annemergmed.2007.08.003. 509 e501
Meurer WJ, Smith BL, Losman ED, Sherman D, Yaksich JD, Jared JD, Malani PN, Younger JG: Real-time identification of serious infection in geriatric patients using clinical information system surveillance. J Am Geriatr Soc. 2009, 57 (1): 40-45. 10.1111/j.1532-5415.2008.02094.x.
Nelson JL, Smith BL, Jared JD, Younger JG: Prospective trial of real-time electronic surveillance to expedite early care of severe sepsis. Ann Emerg Med. 2011, 57 (5): 500-504. 10.1016/j.annemergmed.2010.12.008.
Jaimes F, Garcés J, Cuervo J, Ramírez F, Ramírez J, Vargas A, Quintero C, Ochoa J, Tandioy F, Zapata L, Estrada J, Yepes M, Leal H: The systemic inflammatory response syndrome (SIRS) to identify infected patients in the emergency room. Intensive Care Med. 2003, 29 (8): 1368-1371. 10.1007/s00134-003-1874-0.
Grimes DA, Schulz KF: Uses and abuses of screening tests. Lancet. 2002, 359 (9309): 881-884. 10.1016/S0140-6736(02)07948-5.
Kocher KE, Shane SA, Venkatesh AK, Aronsky D, Asplin BR, Rathlev NK: Interventions to safeguard system effectiveness during periods of emergency department crowding. Acad Emerg Med. 2011, 18 (12): 1313-1317. 10.1111/j.1553-2712.2011.01219.x.
Langley GJ, Moen R, Nolan KM, Nolan TW, Norman CL, Provost LP: The Improvement Guide. 2009, John Wiley & Sons, San Franciso, California
Gerald J, Alsip J, Hicks J, Waldrum M, Dunlap N: Using the EMR to perform continuous, automated, real-time surveillance to identify hospitalized patients at risk of sepsis. Chest. 2011, 140 (4_MeetingAbstracts): 426A-10.1378/chest.1118563. doi:10.1378/chest.1118563
van der Sijs H, Aarts J, Vulto A, Berg M: Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006, 13 (2): 138-147. 10.1197/jamia.M1809.
Dr. Sami Alsolamy (PI) had full access to all study data and takes responsibility for the integrity and the accuracy of the data analysis.
The study was funded by the King Abdullah International Medical Research Centre (KAIMRC).
The authors declare that they have no competing interests.
SS contributed to the study conception and design, data analysis and interpretation, critical revision for important intellectual content, drafting the article, final approval of the article, and statistical analysis. MS contributed to the study conception and design, data analysis and interpretation, drafting the article, final approval of the article, and statistical analysis. MT contributed to the administrative and technical support and critical revision for important intellectual content. HD contributed to the administrative and technical support, conception and design, data analysis and interpretation, critical revision for important intellectual content, and final approval of the article. AMM contributed to the administrative and technical support, data collection and assembly, and final approval of the article. NJ contributed to the data analysis and interpretation, critical revision for important intellectual content, and drafting of the article. FE contributed to the data analysis and interpretation, critical revision for important intellectual content, and drafting of the article. FH contributed to the administrative and technical support and data collection, assembly, analysis and interpretation. AM contributed to the administrative and technical support and to the data collection, assembly, analysis and interpretation. AA contributed to the administrative and technical support, data analysis and interpretation, critical revision for important intellectual content, and final approval of the article. YMA contributed to the study conception and design, administrative and technical support, data collection, assembly, analysis, and interpretation, critical revision for important intellectual content, final approval of the article, and statistical expertise, as well as the obtaining of funding. All authors read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Alsolamy, S., Al Salamah, M., Al Thagafi, M. et al. Diagnostic accuracy of a screening electronic alert tool for severe sepsis and septic shock in the emergency department. BMC Med Inform Decis Mak 14, 105 (2014) doi:10.1186/s12911-014-0105-7
- Clinical decision support
- Sensitivity and specificity
- Septic shock
- Emergency department
- Electronic alert