- Research article
- Open Access
Testing modes of computerized sepsis alert notification delivery systems
BMC Medical Informatics and Decision Making volume 16, Article number: 156 (2016)
The number of electronic health record (EHR)-based notifications continues to rise. One common method to deliver urgent and emergent notifications (alerts) is paging. Despite of wide presence of smartphones, the use of these devices for secure alerting remains a relatively new phenomenon.
We compared three methods of alert delivery (pagers, EHR-based notifications, and smartphones) to determine the best method of urgent alerting in the intensive care unit (ICU) setting. ICU clinicians received randomized automated sepsis alerts: pager, EHR-based notification, or a personal smartphone/tablet device. Time to notification acknowledgement, fatigue measurement, and user preferences (structured survey) were studied.
Twenty three clinicians participated over the course of 3 months. A total of 48 randomized sepsis alerts were generated for 46 unique patients. Although all alerts were acknowledged, the primary outcome was confounded by technical failure of alert delivery in the smartphone/tablet arm. Median time to acknowledgment of urgent alerts was shorter by pager (102 mins) than EHR (169 mins). Secondary outcomes of fatigue measurement and user preference did not demonstrate significant differences between these notification delivery study arms.
Technical failure of secure smartphone/tablet alert delivery presents a barrier to testing the optimal method of urgent alert delivery in the ICU setting. Results from fatigue evaluation and user preferences for alert delivery methods were similar in all arms. Further investigation is thus necessary to understand human and technical barriers to implementation of commonplace modern technology in the hospital setting.
Generation of computerized notifications and alarms by automated detection algorithms for a variety of clinical conditions leads to alarm fatigue, which is one of the most important health technology hazards over past years . Several solutions have been proposed to improve alarm detection conditions in the healthcare setting . In the case of alarm system management, the challenge is to deliver the correct alarm, using the right alarm delivery mode, to the right recipient(s) .
Minimizing the number of clinically insignificant alarms (better pattern recognition), optimizing alarm notification, and response protocols are the goals to address clinical alarm hazards to insure patients receive appropriate care at the time it is needed [1, 4].
Perception of different methods of alert delivery is significantly influenced by complex human cognition factors [5, 6]. In the specific context of the hospital setting, mechanisms of notification or alert delivery have been exploited, including oral communication, charts, loudspeaker alerts, phones, paging, electronic health record (EHR) display, and email . New methods of alert delivery have also led to the development of technology to reduce errors in the hospital setting . 
Successful EHR notification delivery has the potential to reduce errors in the hospital setting [9–12]. However, it is not sufficient for a clinical alert system to be merely capable of generating medically meaningful alerts [4, 13]. This is because implementation of any automated notification or alert system must be performed in the context of information overload and complex task interruption. In ICU setting, even meaningful alerts pose the risk of interruption  and information overload can alter alert perception [15, 16]. There is a need to consider how a system can generate clinically meaningful alerts, while concurrently minimizing information overload and task interruption. To do this, a better understanding of human cognition and user interfaces is required [17, 18]. An ideal communication tool should enable bi-directional, rapid, secure, and non-disruptive transmission of content-rich messages . It should provide specific mechanisms to avoid any potential for protected health information security breach. Knowledge of optimal methods of delivery of urgent alerts in the intensive care unit (ICU) setting, particularly with the goal to shorten time-to-reaction and to decrease alert fatigue, is limited and contradictory [20, 21].
The objective of this study was to compare three methods of alert delivery - pagers, EMR-based notifications, and smartphones - to determine the best method of urgent alerting in the ICU setting.
Study design and setting
This prospective randomized study was performed from October 2015 through December 2015 in the 54 beds ICU at Mayo Clinic in Jacksonville, FL. This ICU includes multidisciplinary beds including medical, surgical, transplant, and neurology critical care services. The Mayo Clinic Institutional Review Board (IRB) has approved this study. There were no changes in usual protocols of patient care. Oral consent was obtained from participating clinicians.
Clinicians with a personal iPhone Operating System (iOS) device (smartphone or tablet) were eligible to participate. One ICU 12-hour shift was treated as a study period for each participant. Participants were already familiar with the EHR electronic sepsis alerts and SSC (Surviving Sepsis Campaign) recommendations [22, 23] as they were routinely used in the ICU.
The automated, EMR-based sepsis detection tool (sepsis sniffer), already validated and described elsewhere , continuously assessed EHR data for sepsis criteria. In a case of automated detection, it generated an alert - a yellow triangular icon appeared within the EHR, indicating individual patient with sepsis. Standard practice includes alerting a nursing team leader by a pager, who makes decision to activate a rapid response team (RRT). For the purposes of the study generated alerts were sent to participants using randomly selected delivery method. These alerts included the following text: “Shock alert: sepsis detected, room XXX”.
Alert delivery and randomization algorithms
Three methods of alert delivery were used: pager, EHR-based monitor display notification, and iOS-based smartphone/tablet (Fig. 1).
At the beginning of study periods, participating clinicians manually triggered a specific flag within the EHR to “claim” hospitalized patients and become an “assigned practitioner”. This flag cleared automatically at the end of each study period. If there was no assigned clinician, as a safety net, an alert was rerouted following standard practice. At the time an alert is generated, an algorithm randomly determines which of 3 routes will be used to deliver that alert to the clinician who has 'claimed' that patient. Participating clinicians could receive more than one alert during one study period.
iOS device pathway description
Sepsis alerts were delivered using a standard practice iOS application at Mayo Clinic (Synthesis Mobile) complied with protected health information (PHI) encryption and HIPAA specifications . This application uses a secured virtual private network (VPN) connection and allows delivery of pop-up notifications. The following conditions were required to exploit iOS device pathway: a) the patient must be ‘claimed’, b) Synthesis Mobile must be running on person’s device, c) VPN connection must be established. Each participant was given instructions before the beginning of the study to ensure that latest Synthesis Mobile Application was installed on their device and how to make sure that the application was running on the background during their shifts. In the case of failure to meet all criteria of eligibility, alerts assigned by the randomization algorithm to iOS were sent by pager instead. Prior to study implementation, notification system was successfully piloted on pagers, the EHR, and iOS devices.
Alert delivery was acknowledged by clinician participants through the EHR using individual personal credentials by entering login and password. The primary outcome was time to acknowledgment, which was defined as the difference between time of alert generation and the time of interaction with the alert in the EHR.
The secondary outcome was time from alert generation to SSC 3-hours bundle completion (‘time-to-goal’) [22, 23, 26]. Also measured were time from acknowledgement to time-to-goal, and timeliness of acknowledgment when time-to-goal was achieved. We assessed clinician participant experience with methods of alert delivery, using a 7-points Likert-like scale from extremely dissatisfied (1) to extremely satisfied (7). A post-shift fatigue was evaluated using the validated 7-points Samn-Perelli (SP) scale scoring fatigue from “1 - fully alert, wide awake” to “7 - completely exhausted, unable to function effectively” [27, 28].
Data handling and statistical analysis
For the power calculation we used mean time to acknowledgement of 13 minutes, based on the study of Moss and colleagues (median response 3 mins, IRQ 1-8 mins, and less than 15 mins for ‘adequate’ response) . For mean time to acknowledgement of 13 mins, with a presumed standard deviation of 7 mins, 240 measurements (alerts) should produce 84% power. The number of unique patients and number of alerts per unique patient directly influence the number of potential clinician participant responses. However, these factors are independent of the power calculation of the number of clinician participant measurements needed.
Data collected by the randomization algorithm was stored in a dedicated table within an institutional research warehouse database. An online electronic survey was conducted at the end of study to measure clinician participant experience and fatigue results. Continuous variables are presented as median ± interquartile range. Categorical variables are presented as count and percentage. Multiple statistical comparisons were performed using the Kruskal-Wallis or chi-square test as appropriate. A two-tailed p-value of less than 0.05 was considered statistically significant. Statistical software JMP 11 (SAS, Cary, NC) was used for all calculations.
Out of 40 potentially eligible clinicians, 23 (58%) agreed to participate in this study. All clinician participants had iOS devices (smartphone or tablet). During the 3 month study period using this sepsis-detection decision algorithm, there were 69 alerts in 67 patients. However, only 46 patients were “claimed”, and a total of 48 alerts were sent by the randomization algorithm (Table 1). All of these alerts were acknowledged.
The study was terminated prematurely because of failure of the iOS arm. All patients in the iOS arm (n = 22) were claimed and participating clinicians was assigned, but the algorithm of alert delivery failed to detect the presence of the dedicated iOS EHR application in all of these cases. This caused all alerts assigned to this arm to be rerouted to the pager arm. The median time to acknowledgement for EHR-embedded notifications was 170 minutes (N = 14) and 102 minutes (N = 34) for alerts delivered by pager (p = 0.15). The SSC bundle was completed for 60% (N = 28) patients. In the case of these patients, the median time from alert generation to SSC bundle completion was 159 minutes for EHR-notifications and 164 minutes for pager notifications. At the end of the study participants were satisfied with the both methods of alert delivery. The median fatigue score at the end of shifts was equal in both arms.
Methods of urgent alert delivery in the ICU setting were evaluated in this randomized prospective study. The study was terminated before target accrual was reached because the iOS randomized arm was not able to deliver alerts from the sepsis sniffer. Time to acknowledgment was almost 2 hours for the pager arm and 2.5 hours for the EHR arm, which was not a statistically significant difference. For patients in both arms for whom the SSC bundle was completed, 60% of alerts were acknowledged before bundle goals. However, this time likely does not represent the time of alert reception by clinicians. The experience from the survey about EHR and pager methods of notification delivery was neutral to positive in both groups. No difference in Samn-Perelli fatigue score was detected between participants in the assigned arms.
Traditional computerized paging system notifications are widely implemented and frequently used. The content of delivered information is limited to a finite number of characters and it is also not possible to send immediate feedback to the sender. Urgent character of alerts does not force clinicians to take immediate action in the EHR. The provider often prioritizes his attention to the patient until the initial goals of care are reached, and thereafter acknowledge the alert in the EHR. Although this EHR login step was important in this study setting to acknowledge alert receipt, it was seemingly not used routinely by clinician participants. This can explain the important lag time between alert generation and alert acknowledgment in the pager arm.
Standard text messaging is discouraged for transfer of confidential information in the healthcare setting due to lack of HIPAA compliance. Advanced two-way systems, such as cellphone or smartphone text message, save time, increase efficiency, facilitate better patient care, decrease callbacks, and reduce interruptions of educational activities .
Smartphones and tablets were found to be a reliable method to deliver time-critical information. However, transferred and encrypted PHI can be viewed by the host server [25, 31] which can lead to HIPAA violation. A secure, VPN-based connection was used in this study, enabling encrypted HIPAA-compliant transmission of PHI. A VPN connection had to be established prior the launch of the Application. However, of 22 correctly assigned alerts, none of them reached the designated clinician because the algorithm was not able to detect an established VPN connection or a running specific application on the iOS device at time of alert. At our institution a VPN connection establishes automatically, it usually only requires to be set up once and then upon entering the building credentials are automatically authenticated. This points our attention to the lack of running application as the most plausible culprit. Several possibilities would explain lack of running application include: failure to install the lasted application, failure to configure their application to allow notifications and failure to launch the application at the beginning of their shift. In a busy clinical environment adding one more step like launching an application during hand-off communication may get overlooked.
Personal mobile devices in the professional settings are used mostly for non-urgent and less time-sensitive purposes and are not considered as primary messengers. One concern that would remain would be the competing priorities of other personal alerts, email, texts or pop-ups from personal business. Recent study  has been shown that mobile device notifications were disruptive and negatively influenced on the performance during attention-demanding task, even when participants did not directly interacted with a mobile device during the task.
Timely acknowledgement of EHR-embedded notifications requires constant physical proximity and the use of personal credentials to log in to the system, or again the use of VPN connections through portable devices. EHR-embedded alerts were studied using computerized prescription order entry (CPOE) systems. A median lag time for the interrupting alerts for drug-drug interactions was 8 seconds and varied from 1 second to 34 mins . In our study for EHR embedded alerts the minimal time to acknowledgment was 14 mins and maximal 34 hours (for one participant). Theoretically any alert sent within an EHR has an instant delivery, and can have a reliable feedback. This is one of strengths of this type of delivery method. As the accuracy of electronically generated alerts is imperfect, the feedback loop for any clinical notifications can help not only to ameliorate quality of care, but also to optimize and diminish alert fatigue [33, 34]. However, working stations with EHR-embedded notifications were inappropriate to display time critical notification messages in the ICU settings  due to limited duration of login authentication intervals, which delayed acknowledgements. ICU providers may not have the ability to monitor the EHR for new alerts, so in this scenario having the pager system seems to be the most reliable from studied methods.
Pager devices remain a gold standard for urgent and non-urgent notifications. Reported use of this highly reliable technology is about 90% out of 200 hospitals . However, the cost of pager mediated alert delivery method seems to be almost 2 times higher than the cost of secure messaging apps.
The urgency and prioritization of transmission using different methods of notification delivery is important to address in the scope of the phenomenon of alert fatigue. However, this study was not able to detect any difference in the fatigue of clinician participants. This was partly the result of a small number of observations and lack of validated methodology to assess alert fatigue in the ICU setting.
This study has several limitations. (1) This was a single-center study at an academic medical center. Well-established biases and potential confounders are known to be present with this particular study design. (2) Simple randomization without blocking was used, which may have led to unequal allocation due to small sample size. (3) HIPAA compliant mechanisms for iOS-based mobile devices are institution-specific and have led to technical failure. Thus, this aspect of the results of this study may not be completely generalizable. (4) Due to iOS arm failure the study was underpowered. (5) The Samn-Perelli scale is a subjective tool and thus may not be sensitive enough to fully address all aspects of alert fatigue. (6) Time to acknowledgement as it was designed, probably does not represent the time when clinicians became aware of the alert, due to the human-factors barriers to implementation of alert studies in the clinical setting. This phenomenon was also explored in a parallel study in the ICU setting (Harrison AM, Thongprayoon C, Aakre C, Jeng J, Dziadzko MA, Gajic O, Pickering B, Herasevich V: Barriers to implementation of an automated severe sepsis alert system in the ICU setting, submitted).
This study did not determine the optimal method of urgent alerts in the ICU settings using HIPAA compliant transmission protocol. Personal iOS-based devices were not reliable tools to deliver alerts using HIPAA compliant methods due to specific network connection requirements. Further investigation is thus necessary to understand human and technical barriers to implementation of prominent wireless electronic technology in the healthcare setting.
Computerized physician order entry
Electronic health record
Health Insurance Portability and Accountability Act
Intensive care unit
iPhone operative system
Institutional review board
Protected health information
Rapid response team
Sepsis survival campaign
Virtual private network
Top 10 Patient Safety Concerns for Healthcare Organisations. Plymouth Meeting: ECRI Institute. 2015. https://www.ecri.org/EmailResources/PSRQ/Top10/2015_Patient_Safety_Top10.pdf.
New R3 Report addresses NPSG.06.01.01. Jt Comm Perspect 2014, 34(2):1
AAMI. 2011 Clinical Alarms Summit. In: AAMI, editor. 2011 Clinical Alarms Summit. Herndon: AAMI; 2011.
Losee Jr RM. Minimizing information overload: The ranking of electronic messages. J Inform Sci. 1989;15(3):179–89.
Donchin Y, Gopher D, Olin M, Badihi Y, Biesky M, Sprung CL, Pizov R, Cotev S. A look into the nature and causes of human errors in the intensive care unit. Crit Care Med. 1995;23(2):294–300.
Zhang J, Johnson TR, Patel VL, Paige DL, Kubose T. Using usability heuristics to evaluate patient safety of medical devices. J Biomed Inform. 2003;36(1–2):23–30.
Gill PS, Kamath A, Gill TS. Distraction: an assessment of smartphone usage in health care work settings. Risk Manag Healthc Policy. 2012;5:105–14.
Marian AA, Dexter F, Tucker P, Todd MM. Comparison of alphabetical versus categorical display format for medication order entry in a simulated touch screen anesthesia information management system: an experiment in clinician-computer interaction in anesthesia. BMC Med Inform Decis Mak. 2012;12:46.
Top 10 Health Technology Hazards Report for 2014. ECRI Institute 2014. https://www.ecri.org/Resources/Whitepapers_and_reports/2014_Top_10_Hazards_Executive_Brief.pdf. Accessed 12 Dec 2016.
Bates DW, Cullen DJ, Laird N, Petersen LA, Small SD, Servi D, Laffel G, Sweitzer BJ, Shea BF, Hallisey R, et al. Incidence of adverse drug events and potential adverse drug events. Implications for prevention. ADE Prevention Study Group. JAMA. 1995;274(1):29–34.
Bates DW, Teich JM, Lee J, Seger D, Kuperman GJ, Ma'Luf N, Boyle D, Leape L. The impact of computerized physician order entry on medication error prevention. J Am Med Inform Assoc. 1999;6(4):313–21.
Raschke RA, Gollihare B, Wunderlich TA, Guidry JR, Leibowitz AI, Peirce JC, Lemelson L, Heisler MA, Susong C. A computer alert system to prevent injury from adverse drug events: development and evaluation in a community teaching hospital. JAMA. 1998;280(15):1317–20.
Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record-based settings. JAMA Intern Med. 2013;1:3.
Hodgetts HM, Jones DM. Reminders, alerts and pop-ups: The cost of computer-initiated interruptions, vol. 4550. Beijing: LNCS; 2007. p. 818–826.
Glassman PA, Belperio P, Simon B, Lanto A, Lee M. Exposure to automated drug alerts over time: effects on clinicians’ knowledge and perceptions. Med Care. 2006;44(3):250–6.
van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13(2):138–47.
Brixey JJ, Tang Z, Robinson DJ, Johnson CW, Johnson TR, Turley JP, Patel VL, Zhang J. Interruptions in a level one trauma center: a case study. Int J Med Inform. 2008;77(4):235–41.
Paxton F, Heaney D, Howie J, Porter BA. A study of interruption rates for practice nurses and GPs. Nurs Stand. 1996;10(43):33–6.
O'Connor C, Friedrich JO, Scales DC, Adhikari NK. The use of wireless e-mail to improve healthcare team communication. J Am Med Inform Assoc. 2009;16(5):705–13.
Rohrig R, Meister M, Michel-Backofen A, Sedlmayr M, Uphus D, Katzer C, Rose T. Online guideline assist in intensive care medicine--is the login-authentication a sufficient trigger for reminders? Stud Health Technol Inform. 2006;124:561–8.
Stothart C, Mitchum A, Yehnert C. The attentional cost of receiving a cell phone notification. J Exp Psychol Hum Percept Perform. 2015;41(4):893–7.
Peake SL, Delaney A, Bailey M, Bellomo R, Cameron PA, Cooper DJ, Higgins AM, Holdgate A, Howe BD, Webb SA, et al. Goal-directed resuscitation for patients with early septic shock. N Engl J Med. 2014;371(16):1496–506.
Yealy DM, Kellum JA, Huang DT, Barnato AE, Weissfeld LA, Pike F, Terndrup T, Wang HE, Hou PC, LoVecchio F, et al. A randomized trial of protocol-based care for early septic shock. N Engl J Med. 2014;370(18):1683–93.
Harrison AM, Thongprayoon C, Kashyap R, Chute CG, Gajic O, Pickering BW, Herasevich V. Developing the surveillance algorithm for detection of failure to recognize and treat severe sepsis. Mayo Clinic Proceedings. 2015;90(2):166–75.
Karasz HN, Eiden A, Bogan S. Text messaging to communicate with public health audiences: how the HIPAA Security Rule affects practice. Am J Public Health. 2013;103(4):617–22.
Mouncey PR, Osborn TM, Power GS, Harrison DA, Sadique MZ, Grieve RD, Jahan R, Harvey SE, Bell D, Bion JF, et al. Trial of early, goal-directed resuscitation for septic shock. N Engl J Med. 2015;372(14):1301–11.
Samn SW, Perelli LP. Estimating aircrew fatigue: a technique with implications to airlift operations. In. Brookes AFB. TX: USAF School of Aerospace Medicine; 1982.
Myers JA, Haney MF, Griffiths RF, Pierse NF, Powell DM. Fatigue in Air Medical Clinicians Undertaking High-acuity Patient Transports. Prehosp Emerg Care. 2015;19(1):36–43.
Moss M, Trow TK, Clardy N. Differences in the response times of pages originating from the ICU. Chest. 1999;116(4):1019–24.
Epstein RH, Dexter F, Rothman B. Communication latencies of wireless devices suitable for time-critical messaging to anesthesia providers. Anesth Analg. 2013;116(4):911–8.
Brooks A. Healthcare texting in a HIPAA-compliant environment. American Academy of Orthopaedic Surgeons News. Available at: www.aaosorg/news/aaosnow/aug12/managing5asp. Accessed 4 Aug 2014.
McDaniel RB, Burlison JD, Baker DK, Hasan M, Robertson J, Hartford C, Howard SC, Sablauer A, Hoffman JM. Alert dwell time: introduction of a measure to evaluate interruptive clinical decision support alerts. J Am Med Inform Assoc. 2015.
Kashiouris M, O'Horo JC, Pickering BW, Herasevich V. Diagnostic performance of electronic syndromic surveillance systems in acute care: a systematic review. Appl Clin Inform. 2013;4(2):212–24.
Fedosov V, Dziadzko MA, Dearani JA, Brown DR, Pickering BW, Herasevich V. Decision Support Tool to Improve Glucose Control Compliance After Cardiac SurgeryAACN. Adv Crit Care. 2016;27:274–282.
TigerText I: The Hidden Costs of Pagers in Healthcare: How Outmoded Technology Is Draining Healthcare IT Budgets. 2016. http://www.tigertext.com/wp-content/uploads/Report-Hidden-Cost-of-Pagers.pdf.
The authors thank Sergiu Plamadeala from the Mayo Clinic Department of Information Technology, who developed and implemented the randomization and alert distribution algorithm.
This research was funded by Critical Care Research Subcommittee, Mayo Clinic, MN.
Availability of data and materials
No additional data are available, no data will be shared. All data are the property of the Mayo Clinic.
The concept of study VH and MAD. MAD, PMF, AMH conduct of the study and data collection. AMH, ICT, BWP, PMF, VH and MAD contributed on the design, analysis, manuscript writing and critical revision and final approval.
No competing financial interests exist.
Consent for publication
Ethics approval and consent to participate
This study was approved by the Mayo Clinic Institutional Review Board (IRB) as minimal-risk, since there were no changes in usual protocols of patient care. Oral consent was obtained from participating clinicians.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Dziadzko, M.A., Harrison, A.M., Tiong, I.C. et al. Testing modes of computerized sepsis alert notification delivery systems. BMC Med Inform Decis Mak 16, 156 (2016). https://doi.org/10.1186/s12911-016-0396-y
- Methods of alert delivery
- Alert fatigue
- Decision support system