Prediction of adverse cardiac events in emergency department patients with chest pain using machine learning for variable selection
© Liu et al.; licensee BioMed Central Ltd. 2014
Received: 9 July 2014
Accepted: 18 August 2014
Published: 23 August 2014
The key aim of triage in chest pain patients is to identify those with high risk of adverse cardiac events as they require intensive monitoring and early intervention. In this study, we aim to discover the most relevant variables for risk prediction of major adverse cardiac events (MACE) using clinical signs and heart rate variability.
A total of 702 chest pain patients at the Emergency Department (ED) of a tertiary hospital in Singapore were included in this study. The recruited patients were at least 30 years of age and who presented to the ED with a primary complaint of non-traumatic chest pain. The primary outcome was a composite of MACE such as death and cardiac arrest within 72 h of arrival at the ED. For each patient, eight clinical signs such as blood pressure and temperature were measured, and a 5-min ECG was recorded to derive heart rate variability parameters. A random forest-based novel method was developed to select the most relevant variables. A geometric distance-based machine learning scoring system was then implemented to derive a risk score from 0 to 100.
Out of 702 patients, 29 (4.1%) met the primary outcome. We selected the 3 most relevant variables for predicting MACE, which were systolic blood pressure, the mean RR interval and the mean instantaneous heart rate. The scoring system with these 3 variables produced an area under the curve (AUC) of 0.812, and a cutoff score of 43 gave a sensitivity of 82.8% and specificity of 63.4%, while the scoring system with all the 23 variables had an AUC of 0.736, and a cutoff score of 49 gave a sensitivity of 72.4% and specificity of 63.0%. Conventional thrombolysis in myocardial infarction score and the modified early warning score achieved AUC values of 0.637 and 0.622, respectively.
It is observed that a few predictors outperformed the whole set of variables in predicting MACE within 72 h. We conclude that more predictors do not necessarily guarantee better prediction results. Furthermore, machine learning-based variable selection seems promising in discovering a few relevant and significant measures as predictors.
KeywordsTriage Major adverse cardiac events Scoring system Random forest
Chest pain is one of the leading causes of visits to the emergency department (ED). Patients with chest pain present a wide range of risk for death and adverse cardiac events. Of great concern is the risk of cardiac arrest that accounts for the majority of early deaths in patients with acute myocardial infarction (AMI) and other adverse cardiac events. Significant hospital resources are dedicated to these high risk patients. Therefore, accurate risk stratification of chest pain patients for the prediction of major adverse cardiac events (MACE) could play an essential role in supporting clinical decisions that allow timely intervention for preventable and treatable complications. It could also allow management of low-risk patients without unnecessary admissions, investigations and monitoring, hence reducing the strain on limited ED resources. Such a risk stratification tool would be useful for chest pain patients presenting to the ED.
Various systems exist for stratifying the risk of acute coronary syndromes (ACS)[4–6]. The thrombolysis in myocardial infarction (TIMI) score and the Global Registry of Acute Coronary Events (GRACE) score were developed to predict the risk of death, reinfarction and revascularization. TIMI and GRACE scores have been validated on an unselected population of chest pain patients at the ED for predicting adverse events. However, both risk scores are often not applicable at the first presentation as not all the variables are measured routinely in the ED. The modified early warning score (MEWS) is another popular tool in Commonwealth countries to identify patients at risk of deterioration[8, 9]. Its process of risk evaluation involves assessment of vital signs and accurate prediction requires some prior training. Furthermore, the above mentioned scoring systems use predefined predictive variables, prohibiting them from adopting new variables that are clinically significant for risk stratification. Several limitations have been reported in current risk scores for prediction of cardiovascular complications[11, 12].
Most scoring systems use clinical vital signs such as heart rate, respiratory rate, blood pressure, temperature and pulse oximetry for risk stratification model derivation, but these physiological measures might not correlate well with short- or long-term clinical outcomes[14, 15]. Furthermore, conventional statistical scoring systems are usually not readily adaptable to new variables. Motivated by the flexibility of machine learning (ML) techniques, we have previously developed an intelligent scoring system for predicting acute cardiac complications and discovered that rapid non-invasive bedside heart rate variability (HRV) combined with clinical signs showed improved prediction performance when compared with the MEWS. However, redundant information likely exists within these predictive variables as not all the variables may contribute to the prediction. Machine learning-based variable selection has been widely used in bioinformatics but received little attention in medical research where traditional statistical methods play a dominant role. In this study, we aim to discover the most relevant variables from HRV and clinical signs for the prediction of MACE using machine learning. We will compare the aforementioned ML score using the selected variables with TIMI and MEWS scores by conducting the receiver operating characteristic (ROC) analysis for performance evaluation.
List of HRV parameters and their definitions
HRV parameter (unit)
Average width of the RR interval
Standard deviation of all RR intervals
Average of the instantaneous heart rate (HR)
Standard deviation of the instantaneous HR
Root mean square of differences between adjacent RR intervals
Number of consecutive RR intervals differing by more than 50 ms
Number and percentage of consecutive RR intervals differing by more than 50 ms
Total number of all RR intervals divided by the height of the histogram of intervals
Baseline width of a triangle fit into the RR interval histogram using a least squares
LF power (ms2)
Power in low frequency range 0.04-0.15 Hz
HF power (ms2)
Power in high frequency range 0.15-0.40 Hz
Total power (ms2)
Total power estimated from RR intervals
LF norm (n.u.)
LF power in normalized units: LF/(Total power-VLF) × 100
HF norm (n.u.)
HF power in normalized units: HF/(Total power-VLF) × 100
Ratio of LF power to HF power
Patients were followed up to discharge from hospital or in-hospital death. The primary outcome was defined as a composite of four major adverse cardiac events (MACE) within 72 h of patient’s arrival at the ED. MACE in this study included death, cardiac arrest, sustained ventricular tachycardia (VT), and hypotension requiring inotropes or intra-aortic balloon pump (IABP) insertion. The outcomes were retrieved from the ED charts and outcome decisions were made by physicians.
Ethics approval was obtained from the Singapore Health Services (SingHealth) Centralized Institutional Review Board (CIRB Ref: 2009/871/C) with a waiver of patient consent. This study was conducted from March 2010 to April 2012 at the Emergency Department (ED) of Singapore General Hospital (SGH).
Predictive variable selection
Variable selection in statistics and machine learning is a process of determining a subset of relevant variables for model construction. Irrelevant variables usually do not contribute to model building and may even degrade the prediction performance. Automatic variable selection methods have been widely adopted in clinical studies such as for predicting acute myocardial infarction mortality.
We have previously studied the use of combined HRV parameters and clinical signs for patient outcome prediction[17, 24] where each variable was assumed to equally contribute to model building. These studies could be refined by means of variable selection such that predictive variables are individually evaluated for their correlations to the primary outcome. Refinement of the model required a novel variable selection method as the distribution of our data was imbalanced; relatively few patients met the primary outcome (29 out of 702). Data imbalance compromises the performance of most machine learning algorithms.
In this study, we proposed a novel variable selection framework based on ensemble learning, in which random forests (RF) was chosen as the independent variable selector for creating the decision ensemble, where the number of trees was 500. RF is an ensemble learning method for classification and regression. It combines many binary decision trees built using several bootstrapped learning samples and choosing randomly at each node a subset of explanatory variables. The RF approach has been shown to be effective in variable selection[27, 28].
The classification and regression training package in R programming language was used to implement RF for variable selection. Data were imported into R from a CSV file where all variables were in continuous format and the primary outcome was in categorical format.
Risk score prediction
In this study only a set of the selected variables was fed into the learning model for risk prediction, which was different from our previous ML method that utilized all variables (15 HRV parameters and eight clinical signs) as the input. The output risk score ranges from 0 to 100 with 0 indicating no risk and 100 indicating the highest risk of MACE within 72 h. To compare the ML score with existing clinical scores, TIMI and MEWS scores were also derived for each patient.
Continuous variables were presented as means (standard deviation) or medians (interquartile range) and were analyzed using the Mann–Whitney test. SPSS version 17.0 (SPSS Inc., Chicago, IL), MATLAB R2009a (Mathworks, Natick, MA), and R version 2.15.1 (R Foundation, Vienna, Austria) were employed for data analysis.
Performance evaluation was carried out with the leave-one-out cross-validation (LOOCV) strategy to get unbiased estimation of the model performance. In this study with 702 samples, 702 iterations were required for performance evaluation. In iteration, one sample was used as the testing sample while the remaining 701 samples were used for training. The risk score prediction process was repeated 702 times so that each sample was tested individually. Risk scores were then obtained for the entire dataset and a threshold was derived to report sensitivity and specificity.
A package developed in MATLAB was used to analyze ECG for HRV, calculate risk scores, and report prediction performance measures. To further evaluate differences in discrimination between models, pair-wise AUC comparisons using bootstrap method were performed with a ROC comparison package. Statistical significance was set at p-value <0.05.
Results and discussion
Characteristics of the recruited patients
No MACE within 72 h
MACE within 72 h
(N = 673)
(N = 29)
Mean age (SD)
Ischemic heart disease
Chronic renal failure
Congestive heart failure
Outcomes of patients with MACE within 72 h of arrival at the ED
Number of patients (%)
One or more severe complications
Sustained ventricular tachycardia
Hypotension requiring inotropes or IABP insertion
Measurements of HRV parameters and clinical signs of recruited patients
No MACE within 72 h (N = 673)
MACE within 72 h (N = 29)
Glasgow Coma Scale
15 (15 to 15)
15 (15 to 15)
Pulse rate (beats/minute)
Respiratory rate (breaths/minute)
Systolic BP (mmHg)
Diastolic BP (mmHg)
Oxygen saturation (%)
2 (0 to 4)
3 (0 to 5)
LF power (ms2)
HF power (ms2)
Total power (ms2)
LF power norm (n.u.)
HF power norm (n.u.)
Top selected variables and their prediction performance
Selected variables for model building
AUC 95% CI
0.553 - 0.773
0.656 - 0.862
SBP, avHR, aRR
0.716 - 0.908
SBP, avHR, aRR, DBP
0.671 - 0.874
SBP, avHR, aRR, DBP, TI
0.660 - 0.865
SBP, avHR, aRR, DBP, TI, LF/HF
0.672 - 0.875
SBP, avHR, aRR, DBP, TI, LF/HF, HF norm
0.664 - 0.869
SBP, avHR, aRR, DBP, TI, LF/HF, HF norm, LF norm
0.666 - 0.870
Furthermore, a linear logistic regression model with forward selection was created where the predicted score ranges from 0 to 1. Six variables including Glasgow Coma Scale, respiratory rate, DBP, pain score, STD, and avHR were chosen for model building. The regression score yielded an AUC of 0.632 (95% CI: 0.564 - 0.7). A cutoff score of 0.11 gave a sensitivity of 58.0% (95% CI: 47.3% - 68.8%) and specificity of 55.7% (95% CI: 51.8% - 59.6%), respectively.
Pair-wise discrimination comparison of AUC values of different risk prediction models
ML (Top 3)
ML (All 23)
ML (Top 3)
ML (All 23)
This study is a continuation of our previous work on applying both HRV and clinical signs for outcome prediction. In this study, we investigated the use of variable selection to choose the most relevant predictors of MACE within 72 h. The ML method was able to more accurately identify patients who met the primary outcome than MEWS and TIMI scores. Only 3 variables including systolic BP, HRV parameters avHR and aRR were required to achieve the best risk prediction performance. While 5-min ECG is not the standard of care, we have demonstrated its feasibility and effectiveness of risk prediction. The analysis of short recordings of ECG HRV has potential to benefit triage in the ED, where timely response is essential. Furthermore, only SBP instead of all eight clinical signs is required for risk prediction, which dramatically simplifies the scoring system and saves a lot of time in measurement.
Many scoring systems have been proposed to measure the risk of adverse events in acute patients. However, no study has yet been done on performance-driven model building with random forest (RF)-based variable selection. With the RF technique, we derived a novel variable selection process integrating the strengths of both machine learning and statistics. Artificial intelligence has been used in the medical field for many years[16, 33]. Due to the fact that positive samples (patients with MACE) are a small proportion of the dataset, common machine learning-based model building usually fails to achieve effective risk analysis. The trained model tends to over-fit due to the predominance of negative samples (patients without MACE). Techniques for handling imbalanced data has been well summarized in He and Garcia. Three major techniques are widely used, namely: sampling methods, cost-sensitive methods, and kernel-based and active learning methods. Each type of method has its pros and cons. Because of its simple yet effective structure, the sampling method was adopted together with random forest to create a novel variable selection algorithm.
A total of 23 variables consisting of 15 HRV and eight clinical signs were investigated for their associations with the primary outcome. The results showed that several HRV parameters and clinical signs were significant in predicting MACE within 72 h. It has also been reported that patients with “normal” vital signs may be more ill than they appeared, and therefore predictors other than traditional vital signs are needed to increase the chance of detecting critically ill patients. We have previously discovered that a combination of HRV and clinical signs outperformed HRV alone and clinical signs alone[17, 24]. It is shown in this study that a few significant predictors were able to improve performance as well as to shorten decision making time. Both RF-based and statistics-based (by means of p-value) variable selection methods were able to pick up significant variables such as SBP, avHR and aRR. It is worth noting that avHR (average of the instantaneous heart rate) and aRR (average width of the RR interval) were correlated and both were selected for building the scoring model. In a conventional statistics approach, these two variables will not appear in the same prediction model. However, in a machine learning method, variable selection is performance-driven where both prediction performance and statistical significance play important roles in the selection process.
With these selected variables, triage (in terms of risk prediction) can be done within a shorter time to determine priority of patients’ treatments. Fast response is of great value in triage because resources are usually limited for all patients to receive immediate attention. In this scenario, a few yet significant variables will benefit the triage from two aspects. From software modeling perspective, they will maintain a balance between model complexity and prediction performance because including too many predictors may lead to a loss in precision, whereas omitting important variables may result in biased prediction. From hardware design perspective, fewer variables that require lesser time and effort to obtain will make a triage device simpler and faster in real-time usage.
Our study has several limitations. Since this is a single-center pilot study at an acute tertiary hospital in Singapore, the number of recruited patients is small and our findings may not be generalizable to other populations. The endpoints in this study are heterogeneous while the event rate is still low. This creates difficulty in associating predictive variables with specific outcomes. The TIMI outcomes (death, AMI, and revascularization within 30 days) might be adopted to standardize model derivation and for comparison across several risk scoring methods in chest pain patients presented to the ED.
While we adopt random forest for variable selection, there are several other techniques available for selecting significant variables. Small sample size may lead to overestimation even though we have adopted the leave-one-out cross-validation scheme. The ideal way is to validate the method on a separate dataset. Given more available data in the future, we will be able to derive a model from one dataset and validate its performance on another. Finally, in variable selection we created the subsets by using 1:1 ratio between MACE and no MACE patient groups, which may potentially lead to bias.
In future work, exploration of other potential predictive variables will be considered. Gender has been reported to be useful for patient outcome prediction; 12-lead ECG changes and bedside qualitative cardiac troponin could also be added to the algorithm to enhance prediction performance. Also, including demographic and medical history variables in the predictive model may be useful. In this study, eight top-ranked variables were chosen where the number was empirically determined to create a trade-off between accuracy and efficiency. Therefore, finding a method to decide the optimal number of variables is necessary so that the proposed variable selection method can be generalized in other applications.
This study expands our understanding that only a few predictors are needed in risk stratification of MACE within 72 h. With the proposed variable selection method, a machine learning scoring system outperforms traditional risk stratification systems such as TIMI and MEWS. We have seen that blood pressure and some HRV parameters achieved better prediction performance than a larger set of clinical signs and HRV parameters. It shows that more predictors do not necessarily guarantee better prediction results. Furthermore, variable selection presents potential to simplify scoring systems with just a few relevant measures as predictors, and the machine learning based scoring system demonstrates its adaptability to changes of variables.
Acute coronary syndrome
Acute myocardial infarction
Area under ROC curve
Coronary artery bypass graft
Centralized Institutional Review Board
Glasgow coma scale
Global registry of acute coronary events
Heart rate variability
Intra-aortic balloon pump
Major adverse cardiac events
Modified early warning score
Percutaneous coronary intervention
Receiver operating characteristic
Singapore general hospital
Singapore health services
- SpO2 :
Support vector machine
Thrombolysis in myocardial infarction
Very low frequency
We would like to thank the contributions from doctors and nurses from the Department of Emergency Medicine, Singapore General Hospital for their assistance in this study.
- McCaig LF, Burt CW: National Hospital Ambulatory Medical Care Survey: 2001 emergency department summary. Adv Data. 2003, 335: 1-29.PubMedGoogle Scholar
- Anderson JL, Adams CD, Antman EM, Bridges CR, Califf RM, Casey DE, Chavey WE, Fesmire FM, Hochman JS, Levin TN, Lincoff AM, Peterson ED, Theroux P, Wenger NK, Wright RS, Smith SC, Jacobs AK, Halperin JL, Hunt SA, Krumholz HM, Kushner FG, Lytle BW, Nishimura R, Ornato JP, Page RL, Riegel B: ACC/AHA 2007 guidelines for the management of patients with unstable angina/non-ST-Elevation myocardial infarction: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines for the Management of Patients With Unstable Angina/Non-ST-Elevation Myocardial Infarction) developed in collaboration with the American College of Emergency Physicians, the Society for Cardiovascular Angiography and Interventions, and the Society of Thoracic Surgeons endorsed by the American Association of Cardiovascular and Pulmonary Rehabilitation and the Society for Academic Emergency Medicine. J Am Coll Cardiol. 2007, 50 (7): e1-e157.View ArticlePubMedGoogle Scholar
- Huikuri HV, Castellanos A, Myerburg RJ: Sudden death due to cardiac arrhythmias. N Engl J Med. 2001, 345 (20): 1473-1482.View ArticlePubMedGoogle Scholar
- Goldman L, Cook EF, Johnson PA, Brand DA, Rouan GW, Lee TH: Prediction of the need for intensive care in patients who come to the emergency departments with acute chest pain. N Engl J Med. 1996, 334 (23): 1498-1504.View ArticlePubMedGoogle Scholar
- Antman E, Cohen M, Bernink P, McCabe C, Horacek T, Papuchis G, Mautner B, Corbalan R, Radley D, Braunwald E: The TIMI risk score for unstable angina/non-ST elevation MI - A method for prognostication and therapeutic decision making. JAMA. 2000, 284 (7): 835-842.View ArticlePubMedGoogle Scholar
- Eagle KA, Lim MJ, Dabbous OH, Pieper KS, Goldberg RJ, Van de Werf F, Goodman SG, Granger CB, Steg PG, Gore JM, Budaj A, Avezum A, Flather MD, Fox KA: A validated prediction model for all forms of acute coronary syndrome: estimating the risk of 6-month postdischarge death in an international registry. JAMA. 2004, 291 (22): 2727-2733.View ArticlePubMedGoogle Scholar
- Lyon R, Morris AC, Caesar D, Gray S, Gray A: Chest pain presenting to the Emergency Department–to stratify risk with GRACE or TIMI?. Resuscitation. 2007, 74 (1): 90-93.View ArticlePubMedGoogle Scholar
- Subbe CP, Kruger M, Rutherford P, Gemmel L: Validation of a modified early warning score in medical admissions. QJM. 2001, 94 (10): 521-526.View ArticlePubMedGoogle Scholar
- Goldhill DR, McNarry AF, Mandersloot G, McGinley A: A physiologically-based early warning score for ward patients: the association between score and outcome. Anaesthesia. 2005, 60 (6): 547-553.View ArticlePubMedGoogle Scholar
- Ong MEH, Ng CHL, Goh K, Liu N, Koh ZX, Shahidah N, Zhang T, Fook-Chong S, Lin Z: Prediction of cardiac arrest in critically ill patients presenting to the emergency department using a machine learning score incorporating heart rate variability compared with the modified early warning score. Crit Care. 2012, 16 (3): R108-View ArticlePubMedPubMed CentralGoogle Scholar
- Manini AF, Dannemann N, Brown DF, Butter J, Bamberg F, Nagurney JT, Nichols JH, Hoffmann U, Rule-Out Myocardial Infarction U: Limitations of risk score models in patients with acute chest pain. Am J Emerg Med. 2009, 27 (1): 43-48.View ArticlePubMedPubMed CentralGoogle Scholar
- Hollander JE, Robey JL, Chase MR, Brown AM, Zogby KE, Shofer FS: Relationship between a clear-cut alternative noncardiac diagnosis and 30-day outcome in emergency department patients with chest pain. Acad Emerg Med. 2007, 14 (3): 210-215.View ArticlePubMedGoogle Scholar
- Ong MEH, Goh K, Fook-Chong S, Haaland B, Wai KL, Koh ZX, Shahidah N, Lin Z: Heart rate variability risk score for prediction of acute cardiac complications in ED patients with chest pain. Am J Emerg Med. 2013, 31 (8): 1201-1207.View ArticlePubMedGoogle Scholar
- Sanchis J, Bodi V, Nunez J, Bosch X, Lorna-Sorio P, Mainar L, Santas E, Minana G, Robles R, Llacer A: Limitations of clinical history for evaluation of patients with acute chest pain, non-diagnostic electrocardiogram, and normal troponin. Am J Cardiol. 2008, 101 (5): 613-617.View ArticlePubMedGoogle Scholar
- Hargarten KM, Aprahamian C, Stueven H, Olson DW, Aufderheide TP, Mateer JR: Limitations of prehospital predictors of acute myocardial infarction and unstable angina. Ann Emerg Med. 1987, 16 (12): 1325-1329.View ArticlePubMedGoogle Scholar
- Pearce CB, Gunn SR, Ahmed A, Johnson CD: Machine learning can improve prediction of severity in acute pancreatitis using admission values of APACHE II score and C-reactive protein. Pancreatology. 2006, 6 (1–2): 123-131.View ArticlePubMedGoogle Scholar
- Liu N, Lin Z, Cao J, Koh ZX, Zhang T, Huang G-B, Ser W, Ong MEH: An intelligent scoring system and its application to cardiac arrest prediction. IEEE Trans Inf Technol Biomed. 2012, 16 (6): 1324-1331.View ArticlePubMedGoogle Scholar
- Liu H, Yu L: Toward integrating feature selection algorithms for classification and clustering. IEEE Trans Knowl Data Eng. 2005, 17 (4): 491-502.View ArticleGoogle Scholar
- Huynh-Thu VA, Saeys Y, Wehenkel L, Geurts P: Statistical interpretation of machine learning-based feature importance scores for biomarker discovery. Bioinformatics. 2012, 28 (13): 1766-1774.View ArticlePubMedGoogle Scholar
- Katz MH: Multivariable analysis: a primer for readers of medical research. Ann Intern Med. 2003, 138 (8): 644-650.View ArticlePubMedGoogle Scholar
- Fawcett T: An introduction to ROC analysis. Pattern Recognit Lett. 2006, 27 (8): 861-874.View ArticleGoogle Scholar
- Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology: Heart rate variability: standards of measurement, physiological interpretation and clinical use. Circulation. 1996, 93 (5): 1043-1065.View ArticleGoogle Scholar
- Austin PC, Tu JV: Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. J Clin Epidemiol. 2004, 57 (11): 1138-1146.View ArticlePubMedGoogle Scholar
- Liu N, Lin Z, Koh ZX, Huang G-B, Ser W, Ong MEH: Patient outcome prediction with heart rate variability and vital signs. J Signal Process Syst. 2011, 64: 265-278.View ArticleGoogle Scholar
- He H, Garcia EA: Learning from imbalanced data. IEEE Trans Knowl Data Eng. 2009, 21 (9): 1263-1284.View ArticleGoogle Scholar
- Breiman L: Random forests. Mach Learn. 2001, 45 (1): 5-32.View ArticleGoogle Scholar
- Genuer R, Poggi J-M, Tuleau-Malot C: Variable selection using random forests. Pattern Recogn Lett. 2010, 31 (14): 2225-2236.View ArticleGoogle Scholar
- Hapfelmeier A, Ulm K: A new variable selection approach using Random Forests. Computational Statistics & Data Analysis. 2013, 60: 50-69.View ArticleGoogle Scholar
- Kuhn M: Building predictive models in R using the caret package. J Stat Softw. 2008, 28 (5): 1-26.View ArticleGoogle Scholar
- Han J, Kamber M: Data Mining: Concepts and Techniques. 2006, San Francisco: Morgan KaufmannGoogle Scholar
- Burges C: A tutorial on support vector machines for pattern recognition. Data Min Knowl Discov. 1998, 2 (2): 121-167.View ArticleGoogle Scholar
- Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Muller M: pROC: an open-source package for R and S + to analyze and compare ROC curves. BMC Bioinformatics. 2011, 12: 77-View ArticlePubMedPubMed CentralGoogle Scholar
- Hanson CW, Marshall BE: Artificial intelligence applications in the intensive care unit. Crit Care Med. 2001, 29 (2): 427-435.View ArticlePubMedGoogle Scholar
- Hong WL, Earnest A, Sultana P, Koh ZX, Shahidah N, Ong MEH: How accurate are vital signs in predicting clinical outcomes in critically ill emergency department patients. Eur J Emerg Med. 2013, 20 (1): 27-32.View ArticlePubMedGoogle Scholar
- Mahmood K, Eldeirawi K, Wahidi MM: Association of gender with outcomes in critically ill patients. Crit Care. 2012, 16 (3): R92-View ArticlePubMedPubMed CentralGoogle Scholar
- Polanczyk CA, Lee TH, Cook EF, Walls R, Wybenga D, Printy-Klein G, Ludwig L, Guldbrandsen G, Johnson PA: Cardiac troponin I as a predictor of major cardiac events in emergency department patients with acute chest pain. J Am Coll Cardiol. 1998, 32 (1): 8-14.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/14/75/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.