The study was approved by Mayo Clinic Institutional Review Board for the use of existing medical records of patients who gave prior research authorization.
Study population
The derivation and validation subsets were obtained retrospectively from two critical care units at Mayo Clinic in Rochester, Minnesota. This heterogeneous population of patients in the medical and surgical ICUs was admitted from January 1, 2010, through December 31, 2011. The total cohort included 6,714 patients. The cohort was reduced to contain only the 2,689 patients who received both invasive and non-invasive mechanical ventilation on their first ICU admission during the study period, excluding 101 patients who did not provide prior research authorization (Figure 1). From this cohort, five patients were excluded because of age restriction (<18 years). A subset of this cohort was used for derivation and consisted of 450 randomly selected patients treated in 2010. This group was further reduced to 83 patients who, with the same criteria, were intubated emergently in the ICU (invasive). The search algorithm was then validated against 450 randomly selected patients treated in 2011, and subsequently reduced to 71 patients with the same criteria from the above-mentioned cohort using JMP statistical software (version 9.0; SAS Institute Inc). The selection of 450 patients for the derivation and validation cohorts was made to minimize the burden for the investigators during manual medical record review while ensuring a robust sample size for both subsets. We used the search algorithm published previously for identification of emergent endotracheal intubations [1].
Manual data extraction strategies (Reference Standard)
The medical records of the derivation and validation groups were manually reviewed by two independent critical care clinicians (N.J.S and V.M.V). Each record was evaluated for mechanical ventilation initiation, and four screening variables were recorded: intubation procedure note time, end-tidal CO2 recording time, peak inspiratory pressure (PIP) time, and positive end-expiratory pressure (PEEP) time. Given that the variables within the present study are only recorded in association with ventilation, their first appearance was recorded as the time point of ventilation initiation. It was necessary to record these variables because no reference standard exists for mechanical ventilation initiation when reviewed retrospectively. The variables analyzed in the current manuscript are translated into time points through the use of our institution’s ICU database [7]. The ventilator parameters are automatically downloaded within the database once the patient is connected to the ventilator. Thus, the mechanical ventilator parameters registered within the database are translated into time points in real-time. While the best outcome metric for a time point would clearly be prospective data capture, this was a retrospective study. PIP was the reference standard adopted in the present study because 1) the difference in mechanical ventilation parameters and intubation procedure note time varied by more than 60 minutes in 40% of the derivation subset, 2) end-tidal CO2 monitoring was present in roughly 15% of the derivation subset, and 3) PEEP was recorded with both noninvasive and invasive modes of mechanical ventilation vs PIP, which more accurately reflected invasive mechanical ventilation. The review included emergent and nonemergent intubations that occurred in the ICU. The review of non-emergent intubations was included as a systematic check to ensure the electronic search algorithm we published previously was accurate. Disagreements between the two reviewers were settled by a third reviewer (V.H.). The research team involved in manual data extraction was not aware of the automated electronic search strategy results.
Automated electronic search strategy
The present retrospective study used the ICU data mart of METRIC [7]. The data mart contains such patient information as demographic characteristics, diagnoses, laboratory testing, flow sheets, clinical testing, and pathologic data gathered from various resources within the institution. It allows the application of search algorithms, such as the one described herein. Data is automatically captured from ventilators through the EMR. The ICU data mart has been validated and is reliable [7, 8].
To develop the electronic search strategy, we first included such variables as intubation procedure note and end-tidal CO2 time to the ‘search query’ from our ICU database. Additional criteria consisted of mechanical ventilation parameters, such as PIP and PEEP times. The electronic search strategy was continuously refined through use of one variable as either intubation procedure note time, end-tidal CO2 time, PIP time, or PEEP time, or any combination of these variables. The variable PEEP was chosen for the automated electronic search rather than the PIP variable or the other parameters for the following reasons: 1) the PEEP variable had a nearly complete dataset unlike the PIP and end-tidal CO2 variables which was missing a significant portion of data within data mart and; 2) PEEP had a more accurate agreement with manual medical record review (using PIP as the reference standard) to within five minutes versus the intubation procedure note. Therefore, the final search algorithm consisted of PEEP with a five-minute restriction and the utilization of datasets with less than 15% missing data as an additional restriction. We choose an error of five minutes as the maximum acceptable error in order to precisely identify the start of mechanical ventilation. Parameters that differed more than five minutes may not accurately capture hemodynamic disturbances associated with emergent intubations.To validate the automated electronic search algorithm, an overall percent agreement plot was constructed by comparing the automated electronic search to the reference standard of comprehensive manual medical record review. When derived, the final electronic search algorithm consisting of PEEP was applied to an independent validation subset. A κ value was generated for both subsets. The automatic search strategy was done independently by a critical care research physician (R.K.). (For a flow chart of the process, see Figure 1.)
Statistical analyses
Given that our outcome data are continuous, as time in minutes, an overall percent agreement between the search algorithm and manual medical record review was recorded. We agreed that a difference of five minutes between our EMR search strategy and the reference standard was acceptable. A Bland-Altman plot would have been ideal with the outcome variable; however, the outcome metric of interest was a time point, and thus an average of time when the majority of values differed by less than one minute did not make logical sense. Therefore, we report an overall percent agreement plot rather than a Bland-Altman plot.
Furthermore, we do not report on sensitivity and specificity of the current algorithm. Given that no gold standard exists and that the reference standard we adopted may be different in other institutions where, for example, end-tidal CO2 is continuously recorded, we felt it was misleading to report on sensitivity and specificity of the current algorithm. If we were to report on sensitivity and specificity, we would have then adopted the PIP variable as the gold standard. However, this may not be the consensus in the greater scientific community.