Clinical evaluation of an interoperable clinical decision-support system for the detection of systemic inflammatory response syndrome in critically ill children

Background Systemic inflammatory response syndrome (SIRS) is defined as a non-specific inflammatory process in the absence of infection. SIRS increases susceptibility for organ dysfunction, and frequently affects the clinical outcome of affected patients. We evaluated a knowledge-based, interoperable clinical decision-support system (CDSS) for SIRS detection on a pediatric intensive care unit (PICU). Methods The CDSS developed retrieves routine data, previously transformed into an interoperable format, by using model-based queries and guideline- and knowledge-based rules. We evaluated the CDSS in a prospective diagnostic study from 08/2018–03/2019. 168 patients from a pediatric intensive care unit of a tertiary university hospital, aged 0 to 18 years, were assessed for SIRS by the CDSS and by physicians during clinical routine. Sensitivity and specificity (when compared to the reference standard) with 95% Wald confidence intervals (CI) were estimated on the level of patients and patient-days. Results Sensitivity and specificity was 91.7% (95% CI 85.5–95.4%) and 54.1% (95% CI 45.4–62.5%) on patient level, and 97.5% (95% CI 95.1–98.7%) and 91.5% (95% CI 89.3–93.3%) on the level of patient-days. Physicians’ SIRS recognition during clinical routine was considerably less accurate (sensitivity of 62.0% (95% CI 56.8–66.9%)/specificity of 83.3% (95% CI 80.4–85.9%)) when measurd on the level of patient-days. Evaluation revealed valuable insights for the general design of the CDSS as well as specific rule modifications. Despite a lower than expected specificity, diagnostic accuracy was higher than the one in daily routine ratings, thus, demonstrating high potentials of using our CDSS to help to detect SIRS in clinical routine. Conclusions We successfully evaluated an interoperable CDSS for SIRS detection in PICU. Our study demonstrated the general feasibility and potentials of the implemented algorithms but also some limitations. In the next step, the CDSS will be optimized to overcome these limitations and will be evaluated in a multi-center study. Trial registration: NCT03661450 (ClinicalTrials.gov); registered September 7, 2018. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-021-01428-7.


INTRODUCTION 3
Scientific and clinical background, including the intended use and clinical role of the index test 3,4 4 Study objectives and hypotheses 3,4

Study design 5
Whether data collection was planned before the index test and reference standard were performed (prospective study) or after (retrospective study) 4 Participants 6 Eligibility criteria 5 + appendix 1   7 On what basis potentially eligible participants were identified (such as symptoms, results from previous tests, inclusion in registry) 5 + appendix 1 8 Where and when potentially eligible participants were identified (setting, location and dates) 5 + appendix 1 9 Whether participants formed a consecutive, random or convenience series 5 + appendix 1 Test methods 10a Index test, in sufficient detail to allow replication 5,6 10b Reference standard, in sufficient detail to allow replication 6 (lines 181-186)

11
Rationale for choosing the reference standard (if alternatives exist) 6,14,15 Where the full study protocol can be accessed 4

30
Sources of funding and other support; role of funders 23 Appendix 2: Details of changes to the published study protocol The study protocol has been previously published. For the conduction of the study, it was modified in three points:  Extension of exclusion criteria: Five recruited patients could not undergo standard data monitoring due to a special treatment (e. g. Berlin Heart*) and were therefore excluded.
 Definition of a shift: Correct detection of a SIRS episode was originally defined as SIRS identified within the correct shift. This would mean that a SIRS episode starting shortly after shift change but detected shortly before the official shift start would be evaluated as false negative. Likewise, the detection at the end of the latter shift would be evaluated as true positive, though it might be eight hours later. Consequently, to reach a more useful definition, shift was defined as +/-4 hours of the correct starting time.
 Adding another case for evaluation: The classical contingency table was amended by a fifth and a sixth case to include aspects of correct timing of episodes. The sixth case was not defined explicitly in the study protocol and covers situations in which a SIRS episode was identified correctly but an additional false positive episode was detected within the same patient's stay (true positive and false positive). The end of the SIRS episode is not taken into account. If the assessor documents additional SIRS episodes with less than 24 hours gap, these episodes will be treated as extension of the first episode and not evaluated as an additional "false positive".
(3  The implemented rules have triggered correctly and the database is also correct. However, clinical expertise has led to a different interpretation of the situation, so that the criteria that may apply do not apply after all (e.g. context factors exist that are not recorded by the system or certain correlations between parameters). 4 Poor data quality / wrong data base used The data base of the parameter is not correct, i.e. the data from the source system may have been transferred incorrectly to the new data repository or data from before the recording on the ICU were used.
The CDSS made the right decision, but did not get the right data / not enough data as input.
3 Body temperature 2 Heart rate 1 Incorrect rule triggering The rule has triggered incorrectly/ did not work or the data was evaluated incorrectly (= not according to the implemented rules). The implemented rules have triggered correctly and the database is also correct. However, clinical expertise has led to a different interpretation of the situation, so that the criteria that may apply do not apply after all (e.g. context factors exist that are not recorded by the system or certain correlations between parameters).

9
Additional, only for primary outcome measure on the level of patient Incorrect start time The SIRS criteria were detected correctly but episode was detected too late/too early (+/-4 hours).

3
Too late The patient's stays were classified as false negative (case 3) 2 Too early The patient's stay was classified as false positive/true positive (case 6)