Application of a practice-based approach in variable selection for a prediction model development study of hospital-induced delirium

Background Prognostic models of hospital-induced delirium, that include potential predisposing and precipitating factors, may be used to identify vulnerable patients and inform the implementation of tailored preventive interventions. It is recommended that, in prediction model development studies, candidate predictors are selected on the basis of existing knowledge, including knowledge from clinical practice. The purpose of this article is to describe the process of identifying and operationalizing candidate predictors of hospital-induced delirium for application in a prediction model development study using a practice-based approach. Methods This study is part of a larger, retrospective cohort study that is developing prognostic models of hospital-induced delirium for medical-surgical older adult patients using structured data from administrative and electronic health records. First, we conducted a review of the literature to identify clinical concepts that had been used as candidate predictors in prognostic model development-and-validation studies of hospital-induced delirium. Then, we consulted a multidisciplinary task force of nine members who independently judged whether each clinical concept was associated with hospital-induced delirium. Finally, we mapped the clinical concepts to the administrative and electronic health records and operationalized our candidate predictors. Results In the review of 34 studies, we identified 504 unique clinical concepts. Two-thirds of the clinical concepts (337/504) were used as candidate predictors only once. The most common clinical concepts included age (31/34), sex (29/34), and alcohol use (22/34). 96% of the clinical concepts (484/504) were judged to be associated with the development of hospital-induced delirium by at least two members of the task force. All of the task force members agreed that 47 or 9% of the 504 clinical concepts were associated with hospital-induced delirium. Conclusions Heterogeneity among candidate predictors of hospital-induced delirium in the literature suggests a still evolving list of factors that contribute to the development of this complex phenomenon. We demonstrated a practice-based approach to variable selection for our model development study of hospital-induced delirium. Expert judgement of variables enabled us to categorize the variables based on the amount of agreement among the experts and plan for the development of different models, including an expert-model and data-driven model. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-023-02278-1.

adjusted R-squared among various models that were developed using the same set of data (the various models had to be presented in the article or supplementary material), ii. by bootstrapping or with cross-validation, or by randomly splitting the sample into a training set and test set and developing the model using the training set, and validating the model using the test set; b. external validation by comparing model performance between an internal dataset (dataset that was used to develop a model) and external dataset (data that was used to validate the model, for example, in a different setting), where both the internal and external datasets came from the same study design.
Studies were excluded if they met any one of the following exclusion criteria: 1. did not have delirium as the (primary) outcome: a. had delirium as a predictor, for example, in a prognostic model of postoperative complications, b. predicted the course of delirium (for example, delirium severity) or outcomes of delirium (for example, post-delirium complications, delirium recovery, delirium survival, etc.), not the occurrence of delirium itselt, 2. developed diagnostic models of hospital-induced delirium (a diagnostic model is a type of a prediction model that estimates probability that an outcome is present at this time, for example, studies developing or validating delirium assessment tools), 3. failed to validate their prediction models either internally or externally using any one of the three ways that are listed in the inclusion criteria, 4. based in the following settings: a. community, including assisted living, b. emergency departments/rooms, c. gynecologic and/or obstetrical units, d. nursing homes/long-term care facilities, e. psychiatric hospitals/units f. rehabilitation units/outpatient rehabilitation facilities; 5. lacked abstracts for the title and abstract screening or full texts for the full-text screening (including through the interlibrary loan system that is offered by our institution).

Selection Process
The selection process consisted of two parts.The first part involved screening of the records that had been identified in the databases by title and abstract against the eligibility criteria.Five percent of the records were independently screened by two researchers (U.A.S. and T.G.R.M.) and the percentage of agreement was calculated.The researchers then discussed any discrepancies and resolved them via consensus.Any remaining discrepancies were presented to the primary investigator (R.J.L.) for a final resolution.The remaining (unscreened) 95% of the records were halved, and each researcher (U.A.S. and T.G.R.M.) independently reviewed one half.A couple of considerations during this step included: (1) If it was unclear in the abstract whether and/or how the model(s) was validated, the record was tentatively included and assessed for appropriate validation in the full text in part 2 of the selection process, and (2) Literature reviews that seemed relevant were tentatively included and individual records were extracted and screened following step 1 and 2.
The second part of the selection process involved full-text screening of the records that had been included in the title and abstract screening against the eligibility criteria.Both researchers (U.A.S. and T.G.R.M.) independently screened 5% of the full texts.The percentage of agreement was calculated.Any discrepancies were first discussed and then resolved by consensus between the two researchers (U.A.S. and T.G.R.M.).Any unresolved discrepancies were presented to the primary investigator (R.J.L.) for a final resolution.The remainder of the full texts was then halved and each researcher (U.A.S. and T.G.R.M.) independently reviewed one half.The articles that each researcher included were then added and included in the final synthesis.S1).

Table S1 Prognostic
Model Development-and-Validation Studies of Hospital-induced Delirium (n = 34) Intensive Care Unit; DC = development cohort; DG = development group; Diff.= difference; DS = development/derivation set; ICDSC = Intensive Care Delirium Screening Checklist; ICU = intensive care unit; NPV = negative predictive value; NuDESC = Nursing Delirium Screening Scale; PPV = positive predictive value; pts = patients; SICU = surgical intensive care unit; TS = test set; VC = validation cohort; VG = validation group; VS = validation set; yrs = years.* Total number from both cohorts for studies with the development and prospective validation cohorts.
Note.This table shows the "Candidate Predictors (Or Index Tests)" domain only.CHARMS = Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies.See TableS1for the references.The literature review included 34 prognostic model development-and-validation studies of hospital-induced delirium (see Table *