The application of machine learning for identifying frailty in older patients during hospital admission

Abstract

Background

Early identification of frail patients and early interventional treatment can minimize the frailty-related medical burden. This study investigated the use of machine learning (ML) to detect frailty in hospitalized older adults with acute illnesses.

Methods

We enrolled inpatients of the geriatric medicine ward at Taichung veterans general hospital between 2012 and 2022. We compared four ML models including logistic regression, random forest (RF), extreme gradient boosting, and support vector machine (SVM) for the prediction of frailty. The feature window as well as the prediction window was set as half a year before admission. Furthermore, Shapley additive explanation plots and partial dependence plots were used to identify Fried’s frailty phenotype for interpreting the model across various levels including domain, feature, and individual aspects.

Results

We enrolled 3367 patients. Of these, 2843 were frail. We used 21 features to train the prediction model. Of the 4 tested algorithms, SVM yielded the highest AUROC, precision and F1-score (78.05%, 94.53% and 82.10%). Of the 21 features, age, gender, multimorbidity frailty index, triage, hemoglobin, neutrophil ratio, estimated glomerular filtration rate, blood urea nitrogen, and potassium were identified as more impactful due to their absolute values.

Conclusions

Our results demonstrated that some easily accessed parameters from the hospital clinical data system can be used to predict frailty in older hospitalized patients using supervised ML methods.

Introduction

Frailty is a common clinical condition in the elderly. It is caused by the gradual loss of physiological reserve capacity of several interconnected organs and the gradual but cumulative capacity loss [1, 2]. It is characterized by the loss of health reserves, reduced physiological functions, and impaired homeostasis, which in turn, increases the individual’s vulnerability. The presence of frailty is associated with an increased risk of negative health outcomes, such as functional decline, falls, fractures, disability, morbidity, dependence, recurrent hospitalizations, institutionalization, reduced quality of life, and death [3,4,5,6]. In high-income countries, frailty affects an estimated 10% of community-dwelling individuals over the age of 65 years (65+), and 25-50% of those aged 85+ [7, 8]. In acute care hospitals, the prevalence of frailty among hospitalized patients, over 65 years of age, is even higher, at nearly 50% [9]. Research has demonstrated that frailty can be delayed or ameliorated through various interventions, including nutritional supplementation and enhanced physical activity [10]. Primary health care plays a vital role in accurately identifying and managing frailty among patients to facilitate early initiation of interventional strategies [11]. Moreover, advanced frailty screening tests [12] can improve the quality of personal care. These may potentially reduce hospitalizations and healthcare costs [11].

In 2001, two key frailty assessment models were introduced. Fried’s frailty phenotype (FFP) assesses physical frailty using five clinical indicators: weak grip, slow walking speed, unintended weight loss, low physical activity, and self-reported exhaustion [13]. The frailty index (FI), based on Rockwood and Mitnitski’s age-related health deficits model, measures age-related health problems including diseases, symptoms, signs, disability, and abnormal lab results. Additionally, the FI-comprehensive geriatric assessment (CGA) was created by selecting items from the CGA questionnaire to establish a clinically useful FI based on cumulative health deficits. Various other frailty instruments, such as the FRAIL scale, clinical frailty scale (CFS), and the Groningen frailty indicator, provide alternative approaches to assess frailty [14].

Currently, there are no universally agreed-upon definitions or tools for diagnosing frailty. As a result, the estimated prevalence of frailty varies significantly depending on the diagnostic tool used [7]. Williamson et al. reported on using supervised machine learning (ML) for analyzing vulnerabilities in electronic medical record (EMR) profiles by using data from the Canadian province of Alberta [15]. This study used the CFS [16] to define frailty and obtained a sensitivity of 0.28 and a specificity of 0.94. Other studies have employed various frailty diagnostic tools to diagnose frailty, while ML methods have been used to classify the frailty extent. Additionally, Hassler et al. used frailty phenotypes to identify frailty and supervised ML methods to infer frailty. Sensitivity estimates for this study ranged from 65.7 to 86.7%, and specificity ranged from 58.1 to 85.6% [17]. Anbagt Sher et al. used the electronic FI [18] to identify frailty, and used EMR data to make frailty inferences using supervised ML methods [19]. The best-performing model has a sensitivity of 97.8% and a specificity of 89.1%. Currently, there have been many studies on prediction models for frailty in primary care settings. In contrast, prediction models for frailty in hospitalized populations are limited.

This study aimed to develop a validated case definition of frailty for the elderly patient admitted to the hospital using ML. Our intention is to improve the quality of treatment and reduce future medical expenses through early diagnosis and timely intervention.

Materials and methods

Data source

The Institutional Review Board (IRB) of the Taichung Veterans General Hospital (TCVGH) (IRB no.: CE22109A) approved this study, ensuring that all procedures adhered strictly to the approved study protocol, following standard regulations and the principles of the Declaration of Helsinki. The study utilized de-identified Electronic Medical Record (EMR) data from TCVGH, allowing the IRB to waive the need for informed consent. The study focused on frailty diagnosis using the FFP criteria [13], which include factors such as fatigue, unintentional weight loss, weakness, sluggishness, and low physical activity.

In this study, the diagnosis of frailty was determined using the FFP criteria, which include fatigue, unintentional weight loss, weakness, sluggishness, and low physical activity. Participants who met any two criteria were defined as “pre-frail”. Patients who met ≥ 3 criteria were classified as “frail” [13].

Constructing a dataset for supervised learning

This retrospective study included patients aged 65 years and older who were admitted to the geriatric ward of the medical center between 2012 and 2022. From an initial cohort of 7,421 patients, we excluded those with incomplete information (42 patients), repeated assessments before discharge (3,353 patients), hospital stays of less than three days (65 patients), and cases where the assessment was conducted more than four days after admission (400 patients). The final dataset comprised 3,367 patients, of which 2,843 were classified as frail and 524 as non-frail. This dataset was notably imbalanced, with a significantly higher number of frail cases.

Fig. 1 presents the study flowchart and sample selection.

Fig. 1

Study flowchart

Data description

Data collected included information from six months before to three days after the hospitalization date. This encompassed blood test reports, diagnosis codes, the number of hospitalizations, emergency visits, multimorbidity frailty index (mFI) [20], injury classification, and treatment orders. The study stipulated that hospitalization must exceed three days to avoid assessment duplications, requiring a minimum of four days between the assessment and previous admission. The primary aim of this study was to develop a binary classification prediction model to identify frailty in patients admitted to the geriatric ward.

Fig. 2 illustrates the frailty prediction task.

Fig. 2

Frailty prediction task

Feature selection and data preprocessing

Feature selection for this study was guided by a review of existing frailty prediction models and adjusted based on clinical relevance and data availability [1, 19, 21, 22]. An initial set of 71 features was selected, including diagnosis codes, drug codes, and doctor’s order codes. To address missing data, features with a missing rate greater than 20% were removed, following the guidelines from Rubin and Allison [23, 24]. Various imputation methods were explored, including mean imputation, k-nearest neighbor (KNN) imputation, and multiple imputation. The impact of these imputation methods on the performance of different classifiers, such as extreme gradient boosting (Xgboost), random forest (RF), logistic regression (LR), and support vector machine (SVM), was analyzed. The results indicated that mean imputation effectively addressed missing data without significantly affecting model performance metrics like sensitivity, specificity, accuracy, AUROC, precision, and F1-score. Consequently, 21 clinical variables were retained for analysis, including demographic data, blood test results, diagnosis codes, and emergency data [25, 26]. For detailed results of each imputation method, see Supplementary Table 2.

Table 1 presents the demographic data including age and gender. Blood test reports included white blood cell count, neutrophil ratio, hemoglobin (Hgb), platelet count, blood urea nitrogen (BUN), creatinine, estimated glomerular filtration rate (eGFR), glucose, sodium, potassium (K), and c-reactive protein (CRP) levels. Diagnosis codes are represented by mFI. Emergency data comprises the number of visits, injury classification and triage, the number of hospitalizations, and treatment orders, such as nasogastric (NG) tube, restriction, Foley catheter, and nothing by mouth.

Modeling

Once data preprocessing was completed, the dataset was divided into a training set (80%) and a test set (20%) for model development. The model was trained using Python version 3.6. Four machine learning (ML) algorithms were employed: Xgboost, RF, LR, and SVM. Hyperparameter optimization was conducted using a grid search approach with 5-fold cross-validation to identify optimal parameter configurations, specifically targeting improvements in the AUROC. Given the significant class imbalance between frail and non-frail groups, class weights were applied to improve model performance. The models were implemented using widely recognized Python libraries: Xgboost for Xgboost, imbalanced-learn for RF, and scikit-learn for both LR and SVM. The Xgboost and RF models represented integrated learning and tree-based methods, respectively [27, 28]. The LR model represented a linear model [29], while SVM represented a non-probabilistic binary linear classifier [30].

Table 1 The demographic data of the study population

Statistical analysis

Traditional statistical methods were employed to analyze the data, with independent t-tests used to compare continuous variables between positive and negative outcomes. The study evaluated the performance of the ML models using sensitivity, specificity, accuracy, AUROC, precision, and F1-score as metrics. AUROC values were categorized as follows: 0.7 to 0.8 as acceptable, 0.8 to 0.9 as good, and greater than 0.9 as excellent. To further interpret model predictions, Shapley Additive Explanations (SHAP) plots were utilized to determine the importance of each feature, while partial dependence plots (PDPs) were used to explore trends related to individual features. The study’s goal was to develop a robust frailty prediction model that could effectively identify frail patients in various clinical settings.

Results

Evaluation of the different models

The present study compared four different algorithms and evaluated the performance of each using our dataset. Table 2; Fig. 3 present the predictive performance of frailty. We used the predictive results from the ML algorithm to represent the risk probability of frailty. The default value was set at 0.5, meaning if the model’s threshold exceeds 0.5, the patient was defined as ‘frail’. When comparing sensitivity and specificity, it was evident that the LR classifier had the highest specificity for both the validation and testing datasets (72.06% and 76.47%, respectively). The SVM showed the highest AUROC on the validation dataset, with a value of 78.05%. Additionally, SVM showed highest precision and F1-score on the testing dataset (94.53% and 82.10%).

Table 2 The predictive performance of frailty using different algorithms

Fig. 3

Comparative performance evaluation of machine learning models. (a) AUROC curves for model discrimination analysis. (b) Calibration curves for model probability calibration

The predictive performance was further assessed based on the AUROC and calibration from the testing data. The AUROC curve provides a clear visualization of a classifier’s diagnostic capability as the discrimination threshold is adjusted. In our study, all models demonstrated robust predictive performance, with AUROC values around 0.8, effectively distinguishing between frail and non-frail individuals. Notably, the RF and SVM models slightly outperformed the others, with the SVM achieving the highest AUROC at 0.811. This indicates that the SVM model is particularly effective in detecting frailty within this dataset. These findings highlight the SVM’s potential in targeted health assessments. Among the evaluated models, the SVM demonstrates the best calibration, making it exceptionally reliable for applications that require precise predictions and risk assessments. Its close adherence to the ideal calibration line ensures that its probability estimates accurately reflect true event likelihoods. In contrast, the Xgboost, RF, and LR models tend to predict higher probabilities than actual outcomes across a wide range of probabilities, consistently positioning above the 45-degree line. This tendency indicates an overestimation of event likelihoods, which could result in a higher rate of false positives in practical settings. Despite adjustments made using class weight to address data imbalance, these models still exhibit a tendency to overestimate, suggesting that other factors might be contributing to this issue.

Clinical features: importance and visualization

Figure 4 presents the feature importance as the strongest predictor to effect frailty. The most important feature was age, followed by Hgb, Foley catheter, NG tube, and so on. By analyzing the SHAP plot [31], we can gain insights into how each feature influences the model’s predictions. A positive SHAP value indicates that as the feature value increases, the model’s predicted outcome also increases. Conversely, a negative SHAP value means that higher feature values lead to a lower predicted outcome. The SHAP summary plot (Fig. 5) illustrates the strength and direction of each feature’s impact in this study. Older age, the presence of NG or Foley catheters, male, higher mFI, higher triage, elevated BUN levels, and lower K levels, were more strongly associated with frailty. On the contrary, being male, higher Hgb levels, higher eGFR, higher K levels, and a higher neutrophil ratio were less strongly associated with frailty.

Fig. 4

Model interpretation and visualization according to feature importance

Fig. 5

The relationship between different features and frailty

After identifying the positive and negative influences of each feature on the model using the SHAP plot, we further analyzed the relationship between a feature’s SHAP value and its actual value using a dependence plot. By setting the SHAP value of 0 as a threshold, we can identify the turning point of the feature value. This turning point serves as a clinical reference, indicating that if the feature value exceeds or falls below this point, it may positively or negatively affect the model’s predictions. Upon reviewing the dependence plots for all features, we found that the turning points of seven features have a significant impact on the model. We will now explain how each of these seven key features influences the probability of failure in the ML model. The seven key features are: including age, gender, BUN, eGFR, Hgb, mFI, and neutrophil ratio. As shown in Fig. 6, ages approximately 78 years and older, BUN level higher than nearly 47, Hgb lower than 12, eGFR lower than 50, neutrophil ratio lower than 70, and mFI higher than 5, were consistently associated with an increased risk of frailty.

Fig. 6

The dependence plots

Discussion

This study investigated 4 algorithms and 21 features, and utilized a six-month window from the time of hospital admission for both features and predictions to build a frailty detection model. Among the five algorithms, SVM showed the best predictive results: accuracy (73.15%), Precision, F1-score (82.10%), and AUROC (0.811%). To gain a deeper understanding, the SHAP and PDP were utilized to interpret the model across various dimensions. Of the 21 features, age was the most predictive of frailty, followed by Hgb, Foley catheter, NG tube, and so on.

The incidence of frailty varies with different diagnostic tools, patient ethnicity, and population. It is estimated that 10% of community residents over 65 years are frail, and the frailty rate is as high as 25-50% among community residents over 85 years of age [13, 32]. In one of our previous retrospective studies, the proportion of frailty among elderly hospitalized patients was as high as 44.7% [33].

Due to the high prevalence and negative health effects of frailty in the elderly, it is very important to clinically diagnose frailty as early as possible and provide timely frailty interventions. In our geriatric medicine ward, we screen patients to identify those who are frail and provide them with CGA. These evaluations are conducted by geriatricians who assess the causes of frailty and determine appropriate interventions, including dental care, nutrition, and rehabilitation. We also guide both patients and caregivers on post-discharge care and health education to ensure tailored support for improved health and well-being.

Due to the detailed nature of these assessments, which required 40 to 60 min to be complete specialized staff, such resources were predominantly available to geriatric wards. However, our newly developed frailty module can detect frailty risks early in the admission process, allowing for timely interventions that can significantly reduce frailty levels in elderly hospitalized patients.

In a recent review, Liotta et al. emphasized the importance of identifying elements that contribute to the effectiveness of health and social care interventions and the long-term viability of healthcare systems, with a focus on public health considerations [34]. A prospective study by Wang et al. showed that providing rehabilitation and nutritional counseling to patients who are hospitalized and frail can improve activities of daily living, and CFS during hospitalization, shorten hospitalization time, reduce medical costs, and lower the 30-day and 90-day readmission rates [35]. Once the module detects frailty, the attending physician can consult with a geriatric specialist. The geriatric medicine team will then perform a comprehensive assessment of the patient. Using the assessment results, along with the patient’s medical and medication history, the geriatric doctor will identify the primary cause of frailty. Based on the patient’s rehabilitation potential and nutritional status, they will implement a plan that includes exercise interventions and nutritional adjustments. After discharge, follow-up visits will be scheduled to monitor improvements in frailty risk and evaluate the effectiveness of the intervention. The plan will be continuously adjusted as needed based on the patient’s progress.

Previous research on ML frailty detection models has mostly targeted community residents and institutional residents. Ambagtsheer et al. designed a frailty prediction system for 10 residential elderly care facilities in Queensland, Australia, based on frailty identification across the performance of 18 specific scenarios using three ML algorithms (SVM, decision tree, and KNN). The results showed that the overall accuracy of ML based on SVM was greater than 75% [19]. Eskandari et al. used a long-short term memory model to detect frailty based on the institutional resident data of the Arizona frailty and fall cohort. Its sensitivity and specificity were both higher than 80% [22]. Sajeev et al. used three algorithms (discriminant analysis, SVM, and RF) to estimate the accuracy of frailty in community-dwelling older adults. When phenotypes are used to define frailty, RF demonstrated the best AUROC (0.722 in selected features, 0.701 in all 63 features) [14]. Aponte-Hao et al. used the Canadian primary care sentinel surveillance network data to predict frailty. The prevalence of frailty in their sample was 18.4%. The Xgboost module demonstrated the best frailty prediction ability (sensitivity 78.14% and specificity 74.41%) [21]. A unique aspect of our study is its focus on detecting frailty in hospitalized patients. This is especially important because frail older adults admitted to hospitals face a higher risk of poor outcomes. Additionally, we make use of “early” and “easily accessible” clinical data during hospitalization to predict frailty, which enhances its practical application in real-world clinical settings. Our research significantly contributes to the field by demonstrating how machine learning can be effectively utilized in hospital environments. This opens up new possibilities for integrating machine learning into hospital screening and intervention programs, addressing a gap in current research on frailty detection in hospitals.

Research conducted by Pablo Ferri and colleagues has demonstrated that handling datasets with significant amounts of missing data can be notably improved by using translation and coding interpolation methods that account for missing information. This approach, in conjunction with tree ensemble classifiers such as RF and gradient boosting, effectively enhances performance [36]. In our study, we adhered to both established practices and specific statistical considerations appropriate for our research context. To preserve the integrity of our statistical analysis and model performance in the face of significant missing data, three geriatric medicine doctors evaluated the importance of each feature before removing any with more than 20% missing data. While we considered various imputation methods, we recognized that the potential bias and variance from significant imputation could outweigh the benefits. Our approach aligns with established statistical guidelines, such as those by Little and Rubin [23] and Allison [24], which suggest removing features with substantial missingness under certain conditions. This cautious strategy ensures more reliable and interpretable results. Our study on frailty detection in hospitalized older adults effectively employs a variety of ML models, including LR, RF, and SVM. This approach enables a detailed analysis of a broad range of clinical data, aiming to significantly enhance early intervention and treatment strategies for this vulnerable population. Our study provides a robust framework, offering a strong comparison point to other research such as that conducted by Sajeev et al. and Blanes-Selva et al., who also utilize ML but with different focuses and contexts. Sajeev et al.‘s research targets community-dwelling adults, using techniques like LR and linear discriminant analysis to identify pre-frailty. Their study emphasizes the importance of lifestyle and environmental factors in the progression of frailty, offering crucial insights for preventive measures in non-hospital settings [14]. Conversely, Blanes-Selva et al. employ advanced models such as gradient boosting machines and deep neural networks [37]. Their research extends beyond frailty detection to include mortality predictions, integrating the assessment of palliative care needs into their models, which greatly benefits overall patient care planning. While all three studies aim to use ML to improve healthcare outcomes for older adults, our research is particularly distinguished by its focus on the acute care setting. This specificity facilitates critical, tailored interventions at the point of hospital admission, thus enhancing the management and treatment of frailty in hospitalized older adults. Each study contributes uniquely to the field, collectively advancing the understanding and management of frailty through innovative ML applications across various healthcare environments.

We utilized the class_weight parameter in our models to address class imbalance. By adjusting this parameter, we were able to assign a higher weight to the minority class, effectively increasing the cost of misclassifying the underrepresented class. This approach helps to balance the influence of each class on the model’s learning process. And, recognizing that accuracy alone can be misleading in the presence of class imbalance, we incorporated additional performance metrics such as precision, recall, F1-score, and the AUROC. These metrics are more informative and provide a better assessment of model performance in imbalanced settings.

To the best of our knowledge, the models developed by our institute are currently the only models available for predicting frailty in hospitalized elderly patients. Compared with the feature selections in previous studies, we chose a relatively streamlined number of features, and most of the features we used were data that could be easily obtained clinically, including objective emergency injury classification and medical treatment orders. As the included features are conveniently acquired, our prediction module has high reproducibility. Importantly, the prediction results presented by our prediction models performed very well.

Limitations

This study has certain research flaws and limitations. First, our sample population was drawn exclusively from a single hospital and a single ward. In the future, we aim to use other attributes of inpatients to refine the features and models. Second, the study sample provided unbalanced data. The application of post hoc calibration techniques could be considered in the future to further refine the probability estimates. Additionally, collecting more data from non-frail patients may help determine whether if the model or its components require modification. Finally, the detection module developed in this study is specifically designed for patients who have undergone blood tests at a single hospital within the past six months. Patients who have not received treatment at this hospital, or who lack recent blood test reports, will be excluded from the study and are ineligible for assessment by this module. While patients entering through the emergency department typically have complete blood reports available during their admission wait, those admitted through outpatient clinics may not have recent blood tests, making them likely to be excluded from the detection process.

Conclusion

Early diagnosis of frailty can help reduce negative medical impact and reduce overall medical costs. This study shows that using clinically accessible data and ML can predict frailty in patients admitted to the hospital.