- Open Access
The development of a web-based app employing machine learning for delirium prevention in long-term care facilities in South Korea
BMC Medical Informatics and Decision Making volume 22, Article number: 220 (2022)
Long-term care facilities (LCFs) in South Korea have limited knowledge of and capability to care for patients with delirium. They also often lack an electronic medical record system. These barriers hinder systematic approaches to delirium monitoring and intervention. Therefore, this study aims to develop a web-based app for delirium prevention in LCFs and analyse its feasibility and usability.
The app was developed based on the validity of the AI prediction model algorithm. A total of 173 participants were selected from LCFs to participate in a study to determine the predictive risk factors for delerium.
The app was developed in five phases: (1) the identification of risk factors and preventive intervention strategies from a review of evidence-based literature, (2) the iterative design of the app and components of delirium prevention, (3) the development of a delirium prediction algorithm and cloud platform, (4) a pilot test and validation conducted with 33 patients living in a LCF, and (5) an evaluation of the usability and feasibility of the app, completed by nurses (Main users).
A web-based app was developed to predict high risk of delirium and apply preventive interventions accordingly. Moreover, its validity, usability, and feasibility were confirmed after app development. By employing machine learning, the app can predict the degree of delirium risk and issue a warning alarm. Therefore, it can be used to support clinical decision-making, help initiate the assessment of delirium, and assist in applying preventive interventions.
This web-based app is evidence-based and can be easily mobilised to support care for patients with delirium in LCFs. This app can improve the recognition of delirium and predict the degree of delirium risk, thereby helping develop initiatives for delirium prevention and providing interventions. Moreover, this app can be extended to predict various risk factors of LCF and apply preventive interventions. Its use can ultimately improve patient safety and quality of care.
Delirium, characterised by its sudden onset, causes changes in consciousness, memory, logical reasoning, concentration, and the performance of activities . Its prevalence is 24.6% among patients older than 65 years who are admitted to acute care facilities, 7.9% among those with dementia or recovering from a stroke , and 29–31% among inpatients in general wards . The prevalence of postoperative delirium varies widely, from 12 to 51%, depending on the type of surgery . In South Korea, the incidence of delirium is 20% among intensive care unit patients , whereas in long-term care facilities (LCFs), the incidence of delirium is only 8.1%. The incidence of delirium that is comorbid with dementia is 39.9% .
Delirium increases the mortality rate [4, 6, 7], incidence of falls, presence of pressure ulcers [8, 9], length of hospital stays, degree of medical burden [10,11,12,13], rate of chronic cognitive impairment, and likelihood of readmission to LCFs [14,15,16]. Early detection and preventive interventions using standardised assessment tools [17, 18] are more effective for delirium than are its treatment and management [4, 19]. However, these measures are rarely applied [20, 21]. In addition, it is important to approach delirium intervention with evidence-based multi-components [19, 22, 23].
Older adults admitted to LCFs often have cognitive, visual, or hearing impairments, which place them at risk for delirium [21, 24]. Usually, LCFs in South Korea are not equipped to provide high-level care for delirium patients and do not have an electronic medical record system, as seen in general hospitals . This hinders systematic approaches, including delirium monitoring. Therefore, a system that predicts the onset of delirium and enables the timely administration of preventive interventions is necessary.
A health program using mobile technology that includes delirium prediction, assessment, and prevention intervention according to delirium risk factors, is advantageous as it can be used by various health providers at a low cost . When the clinical decision support system is integrated into mHealth technology, it improves the accuracy of diagnosis and treatment [26, 27]. In addition, implementing such a system on a mobile device offers portability, the possibility of customisation, and convenience .
Machine learning uncovers useful information hidden within data and is able to generate an explanatory, predictive, or normative tool based on this information [25, 29]. Of the different types of machine learning algorithms, random forest, artificial neural networks, and support vector machines are widely used in various clinical apps. They have been shown to exhibit high performance in disease diagnosis and prognostic predictions [26, 28, 30,31,32].
The purpose of this study is to develop a web-based delirium prevention app (Web_DeliPREVENT_4LCF) that uses machine learning to predict the risk of delirium and provide evidence-based interventions for patients in LCFs. It applies a multi-component program and a capitalised mobile clinical decision support system to support health providers working in LCFs. Furthermore, this study assesses the web app’s usability and feasibility relative to its intended use.
The development process consisted of five phases. The guidelines set forth by the Scottish Intercollegiate Guidelines Network  were used to select the evidence-based delirium risk factors and preventive interventions. Iterative design was performed and cloud computing for machine learning to predict the high risk of delirium. The prediction algorithm was validated using the risk factor data of 33 LCF residents and subsequently implemented into the delirium prevention app . The app was then pilot tested, and its usability and feasibility were analysed.
Two nursing professors, one biomedical engineering professor, two machine learning experts, one app development expert, and three clinical field nurses participated in this study to proceed with its five phases. For step-by-step research progress, the related literature was referred to.
Conceptual framework of the app
The app developed in this study was based on Ahituv’s clinical decision model, an information flow model that involves the observation of a patient, interpretation of a patient's information, drawing of conclusions based on stored clinical knowledge, receipt of clinical advice, and finally, action .
Web_DeliPREVENT_4LCF consists of four domains: patient data input, prediction results for the input patient’s delirium risk factors, delirium assessment using the Short version of the Confusion Assessment Measure (S-CAM), and application of delirium preventive intervention (Fig. 1).
Selecting delirium risk factors and non-pharmacological interventions through an evidence-based literature review
Search strategy and eligibility were determined using Scottish Intercollegiate Guidelines Network guidelines  and applied to delirium risk factors and preventive interventions in the literature. Individual studies, review articles, and published systematic reviews were searched in MEDLINE, PubMed, CINAHL, Cochrane Library, Embase, PROQuest, KoreaMed, KMbase, and KISS (Korean studies Information Service System) using the terms ‘delirium’, ‘prevention’, ‘intervention’, ‘long-term care’, ‘nursing home’, and ‘mobile apps’. The publication year was set between 2010 and 2018. Various guidelines were searched using the search terms ‘delirium’ and ‘prevention.’
Inclusion criteria included a focus on prevention or intervention of delirium in a long-term care setting and mobile apps. Exclusion criteria included non-English publications (except Korean), study protocols, editorials, commentaries, letters, dissertations, and studies that focused exclusively on paediatrics, stroke, alcohol withdrawal, neurosurgery or trauma patient populations or intensive care unit settings.
After reviewing 1507 titles, 194 abstracts and 74 full texts, 23 studies were selected to be included in the literature review. Two researchers reviewed the studies independently and evaluated them critically using the relevant tools such as SIGN checklist RCT, case control studies, cohort studies, diagnostic studies, systematic reviews, and Critical Appraisal Tool for Cross-Sectional Studies (CAT-CSS).
Regarding delirium risk factors (e.g. age, number of comorbidities, pain, and pain medicine use), 24 factors were selected based on the literature review and a previous study . Orientation, environmental interventions, and early avoidance of the intrinsic and extrinsic risk factors for delirium were selected as multi-components and non-pharmacological interventions for delirium prevention (Text box 1).
Prediction algorithm pilot test and iterative design
The validity of the delirium prediction algorithm was evaluated using explainable machine learning models utilizing data from 33 LCF participants between 10 and 25 August 2020. Machine learning algorithms, such as C4.5, CBA, MCAR, and LEM2, and statistical learning algorithms, such as LR, ANNs, SVMs with three kernel functions, and random forest, were validated by paired Wilcoxon signed-rank tests on both macro -averaged F1 and weighted average F1-measures during the 10-times stratified twofold cross-validation . Furthermore, for a user-friendly design, five nurses working at an LCF participated in evaluating the appropriateness and given comments of the app.
The System Usability Scale (SUS), a tool developed by Brooke in 1996  that evaluates the usefulness of apps, was translated into Korean after obtaining permission from the developer. This tool was validated and modified by two professors and clinical experts each in accordance with the purpose of this study (e.g. convenience of using the app, continuous use and usefulness, time-consuming state, use of CAM, delirium prediction, and initiating intervention) and usability and feasibility were assessed using this modified SUS on 32 LCF nurses.
The SUS is a 10-item questionnaire scored on a 5-point Likert scale ranging from 1 (not at all) to 5 (strongly agree). The overall score is calculated by summing the item scores and multiplying the total by 2.5. The score ranges from 0 to 100, with higher scores indicating greater usefulness. A score of 68 or higher is considered above average in terms of usability. At the time of the tool’s development, Cronbach α = 0.85.
Development of the Delirium Prediction Algorithm and Cloud Platform
A knowledge-based reasoning module that enables early screening of delirium based on the data collected from a prospective cohort study  was conducted with older adults in two LCFs (120-bed and 100-bed) in two cities from October 2016 to March 2017. Predictive performance was assessed with macro-averaged accuracy (71.7%), sensitivity (74.4%), specificity (71.6%), and AUC (73.8%) during a statistical five-fold cross-validation experiment. When compared to three machine learning algorithms—random forest, artificial neural network, and support vector machine with a radial basis kernel function—AUC performance improved by 6.2%, 2.1%, and 3.3% on average, respectively .
Figure 2 shows the module of the delirium prevention app designed to operate in the Amazon Web Services environment. The knowledge-based reasoning module is divided into delirium classification rules (Additional file 1: Appendix 1) and reasoning processes (Additional file 1: Appendix 2). Additional file 1: Appendix 2 shows the pre-learned knowledge to screen delirium from non-delirium. This is a delirium classification rule extracted by selecting 24 delirium risk factors (Additional file 1: Appendix 3) from the data collected in a prospective cohort study (N = 173) by Moon and Park  and using the knowledge-based reasoning method employed by Son et al. .
Cases with ‘YES’ responses for attention deficits in Step 2, at least one YES in Box 1 that combines Steps 1 and 2, and at least one YES in Box 2 that combines Steps 3 and 4, were determined as having delirium (+ ; Additional file 1: Appendix 1). When that was the case, the delirium result was presented along with a warning alarm for delirium. The warning alarm is provided as a screen alarm, sound alarm, vibration alarm, or any combination, for healthcare providers as notifications to check delirium results. In this study, screen and vibration alarms were used.
Iterative design and composing the components of Web_DeliPREVENT_4LCF
With regard to the app design, Google and Android markets examined to determine current development trends and ascertain whether there were any prototypes or existing delirium applications. Searching keywords were ‘delirium’, ‘applications’, and ‘predictions’ and expert validity achieved in the content.
First, the app provides usage tips, including instructions, recommendations, and information about delirium, such as its definition, the risk factors, and key intervention points. Second, the user is requested to enter the patient’s information, including sex, birth date, diagnosis, severity of delirium, and 10 applicable risk factors (e.g. aged ≥ 65 years, disease severity, abnormal blood urea nitrogen, dehydration, water-electrolyte imbalance, nutritional imbalance, hypoxia, infection, sleep disorder, surgery with general anaesthesia, and oedema). The app then automatically calculates the risk of delirium using the established equation in a percentage and shows whether the patient belongs to the high-, moderate-, or low-risk group. This allows healthcare providers to predict the onset of delirium and pay closer attention to the patient in advance or effectively prevent the sudden onset of delirium. Delirium risk is classified as high, moderate, and low, with the 33.3rd percentile (1.73%) and 66.7th percentile (3.47%) as the cut-off markers. The support values for 23 delirium-related rules (Nos. 1–23) in the ‘IF’ condition and ‘THEN’ decision were used (Additional file 1: Appendix 1).
Next, the type of delirium (hyperactive, hypoactive, or mixed) is assessed among patients testing positive for delirium in the S-CAM. Written permission was obtained from the developer to use S-CAM in the app . The S-CAM consists of four steps. Step 1 includes two questions that check for acute onset and changes in consciousness; Step 2 checks for attention deficits; Step 3 involves a non-systematic thinking assessment, which estimates whether the patient engages in non-systematic or inconsistent thinking; and Step 4 involves the evaluation of changes in the level of consciousness. Consciousness was evaluated as normal, vigilant, lethargic, stuporous, or comatose, and the result is shown as delirium ( +) or no delirium (−) as embedded assessment rule.
Following the assessment, preventive multi-component interventions are given to high-risk patients testing positive for delirium. The preventive multi-components include orientation interventions, environmental interventions, and early avoidance of risk factors. Orientation interventions include repeated orientation such as: using a clock or calendar, using the patient’s name when providing care (as a reminder); placing familiar objects close to the patient, encouraging family members to visit regularly, early abnormality detection, encouraging regular involvement in the activities of daily living, and providing assistive devices to improve visual and hearing abilities, such as eyeglasses and hearing aids.
However, environmental interventions include the following: using indirect lighting and reduced ambient noise to foster an appropriate sleeping environment, keeping the same nurse in the same ward, distinguishing between day and night by using window curtains or blinds, encouraging patients to continue their hobbies (such as listening to music, playing games, and doing hand-knitting), providing or reading newspapers daily, and encouraging meaningful conversations to stimulate memory and reasoning skills.
The early avoidance of risk factors refers to actions such as: providing an appropriate amount of water and preventing dehydration, pharmacological and non-pharmacological interventions for pain, minimising the use of restraints, a range of active and passive motion exercises, encouraging walking, encouraging drinking water, evaluating nutritional intake, providing non-oral feeding if necessary, the early detection of and intervention for infection, careful use of anticholinergic drugs and opioids and minimising the use of unnecessary drugs, monitoring hypoxia, and preventing constipation, falls, and pressure ulcers. Healthcare providers are asked to mark each intervention item when completed and to calculate the patient’s performance rate. Figure 3 shows a screenshot of the delirium prevention app for patients in LCFs.
Pilot testing and analysis of delirium prediction validity
To assess the validity of the prediction by the developed app, the Web_DeliPREVENT_4LCF app was applied to a sample of 33 adult (≥ 18 years) inpatients at a LCF for two weeks. In the initial stage of the study, inpatients at a long term care facility in Korea were recorded as adults > 18 years of age. However, in majority of the long-term care facilities, they were older than 65 years of age.
Approval was obtained from K University’s Institutional Review Board. Exclusion criteria included the inability to perform S-CAM owing to psychiatric diagnosis, severe visual or hearing impairment, death, transfer on the day of admission, emergency treatment at the time of delirium assessment, and having other hospital affairs. The experimental cases were collected using the mobile delirium prevention app between 10 and 25 August 2020. These cases were examined through the knowledge-based reasoning module to screen for delirium risk. To validate the delirium prediction, a pilot study was performed using multiple machine learning algorithms, including four rule-mining algorithms (C4.5, CBA, MCAR, and LEM2) and four other statistical learning algorithms (LR, ANNs, SVMs with three kernel functions, and random forest) by paired Wilcoxon signed-rank tests on both macro-averaged F1 and weighted average F1-measures during the 10-times stratified twofold cross-validation. Post analysis, the LEM2 algorithm showed the best prediction performance (macro-averaged F1-measure of 49.35%; weighted average F1-measure of 96.55%), sharply identifying patients at risk of delirium. Pairwise comparisons between predictive powers were observed in independent models, where the LEM2 model had moderate or large effect sizes, between 0.4925 and 0.8766, when compared to the LR, ANN, SVM with RBF, and MCAR models .
To evaluate its usability and feasibility, Web_DeliPREVENT_4LCF was approved by the K University Institutional Review Board (IRB No.40525–202,101). Data were collected from 13 to 31 March 2021 for nurses who had been working in LCFs for more than six months. In total, 33 questionnaires were collected. After excluding one questionnaire for having incomplete responses, 32 questionnaires were used for analysis (97% response rate).
The data were analysed using the SPSS WIN 23.0. The general characteristics of the study participants are shown in Table 1, the usability of the app in Table 2, and the feasibility of the app in Table 3.
Regarding the app’s usability, the items stating that a personal smartphone is more convenient than a tablet PC and that it is more convenient to use Wi-Fi than personal data received a high score. The lowest score was given to the item stating that the training for using the app consumes a lot of time.
Regarding the app’s feasibility, the item stating that ‘Initiating care for delirium patients was achieved through the results of the app’s delirium prediction and delirium assessment,’ and that ‘The use of the app has improved the overall knowledge of delirium’ received the highest scores. Particularly, 68.8% of respondents had experience with delirium care, but only 3.1% felt that they would not actively use the tools during delirium care.
This study describes the development of an Android mobile app for delirium prevention among patients at LCFs: Web_DeliPREVENT_4LCF. When patient information and delirium risk factors were entered into the app through a web server connection, the app predicted the risk of delirium (high, moderate, or low) based on a knowledge-based reasoning module. Through the app, healthcare providers can be notified of patients’ risk levels for delirium and assess delirium accordingly using the S-CAM. The app then shows the delirium prevention intervention screen and instructs the provider to apply multi-component interventions.
This app could serve to help LCFs when the awareness of delirium is low and when the facility is poorly equipped for delirium assessment or preventive interventions compared to larger hospitals [6, 39]. Furthermore, by reflecting the importance of real-time information at the point of care in the practice of evidence-based intervention, this app also provides effective communication among health professionals [40, 41].
Web_DeliPREVENT_4LCF was designed to predict current delirium risk levels based on up-to-date patient factors entered into the system, to assess delirium using a validated delirium assessment tool, and to help healthcare providers apply multi-component preventive interventions. Moreover, the entered app data are connected to the web server; thus, the app uses the prediction algorithm installed online to calculate patients’ delirium risks as percentages (%). The predictive accuracy increases as data accumulate (i.e. as higher numbers of patients and risk factors are entered into the system) .
When considering the poor outcomes of delirium among patients within LCFs [6, 42], the use of Web_DeliPREVENT_4LCF increases awareness of delirium among healthcare providers and presents a prediction of patients’ risks, thereby enabling healthcare providers to begin assessing delirium and proper intervention immediately and accurately. Thus, Web_DeliPREVENT_4LCF can be regarded as a mobile clinical decision support system that helps health providers make clinical judgements about delirium care, rather than the concept of a mobile health app used for patient or caregiver education. In addition, the use of the application may improve the quality of care for elderly patients in LCFs in Korea, where assessment and preventive nursing are sledom performed due to low awareness of delirium among healthcare providers. LCFs typically have a somewhat lower CDSS function compared to general hospitals [43, 44], so this is expected to increase the overall care level in LCFs. In future studies, it would be beneficial to develop similar predictive programs for falls, bedsores, hospital associated infections, and other common occurrences in order to improve the quality of care in LCFs [43, 45].
Regarding the tool’s usability and feasibility, surveyed users responded that Web_DeliPREVENT_4LCF improved their knowledge of delirium, raised awareness about the onset of delirium, and facilitated the app of delirium intervention. Most of the nurses had received education on delirium care, and they actually perform delirium care in the LCFs, but the rate of delirium assessment conducted using a standardised tool was remarkably low. Therefore, the use of Web_DeliPREVENT_4LCF is expected to help improve the quality of delirium care, as standardised delirium assessment tools can be easily applied, and this is thought to be helpful for early detection and intervention [15, 42], which is the goal of delirium management.
This app can be used on a smartphone, tablet or PC by healthcare providers, but one limitation of the design was that this app is operated over the Internet. Thus, it may be of limited use when Internet connections are unstable, and it may not yet be feasible in all long-term care hospitals.
A second limitation was that the risk factors of patients must be continuously monitored, periodic updates are not performed automatically, and it is difficult to contact an external technician.
The environment for the patients from whom the random forest data were collected in a previous study  (which were the data used for developing the algorithm) differed from that of the patients enrolled in the pilot study. It can be assumed that the current COVID-19 pandemic had some influence; thus, subsequent studies should recruit a larger study population and continuously apply and analyse prediction algorithms based on new risk factors in this population.
Another limitation was patient information. When entering patient information in the app, only the surname is entered as the patient’s ID. Additionally, when being synchronised with Amazon Web Services, patients’ IDs were encoded again per the secure coding standard. Nevertheless, a fundamental problem, the security of cloud-uploaded data, is socially present; therefore, this should be continuously complemented and updated through app and version upgrades.
Although this study was conducted in such a manner that any potential selection bias would be minimised, the possibility of selection bias occurring should be considered with respect to the app contents and pilot testing. Although the usefulness of the app, as well as its ease of use have been confirmed, there may still be limitations in the generalisation of our findings to the wider population due to the small sample size.
Despite the high incidence of delirium among patients at LCFs, awareness of delirium is still low, and appropriate assessments and interventions are not actively performed in long-term care hospitals. Web_DeliPREVENT_4LCF app presents healthcare providers with timely and convenient predictions of patients’ delirium risks, assisting them in assessing delirium with a validated tool and administering delirium prevention interventions. This app could ultimately contribute to patient safety and quality of care, including lower mortality rates, reduced durations of hospital stay, and lower medical costs.
Availability of data and materials
Long-term care facilities
Short version of the confusion assessment measure
Korean studies information service system
Critical appraisal tool for cross-sectional studies
System usability scale
American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. Washington, DC: American Psychiatric Publishing; 2013.
Reynish EL, Hapca SM, De Souza N, Cvoro V, Donnan PT, Guthrie B. Epidemiology and outcomes of people with dementia, delirium, and unspecified cognitive impairment in the general hospital: prospective cohort study of 10,014 admissions. BMC Med. 2017;15(1):140. https://doi.org/10.1186/s12916-017-0899-0 (PMID: 28747225).
Vasilevskis EE, Han JH, Hughes CG, Ely EW. Epidemiology and risk factors for delirium across hospital settings. Best Pract Res Clin Anaesthesiol. 2012;26(3):277–87. https://doi.org/10.1016/j.bpa.2012.07.003 (PMID: 23040281).
Inouye SK, Westendorp RG, Saczynski JS. Delirium in elderly people. Lancet. 2014;383(9920):911–22. https://doi.org/10.1016/S0140-6736(13)60688-1 (PMID: 23992774).
Moon KJ, Jin Y, Jin T, Lee SM. Development and validation of an automated delirium risk assessment system (Auto-DelRAS) implemented in the electronic health record system. Int J Nurs Stud. 2018;77:46–53. https://doi.org/10.1016/j.ijnurstu.2017.09.014 (PMID: 29035732).
Moon KJ, Park H. Outcomes of patients with delirium in long-term care facilities: a prospective cohort study. J Gerontol Nurs. 2018;44(9):41–50. https://doi.org/10.3928/00989134-20180808-08 (PMID: 30148531).
Tieges Z, Quinn T, MacKenzie L, Davis D, Muniz-Terrera G, MacLullich AMJ, et al. Association between components of the delirium syndrome and outcomes in hospitalised adults: a systematic review and meta-analysis. BMC Geriatr. 2021;21(1):162. https://doi.org/10.1186/s12877-021-02095-z (PMID: 33673804).
Dharmarajan K, Swami S, Gou RY, Jones RN, Inouye SK. Pathway from delirium to death: potential in hospital mediators of excess mortality. J Am Geriatr Soc. 2017;65(5):1026–33. https://doi.org/10.1111/jgs.14743 (PMID: 28039852).
Thomas AN, Balmforth JE. Patient safety incidents describing patient falls in critical care in North West England Between 2009 and 2017. J Patient Saf. 2021;17(2):e71–5. https://doi.org/10.1097/PTS.0000000000000574 (PMID: 30747859).
Brown CH IV, LaFlam A, Max L, Wyrobek J, Neufeld KJ, Kebaish KM, et al. Delirium after spine surgery in older adults: incidence, risk factors, and outcomes. J Am Geriatr Soc. 2016;64(10):2101–8. https://doi.org/10.1111/jgs.14434 (PMID: 27696373).
Moon KJ, Lee SM. The effects of a tailored intensive care unit delirium prevention protocol: a randomized controlled trial. Int J Nurs Stud. 2015;52:1423–32. https://doi.org/10.1016/j.ijnurstu.2015.04.021 (PMID: 26032729).
Moon KJ, Piao J, Jin Y, Lee SM. Is delirium an unrecognized threat to patient safety in Korean intensive care units? J Nurs Care Qual. 2014;29(1):91–8. https://doi.org/10.1097/NCQ.0b013e31829f828b (PMID: 23896778).
Muresan ML, Adamis D, Murray O, O’Mahony E, McCarthy G. Delirium, how does it end? Mortality as an outcome in older medical inpatients. Int J Geriatr Psychiatry. 2016;31(4):349–54. https://doi.org/10.1002/gps.4332 (PMID: 26250650).
Saczynski JS, Marcantonio ER, Quach L, Fong TG, Gross A, Inouye SK, et al. Cognitive trajectories after postoperative delirium. N Engl J Med. 2012;367(1):30–9. https://doi.org/10.1056/NEJMoa1112923 (PMID: 22762316).
Morandi A, Davis D, Bellelli G, Arora RC, Caplan GA, Kamholz B, et al. The diagnosis of delirium superimposed on dementia: an emerging challenge. J Am Med Dir Assoc. 2017;18(1):12–8. https://doi.org/10.1016/j.jamda.2016.07.014 (PMID: 27650668).
Steis MR, Fick DM. Delirium superimposed on dementia: accuracy of nurse documentation. J Gerontol Nurs. 2012;38(1):32–42. https://doi.org/10.3928/00989134-20110706-01 (PMID: 21761816).
Guthrie PF, Rayborn S, Butcher HK. Evidence-based practice guideline: delirium. J Gerontol Nurs. 2018;44(2):14–24. https://doi.org/10.3928/00989134-20180110-04 (PMID: 21761716).
Zhang XM, Jiao J, Xie XH, Wu XJ. The association between frailty and delirium among hospitalized patients: an updated meta-analysis. J Am Med Dir Assoc. 2021;22(3):527–34. https://doi.org/10.1016/j.jamda.2021.01.065 (PMID: 33549566).
Abraha I, Rimland JM, Trotta F, Pierini V, Cruz-Jentoft A, Soiza R, et al. Non-pharmacological interventions to prevent or treat delirium in older patients: clinical practice recommendations the SENATOR-ONTOP series. J Nutr Health Aging. 2016;20(9):927–36. https://doi.org/10.1007/s12603-016-0719-9 (PMID: 27791223).
Bond P, Goudie K. Identifying and managing patients with delirium in acute care settings. Nurs Older People. 2015;27(9):28–32. https://doi.org/10.7748/nop.27.9.28.s19 (PMID: 26511424).
Marcantonio ER. Delirium in hospitalized older adults. N Engl J Med. 2017;377:1456–66. https://doi.org/10.1056/NEJMcp1605501 (PMID: 29020579).
Woodhouse R, Burton JK, Rana N, Pang YL, Lister JE, Siddiqi N. Interventions for preventing delirium in older people in institutional long-term care. Cochrane Database Syst Rev. 2019;4(4):CD009537. https://doi.org/10.1002/14651858.CD009537 (PMID: 24488526).
Hshieh TT, Inouye SK, Oh ES. Delirium in the elderly. Clin Geriatr Med. 2020;36(2):183–99. https://doi.org/10.1016/j.cger.2019.11.001 (PMID: 32222295).
Voyer P, Richard S, McCusker J, Cole MG, Monette J, Champoux N, et al. Detection of delirium and its symptoms by nurses working in a long term care facility. J Am Med Dir Assoc. 2012;13(3):264–71. https://doi.org/10.1016/j.jamda.2010.11.002 (PMID: 21450220).
Marcolino MS, Oliveira JAQ, D’Agostino M, Ribeiro AL, Alkmim MBM, Novillo-Ortiz D. The impact of mHealth interventions: systematic review of systematic reviews. JMIR mHealth uHealth. 2018;6(1): e23. https://doi.org/10.2196/mhealth.8873 (PMID: 29343463).
Mechael PB, Batavia H, Kaonga N, et al. Barriers and gaps affecting mHealth in low and middle income countries: Policy white paper. p. 2010. http://www.globalproblems-globalsolutions-files.org/pdfs/mHealth_Barriers_White_Paper.pdf.
Silveira DV, Marcolino MS, Machado EL, Ferreira CG, Alkmim MBM, Resende ES, et al. Development and evaluation of a mobile decision support system for hypertension management in the primary care setting in Brazil: mixed-methods field study on usability, feasibility, and utility. JMIR mHealth uHealth. 2019;7(3): e9869. https://doi.org/10.2196/mhealth.9869 (PMID: 30907740).
Martínez-Pérez B, de la Torre-Díez I, López-Coronado M, Sainz-de-Abajo B, Robles M, García-Gómez JM. Mobile clinical decision support systems and applications: a literature and commercial review. J Med Syst. 2014;38(1):4. https://doi.org/10.1007/s10916-013-0004-y (PMID: 24399281).
Jauk S, Kramer D, Großauer B, Rienmüller S, Avian A, Berghold A, et al. Risk prediction of delirium in hospitalized patients using machine learning: an implementation and prospective evaluation study. J Am Med Inform Assoc. 2020;27(9):1383–92. https://doi.org/10.1093/jamia/ocaa113 (PMID: 32968811).
Kim KJ, Tagkopoulos I. Application of machine learning in rheumatic disease research. Korean J Intern Med. 2019;34(4):708–22. https://doi.org/10.3904/kjim.2018.349 (PMID: 30616329).
Fei Y, Li WQ. Improve artificial neural network for medical analysis, diagnosis and prediction. J Crit Care. 2017;40:293. https://doi.org/10.1016/j.jcrc.2017.06.012 (PMID: 28647140).
Davoudi A, Ozrazgat-Baslanti T, Ebadi A, et al. Delirium prediction using machine learning models on predictive electronic health records data. Proceedings of the 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering (BIBE). Washington, DC: PMID; 2017. p. 568–73. https://doi.org/10.1109/bibe.2017.00014: 30393788.
Ahituv N. A meta-model of information flow - a tool to support information system theory. Commun ACM. 1987;30:781–91.
Scottish Intercollegiate Guidelines Network. A Guidline developer’s handbook. Eidinburgh; 2008.
Son CS, Kang WS, Lee JH, et al. Machine learning to identify psychomotor behaviors of delirium for patients in long-term care facility. IEEE Xplore. 2021;10:1–12. https://doi.org/10.1109/JBHI.2021.3116967 (PMID: 34596563).
Medlock S, Wyatt J, Patel V, Shortliffe E, Abu-Hanna A. Modeling information flows in clinical decision support: key insights for enhancing system effectiveness. J Am Med Inform Assoc. 2016;23:1001–6. https://doi.org/10.1093/jamia/ocv177 (PMID: 26911809).
Oh J, Cho D, Park J, Na SH, Kim J, Heo J, et al. Prediction and early detection of delirium in the intensive care unit by using heart rate variability and machine learning. Physiol Meas. 2018;39(3): 035004. https://doi.org/10.1088/1361-6579/aaab07 (PMID: 29376502).
Sus BJ. A “quick and dirty” usability scale. In: Usability evaluation in industry. Jordan PW, Thomas B, Weerdmeester BA, et al., editors. Taylor & Francis. ISBN: 9780429157011; 1996. p. 189–94.
Son CS, Kang WS, Lee JH, et al. A classification method of delirium patients using local covering-based rule acquisition approach with rough lower approximation. KIPS Trans Soft Data Eng. 2020;9(4):137–44. https://doi.org/10.3745/KTSDE.2020.9.4.137.
Mosa AS, Yoo I, Sheets L. A systematic review of health care apps for smartphones. BMC Med Inform Decis Mak. 2013;12:67. https://doi.org/10.1186/1472-6947-12-67 (PMID: 22781312).
Ventola CL. Mobile devices and apps for health care professionals: uses and benefits. P T. 2014;39(5):356–64 (PMID: 24883008).
Kosar CM, Thomas KS, Inouye SK, Mor V. Delirium during postacute nursing home admission and risk for adverse outcomes. J Am Geriatr Soc. 2017;65(7):1470–5. https://doi.org/10.1111/jgs.14823 (PMID: 28338215).
Damoiseaux-Volman BA, Medlock S, Ploegmakers KJ, Karapinar-Çarkit F, Krediet CTP, de Rooij SE, et al. Priority setting in improving hospital care for older patients using clinical decision support. J Am Med Dir Assoc. 2019;20(8):1045–7. https://doi.org/10.1016/j.jamda.2019.03.017 (PMID: 31056454).
Damoiseaux-Volman BA, van der Velde N, Ruige SG, Romijn JA, Abu-Hanna A, Medlock S. Effect of interventions with a clinical decision support system for hospitalized older patients: systematic review mapping implementation and design factors. JMIR Med Inform. 2021;9(7):1–11. https://doi.org/10.2196/28023 (PMID: 34269682).
Groshaus H, Boscan A, Khandwala F, Holroyd-Leduc J. Use of clinical decision support to improve the quality of care provided to older hospitalized patients. Appl Clin Inform. 2012;3(1):94–102. https://doi.org/10.4338/ACI-2011-08-RA-0047 (PMID: 23616902).
This research was supported by Basic Research Program through the National Research Foundation of Korea, funded by the Ministry of Education (2019R1I1A3A01060561).
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
. Appendix 1. Development of delirium prediction algorithm utilizing a knowledge-based reasoning module. 1-1. Data characteristics. 1-2. Classification rules to identifying the delirium from non-delirium patients. 1-3. Reasoning procedure to estimate the risk level of delirium.
About this article
Cite this article
Moon, K.J., Son, CS., Lee, JH. et al. The development of a web-based app employing machine learning for delirium prevention in long-term care facilities in South Korea. BMC Med Inform Decis Mak 22, 220 (2022). https://doi.org/10.1186/s12911-022-01966-8
- Clinical decision support system
- Long-term care facility
- Mobile apps
- Rule-based prediction