Application of standardised effect sizes to hospital discharge outcomes for people with diabetes
BMC Medical Informatics and Decision Making volume 20, Article number: 150 (2020)
Patients with diabetes are at an increased risk of readmission and mortality when discharged from hospital. Existing research identifies statistically significant risk factors that are thought to underpin these outcomes. Increasingly, these risk factors are being used to create risk prediction models, and target risk modifying interventions. These risk factors are typically reported in the literature accompanied by unstandardized effect sizes, which makes comparisons difficult. We demonstrate an assessment of variation between standardised effect sizes for such risk factors across care outcomes and patient cohorts. Such an approach will support development of more rigorous risk stratification tools and better targeting of intervention measures.
Data was extracted from the electronic health record of a major tertiary referral centre, over a 3-year period, for all patients discharged from hospital with a concurrent diagnosis of diabetes mellitus. Risk factors selected for extraction were pre-specified according to a systematic review of the research literature. Standardised effect sizes were calculated for all statistically significant risk factors, and compared across patient cohorts and both readmission & mortality outcome measures.
Data was extracted for 46,357 distinct admissions patients, creating a large dataset of approximately 10,281,400 data points. The calculation of standardized effect size measures allowed direct comparison. Effect sizes were noted to be larger for mortality compared to readmission, as well as for being larger for surgical and type 1 diabetes cohorts of patients.
The calculation of standardised effect sizes is an important step in evaluating risk factors for healthcare events. This will improve our understanding of risk and support the development of more effective risk stratification tools to support patients to make better informed decisions at discharge from hospital.
Increasing numbers of hospital inpatients have a co-existent diagnosis of diabetes . These patients are at an increased risk of both readmission  and mortality [3, 4]. Multiple studies have been performed aiming to identify statistically significant risk factors for poor outcomes, when patients with diabetes are discharged from hospital [5, 6]. These studies typically identify risk factors for generalised populations of inpatients with diabetes [7, 8], or for an individual specific cohorts who may have been admitted for a particular condition or group of conditions [9,10,11]. Almost universally, these studies report statistically significant risk factors for an individual outcomes (either readmission or mortality) and report unstandardized effect size measures, usually as odds ratios.
The use of unstandardized effect size measures that report only on individual outcomes for individual patient cohorts, makes effect size comparisons between groups difficult. Comparisons between studies are additionally difficult as, unlike clinical trials, extracted electronic health record data is rarely made available as supplementary material to research articles, due to the risk of inadvertently compromising anonymity of the data. Furthermore, whilst standardised effect sizes can be calculated if both the sample size and standard deviation are given with unstandardised effect statistics in articles, it is recognised that this information is too often incomplete and can be a laborious process across multiple studies even if it is available .
This represents a key challenge in identifying optimal targets and outcome measures for the delivery of interventions to improve the discharge process from hospital for patients with diabetes. In particular, attempts to create risk stratification tools for patients with diabetes at discharge have only had limited success, often reporting only moderate area under the curve (AUC) values and restricted predictive values [13, 14].
Standardised effect size calculations allow the direct comparison between risk factors, outcomes and cohorts. Similar to other statistical tests, there are a range of effect size statistics available, with well over 60 reported in the literature . The appropriate standardised effect size statistic needs to be selected relevant to the variables in question, with d statistics (such as Cohen’s d or) appropriate for continuous dependant variables & predictors, whilst differing tests are relevant for effect size estimates related to categorical date (such as Phi or Cramér’s V) .
Effect sizes are descriptive statistics that support both clinicians and researchers to interpret study findings. The interpretation of effect sizes can be done in isolation against pre-defined published levels of effect or “rules of thumb” . Table 1 demonstrate published rules of thumb for Cohen’s d  and Phi statistics [19, 20]. Importantly however, the importance of any effect is dependent on what is being studied, with for instance very small effect sizes being important in certain circumstances (for example life threating situations) . Effect sizes statistics are also particularly valuable when looking to make comparisons, for example between different predictors, cohorts or variables and in is primarily in this context that standardised effect sizes have utility in considering risk predictors for negative clinical outcomes.
In this study, we extract data from a large tertiary referral centre in order to calculate standardised effect sizes for pre-specified risk factors, across outcome measures and across patient cohorts. This research demonstrates the importance of calculating standardised effect sizes, a practice more typical in the psychological literature than medical literature. It further demonstrates important variation in risk at discharge from hospital for patients with diabetes.
The study adopted a retrospective evaluation of data extracted from electronic health record (EHR) of a large tertiary referral centre, in the West Midlands region of the United Kingdom, for all adult patients discharged from University Hospitals Coventry and Warwickshire NHS Trust with a diagnosis of diabetes, over a 3-year period. Data were extracted for an exemplar set of 10 pre-specified risk factor variables, these variables were selected based on both pre-specification from the published research literature , and the ease of which data for these variables can be extracted from inpatient electronic health records. Ease of extraction considered to ensure the results are generalizable to other healthcare organisations internationally. The selected extracted variables and are listed in Table 2. Outcome variable data were extracted for hospital readmission within 30 days and mortality within 180 days of hospital discharge.
The diagnosis of diabetes was taken from the coding of patients at discharge and, thus, if there was discrepancy in the diagnosis within the record, the latest diagnosis of diabetes at discharge was used. Maternity patients were excluded from the study, due to the differing nature of maternity care and readmission patterns. Patients discharged within the last 6 months of the study period were not evaluated as index patients, to ensure that all patients had a full period of 6 months follow up on the electronic health record to assess for the outcome measures of interest.
The association between risk factor variables and outcomes of interest was analysed using Chi Squared Tests for categorical variables and Student’s t-Test for continuous variables, following adequate assessment for skew and kurtosis to ensure normality. an absolute skew value larger than 2 or an absolute kurtosis (proper) larger than 7 may be used as reference values for determining substantial non-normality .
A p-value of < 0.05 was considered significant. Standardised size was evaluated using Phi coefficient for categorical variables and Cohen’s D for continuous variables. The statistical significance and effect size was evaluated for the following patient cohorts: all patients discharged with a diagnosis of diabetes; all emergency admissions discharged with a diagnosis of diabetes; all emergency admissions discharged with a diagnosis of Type 2 Diabetes; all emergency admissions discharged with a diagnosis of type 1 diabetes; all elective admissions discharged with a diagnosis of diabetes; all elective admissions discharged with a diagnosis of type 2 diabetes; all elective admissions discharged with a diagnosis of type 2 diabetes; all patients with diabetes discharged from surgical care; all patients with type 1 diabetes discharged from surgical care; and all patients discharged with type 2 diabetes from surgical care.
All statistical testing was performed using Microsoft Excel 2016 and IBM’s SPSS v24.
Ethical approval was granted by the local NHS Trust Research Ethics Committee, at University Hospitals Coventry & Warwickshire NHS Trust through the Governance arrangements for Research Ethics Committee Process [Study Ref: GF0220 & GF0335]. Approval was also granted through the University of Warwick’s Biomedical & Scientific Research Ethics Committee [Study Ref: REGO-2017-2114].
Data was extracted for 46,367 distinct patient episodes resulting in discharge from hospital, over the study period, with a diagnosis of diabetes. Table 3 demonstrates the number of patients in each cohort. Table 4 illustrates the statistical significance of each risk factor in relation to readmission per patient cohort, separated into categorical risk factor variables (evaluated with the Chi Squared Test) and continuous variables (evaluated using student’s t-test) for readmission. Table 5 similarly demonstrates statistical significance testing for mortality. Table 6 illustrates the standardised effect sizes of each risk factor related to readmission per patient cohort, separated again into categorical risk factors (evaluated using Phi coefficient) and continuous risk factors (evaluated using Cohen’s D). Table 7 illustrates effect sizes in relation to mortality at 180 days.
Statistically significant associations with readmission at 30 days were found for 46 cohort/risk factors combinations, with 61 statistically significantly associations for mortality at 180 days. Following expectations, the effect size of most risk factors individually on outcomes was small. However, there was significant variation in effect size between risk factors, cohorts and outcome measures. The mean average effect size for categorical values considering readmission (Phi Coefficient) was 0.05, with the mean average effect size for continuous variables being (Cohen’s D) 0.23. Mean average effect size for categorical values, considering mortality, was 0.06 (Phi coefficient) and for continuous variables 0.81 (Cohen’s D). Effect sizes were notably larger for surgical cohorts of patients, in particular surgical patients with T1DM.
We demonstrate that there is substantial variation in the effect sizes, regarding risk factors related to poor outcomes at discharge from hospital for patients with diabetes. Whilst a large number of candidate risk factors have been identified as statistically significant, there is variation in the effect sizes between individual risk factors. Typically, effect sizes for mortality were greater than effect sizes for readmission, suggesting that using the risk factors described here, it may be easier to predict risk related to mortality than readmission. This is particularly interesting, given that readmission is most commonly used as the maker of the success of the discharge process and a typical target for risk predication modelling and risk reduction interventions.
There is also substantial variation in both the statistical significance and effect size of individual risk factors between individual cohorts of patients with diabetes, as well as the overall combined effect sizes between individual patient cohorts. This suggests, again, that risk prediction may be easier for some cohorts of patients, particularly those with Type 1 Diabetes & those attending for surgery. The ability, with which we are able to predict risk from known risk factors, is important in the development and appropriateness of developing risk prediction tools, but also in targeting interventions to those most at need. The targeting of interventions, supported by evidence-based discussions of risk with patients, is essential to individualised sustainable healthcare.
This study does not aim to provide a comprehensive assessment of effect sizes for every patient cohort with diabetes at discharge from hospital, or for every known risk factor for poor outcomes. Rather, we demonstrate the substantial variations of standardised effect sizes between risk factors, outcomes and patient cohorts. It is important to note that there are a number of different effect size measures that could be used, Cohen’s D and Phi were selected as being widely used effect size measures and their suitability for use alongside statistical significance measures. This, therefore, lays important foundations for future research looking to explore individual risk factors, outcomes or cohorts in more depth, before possible future development of rigorous risk prediction models.
The study has a number of limitations. Firstly it is based only at a single centre, albeit with a large patient population over a significant period of time. Secondly, we have not attempted to control individual risk factors at this stage; this approach is however representative of the many studies identifying new risk factors and subsequently reporting with unstandardized effect sizes. It must be remembered that this is an exemplar paper demonstrating the utility of standardised effect sizes and the ability to predict risk may vary depending on other risk factors selected or cohorts analysed.
When considering the utility of standardised effect sizes it is notable that we have used two different effect size statistics (Cohen’s d & Phi coefficient), whilst the primary aim of using standardised effect size statistics is to enable comparison the outputs of the two statistical methods cannot be directly compared due to variations in the “rules of thumb” for their interpretation. Whilst some processes to enable conversion between effect size statistics have been published, there is no accepted approach to converting between all effect size statistics .
We demonstrate the calculation of standardised effect sizes for risk factors related to poor outcomes when patients are discharge from hospital. Whilst individual effect sizes are often small, there is substantial variability between different risk factors, patient cohorts and outcomes. The use of standardised effect sizes allows the easier comparison between such groups, this in turn may facilitate the development of better risk stratification models and risk minimisation interventions. We hope that, as a consequence of this paper, more studies will look to calculate standardised effect sizes when considering risk factors, generating more directly comparable results and enabling more rapid translation into changes to patient care.
Availability of data and materials
Data was generated from the inpatient electronic health record patient-data of University Hospitals Coventry & Warwickshire NHS Foundation Trust. As is typical for data sets of this nature, whilst the information is anonymised, the ethical approval process requires analysis and storage of the raw data on secure NHS equipment due to the risk of inadvertent or indirect breeches to anonymization. The raw data may potentially be available from University Hospitals Coventry & Warwickshire NHS Trust subject to approval, ethical review and secure storage arrangements.
Electronic Health Record
National Health Service
Type 1 Diabetes
Type 2 Diabetes
Wallymahmed M, et al. Hospital in-patients with diabetes: increasing prevalence and management problems. Diabet Med. 2005;22(1):107–9.
Dungan KM. The Effect of Diabetes on Hospital Readmissions. J Diabetes Sci Technol. 2012;6(5):1045–52. https://doi.org/10.1177/193229681200600508.
Bataille V, Ferrières J, Danchin N, Puymirat E, Zeller M, Simon T, Carrié D. Increased mortality risk in diabetic patients discharged from hospital with insulin therapy after an acute myocardial infarction: Data from the FAST-MI 2005 registry. Eur Heart J Acute Cardiovasc Care. 2019;8(3):218–30.
Thomsen RW, et al. Diabetes and 30-day mortality from peptic ulcer bleeding and perforation: a Danish population-based cohort study. Diabetes Care. 2006;29(4):805–10.
Enomoto LM, et al. Risk factors associated with 30-day readmission and length of stay in patients with type 2 diabetes. J Diabetes Complicat. 2017;31(1):122–7.
Ostling S, et al. The relationship between diabetes mellitus and 30-day readmission rates. Clin Diabetes Endocrinol. 2017;3(1):3.
Sonmez H, et al. The readmission rates in patients with versus those without diabetes mellitus at an urban teaching hospital. J Diabetes Complicat. 2017;31(12):1681–5.
Robbins JM, Webb DA. Diagnosing diabetes and preventing rehospitalizations: the urban diabetes study. Med Care. 2006;44(3):292.
Lovecchio F, et al. Do patients with insulin-dependent and noninsulin-dependent diabetes have different risks for complications after arthroplasty? Clin Orthop Relat Res. 2014;472(11):3570–5.
Rubin DJ, et al. Predicting readmission risk of patients with diabetes hospitalized for cardiovascular disease: a retrospective cohort study. J Diabetes Complicat. 2017;31(8):1332–9.
Holscher CM, et al. Unplanned 30-day readmission in patients with diabetic foot wounds treated in a multidisciplinary setting. J Vasc Surg. 2018;67(3):876–86.
Nakagawa S, Cuthill IC. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev. 2007;82(4):591–605.
Rubin DJ, et al. Development and VALIDATION of a novel tool to predict hospital readmission risk among patients with diabetes. Endocr Pract. 2016;22(10):1204–15.
Yu D, et al. Development and external Validation of risk scores for cardiovascular hospitalization and Rehospitalization in patients with diabetes. J Clin Endocrinol Metab. 2018;103(3):1122–9.
Huberty CJ. A history of effect size indices. Educ Psychol Meas. 2002;62(2):227–40.
Fritz CO, Morris PE, Richler JJ. Effect size estimates: current use, calculations, and interpretation. J Exp Psychol Gen. 2012;141(1):2.
Cohen J. Statistical power analysis for the behavioural sciences Hillsdale, vol. 2. NJ: Lawrence Earlbaum Associates; 1988.
Sawilowsky SS. New effect size rules of thumb; 2009.
Kotrlik JW, Williams HA, Jabor MK. Reporting and interpreting effect size in quantitative agricultural education research. J Agric Educ. 2011;52(1):132–42.
Rea LM, Parker RA. Designing and conducting survey research: A comprehensive guide. San Francisco: Wiley; 2014.
Rosnow RL, Rosenthal R. Statistical procedures and the justification of knowledge in psychological science. Am Psychol. 1989;44(10):1276.
Robbins TD, et al. Risk factors for readmission of inpatients with diabetes: a systematic review. J Diabetes Complicat. 2019;33(5):398–405.
Kim H-Y. Statistical notes for clinical researchers: assessing normal distribution (2) using skewness and kurtosis. Restor Dent Endod. 2013;38(1):52–4.
Professor Theodoros N Arvanitis is partially supported by Health Data Research UK, which is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and Wellcome Trust.
Dr. Tim Robbins receives funding through a PhD Studentship at WMG, University of Warwick.
Professor Theodoros N. Arvanitis is partially funded by HDR UK. HDR UK is funded by the UK Medical Research Council, Engineering and Physical Sciences Research Council, Economic and Social Research Council, Department of Health and Social Care (England), Chief Scientist Office of the Scottish Government Health and Social Care Directorates, Health and Social Care Research and Development Division (Welsh Government), Public Health Agency (Northern Ireland), British Heart Foundation and Wellcome Trust.
HDR UK – Medical Research Council – Award Reference HDR-3001.
Ethics approval and consent to participate
Ethical approval was granted by the local NHS Trust Research Ethics Committee, at University Hospitals Coventry & Warwickshire NHS Trust through the Governance arrangements for Research Ethics Committee Process [Study Ref: GF0220 & GF0335]. Approval was also granted through the University of Warwick’s Biomedical & Scientific Research Ethics Committee [Study Ref: REGO-2017-2114]. Due to the nature of the project using retrospective routinely collected patient information, patient consent was not required and approved by both ethics committees.
Consent for publication
No conflicting interests are declared by any authors.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Robbins, T., Lim Choi Keung, S.N., Sankar, S. et al. Application of standardised effect sizes to hospital discharge outcomes for people with diabetes. BMC Med Inform Decis Mak 20, 150 (2020). https://doi.org/10.1186/s12911-020-01169-z