The impact of changes in coding on mortality reports using the example of sepsis

Abstract

Objectives

NHS Digital issued new guidance on sepsis coding in April 2017 which was further modified in April 2018. During these timeframes some centres reported increased sepsis associated mortality, whilst others reported reduced mortality, in some cases coincident with specific quality improvement programmes. We hypothesised that changes in reported mortality could not be separated from changes in coding practice.

Methods

Hospital Episode Statistics from the Admitted Patient Care dataset for NHS hospitals in England, from April 2016 to March 2020 were analysed. Admissions of adults with sepsis: an International Classification of Diseases 10 (ICD-10) code associated with the Agency for Healthcare Research and Quality Clinical Classifications Software class ‘Septicaemia (except in labour)’, were assessed. Patient comorbidities were defined by other ICD-10 codes recorded within the admission episode.

Results

1,081,565 hospital episodes with a coded diagnosis of sepsis were studied. After April 2017 there was a significant increase in admission episodes with sepsis coded as the primary reason for admission. There were significant changes in the case-mix of patients with a primary diagnosis of sepsis after April 2017. An analysis of case-mix, hospital and year treated as random effects, defined a small reduction in sepsis associated mortality across England following the first change in coding guidance. No centre specific improvement in outcome could be separated from these random-effects.

Conclusion

Changes in sepsis coding practice altered case-mix and case selection, in ways that varied between centres. This was associated with changes in centre-specific sepsis associated mortality, over time. According to the direction of change these may be interpreted either as requiring local investigation for cause or as supporting coincident changes in clinical practice. A whole system analysis showed that centre specific changes in mortality cannot be separated from system-wide changes. Caution is therefore required when interpreting sepsis outcomes in England, particularly when using single centre studies to inform or support guidance or policy.

Introduction

Sepsis is defined as life-threatening organ dysfunction due to a dysregulated host response to infection [1]. It is an important cause of morbidity and mortality. In 2015, 123,000 cases of sepsis were reported by NHS England (NHSE) to be associated with 36,900 deaths, [2, 3]. Although the reported rate of sepsis has increased over time [4], there is concern that sepsis remains under-recognised, under-diagnosed and under-recorded [5, 6]. Early recognition and treatment is important as there is evidence that this can reduce mortality [7]. Clinical definitions have therefore been extended to include various features, including reduced conscious level and hypoxia as well as hypotension [1].

Initiatives to improve the identification and management of sepsis have been introduced, including public health campaigns raising awareness of the diagnosis [8]. In England, financial incentives were introduced to promote screening for sepsis in emergency departments in 2015 and acute inpatient settings in 2016 [9]. These were followed by recommendations on the use of NEWS2 to screen for sepsis by the Royal College of Physicians (RCP) in 2017 [10], NHSE in 2018 [11] and the National Institute for Health and Care Excellence (NICE) in 2019 [12].

A range of local responses to improve the recognition of sepsis have emerged in primary and secondary care [13, 14], with reports of best-practice identified by NHSE as associated with major reductions in sepsis associated mortality. However, these innovations coincided with national guidance issued to improve the recording of sepsis within the diagnosis [15]. In April 2017 NHS Digital implemented a change in the guidance on how to code a diagnosis of sepsis, with the aim of increasing the identification of sepsis as the primary diagnosis leading to admission. This involved an emphasis on clinical terminology, so that clinicians’ reference to organ specific sepsis, meaning local infection, was more likely to be coded as sepsis. In addition, when conditions such as pneumonia presented with sepsis, there was an emphasis on coding sepsis in the primary position rather than the underlying condition. A further change in guidance was issued from April 2018, advising that this sequencing of conditions should be left to ‘clinical judgement’ [16]. The effect of these changes on Hospital Episode Statistics (HES) derived outcome metrics [17, 18], such as Hospital Standardised Mortality Ratios (HSMR) and Summary Hospital-level Mortality Indicator (SHMI), have been discussed in a briefing document by Dr Foster Intelligence. After the coding change in 2017, an increase in sepsis associated mortality was reported by many hospitals. This triggered further investigation of cause and effect by those hospitals and by the Care Quality Commission (CQC) [19].

We hypothesised that altered coding practice could have affected the sepsis associated mortality rates reported, without changes in actual outcome. This change would confound the interpretation of practice changes introduced during this period. In order to understand this, we undertook a detailed examination of the relationship between altered coding and patient age, type of admission and secondary diagnoses associated with sepsis, [20] or ‘case-mix’. This was with a view to understanding the factors influencing sepsis associated mortality and determining whether centre specific changes in outcome can be differentiated from changes arising from coding practices.

Methods

Data were extracted from the Hospital Episode Statistics (HES) Admitted Patient Care dataset for the period between April 2016 and March 2020, providing data for 126 NHS acute hospital trusts in England.

Non-identifiable data was accessed and therefore the project did not require specific Human Research Authority (HRA) ethical approval, however, the study was approved by the non-HRA Data Committee at University Hospitals Birmingham NHS Foundation Trust.

Adult patients, aged 18 years or over, with an International Classification of Diseases 10 (ICD-10) code associated with the Agency for Healthcare Research and Quality (AHRQ) Clinical Classifications Software (CCS) class ‘Septicaemia (except in labour)’ recorded in the dominant inpatient episode were identified and is referred to as ‘sepsis’. Patient comorbidities were assessed using the ICD-10 codes recorded within the admission episode, aggregated using the AHRQ CCS categories for ICD-10-CM Diagnoses v2019.1.

Changes to sepsis coding criteria based on recommendations published by NHS Digital were introduced in April 2017 and April 2018. Data were analysed within 12 month sample periods beginning in April each year, providing data for 12 months leading up to the first coding change, the 12 months between coding changes, and the 24 months following the April 2018 coding change.

Statistical analysis

Data analysis was performed through the Quality and Outcomes Research Unit in conjunction with the Health Informatics Department at University Hospitals Birmingham, using SAS/STAT software version 9.4. Models were constructed including cases with septicaemia coded as the dominant diagnosis. A change in prevalence of a comorbidity was determined by modelling the proportion of patients with the comorbidity before and after the coding changes using a logistic regression model with terms for provider, age and sample period.

Mortality status was obtained from the Office for National Statistics. Hospital mortality was modelled using a generalised linear mixed model. Provider and sample period were treated as random effects. The effects of case-mix, provider and sample period were modelled using the terms identified via backwards elimination logistic regression. Age was represented by a natural cubic spline with six knots placed at even percentiles of the data. Deprivation was represented by Quintiles plus a category for unmatched and missing data. Season was represented by a cyclic linear spline with knots in March (Spring), June (Summer), September (Autumn), and December (Winter).

Backwards elimination logistic regression was performed to identify comorbidities that were associated with death. Comorbidity classes that recorded death, for example ‘Sudden death cause unknown’ were excluded prior to analysis, as were comorbidity classes associated with fewer than 10 deaths in each year, to prevent numeric convergence problems and unstable parameters associated with these small groups. The logistic regression model also included demographic variables (age, sex, ethnicity, index of multiple deprivation), type of admission (emergency or non-emergency), time period, season and provider.

Results

The number of admissions where sepsis was recorded within the diagnosis is shown in Table 1, increasing from 199,395 between April 2016 and March 2017 (‘2016–2017’) to 312,780 between April 2017 and March 2018 (‘2017–2018’), 291,110 between April 2018 and March 2019 (‘2018–2019’), and 278,270 between April 2019 and March 2020 (‘2019–2020’).

Table 1 Number of patients with septicaemia recorded within diagnosis

In parallel with these changes in the absolute number of admissions in which sepsis was recorded, the proportion in which it was in the primary position increased in 2017–2018 versus 2016–2017 (56.0% vs. 34.6%, p < 0.005), after the first change in coding guidance. After the second change in coding guidance in April 2018 this proportion fell to 45.3% but remained significantly higher than in 2016–2017 (45.3% vs. 34.6%, p < 0.005). These overall trends were mirrored across different demographic groups, albeit that as age increases the proportion in which sepsis appeared in the primary vs secondary position also increased (Table 2).

Table 2 Demographics of patients with a recorded diagnosis of septicaemia

Inpatient mortality following admission with a diagnosis of sepsis is summarised in Fig. 1. Inpatient mortality associated with sepsis coded in the primary position fell from 17.8% (2016–2017) to 16.8% (2017–2018), 15.7% (2018–2019) and 16.2% (2019–2020). However, changes in coding practice were also associated with significant changes in comorbidities coded in the secondary positions, shown in Additional file 1: Table S1 and summarised in Table 3. Thus for 2017–2018 compared to 2016–2017, 84 diagnostic codes were more common and 27 less common in patients with sepsis in the primary position. Of those that were more common, 53 were associated with increased mortality and 31 with decreased mortality. Of those that were less common, 20 were associated with increased mortality and 7 with decreased mortality.

Fig. 1

Outcome of admission episodes with a diagnosis of septicaemia. Legend. Number of admissions shown for patients with septicaemia as primary diagnosis, or as secondary diagnosis, for survivors and for those who died, presented by month. Data from Hospital Episode Statistics (HES) Admitted Patient Care dataset

Table 3 Prevalence of comorbidities in patients with a primary diagnosis of septicaemia

As there was a change in the case-mix of patients with sepsis in the primary position, case-mix adjusted log odds of death were calculated from the mortality risk of comorbidities. The median log odds of death accounting for all coded comorbidities increased from − 2.21 (2016–2017) to − 2.10 (2017–2018), − 2.15 (2018–2019) and − 2.11 (2019–2020). There was therefore a small but significant increase in the calculated mortality risk of the population with sepsis in the primary position.

Figure 2 presents the findings of the mixed model in which provider and year were treated as random effects. The observed sepsis associated mortality (the log odds ratio for the provider in the year vs the average provider across all years) is plotted against the expected mortality (the within year normal standardised deviates of the case-mix adjusted predicted mortality). In this analysis, mortality falls from 2016–2017 to 2017–2018, and reduces a little further in the subsequent 2 years. This reduction in mortality occurs despite the adjusted mortality risk derived from the coded comorbidities increasing. The shift in mortality is consistent across all but one centre, which exhibited higher than expected mortality across all three years following the coding change. All other centres form a continuous distribution across the years studied; there were no other outlying centres in which the observed mortality significantly differed from that expected in 2018–2019 and 2019–2020.

Fig. 2

Hospital random effects model of case-mix adjusted mortality

Discussion

The recognition and response to sepsis is of concern to patients and healthcare professionals, since it is a significant cause of morbidity and mortality [2, 3]. NHSE have used various means to improve the recognition and treatment of sepsis, most recently mandating the use of NEWS2 in acute hospital trusts to screen for sepsis [10, 11]. The effect of such a screening strategy is poorly understood [21], nevertheless a range of centre specific quality improvement programmes have been described and endorsed by NHSE, in ways that appear to link process change to outcome, resulting in statements such as: ‘Hundreds of lives saved through new tech to spot sepsis’ [22]. This interpretation reflects the results of interrupted time series, during a period in which NHS coding guidelines for sepsis have changed twice, in April 2017 and April 2018. The effects of these changes on standardised measures of hospital mortality have been recognised. Our analysis set out to understand their effects on interpretation of centre specific changes in sepsis associated mortality, analysing Hospital Episode Statistics from 2016 to 2020.

As intended, in April 2017 there was a substantial increase in the number of patients in which sepsis was defined as the primary diagnosis. There was a coincident change in the case-mix which would have been expected to increase mortality associated with a primary diagnosis of sepsis. There were for example, 111 comorbidities associated with a change in sepsis associated mortality, whose prevalence changed following the introduction of the new coding recommendation. Comparing sepsis mortality across these time periods must therefore consider that this altered case-mix will itself impact, and increase, expected mortality, as the population now defined as sepsis has altered. This alteration in comorbidities contributes to complexity in the interpretation of changes in sepsis associated mortality within and between individual centres, in which there may be different population patterns of comorbidity and differences in coding practice, accentuated at times of change in coding as well as clinical practice [17, 19].

The mixed methods analysis performed here separates the effects of case-mix and centre, adjusting for the changes in case-mix that influence sepsis associated mortality, facilitating comparison of effects related to centre to expected distributions. This showed that variation in mortality between centres follows parallel distributions before and after coding changes, suggesting a systemic change that affected sepsis associated mortality across centres. This presentation focuses on observations which are inconsistent with expectations, identifying just one centre where reported mortality is higher than expected in 2018–2019 and 2019–2020. No centre was identified in which sepsis associated mortality was significantly better than expected. Although this does not preclude the possibility that specific interventions have influenced outcomes in individual centres, these would not be of sufficient magnitude to allow their identification within the observed level of random variation, including unpredictable changes in the application of coding guidance. After April 2018, the sequencing of conditions in episodes in which sepsis appears was left to ‘clinical judgement’, so that interactions between coder and clinician further influences local coding. This may itself be affected by conduits for that interaction, including the electronic healthcare record, and the emergence of local applications of specific terminologies, particularly in the context of local sepsis awareness campaigns. These are some of the potential contributors to unaccounted for inter-centre variation in coding for sepsis.

Although not formally proven, it is extremely likely that the shift in mortality from April 2017 is consequent upon systematic increases in coding for sepsis in the primary position, in patients with a lower mortality risk, that is to say with less severe disease, rather than being due to changes in clinical practice. This would be an expected consequence of a policy that sets out to increase coding for sepsis. Furthermore, there was no universal intervention introduced in April 2017 that would otherwise account for such a consistent change towards lower sepsis associated mortality across centres (with one centre excepted). Caution should therefore be exercised when interpreting the outcomes of interventions introduced from 2016 to 2018 in particular. This situation has now been further complicated by the emergence of COVID19.

We set out to understand whether reasonable conclusions could be made regarding the benefits of specific interventions, during a period of rapid change. Our findings illustrate the general issue of comparing outcomes, on the background of changing data definitions and standards over time and across systems [1, 23]. Use of coded data to provide longitudinal comparison and monitoring of outcomes, including mortality, is reliable only where definitions have remained consistent. Change in nationally reported mortality rates for specific diseases has been demonstrated previously following the introduction of new coding systems [24, 25]. The specific case of sepsis is important, given that the benefits of national policy and resulting organisational and individual behaviours, remain unproven [12, 26]. This needs to be considered by individual centres and regulators responding to adverse changes in sepsis associated mortality, as well as policy makers interpreting changes in outcome. This is not simply a question of case-mix but of case selection, changes in phenotype that are not captured within the case-mix.

Assessment of the effects of process or policy change could be improved, independent of the effects of coding, by using underlying data from electronic healthcare records, to consistently define the phenotype of interest. Previous research suggests that utilising clinical data from electronic healthcare records to monitor sepsis may provide more accurate estimates of sepsis incidence and its associated mortality in comparison to coded diagnoses [27]. Also, the NHS would be well placed to implement a systematic approach to the evaluation of interventions in the electronic environment, through cluster randomised studies; an approach well illustrated in a recent study of the automated identification of adults at risk of deterioration in hospitals in Northern California [28]. A more robust approach to the evaluation of practice than longitudinal analysis post policy change.

Our study relies on clinical coding which is recognised to under-report relevant comorbidities [29], vary between hospitals [30] and between diagnoses [31]. This does not alter our conclusions, rather is the reason to account for random effects in our analysis. The findings do not invalidate evaluations of interventions reporting improvements in process.

Conclusion

Changes in the coding of sepsis from medical records in England altered the case-mix and case-selection of patients, altering the expected mortality rate in patients where sepsis was recorded as a primary diagnosis. These changes resulted in systematic and random effects which impact upon the interpretation of centre specific mortality rates over time. This is therefore relevant to local quality management and improvement.

Although this focuses on sepsis, the principle applies to other fields where coding practice is subject to intended change or unrecognised drift. This must be considered in determining any clinical response recommendations arising from uncontrolled evidence.