Logistic regression model can reduce unnecessary artificial liver support in hepatitis B virus-associated acute-on-chronic liver failure: decision curve analysis

Background Several models have been proposed to predict the short-term outcome of acute-on-chronic liver failure (ACLF) after treatment. We aimed to determine whether better decisions for artificial liver support system (ALSS) treatment could be made with a model than without, through decision curve analysis (DCA). Methods The medical profiles of a cohort of 232 patients with hepatitis B virus (HBV)-associated ACLF were retrospectively analyzed to explore the role of plasma prothrombin activity (PTA), model for end-stage liver disease (MELD) and logistic regression model (LRM) in identifying patients who could benefit from ALSS. The accuracy and reliability of PTA, MELD and LRM were evaluated with previously reported cutoffs. DCA was performed to evaluate the clinical role of these models in predicting the treatment outcome. Results With the cut-off value of 0.2, LRM had sensitivity of 92.6 %, specificity of 42.3 % and an area under the receiving operating characteristic curve (AUC) of 0.68, which showed superior discrimination over PTA and MELD. DCA revealed that the LRM-guided ALSS treatment was superior over other strategies including “treating all” and MELD-guided therapy, for the midrange threshold probabilities of 16 to 64 %. Conclusions The use of LRM-guided ALSS treatment could increase both the accuracy and efficiency of this procedure, allowing the avoidance of unnecessary ALSS.


Background
Acute-on-chronic liver failure (ACLF) is characterized by severe jaundice, coagulopathy, hepatic encephalopathy (HE), and high morbidity and mortality. Hepatitis B virus (HBV) infection is the commonest cause of ACLF in the Asian region [1]. Currently, the treatment of HBV-associated ACLF (HBV-ACLF) is based on two main regimesantiviral treatment and artificial liver support system (ALSS) in China [2]. Antiviral therapy with nucleos(t)ide analogues has been proven to significantly improve prognosis and survival rate in HBV-ACLF patients [3]. However, the use of the ALSS has shown a variable range of safety and efficiency in several clinical trials [4][5][6][7][8]. In the last decade, several models have been proposed to predict the short-term outcome of HBV-ACLF after treatment. For instance, antiviral treatment can significantly decrease the 3-month mortality only in patients with the model for end-stage liver disease (MELD) score below 30, compared with those with higher MELD scores [9]. Another study suggested that ACLF patients with lower MELD scores showed significantly improved prognosis compared with those with higher MELD scores [10]. Therefore, the variable results of the ALSS treatment in terms of cost-effectiveness might be explained by the lack of guidance of such predictive models. An ideal prediction model may provide objective information about whether future treatment is likely to result in a favorable outcome or survival.
Recently, some studies revealed higher diagnostic accuracy for predictive models which combined liver dysfunction with etiological factor (e.g. HBV). In particular, Zheng et al., in a population of 452 patients with diagnosis of HBV-ACLF, established the logistic regression model (LRM) score, with an area under the receiving operating characteristic curve (AUC) of 0.844. With the cutoff of 0.2, a sensitivity of 86.7 and specificity of 75.5 % were reported [11]. LRM has shown promising results for prognosis prediction in HBV-ACLF ever since its introduction into clinical application. Yang et al. compared the predictive performance of MELD with that of LRM in a population of 273 HBV-ACLF patients. In ACLF patients with liver cirrhosis (LC), the AUC of LRM (0.851) was comparable with that of MELD (0.840). Yet, in patients with noncirrhotic ACLF, the AUC of LRM (0.897) was significantly higher than that of MELD (0.758) [12].
In a previous study, we reported that ALSS could improve short-and long-term prognosis in patients with HBV-ACLF [13]. Here we reanalyzed the data in order to determine whether better decisions for ALSS treatment could be made with a model (e.g. MELD or LRM) than without, through decision curve analysis.

Study patients
From January 2003 to December 2007, all patients admitted to our hospital with the diagnosis of HBV-ACLF according to the Chinese guidelines [14,15] were screened. Eligible patients were enrolled with the following criteria: (i) aged between 18 and 70 years; (ii) presumptively diagnosed as hepatitis B surface antigen (HBsAg) carrier, chronic hepatitis B (CHB) or HBV-related liver cirrhosis (HBC); (iii) progressive hyperbilirubinemia, with serum total bilirubin (TBil) ≥10 mg/dL; (iv) coagulopathy with plasma prothrombin activity (PTA) ≤40 % or international normalized ratio (INR) >1.5; (v) within 4 weeks from symptom onset complicated by ascites and/or HE. ACLF is further classified into early stage (30 % < PTA ≤ 40 %) and middle-to-end stage (PTA ≤ 30 %) [15]. Some patients were excluded according to the following criteria: acute HBV infection, super-infection with other viruses, chronic liver failure, coexistence of hepatocellular carcinoma (HCC), severe gastrointestinal bleeding, pregnancy, or liver transplant recipients.
The primary endpoint was 3-month survival. The secondary endpoint was survival at 5 years after diagnosis of ACLF.
The medical profiles of the patients were retrospectively analyzed in the present study. Our study protocol was approved by the institutional review board (IRB) of Nantong Third People's Hospital, Nantong University and conducted in accord with the ethical guidelines of the Declaration of Helsinki 1975. Patient consents were waived by the same IRB.

MELD calculation
MELD score (range from 6 to 40) was calculated according to the standard formula [16]. MELD = 11.2 × ln(INR) + 9.6 × ln[creatinine (mg/dL)] + 3.8 × ln[TBil (mg/dL)] + 6.4 (constant for liver disease etiology). Laboratory values of bilirubin, INR or creatinine less than 1 were rounded off to 1, in order to avoid negative scores. Creatinine greater than 4 mg/dL or with renal replacement therapy was capped at 4 mg/dL. In addition, the factor for etiology of liver disease was not used.

ALSS treatment
The patients with HBV-ACLF were assigned randomly to groups either given ALSS combined with standard medical therapy (SMT) or only SMT. The randomization was conducted by the Department of Epidemiology and Medical Statistics, Nantong University based on the SAS module, as we described in a previous study [13].

Statistical analysis
The accuracy of PTA, MELD or LRM was evaluated separately. For each model, the AUC, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), as well as diagnostic odds ratio (DOR), with the 95 % confidence intervals (CIs) were evaluated.
A simple decision tree is constructed with the HBV-ACLF patients in Fig. 1. The splitting parameter is recommendation for ALSS treatment or not. The a, b, c and d showing live or dead cases in each group, give the values of true positive, false positive, false negative, and true negative respectively. A cost-effective analysis, proposed by Vickers [17][18][19], was performed for each model using decision curve analysis (DCA). The net benefit of a model was estimated by the difference between the number of true-positive and false-positive, weighted by the odds of the specific threshold probability (Pt). A threshold value of, for example, 20 % indicates that survival probability of a treated patient is 80:20 = 4 times higher than the complications of an unnecessarily treated patient. Furthermore, using LRM for a particular patient, Pt of 10 % indicates that his possibility of LRM < 0.2 (ALSS treatment recommended) is 90:10 = 9 times higher than that of LRM ≥ 0.2 (ALSS treatment unnecessary). The net benefit of a model compared with another model or with the reference net benefit could be interpreted as the net increase in the proportion of cases identified. We calculate net benefit using the following formula [17] We set a zero net benefit by assuming no patient undergoing ALSS; on the contrary, we calculated the reference by assuming that all patients had undergone ALSS treatment. For any given threshold probability (Pt) cut point, the preferred model would be that with greater net benefit. The reduction in the number of unnecessary ALSS is calculated using the following formula [17]: reduction of avoidable ALSS treatment per 100 patients = (net benefit of the model net benefit of treat all)/[Pt/(1 -Pt)] × 100. The value is net of false negatives and is hence the equivalent to the reduction number of unnecessary ALSS without a decrease in the number of patients with ACLF who duly need ALSS.
All statistical analysis, including the DCA and plots implementation, were done using Stata version 13 (StataCorp, TX, USA). DCA analysis was performed using the code found at https://www.mskcc.org/departments/epidemiologybiostatistics/health-outcomes/decision-curve-analysis-01 according to its tutorials.

Patient characteristics
A total of 283 patients diagnosed with HBV-ACLF were screened for inclusion in this study cohort. Among these, 51 patients were excluded from this study: five for age over 70 years, eight for super-infection with hepatitis E virus, two for coexistence of HCC, seven for liver transplantation, 29 for incomplete data. Thus the final cohort compromised 232 patients with complete medical records.

Decision curve analysis
In Table 3 Decision curves for ACLF treatment, with the 3 analyzed models, were plotted in Fig. 3 to estimate the results in a clinical context. The net benefit of the single marker PTA was similar to the net benefit of random ALSS assignment. LRM-guided ALSS compared with "treating all" strategy lead to a greater net benefit for ever probability threshold starting from 16 %. Besides, it was always higher than the net benefit of MELD-guided strategy. Both the LRM and MELD model showed a superior net benefit compared with the random ALSS assignment. The net benefit of 0 at a Pt of 16 % could be interpreted in terms that if we perform ALSS based on LRM score, the consequence is equivalent to ALSS for all patients. Furthermore, at a Pt of 64 %, the net benefit for the prediction model is 0.321 greater than assuming all patients need ALSS. The net benefit formula was used to calculate which was the equivalent of a net 0.321 × 100/(0.64/0.36) = 18 fewer false positive results per 100 patients. It means that use of the prediction model might result in the equivalent of 18 % fewer ALSS in patients with ACLF, with no increase in the number of patients who need ALSS left untreated. For higher Pt (>64 %), the option to "not treat" is preferred and neither predictive model has value (Fig. 3, Table 4).

Discussion
Artificial liver support system was first applied to treat acute liver failure in 1970s with the attempt to replace certain detoxification function of the liver. On the one hand, the therapy cost is as expensive as nearly $US2500 for each session of ALSS in China. The incidence of adverse events (i.e. bleeding, hypotension, infection, coagulopathy, and catheter-related events) were reported [5,6,8,13,20]. On the other hand, several clinical trials and systemic reviews suggested that ALSS   ALSS artificial liver support system, LRM logistic regression model, Pt threshold probability could reduce mortality in ACLF patients compared with standard medical therapy [21][22][23][24]. Therefore, ALSS has been recommended as one important method for the treatment of ACLF [1,15,25,26]. It has been extensively debated whether to treat all ACLF patients or to treat selected patients. Some efforts have been made toward the identification of factors or models for predicting the prognosis after ALSS. For instance, many factors, including HE, PTA, bilirubin, creatinine, sodium, preexisting cirrhosis and age, were found as independent predictors for the short-term survival rate in ACLF [13]. Several models such as MELD and LRM, with recommended cutoff values, have been proposed to predict the survival outcomes of ACLF patients [11,27]. Since their introduction into clinical practice, the MELD and LRM scores have been tested in quite a few studies. Yet wide range of sensitivity and specificity has been reported in predicting mortality of ACLF patients [9,10,12,28]. Despite the well-known utility of MELD in allocating donor livers [27], the clinical utility of MELD and other models for other treatments remains unclear.
Although discrimination and calibration are essential aspects of a prediction model, they do not evaluate clinical usefulness such as the ability to make better decisions with a model than without. Decision curve analysis is a method for assessing the benefits of a model through a range of patient preferences in accepting risk of overtreatment and undertreatment to facilitate decision making [29,30]. The hypothesis in our study was that we may make better decisions for ALSS treatment with a model (e.g. MELD or LRM) than without. For a prediction model aiming to guide therapeutic decisions, a cut-off is required for the decision threshold. Right at the threshold, the likelihood of benefit, e.g. improved survival as a result of ALSS treatment, exactly balances the likelihood of harm, e.g. adverse events and expensive costs. However, as empirical evidence for the relative weight of benefits and harms is often lacking, it is always not easy to define a threshold.
In this study, we applied the DCA to evaluate the cost/benefit ratio of one single marker (i.e. PTA) and two models (i.e. MELD and LRM). The utility of PTA alone resulted in no more net benefit gain than random ALSS assignment. Using the MELD or LRM scores, some number of unnecessary ALSS treatment could be avoided at the cost that only a small proportion of patients with HBV-ACLF being advised not to undergo ALSS treatment. Starting from the cutoff of 16 %, the net benefit gain of LRM-guided strategy starts to be remarkable. The DCA results showed that for patients with threshold probabilities between 0 and 16 %, relatively preferring for empirical therapy, the net benefit is the greatest if all patients are treated. Across this range of threshold probabilities, patients tend to be more concerned about missing a timely treatment than about receiving unnecessary one. For the midrange threshold probabilities between 16 and 64 %, the LRM-guided ALSS therapy is superior to other strategies, including the MELD score. For higher thresholds (>64 %) at which patients appear be more concerned about unnecessary  ALSS artificial liver support system, ACLF, acute-on-chronic liver failure, LRM logistic regression model, Pt threshold probability of risk treatment than missed one, the option to not treat is preferred and none of the predictive models has value. Admittedly, there are some limitations in our study. First, we applied the decision curve analysis theory retrospectively with our cohort. Second, our findings were based on a small sample size. Finally, the present methodology may be appropriate for point decision making but not necessarily for decisions which reoccur over time, because the scores may frequently change in the natural history of ACLF.

Conclusions
Our findings indicate that the use of LRM-guided ALSS treatment could increase both the accuracy and efficiency of this procedure. Promising results from studies on the novel LRM score for ACLF prognosis may lead to better accuracy when predicting post-treatment outcomes in the near future, allowing the avoidance of unnecessary ALSS.