 Research article
 Open Access
 Open Peer Review
 Published:
Extensions to Regretbased Decision Curve Analysis: An application to hospice referral for terminal patients
BMC Medical Informatics and Decision Makingvolume 11, Article number: 77 (2011)
Abstract
Background
Despite the well documented advantages of hospice care, most terminally ill patients do not reap the maximum benefit from hospice services, with the majority of them receiving hospice care either prematurely or delayed. Decision systems to improve the hospice referral process are sorely needed.
Methods
We present a novel theoretical framework that is based on wellestablished methodologies of prognostication and decision analysis to assist with the hospice referral process for terminally ill patients. We linked the SUPPORT statistical model, widely regarded as one of the most accurate models for prognostication of terminally ill patients, with the recently developed regret based decision curve analysis (regret DCA). We extend the regret DCA methodology to consider harms associated with the prognostication test as well as harms and effects of the management strategies. In order to enable patients and physicians in making these complex decisions in realtime, we developed an easily accessible webbased decision support system available at the point of care.
Results
The webbased decision support system facilitates the hospice referral process in three steps. First, the patient or surrogate is interviewed to elicit his/her personal preferences regarding the continuation of lifesustaining treatment vs. palliative care. Then, regret DCA is employed to identify the best strategy for the particular patient in terms of threshold probability at which he/she is indifferent between continuation of treatment and of hospice referral. Finally, if necessary, the probabilities of survival and death for the particular patient are computed based on the SUPPORT prognostication model and contrasted with the patient's threshold probability. The webbased design of the CDSS enables patients, physicians, and family members to participate in the decision process from anywhere internet access is available.
Conclusions
We present a theoretical framework to facilitate the hospice referral process. Further rigorous clinical evaluation including testing in a prospective randomized controlled trial is required and planned.
Background
Introduction
Hospice services have been proven to provide better quality of care to dying patients[1–3] by optimizing pain relief [4, 5] and reducing emotional stress [1, 6, 7]. Furthermore, hospice care is associated with greater patientfamily satisfaction[8], is shown to be cost effective[9, 10], and most importantly, it has been attributed with increased survival in some patients [11]. Despite these well documented advantages, many terminally ill patients do not reap maximum benefits from hospice care. The fundamental reason for this is related to the less than optimal and frequently poorly timed referral of terminally ill patients to hospice [1, 12]. As a result, many patients die within a few days of referral, or live many years after the referral was made [13].
According to Medicare regulations, a person should be referred to hospice if his/her "life expectancy (LE) is 6 months or less" [1, 14]. Hence, the problem of meaningful referrals relates to the accurate estimation (prognosis) of death within approximately 6 months after evaluation for hospice care. However, statistical models designed to assist physicians in predicting life expectancy (LE), although beneficial [15, 16], so far they failed to improve the quality of care at the end of life [17–21].
One such statistical model is SUPPORT (Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments), designed to calculate the probability of survival over a period of 180 days [22, 23]. Although the SUPPORT model has been well validated [17, 22] for prognostication of LE in terminally ill patients, a controlled trial of SUPPORT failed to demonstrate any impact on the overall quality of care for these patients [17, 20]. We postulate that this lack of impact may be due to the fact that SUPPORT results, were not linked to any decision methodology that would translate the probability of survival to a hospice referral recommendation. Therefore, the full potential of the model's prognostication power remained unexploited.
In this work, we link the SUPPORT prognostication model with the recently developed decision methodology regret DCA[24] to facilitate the hospice referral process. Regret DCA relies on regret theory and decision curve analysis [25] to recommend the optimal management strategy for a patient, accounting for the personal attitudes and values of the particular patient or his/her surrogate.
Furthermore, we extend regret DCA to incorporate harms and effects of treatment as well as harms associated with the prognostication test to the decision model. The presented methodology is integrated into a comprehensive clinical decision support system developed to facilitate the hospice referral process.
Methods
Dataset
In our analysis, we utilized the entire SUPPORT dataset, both development and validation cohorts. The dataset is presented in detail elsewhere [22]. Medical records of 8,329 seriously ill hospitalized adults are included.
Support model
SUPPORT is a multivariable model designed to estimate probability of survival for seriously ill hospitalized patients over a period of the subsequent 180 days. The model variables include the patient's medical condition compatible with one of eight major diagnostic groupings (Acute Respiratory Failure, Multiple Organ System Failure, Chronic Obstructive Pulmonary Disease, Congestive Heart Failure, Hepatic Cirrhosis, Neurological Coma, Lung or Colon Cancer), the patient's current age, number of days in the hospital before study entry, neurologic status, and 11 physiologic measures recorded on day 3 after study entry [22].
The SUPPORT implementation for the estimation of survival probability is detailed in the appendix. Due to the nature of the hospice referral problem we also express the survival probability in terms of mortality. We can convert the estimated survival probability (SP) (equation A2) to probability of death within 180 days (denoted here as p) using the equation:
where SP is the survival probability computed by SUPPORT, i ∈ [1, 8] the patient's disease group, T is the survival time in days, and t is an arbitrary time (typically expressed in days e.g. t ∈ [1,180]).
In terms of accuracy, the SUPPORT model has an area under the receiveroperating characteristics curve (ROC) for prediction of surviving 180 days of 0.79 in the phase I development cohort and 0.78 in the phase II validation cohort [22].
Decision model
Figure 1 depicts the decision tree summarizing the process of hospice referral. The four outcomes and their corresponding utilities (U) shown are:

1.
U _{1}: Refer the patient to hospice and the patient's LE is less than or equal to 6 months (HospLE ≤ 6).

2.
U _{2}: Refer the patient to hospice and the patient's LE is greater than 6 months (HospLE > 6).

3.
U _{3}: Continue treating the patient and the patient's LE is less than or equal to 6 months (RxLE ≤ 6).

4.
U _{4}: Continue treating the patient and the patient's LE is greater than 6 months (RxLE > 6).
p is the probability associated with the presence of an event (e.g. patient's LE ≤ 6 months) as predicted by the SUPPORT model, 1  p is the probability associated with the absence of the same event (e.g. patient's LE > 6 months).
As with any decision, one may come to realize that, in retrospect, an alternative decision would have been preferable. This knowledge may bring a sense of loss or regret [26–32]. In this paper, we use this sense of regret to determine the preferences of the decision maker towards alternative management strategies. Specifically, we employ regret theory to estimate the threshold probability, P _{ t } , at which the decision maker (patient, physician, or family member) is indifferent between continuation of treatment vs. hospice referral. Based on the concept of threshold probability, the patient should be referred to hospice if his/her probability of death is greater than or equal to P _{ t } (e.g. p ≥ P _{ t } ), and he/she should continue receiving curative treatment otherwise (p < P _{ t } ).
The threshold probability is derived as [24]:
In (2) U _{ 1 } U _{ 3 } is associated with regret of omission (e.g. the patient was not referred to hospice, instead he/she continued receiving unnecessary treatment) and U _{ 4 } U _{ 2 } with regret of commission (e.g. the patient was unnecessary referred to hospice instead of continue receiving lifesustaining treatment) [24].
To elicit the decision maker's regret, and therefore threshold probability, we utilize the DVAS (Dual Visual Analogue Scale) method [24]. One visual analogue scale is used to capture the regret associated with failing to refer the patient to hospice (e.g. continue unnecessary treatment) and the second scale to measure the regret associated with unnecessary hospice referral (e.g. failing to provide lifesustaining treatment) (Figure 2).
Elicitation of threshold probability can be achieved through a set of questions such as:

1.
On the scale 0 to 100, where 0 indicates no regret and 100 indicates the maximum regret you could feel, how would you weigh the level of your regret if you were not referred to hospice but instead you continued receiving unnecessary treatment? That is, how much would you regret if you did not reap the benefits of hospice care? *Note that this value corresponds to U_{1}  U_{3}.

2.
On the scale 0 to 100, where 0 indicates no regret and 100 indicates the maximum regret you could feel, how would you weigh the level of your regret, if you were referred to hospice instead of continue receiving necessary lifesustaining treatment? That is, how much would you regret if you sustained harms from hospice care? *Note that this value corresponds to U_{4}  U_{2}.
For example, suppose that the patient  who is aware of his/her terminal condition answers 50 and 25 to the questions 1 and 2 respectively. This means that the patient considers 50/25 = 2 times worse not to be referred to hospice when necessary than receiving an unnecessary hospice referral. The threshold probability for this patient is (equation 2)
Regret DCA and extensions
The clinical problem we face in the situation of hospice referral is how to use reasonably accurate predictions of death, p, coupled with the patient's preferences (as expressed in terms of threshold probability, P _{ t } ) to arrive at the optimal decision for a specific individual. The problem is decomposed into three strategies: (1) act based on the prediction model (SUPPORT) (e.g. refer to hospice if p ≥ P _{ t } and continue treating otherwise), (2) refer all patients to hospice, and (3) continue current treatment for all patients (i.e. refer no patients to hospice).
Each of these strategies may inflict physiological and/or psychological damages to the patient. Specifically, a patient may suffer harms due to a treatment strategy (e.g. adverse effects) or harms due to the prognostication test (e.g. a test requiring invasive procedure). We express these harms as loss in utility associated with actions we may undertake. To that end, we define H _{ Rx } , H _{ Hosp } and H _{ te } as the utility losses due to harms of the treatment, hospice, and prognostic test, respectively.
Figure 3 presents the decision tree describing the overall hospice referral problem. p = P(D +) is the probability that the patient's LE is less than or equal to 6 months as estimated by the prediction model (SUPPORT);1  p = P(D ) is the probability that the patient's LE is greater than 6 months, and U _{ i } , i ∈ [1, 4], are the utilities corresponding to each of the decision model outcomes (detailed in the previous section). The variables Hosp and Rx correspond to referring a patient to hospice and continuing current curative treatment, respectively. Rg, is the regret associated with an action, e.g. Rg(Hosp, D) is the regret one may feel if the patient was referred to hospice when his/her LE was greater than 6 months. Finally, te designates that the patient received a prognostication test.
Considering the decision tree in Figure 3 we can compute the expected regret associated with each decision in terms of the utilities of each possible outcome as follows (detailed derivation is presented in the Appendix):
In addition to harms, equations 3, 4 and 5 incorporate the effects of treatment and hospice care using measures of Relative Risk Reduction: and RRR _{ Rx } RRR _{ Hosp } respectively. The values for these measures are treatment specific and can be acquired from the literature. We have incorporated hospice effects because a recent study [11] has shown that early palliative care for patients with metastatic nonsmall cell lung cancer could increase survival. The variables TP, FP, FN, TN are related to the prognostic capability of the SUPPORT model (see appendix for detailed derivation) [24].
Since the regret of omission and regret of commission have been generalized to include effects and harms related to management strategies and testing, the function of threshold probability (equation 2) becomes:
Where U _{1}  U _{3} + H _{ Rx } H _{ Hosp } corresponds to the regret associated with not referring the patient to hospice when necessary, and U _{4}  U _{2}  H _{ Rx }  H _{ Hosp } corresponds to the regret associated with unnecessary hospice referral.
Choosing the optimal strategy
The optimal strategy is selected as the one which will bring the least amount of regret. The regret DCA algorithm expresses the regret associated with each strategy in terms of threshold probability and is implemented as follows [24]:

1.
Select a value for threshold probability.

2.
Assuming that patients should be referred to hospice if p ≥ P _{ t } and should continue current treatment otherwise, compute #TP and #FP for the prediction model.

3.
Calculate the ERg(SUPPORT)using equation 5.

4.
Calculate ERg(Rx) using equation 4.

5.
Compute the ERg(Hosp)using equation 3.

6.
Repeat steps 1  6 for a range of threshold probabilities.

7.
Graph each expected regret function calculated in steps 35 against each threshold probability.
At each threshold probability, the action with the lowest value of expected regret corresponds to the most desired action. For example, in Figure 4, at a threshold probability equal to 10% (e.g. the patient considers 9 times worse not to be referred to hospice when necessary than to receive an unnecessary hospice referral), the optimal strategy is to refer the patient to hospice.
Figures 5, 6 and 7 depict the regret associated with alternative decision strategies as they relate to different values of hospice effectiveness (Figure 5), treatment effectiveness (Figure 6), and harms due to the prognostication test (Figure 7). As expected, when the harms due to the prognostication test are increased, then the area of threshold probability at which the prognostication model is the optimal decision is reduced (Figure 5). Even though, it is not expected that the SUPPORT model will actually create harms, at least physiological, to the patient, this is not always the case for other diagnostic tests that may be more invasive (e.g. screening for prostate cancer).
As can be seen from Figures 5, 6 and 7 the optimal decision is derived by the SUPPORT model for a rather wide range of threshold probabilities. Therefore, it appears that the SUPPORT model is the superior strategy for the vast majority of decision makers, regardless the effects of the alternative management strategies. However, since the threshold probability expresses the personal preferences of a particular decision maker, it is not unusual for specific patients to have smaller or greater threshold probability values than the majority of decision makers. This is the power of the proposed methodology, which allows for decision making at the individual level. For example, if the decision maker presents a threshold probability greater than ≈92%, the optimal decision would be to continue lifesustaining treatment even if it is deemed not to be effective(Figure 4). Similarly, for small values of threshold probability, the desired action would be to refer the patient to hospice.
Decision Support System
As our theoretical discussion highlighted, decisions about life and death are complex and difficult at both the emotional and cognitive level. Therefore, it is not surprising that the SUPPORT model originally failed to improve the quality of care for terminally ill patients despite its reasonable accuracy in prediction of probability of survival [17, 20]. Any attempt to focus on a single dimension of the complex hospice referral process is not likely to succeed. An accurate prognostic model is only the first step. Having the apparatus to take into account tradeoffs associated with the hospice referral decision while taking into consideration the patients' preferences represent further necessary steps to improve the care of terminally ill patients. In addition, we hypothesize that the SUPPORT intervention failed because it was not available at the point of care in real time. This is because the most desired outcomes are best achieved when decisionmaking occurs in realtime, at the point of care [33, 34].
To facilitate the decision making process for the hospice referral at bedside, we propose a webbased clinical decision support system (CDSS) that computes the probabilities of survival and death for individual patients using the SUPPORT model, elicits personal preferences from patients and/or physicians, and utilizes regret DCA to suggest the optimal decision for a particular patient.
Features
Access
Our goal is to develop a CDSS that can be accessed by everyone and from anywhere regardless the operating system one uses. At the same time, it is desirable to develop a system that can eventually be integrated with various healthcare providers' electronic medical records (EMR). We concluded that a webbased implementation would fulfil such requirements.
Data storage
The CDSS performs the required computations without retaining or transmitting sensitive and identifiable information.
Results
In this section we present a prototype of the CDSS, developed to demonstrate the applicability of our theoretical framework for hospice referral. Each subsection describes the results of the methods shown in the previous section in conjunction with the description of the corresponding module. Figure 8 depicts the logical diagram that outlines the operation of the CDSS. Briefly, the operation begins by interviewing the patient or surrogate to elicit his/her threshold probability. Based on the value of threshold probability equation, the optimal strategy for the particular patient is derived (e.g. refer to hospice, continue treatment, or use of prediction model). If the optimal strategy is to follow the prediction model (SUPPORT), then using the equations A1, A2 and 1, the probabilities of survival and death are computed for the particular patient. The probability of death is then contrasted with the patient's threshold probability and the optimal decision is derived (refer to hospice, or continue treatment). At each step described, the patient selects the level of information he/she wishes to be exposed to. For example, the patient may not wish to know his/her threshold probability or probability of death. Instead he/she wishes to know only the optimal decision regarding his/her condition.
General implementation details
The proposed CDSS is a webbased application residing on the USF Health servers. The web address is http://health.usf.edu/research/ebm/decisionaids.htm. It has been developed based on the Adobe^{®} ColdFusion^{®} application technology and the interface has been designed using html and JavaScript programming languages. The hardware and software requirements from the user's point of view are modest. The system runs on any contemporary computer with net browsing capabilities. However, at this stage the CDSS is not optimized for use with handheld devices. The CDSS consists of 3 different modules as described below.
Elicitation of threshold probability module
The threshold elicitation module consists of the dual visual analogue scales, used to weigh the patient's regret in the case of wrong decisions. Each scale has 100 points where 0 corresponds to no regret and 100 to maximum regret. Depending on the role of the decision maker (e.g. patient/surrogate or physician) two different sets of questions are displayed. These questions are designed to capture the regret of omission and the regret of commission. For the remainder of this paper, we assume that the decision maker is the patient. As in pain scales [35], each visual analogue scale uses facial expressions to graphically represent variations in regret (Figure 9). A summary of the decision maker's preferences is presented for final verification. The threshold probability for the particular patient is derived using equation 2, however is not displayed until the decision maker requests it.
Decision module
The decision module utilizes the decision maker's threshold probability and the regret DCA methodology to derive the optimal decision. For example, the preferences of the patient depicted in Figure 9, correspond to a threshold probability equal to 29%. From Figure 4 the strategy that will bring the least amount of regret is to use the prognostication model (SUPPORT) for the hospice referral recommendation. In this case, the decision module initiates the SUPPORT module.
SUPPORT module
If the optimal strategy derived by the decision module is to utilize the prognostication model, the SUPPORT module is enabled. This module (Figure 10) is used to compute the probability of death for the particular patient based on the SUPPORT prognostication model. Currently, the user inserts all required information to the CDSS. In the future, this information will be captured automatically from the health care provider's electronic medical records system. Data validation restrictions have been imposed to protect the integrity of the collected data.
Once the values of all available variables have been inserted in the corresponding cells, the patient's life expectancy and probabilities of survival and death are computed. The decision module is employed again to display the optimal recommendation.
Decision justification module
The decision justification module explains in detail and at the user's request the reasons that led to a particular recommendation (Figure 11). It contains information regarding the decision maker's threshold probability, the optimal strategy associated with the threshold probability and the patient's probability of death (if applicable). Since people often misinterpret probabilities [36], we complement the results presented in terms of probabilities using frequency format (Figure 11). The latter format is currently considered the best way to represent favorable and unfavorable facts regarding medical interventions [37]. The justification module is highly technical and should only be reviewed by decision makers who wish to know more about their or their patient's condition.
Case Study
Figure 12 summarizes the decision process for a patient whose information is simulated in Figures 10, 11 and 12. The probability of death and the threshold probability of this patient have been computed as 85% and 29% respectively. At a 29% threshold probability, the optimal strategy is to use the prediction model for hospice referral (Figure 4). Therefore, since p > P _{ t } the patient should be referred to hospice. For completeness, all possible decision routes are depicted in Figure 12. The route corresponding to the specific simulated patient is shown using bold arrows.
Discussion
In this article we describe both the theory and application behind a hospice referral clinical decision support system. To the best of our knowledge, this is the first CDSS that integrates two well established methodologies, one for prognostication (SUPPORT) and the other for decision making (regret DCA), to assist with the hospice referral decisionmaking process.
The recently developed regret DCA incorporates the decision maker's preferences towards alternative management strategies from the perspective of regret theory in terms of threshold probability. Such an approach promotes personalized patient care. We anticipate that the regretbased approach is more appropriate for the hospice referral process than other preference elicitation techniques, due to the nature of the problem where there are really no optimal options available the optimal decision can be only considered as the one with the least regret.
Modern cognitive theories increasingly focus on the so called dualprocessing theory in which both intuition (system 1) and analytical, deliberative process (system 2) are important for balancing risks and benefits in the decisionmaking process [38]. We believe that rational decisionmaking should take into account both formal principles of rationality and human intuition about good decisions[24, 39, 40]. One way to accomplish this is to use regret, a cognitive emotion, to serve as the link between systems 1 and 2 [24]. By taking into account the consequences of our actions as well as the circumstances under which we can live with our mistakes we anticipate that the goal of reconciling the formal principles of rationality and human intuitions about good decisions can be met [24, 29, 39, 40]. This is particularly true in the situation of terminally ill patients.
Our webbased CDSS reflects modern cognitive theories to facilitate integration of the decisionmaking ingredients necessary for hospice referral decisions. The CDSS encapsulates all required information for the hospice referral process into a flexible software that can be used at bedside. Obviously, hospice referral decisions are complex and must be exercised with full compassion and deliberation. We advise against the use of our system as an automatic decision making tool that bypasses important personal interactions between the patient and his/her physician. It is important to stress that the elicitation of the threshold probability as described herein reflects the belief (also captured in recent legislation [41]) that patients and their families want to be told the "truth" about the patients' terminal sickness [22, 41, 42] and that physicians have ethical obligations to share this information with patients and their families [41, 42]. Our system should be understood as an aid to facilitate decisions in terminal phases of patient lives.
Our approach has limitations as well. The main limitation of the proposed system remains the complexity of the SUPPORT model. Currently, the system still requires manual entry of data. In addition, failure to enter all data can jeopardize the accuracy of prediction and therefore, the decision process. To cope with this limitation, we plan to integrate our system into various health providers' EMRs. Based on each EMR, specifically designed queries will be used to retrieve lab values and patient demographics to be fed automatically into our system; a process that will reduce the amount of missing values and input errors.
The second limitation of the proposed system is that empirical data are not available to assess how the system actually works in practice. While we plan to undertake empirical testing of the system described here, we believe that a strong theoretical underpinning will enable better hospice referral decisions even in the current form. This is because our system will essentially operationalize the decisionmaking process, which is supposed to occur in every day practice. Nevertheless, we need to firstly, identify the system's feasibility in real life settings and ultimately, if it appears to be usable and assessed favourably by all those involved in the hospicereferral decisionmaking process, to test it in randomized controlled trials against traditional care.
Our future plans include both empirical testing and implementation of multiple additional prognostication models which will be used in parallel to assess optimal decisions regarding hospice referral and take advantage of the regret DCA methodology. We anticipate that for a different range of threshold probabilities these models may perform better than the SUPPORT model. Furthermore, our intent is to develop a separate version of our CDSS optimized for mobile devices.
Conclusions
In this work we have presented the theoretical framework, accompanied by the associated CDSS, to facilitate end of life care decisions. Our work combines the prognostication power of the SUPPORT model, the simplicity of the DVAS methodology in eliciting people's preferences and the effectiveness of regret DCA at evaluating alternative management strategies to resolve the dilemma of choosing traditional vs. palliative care for patients at terminal stages. A clinical evaluation of the CDSS is planned.
Appendix
Support implementation
SUPPORT is implemented in two steps. First, the SUPPORT physiology score is computed based on equation A1 [22].
where:Alb: albumin; Aresp: APACHE III respiration score; Bil: bilirubin; Cr: Creatinine; Na: sodium; PaO _{ 2 } : partial pressure oxygen in arterial blood; MeanBP: mean arterial blood pressure; WBC: white blood cell count in thousands; Temp: temperature in Celsius; HR: heart rate per minute; ARF: Acute respiratory failure; MOSF: Multiple organ failure; Cirrhosis: Cirrhosis; Coma: Coma; Lung: Lung cancer; ColonCancer: Colon cancer; COPD: Chronic obstructive pulmonary disease; CHF: Congestive heart failure. Also:
WBC = 9, if WBC < 9 and {disease group } ≠ ARF /MOSF
WBC = 40, if WBC > 40
Cr = 15, if Cr >15
The second step in implementing the SUPPORT model is to calculate the probability of survival for the individual patient based on equation A2 [22].
where T: survival time in days; t: arbitrary time; S described in Table 1[22] and
where Scoma: SUPPORT coma score (0100); MOSFw/Malig: Multiple organ failure with malignancies; Hday: day in hospital when qualified for study; Cancer: Cancer by comorbidity or primary disease category (0 = no; 1 = present; 2 = metastatic) [22].
Derivation of the Expected Regret functions
As outlined in the Introduction, seriously and terminally ill patients may reap a number of benefits by the hospice program. Nevertheless, after enrollment into hospice, the patient (or the family, or the physician) may feel that this was a wrong decision, and subsequently may regret it. Similarly, the patient may feel regret for the treatment that he/she continues to receive because it is unnecessary, inappropriate, and/or harmful. Figure 3 represents our hospice decision tree in terms of regret from which we can compute the expected values of regret associated with each strategy as follows:
The variables TP, FP, TN, FN are related to the probabilities P (p ≥ P _{ t } ∩ D +), P (p ≥ P _{ t } ∩ D ), P (p < P _{ t } ∩ D ) and P (p < P _{ t } ∩ D +) respectively, and are estimated as follows:

P (p ≥ P _{ t }∩ D +) ≈ the number of patients who will die within 6 months and for whom the prognostic probability is greater than or equal to P _{ t }(with #TP = number of patients with true positive results, $P\left(p\ge {P}_{t}\cap D\phantom{\rule{2.77695pt}{0ex}}+\right)\approx \frac{\#TP}{n}$, where n is the total number of patients in the study).

P (p ≥ P _{ t }∩ D ) ≈the number of patients who will survive for longer than 6 months and for whom the prognostic probability is greater than or equal to P _{ t }(with #FP = number of patients with false positive results, $P(p\ge {P}_{t}\cap D)\approx {\scriptscriptstyle \frac{\#FP}{n}})$.

P (p < P _{ t }∩ D +) ≈the number of patients who will die within 6 months and for whom the prognostic probability is less than P _{ t }(with #FN = number of patients with false negative results, $P(p<{P}_{t}\cap D+)\approx {\scriptscriptstyle \frac{\#FN}{n}})$.

P (p < P _{ t }∩ D ) ≈the number of patients who will survive for longer than 6 months and for whom the prognostic probability is less than P _{ t }(with #TN = number of patients with true negative results, $P(p<{P}_{t}\cap D)\approx {\scriptscriptstyle \frac{\#TN}{n}})$.
To incorporate the effects of alternative treatments (e.g. treatment and hospice care) in equations A3A5 we use the Relative Risk Reduction reported in literature for each strategy as follows:
Since TP + FN = 1 and FP + TN = 1, we have:
Therefore, equation A8 becomes:
Scaling the equations A3, A4 and A9 with the quantity (U _{1}  U _{3} + H _{ Rx }  H _{ Hosp } ) and replacing the expression $\frac{{U}_{4}{U}_{2}\left({H}_{Rx}{H}_{Hosp}\right)}{{U}_{1}{U}_{3}+{H}_{Rx}{H}_{Hosp}}$ with $\frac{{P}_{t}}{1{P}_{t}}$, we derive the final equations for the expected regret (equations 3, 4, and 5).
References
 1.
Christakis NA: Death foretold. Prophecy and prognosis in medical care. 2001, The University of Chicago Press
 2.
Bulkin W, Lukashok H: Rx for dying: the case for hospice. N Engl J Med. 1988, 318 (6): 376378. 10.1056/NEJM198802113180609.
 3.
Christakis NA, Iwashyna TJ: Spousal illness burden is associated with delayed use of hospice care in terminally ill patients. J Palliat Med. 1998, 1 (1): 310. 10.1089/jpm.1998.1.3.
 4.
Goldberg RJ, Mor V, Wiemann M, Greer DS, Hiris J: Analgesic use in terminal cancer patients: report from the National Hospice Study. J Chronic Dis. 1986, 39 (1): 3745. 10.1016/00219681(86)901050.
 5.
Greer DS, Mor V: An overview of National Hospice Study findings. J Chronic Dis. 1986, 39 (1): 57. 10.1016/00219681(86)901025.
 6.
Moinpour CM, Polissar L: Factors affecting place of death of hospice and nonhospice cancer patients. Am J Public Health. 1989, 79 (11): 15491551. 10.2105/AJPH.79.11.1549.
 7.
Kane RL, Klein SJ, Bernstein L, Rothernberg R, Wales J: Hospice role in alleviating the emotional stress of terminal patients and their families. Med Care. 1985, 23 (3): 189197. 10.1097/0000565019850300000001.
 8.
Dawson NJ: Need satisfaction in terminal care settings. Soc Sci Med. 1991, 32 (1): 8387. 10.1016/02779536(91)90131U.
 9.
Kidder D: The effects of hospice coverage on Medicare expenditures. Health Serv Res. 1992, 27 (2): 195217.
 10.
Mor V, Kidder D: Cost savings in hospice: final results of the National Hospice Study. Health Serv Res. 1985, 20 (4): 407422.
 11.
Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, Dahlin CM, Blinderman CD, Jacobsen J, Pirl WF: Early palliative care for patients with metastatic nonsmallcell lung cancer. N Engl J Med. 2010, 363 (8): 733742. 10.1056/NEJMoa1000678.
 12.
Christakis NA: Timing of referral of terminally ill patients to an outpatient hospice. J Gen Intern Med. 1994, 9 (6): 314320. 10.1007/BF02599178.
 13.
Christakis NA, Escarce JJ: Survival of Medicare patients after enrolment in hospice programs. The New England Journal of Medicine. 1996, 335 (3): 172178. 10.1056/NEJM199607183350306.
 14.
 15.
Dawes RM, Faust D, Meehl PE: Clinical versus actuarial judgment. Science. 1989, 243 (4899): 16681674. 10.1126/science.2648573.
 16.
Hastie R, Dawes RM: Rational choice in an uncertain world. 2001, London: Sage Publications Inc
 17.
The Support Investigators: A Controlled Trial to Improve Care for Seriously III Hospitalized Patients: The Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments (SUPPORT). JAMA. 1995, 247 (20): 15911598.
 18.
Cook DJ, Guyatt GH, Jaeschke R, Reeve J, Spanier A, King D, Molloy DW, Willan A, Streiner DL, Canadian Critical Care, Trials G: Determinants in Canadian Health Care Workers of the Decision to Withdraw Life Support From the Critically Ill. JAMA. 1995, 273 (9): 703708. 10.1001/jama.1995.03520330033033.
 19.
Hamel MB, Goldman L, Teno J, Lynn J, Davis RB, Harrell FE, Connors AF, Califf R, Kussin P, Bellamy P: Identification of Comatose Patients at High Risk for Death or Severe Disability. JAMA. 1995, 273 (23): 18421848. 10.1001/jama.1995.03520470050031.
 20.
Teno J, Lynn J, Connors A, Wenger N, Phillips RS, Alzola C, Murphy DP, Desbiens N, WA K: The illusion of endoflife resource savings with advance directives. SUPPORT Investigators. Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatment. J Am Geriatr Soc. 1997, 45 (4): 519520.
 21.
Teno JM, Lynn J, Phillips RS, Murphy D, Youngner SJ, Bellamy P, Connors AFJ, Desbiens NA, Fulkerson W, Knaus WA: Do formal advance directives affect resuscitation decisions and the use of resources for seriously ill patients? SUPPORT Investigators. Study to Understand Prognoses and Preferences for Outcomes and Risks of Treatments. J Clin Ethics. 1994, 5 (1): 2330.
 22.
Knaus WA, Harrell FE, Lynn J, Goldman L, Phillips RS, Connors AF, Dawson NV, Fulkerson WJ, Califf RM, Desbiens N: The SUPPORT Prognostic Model: Objective Estimates of Survival for Seriously Ill Hospitalized Adults. Annals of Internal Medicine. 1995, 122 (3): 191203.
 23.
Lynn J, Knaus WA: Background for SUPPORT. J Clin Epidimiol. 1990, 43 (Suppl): 1S4S.
 24.
Tsalatsanis A, Hozo I, Vickers A, Djulbegovic B: A regret theory approach to decision curve analysis: a novel method for eliciting decision makers' preferences and decisionmaking. BMC Med Inform Decis Mak. 2010, 10: 5110.1186/147269471051.
 25.
Vickers A, Elkin E: Decision curve analysis: a novel method for evaluating prediction models. Med Dec Making. 2006, 26 (6): 565574. 10.1177/0272989X06295361.
 26.
Djulbegovic B, Hozo I: When Should Potentially False Research Findings Be Considered Acceptable?. PLoS Med. 2007, 4 (2): e2610.1371/journal.pmed.0040026.
 27.
Djulbegovic B, Hozo I, Schwartz A, McMasters KM: Acceptable regret in medical decision making. Med Hypotheses. 1999, 53 (3): 253259. 10.1054/mehy.1998.0020.
 28.
Hozo I, Djulbegovic B: When is diagnostic testing inappropriate or irrational? Acceptable regret approach. Med Dec Making. 2008, 28 (4): 540553. 10.1177/0272989X08315249.
 29.
Hozo I, Djulbegovic B: Will insistence on practicing medicine according to expected utility theory lead to an increase in diagnostic testing?. Med Dec Making. 2009, 29: 320322. 10.1177/0272989X09334370.
 30.
Bell DE: Regret in Decision Making under Uncertainty. Operations Research. 1982, 30: 961981. 10.1287/opre.30.5.961.
 31.
Loomes G, Sugden R: Regret theory: an alternative theory of rational choice. Economic J. 1982, 92: 805824. 10.2307/2232669.
 32.
Zeelenberg M, Pieters R: A theory of regret regulation 1.1. J Consumer Psychol. 2007, 17: 2935. 10.1207/s15327663jcp1701_6.
 33.
Olsen LA, Aisner D, McGinnis MJ: The learning healthcare system: Workshop summary (IOM roundtable on evidence based medicine). 2008, Washington: The National Academic Press
 34.
Eden J, Wheatley B, McNeil B, Sox H: Knowing what works in health care: A roadmap for the Nation. 2008, Washington: The National Academies Press
 35.
McCaffery M, Beebe A: Pain: Clinical manual for nursing practice. 1993, Baltimore: VV Mosby Company
 36.
Gigerenzer G, Todd PM, ABCResearchGroup: Simple heuristics that make us smart. 1999, New York: Oxford University Press
 37.
Schwartz LM, Woloshin S, Welch HG: Using a Drug Facts Box to Communicate Drug Benefits and Harms. Annals of Internal Medicine. 2009, 150 (8): 516527.
 38.
Kahneman D: Maps of bounded rationality: psychology for behavioral economics. American Economic Review. 2003, 93: 14491475. 10.1257/000282803322655392.
 39.
Krantz DH, Kunreuther HC: Goals and plans in decision making. Judgement and decision making. 2007, 52 (3): 137168.
 40.
Rawls J: A theory of justice. Revised edition. 1999, Cambridge: Harvard University Press
 41.
Astrow AB, Popp B: The Palliative Care Information Act in real life. N Engl J Med. 2011, 364 (20): 18851887. 10.1056/NEJMp1102392.
 42.
Mack JW, Weeks JC, Wright AA, Block SD, Prigerson HG: Endoflife discussions, goal attainment, and distress at the end of life: predictors and outcomes of receipt of care consistent with preferences. J Clin Oncol. 2010, 28 (7): 12031208. 10.1200/JCO.2009.25.4672.
Prepublication history
The prepublication history for this paper can be accessed here:http://www.biomedcentral.com/14726947/11/77/prepub
Acknowledgements
This work is supported by the Department of Army grant #W81 XWH 0920175 (PI Djulbegovic).
Author information
Additional information
Competing interests
The authors declare that they have no competing interests.
Authors' contributions
All authors contributed equally to this work. All authors have read and approved the final manuscript.
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Rights and permissions
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
About this article
Received
Accepted
Published
DOI
Keywords
 Chronic Obstructive Pulmonary Disease
 Optimal Decision
 Decision Module
 Threshold Probability
 Clinical Decision Support System