Development and preliminary user testing of the DCIDA (Dynamic computer interactive decision application) for ‘nudging’ patients towards high quality decisions

In recent years, numerous patient decision aids (PtDA) have been developed to facilitate informed, value based decisions about treatment options [1]. They have been developed in response to many beneficial treatments or screening strategies which also have negative aspects such as side-effects or high costs. What is best for one patient may be different to another depending on how each values the attributes of each option [2]. Health professionals are often poor proxies of patients’ values [3, 4], and often fail appropriately ‘diagnose’ patient preferences [5]. Patients can also have unrealistic expectations of treatment benefits and harms [6].

PtDAs provide facts about the condition, options, and attributes such as outcomes and probabilities for each option; a value clarification task that helps patients evaluate which attributes matter most to them; and a guide in the steps of deliberation and communication required for the informed patient to reach their goal – concordance between what matters most to them and their chosen option [7]. An updated Cochrane systematic review found that, among 115 studies involving 34,444 participants, PtDA increase patients’ knowledge about treatment options, and reduce their decisional conflict related to feeling uninformed and unclear about their personal values [1].

There have been tremendous advances in the way PtDAs are developed, from the way risks are presented [8], to the use of animated stories to better communicate information [9]. However, there has been comparatively little research on reducing decision errors in people using PtDAs [10]. While in theory, PtDAs should help patients identify the best treatment option for them, research shows that various cognitive biases may result in errors that prevent this in some situations [11–14]. For example, individuals are known to make different choices when their options are framed as gains or losses, preferring a surgical procedure with a 90% survival rate to one with a 10% mortality rate [15]. Studies have shown that individual treatment choices are unduly influenced by whether individuals learn first about potential harms or potential benefits [10], and individuals are intimidated and overwhelmed by options that include numerous rare side-effects leading to irrational decisions [16].

The objective of this study was to develop and user test a computer application that enhances conventional PtDAs to improve the quality of decisions by helping patients overcome common decision errors.

Development of DCIDA

Theoretical motivation

Normative decision theory suggests that for patients to approach treatment or screening decisions rationally they need to weigh benefits and harms using deliberative “compensatory strategies” to make trade-offs [17]. That is, a patient can “compensate” for the negative feature of one option by considering a positive feature. A patient looking at cancer screening options may not want to have annual testing, but would not mind if such frequent testing were non-invasive. While this approach helps people identify the treatment option that matches their informed values, descriptive decision theory has identified numerous errors in peoples’ decisions caused by cognitive biases and simplifying heuristics [12–15]. An understanding that people have two systems for cognitive functioning has provided a framework for understanding these errors and providing effective strategies for improving decision making [18]. System 1 refers to people’s intuitive system, which is typically fast, automatic, effortless, implicit, and emotional. While often useful for simple decisions, they can lead to decision errors for more difficult decisions, such as ones requiring compensatory strategies. System 2 refers to reasoning that is slower, conscious, effortful, explicit, and logical. Recent research suggests that when faced with decisions and information that is unfamiliar, complex, or overwhelming – all common traits targeted by most PtDAs – people can switch to use System 1 functioning, which can lead to decision errors [12, 13].

By reviewing PtDAs contained in the Ottawa repository [19], the project team identified few examples where PtDAs helped patients making trade-offs, and beyond simple value clarification exercises gave patients little help in choosing what was best for them. The team identified four issues common to nearly all PtDAs that were believed could impede the quality decision-making, and for which interventions were feasible: unstable preferences, order effects, overweighting of rare events, and information overload.

1. i)

   Unstable values: Unlike most goods and services, where markets help form stable values and consumers understand the relative value of attributes through a process of trial and error and notion of sacrifice, evidence indicates that patients often have unstable values when it comes to health care [20, 21]. This follows from the fact that the potential benefits and harms of treatment may be unfamiliar to patients and impossible to evaluate without a great deal of information and reflection. The consequence is that people’s decisions may be inconsistent with their true underlying values.

    
2. ii)

   Order effects: For a rational decision maker, the way information is presented should not have an influence on the choice that is made. However, there is evidence that people tend to, for example, remember information presented to them first (primacy effect) or last (recency effect) [22–24]. The decision error in PtDAs caused by this heuristic is demonstrated in a study of women at high risk of breast cancer who were considering tamoxifen, which found that patients who learned first about the risks of the drug thought more favourably of tamoxifen than patients who learned first about the benefits [10]. Nevertheless, harms and benefits must be presented in some order in a PtDA, and therefore, the designer of the PtDA may inadvertently influence the patient to choose a given option.

    
3. iii)

   Overweighting of rare events: Most treatments have multiple complications. PtDAs are expected to inform patients about any treatment complication that is reasonably likely to occur. Although there are no absolute criteria regarding which complications must be included for meeting informed consent, most PtDAs show information on any moderately severe complication that occurs at least 1% of the time, and serious complications that occur even less often than that. Prospect theory describes how people systematically overweigh small probabilities in terms of their impact on decisions. The consequence of this phenomenon is that patients using PtDAs can be scared away from treatments with multiple rare side-effects that rationally would have appeared the best treatment option for them [14, 16].

    
4. iv)

   Information overload: When individuals receive conflicting, incomplete, uncertain, or excessive information, they experience ambiguity and can make contradictory decisions [25]. The role of information overload causing ambiguity in investment decision-making has been well documented. When the complexity of decision-making increases, people tend to expend less effort to actually make their decision, seek others to make decisions for them, or select default options if available [26].

    

A number of promising strategies have been uncovered for overcoming specific decision errors. One approach is to encourage people to use System 2 thinking instead of System 1 by making the information and decision less overwhelming. This can be achieved by focussing attention on the most pertinent information, and by using analytic processes which reduce the number of ‘internal calculations’ which require cognitive effort [27–30]. In the area of PtDAs, the predominant approach has been to employ formal decision analysis techniques which quantifies patients' values and integrates them with probabilistic information [31–33]. While there are many perceived advantages to this prescriptive approach, there are also criticisms. First, ‘optimal’ options derived from decision analysis are reliant on assumptions, theories and inaccuracies in inputs which mean they may not actually prescribe the best course of action for each patient [34]. Second, the current approaches to decision analysis are typically ‘overt’ to be best course of action, and consequently have been argued to be an extension of paternalism, compromising patient autonomy [35].

An alternative strategy is to leverage System 1 thinking by changing the decision environment to maximize the odds that people will make high quality decisions given known biases [26]. For example, it is known that most people have a bias towards inaction, in which providing a default option has been found to be a powerful decision enhancer [36]. These strategies have increasingly been referred to as ‘nudges’, reflecting the unavoidable paternalistic role of the designer of the tool (in this case, PtDAs) in influencing users’ choices [37]. Nudges have been defined as “…any aspect of the choice architecture that alters people’s behavior in a predictable way without forbidding any options or significantly changing their economic incentives…” [37], and so seek to preserve patient autonomy and freedom of choice.

We sought to employ decision analysis more covertly [38] to improve decision-making by testing various ‘nudges’ which help people focus on the information and options that reflect their values, and simplify their trade-offs.

Features

We developed a dynamic computer interactive decision application ( DCIDA - pronounced ‘decider’) to employ some of the strategies described above. The overarching aim of the DCIDA is to present information and the decision to each person in an individualized way in order to maximize their ability to make choices that reflect their own informed, stable values. Acknowledging that it is rare that any decision support system will induce optimal decisions, the goal is to improve the quality of decisions, from what would be made with conventional PtDAs. Multi-criteria decision analysis (MCDA) using a weighted additive model motivates the application. For treatment decisions, this assumes the preferred option is based on the sum of the importance or weight (on a 0–100 scale) of each attribute, say benefit or harm, multiplied by each option’s score (on a 0–100 scale) for that attribute. The treatment with the highest weighted score or expected value indicates the patient’s optimal option.

The application contains the same content and information as a PtDA, explaining the condition, providing information about options and their characteristics (benefits, side-effects, costs etc.) using probabilities and pictographs to describe baseline and incremental absolute risks where appropriate, a value clarification exercise, and a summary of information to help the patient deliberate on the decision along with an opportunity to select the preferred option. However, the way this content and information is structured and organized differs, and where possible individualized to make it simpler for each person to choose what is best for them. Figure 1 compares the pathway a patient would take between a conventional and DCIDA version of a PtDA. For example, the first unique feature of the DCIDA is that in step 1 the value clarification task, which is usually near the end of a conventional PtDA, is moved to the beginning. The objective of the task was to a) provide an opportunity for individuals to reflect on the relative value of attributes and subsequently derive more stable values (see “i) Unstable values” above) and b) to generate the weights for each attribute for use later in the tool. After preliminary testing, we decided to use an interactive form of constant sum exercise (also known as a “budget pie”) which requires users to allocate a certain number of points (often 100) to each attribute in accordance with the relative importance of each. Constant sum exercises have a long history of use and incorporate a number of properties desirable for encouraging compensatory decisions [39], but have been criticized for requiring a higher levels of numeracy [40]. We developed a simpler version that requires users to move multiple sliders, all linked to an interactive pie chart (Figure 2). For individuals with low numeracy, pie charts can be an effective format for communicating the “gist” of health knowledge and treatment choices [41]. As participants navigated through the following steps of the DCIDA version, they could return at any time to the budget pie exercise to change the weights allocated to different attributes of the decision.

After the values clarification task, Step 2 presented participants with detailed information (e.g. the actual risk) for each attribute. Where possible, the information was explained in a simple format, including pictographs where necessary to communicate risks. Importantly, this information was ordered in accordance with the rank of the individual’s weights – obtained from the value clarification exercise. For each attribute, the patient was asked to rate how important the attribute (outcome such as benefit or harm) for each option. This was used to create the score for each attribute. In our preliminary testing we used a simple 5-point scale to derive numerical scores.

In step 3, the summary information for all consequences was displayed. In contrast to a conventional PtDA: 1) the consequences were ordered in accordance with the rank of the individual’s weights – obtained from the value clarification exercise. This aimed to exploit order effects by nudging individuals to focus on the information that would most significantly influence their decision (see “ii) Order effects” above); 2) rows were further sized in proportion to the weights of each consequence, with the most important consequences being presented in wide rows and less important consequences in narrow rows. This served to take attention away from rare events for the majority of people who rated these attributes to have low importance in the value clarification exercise (see “iii) Overweighting of rare events” above); 3) for each consequence, the colour for each option was based on the score, with a lighter shade of grey indicating a more preferred option. Colouring aimed to simplify the information presented (akin to traffic light labelling for the nutritional of food [42]), enabling individuals to process multiple pieces of information and distinguish between harms and benefits (see “iv) Information overload” above); 4) the sum of the weights and scores were used to determine which option would be preferred using MCDA (Figure 3). This indicated the ‘optimal’ choice for a given individual and became the default option for the participant, helping overcome information overload. On the summary page, users were able to select an option other than the default optimal choice; however the presence of a selected default option has proven to help overcome ambiguity [36].

Development

Three PtDAs available to the developers were transformed into DCIDA versions. The goal was to ensure the DCIDA was usable for a broad range of PtDAs, each of which has a different set of individual characteristics. The first PtDA used information on newly developed over the counter medication choices for the treatment of knee osteoarthritis, where it is suspected patients underestimate the risks associated with high doses. The options were no treatment, acetaminophen (lower benefit, but fewer side-effects), and non-steroidal anti-inflammatory drugs (NSAIDs) (greater benefit, but greater side-effects). The second PtDA focussed on treatment options for patients with Obstructive Sleep Apnea (OSA). The primary treatment option is continuous positive airway pressure (CPAP), which is effective but inconvenient to use. Less invasive alternatives such as oral appliances or no treatment are also options. The third PtDA considers chemotherapy options for patients with late stage non-small cell lung cancer (NSCLC). Treatments with higher efficacy are also associated with more frequent and severe side-effects. All three PtDAs were conceived using the Ottawa Framework [43] and were reviewed to International Patient Decision Aid Standards (IPDAS) guidelines [7]. DCIDA versions of each PtDA were made using exactly the same information. An example of the DCIDA version of the PtDAs is available at: http://dcida.cheos.ubc.ca/osa1.

This explorative study investigated the feasibility of using nudges based on MCDA to improve the quality of treatment decisions. Software was developed to enable information from conventional PtDAs to be restructured in how it is presented to the user through dynamic and interactive interfaces. The software was tested for both usability and usefulness in 20 participants. Both aspects of user testing provided some positive results, and provide important information for future development.

There is limited research on the use of behavioural economic approaches to improve patients’ use of PtDA. A study by Ubel et al. found that order effects in decision aids could be unbiased by providing the patient further information (in graphical form) [10]. This is one approach for encouraging System 2 thinking, yet a concern with targeting numerous biases through this general technique is that providing more information can sometimes overwhelm people, causing them to revert back to System 1 thinking [54]. The DCIDA approach has sought to enable users to read less information, but focus on information that will most likely influence their choice.

While there has been substantial attention to ‘nudge’ theory in health [55, 56], to our knowledge this theory has not been tested in PtDAs. Default options have become the predominant ‘nudge’ used in health interventions to date [57], but have typically selected a single default option for all users. For example, organ donation programs may ‘nudge’ patients to enrol by making organ donation the default option. It has been proposed that nudges could be used in PtDAs for conditions where the evidence clearly indicates that one treatment option is superior to the others [58]. This is controversial as most PtDAs are developed for preference sensitive decisions where two or more medically appropriate options exist, and they seek to promote rather than diminish patient autonomy. DCIDA has been designed as a bridge between non prescriptive PtDAs and overtly prescriptive decision analysis tools.

The objective of this study was to examine if there was a difference in response between the two versions of PtDAs. If no difference was observed, we would reject the hypothesis that the DCIDA version had any impact. While we establish some preliminary demonstration of effectiveness, this study alone cannot ascertain whether the impact is real or useful and should be interpreted with caution for three primary reasons. First, participants considered the conventional summary before the DCIDA version, therefore an ordering effect might have been observed whereby they became more informed as they spent more time viewing the information. On average 42 seconds was spent viewing the conventional display versus 31 seconds on the DCIDA display. It is also difficult to disentangle the effects of each aspect of the DCIDA version that differed from the conventional summary, such as the values clarification exercise, the layout, or colour hues. While it would be unfeasible to investigate the impact of each design feature, we have subsequently evaluated the impact of ordering effects in a larger controlled study [59]. A second limitation relates to putting the values clarification exercise at the beginning of the decision aid. We deliberately presented this exercise up front to elicit more stable values from participants. However, recent studies have suggested that it can be problematic to engage in importance weighting too soon in the decision-making process [60, 61]. A related third reason is that we do not know if participants who changed their decision actually made an improved choice. This challenge of measuring the quality of patients choices is a limitation in all research on PtDAs [62].

Further, we acknowledge some limitations to our use of standard measures. With regard to the System Usability Scale (SUS), this measure has been used frequently for usability evaluations of Internet-based interfaces; it is not used typically for evaluations of Internet-based PtDAs. Validation of the scale was based on studies of interfaces up to 18 years old and, thus, there are contextual and design differences between DCIDA and the average tools used to validate the SUS. However, by triangulating the results of our SUS scores with our think aloud and eye-tracking results, we determined that the initial prototype has acceptable usability, though we aim to improve it in future iterations of the prototype and to use and report a newer iteration of the SUS developed by Bangor et al. [63]. Additionally, we chose to use the Uncertainty subscale of the Decisional Conflict Scale and acknowledge that our analysis would have benefitted from also using the Values Clarity subscale [49]. Use of this additional measure would have contributed to our understanding of how DCIDA impacted participants’ ability to arrive at stable values.

While the usability scores and improvements in decisional conflict are encouraging, they suggest there is still opportunity to further improve the tool. At the time data collection, the DCIDA software was in alpha stage, and the results of this research have motivated us to move to a different platform for the beta version. The higher usability results may also be due to the hypothetical nature of the task. Participants did not have the diseases and were aware they were testing a tool. In addition, subjects were majority college-educated who had access to and comfort using computers. These findings may not be generalizable to a more heterogeneous population with lower education or computer proficiency. We are also cognisant that it will be important for the software to be compatible with Internet use on tablets, which will require separate testing.

Given these opportunities for future research, we plan to further explore the influence of the DCIDA in subsequent studies. In these, individuals will be randomized between a conventional PtDA and a DCIDA version and plan to determine if DCIDA’s unique design features lead to improvements in decision quality, including concordance between what matters most to individual patients and their chosen option. We propose to consider carefully how value concordance is measured. There are no standard criteria for studying values concordance and a recent Cochrane review [1] shows that there is substantial heterogeneity among the measures that authors have used to date. We agree with the growing number of researchers calling for further study into the “active ingredients” of values clarification [64] and the creation of standard measures for analyzing values congruence [62, 65]. Such research will assist us in identifying what proportion of people make values congruent decisions when they use DCIDA in comparison to conventional tools. We also plan to ask questions about patients’ attitudes to the role of nudges in making autonomous decisions [38]. Finally, we believe it will be crucial to include patients of varying health literacy, and numeracy to examine the influence of the tool in different patient groups.

The DCIDA has been developed to enhance conventional PtDAs to assist patients in choosing the treatment that is most congruent with their informed values. This paper reports on the theoretical motivation for the DCIDA and then describes an experiment in which the tool is user tested. The results give some empirical support that the DCIDA is understandable to users and that it can help users focus on attributes that are of individual importance to them – to the extent that some participants changed their decisions. A number of valuable insights were learned for improving the next version of the DCIDA. In conclusion, we propose that the DCIDA is a promising approach to improve conventional PtDAs. Further development is required to improve its usability and usefulness; however research on testing preliminary effectiveness on patient decision-making is justified.