- Research article
- Open Access
- Open Peer Review
Variability of the impact of adverse events on physicians’ decision making
BMC Medical Informatics and Decision Making volume 14, Article number: 86 (2014)
Physicians frequently differ in their treatment recommendations. However, few studies have examined the reasons underlying these differences. The objective of this study was to examine whether physicians vary in the importance they attach to specific adverse events for two treatment options found in recent randomized controlled trials to have equivalent efficacy and overall toxicity.
A Max-Diff survey was administered to physicians attending a national scientific conference to quantify the influence of 23 specific adverse events on decision making related to two treatment options for vasculitis. This approach was chosen because it results in greater item discrimination compared to rating scales. We used Hierarchical Bayes modeling to generate the relative importance score for each adverse event and examined the association between physicians’ characteristics and the five most influential factors.
118 physicians completed the survey. The mean age (SD) was 48 years (10); 68% were male and 81% reported spending the majority of time in clinical practice. There was significant variability in the ratings of the relative importance of all adverse events, except those that were mild and easily reversible. We found a positive correlation between increasing physician age with ratings of sepsis (r = 0.29, p = 0.002) and opportunistic infection (r = 0.23, p = 0.016), and an inverse association between age with progressive multifocal leukoencephalopathy (r = - 0.28, p = 0.003). Physician sex, work setting, location, and number of patients with vasculitis seen per year were not associated with the influence of specific adverse events on decision making.
Our findings demonstrate that physicians differ substantially in how they perceive the importance of specific adverse events which may help explain observed unwarranted variability in physicians’ recommendations in clinical practice. Further efforts are needed to ensure that the reasons underlying variability in physicians’ recommendations are transparent.
Physicians often differ in their treatment recommendations. Extant reports of patients receiving significantly different recommendations by physicians practicing in different locations have been published both in the medical literature and the lay press. This variability persists even among experts who are well versed in the same literature as illustrated by the publication of conflicting guidelines by separate organizations . However, few studies have systematically examined the reasons underlying these differences. Variability in recommendations likely occurs more often under conditions of uncertainty where there are few high quality studies available to inform decisions . In these situations, indirect evidence is frequently relied upon to inform decision making in clinical practice. This is especially true for less common conditions, such as vasculitis.
Anti-neutrophil cytoplasmic antibodies (ANCA)-associated vasculitis (AAV) refers to a group of life- and organ-threatening diseases characterized by inflammation of small blood vessels, most commonly involving the skin, lungs, and kidneys. When vasculitis was first described, glucocorticoids were the only available treatment. The introduction of cyclophosphamide in the 1970’s resulted in significant improvements in overall survival of patients with vasculitis. However, cyclophosphamide is associated with significant toxicity including infection, infertility, and cancer, and physicians have long sought safer alternatives.
In 2010, two randomized controlled trials, Rituximab in ANCA-Associated Vasculitis (RAVE) and Rituximab versus Cyclophosphamide in ANCA-Associated Vasculitis (RITUXVAS) [3, 4], demonstrated that rituximab is as effective as cyclophosphamide at inducing remission in patients with moderate to severe disease. Moreover, there were no differences in the rate of adverse events between the two groups. These results provided physicians with an alternative treatment option for AAV for the first time in approximately 40 years.
Given the equivalent rates of remission and toxicity induced by both treatment regimens, the decision of which treatment to initiate (assuming the patient has no contraindications to either option) is likely strongly influenced by judgments related to the impact of specific adverse events. The RAVE trial was designed as a non-inferiority trial because physicians believed rituximab to be a potentially safer alternative. This belief was based on a large body of indirect evidence generated from randomized trials and observational studies of patients with different diseases or using different treatment regimens demonstrating a lower risk of infection associated with rituximab compared to cyclophosphamide [5–26]. Moreover, unlike cyclophosphamide, rituximab is not associated with a risk of infertility or cancer. However, the RAVE and RITUXVAS trials, which are the only head-to-head comparisons of cyclophosphamide and rituximab in any disease, demonstrated no significant differences in toxicity between the two treatment arms, including serious infection or rates of hospitalization. Several of the major toxicities of concern for cyclophosphamide, including ovarian failure and bladder cancer, usually manifest after either higher cumulative doses of cyclophosphamide or longer follow-up times than present in the trials under discussion. Despite these limitations, the RAVE and RITUXVAS trials led not only to the approval of rituximab for use in AAV by the US Food and Drug Administration and regulatory agencies in other countries, but also rapid widespread adoption of the drug’s use as an remission induction agent in AAV.
The objective of this study was to quantify the influence of specific adverse events on physicians’ treatment decisions for patients with newly-diagnosed AAV. Relative importance of specific attributes are most commonly measured using rating or ranking tasks. Visual analogue, Likert, or numeric rating scales are relatively easy to perform and permit inclusion of a large number of items. However, such rating scales are influenced by response biases, including social desirability bias and extreme response bias . The latter is particularly problematic in healthcare research where subjects tend to rate all items as “Extremely Important”. Ranking tasks provide better discrimination, but significantly limit the number of items that can be included in a survey. In this study we used Sawtooth Software’s MaxDiff program as an alternative to a rating or ranking task. This approach asks respondents to choose the best (or the most important) item and/or the worst (or least important) item from a series of sets containing different combinations of items from a master list (see Figure 1 for an example). This method has several important advantages over rating scales including the ability to handle a large number of attributes, greater item discrimination, and avoidance of scale-related response bias .
We administered a MaxDiff survey to a convenience sample of physicians attending the 2010 American College of Rheumatology National Scientific Meeting . The survey was designed using Sawtooth Software (Sawtooth Software SSI Web version 6.4) and included 23 adverse events associated with either RTX and/or cyclophosphamide. Participants were given the following instructions:
“On the following pages you will be asked to consider (given your knowledge about the severity and probability of each of the adverse events) how much each adverse event influences your decision about which medication to prescribe for a 50 year old post-menopausal woman with newly diagnosed severe ANCA-associated vasculitis. Assume the patient is hospitalized with pulmonary and renal involvement”. We described the patient as being a postmenopausal woman so as to eliminate fertility concerns, which are extremely variable and difficult to control for in a standardized scenario. The survey asked respondents to choose the most important item from 20 sets containing different combinations of five items from the master list of 23 adverse events (see example in Figure 1).
The survey design ensured that each item 1) was shown approximately an equal number of times, 2) was paired with each other item an equal number of times, and 3) appeared in each position an equal number of times. We also collected data on the following physician characteristics: age, sex, number of years of experience, work setting, and the number of patients with vasculitis seen per year.
We used Hierarchical Bayes modeling to generate the mean relative importance score for each adverse event. The relative importance scores sum to 100, and reflect the impact of each characteristic on physicians’ decision making. For each respondent, utilities (zero-centered values) were calculated for each level of each attribute using Hierarchical Bayes (HB) modeling (Sawtooth CBC/HB system for hierarchical Bayes estimation version 4.0). HB modeling has the advantage that it can better incorporate heterogeneity between respondents’ choices. In HB modeling, the sample averages (prior information) are used to update the individual utilities in a number of iterations until the sample averages stop changing between iterations. After this convergence, the cycle is run several thousand more times and the estimates of each iteration are saved and averaged. Raw scores were rescaled to a 0 to 100 on a ratio scale . The associations between physician characteristics and ratings were made using Pearson’s correlation coefficient and chi-square tests for continuous and categorical variables, respectively. This study qualified for exemption as determined by the Yale Human Investigation Committee.
One hundred and eighteen physicians completed the survey: 75 from North America and 22 from Europe. The mean age (SD) was 48 (10) years; 68% were male; 81% reported spending the majority of time in clinical practice; 39% work in an academic setting; 7% see fewer than 1 patient with AAV per year, 46% between 1 and 5 patients with AAV per year; 22% between 6 and 10 patients with AAV per year, and 25% see more than 10 patients with AAV per year.
Physicians’ relative importance scores for each adverse event are provided in Table 1. Overall, physicians’ decisions were most strongly influenced by the risk of infection and cancer. As noted by the ranges in Table 1 and the histograms illustrating the distributions of the mean relative importance of each adverse event (Figure 2), there was significant variability in the ratings of all adverse events, except those that were felt to be the least important. Figure 3 provides a full sized example of a mild and serious adverse event.
We examined the association between physicians’ characteristics and the five of the most influential factors: sepsis, lymphoma, opportunistic infection, serious infusion reaction, and progressive multifocal leukoencephalopathy. There was a positive correlation with increasing physician age with ratings of sepsis (r = 0.29, p = 0.002) and opportunistic infection (r = 0.23, p = 0.016). In contrast, older rheumatologists’ decisions were less influenced by progressive multifocal leukoencephalopathy (r = - 0.28, p = 0.003). The association between age and lymphoma was of borderline significance (r = - 0.19, p = 0.05) and age was not related to perceived importance of serious infusion reaction (r = 0.02, p = 0.9).Ratings of all adverse events by age group (up to 39, 40-55, 56+ years of age) are illustrated in Figure 4. Compared to younger physicians, older physicians were more strongly influenced by the risk of serious infection-related adverse events (except for progressive multifocal leukoencephalopathy) and less strongly influenced by malignancy. Physician sex, work setting, location, and number of patients with vasculitis seen per year were not associated with the impact of sepsis, lymphoma, opportunistic infection, serious infusion reaction, and progressive multifocal leukoencephalopathy on decision making.
The findings from this study demonstrate that physicians differ significantly in the impact that specific adverse events have on their decision making. With the exception of reversible and easily treatable adverse events, histograms showed wide distributions for all adverse events related to the two treatment options examined. Ratings were not related to clinical experience. However, we did find differences by age group, with physicians greater than 56 years old being more influenced by the risks of all infection-related adverse events, except for progressive multifocal leukoencephalopathy, than their younger counterparts. While the data do not explain the reasons underlying this pattern, it is possible that the differential impact between unspecified serious infections and progressive multifocal leukoencephalopathy is due to the relative recent attention focusing on the latter [31–33].
The RAVE  and RITUXVAS  head-to-head randomized controlled trials demonstrated no significant differences in the risk of infection between the two regimens. Despite these results, physicians ranked the risk of infection as the most important factor influencing their treatment decision. This finding was especially pronounced among older physicians. Physicians may undervalue risk estimates generated from randomized trials, because selective eligibility criteria, strict protocols, and short-term follow-up minimize adverse events. In addition, many of the risks associated with immunosuppressive medications are rare and would not be expected to occur in trials with relatively small numbers of patients. Small numbers and short duration of follow-up limit the usefulness of the toxicity data generated from head-to-head randomized controlled trials, however, the use of indirect data also has significant limitations. Most notably, the indirect data available for cyclophosphamide were generated from trials using higher doses of the drug and for much longer periods of time compared to the regimens currently used. This is particularly important because the most worrisome adverse events associated with cyclophosphamide (i.e. infection, infertility, and malignancy) are dose dependent.
There are several important limitations of this study. The use of standardized scenarios, enabled us to systematically vary patient characteristics, but this approach cannot replicate the complexity of decision making in clinical practice. In addition, although we surveyed a large number of physicians, most of whom spend the majority of their time in clinical practice, the study population represents a convenience sample of consecutively approached volunteers, which limits generalizability. Future research should examine whether differences in decision making predict differences in treatment choices made in clinical practice.
Ideally, despite physicians differing in their judgments, patients should all receive the same information describing potential treatment options and the reasons underlying variability in physicians’ recommendation should be transparent. While we did not measure actual treatment decisions, these findings suggest that physicians differ substantially in how they perceive the importance of specific adverse events which may help explain the frequently observed unwarranted variability in physicians’ recommendations.
Kachalia A, Mello MM: Breast cancer screening: conflicting guidelines and medicolegal risk. JAMA. 2013, 309: 2555-2556. 10.1001/jama.2013.7100.
GRADE Working Group: Grading quality of evidence and strength of recommendations. BMJ. 2004, 328: 1490-1497.
Stone JH, Merkel PA, Spiera R, Merkel PA, Spiera R, Seo P, Langford CA, Hoffman GS, Kallenberg CGM, St Clair EW, Turkiewicz A, Tchao NK, Webber L, Ding L, Sejismundo LP, Mieras K, Weitzenkamp D, Ikle D, Seyfert-Margolis V, Mueller M, Brunetta P, Allen NB, Fervenza FC, Geetha D, Keogh KA, Kissin EY, Monach PA, Peikert T, Stegeman C, Ytterberg SR, Specks U, for the RAVE-ITN Research Group: Rituximab versus cyclophosphamide for ANCA-associated vasculitis. N Engl J Med. 2010, 363: 221-232. 10.1056/NEJMoa0909905.
Jones RB, Tervaert JWC, Hauser T, Luqmani R, Morgan MD, Peh CA, Savage CO, Segelmark M, Tesar V, van Paassen P, Walsh D, Walsh M, Westman K, Jayne DRW, for the European Vasculitis Study Group: Rituximab versus cyclophosphamide in ANCA-associated renal vasculitis. N Engl J Med. 2010, 363: 211-220. 10.1056/NEJMoa0909169.
Anolik JH, Barnard J, Cappione A, Pugh-Bernard AE, Felgar RE, Looney RJ, Sanz I: Rituximab improves peripheral B cell abnormalities in human systemic lupus erythematosus. Arthritis Rheum. 2004, 50: 3580-3590. 10.1002/art.20592.
Balow JE: Choosing treatment for proliferative lupus nephritis. Arthritis Rheum. 2002, 46: 1981-1983. 10.1002/art.10466.
Boren EJ, Cheema GS, Naguwa SM, Ansari AA, Gershwin ME: The emergence of progressive multifocal leukoencephalopathy (PML) in rheumatic diseases. J Autoimmun. 2008, 30: 90-98. 10.1016/j.jaut.2007.11.013.
Boumpas DT, Austin HA, Balow JE, Vaughan EM, Yarboro CH, Klippel JH, Steinberg AD: Controlled trial of pulse methylprednisolone versus two regimens of pulse cyclophosphamide in severe lupus nephritis. Lancet. 1992, 340: 741-745. 10.1016/0140-6736(92)92292-N.
Cohen P, Pagnoux C, Mahr A, Arène JP, Mouthon L, Le Guern V, André MH, Gayraud M, Jayne D, Blöckmans D, Cordier JF, Guillevin L, French Vasculitis Study Group: Churg-Strauss syndrome with poor-prognosis factors: a prospective multicenter trial comparing glucocorticoids and six or twelve cyclophosphamide pulses in forty-eight patients. Arthritis Rheum. 2007, 57: 686-693. 10.1002/art.22679.
De Groot K, Harper L, Jayne DR, Flores Suarez LF, Gregorini G, Gross WL, Luqmani R, Pusey CD, Rasmussen N, Sinico RA, Tesar V, Vanhille P, Westman K, Savage CO, EUVAS (European Vasculitis Study Group): Pulse versus daily oral cyclophosphamide for induction of remission in antineutrophil cytoplasmic antibody-associated vasculitis. Ann Intern Med. 2009, 150: 670-680. 10.7326/0003-4819-150-10-200905190-00004.
De Groot K, Rasmussen N, Bacon PA, Tervaert JW, Feighery C, Gregorini G, Gross WL, Luqmani R, Jayne DR: Randomized trial of cyclophosphamide versus methotrexate for induction of remission in early systemic antineutrophil cytoplasmic antibody-associated vasculitis. Arthritis Rheum. 2005, 52: 2461-2469. 10.1002/art.21142.
Fauci AS, Haynes BF, Katz P, Wolff SM: Wegener’s granulomatosis: prospective clinical and therapeutic experience with 85 patients for 21 years. Ann Internal Med. 1983, 98: 76-85. 10.7326/0003-4819-98-1-76.
Faurschou M, Sorensen IJ, Mellemkjaer L, Loft AG, Thomsen BS, Tvede N, Baslund B: Malignancies in Wegener’s granulomatosis: incidence and relation to cyclophosphamide therapy in a cohort of 293 patients. J Rheumatol. 2008, 35: 100-105.
Ginzler EM, Dooley MA, Aranow C, Ginzler EM, Dooley MA, Aranow C, Kim MY, Buyon J, Merrill JT, Petri M, Gilkeson GS, Wallace DJ, Weisman MH, Appel GB: Mycophenolate mofetil or intravenous cyclophosphamide for lupus nephritis. N Engl J Med. 2005, 53: 2219-2228.
Gunnarsson I, Sundelin B, Jonsdottir T, Jacobson SH, Henriksson EW, van Vollenhoven RF: Histopathologic and clinical outcome of rituximab treatment in patients with cyclophosphamide-resistant proliferative lupus nephritis. Arthritis Rheum. 2007, 56: 1263-1272. 10.1002/art.22505.
Houssiau FA, Vasconcelos C, D’Cruz D, Sebastiani GD, Garrido Ed Ede R, Danieli MG, Abramovicz D, Blockmans D, Mathieu A, Direskeneli H, Galeazzi M, Gül A, Levy Y, Petera P, Popovic R, Petrovic R, Sinico RA, Cattaneo R, Font J, Depresseux G, Cosyns JP, Cervera R: Immunosuppressive therapy in lupus nephritis: the Euro-Lupus Nephritis Trial, a randomized trial of low-dose versus high-dose intravenous cyclophosphamide. Arthritis Rheum. 2002, 46: 2121-2131. 10.1002/art.10461.
Jayne D: Review article: progress of treatment in ANCA-associated vasculitis. Nephrology. 2009, 14: 42-48. 10.1111/j.1440-1797.2009.01101.x.
Jayne DR, Rasmussen N: Treatment of antineutrophil cytoplasm autoantibody-associated systemic vasculitis: initiatives of the European community systemic vasculitis clinical trials study group. Mayo Clin Proc. 1997, 72: 737-747. 10.1016/S0025-6196(11)63594-5.
Merrill JT, Neuwelt CM, Wallace DJ, Shanahan JC, Latinis KM, Oates JC, Utset TO, Gordon C, Isenberg DA, Hsieh HJ, Zhang D, Brunetta PG: Efficacy and safety of rituximab in moderately-to-severely active systemic lupus erythematosus: the randomized, double-blind, phase II/III systemic lupus erythematosus evaluation of rituximab trial. Arthritis Rheum. 2010, 62: 222-233. 10.1002/art.27233.
Mok CC, Ho CTK, Chan KW, Lau CS, Wong RWS: Outcome and prognostic indicators of diffuse proliferative lupus glomerulonephritis treated with sequential oral cyclophosphamide and azathioprine. Arthritis Rheum. 2002, 46: 1003-1013. 10.1002/art.10138.
Mukhtyar C, Flossmann O, Hellmich B, Bacon P, Cid M, Cohen-Tervaert JW, Gross WL, Guillevin L, Jayne D, Mahr A, Merkel PA, Raspe H, Scott D, Witter J, Yazici H, Luqmani RA, European Vasculitis Study Group (EUVAS): Outcomes from studies of antineutrophil cytoplasm antibody associated vasculitis: a systematic review by the European league against rheumatism systemic vasculitis task force. Ann Rheum Dis. 2008, 67: 1004-1010. 10.1136/ard.2007.071936.
Radis CD, Kahl LE, Baker GL, Wasko MC, Cash JM, Gallatin A, Stolzer BL, Agarwal AK, Medsger TA, Kwoh CK: Effects of cyclophosphamide on the development of malignancy and on long-term survival of patients with rheumatoid arthritis a 20-year followup study. Arthritis Rheum. 1995, 38: 1120-1127. 10.1002/art.1780380815.
Somers EC, Marder W, Christman GM, Ognenovski V, McCune WJ: Use of a gonadotropin-releasing hormone analog for protection against premature ovarian failure during cyclophosphamide therapy in women with severe lupus. Arthritis Rheum. 2005, 52: 2761-2767. 10.1002/art.21263.
Stasi R, Stipa E, Del Poeta G, Amadori S, Newland AC, Provan D: Long-term observation of patients with anti-neutrophil cytoplasmic antibody-associated vasculitis treated with rituximab. Rheumatology. 2006, 45: 1432-1436. 10.1093/rheumatology/kel098.
Talar-Williams C, Hijazi YM, Walther MM, Linehan WM, Hallahan CW, Lubensky I, Kerr GS, Hoffman GS, Fauci AS, Sneller MC: Cyclophosphamide-induced cystitis and bladder cancer in patients with Wegener Granulomatosis. Ann Intern Med. 1996, 124: 477-484. 10.7326/0003-4819-124-5-199603010-00003.
Tamimoto Y, Horiuchi T, Tsukamoto H, Otsuka J, Mitoma H, Kimoto Y, Nakashima H, Muta K, Abe Y, Kiyohara C, Ueda A, Nagasawa K, Yoshizawa S, Shimoda T, Harada M: A dose-escalation study of rituximab for treatment of systemic lupus erythematosus and Evans’ syndrome: immunological analysis of B cells, T cells and cytokines. Rheumatology. 2008, 47: 821-827. 10.1093/rheumatology/ken071.
van Herk H, Poortinga YH, Verhallen TMM: Response styles in rating scales. J Cross Cult Psychol. 2004, 35: 346-360. 10.1177/0022022104264126.
Flynn T, Louviere J, Peters T, Coast J: Best-worst scaling: what it can do for health care research and how to do it. J Health Econ. 2007, 26: 171-189. 10.1016/j.jhealeco.2006.04.002.
The MaxDiff/Web v6.0 Technical Paper:http://www.sawtoothsoftware.com/download/techpap/maxdifftech.pdf,
Orme B: Hierarchical Bayes regression analysis: technical paper. 2003, Sequim: Sawtooth Software, Available from: http://www.sawtoothsoftware.com
Post Marketing Reviews: Rituximab (marketed as Rituxan): Progressive Multifocal Leukoencephalopathy (PML). 2007, Available from: http://www.fda.gov/Drugs/DrugSafety/DrugSafetyNewsletter/ucm115974.htm
Bharat A, Xie F, Baddley JW, Beukelman T, Chen L, Calabrese L, Delzell E, Grijalva CG, Patkar NM, Saag K, Winthrop KL, Curtis JR: Incidence and risk factors for progressive multifocal leukoencephalopathy among patients with selected rheumatic diseases. Arthritis Care Res. 2012, 64: 612-615. 10.1002/acr.21564.
Molloy ES: PML and rheumatology: the contribution of disease and drugs. Clevel Clin J Med. 2011, 78 (Suppl 2): S28-S32.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/14/86/prepub
Research reported in this publication was also supported by the National Institute of Arthritis and Musculoskeletal and Skin Diseases, part of the National Institutes of Health, under Award Number AR060231-01. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Dr. Fraenkel is supported by NIAMS K24 AR060231-01.
Dr. Merkel has conducted research partially funded by Genentech, Inc.
Dr. Merkel has conducted research partially funded by Genentech, Inc.
RC participated in designing the survey, collecting and analyzing the data and writing the MS. PM participated in designing the survey, analyzing the data and writing the MS. EW participated in designing the survey, collecting the data and writing the MS. LF conceived of the study and participated in designing the survey, collecting and analyzing the data and writing the MS. All authors read and approved the final manuscript.
About this article
Cite this article
Cozmuta, R., Merkel, P.A., Wahl, E. et al. Variability of the impact of adverse events on physicians’ decision making. BMC Med Inform Decis Mak 14, 86 (2014) doi:10.1186/1472-6947-14-86
- Drug toxicity
- Best-worst scaling