Accuracy validation of adjuvant! online in Taiwanese breast cancer patients - a 10-year analysis
BMC Medical Informatics and Decision Making volume 12, Article number: 108 (2012)
Adjuvant! Online (http://www.adjuvantonline.com) is an Internet-based software program that allows clinicians to make predictions about the benefits of adjuvant therapy and 10-year survival probability for early-stage breast cancer patients. This model has been validated in Western countries such as the United States, United Kingdom, Canada, Germany, and Holland. The aim of our study was to investigate the performance and accuracy of Adjuvant! Online in a cohort of Taiwanese breast cancer patients.
Data on the prognostic factors and clinical outcomes of 559 breast cancer patients diagnosed at the National Cheng Kung University Hospital in Tainan between 1992 and 2001 were enrolled in the study. Comprehensive demographic, clinical outcome data, and adjuvant treatment data were entered into the Adjuvant! Online program. The outcome prediction at 10 years was compared with the observed and predicted outcomes using Adjuvant! Online.
Comparison between low- and high-risk breast cancer patient subgroups showed significant differences in tumor grading, tumor size, and lymph node status (p < 0.0001). The mean 10-year predicted death probability in 559 patients was 19.44%, and the observed death probability was 15.56%. Comparison with the Adjuvant! Online-predicted breast cancer-specific survival (BCSS) showed significant differences in the whole cohort (p < 0.001). In the low-risk subgroup, the predicted and observed outcomes did not differ significantly (3.69% and 3.85%, respectively). In high-risk patients, Adjuvant! Online overestimated breast cancer-specific survival (p = 0.016); the predicted and observed outcomes were 21.99% and 17.46%, respectively.
Adjuvant! Online accurately predicted 10-year outcomes and assisted in decision making about adjuvant treatment in low-risk breast cancer patients in our study, although the results were less accurate in the high-risk subgroup. Development of a prognostic program based on a national database should be considered, especially for high-risk breast cancer patients in Taiwan.
Breast cancer treatments are based mainly on international guideline recommendations (e.g., the National Comprehensive cancer Network (NCCN) guidelines or St. Gallen consensus)[1–3]. Improvements in the efficacy of adjuvant chemotherapy have also improved the prognosis for early breast cancer. Adjuvant drug therapy can extend survival in breast cancer patients, although the balance between risks and benefits must be considered. The decision to prescribe adjuvant systemic therapy for early breast cancer patients is complex and requires consideration of conditions such as the patient’s and tumor characteristics, treatment effectiveness, and clinician and patient preferences. However, inaccurate outcome predictors can lead to the over- or undertreatment of patients. Recently, several statistical models or programs have been developed to predict outcomes for early breast cancer patients[5–10]. These programs can help to inform patients and allow them to choose a more individualized treatment strategy based on a number of prognostic factors.
Adjuvant! Online is a Web-based, open-access computer program that predicts patient outcome at 10 years in breast cancer patients. The development of the program is based on information from the Surveillance, Epidemiology, and End Results (SEER) registry. The SEER registry includes data for about 10% of breast cancer patients in the USA. The program analyzed the outcome of 10-year survival probabilities, risk of relapse, and survival using patient information and tumor characteristics such as age, tumor size, tumor grade, estrogen receptor (ER) status, and lymph node status.
Adjuvant! Online was validated successfully by Olivotto and colleagues in a population-based series of 4083 early-stage breast cancer patients in Canada. They concluded that the Adjuvant! Online program accurately predicted the observed breast cancer-related death outcome within 2% in the whole study population. However, Asian populations might differ from those in the USA or Canada, and the accuracy of Adjuvant! Online in predicting outcomes in the Asian, particularly the Taiwanese population is unknown.
The aims of our study were to validate the accuracy of Adjuvant! Online, to predict outcomes in different patient groups, and to distinguish individuals who will need different prediction methods in a cohort of Taiwanese patients with breast cancer treated at a multidisciplinary tumor board in a university hospital.
Between January 1, 1992, and December 31, 2001, 652 breast cancer patients were diagnosed and treated at National Cheng Kung University Hospital (NCKUH) in Tainan, Taiwan. The final cohort involved in the subset of 559 patients fulfilled the following criterion: follow-up period > 10 years or death due to breast cancer within 10 years from the diagnosed date. Ninety-three patients were excluded because of loss to follow-up or death from other causes. In all analyzed cases, 472 patients whose follow-up period was > 10 years were classified as “survived” (84.44%), and the other 87 patients who died of breast cancer within 10 years were classified as “died” (15.56%). Those who died of breast cancer beyond the 10-year follow-up were also classified as survived. Ethical approval was provided by the Human Experiment and Ethics Committee of the National Cheng Kung University Hospital (A-ER-101-013).
The demographic data included age at the time of presentation for cancer, and the pathological findings included tumor size, axillary lymph node status, tumor histological grade, and ER status. All pathological specimens were reviewed by breast pathologists at NCKUH. Tumor size was measured using pathological reports from NCKUH. The Scarff-Bloom-Richardson (SBR) system was used for tumor grading and was based on the following morphological features: nuclear pleomorphism of tumor cells, degree of tumor tubule formation, and tumor mitotic activity. To determine the ER status, immunohistochemistry was performed on formalin-fixed, paraffin-embedded breast cancer tissue samples from the patients. Positive ER status was defined as nuclear staining > 1%. Immunohistochemistry was performed using anti-ER (clone 6 F11, Ventana Medical Systems, Strasbourg, France). Postoperative adjuvant chemotherapy was performed according to the NCCN and St. Gallen guidelines.
Postoperative adjuvant chemotherapy was performed according to NCCN and St. Gallen guidelines. Between the period 1992 to 1995, CMF regimen was used as standard chemotherapy (cyclophosphamide 500 mg/m2, methotrexate 50 mg/m2 and 5-FU 500 mg/m2). Since 1996, FEC regimen (5-FU 500 mg/m2, epirubicin 100 mg/m2 and cyclophosphamide 500 mg/m2) was given in the majority of patients. Since 1998, for high risk patients, TEC regimen (Taxotere 75 mg/m2, epirubicin 75 mg/m2 and cyclophosphamide 500 mg/m2) was performed.
χ 2 and two-sample independent t tests were used to compare variables between the two subgroups. Data for demographic and tumor characteristics for each patient, such as age, ER status, grade, tumor size, lymph node status, and treatment modalities (chemotherapy or/and hormone therapy) were entered into the Adjuvant! Online program (version 8.0), which produced a 10-year predicted probability for death due to breast cancer. For the comorbidity item “average for age” was imputed for all patients. The Hosmer–Lemeshow test was used to assess whether the predicted probabilities matched the observed probabilities in subgroups of the patient population. The difference in the number of categories reflects the different subgroup sizes. P values < 0.05 were considered significant. The observed probability was regressed on the predicted probability, and R 2 was calculated to measure the goodness of fit of the linear regression line. All statistical analyses were performed with SPSS 17.0 software.
The patients’ characteristics and the distribution of the whole patient cohort is described in Table1. The mean 10-year predicted death probability in 559 patients was 19.44%, and the observed death probability was 15.56% (p < 0.001). The predicted event rate evaluated using the Adjuvant! Online formula significantly overestimated the real event rate in the patients in Taiwan. We used the linear regression model to adjust for the difference between the predicted and observed probability by calculating the predicted probability using the Adjuvant! Online formula as the first step, inserting the predicted value into the regression formula, and then estimating the observed probability. Compared with the predicted probability, the estimated observed probability was closer to the real probability. The calculated R 2 was 0.843, showing excellent fit of the regression model.
However, in our study, the model was overoptimistic in the high risk patients. We categorized the patients into low- and high-risk subgroups. Patients with all of following prognostic factors were classified as low-risk: aged ≥ 35 years and had a tumor size ≤ 2 cm, negative lymph node status, positive ER/PR receptor status, and tumor grade I. The patients with any one of the following prognostic factors will be classified as high-risk: aged < 35 years or had a tumor size > 2 cm or positive lymph node status or tumor grade II or III or negative ER/PR receptor status. Comparison between the low- and high-risk groups showed significant differences in tumor grade, tumor size, and lymph node distribution (p < 0.0001), which justified the division of the patients into the two subgroups. The mean age did not differ significantly between the two groups (p = 0.056). (Table2).
In 78 low-risk patients, the mean predicted death probability was 3.69%, which did not differ significantly from the 3.85% observed death probability (ratio of predicted/observed risk = 0.96; p = 0.099). The Adjuvant! Online probability accurately predicted the outcomes of patients in the low-risk group. However, the difference between predicted and observed death probabilities was significant in the 481 high-risk patients. A linear regression model was also constructed to adjust for the difference between the predicted and observed probabilities in the high-risk subgroup. The calculated R 2 was 0.86, showing excellent fit of the regression model (Figure1). The Adjuvant! Online model showed an overestimation of breast cancer-specific survival in Taiwanese high risk breast cancer patients (ratio of predicted/observed risk = 1.26; p = 0.016) (Figure2).
The rate of loss to follow-up was 14.26% because some patients were lost of follow-up after 5 years hormonal treatment. These patients were excluded from our data analysis, and our more conservative data analysis method might have overestimated the death probability.
Adjuvant treatment for postoperative early breast cancer patients remains a great challenge for physicians and patients who must consider both the risks and benefits of treatment, possible comorbidities, and especially the desire to maintain quality of life. Several tools to support decisions have been developed[5, 7, 8, 14, 15]. One such tool, Adjuvant! Online, is a computerized, Web-based program that predicts recurrence and mortality risk and the benefit of adjuvant treatment in early breast cancer patients. The program is based on the database from the US SEER tumor registry database. The SEER tumor registry collected information from about 10% of all breast cancer cases in the USA. The database used for Adjuvant! Online included information such as the patients’ demographics and tumor characteristics (tumor size, the number of positive nodes, tumor grade), and survival in postoperative breast cancer patients aged 20 to 79 years between 1988 and 1992. After entering these data, the program calculated the annual breast cancer mortality rates and produced data for comparison with the database from the SEER tumor registry. These data were used to predict the 10-year survival. Adjuvant! Online can be used to provide recommendations for adjuvant systemic therapy after considering the estimated 10-year overall survival (OS), breast cancer-specific survival (BCSS), and event-free survival (EFS) However, the lack of prognostic power of Adjuvant! Online in different populations raises questions about the combination of prognostic factors and accuracy of patients’ characteristics. Our goal was to determine the accuracy of the program applied to an Asian population and, if the program is accurate, which subgroup should be included.
The tool has been validated and used by oncologists in different countries including Canada, Germany, Holland, and the United Kingdom (UK)[12, 16–19]. In an analysis of 4083 early breast cancer patients in Canada, Olivotto et al. showed that the overall predicted and observed 10-year outcomes were within 2% for OS, BCSS, and EFS. The Adjuvant! Online program was also validated in small cohorts of patients in Germany. The increased use of the program by physicians and the positive results in Western countries prompted us to analyze the accuracy of the program in an Asian population.
In the present study, the difference between the predicted and observed outcomes in the low-risk cohort was about 1%. We conclude that Adjuvant! Online is an accurate tool for predicting the outcome in low-risk breast cancer patients in the Taiwanese population. By contrast, we observed a large discrepancy between our prediction and that of Adjuvant! Online in the high-risk population. That is, Adjuvant! Online underestimated the mortality risk in the high-risk subgroup of Taiwanese breast cancer patients. Considering this discrepancy, we suggest that a correction factor of 1.259 might be justified for high-risk patients. Variations in the program validation may differ between countries and ethnic factors are known to be determining factors that can influence the decision about and outcomes of adjuvant treatment. The program should be validated in different countries and ethnic groups before wider application of these data.
Campbell et al. showed that the recommendations for adjuvant treatment made by a UK-based multidisciplinary team using Adjuvant! Online would improve decision making in only a minority (2.5%) of patients. Our results are similar to those of Campbell et al. In a cohort of 1065 British early breast cancer patients, Campbell et al. found that Adjuvant! Online significantly overestimated OS (5.54%, p < 0.001), BCSS (4.53%, p < 0.001), and EFS (3.51%, p = 0.001). They concluded that overestimation of BCSS implies that Adjuvant! Online underestimates breast cancer mortality for women in the UK. One possible reason for this difference may be that the breast cancer mortality rate is lower in the USA than in the UK.
A Dutch validation study of 5380 breast cancer patients showed no significant differences between the 10-year observed OS and BCSS compared with the Adjuvant! Online-predicted OS and BCSS (p = 0.87 and p = 0.18, respectively). However, in the subgroup with patients aged < 40 years, Adjuvant! Online overestimated the predicted OS and BCSS (4.2%, p = 0.04 and 4.7%, p = 0.01, respectively). In the patient subgroup older than 69 years, Adjuvant! Online overestimated the predicted OS (3.4%, p = 0.05). Finally, the program underestimated predicted BCSS in the patient subgroup with 1–3 lymph node metastases (3.1%, p = 0.002). Our data are consistent with these earlier findings in that we found that Adjuvant! Online predicted BCSS accurately in some patient subgroup.
Compared with other studies that validated the accuracy and performance of Adjuvant! Online, our study might be considered to have had a small sample size. However, one advantage in our study is the low proportion of missing data. Data for tumor size, lymph node status, ER status, and tumor grade were collected routinely and rechecked by a clinician and data manager, and our missing data rate was around 25%. Our missing data rate is similar to the publication from Bhoo-Pathy et al. (25%). In the study by Mook et al., 3104 of 5380 patients (58%) had data missing for ER status or tumor grade. However, they concluded that including the patients with incomplete data did not influence the prediction of outcomes. The program can also predict disease outcome without information about ER status. By contrast, Adjuvant! Online tend to underestimate the BCSS in ER-negative breast cancer patients. Mook et al. concluded that this result was observed in patients undergoing hormonal treatment. Another limitation of our study is the adjuvant treatment modalities. The period of data recruitment for this study was from 1992 to 2001. According to the NCCN guidelines and policy of Taiwan national health insurance, standard adjuvant chemotherapy comprised CMF (cyclophosphamide, methotrexate, and fluorouracil) or an anthracycline-based regimen, and standard adjuvant endocrine therapy comprised tamoxifen for HR-positive patients. However, the treatment varies between countries, especially in relation to different national policies for the use of treatments, such as taxane-containing regimen chemotherapy or aromatase inhibitors.
This validation study of Adjuvant! Online applied to a university hospital-based cohort of Taiwanese breast cancer patients showed that the calculated predictions by Adjuvant! Online matched the observed outcomes and that the predictions are suitable when applied to low-risk, early breast cancer patients. We agree that Adjuvant! Online is a useful tool for doctors and patients, but it should be used only by those aware of its limitations, as advised in the program’s instructions and as reported elsewhere.
Goldhirsch A, Ingle JN, Gelber RD, Coates AS, Thurlimann B, Senn HJ: Thresholds for therapies: highlights of the St Gallen International Expert Consensus on the primary therapy of early breast cancer 2009. Ann Oncol. 2009, 20 (8): 1319-1329. 10.1093/annonc/mdp322.
Goldhirsch A, Wood WC, Coates AS, Gelber RD, Thurlimann B, Senn HJ: Strategies for subtypes--dealing with the diversity of breast cancer: highlights of the St. Gallen International Expert Consensus on the Primary Therapy of Early Breast Cancer 2011. Ann Oncol. 2011, 22 (8): 1736-1747. 10.1093/annonc/mdr304.
National Comprehensive Cancer Network Web Site.http://www.nccn.org/index.asp,
Early Breast Cancer Trialists' Collaborative Group (EBCTCG): Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: an overview of the randomised trials. Lancet. 2005, 365 (9472): 1687-1717.
Galea MH, Blamey RW, Elston CE, Ellis IO: The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Res Treat. 1992, 22 (3): 207-219. 10.1007/BF01840834.
Lyman GH, Cosler LE, Kuderer NM, Hornberger J: Impact of a 21-gene RT-PCR assay on treatment decisions in early-stage breast cancer: an economic analysis based on prognostic and predictive validation studies. Cancer. 2007, 109 (6): 1011-1018. 10.1002/cncr.22506.
Ravdin PM, Siminoff LA, Davis GJ, Mercer MB, Hewlett J, Gerson N, Parker HL: Computer program to assist in making decisions about adjuvant therapy for women with early breast cancer. J Clin Oncol. 2001, 19 (4): 980-991.
Siminoff LA, Gordon NH, Silverman P, Budd T, Ravdin PM: A decision aid to assist in adjuvant therapy choices for breast cancer. Psychooncology. 2006, 15 (11): 1001-1013. 10.1002/pon.1040.
Chang TW, Kuo YL: A model building exercise of mortality risk for Taiwanese women with breast cancer. BMC Med Inform Decis Mak. 2010, 10: 43-10.1186/1472-6947-10-43.
Adjuvant! Online Web Site.http://www.adjuvantonline.com,
National Cancer Institute: Surveillance, Epidemiology, and End Results (SEER) Web Site .http://seer.cancer.gov/,
Olivotto IA, Bajdik CD, Ravdin PM, Speers CH, Coldman AJ, Norris BD, Davis GJ, Chia SK, Gelmon KA: Population-based validation of the prognostic model ADJUVANT! for early breast cancer. J Clin Oncol. 2005, 23 (12): 2716-2725.
Hosmer D, Lemeshow S: Applied logistic regression. 1989, New York: Wiley
Lundin M, Lundin J, Burke HB, Toikkanen S, Pylkkanen L, Joensuu H: Artificial neural networks applied to survival prediction in breast cancer. Oncology. 1999, 57 (4): 281-286. 10.1159/000012061.
Hornberger J, Cosler LE, Lyman GH: Economic analysis of targeting chemotherapy using a 21-gene RT-PCR assay in lymph-node-negative, estrogen-receptor-positive, early-stage breast cancer. Am J Manag Care. 2005, 11 (5): 313-324.
Campbell HE, Taylor MA, Harris AL, Gray AM: An investigation into the performance of the Adjuvant! Online prognostic programme in early breast cancer for a cohort of patients in the United Kingdom. Br J Cancer. 2009, 101 (7): 1074-1084. 10.1038/sj.bjc.6605283.
Cufer T: Which tools can I use in daily clinical practice to improve tailoring of treatment for breast cancer? The 2007 St Gallen guidelines and/or Adjuvant! Online. Ann Oncol. 2008, 19 (7): vii41-vii45. 10.1093/annonc/mdn439.
Nowak V, Ah-See M, Ravichandran D: Breast cancer multidisciplinary team adjuvant therapy decision making and adjuvant! Online. Clin Oncol (R Coll Radiol). 2010, 22 (1): 87-88. 10.1016/j.clon.2009.11.002.
Schmidt M, Victor A, Bratzel D, Boehm D, Cotarelo C, Lebrecht A, Siggelkow W, Hengstler JG, Elsasser A, Gehrmann M: Long-term outcome prediction by clinicopathological risk classification algorithms in node-negative breast cancer–comparison between Adjuvant!, St Gallen, and a novel risk algorithm used in the prospective randomized Node-Negative-Breast Cancer-3 (NNBC-3) trial. Ann Oncol. 2009, 20 (2): 258-264.
Mook S, Schmidt MK, Rutgers EJ, van de Velde AO, Visser O, Rutgers SM, Armstrong N, van't Veer LJ, Ravdin PM: Calibration and discriminatory accuracy of prognosis calculation for breast cancer with the online Adjuvant! program: a hospital-based retrospective cohort study. Lancet Oncol. 2009, 10 (11): 1070-1076. 10.1016/S1470-2045(09)70254-2.
Bhoo-Pathy N, Yip CH, Hartman M, Saxena N, Taib NA, Ho GF, Looi LM, Bulgiba AM, van der Graaf Y, Verkooijen HM: Adjuvant! Online is overoptimistic in predicting survival of Asian breast cancer patients. Eur J Cancer. 2012, 48 (7): 982-989. 10.1016/j.ejca.2012.01.034.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/12/108/prepub
We thank Liao, Tzu-Chi for her excellent assistance in the statistic analysis.
The authors declare that they have no competing interests.
KYL and CTW designed the concept of this study, drafted the manuscript and performed treatment. CTW and KYL collected the data and performed the statistical analysis. KYL, CDR and CTW approved the final manuscript. KYL, CDR and CTW designed the concept of this study and provided treatment coordination. All authors read and approved the final manuscript.
About this article
Cite this article
Yao-Lung, K., Dar-Ren, C. & Tsai-Wang, C. Accuracy validation of adjuvant! online in Taiwanese breast cancer patients - a 10-year analysis. BMC Med Inform Decis Mak 12, 108 (2012). https://doi.org/10.1186/1472-6947-12-108