Accuracy validation of adjuvant! online in Taiwanese breast cancer patients - a 10-year analysis
© Kuo et al.; licensee BioMed Central Ltd. 2012
Received: 10 March 2012
Accepted: 7 September 2012
Published: 17 September 2012
Adjuvant! Online (http://www.adjuvantonline.com) is an Internet-based software program that allows clinicians to make predictions about the benefits of adjuvant therapy and 10-year survival probability for early-stage breast cancer patients. This model has been validated in Western countries such as the United States, United Kingdom, Canada, Germany, and Holland. The aim of our study was to investigate the performance and accuracy of Adjuvant! Online in a cohort of Taiwanese breast cancer patients.
Data on the prognostic factors and clinical outcomes of 559 breast cancer patients diagnosed at the National Cheng Kung University Hospital in Tainan between 1992 and 2001 were enrolled in the study. Comprehensive demographic, clinical outcome data, and adjuvant treatment data were entered into the Adjuvant! Online program. The outcome prediction at 10 years was compared with the observed and predicted outcomes using Adjuvant! Online.
Comparison between low- and high-risk breast cancer patient subgroups showed significant differences in tumor grading, tumor size, and lymph node status (p < 0.0001). The mean 10-year predicted death probability in 559 patients was 19.44%, and the observed death probability was 15.56%. Comparison with the Adjuvant! Online-predicted breast cancer-specific survival (BCSS) showed significant differences in the whole cohort (p < 0.001). In the low-risk subgroup, the predicted and observed outcomes did not differ significantly (3.69% and 3.85%, respectively). In high-risk patients, Adjuvant! Online overestimated breast cancer-specific survival (p = 0.016); the predicted and observed outcomes were 21.99% and 17.46%, respectively.
Adjuvant! Online accurately predicted 10-year outcomes and assisted in decision making about adjuvant treatment in low-risk breast cancer patients in our study, although the results were less accurate in the high-risk subgroup. Development of a prognostic program based on a national database should be considered, especially for high-risk breast cancer patients in Taiwan.
KeywordsAdjuvant! Online Breast cancer Mortality prediction Taiwan
Breast cancer treatments are based mainly on international guideline recommendations (e.g., the National Comprehensive cancer Network (NCCN) guidelines or St. Gallen consensus)[1–3]. Improvements in the efficacy of adjuvant chemotherapy have also improved the prognosis for early breast cancer. Adjuvant drug therapy can extend survival in breast cancer patients, although the balance between risks and benefits must be considered. The decision to prescribe adjuvant systemic therapy for early breast cancer patients is complex and requires consideration of conditions such as the patient’s and tumor characteristics, treatment effectiveness, and clinician and patient preferences. However, inaccurate outcome predictors can lead to the over- or undertreatment of patients. Recently, several statistical models or programs have been developed to predict outcomes for early breast cancer patients[5–10]. These programs can help to inform patients and allow them to choose a more individualized treatment strategy based on a number of prognostic factors.
Adjuvant! Online is a Web-based, open-access computer program that predicts patient outcome at 10 years in breast cancer patients. The development of the program is based on information from the Surveillance, Epidemiology, and End Results (SEER) registry. The SEER registry includes data for about 10% of breast cancer patients in the USA. The program analyzed the outcome of 10-year survival probabilities, risk of relapse, and survival using patient information and tumor characteristics such as age, tumor size, tumor grade, estrogen receptor (ER) status, and lymph node status.
Adjuvant! Online was validated successfully by Olivotto and colleagues in a population-based series of 4083 early-stage breast cancer patients in Canada. They concluded that the Adjuvant! Online program accurately predicted the observed breast cancer-related death outcome within 2% in the whole study population. However, Asian populations might differ from those in the USA or Canada, and the accuracy of Adjuvant! Online in predicting outcomes in the Asian, particularly the Taiwanese population is unknown.
The aims of our study were to validate the accuracy of Adjuvant! Online, to predict outcomes in different patient groups, and to distinguish individuals who will need different prediction methods in a cohort of Taiwanese patients with breast cancer treated at a multidisciplinary tumor board in a university hospital.
Between January 1, 1992, and December 31, 2001, 652 breast cancer patients were diagnosed and treated at National Cheng Kung University Hospital (NCKUH) in Tainan, Taiwan. The final cohort involved in the subset of 559 patients fulfilled the following criterion: follow-up period > 10 years or death due to breast cancer within 10 years from the diagnosed date. Ninety-three patients were excluded because of loss to follow-up or death from other causes. In all analyzed cases, 472 patients whose follow-up period was > 10 years were classified as “survived” (84.44%), and the other 87 patients who died of breast cancer within 10 years were classified as “died” (15.56%). Those who died of breast cancer beyond the 10-year follow-up were also classified as survived. Ethical approval was provided by the Human Experiment and Ethics Committee of the National Cheng Kung University Hospital (A-ER-101-013).
The demographic data included age at the time of presentation for cancer, and the pathological findings included tumor size, axillary lymph node status, tumor histological grade, and ER status. All pathological specimens were reviewed by breast pathologists at NCKUH. Tumor size was measured using pathological reports from NCKUH. The Scarff-Bloom-Richardson (SBR) system was used for tumor grading and was based on the following morphological features: nuclear pleomorphism of tumor cells, degree of tumor tubule formation, and tumor mitotic activity. To determine the ER status, immunohistochemistry was performed on formalin-fixed, paraffin-embedded breast cancer tissue samples from the patients. Positive ER status was defined as nuclear staining > 1%. Immunohistochemistry was performed using anti-ER (clone 6 F11, Ventana Medical Systems, Strasbourg, France). Postoperative adjuvant chemotherapy was performed according to the NCCN and St. Gallen guidelines.
Postoperative adjuvant chemotherapy was performed according to NCCN and St. Gallen guidelines. Between the period 1992 to 1995, CMF regimen was used as standard chemotherapy (cyclophosphamide 500 mg/m2, methotrexate 50 mg/m2 and 5-FU 500 mg/m2). Since 1996, FEC regimen (5-FU 500 mg/m2, epirubicin 100 mg/m2 and cyclophosphamide 500 mg/m2) was given in the majority of patients. Since 1998, for high risk patients, TEC regimen (Taxotere 75 mg/m2, epirubicin 75 mg/m2 and cyclophosphamide 500 mg/m2) was performed.
χ 2 and two-sample independent t tests were used to compare variables between the two subgroups. Data for demographic and tumor characteristics for each patient, such as age, ER status, grade, tumor size, lymph node status, and treatment modalities (chemotherapy or/and hormone therapy) were entered into the Adjuvant! Online program (version 8.0), which produced a 10-year predicted probability for death due to breast cancer. For the comorbidity item “average for age” was imputed for all patients. The Hosmer–Lemeshow test was used to assess whether the predicted probabilities matched the observed probabilities in subgroups of the patient population. The difference in the number of categories reflects the different subgroup sizes. P values < 0.05 were considered significant. The observed probability was regressed on the predicted probability, and R 2 was calculated to measure the goodness of fit of the linear regression line. All statistical analyses were performed with SPSS 17.0 software.
Patient characteristics in the whole cohort
Total N = 559 (%)
Less than expected cycles
Patient characteristics in the low- and high-risk subgroups
Low Risk (N = 78)
High Risk (N = 481)
Less than expected cycles
The rate of loss to follow-up was 14.26% because some patients were lost of follow-up after 5 years hormonal treatment. These patients were excluded from our data analysis, and our more conservative data analysis method might have overestimated the death probability.
Adjuvant treatment for postoperative early breast cancer patients remains a great challenge for physicians and patients who must consider both the risks and benefits of treatment, possible comorbidities, and especially the desire to maintain quality of life. Several tools to support decisions have been developed[5, 7, 8, 14, 15]. One such tool, Adjuvant! Online, is a computerized, Web-based program that predicts recurrence and mortality risk and the benefit of adjuvant treatment in early breast cancer patients. The program is based on the database from the US SEER tumor registry database. The SEER tumor registry collected information from about 10% of all breast cancer cases in the USA. The database used for Adjuvant! Online included information such as the patients’ demographics and tumor characteristics (tumor size, the number of positive nodes, tumor grade), and survival in postoperative breast cancer patients aged 20 to 79 years between 1988 and 1992. After entering these data, the program calculated the annual breast cancer mortality rates and produced data for comparison with the database from the SEER tumor registry. These data were used to predict the 10-year survival. Adjuvant! Online can be used to provide recommendations for adjuvant systemic therapy after considering the estimated 10-year overall survival (OS), breast cancer-specific survival (BCSS), and event-free survival (EFS) However, the lack of prognostic power of Adjuvant! Online in different populations raises questions about the combination of prognostic factors and accuracy of patients’ characteristics. Our goal was to determine the accuracy of the program applied to an Asian population and, if the program is accurate, which subgroup should be included.
The tool has been validated and used by oncologists in different countries including Canada, Germany, Holland, and the United Kingdom (UK)[12, 16–19]. In an analysis of 4083 early breast cancer patients in Canada, Olivotto et al. showed that the overall predicted and observed 10-year outcomes were within 2% for OS, BCSS, and EFS. The Adjuvant! Online program was also validated in small cohorts of patients in Germany. The increased use of the program by physicians and the positive results in Western countries prompted us to analyze the accuracy of the program in an Asian population.
In the present study, the difference between the predicted and observed outcomes in the low-risk cohort was about 1%. We conclude that Adjuvant! Online is an accurate tool for predicting the outcome in low-risk breast cancer patients in the Taiwanese population. By contrast, we observed a large discrepancy between our prediction and that of Adjuvant! Online in the high-risk population. That is, Adjuvant! Online underestimated the mortality risk in the high-risk subgroup of Taiwanese breast cancer patients. Considering this discrepancy, we suggest that a correction factor of 1.259 might be justified for high-risk patients. Variations in the program validation may differ between countries and ethnic factors are known to be determining factors that can influence the decision about and outcomes of adjuvant treatment. The program should be validated in different countries and ethnic groups before wider application of these data.
Campbell et al. showed that the recommendations for adjuvant treatment made by a UK-based multidisciplinary team using Adjuvant! Online would improve decision making in only a minority (2.5%) of patients. Our results are similar to those of Campbell et al. In a cohort of 1065 British early breast cancer patients, Campbell et al. found that Adjuvant! Online significantly overestimated OS (5.54%, p < 0.001), BCSS (4.53%, p < 0.001), and EFS (3.51%, p = 0.001). They concluded that overestimation of BCSS implies that Adjuvant! Online underestimates breast cancer mortality for women in the UK. One possible reason for this difference may be that the breast cancer mortality rate is lower in the USA than in the UK.
A Dutch validation study of 5380 breast cancer patients showed no significant differences between the 10-year observed OS and BCSS compared with the Adjuvant! Online-predicted OS and BCSS (p = 0.87 and p = 0.18, respectively). However, in the subgroup with patients aged < 40 years, Adjuvant! Online overestimated the predicted OS and BCSS (4.2%, p = 0.04 and 4.7%, p = 0.01, respectively). In the patient subgroup older than 69 years, Adjuvant! Online overestimated the predicted OS (3.4%, p = 0.05). Finally, the program underestimated predicted BCSS in the patient subgroup with 1–3 lymph node metastases (3.1%, p = 0.002). Our data are consistent with these earlier findings in that we found that Adjuvant! Online predicted BCSS accurately in some patient subgroup.
Compared with other studies that validated the accuracy and performance of Adjuvant! Online, our study might be considered to have had a small sample size. However, one advantage in our study is the low proportion of missing data. Data for tumor size, lymph node status, ER status, and tumor grade were collected routinely and rechecked by a clinician and data manager, and our missing data rate was around 25%. Our missing data rate is similar to the publication from Bhoo-Pathy et al. (25%). In the study by Mook et al., 3104 of 5380 patients (58%) had data missing for ER status or tumor grade. However, they concluded that including the patients with incomplete data did not influence the prediction of outcomes. The program can also predict disease outcome without information about ER status. By contrast, Adjuvant! Online tend to underestimate the BCSS in ER-negative breast cancer patients. Mook et al. concluded that this result was observed in patients undergoing hormonal treatment. Another limitation of our study is the adjuvant treatment modalities. The period of data recruitment for this study was from 1992 to 2001. According to the NCCN guidelines and policy of Taiwan national health insurance, standard adjuvant chemotherapy comprised CMF (cyclophosphamide, methotrexate, and fluorouracil) or an anthracycline-based regimen, and standard adjuvant endocrine therapy comprised tamoxifen for HR-positive patients. However, the treatment varies between countries, especially in relation to different national policies for the use of treatments, such as taxane-containing regimen chemotherapy or aromatase inhibitors.
This validation study of Adjuvant! Online applied to a university hospital-based cohort of Taiwanese breast cancer patients showed that the calculated predictions by Adjuvant! Online matched the observed outcomes and that the predictions are suitable when applied to low-risk, early breast cancer patients. We agree that Adjuvant! Online is a useful tool for doctors and patients, but it should be used only by those aware of its limitations, as advised in the program’s instructions and as reported elsewhere.
We thank Liao, Tzu-Chi for her excellent assistance in the statistic analysis.
- Goldhirsch A, Ingle JN, Gelber RD, Coates AS, Thurlimann B, Senn HJ: Thresholds for therapies: highlights of the St Gallen International Expert Consensus on the primary therapy of early breast cancer 2009. Ann Oncol. 2009, 20 (8): 1319-1329. 10.1093/annonc/mdp322.View ArticlePubMedPubMed CentralGoogle Scholar
- Goldhirsch A, Wood WC, Coates AS, Gelber RD, Thurlimann B, Senn HJ: Strategies for subtypes--dealing with the diversity of breast cancer: highlights of the St. Gallen International Expert Consensus on the Primary Therapy of Early Breast Cancer 2011. Ann Oncol. 2011, 22 (8): 1736-1747. 10.1093/annonc/mdr304.View ArticlePubMedPubMed CentralGoogle Scholar
- National Comprehensive Cancer Network Web Site.http://www.nccn.org/index.asp,
- Early Breast Cancer Trialists' Collaborative Group (EBCTCG): Effects of chemotherapy and hormonal therapy for early breast cancer on recurrence and 15-year survival: an overview of the randomised trials. Lancet. 2005, 365 (9472): 1687-1717.View ArticleGoogle Scholar
- Galea MH, Blamey RW, Elston CE, Ellis IO: The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Res Treat. 1992, 22 (3): 207-219. 10.1007/BF01840834.View ArticlePubMedGoogle Scholar
- Lyman GH, Cosler LE, Kuderer NM, Hornberger J: Impact of a 21-gene RT-PCR assay on treatment decisions in early-stage breast cancer: an economic analysis based on prognostic and predictive validation studies. Cancer. 2007, 109 (6): 1011-1018. 10.1002/cncr.22506.View ArticlePubMedGoogle Scholar
- Ravdin PM, Siminoff LA, Davis GJ, Mercer MB, Hewlett J, Gerson N, Parker HL: Computer program to assist in making decisions about adjuvant therapy for women with early breast cancer. J Clin Oncol. 2001, 19 (4): 980-991.PubMedGoogle Scholar
- Siminoff LA, Gordon NH, Silverman P, Budd T, Ravdin PM: A decision aid to assist in adjuvant therapy choices for breast cancer. Psychooncology. 2006, 15 (11): 1001-1013. 10.1002/pon.1040.View ArticlePubMedGoogle Scholar
- Chang TW, Kuo YL: A model building exercise of mortality risk for Taiwanese women with breast cancer. BMC Med Inform Decis Mak. 2010, 10: 43-10.1186/1472-6947-10-43.View ArticlePubMedPubMed CentralGoogle Scholar
- Adjuvant! Online Web Site.http://www.adjuvantonline.com,
- National Cancer Institute: Surveillance, Epidemiology, and End Results (SEER) Web Site .http://seer.cancer.gov/,
- Olivotto IA, Bajdik CD, Ravdin PM, Speers CH, Coldman AJ, Norris BD, Davis GJ, Chia SK, Gelmon KA: Population-based validation of the prognostic model ADJUVANT! for early breast cancer. J Clin Oncol. 2005, 23 (12): 2716-2725.View ArticlePubMedGoogle Scholar
- Hosmer D, Lemeshow S: Applied logistic regression. 1989, New York: WileyGoogle Scholar
- Lundin M, Lundin J, Burke HB, Toikkanen S, Pylkkanen L, Joensuu H: Artificial neural networks applied to survival prediction in breast cancer. Oncology. 1999, 57 (4): 281-286. 10.1159/000012061.View ArticlePubMedGoogle Scholar
- Hornberger J, Cosler LE, Lyman GH: Economic analysis of targeting chemotherapy using a 21-gene RT-PCR assay in lymph-node-negative, estrogen-receptor-positive, early-stage breast cancer. Am J Manag Care. 2005, 11 (5): 313-324.PubMedGoogle Scholar
- Campbell HE, Taylor MA, Harris AL, Gray AM: An investigation into the performance of the Adjuvant! Online prognostic programme in early breast cancer for a cohort of patients in the United Kingdom. Br J Cancer. 2009, 101 (7): 1074-1084. 10.1038/sj.bjc.6605283.View ArticlePubMedPubMed CentralGoogle Scholar
- Cufer T: Which tools can I use in daily clinical practice to improve tailoring of treatment for breast cancer? The 2007 St Gallen guidelines and/or Adjuvant! Online. Ann Oncol. 2008, 19 (7): vii41-vii45. 10.1093/annonc/mdn439.PubMedGoogle Scholar
- Nowak V, Ah-See M, Ravichandran D: Breast cancer multidisciplinary team adjuvant therapy decision making and adjuvant! Online. Clin Oncol (R Coll Radiol). 2010, 22 (1): 87-88. 10.1016/j.clon.2009.11.002.View ArticleGoogle Scholar
- Schmidt M, Victor A, Bratzel D, Boehm D, Cotarelo C, Lebrecht A, Siggelkow W, Hengstler JG, Elsasser A, Gehrmann M: Long-term outcome prediction by clinicopathological risk classification algorithms in node-negative breast cancer–comparison between Adjuvant!, St Gallen, and a novel risk algorithm used in the prospective randomized Node-Negative-Breast Cancer-3 (NNBC-3) trial. Ann Oncol. 2009, 20 (2): 258-264.View ArticlePubMedGoogle Scholar
- Mook S, Schmidt MK, Rutgers EJ, van de Velde AO, Visser O, Rutgers SM, Armstrong N, van't Veer LJ, Ravdin PM: Calibration and discriminatory accuracy of prognosis calculation for breast cancer with the online Adjuvant! program: a hospital-based retrospective cohort study. Lancet Oncol. 2009, 10 (11): 1070-1076. 10.1016/S1470-2045(09)70254-2.View ArticlePubMedGoogle Scholar
- Bhoo-Pathy N, Yip CH, Hartman M, Saxena N, Taib NA, Ho GF, Looi LM, Bulgiba AM, van der Graaf Y, Verkooijen HM: Adjuvant! Online is overoptimistic in predicting survival of Asian breast cancer patients. Eur J Cancer. 2012, 48 (7): 982-989. 10.1016/j.ejca.2012.01.034.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/12/108/prepub