This article has Open Peer Review reports available.
Automated systems to identify relevant documents in product risk management
© Wee et al; licensee BioMed Central Ltd. 2012
Received: 19 July 2011
Accepted: 2 March 2012
Published: 2 March 2012
Product risk management involves critical assessment of the risks and benefits of health products circulating in the market. One of the important sources of safety information is the primary literature, especially for newer products which regulatory authorities have relatively little experience with. Although the primary literature provides vast and diverse information, only a small proportion of which is useful for product risk assessment work. Hence, the aim of this study is to explore the possibility of using text mining to automate the identification of useful articles, which will reduce the time taken for literature search and hence improving work efficiency. In this study, term-frequency inverse document-frequency values were computed for predictors extracted from the titles and abstracts of articles related to three tumour necrosis factors-alpha blockers. A general automated system was developed using only general predictors and was tested for its generalizability using articles related to four other drug classes. Several specific automated systems were developed using both general and specific predictors and training sets of different sizes in order to determine the minimum number of articles required for developing such systems.
The general automated system had an area under the curve value of 0.731 and was able to rank 34.6% and 46.2% of the total number of 'useful' articles among the first 10% and 20% of the articles presented to the evaluators when tested on the generalizability set. However, its use may be limited by the subjective definition of useful articles. For the specific automated system, it was found that only 20 articles were required to develop a specific automated system with a prediction performance (AUC 0.748) that was better than that of general automated system.
Specific automated systems can be developed rapidly and avoid problems caused by subjective definition of useful articles. Thus the efficiency of product risk management can be improved with the use of specific automated systems.
Product risk management involves critical assessment of health product safety issues by evaluating their risk-benefit ratio, on occasion followed by taking appropriate regulatory actions to mitigate these safety concerns . Environmental scanning of safety information of health products is usually conducted through several sources to ensure thorough search and extensive coverage. Firstly, local trends of adverse events incidence can be obtained from spontaneous adverse events reporting. These adverse events are observed during clinical practice and reported to the regulatory agency by healthcare professionals or by pharmaceutical manufacturers. Information on product safety may also be obtained from safety alerts disseminated by different regulatory authorities such as the Therapeutic Goods Administration in Australia, the Food and Drug Administration in the United States of America and the European Medicines Agency in Europe. Regulatory authorities all over the world work closely together and alert each other of any safety concerns raised. Another source of safety information will be the primary literature where results of clinical trials, case reports and other safety-related studies may be reported.
Primary literature remains as a valuable source of drug safety information, especially for newer drugs where there is little regulatory experience with them. With intensive research in medical sciences, primary literature is generated at a very rapid rate. In PubMed alone, more than 600,000 articles are added each year from over 5000 journals . Although, there is voluminous amount of information available, only a small portion is useful for risk assessment. For instance, a search in PubMed for tumour necrosis factor-alpha (TNF-α) blockers using the search terms 'adalimumab', 'infliximab' and 'etanercept' in Medical Subject Headings (MeSH)  retrieved close to 4000 articles. However, only about 700 (17.5%) contain valuable information for risk assessment work. Thus, it is time-consuming and inefficient to manually sieve through this large number of articles and identify those that are valuable to product risk assessment. Hence, the ability to expedite this process of useful literature identification can contribute to risk assessment efficiency.
Text mining as a potential solution
Text mining is defined as the process of retrieving or extracting small nuggets of relevant information from large collections of textual data . It is a powerful tool to identify word usage patterns and has already been effectively deployed in many areas such as email classification [5, 6], legal/business applications [7, 8] and biomedical text analysis[9, 10]. In one study, Agarwal and Yu had shown that the use of text mining was able to achieve 91.95% of accuracy in automatically classifying sentences from biomedical full text into introduction, methods, discussion and conclusion categories . Wang et al showed that text mining was able to achieve 95% sensitivity and specificity in 51.5% of abstracts that were automatically classified for the purpose of Immune Epitope Database . Recently, text mining was used to improve systematic reviews of adverse drug reactions by identifying such articles from medical literature with a recall of 70% and precision of 21% . Hence, text mining had been shown to be a powerful and useful tool in the automated classification of textual data. It will be interesting to explore text mining as a potential solution for developing an automated system to identify relevant documents for product risk management work.
In this work, two automated systems were developed and explored for their usefulness in identifying 'useful' articles from their titles and abstracts (henceforth referred to as just abstracts) in the PubMed database. The first system used only general terms that were found in the abstracts as predictors and thus is able to identify 'useful' articles regardless of the drug class. However such general automated system may not have sufficiently high accuracy for routine risk assessment work. Thus, a second system which was specific for a particular drug class was developed and tested. During routine work, evaluators will manually classify a small number of articles related to the drug class of interest. The second system will then learn from the abstracts of these articles and develop a model that is specific for that drug class.
In order to develop the two automated systems, large amounts of journal articles have to be manually classified. Ideally, a large number of articles from different drug classes should be used, especially for the development of the general automated system. However, it is tedious and impractical to perform manually classification of such a large number of articles. Thus, in this study, only articles on TNF-α blockers were manually classified and used to develop the two automated systems. TNF-α blockers were chosen because they have a relatively small corpus of literature, which make it suitable for manual classification.
TNF-α blockers are biologics that are indicated for several autoimmune diseases such as psoriasis, rheumatoid arthritis and Crohn's disease, and are playing an emerging role under circumstances when these disease conditions are refractory to conventional therapies . However, intensive post-marketing surveillance [15, 16] and case reports [17–19] reveal rare but severe adverse effects such as opportunistic infections, reactivation of latent infections, new-onset psoriasis and lymphomas. It is unsure whether these adverse effects are due to predisposition by the underlying diseases or the adverse effects of TNF-α blockers per se, therefore post-marketing surveillance is of paramount importance in monitoring the safety profile this group of drugs.
Comparison of performance of various algorithms
Comparison of performance of different algorithms using general predictors on the validation set
K-nearest neighbor (k = 3)
SVM (gamma = 1.0, C = 0.0)
Comparison of performance of different types of frequencies
Comparison of performance of SVM models using different types of frequencies on the validation set
Type of frequency
General automated system
The black line shows the cumulative percentage of 'useful' articles with the assumption that 'useful' papers would be distributed at random in each 10% sample of the set. This represents the case where the general automated system was not used. In such a scenario, identification of 'useful' papers will follow a linear progression correlated to the total number of examined papers.
The line graph shows the cumulative percentage of 'useful' articles when the general automated system was used to present papers to the evaluators in decreasing order of confidence. The confidence values were computed by the SVM model for each article. The percentages for both the black line and line graph can be read using the vertical axis on the right. For instance, 46.4% (102 out of 220) and 70.9% (156 out of 220) of the total number of 'useful' articles were among the first 10% and 20% of the articles presented to the evaluators by the general automated system respectively. The difference between the line graph and black line shows that the general automated system was able to improve the identification of 'useful' articles by using the SVM confidence values to rank the articles, compared to the case when the automated system was not used.
Each red bar represents the number of 'useful' articles contained in every 10% of the total articles presented to the evaluators by the system. For example, the first bar showing 102/119 shows that there were 102 'useful' articles among the first 119 articles that were presented to the evaluators by the general automated system. In the second bar, 54 'useful' articles were present among the next 119 articles.
Specific automated system
Comparison of performance of models trained on various training set sizes using all predictors on the validation set
Training set size
Ratio of useful: non-useful articles
Comparison of performance of various algorithms
Different supervised learning algorithms were used to develop various prediction models. The results showed that the model developed using SVM had the best prediction performance compared to those developed using other algorithms. This is in concordance with other studies which frequently showed that models developed using SVM outperforms those developed using other learning algorithms .
Comparison of performance using different types of frequencies
The prediction performance of models developed using either word occurrence, binary frequency or TF-IDF of predictors were compared. The model developed using TF-IDF showed the best performance, followed by the model using word occurrence and lastly the model using binary frequency. In the computation of TF-IDF, terms that occurred less frequently in the training set were given higher values and those that occurred very frequently were given lower values. A possible reason for the better prediction performance of the model developed using TF-IDF is that those words that occur less frequently play a key role in the classification of 'useful' and 'non-useful' articles. In addition, binary frequency can only provide information on the presence or absence of a term and not its occurrence frequency. Thus, the poorer performance of the model developed using binary frequency suggested that the classification of the articles may be dependent on the frequency of term occurrence.
General automated system
A general automated system was developed using SVM and TF-IDF of general predictors. Only general predictors were used to develop the system because predictors that are specific for TNF-α blockers are usually not present in articles on other drug classes. Thus, the use of these specific predictors in an automated system will add noise to the system, which will reduce its generalizability.
The results showed that the general automated system was able to rank the articles such that approximately 70% of the 'useful' articles on TNF-α blockers were found in the first 20% of TNF-α blockers articles. The generalizability performance for classifying articles on four other drug classes were lower, which suggests that there could be other general predictors which were not present in the abstracts of articles on TNF-α blockers. Thus there is a need to periodically update the general predictors used in the general automated system with newly classified abstracts of articles from different drug classes in order to maintain or improve its generalizability performance.
The advantage of the general automated system is that the evaluators need not manually classify any articles before using the system. A major limitation of the general automated system is that the system is trained using abstracts that were manually classified into 'useful' and 'non-useful'. Although a systematic approach (Additional file 1) was used in this study to classify the articles, the classification scheme may not be applicable for different drug classes or different risk assessment tasks. In fact, it is important to note that product risk management involves using a variety of information sources and primary literature is just one of the sources. Thus different risk assessment tasks may have slightly different definition of 'useful' and 'non-useful' articles. This subjectiveness in classification of 'useful' articles may limit the applicability of the general automated system.
Specific automated system
The general trend of the prediction performance of specific automated systems developed using training sets of various sizes is that an increase in training set size correlated with improved performance of the automated system. The AUC of the system trained using 36 articles appeared to be an anomaly from this general trend. This could be due to its ratio of 'useful' to 'non-useful' articles, which was much more unbalanced compared to those in the rest of the training sets. Studies have shown that models developed from highly unbalanced training sets tend to have poorer prediction performance than those developed from balanced training sets [21, 22].
The advantage of the specific automated system is that a new automated system will be developed for each new drug class or new risk assessment task. This resolves problems caused by the subjective definition of 'useful' and 'non-useful' articles and thus prevent the potential poor generalizability associated with the general automated system. This study shows that manual classification of just 20 articles is sufficient to develop a specific automated system. Based on our experience, an experienced evaluator can easily manually classify 20 articles within a short time of approximately 30 minutes. Thus the use of such specific automated system is feasible for routine risk assessment work.
One disadvantage is that in the selection of articles for manual classification, 'useful' articles may not be selected by the Kennard and Stone sampling method or the selection may result in a highly unbalanced 'useful' to 'non-useful' articles ratio in the training set which may potentially decrease the performance of the automated system. The lower performance of the automated system trained using 36 articles could constitute an example of such problem.
A potential solution is to use the general automated system to perform the initial selection of articles that are most likely to be 'useful' and articles that are most likely to be 'non-useful' for manual classification. Although the general automated system may not have high generalizability performance, using it will still increase the likelihood of achieving a more balanced 'useful' to 'non-useful' articles ratio in the training set, compared to the alternative option of not using it.
Potential application of automated systems in routine risk assessment work
Current limitations and potential for improving the automated systems
In this study, parameter selection of the SVM algorithm was guided by determining the AUC of the automated systems on the testing set. An implicit assumption for AUC values is that both false positives and false negatives are equally problematic. This is clearly not the case for risk assessment task as false positives will only result in additional workload for the evaluators and will be identified during manual review of the results. In comparison, false negatives may result in important articles not being examined. Thus, future studies could explore using a customized performance measure which weighs false positives and false negatives differently to guide parameter selection.
The automated systems were developed using 'useful' and 'non-useful' articles. This assumes that all 'useful' articles are equally valuable. This is an over-simplification as some articles are more 'useful' than others. However, it is difficult to determine the degree of 'usefulness' for an article and thus it is not possible to develop regression models for use in the current automated systems. Future work could attempt to address this limitation by developing methods to ease the determination of the degree of 'usefulness' for an article.
In this study, it was assumed that important information required for classification of the articles will be summarized in the abstracts. However, authors may describe safety issues in the article but did not include them in the abstracts. In addition, several articles may discuss about similar safety issues but not all will include warnings and precautions on the use of these drugs in the abstracts. Hence, not all potentially 'useful' articles could be identified by the two automated systems which use only the abstracts. A potential solution is to develop and apply the automated systems on full articles . However, the presence of several obstacles made it challenging to make use of full articles in such automated system . Firstly, bulk download of full articles is difficult to automate. Moreover, copyright and fair use issues make retrieval of full text of all entries indexed on PubMed unfeasible. In addition, retrieved documents are often in PDF or HTML format and will be required to be converted into plain text prior to being used for text mining. Unfortunately, this conversion is not always accurate. Furthermore, symbols (e.g. 'ε' and 'α') are frequently used in full articles of biomedical literature and they require replacement with their spelled names. Such replacements have usually been done in their abstracts, which makes preparation of abstracts for feeding into automated systems less tedious.
A potential method to improve the performance is to explore semantic features. Semantics refers to the study of 'meanings' linked to their words in linguistic studies . Semantic features could be applied by using MeSH and the Unified Medical Language System (UMLS) concepts and semantic types [12, 20]. UMLS is the most extensive known database of synonyms and concepts relations of biomedical and health-related terms, maintained by National Library of Medicine. It can be used to map related concepts to words in text mining applications  and has been used to extract disease drug knowledge from biomedical and clinical documents [27, 28]. This addition of meanings and concepts of the terms linked to their respective concepts may potentially boost the performance of the model.
Another method to improve performance is to use MeSH terms in addition to the abstracts for creating the predictors. The reason for using MeSH terms is because it contained information on the topics of the articles, which may be useful for the classification of the articles by the automated systems. For example, articles related to pharmacoeconomics are often focused on the decrease in cost burden to patients and to society but not on the adverse drug reactions. On the other hand, articles related to pharmacokinetics of a drug may occasionally be useful as they may report on dose-dependent adverse drug reactions. The inclusion of MeSH terms had been found to improve the prediction performance of the epitope model  and thus could possibly be useful for improving the performance of the automated systems.
In this study, a general automated system for identifying relevant journal articles for product risk assessment work was developed using a large set of abstracts of articles on TNF-α blockers, which were manually classified as 'useful' or 'non-useful'. In addition, the feasibility of developing a specific automated system for each drug class was explored. The results showed that the general automated system was generally able to rank 'useful' articles ahead of 'non-useful' articles, though that ability was poorer on drug classes other than TNF-α blockers. In addition, the subjectiveness in determining 'useful' and 'non-useful' articles may preclude adoption of the general automated system for routine product risk management work. In comparison, a specific automated system with better prediction performance than the general automated system could be developed using approximately 20 abstracts. Since a new specific automated system has to be developed for each new risk assessment task, it will avoid the problem that is caused by the subjective definition of 'useful' and 'non-useful' articles. Hence, the results from this study shows that text mining is suitable to improve the efficiency of literature search performed during comprehensive risk assessment. It is important to note that the use of such automated systems will not make risk management more effective but instead is to help access all data in the medical literature that is relevant to risk management reviews more effectively.
Using the search terms 'adalimumab', 'infliximab' and 'etanercept' in MeSH , a total of 3966 abstracts of articles relevant to TNF-α blockers that were indexed on PubMed were retrieved (as of 11 September 2009). The retrieved articles undergo manual classification into 'useful' and 'non-useful' categories. In this study, 'useful' articles are defined as those containing information that may potentially warrant a regulatory action like an update on the contraindications, warnings and precautions on the use of the drug. These articles may report on newly encountered adverse events, incidences of rare adverse events, mechanisms of action of certain adverse events or any other relevant data. For example, a review that shows TNF-α blockers may possibly be causative agents in congenital abnormalities is considered as a 'useful' article . 'Non-useful' articles refer to any other articles that do not contain any of the above mentioned relevant' information. Additional file 1 shows the classification algorithm used in this study to classify uncomplicated cases. For complicated and ambiguous cases, the full articles were retrieved for further review to ensure accurate classification. A total of 693 articles were classified as 'useful' (17.5%) and 3273 articles were classified as 'non-useful' (82.5%).
where n i, j is the number of occurrences of predictor i in document j, N is the total number of documents and n i is the number of documents in which predictor i occurs. In TF-IDF, predictors that were common in an abstract j but were less common in all the other abstracts were considered as more important to abstract j . These three types of frequencies were used in this study to determine the optimum type for classifying articles for risk assessment work.
Supervised machine learning
Supervised learning methods were used in this study to develop the models as it had been shown to have better performance than unsupervised learning methods in other studies . In supervised learning, the prediction model was trained to recognize patterns in the abstracts of both 'useful' and 'non-useful' articles. Supervised learning algorithms such as logistic regression , k-nearest neighbors , Naive Bayes  and support vector machine (SVM) [35, 36] implemented in the software RapidMiner 5.0  were used to develop the models. These algorithms have been extensively described elsewhere and thus only a brief description is provided for SVM, which was found to have the best performance in this study.
The SVM model was also able to compute a confidence value for the classification of an unlabeled article as a 'useful' one. This can be used to rank the articles such that those with higher confidence values were shown to the evaluators first.
General automated system
A general automated system was developed using the training set and the set of general predictors. General predictors were used because they are non-specific to TNF-α blockers and thus may improve the generalizability of the system compared to a system developed using all available predictors. The testing set was used to guide parameter selection for the modeling algorithms. The validation set was used to validate the performance of the general automated system and the generalizability set was used to assess its generalizability.
Specific automated system
In this study, the feasibility of creating a separate automated system for each drug class or each new risk assessment task during routine risk assessment work was also investigated. Evaluators would manually classify a small number of articles related to a particular drug class and use their abstracts to develop the specific automated system. The system will then be able to automatically classify the remaining articles. In this study, the minimum number of abstracts required to train a model with satisfactory performance was determined by training models using 2 to approximately 100 abstracts from dataset A. The remaining abstracts in dataset A were used as the testing set. The selection of the abstracts was done by using Kennard and Stone sampling . In this sampling technique, the abstracts that were most different in terms of their extracted terms from each other in dataset A were selected for training the models. The models were then validated using the validation set to assess their prediction performance.
Measuring prediction performance
The performances of the models were measured using area under the receiver operating characteristic curve (AUC) , which is frequently used to evaluate prediction models in the biomedical informatics field . AUC quantifies the probability that the model will classify a randomly chosen 'useful' article higher than a randomly chosen 'non-useful' article. Hence, a higher AUC value would indicate better performance of the model.
Lift charts were created for the general automated system. A lift chart is a graphical representation of the ranking efficiency of a prediction model. The ranks of the articles were determined using the confidence values provided by the general automated system for each unlabelled article. A good prediction model will be able to give higher ranks to 'useful' articles than 'non-useful' articles.
This work was supported by the National University of Singapore (NUS) start-up grant (R-148-000-105-133 to Chun Wei Yap) and NUS Department of Pharmacy Final Year Project grant (R-148-000-003-001 to Xue Ting Wee).
- Bull J: US Activities in Risk Management of Pharmaceutical Products. Pharmacovigilance. Edited by: Mann RD, Andrews EB. 2007, Chichester, West Sussex, England: John Wiley & Sons, 2Google Scholar
- Fact sheet: MEDLINE®. [http://www.nlm.nih.gov/pubs/factsheets/medline.html]
- Medical Subject Headings (MeSH®). [http://www.nlm.nih.gov/mesh/]
- Spasic I, Ananiadou S, McNaught J, Kumar A: Text mining and ontologies in biomedicine: making sense of raw text. Brief Bioinform. 2005, 6: 239-251. 10.1093/bib/6.3.239.View ArticlePubMedGoogle Scholar
- Grieser L, Hippner H, Wilde KD: E-Mail bounce management using text mining. 42nd Hawaii International Conference on System Sciences. 2009, 1-10.Google Scholar
- Sahami M, Dumais S, Heckerman D, Horvitz E: A bayesian approach to filtering junk e-mail. AAAAI'98 Workshop on Learning for Text Categorisation. 1998, 55-62.Google Scholar
- Sakurai S, Ueno K: Analysis of daily business reports based on sequential text mining method. IEEE International Conference on Systems, Man and Cybernetics. 2004, 2004: 3279-3284.Google Scholar
- Ticom AAM, de Souza B, de Lima LP: Text mining and expert systems applied in labor laws. Seventh International Conference on Intelligent Systems Design and Applications. 2007, 2007: 788-792.View ArticleGoogle Scholar
- Lu Z: PubMed and beyond: a survey of web tools for searching biomedical literature. Database (Oxford). 2011, 2011: baq036-View ArticleGoogle Scholar
- Trieschnigg D, Pezik P, Lee V, de Jong F, Kraaij W, Rebholz-Schuhmann D: MeSH Up: effective MeSH text classification for improved document retrieval. Bioinformatics. 2009, 25 (11): 1412-1418. 10.1093/bioinformatics/btp249.View ArticlePubMedPubMed CentralGoogle Scholar
- Agarwal S, Yu H: Automatically classifying sentences in full-text biomedical articles into Introduction, Methods, Results and Discussion. Bioinformatics. 2009, 25 (23): 3174-3180. 10.1093/bioinformatics/btp548.View ArticlePubMedPubMed CentralGoogle Scholar
- Wang P, Morgan A, Zhang Q, Sette A, Peters B: Automating document classification for the Immune Epitope Database. BMC Bioinforma. 2007, 8 (1): 269-10.1186/1471-2105-8-269.View ArticleGoogle Scholar
- Amardeilh F, Bousquet C, Guillemin-Lanne S, Wiss-Thebault M, Guillot L, Delamarre D, Lillo-Le Louet A, Burgun A: A knowledge management platform for documentation of case reports in pharmacovigilance. Stud Health Technol Inform. 2009, 150: 517-521.PubMedGoogle Scholar
- Lacy CF, Armstrong LL, Goldman MP, Lance LL: Drug Information Handbook with International Trade Names Index. 2009, Hudson, Ohio: Lexi-Comp Inc, 18Google Scholar
- Caviglia R, Boskoski I, Cicala M: Long-term treatment with infliximab in inflammatory bowel disease: safety and tolerability issues. Expert Opin Drug Saf. 2008, 7 (5): 617-632. 10.1517/147403220.127.116.117.View ArticlePubMedGoogle Scholar
- Garcia-Vidal C, Rodriguez-Fernandez S, Teijon S, Esteve M, Rodriguez-Carballeira M, Lacasa JM, Salvador G, Garau J: Risk factors for opportunistic infections in infliximab-treated patients: the importance of screening in prevention. Eur J Clin Microbiol Infect Dis. 2009, 28 (4): 331-337. 10.1007/s10096-008-0628-x.View ArticlePubMedGoogle Scholar
- Carter JD, Gerard HC, Hudson AP: Psoriasiform lesions induced by tumour necrosis factor antagonists: a skin-deep medical conundrum. Ann Rheum Dis. 2008, 67 (8): 1181-1183. 10.1136/ard.2007.082842.View ArticlePubMedGoogle Scholar
- Komatsuda A, Wakui H, Nimura T, Sawada K: Reversible infliximab-related lymphoproliferative disorder associated with Epstein-Barr virus in a patient with rheumatoid arthritis. Mod Rheumatol. 2008, 18 (3): 315-318. 10.1007/s10165-008-0053-0.View ArticlePubMedGoogle Scholar
- Nakashima C, Tanioka M, Takahashi K, Miyachi Y: Diffuse large B-cell lymphoma in a patient with rheumatoid arthritis treated with infliximab and methotrexate. Clin Exp Dermatol. 2008, 33 (4): 437-439. 10.1111/j.1365-2230.2007.02683.x.View ArticlePubMedGoogle Scholar
- Lee M, Wang W, Yu H: Exploring supervised and unsupervised methods to detect topics in biomedical text. BMC Bioinforma. 2006, 7 (1): 140-10.1186/1471-2105-7-140.View ArticleGoogle Scholar
- Japkowicz N: The class imbalance problem: Significance and strategies. Proceedings of the 2000 International Conference on Artificial Intelligencel. 2000, 2000: 111-117.Google Scholar
- Qiong G, Zhihua C, Li Z, Bo H: Data mining on imbalanced data sets. International Conference on Advanced Computer Theory and Engineering. 2008, 2008: 1020-1024.Google Scholar
- Müller H-M, Kenny EE, Sternberg PW: Textpresso: an ontology-based information retrieval and extraction system for biological literature. PLoS Biol. 2004, 2 (11): e309-10.1371/journal.pbio.0020309.View ArticlePubMedPubMed CentralGoogle Scholar
- Hai H, Leibman MN, Mural RJ: Biomedical informatics in transitional researc. 2008, Boston and London: Artech House, 1Google Scholar
- Knuth DE: Semantics of context-free languages. Theory of Computing Systems. 1968, 2 (2): 127-145.Google Scholar
- Unified Medical Language System® (UMLS®). [http://www.nlm.nih.gov/research/umls/]
- Zeng Q, Cimino JJ: Automated knowledge extraction from the UMLS. Proc Amia Symp. 1998, 568-572.Google Scholar
- Chen ES, Hripcsak G, Xu H, Markatou M, Friedman C: Automated acquisition of disease drug knowledge from biomedical and clinical documents: an initial study. J Am Med Inform Assoc. 2008, 15 (1): 87-98.View ArticlePubMedPubMed CentralGoogle Scholar
- Carter JD, Ladhani A, Ricca LR, Valeriano J, Vasey FB: A safety assessment of tumor necrosis factor antagonists during pregnancy: a review of the food and drug administration database. J Rheumatol. 2009, 36 (3): 635-641. 10.3899/jrheum.080545.View ArticlePubMedGoogle Scholar
- Statistica 64. 2009, United States: Statsoft Inc, 9.0Google Scholar
- Omniviz. 2009, Cambridge, United Kingdom: Biowisdom, 6Google Scholar
- Manning CD, Schutze H: Foundations of statistical natural language processing. 1999, London, England; Cambridge, Massachusetts: The MIT PressGoogle Scholar
- Hosmer D, Lemeshow S: Applied logistic regression. 1989, New York: WileyGoogle Scholar
- Fix E, Hodges JL: Discriminatory analysis: non-parametric discrimination: consistency properties. 1951, Texas: USAF School of Aviation Medicine, Randolph Field, 261-279.Google Scholar
- Vapnik VN: The nature of statistical learning theory. 1995, New York: SpringerView ArticleGoogle Scholar
- Burges CJC: A tutorial on support vector machines for pattern recognition. Data Min Knowl Disc. 1998, 2 (2): 127-167.View ArticleGoogle Scholar
- RapidMiner. 2009, Dortmund, Germany: Rapid-I, 5.0Google Scholar
- Sun A, Lim E-P, Liu Y: On strategies for imbalanced text classification using SVM: a comparative study. Decis Support Syst. 2009, 48 (1): 191-201. 10.1016/j.dss.2009.07.011.View ArticleGoogle Scholar
- Kennard RW, Stone L: Computer aided design of experiments. Technometrics. 1969, 11: 137-148. 10.2307/1266770.View ArticleGoogle Scholar
- Fawcett T: An introduction to ROC analysis. Pattern Recogn Lett. 2006, 27 (8): 861-874. 10.1016/j.patrec.2005.10.010.View ArticleGoogle Scholar
- Lasko TA, Bhagwat JG, Zou KH, Ohno-Machado L: The use of receiver operating characteristic curves in biomedical informatics. J Biomed Inform. 2005, 38 (5): 404-415. 10.1016/j.jbi.2005.02.008.View ArticlePubMedGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/12/13/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.