This article has Open Peer Review reports available.
Improved de-identification of physician notes through integrative modeling of both public and private medical text
- Andrew J McMurry†1, 2, 3, 5Email author,
- Britt Fitch†1Email author,
- Guergana Savova2,
- Isaac S Kohane1, 2, 4 and
- Ben Y Reis1, 2
© McMurry et al.; licensee BioMed Central Ltd. 2013
Received: 2 November 2012
Accepted: 25 September 2013
Published: 2 October 2013
Physician notes routinely recorded during patient care represent a vast and underutilized resource for human disease studies on a population scale. Their use in research is primarily limited by the need to separate confidential patient information from clinical annotations, a process that is resource-intensive when performed manually. This study seeks to create an automated method for de-identifying physician notes that does not require large amounts of private information: in addition to training a model to recognize Protected Health Information (PHI) within private physician notes, we reverse the problem and train a model to recognize non-PHI words and phrases that appear in public medical texts.
Public and private medical text sources were analyzed to distinguish common medical words and phrases from Protected Health Information. Patient identifiers are generally nouns and numbers that appear infrequently in medical literature. To quantify this relationship, term frequencies and part of speech tags were compared between journal publications and physician notes. Standard medical concepts and phrases were then examined across ten medical dictionaries. Lists and rules were included from the US census database and previously published studies. In total, 28 features were used to train decision tree classifiers.
The model successfully recalled 98% of PHI tokens from 220 discharge summaries. Cost sensitive classification was used to weight recall over precision (98% F10 score, 76% F1 score). More than half of the false negatives were the word “of” appearing in a hospital name. All patient names, phone numbers, and home addresses were at least partially redacted. Medical concepts such as “elevated white blood cell count” were informative for de-identification. The results exceed the previously approved criteria established by four Institutional Review Boards.
The results indicate that distributional differences between private and public medical text can be used to accurately classify PHI. The data and algorithms reported here are made freely available for evaluation and improvement.
Physician's notes contain information that may never be recorded in a coded format in the patient health record [1–3], such as family history , smoking history [5–7], and descriptions of lab results [8, 9]. Nonetheless, the “uncoded” information buried in physician notes is so valuable that numerous attempts have been made towards indexing and sharing notes for research use. However, since physician notes can contain patient names, home addresses, social security numbers, and other types of Protected Health Information (PHI) , vast quantities of doctors’ notes have gone largely unused for medical research studies. Methods to simultaneously protect patient privacy and increase research utility are needed – as the number of electronic health record systems increases and with it the opportunity to study larger numbers of patients [11–13].
Existing methods for de-identifying medical texts range from simple rule-based systems to sophisticated machine learning algorithms [14, 15]. The majority of currently implemented methods are rule-based systems that match patterns and dictionaries of expressions that frequently contain PHI . The advantage of rule-based systems is that experts can quickly define rules and iteratively fine tune them to achieve higher accuracy. While rule-based systems have shown high recall in some settings , they often have the disadvantage of hard coding rules to a specific note format or physician writing style, resulting in poor performance in other contexts. Adjusting existing rule systems for use at other medical centers is often too costly, limiting broad use across institutions. This problem is well recognized [14, 15], and has prompted efforts using an alternative, machine learning approach. Rather than using the expert to author rules, the rules for PHI removal are “learned” by training an algorithm using human annotated examples (i.e. a supervised learning task). For example, competitors in the i2b2 de-identification challenge  were asked to train or tune their algorithms on one set of human annotated notes and then validate their best model on a separate set of annotated notes. Generally, the highest scoring algorithms used machine learning methods such as conditional random fields [17–20], decision trees , support vector machines [22, 23], and meta-classifiers that combine and weight different strategies for high-recall with false positive filtering .
The work reported here was trained and validated on the same i2b2 challenge datasets, which allows for comparison to prior work. Our algorithm performed favorably with regards to recall, albeit with lower precision (see results). The primary difference between our method and other top scores in the i2b2 challenge is the extensive use of publicly available medical texts to learn the distributional characteristics of individual PHI tokens that could appear in any type of physician note. This is in contrast to other models that use features that may be specific to the style and format of discharge summaries such as section headings , sentence position , and longer token sequences [17–20]. We show that publicly available medical texts provide an informative background distribution of sharable medical words, a property that is largely underutilized in patient privacy research.
Instead of trying to recognize PHI words in physician notes, we reversed the problem towards recognizing non-PHI words. We asked, “what are the chances that a word or phrase would appear in a medical journal or medical dictionary? What are the lexical properties of PHI words? To what extent can we use publicly available data to recognize data that is private and confidential?”
While human annotated datasets of PHI are few in number and difficult to obtain, examples of public medical text are broadly available and generally underutilized for de-identification. By definition, medical journal publications provide the distributional evidence for words that are not PHI. Of course, some medical words will end up being proper names but the public corpora provide a heuristic measure of likelihood that we exploit as described below. In this context, relatively fewer human annotated examples are treated as approximations of the distributional properties of PHI. Lexical comparisons between PHI words and non-PHI words reveal that PHI words are generally nouns and numbers – whereas verbs and adjectives are probably ok to share -- especially medically relevant verbs and adjectives that are of more relevant to research studies. Publicly available lists of suspicious words and expert rules are also incorporated into this algorithm, such as US census data and the Beckwith regex list . We combine the discrimination power of these complementary perspectives to achieve improved de-identification performance. As an additional safeguard, notes can be indexed  and later searched using coded medical concepts, thereby reducing the number of full-text reports that need to be shared in early phases of research .
Different types of PHI
Types of PHI and their risk to patient confidentiality
We anticipated that each type of PHI would have a unique set of association rules. For example, patient names are nouns whereas medical record numbers are numbers. Learning different association rules  for each type of PHI has the added benefit that additional weight can be placed on highest risk elements, such as the patient name or home address. All PHI types are generally represented as nouns and numbers with low term frequencies, low occurrence in medical controlled vocabularies, and non-zero regular expression matches of some type. Non-PHI words generally have higher term frequencies, higher occurrence in medical vocabularies and near zero matches in regular expressions of any type.
Feature set construction
Apache cTAKES and Apache UIMA provide the foundation for the Scrubber pipeline. The data processing pipeline is provided by Apache UIMA project , an engineering framework commonly used in Natural Language Processing . Of note, UIMA does not provide any pre-built components for text processing, it provides the main “scaffolding” and flow between user developed components. In the lexical phase, Apache cTAKES splits each document into sentences  and determines the part of speech for each token. Apache cTAKES is especially appropriate because it has been extensively trained on medical documents . In the term frequency phase, the count of each token is retrieved from a corpus of open access medical publications previously annotated by Apache cTAKES. In the dictionary phase, each fragment is compared against phrases in publicly available sources, such as ICD9 diagnoses and LOINC laboratory concepts.
Complete list of all 28 features annotated by the NLP pipeline
Part of Speech
Term Frequency (Token)
# matches HL7 2.5
# matches US Census Names
Part of Speech (Binned)
Term Frequency (Token, Part of Speech)
# matches HL7 3.0
# matches ICD9 CM
# matches for pattern HOSPITAL
Word or Number
# matches ICD10 CM
# matches for pattern AGE
# matches ICD10 PCS
# matches for pattern DATE
# matches LOINC
# matches for pattern DOCTOR
# matches MESH
# matches for pattern LOCATION
# matches RXNORM
# matches for pattern PATIENT
# matches SNOMED
# matches for pattern ID
# matches COSTAR
# matches for pattern PHONE
# consectutive tokens any dictionary
# consecutive tokens any pattern
The feature set is then processed through Weka  using a J48 decision tree  classification algorithm, a popular open source implementation of the C4.5 decision tree algorithm. J48 was chosen for several reasons. First, decision trees do not require “binning” value ranges to be effective . This was useful because the correct value ranges were not known prior to classifier training. Second, decision trees can build a model for multiple class types. This is important because different types of PHI have different rules associated with them. For example, patient names are nouns whereas medical record numbers are numbers. A binary classifier would ignore these characteristic differences across PHI types and likely cause more errors.
The primary data used for training and testing was the I2B2 de-id challenge data . This data consists of 669 training cases and 220 testing cases. The cases are a fully annotated gold standard set of discharge summaries. To calculate frequencies of word occurrences, we randomly selected 10,000 publicly available peer reviewed medical publications. This was necessary as many valid word tokens appear only once or not at all in any random selection of physician notes. Using more than 10,000 publications for training did not alter performance, and was computationally feasible using inexpensive commodity hardware.
On average there were 520 words (tokens) per case, and an average of 39 PHI words per case. As expected, most word tokens were not patient identifiers (PHI) -- the ratio of PHI words to non-PHI words was 1:15. Training a classifier using all of the available training instances would highly favor non-PHI classifications . To address this issue, the training set was compiled using all of the PHI words and an equally sized random selection of non-PHI words.
The training model was applied to an independent validation corpus of 220 discharge summaries from the i2b2 de-id challenge. The goal of this method is to produce a result meeting stated IRB requirements of high recall at the cost of precision. This method achieved its goal of very high recall (98%) and F10 (98%) albeit at the cost of a lower precision (62%) and F1 (76%). Compared to the seven participant groups in the original i2b2 de-identification challenge the scores reported in this paper would have placed first overall in recall and last overall in precision. The low precision is expected for several reasons. First, we did not tailor dictionaries and patterns for the i2b2 corpus and elected for an out-of-the-box approach which is more likely to mimic an initial deployment scenario. Secondly, our IRB stated that the most important metric for patient privacy is recall. In this context, recall is the percentage of confidential tokens that were removed relative to confidential tokens that were missed (Equation 1). Precision is the percentage of confidential tokens that were removed relative to the number of public tokens that were erroneously removed (Equation 2). The automated performance matches or exceeds that of two human evaluators  and preserves the readability of the original text .
The words “of”, “and”, “home”, “hospital”, and “services” were overwhelmingly the most commonly missed PHI words. These common words account for 124 of 173 partial misses, and pose little to no risk to patient privacy.
We performed a manual review of each misclassification and determined that no unique identifiers were left fully intact. Partial redactions – such as properly removing the patient last name but missing the patient first name were rare (13 word tokens in 12 cases). Lower risk identifiers such as hospital name and date of treatment were also rare. Only two dates and 2 hospital names were left fully intact.
Every type of PHI is a noun or number (Figure 4). Interestingly, this fact alone yielded 96% recall (Figure 3). However, many naturally occurring words and medically relevant concepts can also appear as nouns and numbers. To distinguish PHI from nouns and numbers that are naturally occurring, a term frequency calculation was applied. Similarly, nouns and numbers with medical relevance were distinguished by their presence in one or more medical vocabularies.
Medical publications do not refer to individually named patients. Even in medical case studies, the patient name, home address, phone number, and medical record number must be withheld in accordance with law. This guarantees that all high-risk PHI elements in Table 1 will not be present in the publication dataset. It was therefore not surprising to find that patient specific identifiers were not frequently reported in the text of medical publications. As a result, classification of PHI using only term frequency features and part of speech yielded high scrubbing performance with 92% recall.
Ten vocabularies in the Unified Medical Language System were selected in order to span a very wide range of demographic terms, diagnoses, lab tests, medication names, and procedures. Surprisingly, a decision tree trained only to distinguish PHI from medical concepts yielded very high recall (94%), albeit with poor precision (12%). This suggests that there is almost no overlap between medical concepts and patient identifiers. These findings provide evidence that automatic retrieval of coded medical concepts (autocoding) is also useful for de-identification. In this way, parallel autocoding and de-identification provides maximum research utility while minimizing the risk of patient disclosure (Additional file 1: Table S2).
Regular Expressions yielded the most balanced ratio of recall (80%) to precision (60%) of any feature group tested in isolation (Figure 3). This matches our experience using a previous version of the HMS Scrubber in new medical center settings without customization and without inspecting the pathology report header . We expected the regular expressions to outperform all other feature groups with respect to dates, phone numbers, and ages but this was not the case. This either means that we used Beckwith’s regular expression rules incorrectly or there are more ways to express these simple concepts than one might expect. Nevertheless, regular expressions slightly improved the overall classification specificity. The only changes to Beckwith’s regular expressions was the addition of one pattern for date, two for hospital names, and three for IDs.
Quantifying the distance between public and private medical texts
Open access medical journals provide a heuristic measure of the words and phrases used to describe medical topics. Estimating the distributions of non-PHI tokens is therefore informative for recognizing PHI. To quantify this relationship, a vector space model [44–46] was created for journal publications and the i2b2 datasets. First, each dataset was annotated as described in “feature set construction”. Second, each numeric feature was normalized in the range between zero and one and mean subtracted. Third, the principal components were calculated for each part of speech (using defaults of Matlab pca method). Principal components were selected if they had >1 explained variance for a total explained variance of 99%. Fourth, the principal components were then compared in vector space by measuring the dot product (Equation 4).
The i2b2 Challenge Data includes surrogate names that were constructed by permuting the syllables of real names in the US Census. This means that the provided names of patients, doctors, and hospitals are highly unlikely to be realistic examples, which could give an unnatural advantage to term frequency calculations and thus artificially improve classifier performance.
To verify that this was not the case, the i2b2 surrogate names were replaced with real names from the Medicare General Hospital Information file  and the US patent office list of US inventors . Each hospital name in the i2b2 training and test datasets was replaced with a hospital name from Medicare. Each patient and doctor name in the i2b2 data was replaced with a randomly selected name from the list of US inventors. In total, Medicare provided 4838 unique hospital names and the USPTO provided 4301229 inventors (1473329 unique).
Classifier validation results
Boosted + FP filtering
Can we use vast quantities of public medical text to de-identify confidential information within private physician notes? Can we accelerate the rate of sharing physician notes for research without compromising patient confidentiality? Can we achieve these goals while respecting the challenges and responsibilities among hospital privacy boards? These questions motivated the authors to compare public and private medical texts to learn the distributions and lexical properties of Protected Health Information. The results of this experiment show that publicly available medical texts are highly informative for PHI recognition, resulting in performance that is likely to be approved for research use among by hospital review boards. The vast majority of misclassifications were common words appearing in hospital names, which pose minimal risk to patient privacy. A useful byproduct of this de-identification process is that coded medical concepts  are also stored for later search  and retrieval . This approach to de-identification both reduces unauthorized disclosures and increases authorized use , a position previously confirmed by numerous hospital privacy boards [16, 26, 50].
Comparing public and private text sources reveals interesting properties of PHI. Words in physician notes that frequently appear in medical journal publications and concept dictionaries are highly unlikely to contain PHI. Conversely, words in physician notes that are nouns and numbers are more likely to contain PHI. It is interesting to speculate just how far publicly available text can be leveraged for de-identification tasks, and we encourage other researchers to use our annotated datasets and open source software for use in their own medical studies.
In a state of the art review of de-identification, Ozuner and Szolovits appropriately ask “how good is good enough? ” In this study, we sought to achieve performance levels that were already considered satisfactory by hospital privacy boards  with minimal investment. Numerous tradeoffs were made to achieve this goal. First, recall was strongly favored over precision, especially for patient names and ID numbers that have highest risk of disclosure. Second, we favored default configuration over hospital-specific human refinement. In our experience, site-specific modification of patient names lists and regular expressions can be laborious and can lead to “overscrubbing” information that is valuable for research. Third, we needed the algorithm to run on a single computer using commodity hardware, both to satisfy IRB concerns over data-duplication and reuse hardware already in place. Fourth, we wanted to make as few assumptions as possible about the training set to avoid unnecessary overfitting.
There were several limitations to this study. Term frequency calculations were performed for single word tokens. Increasing the term frequency to use two or more words might improve patient name recognition. For example, patients are more likely to have a first or last name in common with an author than a full name. Similarly, patient home addresses are highly unlikely to be found in published medical journals. However, common and rare word sequences can vary considerably across the different types and formats of physician notes and journal publications. We chose instead to err on the side of caution and use a single token model rather than ngrams or conditional random fields.
There is also the potential that we too have overfit our model to training examples and were fortunate enough to have the model validated in an independent sample. There are several cases where classifying PHI in new physician notes could be significantly less accurate. PHI words and phrases that frequently appear in medical publications and dictionaries are the most difficult to classify, although the number of times this occurs appears negligible. Incoherently written physician notes may be difficult to tag for part of speech, which would likely degrade classifier accuracy. Datasets that have different probability distributions and term frequency could also pose problems. In each of these potentially limiting examples, a new corpus would have to be characteristically different from the testing and training examples studied here.
We recommend that this de-identification method be used according to procedures that were previously acknowledged by four hospital IRBs [16, 26]. The recommended workflow is as follows. Physician notes are de-identified and autocoded such that the scrubbed report is saved in a secured database and searchable according to medical vocabularies. Search access is limited to authorized investigators affiliated with the institution hosting the data, and under no circumstances should the textual data be made available for public download. Searching for patient cohorts matching study criteria occurs in an anonymized manner, meaning that only counts are returned with the first level of access. After finding a cohort of interest, an investigator may apply for access to review the deidentified cases. By increasing the level of access commensurate with the needs of a study , the risk to patient disclosure is minimized while allowing many investigators the ability to query and browse the valuable collection medical notes. The methods proposed here can be put to practical use today to help unlock the tremendous research potential of vast quantities of free-text physician notes accumulating in electronic medical record systems worldwide .
The authors would like to acknowledge the National Cancer Institute for funding and evaluating this work for use in other medical centers.
We would especially like to thank Andy Glass, Sheila Taube, Pierre Lechance for their contributions and support. The authors would also like to thank Bruce Beckwith for insights and commentary in the authorship of this report. We would also like to thank the cTAKES team for providing extensible language processing tools.
Deidentified clinical records used in this research were provided by the i2b2 National Center for Biomedical Computing funded by U54LM008748 and were originally prepared for the Shared Tasks for Challenges in NLP for Clinical Data organized by Dr. Ozlem Uzuner, i2b2 and SUNY.
Editors: please note that Britt Fitch and Andrew McMurry are sharing first authorship of this report.
- Uzuner O, Solti I, Cadag E: Extracting medication information from clinical text. J Am Med Inform Assoc. 2010, 17 (5): 514-518. 10.1136/jamia.2010.003947.View ArticlePubMedPubMed CentralGoogle Scholar
- Uzuner O: Recognizing obesity and comorbidities in sparse data. J Am Med Inform Assoc. 2009, 16 (4): 561-570. 10.1197/jamia.M3115.View ArticlePubMedPubMed CentralGoogle Scholar
- Liao KP, et al: Electronic medical records for discovery research in rheumatoid arthritis. Arthritis Care Res. 2010, 62 (8): 1120-1127. 10.1002/acr.20184.View ArticleGoogle Scholar
- Goryachev S, Kim H, Zeng-Treitler Q: Identification and extraction of family history information from clinical reports. AMIA … Annual Symposium proceedings/AMIA Symposium. Proc AMIA Symp. 2008, 2008: 247-251.PubMed CentralGoogle Scholar
- Zeng QT, et al: Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system. BMC Med Inform Decis Mak. 2006, 6: 30-10.1186/1472-6947-6-30.View ArticlePubMedPubMed CentralGoogle Scholar
- Savova GK, et al: Mayo clinic NLP system for patient smoking status identification. J Am Med Inform Assoc. 2008, 15 (1): 25-28.View ArticlePubMedPubMed CentralGoogle Scholar
- Heinze DT, et al: Medical i2b2 NLP smoking challenge: the A-Life system architecture and methodology. J Am Med Inform Assoc. 2008, 15 (1): 40-43.View ArticlePubMedPubMed CentralGoogle Scholar
- Patel AA, et al: Availability and quality of paraffin blocks identified in pathology archives: a multi-institutional study by the Shared Pathology Informatics Network (SPIN). BMC Cancer. 2007, 7: 37-10.1186/1471-2407-7-37.View ArticlePubMedPubMed CentralGoogle Scholar
- Hoshida Y, et al: Gene expression in fixed tissues and outcome in hepatocellular carcinoma. N Engl J Med. 2008, 359 (19): 1995-2004. 10.1056/NEJMoa0804525.View ArticlePubMedPubMed CentralGoogle Scholar
- Services, U.S.D.o.H.H: Health Information Portability and Accountability act. 1996, Available from: http://www.hhs.gov/ocr/privacy/ Google Scholar
- Kohane IS, Churchill SE, Murphy SN: A translational engine at the national scale: informatics for integrating biology and the bedside. J Am Med Inform Assoc. 2012, 19 (2): 181-185. 10.1136/amiajnl-2011-000492.View ArticlePubMedGoogle Scholar
- Kohane IS: Using electronic health records to drive discovery in disease genomics. Nat Rev Genet. 2011, 12 (6): 417-428. 10.1038/nrg2999.View ArticlePubMedGoogle Scholar
- McMurry AJ, et al: SHRINE: enabling nationally scalable multi-site disease studies. PLoS One. 2013, 8 (3): e55811-10.1371/journal.pone.0055811.View ArticlePubMedPubMed CentralGoogle Scholar
- Uzuner O, Luo Y, Szolovits P: Evaluating the state-of-the-art in automatic de-identification. J Am Med Inform Assoc. 2007, 14 (5): 550-563. 10.1197/jamia.M2444.View ArticlePubMedPubMed CentralGoogle Scholar
- Meystre SM, et al: Automatic de-identification of textual documents in the electronic health record: a review of recent research. BMC Med Res Methodol. 2010, 10: 70-10.1186/1471-2288-10-70.View ArticlePubMedPubMed CentralGoogle Scholar
- Beckwith BA, et al: Development and evaluation of an open source software tool for deidentification of pathology reports. BMC Med Inform Decis Mak. 2006, 6: 12-10.1186/1472-6947-6-12.View ArticlePubMedPubMed CentralGoogle Scholar
- Wellner B, et al: Rapidly retargetable approaches to de-identification in medical records. J Am Med Inform Assoc. 2007, 14 (5): 564-573. 10.1197/jamia.M2435.View ArticlePubMedPubMed CentralGoogle Scholar
- Aramaki EIT, Miyo K, Ohe K: Automatic Deidentification by using Sentence Features and Label Consistency. in i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data. 2006, Washington, DC: i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data, 10-11.Google Scholar
- Aberdeen J, et al: The MITRE Identification Scrubber Toolkit: design, training, and assessment. Int J Med Inform. 2010, 79 (12): 849-859. 10.1016/j.ijmedinf.2010.09.007.View ArticlePubMedGoogle Scholar
- Lafferty J: Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, in Proceedings of the 18th International Conference on Machine Learning 2001 (ICML 2001). 2001, San Francisco, CA: Morgan Kaufmann Publishers Inc., 282-289.Google Scholar
- Szarvas G, Farkas R, Busa-Fekete R: State-of-the-art anonymization of medical records using an iterative machine learning framework. J Am Med Inform Assoc. 2007, 14 (5): 574-580.View ArticlePubMedPubMed CentralGoogle Scholar
- Uzuner O, et al: A de-identifier for medical discharge summaries. Artif Intell Med. 2008, 42 (1): 13-35. 10.1016/j.artmed.2007.10.001.View ArticlePubMedGoogle Scholar
- Hara K: Applying a SVM Based Chunker and a Text Classifier to the Deid Challenge. in i2b2 Workshop on Challenges in Natural Language Processing for Clinical Data. 2007, Washington, DC: i2b2 Workshop on Challenges in Natural Language Processing for Clinical DataGoogle Scholar
- Ferrandez O, et al: BoB, a best-of-breed automated text de-identification system for VHA clinical documents. J Am Med Inform Assoc. 2013, 20 (1): 77-83. 10.1136/amiajnl-2012-001020.View ArticlePubMedGoogle Scholar
- Berman JJ: Doublet method for very fast autocoding. BMC Med Inform Decis Mak. 2004, 4: 16-10.1186/1472-6947-4-16.View ArticlePubMedPubMed CentralGoogle Scholar
- McMurry AJ, et al: A self-scaling, distributed information architecture for public health, research, and clinical care. Journal of the American Medical Informatics Association: JAMIA. 2007, 14 (4): 527-533. 10.1197/jamia.M2371.View ArticlePubMedPubMed CentralGoogle Scholar
- U.S. Department of Health and Human Services, N: De-identifying Protected Health Information Under the Privacy Rule. 2007, 2/2/2007 [cited 2012 4/3/2012]; Available from: http://privacyruleandresearch.nih.gov/pr_08.asp Google Scholar
- Allwein EL, Schapire RE, Singer Y: Reducing multiclass to binary: a unifying approach for margin classifiers. J Mach Learn Res. 2001, 1: 113-141.Google Scholar
- Savova GK, et al: Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc. 2010, 17 (5): 507-513. 10.1136/jamia.2009.001560.View ArticlePubMedPubMed CentralGoogle Scholar
- Bodenreider O: The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004, 32 (Database issue): D267-270.View ArticlePubMedPubMed CentralGoogle Scholar
- Bureau, U.S.C: Frequently Occurring First Names and Surnames From the 1990 Census. 1990, Available from: http://www.census.gov/genealogy/names/ Google Scholar
- David Ferrucci AL: UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat Lang Eng. 2004, 10 (3–4): 327-348.View ArticleGoogle Scholar
- Nadkarni PM, Ohno-Machado L, Chapman WW: Natural language processing: an introduction. J Am Med Inform Assoc. 2011, 18 (5): 544-551. 10.1136/amiajnl-2011-000464.View ArticlePubMedPubMed CentralGoogle Scholar
- Zhang T: Updating an NLP System to Fit New Domains: an empirical study on the sentence segmentation problem, IBM T.J. 2003, Stroudsburg, PA: Proceeding CONLL '03 Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, 56-62.Google Scholar
- Reshef DN, et al: Detecting novel associations in large data sets. Sci. 2011, 334 (6062): 1518-1524. 10.1126/science.1205438.View ArticleGoogle Scholar
- Lin F, Cohen WW: A Very Fast Method for Clustering Big Text Datasets, in Proceedings of the 2010 conference on ECAI 2010: 19th European Conference on Artificial Intelligence. 2010, Amsterdam, The Netherlands: IOS Press, 303-308.Google Scholar
- Frey BJ, Dueck D: Clustering by passing messages between data points. Sci. 2007, 315 (5814): 972-976. 10.1126/science.1136800.View ArticleGoogle Scholar
- Dhillon IS, Guan Y: Information Theoretic Clustering of Sparse Co-Occurrence Data, in Proceedings of the Third IEEE International Conference on Data Mining 2003. Proc IEEE Comput Soc Bioinform Conf. 2003, 517-Google Scholar
- Mark Hall EF, Geoffrey H, Bernhard P, Peter R, Ian H: Witten, The WEKA data mining software: an update. SIGKDD Explor. 2009, 11 (1): 10-18. 10.1145/1656274.1656278.View ArticleGoogle Scholar
- Quinlan JR: C4.5: programs for machine learning. 1993, San Francisco, CA: Morgan KaufmannGoogle Scholar
- Ying Yang GW: Proportional k-Interval Discretization for Naive-Bayes Classifiers. ECML01: 12th European Conference on Machine Learning. 2001, Berlin, Heidelberg: Springer-Verlag, 564-575.Google Scholar
- Chen Y: Learning Classifiers from Imbalanced, Only Positive and Unlabeled Data Sets. CS573 Project, (2009). 2009, Ames, IA: Department of Computer Science Iowa State UniversityGoogle Scholar
- Neamatullah I, et al: Automated de-identification of free-text medical records. BMC Med Inform Decis Mak. 2008, 8: 32-10.1186/1472-6947-8-32.View ArticlePubMedPubMed CentralGoogle Scholar
- Liangcai S, et al: Efficient SPectrAl Neighborhood blocking for entity resolution. in Data Engineering (ICDE), 2011 IEEE 27th International Conference on. 2011, Washington, DC: IEEE Computer SocietyGoogle Scholar
- Hao Z, et al: SVM-KNN: Discriminative Nearest Neighbor Classification for Visual Category Recognition. in Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on. 2006, Washington, DC: IEEE Computer SocietyGoogle Scholar
- Toussaint G: Proximity Graphs for Nearest Neighbor Decision Rules: Recent Progress. in Proceedings of the 34th Symposium on the INTERFACE. 2002, Quebec, Canada: School of Computer Science McGill University MontrealGoogle Scholar
- Services, C.f.M.M: Official Hospital Compare Data. 2013, Available from: https://data.medicare.gov/data/hospital-compare Google Scholar
- Research, N.B.o.E: Individual inventor records. 1999, Available from: http://www.nber.org/patents/ainventor.zip Google Scholar
- Wu ST, et al: Unified Medical Language System term occurrences in clinical notes: a large-scale corpus analysis. J Am Med Inform Assoc. 2012, 19 (e1): e149-e156. 10.1136/amiajnl-2011-000744.View ArticlePubMedPubMed CentralGoogle Scholar
- Drake TA, et al: A system for sharing routine surgical pathology specimens across institutions: the Shared Pathology Informatics Network. Hum Pathol. 2007, 38 (8): 1212-1225. 10.1016/j.humpath.2007.01.007.View ArticlePubMedGoogle Scholar
- Clifton C, et al: Tools for privacy preserving distributed data mining. SIGKDD Explor Newsl. 2002, 4 (2): 28-34. 10.1145/772862.772867.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/13/112/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.