Pre-trained Language model Version | Corpus | Pre-training |
---|---|---|
UMLSBert_ENG [25] | Pubmed + UMLS | Continual pretraining |
 |  | + weight adjustement |
biobert-base-cased-v1.1 [26] | PubMEd | Continual pretraining |
bluebert_pubmed_uncased | Pubmed + MIMIC III notes | Continual pretraining |
_L-12_H-768_A-12 [27] | Â | Â |
scibert_scivocab_uncased [28] | Semantic Scholar | From scratch |
Bio_clinicalBERT [29] | MIMIC III notes | Continual pretraining |
BERT-base-uncased [30] | Wikipedia | From scratch |
BiomedNLP-PubMedBERT-base- | Pubmed | From scratch |
uncased-abstract-fulltext [31] | Â | Â |