Table 2 The different pre-trained language models considered in the evaluation, their version in the HuggingFace repository and the type of pre-training
Pre-trained Language model Version | Corpus | Pre-training |
|---|---|---|
UMLSBert_ENG [25] | Pubmed + UMLS | Continual pretraining |
| Â | Â | + weight adjustement |
biobert-base-cased-v1.1 [26] | PubMEd | Continual pretraining |
bluebert_pubmed_uncased | Pubmed + MIMIC III notes | Continual pretraining |
_L-12_H-768_A-12 [27] | Â | Â |
scibert_scivocab_uncased [28] | Semantic Scholar | From scratch |
Bio_clinicalBERT [29] | MIMIC III notes | Continual pretraining |
BERT-base-uncased [30] | Wikipedia | From scratch |
BiomedNLP-PubMedBERT-base- | Pubmed | From scratch |
uncased-abstract-fulltext [31] | Â | Â |