Table 6 Performance characteristics of methods in detection of actionable reports without and with oversampling of positive samples in the training data
Method | Use of order information | Oversampling | AUPRC | Average precision | AUROC | F1 score |
|---|---|---|---|---|---|---|
LR | ( −) | ( −) | 0.4574 | 0.4224 | 0.9351 | 0.1052 |
( +) | 0.3166 | 0.3167 | 0.8036 | 0.0474 | ||
( +) | ( −) | 0.4218 | 0.4580 | 0.8992 | 0.0763 | |
( +) | 0.4214 | 0.4221 | 0.9277 | 0.1089 | ||
GBDT | ( −) | ( −) | 0.4813 | 0.4816 | 0.8986 | 0.0975 |
( +) | 0.4854 | 0.4858 | 0.9335 | 0.0841 | ||
( +) | ( −) | 0.4767 | 0.4771 | 0.9133 | 0.0699 | |
( +) | 0.4874 | 0.4878 | 0.9307 | 0.0920 | ||
LSTM | ( −) | ( −) | 0.4617 | 0.4620 | 0.9331 | 0.0916 |
( +) | 0.4188 | 0.4194 | 0.9262 | 0.0818 | ||
( +) | ( −) | 0.4265 | 0.4272 | 0.9277 | 0.0797 | |
( +) | 0.4086 | 0.4066 | 0.9255 | 0.0795 | ||
BERT | ( −) | ( −) | 0.5138 | 0.5142 | 0.9516 | 0.1030 |
( +) | 0.4256 | 0.4273 | 0.9464 | 0.1190 | ||
( +) | ( −) | 0.5153 | 0.5157 | 0.9497 | 0.1183 | |
( +) | 0.4549 | 0.4559 | 0.9441 | 0.0953 |