Skip to main content

Table 3 Evaluation results of Text-to-UMLS linking on validation and testing data from MIMIC-III discharge summaries

From: Ontology-driven and weakly supervised rare disease identification from clinical notes

 

validation (n=142+/400)

test (n=187+/673)

test, seen in WS (n=80+/499) i.e. both rules [not] satisfied

test, unseen in WS (n=107+/174) i.e. only one rule satisfied

Text to UMLS

P

R

\(F_{1}\)

P

R

\(F_{1}\)

P

R

\(F_1\)

P

R

\(F_1\)

SemEHR [15]

35.5

100.0

52.4

27.8

100.0

43.5

16.0

100.0

27.6

61.5

100.0

76.2

+ rules

80.9

89.4

84.9

68.6

94.7

79.6

83.3

87.5

85.4

61.5

100.0

76.2

+ WS (rules+BERT)

92.0

89.4

90.7

81.4

91.4

86.1

83.3

87.5

85.4

80.2

94.4

86.7

+ SS (anns+BERT)

-

-

-

88.4

93.6

90.9

87.7

88.8

88.2

88.9

97.2

92.9

  1. The column statistics (n=\(N_+\)+/N) show the number of positive data \(N_+\) and all samples N in the dataset. SemEHR has a perfect reference recall, because all candidate mention-UMLS pairs were created using the tool. WS, weak supervision; SS, strong supervision. BlueBERT-base (PubMed+MIMIC-III) was used as the BERT model. The best scores, either or not considering strong supervision (SS), are bolded