Table 2 Overall performance of queries in the 361-patient manually abstracted cohort
Image annotation | ||||
|---|---|---|---|---|
Image type: any relevant image | Gold standard manual abstraction | |||
Relevant study | Not relevant study | |||
Automated Lucene tool result | Relevant study | 2548 | 133 | |
Not relevant study | 463 | 47 | ||
Sensitivityβ (95% CI) | Specificityγ (95% CI) | PPVδ (95% CI) | F1 scoreε | |
85% (83–86%) | 26% (20–33%) | 95% (94–96%) | 0.90 | |
Indication: surveillance | Gold standard manual abstraction | |||
Surveillance study | Not surveillance study | |||
Automated Lucene tool result | Surveillance study | 690 | 292 | |
Not surveillance study | 264 | 1763 | ||
Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) | F1 score | |
72% (69–75%) | 86% (84–87%) | 70% (67–73%) | 0.71 | |
Finding: suspicious | Gold standard manual abstraction | |||
Suspicious finding on study | No suspicious finding on study | |||
Automated Lucene tool result | Suspicious finding on study | 247 | 755 | |
No suspicious finding on study | 84 | 1922 | ||
Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) | F1 score | |
75% (70–79%) | 72% (70–73%) | 25% (22–27%) | 0.37 | |
Finding: recurrence only | Gold standard manual abstraction | |||
Recurrence on study | No recurrence on study | |||
Automated Lucene tool result | Recurrence on study | 105 | 353 | |
No recurrence on study | 19 | 2531 | ||
Sensitivity (95% CI) | Specificity (95% CI) | PPV (95% CI) | F1 score | |
85% (77–91%) | 88% (87–89%) | 23% (19–27%) | 0.36 | |