Table 4 Performance Comparison between 525 Human Manual Rating and Deep Learning Model. Manual performance 526 is reported as percent agreement. Automated performance is reported as Implementation Accuracy (see Table 3).
Question | Manual Performance | Automated Performance | ||
|---|---|---|---|---|
| Â | DISCERN | HONcode | HEA BioBERT | |
| Â | 2 raters | 3 raters | 80% coverage | 100% coverage |
Q4: References (HoN: Reference) | 96% | 89% | 87% | 84% |
Q5: Date (HoN: Date) | 88% | 80% | 87% | 83% |
Q9: How Treatment Works | 92% | Â | 82% | 78% |
Q10: Treatment Benefits | 95% | Â | 83% | 77% |
Q11: Tt. Risks (HoN: Justifiability) | 97% | 74% | 91% | 81% |
average | 94% | 81% | 86% | 81% |