Table 4 Performance Comparison between 525 Human Manual Rating and Deep Learning Model. Manual performance 526 is reported as percent agreement. Automated performance is reported as Implementation Accuracy (see Table 3).
| Question | Manual Performance | Automated Performance | ||
|---|---|---|---|---|
| DISCERN | HONcode | HEA BioBERT | ||
| 2 raters | 3 raters | 80% coverage | 100% coverage | |
| Q4: References (HoN: Reference) | 96% | 89% | 87% | 84% |
| Q5: Date (HoN: Date) | 88% | 80% | 87% | 83% |
| Q9: How Treatment Works | 92% | 82% | 78% | |
| Q10: Treatment Benefits | 95% | 83% | 77% | |
| Q11: Tt. Risks (HoN: Justifiability) | 97% | 74% | 91% | 81% |
| average | 94% | 81% | 86% | 81% |