Fig. 3

The results of the system by the threshold of confidence rate

(a) Precision, recall, and F-measure do not vary with the expertise of the question authors. Advanced group: Pharmacists with advanced knowledge and experience of Kampo medicines; Basic group: Pharmacists with basic knowledge and experience of Kampo medicines

(b) The system was able to provide the correct answer for 90.3% (89.4% for true positive plus 0.9% for true negative) of the validation questions. 〇 means the system answers were expected. △ means the system answers were “I do not have an answer.” × means the system answers were unexpected. TP: True positive; FN: False negative; TN: True negative; FN: False negative

(c) Number of questions with correct answers listed in the system by the threshold of confidence rate of why the question did not lead to the correct answer. The main reason why questions did not lead to the correct answer even though the correct answer was listed in the system is the Misclassification of Category-sorting NLC® and Category NLC®