These findings generally supported our redesigned prototype in multiple dimensions, but were mixed. Design enhancements to the VHA's existing CRC screening clinical reminder positively impacted PCPs' workflow integration and usability ratings in terms of finding the patient's relevant data, as well as in providing helpful patient education materials. However, the redesigned prototype showed no difference in terms of perceived workload, as measured by the NASA TLX. Also, the improvements demonstrated by the usability survey were related only to the three questions specific to CRC screening design enhancements. Of these three, only the question about patient education reflected a functionality that was available in the redesigned prototype, but not in the existing reminder. Both the redesigned and current CRC clinical screening reminder may inform the patient's cancer screening history and status. The specificity of these appended items to the CSUQ regarding CRC screening was likely main reason why the results differed from the validated CSUQ survey, since the three questions added were specifically targeted to assess the issues regarding CRC screening that our design changes were meant to address.
The more general usability statements on the CSUQ about simplicity, efficiency, learnability, error recovery, overall satisfaction, and other dimensions of usability, did not produce significantly improved ratings for the resigned prototype over the current design. Adding the new design features in the redesigned prototype corresponded to additional mouse clicks and mouse movement, as recorded by the Morae software, and likely contributed to the lack of significant improvement in general usability. In terms of usability, potential advantages of the new timeline visual and patient education resource may have been off-set by the additional steps needed to access these features. These findings suggest that, while participants were supportive of the design changes to include the timeline visual and patient education resource, there is still room to improve the overall usability of the CRC screening clinical reminder and how it is integrated within the EHR.
Specific participant verbalizations provide additional insight why general usability ratings for simplicity, efficiency, and other usability constructs, were not significantly improved with the redesigned prototype. For example, two participants directly commented that the timeline visual for CRC history should be integrated at the same level as the CRC reminder (i.e., without having to click on an additional button). One participant stated: "It would be better if you had it [the timeline] over here on the window itself [next to the reminder dialog] because otherwise you have to click back and forth and it's hard to remember. Especially which somebody with a complicated history and you got to go from one thing to another, that doesn't work. So why not put it over here, you know, same window."
The Workflow Integration Survey received significantly higher (better) ratings for the redesigned prototype than the current design for each of the four survey subscales (navigation, functionality, ease of use, and workload). Interestingly, the NASA TLX did not show significant differences for workload. This suggests that perhaps the Workflow Integration Survey may have been more sensitive to detect differences in workload than the NASA TLX. Alternatively, 'workload' may have represented a different construct in the two instruments. The items from the Workflow Integration Survey that comprise the "workload" subscale are items 4, 8, and 12 (see additional file 1: Workflow Integration Survey). In contrast, The NASA TLX has several items that measure specific dimensions of workload: mental demand, physical demand, temporal demand, performance, effort, and frustration. There are several differences between the "workload" construct in the Workflow Integration Survey and workload as measured by the NASA TLX. The NASA TLX includes specific constructs not covered by the Workload Integration Survey subscale for workload (e.g., frustration). Also, the Workload Integration Survey does not distinguish between mental effort and physical effort. These differences suggest the instruments measure different but related constructs and may explain the difference in results between the two.
Analysis of the qualitative data revealed broad support for both the timeline visual and patient education design features. In addition, participants offered feedback about potential enhancements that may further increase acceptance and usability of these features. For example, while six of the PCP's directly expressed a positive experience with the timeline visual, five PCPs were skeptical about the reliability and accuracy of the results in the timeline. This finding represents a lack of trust in the quality or authenticity of the underlying data; potential, related solutions may be to (1) increase the quality of the underlying data and (2) transparently provide information regarding the data's source. To increase trust in the data, we envision providing additional options in the visual timeline to increase the transparency of the data. In this way, providers can investigate, for example, the specific details of a patient's colonoscopy that was performed outside of the VA. These additional details may include the contact information for the facility and physician who performed the test. The additional details may also improve the overall quality of the data.
In the case of the patient education resource, PCPs generally found the resource to be helpful, but five PCPs expressed a preference for the patient education form to be available in their clinics as a pre-printed form rather than having to access and print it from the CDS as needed. One obvious issue with our redesigned prototype was the inability of all but one PCP to recognize that the specific results of abnormal tests (i.e., colonoscopy pathology reports) were available from the timeline by clicking on a green box (see Figure 3). Provider awareness could be increased by making the box resemble a button, or provide mouse-over functionality, and to provide surface descriptors (i.e., pathology) about what data resides in the next layer of information.
These results underscore the importance of iteration in design. Ideally, more than one laboratory simulation should be conducted prior to implementation. The next logical step in the design process should be to further improve the redesigned prototype, based on the results of this study, and then to repeat the experiment to determine if further improvements to usability and perceived workload are demonstrated. In our future work, we intend to pursue these tasks: building on our current design, conducting further laboratory testing of design changes, and ultimately testing in live clinical environments.