The findings of this study showed that most features of our prototype Q&A network tool were usable, although some of them could be improved. The majority of the tasks were executed effectively, whereas task efficiency and satisfaction varied. Participants helped to identify various possibilities for improvement, including features such as the process of asking a question, searching for an answer by category, obtaining technical help from the moderator or help function and expert registration. As a result, in the revised version of ArboAntwoord, launched in October 2008, we limited the amount of scrolling and added tracking (steps) to the questioning process, allowed questioners to select experts themselves, redefined (sub)categories, moved the hyperlink to the help function in the header, registered experts ourselves and presented the rules for expert participation.
The results of our study further suggested that an online network tool is an applicable information tool for the OSH field. Some questioners preferred to consult a familiar expert in as well. The tool was stated to be applicable for non-urgent health problems and for gathering additional information. The experts stated that the system might assist in observing new OSH trends and might facilitate contact between questioners from the working population and experts. Hosting and support by a trustworthy professional organisation, anonymity, timely answers and effective promotion campaigns were mentioned as important requirements for use. Usability findings and participant remarks on online Q&A target groups indicate that online Q&A network tools are not applicable for people with no or only limited computer or internet experience. To provide OSH information to this sub-set of workers, asking a question directly or indirectly through a coordinator by telephone could be an alternative.
Little is known about the applicability and usability of Q&A tools and similar online networks for high quality information and knowledge in healthcare, although similar tools, such as tele-consulting systems and patient forums, have been discussed in the literature. Q&A tools are different in some respects (i.e. they include more or less extensive network features, self-selection of experts by questioners, e-mail notification and an easily accessible public database). Notwithstanding differences, comparison is useful. Marco et al. studied an "ask-the-expert-service" of a consumer-oriented website on HIV-AIDS. Despite the fact that there was only one expert answering questions, the authors concluded that there was a great demand for online "ask-the-expert" services. This opinion shared by Umefjord et al. who studied a similar service for enquiries related to health or diseases. These researchers found that an ask-the-expert-service was mostly used because of anonymity and convenience. Asking the questions and viewing the answers at a self-chosen time was highly appreciated. Other important reasons for use were to become better informed, to obtain a second opinion and to present embarrassing concerns and worries anonymously. Similar reasons for use were found by Himmel et al., who studied an expert forum on infertility. The importance of a "second opinion" was also brought forward as a reason for seeking tele-advice by Eysenbach et al, who studied patients asking questions mainly in the field of dermatology (unsolicited e-mails sent to physicians). Marco et al. stated that the facilitating conditions for the success of an ask-the-expert-service were anonymity, free access and timely answers. Massone et al., who studied a non-commercial tele-consulting system in the field of dermatology, concluded that these systems are promising when they are non-commercial, discretionary, multilingual and open-access in nature. Important reasons for using an ask-the-expert-service are the easy access and the additional information or second opinion about specific health issues or interventions[24–27]. Both reasons were confirmed by several participants in our study. Other facilitating reasons for use, such as anonymity and timely answers, are also in accordance with the results in this evaluation.
The strength of our study lies in the user-centred design used to evaluate and improve important features of this new Q&A network tool for OSH before implementation. However, the study has several limitations as well. First, the sample was limited and unevenly distributed with respect to age and sex, which may lead to overestimation or underestimation in the study results. In addition, the sample was not entirely representative for purposes of assessing applicability. It would have been better to recruit a larger sample from different settings, particular organisations or occupations, who actually had (answered) OSH questions. Second, the test took place in a field setting: the participants' workplace. Possible differences in this setting such as screen size, internet connection speed and keyboard features can result in dissimilar experiences and different research results. However, the advantage of a field test is that it represents the real life situation better than a laboratory experiment. A third limitation is the data collection method. Observation, for example, has advantages and disadvantages in comparison with methods such as video recordings. The investigator might miss some navigation paths, resulting in an overestimation of task efficiency. Moreover, the observer may somehow influence a participant. Sitting just behind a participant may create a feeling of being rushed, which may lead to mistakes. A usability laboratory can facilitate in more rigorous data collection. Next, evaluating a system that was also developed by the evaluators could raise a conflict of interest. For example, interviewees knew that we were developing a new information tool. This knowledge could have elicited gratifying responses. We tried to overcome this by creating an open atmosphere, in which participants were encouraged to find usability problems. Another limitation is the think-aloud protocol applied in this study. Thinking aloud during usability tests can facilitate finding problems, as it reflects the actual use of a feature rather than the participant's judgment[19, 31]. Therefore, some authors have noticed that think-aloud interviews can impede the discovery of usability problems and task performance.