Our case study has focused on the user research that ran alongside the development of an imaging device and has considered the way in which the results of this were taken into account by the company. The final stages of this development project are still to unfold and therefore we are not able to draw firm conclusions as to whether these issues were a significant factor in the success or failure of this device. However the focus of this paper has been one exploring the way in which user needs research was or was not integrated into the product development process and this focus is distinct from the issue of whether or not the product ultimately proves to be a success.
It is difficult to draw any firm conclusions as to whether greater consideration of the results of the user research would have benefitted this project. However, a number of the user requirements that were identified in the early user research were at variance with the direction and focus of technological development. For example, it was determined at the beginning of the project that the device should be portable so that it could be carried around by healthcare staff; the phrase “camcorder sized” was frequently used by the development team. As a result a main focus of the technological development was on developing a device that was light and small. However, the interview data suggested that manoeuvrability at the point of capturing the image was more important than portability. The clinical users reported that getting the device to patient’s side would not be a the problem, as organisational protocols meant that it would be likely that the device would stay in a particular clinic or department, but that once the device was next to the patient it would be essential that the device could be easily positioned to quickly capture the image of the required body area. This suggests that the effort and investment that was directed to the considerable technological challenge of producing a small device may have been a poor use of resources.
There was no evidence that the user research had a significant effect on the subsequent product development process. Our analysis of this case study suggests that a range of factors can be identified as contributing to this situation.
Integration of user needs with technology development
It was clear that the user needs part of the product development process was viewed as an isolated part of the project and separate to the technology development. The evidence provided by the requirements elicitation study was not integrated with the technology development process. This was also observed at the end of the project, when the developers requested the technical results of the prototype evaluation but not the results of the usability evaluation.
This chimes with the work of Chess [26] who provides a useful framework for exploring organisational responsiveness. She conceptualises the possible impact of organisational structure and processes on the way in which knowledge is exchanged within an organisation – and thus on the impact that this knowledge has on organisational actions. A key determinant of this is how different organisational functions are ‘coupled’, i.e. the way in which one function constrains another. In the product development scenario described in the case study it is clear that the user-facing functions (or at least those that are formally oriented around user needs) were loosely coupled to those that were related to development of the technology. The two different functions did not constrain the other, with each able to proceed independently of the other. This was evidenced by the lack of planned interdependencies between the user research and the rest of development. This is exemplified by the fact that at the end of the project, the technical results of the prototype evaluation were requested but not the usability results, suggesting that the developers did not believe that the usability results would affect future decisions about the development of the technology. There are a number of reasons why this type of situation may exist during the product development cycle. Within large organisations different aspects of development will often be performed by dedicated and separate departments and if there is poor communication or a lack of interaction between departments this could result in different aspects being de-coupled. Within a small company or a close-knit project team, such as the one in the case study, this type of situation should be less likely to occur as there should theoretically be greater involvement and awareness by all of the developing research results. However, although this was the case in this case study, there seemed to be consensus within the development team that the user needs research was not a critical aspect of development and that the results of this work would not significantly influence the development of the technology. As a result no contingencies in relation to the user needs work were identified.
We suggest that adopting a formal new product process to monitor progress and aid decision making may have led to improved integration of the different strands of this development project. The use of such processes has consistently been shown to be associated with successful product development across a variety of industries [27]. It is noteworthy that the funders of this research – the UK TSB, who support innovation in UK business by investing more than £500 million each year – did not either require or recommend that the project team adopt such a strategy for ensuring due consideration is given to the results of the user related research that is required.
Framing research aims
Arguably the way in which the aims of the user research were framed by the development team was inappropriate, both in relation to the characteristics of users and the timing of the research.
Characteristics of users
In the original product development plan no early stage of capturing clinical needs was planned. Although the developers were open to this when it was suggested, they believed that the main aim of user research was to identify design features for the new device to inform the development of a prototype. This was thus the primary aim of the requirements elicitation. The results of the case study demonstrate that this was a somewhat unrealistic aim. The data collected from the interviews indicated that users primarily expressed their reflections on the device in terms of the needs, difficulties and problems that they experienced. These findings support previous research that has shown that users find it difficult to provide design suggestions, particularly when unprompted by an existing prototype, as this requires a high degree of foresight [28, 29]. It has also been shown that this is a particular issue for radical innovations when users do not have knowledge or experience of a similar existing product to draw on [30], which was the case with this study. Unsurprisingly, in reflecting on design features, users are also likely to draw heavily upon the contexts in which would they be required to use the device.
From this, we propose that the strategy of developers at the early stages of medical device development should be on identifying clear clinical needs and requirements. Once this has been done it is then the role of the developers and designers to provide a device that provide solutions to these needs, which can then be tested and evaluated by users and hopefully provide the required triggers for a discussion of potential improvements and refinements. In reality of course, many devices are the outcome of a new technological capability and a matching clinical need then needs to be found to allow exploitation of this. In these circumstances it is arguably all the more important that formal processes that allow the integration of revealed clinical needs with technological development are adopted.
Timing of research
The user research commissioned by the development team did meet one of the main recommendations of the guidance and literature in this area: that research should be conducted early in the development process [31]. However our results suggest that this was negated by the lack of mechanisms for feeding the data into a product development process that was dominated by technological requirements.
The results of the requirements elicitation suggested that the fundamental concept for the device should be changed. The nature and scope of these findings was not anticipated by the developers. They rather believed that the results would confirm their belief on the clinical need for the device and provide a consensus on what the look and feel of the device should be. This suggests that it would be valuable for those doing user focused research to be clear about the potential range and scope of findings (for example, that the clinical need is different than anticipated, that unanticipated features of the device are of importance etc.). By the same token, developers need to consider and articulate what their latitude is for taking account of such findings within the product development process. Clearly it is naive to expect that all findings can be taken on board. Where time or money preclude a change in direction however, it is arguably important to document how and why this decision has been made as part of the formal stage gate process so that information is made available throughout and decisions revisited if necessary. Such a process also allows possible risks to commercial success to be identified and documented so that they can be monitored as development continues.
In our case study it appeared that neither the organisational culture nor the project management processes facilitated full consideration of the implications of the research results. That the developers did not expect the results to significantly contradict their beliefs was possibly due to their lack of familiarity with this type of research as well as to the researcher not making the potential impact of the research on the development plan clear. The researcher did not explicitly describe the scope and implications of the work, nor pose the question as to what they would do if the results were not in line with their expectations.
Even where developers are not aware of the potential benefits of user needs research a formal stage gate process would provide the opportunity for objective review of the research and clear documentation of the decisions that have been made on the basis of it. In this domain, where the assumption is that technological development has primacy, such a process provides a formal opportunity to articulate whether the user needs research that has been conducted requires any subsequent change in the development pathway.
Significance of user research results
Another possible reason for the user research results not being fully adopted by the development team may have been the degree to which the findings contradicted the concept for the device. The concept, developed as the result of informal discussions with a small number of clinicians, had been the basis for planning the entire project, including securing the funding and as a result it is reasonable to assume that the developers were committed to this concept. It is possible therefore that cognitive biases, specifically confirmation bias, may have affected how the development team processed the results of the user research, which contradicted this concept. Confirmation bias, identified by Nickerson [32] as the “single problematic aspect of human reasoning that deserves attention above all others”, is a well-established phenomenon where decision makers have been shown to search for or interpret information in a way that confirms their hypothesis, and place less importance or ignore evidence that would contradict it [33]. This may also explain why the developers did not request the complete results of the second user research study.
Another cognitive bias that may also have affected how the results were processed is Status Quo Bias [34]: the tendency to maintain the status quo when faced with a complex decision. Recent research on this phenomena has shown that the more difficult the decision, the more likely it is that the decision-maker will do nothing and maintain the status quo [35].
Again, adopting a formal new product process may be a strategy to counter this, resulting in a more objective approach being taken to the synthesis and uptake of research evidence. This is arguably even more pertinent for SME developers where there are likely to be fewer people involved in the development of each device perhaps leading to less objectivity during decision making.
Communicating research results
The way that the results of the first user research study were disseminated to the development team by the researcher (JM) may not have been effective in conveying the significance or breadth of the data. The full results were contained in a lengthy document that would have been time consuming for the development team to read and may also have led to the quotes losing some of their significance due to being presented as text. Playing the audio clips of these quotes may have been a more effective at conveying this information. In addition, although the most significant findings of the requirements elicitation were presented in a presentation to the development team by JM, this was done during a normal project meeting where it was one item on a full agenda. It may have been more effective to have scheduled a special meeting where the development team had more time to listen to and discuss the results.
Related to this issue is the fact that the user research was performed by an external researcher who was responsible for disseminating the information to the development team. It has been suggested that such an approach is undesirable as it inevitably results in some filtering and distortion of the information [36, 37] and that developers should be brought into direct contact with users rather than merely hearing or reading about them through intermediaries [38]. In addition to this, the risk and rewards of commercialising this device lay mainly with the lead organisation and therefore it is possible that the external researcher was not sufficiently motivated to reinforce the results of this work and to press for their prioritisation during development. It is possible that this type of situation may arise in other development projects when contracting user research (or other aspects of development) to external consultancies. It is the responsibility of the developer to establish processes that will allow all information streams to be heard and appropriately prioritised.
Nature of obligations to conduct user research
The main motivation for including user research a part of this project was that it was a requirement of the research funders. It is of note, however that although the plan for the user research proposed by the development team during the application process was extremely brief and vague, this met the funding requirements. No recommendations or guidance were provided to the developers as to what might constitute meaningful user research, nor was any evidence needed that the results of this research would be considered within the product development process.
The present research has suggested that it is not sufficient to plan to conduct user research and that this should not serve as the ‘trump card’ for funding applications. Indeed we would argue that the current simplistic requirement of some funders and regulators that users should be involved in the product development process may in fact be counterproductive. It may lead to this work becoming a box-ticking exercise, conducted to satisfy funders, rather than being a thoughtful consideration of how to conduct user research that firstly, allows the necessary information on user needs and requirements to be collected and secondly, allows the best product development decisions to be made. Taking this further, one might suggest that where user research is framed inappropriately (e.g. focusing on detailed device characteristics) that this may lead to the development path becoming more set and less amenable to change (greater lock-in) than if the user research was not done at all. It is our position that research funders should focus more on ensuring that the aims, approach, timing and integration of user research are appropriately specified and that the development decisions that are made on the basis of the research are visible.
Often the need for confidentiality leads to a reluctance to publish results and this leads to a lack of knowledge development in this area. We concur with Buckle [1] that a growing body of evidence and good-practice examples in this area would lead to a raised awareness amongst developers of how to effectively conduct and manage user research as part of a wider product development process.
Finally it is useful to reflect on what ‘good’ user needs research looks like. In this case study, in the final analysis it was not rated highly by the developer simply because the results it produced did not concur with developer expectations. In the development of technology there is a clear end point - acceptable functionality – yet the process by which this is reached remains largely invisible. In contrast, the end point of user needs research is unknown; rather research quality is a function of the process. Researchers should make developers aware that the research they conduct should be subject to evaluation and scrutiny in relation to the rigour of the process of data collection and analysis rather than on the substance of the findings per se.
Recommendations
-
1.
The guidance currently available to developers is somewhat simplistic. Funders of medical device research and development should provide developers with more detailed guidance on how to design effective user research studies. For example, what the aims of research at each stage of development should be, how it should be conducted, and what the implications of the research may be.
-
2.
User research in the early stages of medical device development should be kept broad and aim to ‘open up’ device considerations with a focus on identifying as many clinical needs as possible. Developers should wait until a prototype is available before collecting user opinions on specific design features.
-
3.
A formal new product process such as a stage-gate model should be adopted that allow for articulation and documentation, at pre-determined points, of the contingencies between technological development and the decisions that are made about user requirements.
-
4.
Developers should take care when contracting out parts of development that these are objectively appraised and considered by, for example, a senior member of the development team becoming a ‘champion’ for this work.