Constructive eHealth evaluation: lessons from evaluation of EHR development in 4 Danish hospitals
© The Author(s). 2017
Received: 22 December 2016
Accepted: 11 April 2017
Published: 20 April 2017
Information and communication sources in the healthcare sector are replaced with new eHealth technologies. This has led to problems arising from the lack of awareness of the importance of end-user involvement in eHealth development and of the difficulties caused by using traditional summative evaluation methods. The Constructive eHealth evaluation method (CeHEM) provides a solution to these problems by offering an evaluation framework for supporting and facilitating end-user involvement during all phases of eHealth development. The aim of this paper is to support this process by sharing experiences of the eHealth evaluation method used in the introduction of electronic health records (EHR) in the North Denmark Region of Denmark. It is the first time the fully developed method and the experiences on using the CeHEM in all five phases of a full lifecycle framework is presented.
A case study evaluation of the EHR development process in the North Denmark Region was conducted from 2004 to 2010. The population consisted of clinicians, IT professionals, administrators, and vendors. The study involved 4 hospitals in the region. Data were collected using questionnaires, observations, interviews, and insight gathered from relevant documents.
The evaluation showed a need for a) Early involvement of clinicians, b) The best possible representation of clinicians, and c) Workload reduction for those involved. The consequences of not providing this were a lack of ownership of decisions and negative attitudes towards the clinical benefits related to these decisions.
Further, the result disclosed that by following the above recommendations, and by providing feedback to the 4 actor groups, the physicians’ involvement was improved. As a result they took ownership of decisions and gained a positive attitude to the clinical benefits.
The CeHEM has proven successful in formative evaluation of EHR development and can point at important issues that need to be taken care of by management. The method provides a framework that takes care of feedback and learning during eHealth development. It can thus support successful eHealth development in a broader context while building on a well-known success factor: end-user involvement in eHealth development.
KeywordsUser involvement End-user participation Formative evaluation eHealth development Evaluation method Constructive eHealth evaluation Participation End-user Health information technology Organizational factors
During the past few decades, centralised eHealth technologies have replaced previous information and communication sources throughout the healthcare sector at an ever-increasing rate in large parts of the world. The objective is to use information and communication technology (ICT) to improve patient health and safety and the quality of treatment. However, besides its benefits, this development has also resulted in challenges to the organization and to evaluation-methodologies.
Although challenges related to e.g., availability, structure, and overview of patient-related information may have been solved by the introduction of eHealth technologies, other problems of an organizational nature have been introduced, as the focus in eHealth development is often directed towards the technological aspects, while organizational aspects are underrated [1–7]. End-user involvement in design and implementation is an example of an important organizational issue that requires special attention for a successful outcome [2, 3, 6, 8–12]. Experience has shown such involvement to be a precondition for achieving the positive clinical benefits of new eHealth technologies [1, 10, 13], as well as joint ownership [11, 12, 14–16] - both well-known success factors in eHealth development. The importance of end-user involvement in technological development processes in general has been recognized since the 1950s, when members of the London Tavistock Institute formulated the “sociotechnical approach”, according to which both users and the technical aspects of work processes must be accommodated to achieve joint optimization [17, 18]. During the following decades this sociotechnical approach was further developed [18, 19], and Enid Mumford established the “ETHICS” method that integrated the approach into IT system development [9, 20]. According to Mumford, the overall objective is to ensure that equal weight is given to technical and human factors in the design of technology, and that future users are involved in the design process [9, 21]. However, the concept ‘end-user involvement’ spans a broad scale, from end-users acting as consultants during the late phases of the technological development lifecycle to end-users exerting influence on the decisions made during all phases in the lifecycle, from innovation to operation. Experience shows that, to achieve the success factors mentioned above, the involvement must be at the first, more comprehensive, end of the scale: end-users must be involved at the earliest possible stage as their ability to influence decisions already taken gradually decreases as the development process progresses [15, 22–25].
Although organizational issues in eHealth development have been widely discussed in recent decades, eHealth implementations continue to face problems caused by a lack of awareness of their importance [13, 26–28]. A recent example is given by the evaluation of the implementation of a new nationwide EHR system in England, which has displayed limited or no benefits, mainly because of the absence of user involvement in the decisions made during the development process . This calls for new methods in the application of end-user involvement in the practice of eHealth development.
The evaluation-methodological challenges concerns the continued preference for summative methods which are facing fundamental difficulties as e.g., that the end goals must be fixed, the outcomes and the indicators be stable, and no intervening variables can be present [29–32]. However, eHealth development rarely fulfils these criteria, because a) eHealth development is a lengthy process as the technology often changes during the process, jeopardizing the evaluation methodologically; b) Traditional summative evaluation cannot be carried out until the technology is in operation - that is, when it is almost fully developed, making major changes difficult or expensive to implement should the evaluation show a need for this; and c) Traditional summative evaluation methods fail to accommodate the complexity of the healthcare sector with respect to differences in work practices across wards, clinical specialties, professional groups, etc. [22, 33, 34]. This calls for new methods for evaluating eHealth.
The new Constructive eHealth evaluation method (CeHEM) aims specifically to be used in evaluating eHealth in context, and offers a solution to the challenges outlined above. The method is inspired by the Constructive Technology Assessment (CTA) methods, which are formative evaluation methods to assess technology in general. The aim is to provide feedback on the strengths and weaknesses including the views of different actor groups, during the entire technological development lifecycle [35, 36]. The CTA concept, introduced by Leyten and Smits in 1987, has a specific focus on creating strong links between the technological development process and the user environment into which the technology is to be applied by establishing networks between relevant actor groups [37, 38], and shift the focus away from the various outcomes of fully developed technologies to also include relevant actor groups and the process itself [22, 33, 34]. The methods are typically inspired by interdisciplinary research perspectives from cognitive psychology, computer science, systems engineering, etc. . During the 1990s, researchers in the Netherlands and Scandinavia in particular introduced CTA methods into the healthcare sector. The need for technology assessment increased significantly during these years because of the strong growth of technology in the sector. Thus, a number of experiences of using CTA methods within the healthcare sector were described during this decade [22, 40–48].
To give the end-users as much influence as possible over the decisions taken, they should be involved in the CeHEM during the first eHealth development lifecycle [15, 22–25]. However, the method can also be used in lifecycle analysis to follow, if the evaluation involves the redesign of an eHealth system. This will of course considerably reduce end-user influence on decision-making.
Introduction to the constructive ehealth evaluation method
The CeHEM offers a full-lifecycle evaluation framework to support and facilitate end-user involvement during the eHealth lifecycle. It includes methods to guide the process during all phases, and a modified summative evaluation. It draws attention to evaluation as an independent activity in parallel with other project activities.
The evaluation activities are prefaced by contextual evaluation questions: “what, why, when, who, and how”. If available data do not allow these questions to be answered, the evaluation should not start [64, 65].
Phase 1: Research and planning phase
1a) Identification of relevant actor groups: Obvious actor groups in eHealth development include: clinicians, management, vendors, and IT professionals. However, to identify all relevant groups, partner analysis is recommended [66, 67].
1b) Selection of representatives of the identified groups: In order to support the achievements of ownership of the decisions taken during the eHealth development lifecycle, representatives of the various actor groups should be identified and chosen by peers [8, 16]. At the regional and hospital level, it is recommended that the professional associations make this selection. At ward level, the participatory method can be recommended .
1c) Technology-carrier analysis: Identifying possible conflicts of interests and carrying out power analyses of the interactions between actor groups can be an important part of Phase 1. To secure influence, six conditions need to be fulfilled in relation to the technology in question: interest in the technology, the power to carry through these interests, an organization to support them, and information on, access to, and knowledge of the technology .
Phase 2: Design phase
2a. Identifying needs: Needs analysis of basic functionalities should be carried out to identify the requirements that the new eHealth technology must meet.
2b. Requirement specifications: In 2005 the European Union’s Competitive Dialogue Process, was introduced as a new way of handling procurement. This method aims at highly complex areas where the provider (the eHealth project management) has difficulties in defining legal, technical, and/or financial matters related to the project. To ensure better-informed decisions, the Competitive Dialogue Process allows end-users to pose questions to the vendors on technical and other matters and hence supports end-user involvement in decision-making.
Phase 3: Development phase
The needs and requirements from Phase 2 are here converted into a first version of the new eHealth system, a pre-prototype. This process is made from collaboration and feedback between involved stakeholders. The User Innovation Management method, which is inspired by user-centred and participatory design methods [69–71] is recommended to facilitate the outlines of the possible technical configurations of the new eHealth system, as this method supports end-user involvement in the design of a new technology . The pre-prototype is tested in usability laboratories, simulation labs or in clinical set-ups to assess, for example, user-friendliness in work practice. Jacob Nielsen’s ten heuristics for usability studies may be used . The outcome of Phase 3 is a prototype.
Phase 4: Implementation and diffusion phase
Here the prototype is implemented for test in different clinical contexts. Working groups are formed at ward level to customize the prototype to specific clinical needs. The methods recommended for Phase 1b also apply here. During this phase, the system is tested when used in daily clinical practice. The test in clinical practise continues until the feedback between the ward working group has shown that ward-level customization is fulfilled and that end-users have accepted the result.
Phase 5: Modified summative evaluation
The modified summative evaluation includes the measurement of a) The before–after technical and/or economic quantitative changes and/or b) Qualitative and quantitative assessments of organizational changes - e.g., end-user satisfaction. In contrast to traditional summative evaluation methods, not all of the outcome measures and indicators in the CeHEM are unchangeable from Phase 1. The objective of the evaluation will guide which measurements are included, which are unchanged and which are reconsidered during Phase 1–4. In large eHealth development processes, it is difficult to predict which outcomes and indicators are relevant – especially for those of an organizational nature - until an actual prototype has been implemented, customized, and tested in a specific context because of, for example, the length of the process and the organizational complexity of healthcare. Thus, these outcome measures and indicators are not decided on until the prototype is customized, tested and retested, and accepted by all actors - i.e., in the last part of Phase 4 - with a special focus on the end-users. When this is reached, all outcome measures and indicators need to be fixed, and, for methodological reasons, the eHealth system should not be further adjusted until the modified summative evaluation is completed. While the technical and economic aspects of the summative evaluation should take place at national, regional, or hospital level (depending on the objective of the evaluation), the end-user assessment is best performed at ward level on account of differences between wards with respect to clinical specialties, professional groups, and work practices.
The overall responsibility of the evaluator is to actively facilitate learning and feedback between the relevant actor groups during all phases in the eHealth development lifecycle, in particular focusing on end-users. The evaluator’s role is thus to ensure that the end-users are meaningfully involved in decision-making in all phases of the lifecycle.
The Constructive eHealth Evaluation Method (CeHEM): phases, objective, activity, and methods
Justify a constructive evaluation need
Identify: Why… When… Who… How…
Health care Management meetings based on results from summative evaluation of existing eHealth system performance
Research and planning
Set-up a competent project organisation to provide solid health care and ICT professional insight to management and CIO
a) Identification of relevant actor groups
b) Selection of representatives of the identified groups
c) Technology carrier analysis
d) Identify indicators for final summative evaluation
Observation of work practices
When users identify users
Identify appropriate system developer based on the identified requirement specifications
Identifying user needs
Draft requirement specifications
User centred and participatory design methods e.g., User Innovation Management method (UIM)
Design of proto type
adjust and re-design
Information meetings with department management from different clinical context
Select appropriate clinical context/wards for pilot test
Pilot test of Pre-prototype
Feed-back structure for errors and changes re-design
Meetings with management from different clinical context
Presentation of prototype
Selected Interviews for feedback
Observation of test
Implementation and diffusion
Test of proto-type in clinical setting
Form working group at department and/or ward level
a) Identification of local actor groups relevant for the clinical settings
b) Selection of representatives of the identified groups
Test prototype functionalities
Observation of work practices
Modified summative evaluation
Before and after technical, organizational and economic changes
End user assessment
Measure based on indicators
Other formative evaluation methods
Existing formative evaluation methods have a different focus, and are covering only parts of the eHealth lifecycle or differ from the CeHEM in other ways.
In the 1990s, Bødker et al. introduced the MUST Method for Participatory Design, aimed at supporting end-user involvement in information technology development. However, it focused solely on supporting end-users during the design phase. Besides, the method focuses on ICT development in general, as it is intended to support IT professionals responsible for the design phase [14, 75].
Kaufman et al. have suggested a full-lifecycle evaluation framework for eHealth development in which a five-stage development model is presented. The framework, which has a user-centred approach to design, aims to provide a heuristic for matching the stages of system design and the levels of evaluation. In contrast to the CeHEM, the main focus is not on supporting end-user involvement, but rather on structuring research in health informatics by providing a rigorous evaluation at each stage in the lifecycle .
Full-lifecycle frameworks targeting eHealth evaluation have also been proposed by Catwell and Sheikh, Clarke et al., and van Gemert-Pijnen. However, their methods differ from the CeHEM in several ways. Focusing on improving eHealth development or on implementation, they provide feedback between all groups involved, without any specific focus on end-users [5, 32, 77]. Besides, no tools are provided to guide the process through the different phases of eHealth development and, as has been emphasized by both Nykänen et al.  and Høstgaard , they do not support evaluation as an independent activity that takes place in parallel with other project activities.
Greenhalgh and Russell have offered a different approach based on ten alternative principles for eHealth evaluation . Their principles are generally aligned with those of the CeHEM, but are presented in an abstract and generalized way, as they invite other researchers to help refine them.
Harrison et al. offer a new framework: the ISTA (Interactive Sociotechnical Analysis) framework and a typology specifying important relationships between new e-Health, workflows, clinicians, and organizations. The framework emphasizes the potential of these relations to produce unintended consequences. The objective is to make clinicians, managers, and designers aware of the risk of such unfolding consequences and to spot them through formative evaluation focused at the so-called “HIT in use” during implementation . This framework differs from the CeHEM in several ways, as it is not a life-cycle evaluation framework in the proper sense but an interactive sociotechnical analysis.
Actor Network Theory has been used to evaluate complex IT-systems in healthcare service organisations . They have proven useful in appreciating the complexity of reality and to understand social effects among different actors in a network. Their particular strength is of analytic nature, but provides little guidance for participatory activities.
Cresswell, Bates, and Sheikh advocate a lifecycle perspective of eHealth development - that is, formative evaluation methods - as a solution to the problems caused by traditional summative methods. They suggest ten important considerations for eHealth developments to succeed, focusing on involving end-users throughout the eHealth lifecycle to achieve co-ownership and commitment to the outcomes of the eHealth development process . Their considerations are fully in line with the view on eHealth development on which the CeHEM is based Table 1.
The CeHEM offers a structured analytical framework and methods for meeting the need for new eHealth evaluation methods, as well as the need for end-user involvement in eHealth development and implementation. The importance of involving end-users has been increasingly widely recognized during recent decades [2, 4, 14, 16, 50, 81]. This is underpinned by the European Union’s newly published eHealth Task Force Report, “Redesigning Health in Europe in 2020”, in which a recommendation of end-user involvement features strongly . Hence, the objective of this paper is to support successful eHealth innovation, development, and implementation by sharing the experiences gained from using the CeHEM during the evaluation of the EHR development process at regional level in the North Denmark Region in Denmark. This paper presents for the first time the fully developed method and the experiences on using the Constructive eHealth evaluation method during all five phases of the method.
In Denmark, hospitals are public, financed by taxation and governed by the five regional authorities, each of which has their own EHR based on different development strategies and platforms. The Constructive eHealth evaluation method was used in a case-study evaluation during the development of a new EHR in the North Denmark Region. The authors were commissioned by the regional IT board to carry out Phases 1–4 of the CeHEM and the end-user assessment part of Phase 5. The technical and economic parts of Phase 5 were performed by the EHR project management and the regional IT department. The detailed results are confidential and available only to the regional IT board and the EHR project management.
The design was a case-study evaluation of the first EHR development lifecycle in the region using the CeHEM. The case object was all phases in the first EHR development lifecycle in the region, and was defined in time from 2004 to 2010 and in place as described below in the sub-section: “Setting” .
The objective of the formative evaluation (Phases 1–4) was to support feedback between the identified actor groups - with a specific focus on the clinicians - to assist learning and to support end-user involvement. The objective of the modified summative evaluation (Phase 5) was to provide a survey of the clinicians’ assessment of the clinical benefits of the new EHR after implementation.
Identifying the relevant actor groups and recruiting participants is part of Phase 1 in the CeHEM, and is described in the Results section. In the present study, the term “clinicians” refers to physicians, nurses, and midwifes. However, there was a specific focus on the physicians, as several studies have demonstrated the importance of this end-user group to successful EHR development in that their acceptance of the new system is crucial to whether or not it is applied as intended in clinical work practice (Nøhr C, Høstgaard AM, Botin L, Kjær-Andersen S: Evaluering af GEPKA- projektet, Delrapport 2, Klinisk afprøvning, unpublished; [84, 85]). In Denmark, all physicians, including senior physicians, are expected to enter data into the EHR themselves as part of their work practice.
Phases 1 and 2: Clinicians (4 nurses, 2 physicians; after 18 months, 8 physicians), secretaries (1), IT professionals (8), administrators (the regional IT board (12 members of whom 2 were clinicians) and the EHR project management (5 members of whom 1 was a clinician)), and vendors (4). The clinicians, the secretary, and the IT professionals were all members of the so-called “EHR working group”, which was established at the start of Phase 1 with a responsibility for carrying out tasks from Phases 1 and 2.
Phase 3: All ward-level clinicians and secretaries, IT professionals (3), administrators (the regional IT board, the EHR project management), and a vendor (1).
Phase 4: All ward-level clinicians and secretaries, four IT professionals, administrators (the regional IT board and the EHR project management), and a vendor (1).
Phase 5 : All clinicians participating in Phase 4.
Phases 1 and 2: An office at Aalborg Hospital (meetings between the EHR working group and the vendors, one at a time)
Phase 3: One ward at Aalborg Hospital
Phase 4: Two wards at Aalborg Hospital, one ward at Frederikshavn Hospital, Frederikshavn, one ward at Himmerland Hospital, Hobro, and one ward at Himmerland Hospital, Farsø (either medical or surgical wards).
Phase 5: All Phase 4 wards.
Data collection for the formative evaluation, Phases 1–4
Data on the formative evaluation were collected using observation, interviews, and documents analysis. The overall perspective was on the interactions between the clinicians and the new EHR. However, the focus changed from phase to phase, as is elaborated below.
During Phases 1 and 2, non-participant observations were carried out at all EHR working group meetings (19 in total). During Phase 3, non-participant observation was carried out at one ward, and during Phase 4 at four wards, supplemented by non-participant video observation. During Phases 1 and 2, the various interests and the balance of power between the different actor groups in the meetings were in focus, particularly with respect to the physicians. During Phases 3 and 4, the focus was on the interactions between the clinicians and the new EHR. This was studied during morning and afternoon shifts in the case of the physicians, at morning and afternoon clinical handovers in the case of the nurses, and during ward rounds for both physicians and nurses. For Phases 3 and 4, an observation guide was developed on the basis of preliminary visits to the respective wards to help the researchers maintain a shared perspective and focus. During these phases, observation notes were compared and discussed among the researchers immediately after completion of the observations. In case of disagreement, consensus was reached by discussion.
During Phases 1 and 2, we did semi-structured interviews with 11 physicians, two IT professionals, and two administrators. During Phases 3 and 4, we did two focus group interviews at each ward, with six to eight informants in each session. The interviews in Phases 1–2 and Phases 3–4 had the same focus as the observations. All interviews were informed by an interview guide that had been pilot-tested on individuals comparable to the informants and revised before use in the study. All interviews lasted 15–60 min and were recorded and fully transcribed. All transcriptions were validated by the informants. No requests for changes were made.
During phase 1–5, the researchers had access to relevant documents including minutes of meetings in the EHR working group, project plans, and procurement documents from vendors.
Data collection for the modified summative evaluation, Phase 5
In the case of the North Denmark region, the objective of the modified summative evaluation was to provide a survey of the clinicians’ assessment of the clinical benefits of the new EHR after implementation. Thus, no actual before-after study was requested. Two surveys were conducted during Phase 5, one at the start and one at the end of Phase 4, in all five wards participating in Phase 4. The first survey aimed at measuring clinicians’ expectations of the clinical benefits of the new EHR prior to its implementation. The questionnaire was divided into: 1) Basic information on the informants; 2) Questions relating to the outcome measure: knowledge of the new EHR prior to implementation (the questions concerned indicators of this, with space provided for open answers); and 3) Questions relating to the outcome measure: the clinicians’ expectations of the clinical benefits of the new EHR (the questions concerned indicators of this, with space provided for open answers). The second survey measured clinicians’ assessment of the clinical benefits of the EHR after its implementation. The questionnaire for this survey was divided into: 1) Basic information; 2) Questions relating to the overall outcome measure - i.e., the clinical benefits of the new EHR, as measured by questions concerning the following indicators: the accessibility of clinical data, system performance, customization, and effectiveness. The formulation of indicators was guided by the objective of the survey, responses to the first survey, and experience from previous healthcare evaluation studies. Before use in the study, both questionnaires were pilot-tested on groups comparable to the respondents and revised.
Data analysis for the formative evaluation, Phases 1–4
All data collected for the formative evaluation were analysed using a hermeneutic strategy . All three authors collaborated on the analysis. In case of disagreement, consensus was reached by discussion. The ATLAS software program was used for structuring the analysis process .
Data analysis for the modified summative evaluation, Phase 5
Frequency analyses of the physicians’ and the nurses’ expectations of the new EHR (first survey) and of their assessment of its clinical benefits (second survey) were performed at ward level using the statistical program SPSS. The open answers were analysed according to the strategy described for the formative evaluation.
The results of the evaluation in the North Denmark Region using the Constructive eHealth evaluation method are specified phase by phase. The evaluation of Phases 1 and 2 focused on the communications between the actor groups, specifically on the physicians. However, the evaluators (the authors) were not actively involved in the evaluation of these phases, in contravention of the ideas of the CeHEM. The results and the discussion provided for Phases 1 and 2 therefore focus on the actions actually taken during these phases, as compared to the recommendations in the CeHEM. Phases 3, 4 and 5 were done following the CeHEM.
The responses to the questions of “what, why, when, who, and how” are provided in the Methods section.
Phase 1: Research and planning phase
1a) Identification of relevant actor groups and 1b) selection of representatives of these groups: The CeHEM recommendations for Phases 1a and 1b were not followed in this case study. The EHR working group was established by the project management, who selected its members at the start of Phase 1 based on the staffs’ experience with participation in previous IT projects in the region. The group included clinicians (2 physicians and 4 nurses), secretaries (1), and IT professionals (8). Eighteen months later, the physicians’ professional organization at the hospital objected to all medical specialties, age levels, and locations only being represented by two colleagues. They feared that physician’s professional interests were not adequately represented. As a result, the group was supplemented with six physicians selected by their regional association. However, it was difficult for the newcomers. They had little chance to catch up with the knowledge acquired by the rest of the group during the previous 18 months - that is, to acquaint themselves with the extensive and very technical procurement material. This meant that they did not feel competent to make informed decisions. Thus, their decisions were primarily based on advice from the IT professionals. This caused the physicians to lack ownership . As a consequence of a decision taken by the regional IT board workload reduction was not an option. Hence, the physicians were expected to work and as an extra task do the work associated with the EHR development process. This was possible only for the senior physicians at Aalborg Hospital - and only partly in their case - who could plan their daily work more flexibly than their younger colleagues and who did not have to spend time on transport to the meetings, which were held at Aalborg Hospital. As a consequence, seven out of the eight physicians in the working group were senior physicians from Aalborg Hospital. Time pressure caused the single physician who worked at a smaller regional hospital to leave the group after attending only one meeting. Younger physicians, who make up more than half of the physicians in the region, and physicians from the smaller regional hospitals, thus had no representation during Phases 1 and 2 .
1c) Technology-carrier analysis The technology-carrier analysis was not performed as a part of Phase 1 - that is, before the need assessment for the new eHealth technology had been carried out, as prescribed by the CeHEM, but during Phase 2a (identifying needs). This will be dealt with in the Discussion section. Only a synthesis of the individual analysis of the six conditions required for true involvement in the development of the new EHR in the region will be provided in this paper, as a detailed analysis has previously been given . The synthesis shows that the physicians did not become truly involved in the process, mainly because of their workload, which affected all six conditions required. Time pressures thus had the effect of reducing their role to that of informing the group of clinical needs. Hence they exerted little influence on most decisions made during Phases 1 and 2 of the EHR development lifecycle .
Phase 2: Design phase
2a) Identifying needs The evaluation revealed differences in the interest shown by the three main actor groups (clinicians, IT professionals, and administrators) in the basic functionalities of the new EHR. The main interest of the clinicians (and of the physicians in particular) was the clinical benefits and high user-friendliness. In contrast, IT professionals focused on the administrative functions, which were in line with the interest of the administrators, who focused on compliance with national requirements for clinical data handling. This means that, although the groups shared the same overall vision - that the new EHR improved patient health and safety and the quality of treatment - their identification of needs for the basic functionalities reflected their own interests. However, because of their underrepresentation during the first 18 months, the physicians were unable to specify their clinical interests. As a result, the clinical interests were formulated by a nurse administrator with clinical background and IT professionals working full-time on the EHR development project. The requirement specifications were subsequently formulated on this basis .
2b) Requirement specifications The detailed requirement specifications for the new EHR were formulated through a Competitive Dialogue Process aimed at complex IT projects. This allowed all participants in the working group to ask questions of the vendors until they felt they were capable of making informed decisions. The Competitive Dialogue Process consisted of four phases:
The prequalification phase during which the EHR project management invited vendors to apply for participation in the process. Based on criteria formulated by the EHR project management, four vendors were invited to compete against each other in developing quotations for the new EHR.
The dialogue phase, involving the participants in the EHR working group and each of the four vendors, one at a time.
The quotation phase, during which provisional quotations for the new EHR were presented by each of the four vendors based on discussions in the dialogue phase. Based on these quotations, the dialogue continued until the working group had no more questions after which they formulated their final requirement specifications.
The decision phase, in which the four vendors tendered their final quotations.
The support of end-user involvement enabled by the Competitive Dialogue Process ensures that it is in line with the CeHEM. The clinicians, in particular the physicians, could thus acquaint themselves with technological issues through dialogue with vendors. However, despite their very positive attitude to the Competitive Dialogue Process, the consequences of the physicians’ lack of real influence during Phases 1 and 2a were evident throughout. Hence, their choice of vendor relied heavily on advice from the group’s IT professionals and, as a consequence, the physicians felt little ownership of this decision .
Phase 3: Development phase
During the development phase, a new working group was established at one ward. The representatives were chosen by and among management and staff. The group developed a pre-prototype in preparation for the prototype proper. Different functionalities (e.g., the user interface) were designed and developed using mock-ups and games (inspired by the User Innovation Management method ) in close collaboration between the actor groups; these were tested and retested on the basis of feedback and learning facilitated by the evaluators. The feedback took place through personal contacts, meetings, and a number of reports to the EHR project management and the IT board. This process continued for several months until all actors involved - the physicians in particular - had accepted the result: an actual prototype of the new EHR (Høstgaard AM: Evaluering af Præpilottest af “Klinisk Proces” på Infektionsmedicinsk afd. Aalborg Sygehus Syd, unpublished).
Phase 4: Implementation and diffusion phase
“For us the Partogram is the main work document, where we enter our record notes – that is the Partogram, and it is not included (in the EHR)” (midwife at focus group interview).
“And this program (the Partogram) is the most important” (midwife at focus group interview).
“…it (rebooting all computers) was an integrated part of the secretary’s work tasks at night shifts, when there were two….then some reorganization took place, and suddenly there is only one secretary at night, and she has plenty to do, because she has functions related to other wards as well as a consultant function for the entire hospital. Thus, it just fell out, and nobody took any notice of it, which means, that the computers are no longer rebooted” (nurse at focus group interview).
“… many times it (the pocket-size EHR system guide) says: choose something, and then it is in fact a sub-thing beneath the thing, it tells you to choose – do you understand? It is not comprehensive, and it skips several steps” (nurse at focus group interview).
When the evaluators reported this problem to the EHR project management, a procedure for correcting mistakes in the guide supporting the system was established .
Phase 4 went on at all five wards until user feedback had convinced the management that the respective prototypes were ready for the summative evaluation in Phase 5.
Phase 5: Modified summative evaluation
The modified summative evaluation took place at each of the five wards participating in Phase 4. Thus, when the EHR prototype had been customized, tested and retested, and finally accepted by the end-users at the five wards, no more changes of the outcome measures and indicators - and no further adjustment of the prototypes - took place until the modified summative evaluation was completed. The response rate in the first survey at the start of Phase 4 (which had the aim of identifying the clinicians’ expectations of the system) varied across the five wards; for physicians between 39 and 100%, for nurses between 41 and 53%. The results indicated three key issues across wards and professions: fast and easy access to clinical data, good overview of clinical data, and access to the EHR from multiple PCs. The second survey was carried out at the end of Phase 4 and aimed to establish the clinicians’ assessment of the clinical benefits of the new EHR; the physicians’ response rate varied between 35 and 77%, the nurses’ between 27 and 43%. Across all five wards, the results showed less positive attitudes among the physicians than among the nurses with respect to the following indicators: accessibility of clinical data, system performance, and effectiveness - with no correlation between the response rate and attitudes. At three of the five wards, the physicians were significantly less positive than the nurses; at two of these, the system was not used during ward rounds, only for access to historical data in acute admissions, while at the last, the physicians did not use the system at all because they felt it hampered their clinical work. At the remaining two of the five wards, the system was used as intended although the physicians were slightly less positive than the nurses. With respect to the last indicator, customization, the physicians and the nurses were equally positive - with a tendency for the physicians to be the most positive.
Besides the results mentioned above, the evaluation showed how a number of technical problems persisted, relating for example to log-on procedures, uptimes, and response times. Therefore, the evaluators (the authors) final recommendation to the management was to postpone further implementation and diffusion of the new EHR to other hospital wards until all these issues had been resolved - and hence, take no further steps until the system was able to meet the clinical needs. The management decided to wait for another year before implementing the system at all hospital wards in the region. During this year, several functionalities were adjusted. This may be assumed to have prevented a number of problems from occurring at later stages . Although the implementation started before 2010 no final overall summative evaluation of the EHR development process in the North Denmark Region has yet - in 2016 - taken place.
Discussion of results
The evaluation of Phases 1 and 2 was not a Constructive eHealth evaluation in the strict sense, as the evaluators were not actively involved and the technology-carrier analysis was performed as part of Phase 2a rather than as part of Phase 1c. However, the results of the evaluation strongly support the recommendation of the Constructive eHealth evaluation method. The case study not only points out when the involvement of the stakeholders are omitted but also specifically recall what the consequences can be. This is a natural result of an iterative development of a methodology. The results thus support the experiences gained from earlier empirical studies by DaCHI (Høstgaard AM: Evaluering af Præpilottest af “Klinisk Proces” på Infektionsmedicinsk afdeling Aalborg Sygehus Syd, unpublished [88–91]) and from the literature on the preconditions for achieving real involvement of the clinicians. The important prerequisites for achieving joint ownership of decisions taken during eHealth development thus appear to be: a) Early involvement of clinicians [15, 22–25], b) The best possible representation of all groups of clinicians [8, 9, 92], and c) Workload reduction [50, 93]. The evaluation revealed that the physicians in the EHR working group gained little ownership of the decisions taken during phase 1 and 2 because the above pre-requisites were not met. The use of the Competitive Dialogue Process in Phase 2b, which supported the physicians’ involvement, had little effect on the overall result of the evaluation. We argue that most of the problems caused by not involving enough physicians from different specialities could have been avoided if the recommendations of the CeHEM had been followed. This also appears to be relevant in particular to early participation, to the methods for identifying and selecting participants of the EHR working group and to the support given by the evaluators to the physicians during this process. Thus, a higher degree of ownership among the physicians of the decisions taken - a well-known success factor in eHealth development - could probably have been achieved - despite the absence of workload reduction.
The results of the evaluation also indicate a need to perform a technology-carrier analysis in Phase 1c - that is, before the needs for the basic functionalities in the new EHR are defined in Phase 2a. In the case of the North Denmark Region, it is very important to note that most of the physicians’ clinical needs were formulated by non-physicians, with a great risk that the group’s needs were not fully met, as experience shows that only the members of a professional group can formulate the specific needs of that group [8, 71, 94]. We argue that this risk could have been significantly diminished if the EHR project management had been alerted to the results of the technology-carrier analysis at an earlier stage, as such results have been shown to be good markers of end-users’ ability to show real involvement in decision-making in subsequent phases . This argument is based on the observation that the regional IT board and the EHR project management were very responsive to the potential consequences of the physicians’ lack of ownership of the decisions taken. This was evident when, at the end of Phase 2, they were presented with the results of the evaluation of Phases 1 and 2, which revealed their poor involvement. This caused the leadership to focus strongly on supporting the clinicians’ involvement during the following phases by letting them chose their own representatives and having them all participate from the start. Although workload reduction was not an option, this did not affect the results of the evaluation to the same extent because the demands put on their time were less heavy than in the earlier phases.
During Phases 3 and 4, the evaluator-facilitated feedback between the actor groups gave emphasis to the physicians’ role. As a result, they were more heavily involved in the design of the prototype and its customization in different wards. Their ownership of the decisions taken during this phase was likewise much stronger . The evaluators’ role during these phases was very important, as it most certainly prevented a number of problems from occurring later. Moreover, the experiences gained in the evaluation of Phase 4 showed the importance of using external evaluator/evaluators in order that responsibility for the evaluation is separated from other roles in the project and the hospitals to avoid conflicts of interest. Had the evaluation been an intrinsic part of the overall project activities, some of the problems identified would likely not have been discussed openly. This also stresses the importance of constructive exchange of feedback between actor groups, as this is a precondition for learning from the mistakes, rather than repeating them.
The less positive attitudes of physicians (in contrast to nurses) revealed by the summative evaluation applied to all five wards and were concerned with such basic functionalities of the EHR as accessibility to clinical data, system performance, and effectiveness. At three wards, the physicians’ attitudes to these indicators were negative. During Phases 1 and 2, it was decided which indicators to use. At that time, the physicians’ involvement in decision-making was very limited. Thus, even though no cause and effect relationship can be claimed, the summative evaluation show negative consequences of having the needs analysis and the requirement specifications formulated by other professional groups (e.g., nurses). Only at two wards was the system used by the physicians as intended, while at two other wards, it was used to a very limited extent, and at the last ward it was not used at all. The results thus show a correlation between the clinicians’ involvement in the decision-making during Phases 1 and 2 and positive attitudes towards the basic functionalities, along with a higher degree of use as intended . With respect to the indicator: customization, the physicians and the nurses were equally positive - the physicians slightly more so. This indicator is related to the decisions taken during Phases 3 and 4, when the physicians were actively involved in the customization process. Thus, these results also stress the importance of the physicians’ involvement in decision-making with respect to attaining ownership, as well as clinical benefits as a result of these decisions.
Discussion of methods
During Phases 1 and 2 in the evaluation of the EHR development, the authors role as evaluators meant that they were working close to the participants in the EHR working group for a longer period. This poses a risk that they were affected by group members’ attitudes - causing difficulties with keeping an objective view of the interactions between the actors. To remedy this, triangulation of methods was used together with a thorough description of all activities throughout the process to achieve the best possible transparency. Additionally, the authors were given access to all meetings during Phases 1 and 2 and to most documents related to these phases, which enabled the validation of statements from participants.
The response rate for the second survey, on which most of the results of the summative evaluation are based, varied between 35 and 77% for the physicians and between 27 and 43% for the nurses. This indicates a need for caution in generalizing the results across wards. However, they are strongly supported by the results of the focus group interviews that were carried out in all five wards at the end of Phase 4. The informants were selected by the clinicians themselves, to secure overall representation of the attitude among the respective groups of clinicians.
The experiences that formed the basis of the development of the CeHEM were gained through evaluation studies in Denmark. However, according to Berg et al., interaction between different actor groups is not a country-specific phenomenon, as it is found in eHealth development in general . The CeHEM is thus capable of assisting in successful eHealth innovation, development, and implementation across borders. A barrier to the wider use of formative evaluation methods, such as the CeHEM, lies in their demands on time and money, compared to traditional summative methods. However, the gains achieved by using this method clearly compensate for this, as this paper has demonstrated.
End-user involvement is an established, though rarely achieved, success factor in eHealth development. The development of the Constructive eHealth evaluation presented in this paper has been an iterative process with input from earlier research and the literature. It offers as such a full evaluation framework for the entire eHealth development lifecycle. It improves the adoption of eHealth in healthcare environments by supporting and facilitating end-user involvement during all phases. The method further offers guidance through all lifecycle phases, a modified summative evaluation, and, essentially, support for evaluation as an independent activity separated from other project activities.
The CeHEM has proven effective in evaluating EHR development in the North Denmark Region in Denmark. The evaluation made clear the consequences of failure to adhere to recommendations regarding the preconditions needed to achieve the real involvement of clinicians to ensure that functionalities are articulated by those who will be the primary system users, and hence to achieving joint ownership of the decisions taken during eHealth development. The recommendations are: a) Early involvement of the clinicians, b) The best possible representation of all groups of clinicians, and c) Workload reduction [50, 92]. The consequent lack ownership of the decisions taken and negative attitudes towards the clinical benefits related to the incomplete adherence. The evaluation also showed that, when the recommendations were followed and continual feedback to all actor groups was provided, the clinicians became truly involved in decision-making, which resulted in that they took ownership of decisions and gained a positive attitude to the clinical benefits related to these decisions. Finally, as a result of the evaluation, management decided to slow down the original plans for the EHR development process to ensure the best possible outcome of the process. By taking into account the feedback given by all actor groups involved, including management, problems have been avoided.
The CeHEM can be used across countries, as studies show that most interactions between user groups are not limited to one country. The method is applicable as a framework for achieving valuable information and learning during eHealth development in a broader context, thereby supporting successful eHealth development while building on one of the most well-known success factors: the true involvement of end-users in eHealth development.
Constructive eHealth evaluation method
The competitive dialogue process
Danish Centre for Health Informatics, AAU
Electronic health record
Information and communication technology
We would like to thank everyone who participated in the Constructive eHealth evaluation of the EHR in the North Denmark Region for making possible the evaluation, and hence this paper.
This research received no specific grant form any funding agency in the public, commercial or not-for-profit sectors.
Availability of data and materials
The datasets generated and analyzed during the current study are not publicly available due to the policy of confidentiality of the North Denmark region but are available from the corresponding author on reasonable request.
This paper presents the overall results of a number of empirical studies, each of which has been reported in prior publications [14, 63, 88–91], hence the data on which this paper is based, have been published before. However, the former publications have been reported in Danish [50, 89], in (Høstgaard AM: Evaluering af Præpilottest af “Klinisk Proces” på Infektionsmedicinsk afd. Aalborg Sygehus Syd (Evaluation of the Pre-pilot test of “Clinical Process” module at the Department of Infectious Diseases), unpublished) or have reported only parts of the data [63, 88, 90, 91]. This paper presents for the first time the fully developed method and the experiences on using the Constructive eHealth evaluation method during all five phases of the method. Thus, phase 5 (the summative evaluation) has never been reported in any publication before. Besides, in contrast to former publications, it deals with two problem-areas: the lack of end-user involvement and the use of traditional summative evaluation methods in eHealth evaluation.
The eHealth evaluation model was conceptualized, designed and developed by AMH, CN, and PB. AMH, CN and PB analysed and interpreted the data referred to in this paper. AMH wrote substantial sections of the manuscript. CN and PB provided feedback and revised the manuscript critically before submission. All authors read, contributed to and approved the final manuscript.
AMH: DDS, MPH, PhD. Associate professor, Department of Health Science and Technology, Aalborg University, Denmark. PB: Cand. Tech. Soc. Ph.D. Associate professor, Department of Planning, Danish Centre for Health Informatics, Aalborg University, Denmark. CN: MSc, PhD. Professor, Department of Planning, Danish Centre for Health Informatics, Aalborg University, Denmark.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
According to the Danish Act on Research Ethics Review of Health Research Projects, Section 14, No. 2 , permission to commence from the local Research Ethics Committee is only required for studies involving human biological material. The North Denmark Region Committee on Health Research Ethics has confirmed that no permission is needed for this project, entitled “Constructive eHealth evaluation: Lessons from evaluation of EHR development in 4 Danish hospitals”. To adhere to the Danish Act on the Processing of Personal Data, Chapter 4, Paragraph 6, Section 1 , informed verbal consent to participate was obtained from all participants prior to data collection.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc. 2003;10(3):229–34.
- Cresswell KM, Bates DW, Sheikh A. Ten key considerations for the successful implementation and adoption of large-scale health information technology. J Am Med Inform Assoc [Internet]. 2013;20(e1):9–13. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23599226%5Cn. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC3715363.View ArticleGoogle Scholar
- Edmondson AC. Framing for learning: lessons in successful technology implementation. Calif Manage Rev [Internet]. 2003;45(2):34–54. Available from: http://search.ebscohost.com/login.aspx?direct=true&db=bsh&AN=9463628&site=ehost-live%5Cn. http://content.ebscohost.com.turing.library.northwestern.edu/ContentServer.asp?T=P&P=AN&K=9463628&S=R&D=bsh&EbscoContent=dGJyMNHr7ESep7c4zdnyOLCmr0yep7BSsqi4TLCWxWXS&Con.View ArticleGoogle Scholar
- Pagliari C. Design and evaluation in ehealth: challenges and implications for an interdisciplinary field. J Med Internet Res. 2007;9(2):e15.
- van Gemert-Pijnen JEWC, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res. 2011;13(4):e111.
- Seemann J, Dinesen B, Gustafsson J. Interorganizational innovation in systemic networks: TELEKAT findings. Innov J. 2013;18(3):1-18.
- Berg M, Aarts J, Van der Lei J. ICT in health care: sociotechnical approaches. Methods Inf Med. 2003;42(4):297–301.
- Kensing F, Blomberg J. Participatory design: issues and concerns. Comput Support Coop Work [Internet]. 1998;7(3–4):167–85. Available from: http://link.springer.com/10.1023/A:1008689307411.View ArticleGoogle Scholar
- Mumford E. Designing human systems for new technology : the ETHICS method. Manchester: Manchester Business School; 1983. p. 108.
- Berg M. Implementing information systems in health care organizations: myths and challenges. Int J Med Inform [Internet]. 2001;64(2–3):143–56. Available from: http://www.ncbi.nlm.nih.gov/pubmed/11734382.View ArticleGoogle Scholar
- Lorenzi NM, Riley RT. Organizational Aspects of Health Informatics. New York: Springer Verlag; 1995.View ArticleGoogle Scholar
- Lorenzi NM, Riley RT, Blyth AJC, Southon G, Dixon BJ. Antecedents of the people and organizational aspects of medical informatics. J Am Med Informatics Assoc [Internet]. 1997;4(2):79–93. Available from: http://www.jamia.org/cgi/content/abstract/4/2/79.View ArticleGoogle Scholar
- Robertson A, Cresswell K, Takian A, Petrakaki D, Crowe S, Cornford T, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: qualitative analysis of interim results from a prospective national evaluation. BMJ. 2010;341:c4564.View ArticlePubMedPubMed CentralGoogle Scholar
- Bødker K, Kensing F, Simonsen J. Participatory IT design : designing for business and workplace realities. London: MIT Press; 2004.
- Johnson CW. Why did that happen? Exploring the proliferation of barely usable software in healthcare systems. Qual Saf Health Care [Internet]. 2006;15 Suppl 1:i76–81. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17142614%5Cn. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC2464864.View ArticleGoogle Scholar
- Paré G, Sicotte C, Jacques H, Jacques H. The effects of creating psychological ownership on physicians’ acceptance of clinical information systems. J Am Med Inform Assoc [Internet]. 2006;13(2):197–205. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16357351.View ArticleGoogle Scholar
- Our History - The Tavistock Institute [Internet]. Available from: http://www.tavinstitute.org/who-we-are/our-history/.
- Trist E. The Evolution of socio-technical systems: a conceptual framework and action research program. In: Conference on Organizational Design and Performance. 1980. p. 1–67.
- Cherns A. The principles of sociotechnical design. Hum Relations. 1976;29(8):783.View ArticleGoogle Scholar
- Mumford E, Weir M. Computersystems in work design: the ETHICS method. New York: John Wiley; 1979.Google Scholar
- Mumford E. The story of socio-technical design: reflections on its successes, failures and potential. Inf Syst J. 2006;16:317–42.View ArticleGoogle Scholar
- Müller Jens, Kjaer-Rasmussen Jens NC. Proactive technology assessment as a tool in HIS development. In: Barber B, editor. MEDINFO 1989. Singapore: North Holland; 1989. p. 306.
- Protti D, Protti D. What can the American electronic health record (EHR) pioneers tell us about what it takes to be successful? Healthc Manag Forum [Internet]. 2002;15(2):33–5. Available from: http://linkinghub.elsevier.com/retrieve/pii/S0840470410605789, [cited 2016 Nov 2].View ArticleGoogle Scholar
- Edwards PK. Conceptual and methodological issues in evaluating emergent programs. Eval Program Plann. 1987;10(1):27–33.View ArticleGoogle Scholar
- Hornik R. Shedding some light on evaluation’s myths. Dev Commun Rep. 1980;29:1–4.Google Scholar
- Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med [Internet]. 2011;8(1):e1000387. Available from: http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1000387.
- Sheikh A, Cornford T, Barber N, Avery A, Takian A, Lichtner V, et al. Implementation and adoption of nationwide electronic health records in secondary care in England: final qualitative results from prospective national evaluation in "early adopter" hospitals. BMJ [Internet]. 2011;343(1):d6054. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22006942, [cited 2016 Nov 2].View ArticleGoogle Scholar
- Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA. 2005;293(10):1261–3.View ArticlePubMedGoogle Scholar
- Ammenwerth E, de Keizer N. An inventory of evaluation studies of information technology in health care trends in evaluation research 1982–2002. Methods Inf Med [Internet]. 2005;44(1):44–56. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15778794.Google Scholar
- Douma KFL, Karsenberg K, Hummel MJM, Bueno-de-Mesquita JM, van Harten WH. Methodology of constructive technology assessment in health care. Int J Technol Assess Health Care [Internet]. 2007;23(2):162–8. Available from: http://www.journals.cambridge.org/abstract_S0266462307070262.View ArticleGoogle Scholar
- Dahler-Larsen P, Krogstrup HK. Evaluering og institutionelle standarder: Nyinstitutionelle betragtninger af evaluering som vor tids ritual. Politica. 2000;32(3):283-98.
- Catwell L, Sheikh A. Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Medicine. 2009;6(8)e1000126.
- Bijker WE. How is technology made?-That is the question! Cambridge J Econ. 2009;34(1):63–76.View ArticleGoogle Scholar
- Schot J, Rip ARIE. The past and future of constructive technology assessment. Technol Forecast Soc Change. 1997;268(1996):251–68.View ArticleGoogle Scholar
- Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17(2):204–20.View ArticleGoogle Scholar
- Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21(Suppl2):S1-S8.
- Smits R, Leyten J, Den Hertog P. Technology assessment and technology policy in Europe: new concepts, new goals, new infrastructures. Policy Sci. 1995;28(3):271–99.View ArticleGoogle Scholar
- Smits R, Leyten J. Key issues in the institutionalization of technology assessment. Development of technology assessment in five European countries and the USA. Futures. 1988;20(1):19–36.View ArticleGoogle Scholar
- Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform. 2004;37(1):56–76.View ArticlePubMedGoogle Scholar
- Brender J, Brender J. Trends in assessment of IT-based solutions in healthcare and recommendations for the future. Int J Med Inform [Internet]. 1998;52(1–3):217–27. Available from: http://linkinghub.elsevier.com/retrieve/pii/S1386505698001403, [cited 2016 Nov 6].View ArticleGoogle Scholar
- Berleur J, NT N, R D. Technology Assessment in Medical and Health Care Informatics: A Clarification of the Concepts. In: EMSJ van G, Talmon J, editors. Assessment and Evaluation of Information technology in Medicine. Amsterdam: IOS Press; 1995. p. 21–34.
- Vissers MC, Talmon JL. Evaluating the MACRO solution. Multimedia application for clinical research in oncology. Stud Health Technol Inform. 1997;43 Pt B:741–5.PubMedGoogle Scholar
- Nøhr C, Bernstein K. Implementation and diffusion of medical informatics; the case of electronic prescribing communication between practitioners and pharmacies. Prim Heal Care Spec Gr Br Comput Soc. 1992;9:14–8.Google Scholar
- Nøhr C. Computer-assisted decision support - improved health? A prospective technology assessment. Aalborg: Aalborg University; 1991.Google Scholar
- Nøhr C, Bernstein K. Can HIS be developed without organizational change? The creative potential in user participation. In: MEDINFO 89. Amsterdam: Elsevier Science; 1989. p. 5.
- Nøhr C. Involving end-users in prospective technology assessment. The case of computer based information technology. In: The sixth meeting in the international society of technology assessment in health Care (ISTAHC). Cambridge; 1990:6(1).
- Nøhr C. Social Carriers of Technology. An Actor Oriented View on Diffusion. In: The Seventh Meeting in the International Society of Technology Assessment in Health Care. Helsinki Finland; 1991.
- Nøhr C, Wiegmann I. Technology assessment in danish hospitals. A matter og technology agreements. In: Technology and Democracy: The Use and Impact of Technology Assessment in Europe. Cambridge: Cambridge University Press; 1992.
- Edquist C, Edqvist O. Social carriers of techniques for development. J Peace Res. 1979;16:313–31.View ArticleGoogle Scholar
- Høstgaard A.M. Succeskriterier og barrierer i Sundheds-it udvikling. Case studie af EPHudbudsprocessen i Region Nordjylland med fokus på slutbruger indvolvering (Success factors and barriers in Healthcare IT development. Case study of the EPH tender process in The North Denmark Region with a focus on end-user's involvement). Delft: River Publishers; 2015. Available at: http://www.riverpublishers.com/pdf/ebook/RP_E9788793379343.pdf.
- Müller J, Kjær-Rasmussen J, Nøhr C. Perspektiver for edb-teknologi i sygehusvæsenet. Aalborg: Centertrykkeriet AUC; 1988. ISBN: 87-89088-06-9.
- Checkland P. Soft systems methodology: a 30-year retrospective. Syst Res Behav Sci [Internet]. 1999;58(3):11–58. Available from: http://books.google.com/books?id=R2dQAAAAMAAJ&pgis=1.Google Scholar
- Checkland P. Systems thinking, systems practice. 1st. ed. Wiley J, editor. Systems thinking systems practice. Chichester; 1981. 356 p.
- Checkland P, Scholes J. Soft systems methodology in action. NY: Wiley; 1999. p. 418.
- Bertelsen P, Madsen I. “Participatorisk stuegangsanalyse med fokus på journalanvendelse: medarbejderindflydelse på arbejdsorganisering før og efter EPJ” (Participatory ward-analysis with focus on how to use the EHR: the degree of user- influence on the organization of work befor. In: Nøhr C, Høstgaard AM, editors. Håndbog i organisatoriske forandringer i forbindelse med implementering af elektroniske patientjournaler på sygehuse (Handbook in organizational changes when implementing EHR on hospitals). Aalborg: Aalborg University Press; 2004. p. 77–93.
- Nøhr C, Sørensen M, Kushniruk A. Method for testing a CPOE system in the medication process in a cardiology ward. In: Safran C, Reti H, Marin HF, editors. MEDINFO 2010 : Proceedings of the 13th World Congress on Medical Informatics. Cape Town: Ios Press; 2010. p. 183–7.Google Scholar
- Nøhr C, Botin L, Porot M. Hvordan styres operationsprogrammer for patienter. Tidsskr Dansk Sundhedsvaesen. 2006;82(7):244–7.Google Scholar
- Nøhr C, Botin L, Botin L. Methodology for analysis of work practice with video observation. Stud Health Technol Inform [Internet]. 2007;130:291–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17917202, [cited 2016 Nov 8].Google Scholar
- Nøhr C, Boye N. Towards computer supported clinical activity: a roadmap based on empirical knowledge and some theoretical reflections. In: Human, social, and organizational aspects of health information systems [Internet]. 2008. p. 67–83. Available from: http://services.igi-global.com/resolvedoi/resolve.aspx?. doi: 10.4018/978-1-59904-792-8.
- Elliott G. Global business information technology : an integrated systems approach. Exeter: Addison Wesley; 2004. p 503.
- Barber N, Cornford T, Klecun E, Lichtner V, Takian A. User-centered healthcare IT: meaningful or meaningless? 2011. Available from: http://eprints.lse.ac.uk/39256/, [cited 2016 Nov 8].Google Scholar
- Høstgaard AMB. Constructive eHealth evaluation: In: Reshaping medical practice and care with health information systems [Internet]. IGI Global; 2016 [cited 2016 Nov 8]. p. 44–86. Available from: http://services.igi-global.com/resolvedoi/resolve.aspx?. doi: 10.4018/978-1-4666-9870-3.ch003.
- Høstgaard AMB, Bertelsen P, Petersen LS, Nøhr C, et al. Constructive technology assessment for HIT development: learning, feedback and user involvement. In: Billika G, Bygholm A, Dencker M, Fossum M, Galster G, Hartvigsen G, editors. Scandinavian Conference on Health Informatics 2013 [Internet]. Linköping: Linköping University Electronic Press; 2013. p. 33–7. Available from: http://shi2013.org/, [cited 2016 Nov 8].Google Scholar
- Brender J. Handbook of evaluation methods for health informatics [Internet]. 2006. p. 253–320. Available from: http://www.sciencedirect.com/science/article/pii/B9780123704641500113.Google Scholar
- Yusof MM, Papazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform. 2008;77(6):377–85.View ArticlePubMedGoogle Scholar
- Kørnøv L. Public participation. In: Kørnøv L, Thrane M, Remmen A, Lund H, editors. Tools for sustainable development. Aalborg: Aalborg Universitets Forlag; 2007. p. 721–38.Google Scholar
- Mitchell RK, Agle BR, Wood DJ. Toward a theory of stakeholder identification and salience: defining the principle of who and what really counts. Acad Manag Rev. 1997;22(4):853–86.Google Scholar
- Kanstrup AM, Bertelsen P. Participatory IT-support. In: Proceedings of the ninth conference on participatory design: expanding boundaries in design - volume 1 [Internet]. New York: ACM; 2006. p. 87–94. (PDC’06). Available from: http://doi.acm.org/10.1145/1147261.1147275.
- Ehn P. Work-oriented design of computer artifacts. Lawrence Erlbaum Assoc: Umeå; 1988.Google Scholar
- Greenbaum J. PD a personal statement. Commun ACM. 1993;36(6):47.View ArticleGoogle Scholar
- Brandt E. Designing exploratory design games: a framework for participation in participatory design? In: Participatory design conference. 2006.
- Bertelsen P, Kanstrup AM. User innovation management a handbook. 1st ed. Aalborg, Denmark: Aalborg University Press; 2011.Google Scholar
- Nielsen J. 10 Usability heuristics for user interface design [Internet]. 1995 [cited 2016 Nov 20]. Available from: https://www.nngroup.com/articles/ten-usability-heuristics/.
- Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir M-C, et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform [Internet]. 2011;80(12):815–27. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21920809, [cited 2016 Nov 8].View ArticleGoogle Scholar
- Kensing F, Simonsen J, Bodker K. MUST: a method for participatory design. In: Kensing F, Dykstra-Erickson E, Blomberg J, editors. The fourth beinnal conference on participatory design Cambridge. Massachusetts: CPSR; 1996. p. 129–40.
- Kaufman D, Roberts WD, Merrill J, Lai TY, Bakken S. Applying an evaluation tramework tor health information system design, development, and implementation. Nurs Res. 2006;55(2):S37–42.View ArticlePubMedGoogle Scholar
- Clarke K, O’Moore R, Smeets R, Talmon J, Brender J, McNair P, et al. A methodology for evaluation of knowledge-based systems in medicine. Artif Intell Med [Internet]. 1994;6(2):107–21. Available from: http://linkinghub.elsevier.com/retrieve/pii/093336579490040X. [cited 2016 Nov 8].
- Greenhalgh T, Russell J. Why do evaluations of eHealth programs fail? An alternative set of guiding principles. PLoS Med. 2010;7(11):e1000360.
- Harrison MI, Koppel R, Har-Lev S. Unintented consequences of information technologies in health care - an interactive sociotechnical analysis. J Am Med Inform Assoc. 2007;14:542–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Cresswell KM, Worth A, Sheikh A. Actor-network theory and its role in understanding the implementation of information technology developments in healthcare. BMC Med Inform Decis Mak. 2010;10:67.View ArticlePubMedPubMed CentralGoogle Scholar
- Kushniruk AW, Turner P. Who’s users? Participation and empowerment in socio-technical approaches to health IT developments. In: Studies in Health Technology and Informatics. 2011. p. 280–5.Google Scholar
- European Commission. eHealth task force report – redesigning health in Europe for 2020 [Internet]. Luxembourg: European Commission; 2012. Available from: http://www.e-health-com.eu/fileadmin/user_upload/dateien/Downloads/redesigning_health-eu-for2020-ehtf-report2012_01.pdf.Google Scholar
- Stake R. The Art of Case Study Research. Calif: SAGE publishing; 1995. p. 192.
- Nøhr C, Høstgaard AMB, Vingtoft S. Brugererfaringer fra GEPKA-projektet på afdeling M, Århus Sygehus. Aalborg: Virtual Centre for Health Informatics; 2005.Google Scholar
- Scott JT, Rundall TG, Vogt TM, Hsu J. Kaiser permanente’s experience of implementing an electronic medical record: a qualitative study. BMJ Br Med J [Internet]. 2005;331(7528):1313–6. Available from: http://www.jstor.org/stable/25455558.View ArticleGoogle Scholar
- Brinkmann S, Kvale S. InterViews: learning the craft of qualitative research interviewing. Thirdth ed. Los Angeles: SAGE Publications; 2014.Google Scholar
- Muhr T. ATLAS TI 5.0 users guide. manag educ. 2004.Google Scholar
- Høstgaard A, Bertelsen P, Nøhr C. Methods to identify, study and understand end-user participation in HIT development. BMC Med Inform Decis Mak [Internet]. 2011;11(1):57. Available from: http://www.biomedcentral.com/1472-6947/11/57.View ArticleGoogle Scholar
- Høstgaard AM, Bertelsen P, Nøhr C. Evaluering af pilottest af Klinisk Proces “Vieweren”. Aalborg: Virtual Centre for Health Informatics, Aalborg University; 2011.Google Scholar
- Høstgaard AMB, Bertelsen P, Nøhr C. How are clinicians involved in EHR planning? A process analysis case study of a region in Denmark. In: Safran C, Reti S, Marin HF, editors. MEDINFO 2010: Proceedings of the 13th World Congress on Medical Informatics. Cape Town: Ios Press; 2010. p. 121–5.Google Scholar
- Høstgaard AMB. End-User Participation in Health IT Development. In: Belkhamza Z, Wafa SA, editors. Measuring Organizational Information Systems Success [Internet]. Hershey: IGI Global; 2012. [cited 2016 Nov 8]. p. 318–40. Available from: http://services.igi-global.com/resolvedoi/resolve.aspx?. doi: 10.4018/978-1-4666-0170-3.ch016
- Avison D, Fitzgerald G. Effective Technical and Human Implementation of Computer-based System (ETHICS). In: Avison DE, Fitzgerald G, editors. Information system development: methodologies, techniques and tools. Berkshire: The McGraw-Hill Companies; 1995. p. 353–64.Google Scholar
- Simonsen J, Hertzum M. A Regional PD Strategy for EPR Systems: Evidence- Based IT Development. In: Jacucci G, Kensing F, Wagner I, Blomberg J, editors. PDC 2006: Ninth Biennial Conference on Participatory Design, Vol II. CPSR, p.125–128.
- Simonsen J, Hertzum M. Participative design and the challenges of large-scale systems: extending the iterative PD approach. Proc Tenth Anniv Conf Particip Des. 2008;2008:1–10.Google Scholar
- Berg M, Winthereik BR. Waiting for Godot: episodes from the history of patient records. In: Health Information Management: Integrating Information and Communication Technology in Health Care Work. 2003. p. 234.View ArticleGoogle Scholar
- The National Research Ethics Committee. Act on research ethics review of health research projects [Internet]. Copenhagen; 2011. Available from: http://www.nvk.dk/english/act-on-research.
- The Danish Data Protection Agency. The act on processing of personal data [Internet]. Copenhagen; 2014. Available from: https://www.datatilsynet.dk/english/the-act-on-processing-of-personal-data/read-the-act-on-processing-of-personal-data/compiled-version-of-the-act-on-processing-of-personal-data/.