An e-mail based, three-round decision Delphi study was performed to generate responses and achieve consensus in a selected sample of nationally recognized eHealth experts. Informed consent was obtained from each participant and approval was obtained from Slovene National Ethical Committee.
The Delphi survey was developed in the 1950s. It is a structured and multistage process where a panel of experts are invited to take part in a series of rounds to identify, clarify, and finally achieve consensus on a particular issue. Each subsequent set of non-leading, unambiguous statements is built on the responses to the preceding ones. Consensus is sought through the feedback of information and iteration, and the process is terminated when consensus is reached.
Anonymity offered by Delphi can reduce the inhibition normally occurring in decision-making as individuals are more open with their answers .
There were two main reasons to choose Delphi method as our research tool. Firstly, the Delphi technique has been found effective in the past in raising and measuring group consensus about medical information technology usage within health care [25–29]. Secondly, the outcome in the decision Delphi is focused on decision making in fields that are strongly susceptible to change and where one or more of the following occurs: (1) there is more influence by individual decision makers than by underlying rules; (2) the field of interest is relatively new or is driven by new developments; or (3) the scientific field of interest is small and relatively self-contained. The panel should include a high percentage of decision makers in the considered field [30, 31].
In primary healthcare as well as in eHealth it is often the method of choice for developing consensus on national level [10, 32–37].
Organization of the study
The monitoring of the study was performed by a steering committee that identified the experts to be invited to participate in the Delphi process and that also oversaw the project.
The project group, consisting of two experts from PHC and two medical informatics (MI) specialists, prepared and analyzed the questionnaires. Communication with participants was performed via e-mail with anonymity of written answers assured.
The sample included two groups of experts. The selection criteria were:
At least 10 years of professional work in primary health care or 5 years in medical informatics,
High position in a professional hierarchy (professional association or institution), and/or
Adequate knowledge of the organization of Slovenian PHC.
The estimate of sample size was based on literature review and reports from similar studies [10, 31–33, 35, 36, 38, 39]. We anticipated approximately an 80% response rate from 13 invited experts in the both fields - of PHC and medical informatics - this would ensure the required significance.
PHC experts were recommended by the Department of Family Medicine, Medical Faculty, University of Ljubljana. Medical informatics specialists were recommended by members of the Healthcare Informatics Council, an advisory board to the Slovene Ministry of Health. All recommendations were approved by the steering committee, that checked their references.
Delphi questionnaire development
The first-round questionnaire consisted of semi-structured questions. They were generated based on a literature review performed by a group of experts and from the input of seven focus groups. These focus groups included: three PHC physician groups, two nurse groups and two patient groups.
Questions were grouped into three main themes:
1) Availability and use of contemporary information technology in family medicine practice, with next sub-themes: "an expected level of computerization in PHC practices" and "the role of state institutions".
2) Expectations from contemporary ICT, with sub-themes: "quality of health care services", "health care costs", "satisfaction of medical staff and users of health care services", "paperless healthcare" and "data management and safety of medical data".
3) Computer supported decision-making, electronic communication with patients and between health care levels with sub-themes: "e-communication between patient and health care provider", "e-communication among different health care providers" and "computer supported decision-making".
The draft questionnaire was given to a group of four experts for comments (two from PHC and two from medical informatics). They were asked to rate the questions in two dimensions, clarity and importance, and to comment on the questions. The questions acceptable to all experts were included in the first questionnaire.
The questions were in the form of statements and levels of agreement with the statements were assessed on a 1 to 9 Likert scale.
The Delphi consensus process
Steering committee planned at least three Delphi rounds. Additional rounds were to be performed in case of less than 80% agreement on all items after round three. Agreement for single items was defined by: (1) median value of six or higher, (2) inter-quartile rank (IR) of three or less, and (3) no statically significant differences between MI and PHC subgroup of experts.
The main task of all invited experts was to assess the clarity and importance of each question. This included comments about the questions, including particular terminology and vocabulary related to ICT, medical and administrative terms.
The first round of questionnaires was analyzed using simple statistics: mean, standard deviation (SD), and rank. After this analysis, the questionnaire was modified according to the following criteria.
The exclusion criteria for a statement were: (1) median - all expert ranks lower than five; or (2) IR - higher than five; or (3) median - lower than six, with IR equal or higher than four.
The criteria for modification were: (1) median - six or seven, with IR from three to five with relevant written arguments from the panellists; (2) same meaning of the question; (3) consensus of all strategic project group members; and (4) no limitation on the number of modified questions.
The criteria for adding new questions after the first round were the following: (1) only questions developed by first round participants, which had been reviewed and confirmed by the strategic project group; and (2) no more than 15% of the total number of first-round questions.
Criteria for previously defined exclusion, modification and addition applied to both assessment areas, namely clarity and importance of each statement.
The second round questionnaires were also sent by e-mail. The experts were asked to assess the importance and feasibility of each suggested measure and to complete a short demographic survey. Levels of agreement were again assessed on a 1 to 9 Likert scale. No feedback was provided concerning the results of the first round. The experts that didn't return the questionnaire were reminded by phone after 10 days. The responses were analyzed in order to develop the third questionnaire (except for newly generated questions), based on interquartile range (IR), median, and mean. The criteria for the exclusion or modification of a particular question after the second round were the same as in the first round. In addition, the number of modified questions was limited to 10.
The criteria for the addition of new questions after the second round were the following: (1) relevant arguments by second-round panelists on a particular topic; (2) consensus of all strategic project group members; and (3) limited to 10% of all second-round questions.
In the third round, the participants were asked to review their responses from the previous round and to contrast them using consensus results of the group. The questionnaire was also accompanied with statistical data and comments from the second round. The following information was included: (1) median; (2) consensus; and (3) IR. The respondents' initial ratings in the second round questionnaire were highlighted. They could then change their initial rating. The participants whose score on any statement was outside the group consensus were asked to briefly explain the reasons for their position.
Two methods were used to assess consensus building. A rating-scale was used to indirectly measure agreement (based on IR) and inter-round comparison of results. Comparison of subgroup results for each question was also performed using non-parametric statistical technique (Wilcoxon rank sum test).
After the last round, descriptive statistics was used to analyze the final results [40, 41].
In the second step differences between subgroups of PHC professionals and medical informatics experts were analysed using the Wilcoxon rank sum test.
The third step of the analysis involved Cronbach's alpha analysis to determine the internal consistency of answers in both rounds. Finally, the Wilcoxon signed-rank sum test was used to determine the overall success of the Delphi process. Statistical differences on the importance and feasibility of all rated statements in the second and third round were analyzed.
All quantitative data analyses were performed using Data Analysis Plus 3.0 Statistical Software ad-ins for Microsoft Excel .
After the descriptive statistical analysis of answers to third round statements, they were ranked by order according to median, IR, SD and mean. The classification was made on both rating dimensions, namely importance and feasibility.