Many preventable diseases such as ischemic heart diseases and breast cancer prevail at a large scale in the general population. Computerized decision support systems are one of the solutions for improving the quality of prevention strategies.
The system called EsPeR (Personalised Estimate of Risks) combines calculation of several risks with computerisation of guidelines (cardiovascular prevention, screening for breast cancer, colorectal cancer, uterine cervix cancer, and prostate cancer, diagnosis of depression and suicide risk). We present a qualitative evaluation of its ergonomics, as well as it's understanding and acceptance by a group of general practitioners. We organised four focus groups each including 6–11 general practitioners. Physicians worked on several structured clinical scenari os with the help of EsPeR, and three senior investigators leaded structured discussion sessions.
The initial sessions identified several ergonomic flaws of the system that were easily corrected. Both clinical scenarios and discussion sessions identified several problems related to the insufficient comprehension (expression of risks, definition of familial history of disease), and difficulty for the physicians to accept some of the recommendations.
Educational, socio-professional and organisational components (i.e. time constraints for training and use of the EsPeR system during consultation) as well as acceptance of evidence-based decision-making should be taken into account before launching computerised decision support systems, or their application in randomised trials.
In France, a considerable number of deaths are due to preventable diseases. Ischemic heart diseases and cerebrovascular diseases still account for more than 80,000 premature deaths and breast cancer, more than 10,000 deaths. The control of the main cardiovascular risk factors, such as hypertension, remains poor, both because of lack of awareness and insufficient blood pressure control . The objectives of breast cancer screening (mammography every 2 years in women aged between 50 and 69 years recommended in French guidelines) are not completely attained either .
In France, the promotion of prevention strategies in the ambulatory setting remains a national challenge. Several initiatives have been undertaken by Public Health authorities and by Health Insurance National Office, to prompt general practitioners to modify their prevention practices and improve their adherence to national guidelines. The development of the EsPeR system (Personalised Estimate of Risks) is one of these actions. It relies on three hypotheses:
1. for a given individual, an accurate estimation of his (her) health risks allows a more pertinent choice of prevention interventions
2. presentation of a patient-customized clinical guideline, will promote its implementation.
3. the combination of risk estimations and patient-specific guidelines embedded in a computerised decision aid can promote physician application of guideline recommendations during consultations and stimulate the communication with the patient.
The objective of this paper is to describe EsPeR, a computerised decision support system designed to help prevention practices by use of risk estimates and guidelines and to present a pilot qualitative evaluation of its ergonomics, understanding and acceptance by general practitioners.
Design of EsPeR
The EsPeR system is designed for health professionals and it is freely accessible on the Internet, after a simple identification process. An interactive decision support system allows an estimation of risks and a presentation of clinical guidelines. A non-interactive part describes the scientific rationale and background of the system (the documentation part) and provides its sources of medical and epidemiological knowledge (the library part). The following functional description focuses on the decision support system.
The decision support system on the EsPeR website includes four primary functions:
1. Presentation of the 10 highest risks of avoidable causes of death according to demographic characteristics (age, sex and region of residence).
2. Estimation of specific risks, using published risk prediction models.
3. Individualized presentation of French guidelines on cardiovascular risk management and cancer screening
4. Printable summary of prevention messages for physicians and patients.
Highest risks of death
For a specific patient, the user is asked to enter the system via a screen that presents the 10 highest risks of avoidable causes of death according to demographic characteristics (age, sex and region of residence) (see figure 1). The total 10-year mortality risk is displayed along with the absolute risk and the relative part of risk for each cause. These probabilities have been computed from 1) the French national statistics on the causes of deaths (along with available explicative variables (age, sex and region of residence) and 2) the French population data given by 1990 national census and intermediate estimates by INSEE (the French national Institute of Statistics and Economics). The probability of death from one cause, at a given age is estimated thanks to the method used by INSEE, based on a the use of fictitious cohort, which allows to estimate prospective probabilities from the transformation of transversal observational data . The aim of this part is to suggest a hierarchy of preventable risks that should be adjusted according to the patient specific risk factors (e.g. consumption of tobacco and alcohol, blood pressure level, etc.).
Specific risks and individualized guidelines
After this first screen the physician can choose the specific risks he(she) wants to further estimate and consider for prevention in his(her) patient (cardiovascular risk, alcohol abuse screening, breast cancer, colorectal cancer, uterine cervix and prostate cancer screening, depression and suicide risk). A data input form is dynamically created with relevant clinical and biological items, once the physician has selected the risks (Figure 2). For breast cancer and cardiovascular disease, the form is integrated with a module which allows to create a family tree, enter disease history for each person in the pedigree and estimate familial risk of the patient  (Figure 3). The validation of the form provides a risk estimate for each selected risk, and a presentation of guidelines.
The method of implementation of guidelines has been previously described . It relies on a generic guideline representation model derived from the GLIF 2 (Guideline Interchange Format) model , and the use of XML as a language of representation for any specific guideline, according to a specific DTD (document type definition) of this model. An inference engine implements the guideline model and executes any specific guideline represented in XML.
All original guidelines have been analysed to specify a decision algorithm, represented in XML language, and a list of elementary messages (e.i. extracts of guideline for a given patient profile). Two physicians experienced in medical informatics performed these analysis and specifications. For a given patient, the presentation of guideline consists in 1) the list of variables (i.e. the patient's characteristics) used in the decision algorithm, and 2) the successive elementary recommendations related to his(her) characteristics.
All knowledge used in the system has been selected on the basis of extensive validation (according to evidence-based medicine methodology). Cardiovascular risks and breast cancer risk are computed thanks to published risk prediction models [7, 8]. We considered the Framingham models as sufficiently validated for cardiovascular risk stratification in various cohorts, including French ones [9–11]. Since the Gail model for breast cancer has never been tested on any French data, we chose to only present an estimate of relative risk. Provided guidelines have been selected from those published by the French National Agency for Health Evaluation (ANAES) for:
• Diagnosis and management of hypertension, hyperlipidemia, and type 2 diabetes
• Screening for breast cancer, colorectal cancer, cervical cancer, and prostate cancer
• Diagnosis of depression and suicide risk
• Diagnosis and management of alcohol abuse.
Finally, a printable summary of all risk estimates and guidelines is made accessible for both physician and patient, with words and format adapted to each.
The EsPeR system is based on a 3-tier client-server architecture. The concepts used by the decision support modules (patient data, risks estimates) are implemented in JAVA classes; these classes encapsulate relational databases with SQL queries and generate with the results HTML pages by using Active Server Pages technology.
Following published recommendations , we designed a pilot qualitative study to evaluate the ergonomics, understanding and acceptance of the interactive decision support system of EsPeR by general practitioners. The main objective of this preliminary evaluation in laboratory was to appreciate how general practitioners could use and react to EsPeR and to go in depth with them into their reactions and beliefs. A secondary objective of this study was to prepare a larger qualitative study to be organised in ambulatory practice, with use of EsPeR during consultation.
The study was performed in two sessions, each consisting in two focus groups. The first session was organised in January 2002 with the version 1 of the system and the second one in October 2002, with the version 2, improved according to several results of the first session. The differences between the first version and the second one concern the number of provided guidelines and the global ergonomics and navigation within the site. All focus groups were animated by three investigators (GC, IC and TD for the first and second ones, GC, IC and LL for the third and fourth ones)
Each focus group was divided into four phases: 1) a structured discussion to evaluate skills and culture on prevention (at the beginning of the session) 2) a presentation of the project and short training to EsPeR (1 hour), 3) an individual work on 6 to 12 structured clinical scenarios with or without EsPeR (2 to 4 hours) and 4) a structured discussion to draw a consensus synthesis on the evaluation of EsPeR (at the end of the session) (1 to 2 hours).
The same investigator (GC) conducted discussion during phases 1 and 4 in all groups, according to outlines presented in Table 1, while the others (IC and TD for the 1st session in January and IC and LL for the 2nd session in October) took notes of the discussion. The later were asked to transcribe original comments of physicians as faithfully as possible, rather than their own interpretations. Both topics of discussion (prevention and synthesis of evaluation) were also addressed thanks to a structured individual questionnaire filled out by physicians before the corresponding discussion.
During the second phase, the three investigators observed the physicians' behaviour in using the system. They collected physicians' remarks, difficulties and suggestions, along with their personal comments or opinion on these reactions. Structured clinical scenarios were formulated according to a priori hypotheses on physician's practices or beliefs, which could be in contradiction with the scientific content of EsPeR. For example, as current French guideline recommend systematic mammography screening of breast cancers in women aged 50 to 70 years, we depicted a scenario of a 43 aged women with average personal risk of breast cancer according to the Gail model (particularly, with one history of breast cancer in her mother, diagnosed at the age of 70). Physicians were asked to answer to each scenario on a structured form, about what would be their estimates of health risks and prevention strategies, whether concordant or not with EsPeR, used as decision aid.
Recruitment of general practitioners
Physicians were recruited differently for both sessions. For the January session, we requested volunteers in two different professional groups by the intermediate of the executive manager of each group (the National French Federation of Mutuality, which manages non for profit health centres with full-time of part time salaried GPs, and the National College of Teaching GPs). For the October session, we recruited volunteers by a mail shot to physicians who practice in two administrative departments of France (n = 437 in Indre et Loire, n = 135 in Seine et Marne) and who already participated in a prevention program organized by the CANAM (Health Insurance National Office for non salaried workers). Volunteers were asked to have minimal computer and Internet skills. In each focus group, the participants were not supposed to know each other and had no working contacts with the investigators before the sessions. All GPs were paid for their participation in the study.
Data collection and analysis
Demographic and professional characteristics of participants were collected by a questionnaire mailed to each participant before the session.
Evaluation data were collected:
1. During the discussion phases: from discussion notes collected by two investigators
2. During individual work on clinical scenarios: from standardized structured answer form filled out by each physician for all scenarios, with open commentaries and observation notes from the three investigators
3. Using two structured questionnaires evaluating the degree of knowledge (beginning of the session) and the satisfaction of physicians with EsPeR (end of the session).
Observations and discussion notes collected by the different investigators were all pooled together, with indication of the origin of each item (original comments by physicians or interpretation by investigators). They were summarized by one investigator (IC) and validated by two others (by GC and TD in January then by GC and LL in October). This qualitative verbatim of the focus group was further classified into:
• Ergonomics of the system (navigation, speed, intuitiveness)
• Understanding of contents
• Acceptance of advices (barriers and facilitators to put guidelines into practice).
We also checked the consistency of this verbatim with answers to questionnaires and we tried to infer from this verbatim some pragmatic interpretation potentially leading to concrete improvement of the EsPeR system.
Finally, we analysed the answers to each individual clinical scenario as concordant or discordant with the answers recommended in EsPeR. We expressed this concordance in terms of "acceptance" ("not acceptable": no concordance with the recommendation of EsPeR and negative comments from physicians; "intermediate": no concordance but no negative comments or concordance but negative comments; "acceptable": concordance with the recommendation of EsPeR and positive comments). We tried to explain this acceptance in the light of the verbatim of the focus group.
The four focus groups were performed respectively with a total of 36 participants (11 and 11 in January session, 8 and 6 in October session). Mean age was 48 years, 89% were male and 80% used electronic medical record software. The four groups were different in terms of professional setting and use of informatics (Table 2).
Problems of ergonomics
Several problems of ergonomics were clearly identified with the two first groups in January (both through observation by investigators and reports by physicians). These problems concerned:
Navigation in the family tree module
Navigation between data forms and results (risk and guidelines)
Absence of archive of medical record and absence of an export function to avoid duplicate data entry (in EsPeR and in personal EMR software)
Some solutions to these problems were developed for the version 2 and GPs of the October sessions found the ergonomics and navigation correct, easy to use and fast enough (see Table 3). Physicians needed 15 to 20 minutes to work on one clinical scenario with EsPeR.
Problems of understanding
The physicians' understanding of risks was highly variable. Several of them were not familiar with a quantitative estimate of risk. They did not know how to apply the absolute risk provided in EsPeR and asked for explanations. Few participants asserted being more confident in their own risk estimate. Few GPs did not know clinical practice guidelines. Most had heard about them but did not refer to them, and few both read and used them routinely (Table 3).
Acceptance of EsPeR content
The acceptance of the guideline messages was evaluated from GPs answers to clinical scenarios (adherence or no adherence to the messages), from their reactions observed during individual work and from the comments reported in discussion notes during open group discussion (Table 3). The cardiovascular guidelines were in general better accepted than guidelines on cancer screening. Several recommendations given by EsPeR concerning cancers screening were perceived as inappropriate because of patients' demands and social or organizational pressures.
Overall, physicians perceived the system as useful. Few participants would have liked to be guided more directly to select the risks that should be considered in priority for prevention for a given patient.
Table 4 presents examples of the acceptance of some recommendations of EsPeR, for different clinical situations addressed in scenarios, along with plausible explanations and solutions inferred from discussion notes and from physicians' comments.
The present paper describes a computerised system devoted to the prevention of avoidable morbidity, based on risk assessment and valid recommendations. The main objective of EsPeR is to put in perspective the different health risks for one individual and to present guideline messages appropriate for this individual. The hierarchy of risks currently relies on the presentation of French mortality data and is therefore poorly "individualized", taking into account age, sex and region of residence. A more accurate estimate of several risks is possible thanks to specific models such as the Framingham model. However, these models have been demonstrated as being adequately discriminent but poorly calibrated in French population [10, 13]. Other models have been recently published thanks to the SCORE project and are recommended by the Third Joint Task Force of European Societies on Cardiovascular prevention [14, 15]. We intend to integrate them in EsPeR. We also experienced difficulties in implementing the French guidelines on cardiovascular prevention because of the heterogeneity of recommended tools for risk stratification.
We ran a pilot qualitative study in laboratory, in order to observe physicians' reaction and behaviour while using EsPeR and to find with them some explanations. We interpreted the collected verbatim and observations as potential barriers and facilitators for the diffusion of this tool. No focus group session was recorded or videotaped and no social scientist was involved in the study: the groups were animated by three different investigators among whom one was not involved directly in the design of EsPeR and supposed to be "neutral". Recruitment was different for both sessions (in January and October) and led to very different groups which interestingly suggested that different kinds of reactions may pertain to different backgrounds (socio-professional environment, use of informatics, etc.).
Despite these limitations, ergonomics problems were easily identified thanks to the first two focus groups. Corrective measures were taken in the version 2 and led to solve these problems, as verified with the second session of focus groups. The main problems pertained to the content of EsPeR and presentation of knowledge. Analysis of these results led to several actions. The first one consisted in giving a feedback to the ANAES (the French national agency in charge of the development of guidelines) on difficulties encountered while computerizing guidelines. This resulted in a partnership between our laboratory and the ANAES to set up a guideline development framework designed for the guideline authors, in order for them to write directly and unequivocally "computerizable" guidelines. Our results also suggest that improving the quality of preventive strategies in the primary care setting requires considering the multiple competing demands faced by patients and physicians during each consultation. The promotion of the EsPeR system, designed for assisting physicians in setting rational priorities in prevention strategies, should therefore include both an educational intervention (i.e. teaching evidence and the concept of risk for increasing physician's confidence in recommendations) and an organisational intervention (i.e. increase of the consultation duration to allow the complete use of the computerised decision support system, or promote the delegation of certain tasks to nurse practitioners).
The key assumption made by people developing decision aids is that providing decision support (recommendations, alerts, reminders, calculations) as well as useful clinical information at the point of care can reduce unwarranted variations in care and improve patients' health. It is generally acknowledged that computerised reminders and alerts improve the practice of simple preventive measures (screening, vaccinations...) . However, more complex decision support systems such as computerised guidelines , or calculation of cardiovascular risk  did not improved quality of care in several recent well designed randomised trials. These findings can have several explanations: either the system does not work, or it is not used because it does not fit the physicians' needs or else it is in conflict with physicians beliefs and constraints. In keeping with this idea, some authors encourage evaluating computerised decision support systems not only with randomised trials . In some examples, a qualitative study helps to explain the negative of poor results of a randomised controlled trial [17, 20]. We therefore plan a qualitative on-field evaluation of the system to assess its usability and its acceptance by general practitioners on larger scale, in two different regions of France, before considering evaluating its impact in a randomised trial.
We presented an original computerized decision support system for medical prevention based on concepts like risks and complex knowledge found in clinical practice guidelines. The pilot qualitative evaluation of this system reinforces the need of training on its content, prior to its on-field evaluation. The main perspective of this work is the design of a larger qualitative study to evaluate the utilization and barriers to acceptance of the EsPeR system in the day to day practice. Potential important implication of this project could be the use of EsPeR as a structuring tool for prevention strategies in ambulatory medicine.
Lang T, de Gaudemaris R, Chatellier G, Hamici L, Diene E: Prevalence and therapeutic control of hypertension in 30,000 subjects in the workplace. Hypertension. 2001, 38: 449-454.
Giorgi R, Gouvernet J, Jougla E, Chatellier G, Degoulet P, Fieschi M: The use of the personalized estimate of death probabilities for medical decision making. Comput Biomed Res. 2000, 33: 75-83. 10.1006/cbmr.1999.1531.
Ohno-Machado L, Gennari JH, Murphy SN, Jain NL, Tu SW, Oliver DE, Pattison-Gordon E, Greenes RA, Shortliffe EH, Barnett GO: The guideline interchange format: a model for representing guidelines. J Am Med Inform Assoc. 1998, 5: 357-372.
Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ: Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst. 1989, 81: 1879-1886. 10.1093/jnci/81.24.1879.
Dréau H, Colombet I, Degoulet P, Chatellier G: Identification of patients at high cardiovascular risk: a critical appraisal of applicability of statistical risk prediction models. Meth Inform Med. 2001, 40: 6-11.
Bastuji-Garin S, Deverly A, Moyse D, Castaigne A, Mancia G, de Leeuw PW, Ruilope LM, Rosenthal T, Chatellier G: The Framingham prediction rule is not valid in a European population of treated hypertensive patients. J Hypertens. 2002, 20: 1973-1980. 10.1097/00004872-200210000-00016.
D'Agostino R. B., Sr., Grundy S, Sullivan LM, Wilson P: Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. Jama. 2001, 286: 180-187.
Giacomini MK, Cook DJ: Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 2000, 284: 357-362. 10.1001/jama.284.3.357.
Conroy RM, Pyorala K, Fitzgerald AP, Sans S, Menotti A, De Backer G, De Bacquer D, Ducimetiere P, Jousilahti P, Keil U, Njolstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM: Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003, 24: 987-1003. 10.1016/S0195-668X(03)00114-3.
De Backer G, Ambrosioni E, Borch-Johnsen K, Brotons C, Cifkova R, Dallongeville J, Ebrahim S, Faergeman O, Graham I, Mancia G, Manger Cats V, Orth-Gomer K, Perk J, Pyorala K, Rodicio JL, Sans S, Sansoy V, Sechtem U, Silber S, Thomsen T, Wood D: European guidelines on cardiovascular disease prevention in clinical practice. Third Joint Task Force of European and Other Societies on Cardiovascular Disease Prevention in Clinical Practice. Eur Heart J. 2003, 24: 1601-1610. 10.1016/S0195-668X(03)00347-6.
Shea S, DuMouchel W, Bahamonde L: A meta-analysis of 16 randomized controlled trials to evaluate computer- based clinical reminder systems for preventive care in the ambulatory setting [see comments]. J Am Med Inform Assoc. 1996, 3: 399-409.
Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I: Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002, 325: 941-10.1136/bmj.325.7370.941.
Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ: Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000, 320: 686-690. 10.1136/bmj.320.7236.686.
Kaplan B: Evaluating informatics applications--some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inf. 2001, 64: 39-56. 10.1016/S1386-5056(01)00184-8.
Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M: Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. Bmj. 2003, 326: 314-10.1136/bmj.326.7384.314.
This project has been developed thanks to a grant by the Caisse Nationale d'Assurance Maladie des Travailleurs Indépendants (CANAM). We thank Catherine Wajs for her help in revising the manuscript and all other members of the EsPeR Group for their contribution to the project: Pr P Degoulet, Pr M Fieschi, Dr J Gouvernet, Dr R Giorgi, Dr JC Dufour, E Jougla, E Michel, Y Xu.
Authors and Affiliations
SPIM, Broussais Hotel Dieu University, ERM 202, INSERM, Paris, France
IC, GC, and JM wrote the manuscript. IC and GC designed and organized the study. IC, TD, LL and GC animated the focus groups. SZ ensured software development and maintenance for the EsPeR project and followed the focus group for technical support. The EsPeR Group collaborated to the design of the software
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
Colombet, I., Dart, T., Leneveut, L. et al. A computer decision aid for medical prevention: a pilot qualitative study of the Personalized Estimate of Risks (EsPeR) system.
BMC Med Inform Decis Mak3, 13 (2003). https://doi.org/10.1186/1472-6947-3-13