This article has Open Peer Review reports available.
A computer decision aid for medical prevention: a pilot qualitative study of the Personalized Estimate of Risks (EsPeR) system
© Colombet et al; licensee BioMed Central Ltd. 2003
Received: 02 July 2003
Accepted: 27 November 2003
Published: 27 November 2003
Many preventable diseases such as ischemic heart diseases and breast cancer prevail at a large scale in the general population. Computerized decision support systems are one of the solutions for improving the quality of prevention strategies.
The system called EsPeR (Personalised Estimate of Risks) combines calculation of several risks with computerisation of guidelines (cardiovascular prevention, screening for breast cancer, colorectal cancer, uterine cervix cancer, and prostate cancer, diagnosis of depression and suicide risk). We present a qualitative evaluation of its ergonomics, as well as it's understanding and acceptance by a group of general practitioners. We organised four focus groups each including 6–11 general practitioners. Physicians worked on several structured clinical scenari os with the help of EsPeR, and three senior investigators leaded structured discussion sessions.
The initial sessions identified several ergonomic flaws of the system that were easily corrected. Both clinical scenarios and discussion sessions identified several problems related to the insufficient comprehension (expression of risks, definition of familial history of disease), and difficulty for the physicians to accept some of the recommendations.
Educational, socio-professional and organisational components (i.e. time constraints for training and use of the EsPeR system during consultation) as well as acceptance of evidence-based decision-making should be taken into account before launching computerised decision support systems, or their application in randomised trials.
In France, a considerable number of deaths are due to preventable diseases. Ischemic heart diseases and cerebrovascular diseases still account for more than 80,000 premature deaths and breast cancer, more than 10,000 deaths. The control of the main cardiovascular risk factors, such as hypertension, remains poor, both because of lack of awareness and insufficient blood pressure control . The objectives of breast cancer screening (mammography every 2 years in women aged between 50 and 69 years recommended in French guidelines) are not completely attained either .
In France, the promotion of prevention strategies in the ambulatory setting remains a national challenge. Several initiatives have been undertaken by Public Health authorities and by Health Insurance National Office, to prompt general practitioners to modify their prevention practices and improve their adherence to national guidelines. The development of the EsPeR system (Personalised Estimate of Risks) is one of these actions. It relies on three hypotheses:
1. for a given individual, an accurate estimation of his (her) health risks allows a more pertinent choice of prevention interventions
2. presentation of a patient-customized clinical guideline, will promote its implementation.
3. the combination of risk estimations and patient-specific guidelines embedded in a computerised decision aid can promote physician application of guideline recommendations during consultations and stimulate the communication with the patient.
The objective of this paper is to describe EsPeR, a computerised decision support system designed to help prevention practices by use of risk estimates and guidelines and to present a pilot qualitative evaluation of its ergonomics, understanding and acceptance by general practitioners.
Design of EsPeR
The EsPeR system is designed for health professionals and it is freely accessible on the Internet, after a simple identification process. An interactive decision support system allows an estimation of risks and a presentation of clinical guidelines. A non-interactive part describes the scientific rationale and background of the system (the documentation part) and provides its sources of medical and epidemiological knowledge (the library part). The following functional description focuses on the decision support system.
The decision support system on the EsPeR website includes four primary functions:
1. Presentation of the 10 highest risks of avoidable causes of death according to demographic characteristics (age, sex and region of residence).
2. Estimation of specific risks, using published risk prediction models.
3. Individualized presentation of French guidelines on cardiovascular risk management and cancer screening
4. Printable summary of prevention messages for physicians and patients.
Highest risks of death
Specific risks and individualized guidelines
The method of implementation of guidelines has been previously described . It relies on a generic guideline representation model derived from the GLIF 2 (Guideline Interchange Format) model , and the use of XML as a language of representation for any specific guideline, according to a specific DTD (document type definition) of this model. An inference engine implements the guideline model and executes any specific guideline represented in XML.
All original guidelines have been analysed to specify a decision algorithm, represented in XML language, and a list of elementary messages (e.i. extracts of guideline for a given patient profile). Two physicians experienced in medical informatics performed these analysis and specifications. For a given patient, the presentation of guideline consists in 1) the list of variables (i.e. the patient's characteristics) used in the decision algorithm, and 2) the successive elementary recommendations related to his(her) characteristics.
All knowledge used in the system has been selected on the basis of extensive validation (according to evidence-based medicine methodology). Cardiovascular risks and breast cancer risk are computed thanks to published risk prediction models [7, 8]. We considered the Framingham models as sufficiently validated for cardiovascular risk stratification in various cohorts, including French ones [9–11]. Since the Gail model for breast cancer has never been tested on any French data, we chose to only present an estimate of relative risk. Provided guidelines have been selected from those published by the French National Agency for Health Evaluation (ANAES) for:
• Diagnosis and management of hypertension, hyperlipidemia, and type 2 diabetes
• Screening for breast cancer, colorectal cancer, cervical cancer, and prostate cancer
• Diagnosis of depression and suicide risk
• Diagnosis and management of alcohol abuse.
Finally, a printable summary of all risk estimates and guidelines is made accessible for both physician and patient, with words and format adapted to each.
The EsPeR system is based on a 3-tier client-server architecture. The concepts used by the decision support modules (patient data, risks estimates) are implemented in JAVA classes; these classes encapsulate relational databases with SQL queries and generate with the results HTML pages by using Active Server Pages technology.
Following published recommendations , we designed a pilot qualitative study to evaluate the ergonomics, understanding and acceptance of the interactive decision support system of EsPeR by general practitioners. The main objective of this preliminary evaluation in laboratory was to appreciate how general practitioners could use and react to EsPeR and to go in depth with them into their reactions and beliefs. A secondary objective of this study was to prepare a larger qualitative study to be organised in ambulatory practice, with use of EsPeR during consultation.
The study was performed in two sessions, each consisting in two focus groups. The first session was organised in January 2002 with the version 1 of the system and the second one in October 2002, with the version 2, improved according to several results of the first session. The differences between the first version and the second one concern the number of provided guidelines and the global ergonomics and navigation within the site. All focus groups were animated by three investigators (GC, IC and TD for the first and second ones, GC, IC and LL for the third and fourth ones)
Each focus group was divided into four phases: 1) a structured discussion to evaluate skills and culture on prevention (at the beginning of the session) 2) a presentation of the project and short training to EsPeR (1 hour), 3) an individual work on 6 to 12 structured clinical scenarios with or without EsPeR (2 to 4 hours) and 4) a structured discussion to draw a consensus synthesis on the evaluation of EsPeR (at the end of the session) (1 to 2 hours).
Outlines for structured discussion in focus groups
Discussion on prevention
Physician point of view
• Ergonomics and adoption of the computer decision support system (by main functionalities)
What is prevention?
Knowledge on risk (vocabulary, interpretation)
• Patient medical record (PMR)
Knowledge on guidelines
Current use of guidelines
Part of prevention in their practice
Interoperability with personal practice PMR
• Patient point of view (as perceived by physicians)
• Presentation of risks
Expectations for prevention
Were they puzzled by risk estimates?
Attitudes and adherence to the process of risk control
Perceived utility of risk estimates for the care of individual patients
• Expectations of physicians for computer decision support systems:
• Presentation of guidelines
Perceived appropriateness of advices in GPs field practice
For medical decision making
To be usable while facing the patient
• Printable synthesis for patient
For communication with patients
Usability for communication with patients
For continuous medical education
During the second phase, the three investigators observed the physicians' behaviour in using the system. They collected physicians' remarks, difficulties and suggestions, along with their personal comments or opinion on these reactions. Structured clinical scenarios were formulated according to a priori hypotheses on physician's practices or beliefs, which could be in contradiction with the scientific content of EsPeR. For example, as current French guideline recommend systematic mammography screening of breast cancers in women aged 50 to 70 years, we depicted a scenario of a 43 aged women with average personal risk of breast cancer according to the Gail model (particularly, with one history of breast cancer in her mother, diagnosed at the age of 70). Physicians were asked to answer to each scenario on a structured form, about what would be their estimates of health risks and prevention strategies, whether concordant or not with EsPeR, used as decision aid.
Recruitment of general practitioners
Physicians were recruited differently for both sessions. For the January session, we requested volunteers in two different professional groups by the intermediate of the executive manager of each group (the National French Federation of Mutuality, which manages non for profit health centres with full-time of part time salaried GPs, and the National College of Teaching GPs). For the October session, we recruited volunteers by a mail shot to physicians who practice in two administrative departments of France (n = 437 in Indre et Loire, n = 135 in Seine et Marne) and who already participated in a prevention program organized by the CANAM (Health Insurance National Office for non salaried workers). Volunteers were asked to have minimal computer and Internet skills. In each focus group, the participants were not supposed to know each other and had no working contacts with the investigators before the sessions. All GPs were paid for their participation in the study.
Data collection and analysis
Demographic and professional characteristics of participants were collected by a questionnaire mailed to each participant before the session.
Evaluation data were collected:
1. During the discussion phases: from discussion notes collected by two investigators
2. During individual work on clinical scenarios: from standardized structured answer form filled out by each physician for all scenarios, with open commentaries and observation notes from the three investigators
3. Using two structured questionnaires evaluating the degree of knowledge (beginning of the session) and the satisfaction of physicians with EsPeR (end of the session).
Observations and discussion notes collected by the different investigators were all pooled together, with indication of the origin of each item (original comments by physicians or interpretation by investigators). They were summarized by one investigator (IC) and validated by two others (by GC and TD in January then by GC and LL in October). This qualitative verbatim of the focus group was further classified into:
• Ergonomics of the system (navigation, speed, intuitiveness)
• Understanding of contents
• Acceptance of advices (barriers and facilitators to put guidelines into practice).
We also checked the consistency of this verbatim with answers to questionnaires and we tried to infer from this verbatim some pragmatic interpretation potentially leading to concrete improvement of the EsPeR system.
Finally, we analysed the answers to each individual clinical scenario as concordant or discordant with the answers recommended in EsPeR. We expressed this concordance in terms of "acceptance" ("not acceptable": no concordance with the recommendation of EsPeR and negative comments from physicians; "intermediate": no concordance but no negative comments or concordance but negative comments; "acceptable": concordance with the recommendation of EsPeR and positive comments). We tried to explain this acceptance in the light of the verbatim of the focus group.
Characteristics of participants (figures are numbers)
Group A (n = 11)
Group B (n = 11)
Group A (n = 8)
Group B (n = 6)
Age, median (IQR)
Sex (number of males)
Use of computer during consultation
Use of an electronic medical record (EMR)
Use of EMR during consultation
Use of decision support systems for diagnosis
Use of decision support systems for prescription
Problems of ergonomics
Several problems of ergonomics were clearly identified with the two first groups in January (both through observation by investigators and reports by physicians). These problems concerned:
Navigation in the family tree module
Navigation between data forms and results (risk and guidelines)
Absence of archive of medical record and absence of an export function to avoid duplicate data entry (in EsPeR and in personal EMR software)
Extracts of discussion notes and written open comments by participating physicians
Discussion notes on prevention
"individual prevention or population prevention?"
"to inform the patient"
"to educate the patient for health: nutrition, etc."
"cancer prevention consists in regular and systematic exam, more specifically according to particular risks of patients"
"a lot of time !" "the activity of prevention is guided more by a the reasons of consultations or the circumstances than by any structured prevention plan."
Demand of patients:
"there is an harmful role of the medias: sometime, patients are informed before we are"
"patients would accept messages of prevention if we had more time with them"
"there are too many of them" "we lack time to read them"
"we can agree with them but not use them"
"general practitioners are not involved enough in their development" "guidelines avoid to do bibliography and the ANAES is independent from Health Insurance institutions"
"Prevention is included in the time of regular consultation, according to its context"
"to use appropriate means to avoid accidents and diseases"
"a specific consultation for prevention would be twice longer than regular consultation"
"ANAES guidelines are more acceptable for us when we have the opportunity to work on them in our continuous medical education group"
"we trust the ANAES guidelines more than experts opinions"
"there are a strong pressure by the media on patients concerning PSA screening"
Written open comments on clinical scenario answer forms
"the keyboarding is too long in the family history module"
"EsPeR does not take into account the treatment of risk factors"
"Two myocardial infarctions and one stroke in the family (uncles and aunts) do not constitute a familial risk"
"there is no estimated familial risk despite her mother had breast cancer"
"I will prescribe a mammography, even though EsPeR tells me it is not recommended."
"Don't forget that we care of a patient who does not care about his(her) probability or about the cost of his(her) screening test"
"EsPeR helps to be aware of intensity of risks",
"EsPeR helps to balance risk of cardiovascular diseases and risk of cancer",
"I thought I knew the guidelines... finally: I don't",
"I did not find any answer with EsPeR"
"We cannot refuse a mammography screening to a 43 years aged women, even though screening is recommended at 50 years of age."
"There is a strong incentive from laboratories to use PSA for systematic prostate cancer screening" * (it is not recommended by guideline in EsPeR)
"I am questioned in my idea on the pertinence of screening"
Discussion notes on evaluation synthesis
"EsPeR is not ready for use in consultation"
"This tool is not appropriate for daily practice"
"EsPeR is easy to use, but difficult to integrate in one consultation"
"EsPeR allow to learn"
"Criteria used to define risk are too strict. In practice, we use fuzzy criteria"
"We trust our clinical experience to estimate risks" "the statistical truth is not the clinical truth"
"I need to adapt guideline to my personal practice"
"cognitive interest of the individualisation of guidelines brought at the moment of decision"
"I was able to integrate the concepts of risk only when I started working with EsPeR.
"I learned something today...I have practiced prevention for 23 years, only based on my common sense"
"The presentation of mortality data provides an interesting tool to communicate with the patient"
"We need a framework to interpret risks or some labels; we do not know how to relate cardiovascular risk with mortality risk"
"more warnings and more active alerts are needed from the estimations of risks"
"highlight individual messages better in guideline messages"
"we often overestimate familial risks"
Problems of understanding
The physicians' understanding of risks was highly variable. Several of them were not familiar with a quantitative estimate of risk. They did not know how to apply the absolute risk provided in EsPeR and asked for explanations. Few participants asserted being more confident in their own risk estimate. Few GPs did not know clinical practice guidelines. Most had heard about them but did not refer to them, and few both read and used them routinely (Table 3).
Acceptance of EsPeR content
The acceptance of the guideline messages was evaluated from GPs answers to clinical scenarios (adherence or no adherence to the messages), from their reactions observed during individual work and from the comments reported in discussion notes during open group discussion (Table 3). The cardiovascular guidelines were in general better accepted than guidelines on cancer screening. Several recommendations given by EsPeR concerning cancers screening were perceived as inappropriate because of patients' demands and social or organizational pressures.
Overall, physicians perceived the system as useful. Few participants would have liked to be guided more directly to select the risks that should be considered in priority for prevention for a given patient.
Degree of acceptance of guideline messages, possible interpretation and solutions
Examples of clinical scenarios (and corresponding guideline message)
Acceptance of EsPeR advice#
Characterization of the problem of acceptance
Tobacco smoking (only the last year consumption is taken into account in EsPeR)
Lack of knowledge (cardiovascular risk returns to baseline after smoking cessation)
More explicit reasoning and detailed explanations
Familial cardiovascular risk of a 60-year old man with 2 brothers who had myocardial infarction at more than 70 years of age... (no familial risk according to the definition used in the guideline)
Over estimation of familial risk. Knowledge and evidence disagree with common sense and inherited cultural belief
More explanation and information on evidence
Absence of familial risk of breast cancer in a woman whose mother had a breast cancer at 60 and had no other family member having had a cancer
Knowledge and evidence disagree with common sense and inherited cultural belief
More explanation and information on evidence
Breast cancer screening in a 43-year old women at average risk (mammography not recommended as systematic screening, genetic screening only if high familial risk)
Over-estimation of familial risk. High pressure (anxiety) of patients for screening without knowledge on benefit and risk of screening
More accurate information targeted on both the physician and the patient
Colorectal cancer screening by colonoscopy in a 60-year old man whose father is dead from colorectal cancer at 80 (not recommended in average risk patients except in research programs)
Over-estimation of familial risk. Over estimation of the benefit/risk of colonoscopy
More explanation and information on evidence
Cervical cancer screening in a 55-year old women (pap smear recommended until 70 years of age)
Prostate cancer mass screening in men aged more than 50 (screening not recommended)
Poor knowledge of evidence
More information on evidence
The present paper describes a computerised system devoted to the prevention of avoidable morbidity, based on risk assessment and valid recommendations. The main objective of EsPeR is to put in perspective the different health risks for one individual and to present guideline messages appropriate for this individual. The hierarchy of risks currently relies on the presentation of French mortality data and is therefore poorly "individualized", taking into account age, sex and region of residence. A more accurate estimate of several risks is possible thanks to specific models such as the Framingham model. However, these models have been demonstrated as being adequately discriminent but poorly calibrated in French population [10, 13]. Other models have been recently published thanks to the SCORE project and are recommended by the Third Joint Task Force of European Societies on Cardiovascular prevention [14, 15]. We intend to integrate them in EsPeR. We also experienced difficulties in implementing the French guidelines on cardiovascular prevention because of the heterogeneity of recommended tools for risk stratification.
We ran a pilot qualitative study in laboratory, in order to observe physicians' reaction and behaviour while using EsPeR and to find with them some explanations. We interpreted the collected verbatim and observations as potential barriers and facilitators for the diffusion of this tool. No focus group session was recorded or videotaped and no social scientist was involved in the study: the groups were animated by three different investigators among whom one was not involved directly in the design of EsPeR and supposed to be "neutral". Recruitment was different for both sessions (in January and October) and led to very different groups which interestingly suggested that different kinds of reactions may pertain to different backgrounds (socio-professional environment, use of informatics, etc.).
Despite these limitations, ergonomics problems were easily identified thanks to the first two focus groups. Corrective measures were taken in the version 2 and led to solve these problems, as verified with the second session of focus groups. The main problems pertained to the content of EsPeR and presentation of knowledge. Analysis of these results led to several actions. The first one consisted in giving a feedback to the ANAES (the French national agency in charge of the development of guidelines) on difficulties encountered while computerizing guidelines. This resulted in a partnership between our laboratory and the ANAES to set up a guideline development framework designed for the guideline authors, in order for them to write directly and unequivocally "computerizable" guidelines. Our results also suggest that improving the quality of preventive strategies in the primary care setting requires considering the multiple competing demands faced by patients and physicians during each consultation. The promotion of the EsPeR system, designed for assisting physicians in setting rational priorities in prevention strategies, should therefore include both an educational intervention (i.e. teaching evidence and the concept of risk for increasing physician's confidence in recommendations) and an organisational intervention (i.e. increase of the consultation duration to allow the complete use of the computerised decision support system, or promote the delegation of certain tasks to nurse practitioners).
The key assumption made by people developing decision aids is that providing decision support (recommendations, alerts, reminders, calculations) as well as useful clinical information at the point of care can reduce unwarranted variations in care and improve patients' health. It is generally acknowledged that computerised reminders and alerts improve the practice of simple preventive measures (screening, vaccinations...) . However, more complex decision support systems such as computerised guidelines , or calculation of cardiovascular risk  did not improved quality of care in several recent well designed randomised trials. These findings can have several explanations: either the system does not work, or it is not used because it does not fit the physicians' needs or else it is in conflict with physicians beliefs and constraints. In keeping with this idea, some authors encourage evaluating computerised decision support systems not only with randomised trials . In some examples, a qualitative study helps to explain the negative of poor results of a randomised controlled trial [17, 20]. We therefore plan a qualitative on-field evaluation of the system to assess its usability and its acceptance by general practitioners on larger scale, in two different regions of France, before considering evaluating its impact in a randomised trial.
We presented an original computerized decision support system for medical prevention based on concepts like risks and complex knowledge found in clinical practice guidelines. The pilot qualitative evaluation of this system reinforces the need of training on its content, prior to its on-field evaluation. The main perspective of this work is the design of a larger qualitative study to evaluate the utilization and barriers to acceptance of the EsPeR system in the day to day practice. Potential important implication of this project could be the use of EsPeR as a structuring tool for prevention strategies in ambulatory medicine.
This project has been developed thanks to a grant by the Caisse Nationale d'Assurance Maladie des Travailleurs Indépendants (CANAM). We thank Catherine Wajs for her help in revising the manuscript and all other members of the EsPeR Group for their contribution to the project: Pr P Degoulet, Pr M Fieschi, Dr J Gouvernet, Dr R Giorgi, Dr JC Dufour, E Jougla, E Michel, Y Xu.
- Lang T, de Gaudemaris R, Chatellier G, Hamici L, Diene E: Prevalence and therapeutic control of hypertension in 30,000 subjects in the workplace. Hypertension. 2001, 38: 449-454.View ArticlePubMedGoogle Scholar
- Ganry O, Peng J, Dubreuil A: [Evaluation of mass screening for breast cancer in the Somme district (France) between 1990 and 1996]. Rev Epidemiol Sante Publique. 1999, 47: 335-341.PubMedGoogle Scholar
- Giorgi R, Gouvernet J, Jougla E, Chatellier G, Degoulet P, Fieschi M: The use of the personalized estimate of death probabilities for medical decision making. Comput Biomed Res. 2000, 33: 75-83. 10.1006/cbmr.1999.1531.View ArticlePubMedGoogle Scholar
- Colombet I, Xu Y, Jaulent MC, Desages D, Degoulet P, Chatellier G: A generic computerized method for estimate of familial risks. Proc AMIA Symp. 2002, 175-179.Google Scholar
- Dart T, Xu Y, Chatellier G, Degoulet P: Computerization of guidelines: towards a "guideline markup language". Medinfo. 2001, 10: 186-190.Google Scholar
- Ohno-Machado L, Gennari JH, Murphy SN, Jain NL, Tu SW, Oliver DE, Pattison-Gordon E, Greenes RA, Shortliffe EH, Barnett GO: The guideline interchange format: a model for representing guidelines. J Am Med Inform Assoc. 1998, 5: 357-372.View ArticlePubMedPubMed CentralGoogle Scholar
- Anderson KM, Wilson PW, Odell PM, Kannel WB: An updated coronary risk profile. A statement for health professionals. Circulation. 1991, 83: 356-362.View ArticlePubMedGoogle Scholar
- Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ: Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst. 1989, 81: 1879-1886. 10.1093/jnci/81.24.1879.View ArticlePubMedGoogle Scholar
- Dréau H, Colombet I, Degoulet P, Chatellier G: Identification of patients at high cardiovascular risk: a critical appraisal of applicability of statistical risk prediction models. Meth Inform Med. 2001, 40: 6-11.PubMedGoogle Scholar
- Bastuji-Garin S, Deverly A, Moyse D, Castaigne A, Mancia G, de Leeuw PW, Ruilope LM, Rosenthal T, Chatellier G: The Framingham prediction rule is not valid in a European population of treated hypertensive patients. J Hypertens. 2002, 20: 1973-1980. 10.1097/00004872-200210000-00016.View ArticlePubMedGoogle Scholar
- D'Agostino R. B., Sr., Grundy S, Sullivan LM, Wilson P: Validation of the Framingham coronary heart disease prediction scores: results of a multiple ethnic groups investigation. Jama. 2001, 286: 180-187.View ArticlePubMedGoogle Scholar
- Giacomini MK, Cook DJ: Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 2000, 284: 357-362. 10.1001/jama.284.3.357.View ArticlePubMedGoogle Scholar
- Laurier D, Nguyen PC, Cazelles B, Segond P: Estimation of CHD risk in a French working population using a modified Framingham model. The PCV-METRA Group. J Clin Epidemiol. 1994, 47: 1353-1364.View ArticlePubMedGoogle Scholar
- Conroy RM, Pyorala K, Fitzgerald AP, Sans S, Menotti A, De Backer G, De Bacquer D, Ducimetiere P, Jousilahti P, Keil U, Njolstad I, Oganov RG, Thomsen T, Tunstall-Pedoe H, Tverdal A, Wedel H, Whincup P, Wilhelmsen L, Graham IM: Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003, 24: 987-1003. 10.1016/S0195-668X(03)00114-3.View ArticlePubMedGoogle Scholar
- De Backer G, Ambrosioni E, Borch-Johnsen K, Brotons C, Cifkova R, Dallongeville J, Ebrahim S, Faergeman O, Graham I, Mancia G, Manger Cats V, Orth-Gomer K, Perk J, Pyorala K, Rodicio JL, Sans S, Sansoy V, Sechtem U, Silber S, Thomsen T, Wood D: European guidelines on cardiovascular disease prevention in clinical practice. Third Joint Task Force of European and Other Societies on Cardiovascular Disease Prevention in Clinical Practice. Eur Heart J. 2003, 24: 1601-1610. 10.1016/S0195-668X(03)00347-6.View ArticlePubMedGoogle Scholar
- Shea S, DuMouchel W, Bahamonde L: A meta-analysis of 16 randomized controlled trials to evaluate computer- based clinical reminder systems for preventive care in the ambulatory setting [see comments]. J Am Med Inform Assoc. 1996, 3: 399-409.View ArticlePubMedPubMed CentralGoogle Scholar
- Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I: Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002, 325: 941-10.1136/bmj.325.7370.941.View ArticlePubMedPubMed CentralGoogle Scholar
- Montgomery AA, Fahey T, Peters TJ, MacIntosh C, Sharp DJ: Evaluation of computer based clinical decision support system and risk chart for management of hypertension in primary care: randomised controlled trial. BMJ. 2000, 320: 686-690. 10.1136/bmj.320.7236.686.View ArticlePubMedPubMed CentralGoogle Scholar
- Kaplan B: Evaluating informatics applications--some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inf. 2001, 64: 39-56. 10.1016/S1386-5056(01)00184-8.View ArticleGoogle Scholar
- Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M: Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care. Bmj. 2003, 326: 314-10.1136/bmj.326.7384.314.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/3/13/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.