- Research article
- Open Access
- Open Peer Review
Factors influencing the implementation of clinical guidelines for health care professionals: A systematic meta-review
BMC Medical Informatics and Decision Making volume 8, Article number: 38 (2008)
Nowadays more and more clinical guidelines for health care professionals are being developed. However, this does not automatically mean that these guidelines are actually implemented. The aim of this meta-review is twofold: firstly, to gain a better understanding of which factors affect the implementation of guidelines, and secondly, to provide insight into the "state-of-the-art" regarding research within this field.
A search of five literature databases and one website was performed to find relevant existing systematic reviews or meta-reviews. Subsequently, a two-step inclusion process was conducted: (1) screening on the basis of references and abstracts and (2) screening based on full-text papers. After that, relevant data from the included reviews were extracted and the methodological quality of the reviews was assessed by using the Quality Assessment Checklist for Reviews.
Twelve systematic reviews met our inclusion criteria. No previous systematic meta-reviews meeting all our inclusion criteria were found. Two of the twelve reviews scored high on the checklist used, indicating only "minimal" or "minor flaws". The other ten reviews scored in the lowest of middle ranges, indicating "extensive" or "major" flaws.
A substantial proportion (although not all) of the reviews indicates that effective strategies often have multiple components and that the use of one single strategy, such as reminders only or an educational intervention, is less effective.
Besides, characteristics of the guidelines themselves affect actual use. For instance, guidelines that are easy to understand, can easily be tried out, and do not require specific resources, have a greater chance of implementation.
In addition, characteristics of professionals – e.g., awareness of the existence of the guideline and familiarity with its content – likewise affect implementation.
Furthermore, patient characteristics appear to exert influence: for instance, co-morbidity reduces the chance that guidelines are followed.
Finally, environmental characteristics may influence guideline implementation. For example, a lack of support from peers or superiors, as well as insufficient staff and time, appear to be the main impediments.
Existing reviews describe various factors that influence whether guidelines are actually used. However, the evidence base is still thin, and future sound research – for instance comparing combinations of implementation strategies versus single strategies – is needed.
Evidence based medicine (EBM) is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients . EBM is increasingly embraced by professional associations, as well as by international health care organizations, such as the WHO [2, 3]. Clinical guidelines, in the sense of scientifically developed statements to assist practitioner and patient decision-making about appropriate care for specific clinical conditions, may be important instruments to shape evidence based medicine [1, 4]. Professionals can use guidelines for decision-making at the bedside of individual patients. The guidelines may provide instructions on which diagnostic or screening tests or interventions to be used . Guidelines are also increasingly regarded as being an indispensable part of professional quality systems . These may, for instance, involve continuing professional education, peer review and audit procedures . Accordingly, in many ways, clinical guidelines help practitioners to improve their professional practice and the quality of care and – subsequently – patients' outcomes.
In addition, guidelines may empower patients to make more informed health care choices .
Although the development of guidelines for medical staff, nursing staff and/or other health care professionals has gained momentum in recent years, this does not necessarily mean that the recommendations described in the guidelines are actually followed [4, 9–12]. For instance, Grol et al.  concluded in an observational study on ten Dutch guidelines, that guideline recommendations were followed by GPs in an average of 61% of the relevant decisions. In addition, Bauer  analysed 41 studies on the implementation of clinical guidelines in the field of mental health care, including depression, schizophrenia and addiction. Guideline adherence was found in 27% of the cross-sectional and pre-post studies and in 67% of the controlled trials under review. Several of these studies showed that after the cessation of specific implementation strategies, adherence rates returned to baseline levels.
This article primarily aims to give insight into factors that negatively or positively influence the implementation of clinical guidelines. Since many guidelines are currently being developed, but may not always be successfully implemented, this is a relevant topic.
A second aim of this systematic meta-review is to shed light on the "state-of-the-art" regarding research within this field. Meta-reviews in particular are appropriate for describing whether the current evidence base is complete or incomplete, since evidence from relevant previous systematic reviews or meta-reviews is synthesized. The reason for including only systematic reviews or meta-reviews is because this kind of research generally provides more evidence than separate empirical studies.
This systematic meta-review addresses the following research question:
What evidence exists regarding factors that influence the implementation of clinical guidelines either negatively or positively?
In this article, the term "implementation of" guidelines is sometimes replaced by terms like "use of", "adherence to" or "compliance with" (depending on the terminology used in the publications we are referring to).
Searches in databases
To find relevant publications, we developed a search strategy in cooperation with an experienced librarian. The search strategy was developed first for Pubmed, to be adapted later to search other databases (with no time period limitation). Table 1 displays the search strategy for Pubmed. This strategy was adapted for the other literature databases cited in Table 2. All searches were executed in November 2006.
The references resulting from the searches were entered in Reference Manager and within this program duplicates were removed. Without duplicates, 885 references remained.
In order to assess whether the references found were indeed relevant, we formulated the following criteria concerning types of studies, target groups and variables.
Studies: only systematic reviews or meta-reviews were eligible for inclusion. We considered a review or meta-review to be systematic if at least two of the following three criteria were satisfied: (a) search terms are presented; (b) Pubmed/Medline, at least, has been searched; (c) the methodological quality of the included studies has been assessed by the reviewer(s).
Target groups: the guidelines mentioned in the reviews or meta-reviews should be aimed at medical staff, nurses or other professionals in health care
Variables: the systematic reviews or meta-reviews should discuss factors that influence guideline implementation either positively or negatively.
Reviews or meta-reviews that exclusively focussed on consensus procedures or consensus-based guidelines were not included (e.g. [15–17]). By consensus-based guidelines we mean guidelines that have been developed exclusively on the basis of consensus procedures, without a systematic analysis of relevant scientific literature. We decided to rule out (meta-)reviews that focussed exclusively on consensus-based guidelines, because we assume that clinical guidelines should be based as much as possible on scientific evidence .
Additionally, reviews or meta-reviews that do not differentiate in their conclusions between clinical guidelines and other professional interventions, such as continuing education or comprehensive quality programs, were excluded [19–23].
No exclusion criteria were applied regarding language or search period.
The inclusion process took place in the following steps.
Step 1 – screening of titles and abstracts
Titles and abstracts of the references found (n = 885) were screened independently by a first and second meta-reviewer (ALF and MCS), to check whether these publications satisfied the inclusion criteria. In this phase the two reviewers agreed in virtually 100% of the cases. For the references selected by both reviewers as well as for the references selected by only one reviewer (a total of 84), we tried to track down or download the full text.
Step 2 – screening based on full texts
Next, the full texts were assessed independently by the first and second meta-reviewer (ALF and MCS) using the inclusion criteria cited.
In this phase, we conducted a manual search in the reference lists of the full text papers. Subsequently, we were able to add another 24 potentially relevant references. The full texts of these additional references were studied as well, which brings the total number of full texts examined to 108.
When the first and the second reviewer did not agree in this phase on inclusion or exclusion (initial disagreement existed in this phase for only two publications), agreement was reached on the basis of discussion between the two reviewers.
Table 3 shows the steps of the inclusion process:
The methodological quality of the twelve included studies was then assessed by two meta-reviewers (ALF and PM) independently, with the Quality Assessment Checklist for Reviews [43, 44] (see Additional file 2). This checklist is used quite frequently for assessing systematic reviews methodologically [45, 46]. The checklist is one of the few found for which psychometric properties had been documented  and which had been shown to meet several important criteria, such as inter observer reliability and coverage of the items in the QUORUM statement for reporting systematic reviews . The overall scores on this checklist range from "extensive flaws" (score 1 or 2), to "major flaws" (score 3 or 4), "minor flaws" (score 5 or 6) and "minimal flaws" (score 7).
When the overall score of the meta-reviewers' assessments for a particular review study differed, the average of the mutual overall score was calculated. In the case of large differences (> 1 point) in mutual overall scores, agreement was reached on the basis of discussion between the meta-reviewers. The scores of the appraisal are shown in Table 4. The checklist and an explanation of its use are contained in Additional file 2.
Data analysis and synthesis
The data from the twelve included reviews are presented descriptively following the structure of Additional file 1. Factors that influence, or could influence, the implementation of clinical guidelines are classified in the following categories:
* characteristics of the guidelines;
* characteristics of the implementation strategies;
* characteristics of professionals;
* characteristics of patients;
* characteristics of the environment.
Data extraction was performed by the first reviewer (ALF) and the data were subsequently checked by another reviewer (AJEDeV or PM).
The reviews included determined whether a factor was described as having a positive or a negative influence on guideline implementation. Because of the large variety of factors described and methods used, no quantitative pooling was performed across the reviews. Moreover, pooling was not possible since the large majority of the reviews studied did not provide numbers, e.g. in the form of effect sizes. Conclusions for the meta-review were therefore based on the conclusions and results presented in the reviews.
In the Results section as well as in the Conclusion section our findings are discussed in relation to the methodological quality of these review studies.
General description of the reviews and the guidelines
The twelve included studies are all systematic reviews; no previous systematic meta-reviews were found that completely satisfied our inclusion criteria.
Nearly all twelve review studies concerned English language studies only, although this was not always the result of an explicitly stated exclusion criterion regarding language. Two reviews did however also include non-English studies: the German language review by Sachs  and the French language review by Saillour-Glenisson and Michel , which also included publications in German and French respectively. The aforementioned study by Sachs  is also the most recent review included. The review by Grilli and Lomas  is the oldest.
The reviews included had varying objectives: some aimed to map the success and failure factors or the most effective implementation strategies; others focussed primarily on charting the effects of clinical guidelines (see aims in Additional file 1, column 1). In these cases, establishing which factors had influenced successful implementation was not the primary focus, although the reviews provided relevant information in this regard.
There is also limited overlap in the subjects to which the guidelines relate. For instance, the guidelines in the review by Gross and Pujat  had a rather narrow focus and described recommendations for the use of antibiotics. Likewise, the reviews by Simpson et al.  and Tooher et al.  dealt with a specific topic, viz. the treatment of pneumonia and pressure ulcers respectively. Other reviews had a broader focus and concerned a number of guidelines on preventive or curative treatments and various diseases (see Additional file 1).
In most instances, the main target groups of the guidelines appear to be physicians. The majority of review studies do not explicitly state who the main target groups are, or whether the clinical guidelines are mono-disciplinary or multi-disciplinary (see Additional file 1, column 1). The descriptions and results often show only indirectly that physicians were the main target group. By contrast, the reviews of Sachs  and Thomas et al.  made it very clear that they were aiming at other target groups: Sachs  concentrated on guidelines for nursing staff and Thomas et al.  dealt with guidelines for nursing staff, midwives and/or other allied health care professionals.
Methodological characteristics of the reviews
As stated earlier, for the structured methodological assessment of the twelve reviews we used the Quality Assessment Checklist for Reviews of Oxman and Guyatt (See Additional file 2).
Two reviews received a high score based on the Quality Assessment Checklist for Reviews: the review by Thomas et al.  received a score of 5.5, and that of Grimshaw et al. [41, 42] scored a 7. These scores reflect "minor flaws" and "minimal flaws" respectively on the checklist. The high scores are related to the fact that the searches by these reviewers were extensive and well documented. For instance, they described their search strategy; their search for evidence was comprehensive; and they applied clear inclusion criteria. They also took measures to prevent selection bias, by involving more than one reviewer in the selection process. In addition, in these two reviews the methodological quality of the included studies was systematically assessed [41, 42, 56].
The remaining ten reviews received a score of between 1.5 and 4.5, which relate to the lowest or middle ranges of the checklist, indicating "extensive" to "major" flaws (for the interpretation of these scores also see Additional file 2in relation to Table 4).
Additional file 1 (column 1) shows that most reviews included quantitative studies with comparative designs (RCTs, CCTs, pre-test post-test studies), enabling the reviewers to make statements about the differences between (the effects of) implementation strategies. This is not the case with the review by Cabana et al.  which included only surveys and qualitative studies, in order to disclose the barriers to the implementation of clinical guidelines. The reviews of Saillour-Glenisson and Michel  are also slightly different; they included both quantitative and qualitative studies to reveal such barriers. Grilli and Lomas  included quantitative studies, but in the studies with pre-measurements and post-measurements they only looked at the post-measurements, in order to be able to write about adherence to clinical guidelines.
All twelve reviews not only searched Pubmed/Medline but also studied other sources, and they often used explicit inclusion criteria (see Additional file 1, column 2). Remarkably, most reviews pay no attention to preventing selection bias or to methodological assessment. In addition, information on which methods were used to synthesize study results and to reach conclusions was not, or was only partially provided in the reviews.
Factors influencing implementation: characteristics of the guidelines
The most frequently described guideline characteristic concerns complexity. Guidelines that are easy to understand, can easily be tried out, and do not require specific resources have a greater chance of being used (Davis and Taylor-Vaisey , Grilli and Lomas , Saillour-Glenisson & Michel  and Simpson et al. )
Other influential guideline characteristics are also described, although not so frequently as the factor "complexity of the guideline". For instance, the review of Saillour-Glenisson & Michel  concluded that adherence to evidence based guidelines appears to be higher than is the case for guidelines lacking a clear scientific base. In addition, Sachs  concluded that when guidelines are developed by the target group (in that case nurses) and experts, this enhances the chance of successful implementation. However, Davies et al.  maintain that the findings are contradictory with regard to whether guidelines that are developed by end users (amongst others) are more often used.
For other – not frequently described – influential guideline characteristics, see Additional file 1 (column 3).
In interpreting the results, it should however be taken into account that all reviews describing the influence of guideline characteristics [49–53, 57] have a relatively low methodological score (4.5 or lower) on the Quality Assessment Checklist for Reviews. This implies a high likelihood of "extensive" or "major" flaws in the results and conclusions.
Characteristics of the implementation strategies
Almost all reviews examined the characteristics of the implementation strategies in relation to the use of the guideline (see Additional file 1).
In the review of Grimshaw et al. [41, 42], 235 studies with comparative designs (RCTs, CCTs etc.) on implementation strategies regarding guidelines with a broad variety of topics were analysed. As already described, the Grimshaw review received a high methodological score, namely 7, on the checklist used. This review [41, 42] described the effects of several implementation strategies. For instance, the authors investigated the effects of combined educational materials and meetings as well as combinations of educational materials and audit and feedback. Other combinations were also investigated (see Additional file 1). For most combinations as well as for most single strategies, Grimshaw et al. [41, 42] found some effects. However, they also stated that the effects are often modest and the evidence base sparse. Although these authors described some studies that found more effects of combined strategies than of single ones, one of their main conclusions was that there is no evidence that multi-faceted strategies are more effective than single ones. In addition, they concluded that there was no significant relationship between the number of components of multi-faceted strategies and the effects measured.
As stated earlier, Thomas et al.  were the authors of the other review with a high methodological score (5.5) on the Quality Assessment Checklist for Reviews. These researchers  only focussed on studies regarding guidelines for nurses or allied health professionals and only described three studies relevant to the questions in our meta-review. These three studies were all compromised by a small sample size or "unit of analysis" errors, and the conclusion was that insufficient evidence existed about the effectiveness of different implementation strategies with regard to guidelines aimed at nurses or allied health professionals.
Some reviews with lower methodological scores (scoring 4.5 or lower on the checklist) did not point in the same direction as Grimshaw et al. [41, 42]. For instance, Bauer et al.  concluded that multi-faceted and intensive strategies, involving system redesign or additional resources (e.g. regarding additional consultation or case management) seem to be most successful in improving adherence to mental health guidelines. Davis and Taylor-Vaisey  focussed on guidelines with a variety of subjects and concluded that single strategies (e.g. reminder systems) may be effective, but strategies involving two or more interventions often appear to have greater impact. Besides, Tooher , reaches the conclusion that with regard to guidelines on pressure ulcers, the more comprehensive the implementation strategies are (that is, the greater the variety and breadth of the strategies) the more effective their implementation seems to be in the long term.
For single strategies only, there appears to be insufficient evidence to reach conclusions about the relative effectiveness of different implementation strategies in different contexts or circumstances (Grimshaw et al. [41, 42] and Davies et al. ) However, according to the review of Davies et al. , strategies requiring active professional participation, and strategies that are closely related to clinical decision- making are more likely to lead to successful implementation In other words: strategies that are closer to the end user and more integrated into the process of health care delivery appear to be most successful.
Characteristics of professionals
Six of the twelve reviews paid attention to certain characteristics of professionals in relation to the implementation of clinical guidelines [49–52, 57, 58]. All these reviews received a methodological score of 4.5 or lower, indicating a high chance of "extensive" or "major" flaws.
The review of Cabana et al.  was the most detailed, and described characteristics of physicians in relation to the adoption of clinical guidelines on a number of issues (see Additional file 1, column 3). One of the conclusions of this review is that a lack of awareness, limited familiarity and a lack of agreement with guidelines are the main barriers to guideline adoption. The results of the Cabana review  point largely in the same direction as the reviews of Saillour-Glenisson and Michel  and Simpson et al. . Also these reviews conclude that the main barrier can be found in the simple fact that physicians are sometimes not aware of the existence of particular guidelines. In addition, three reviews also mention age and/or experience as determinants: young professionals or less experienced ones would be more inclined to use guidelines than older, experienced professionals [50, 52, 57]. Other characteristics of professionals that influence implementation are listed in Additional file 1 (column 3).
Characteristics of patients
Four reviews with a methodological score of 4.5 or lower on the Quality Assessment Checklist for Reviews described the influence of a rather limited number of patient characteristics [49, 50, 52, 57]. Cabana et al.  concluded that patient-related characteristics may include the fact that some patients perceive no need for guideline recommendations or may even resist them. Saillour-Glenisson and Michel  also described resistance of patients towards the recommendations as a factor negatively affecting the adoption of clinical guidelines.
In addition, Davis and Taylor-Vaisey  refer to patients with co-morbidity: in the case of such patients the chance is greater that professionals do not adhere to guidelines. Additional relevant patient characteristics are listed in Additional file 1.
Limited time and personnel resources as well as work pressure [49–52] are rather frequently cited environmental characteristics said to negatively influence guideline implementation. A negative attitude or limited support from "peers" or superiors also has a negative influence [51, 52, 55, 57]. Other environmental characteristics are stated in Additional file 1.
Factors influencing the implementation of guidelines
The most frequently investigated guideline characteristic concerns "complexity". Several of the systematic reviews included in our meta-review indicated that when a guideline can be relatively easily understood and tried out, the chance is greater that the guideline will be used.
It is important therefore for guideline developers to take into account the complexity of the guidelines. Particularly for developers of multi-disciplinary guidelines directed at several target groups with varying educational levels and backgrounds (e.g. physicians, nurses, patients), it is a challenge to describe recommendations that are understandable and usable for all target groups.
The finding in the Sachs review , that involving the targeted professionals already in the development phase enhances the chance of successful implementation, may be relevant for guideline developers as well. Prominent groups like the WHO Advisory Committee on Health Research  and the AGREE Collaboration also recommend that groups that develop guidelines should be broadly composed and include all relevant health professionals. In addition, involvement of the target group may imply that the guideline is first being tested in practice before large-scale implementation takes place .
Still, Davies et al.  assert that findings are not always unanimous with regard to whether guidelines that are developed by end users (amongst others) are more often used. Future research will have to provide more insight into this issue.
Our meta-review also describes specific influential characteristics of professionals. Implementers of guidelines and policy makers in particular should take into account that implementation may be hampered by the simple fact that professionals are often unaware that the guidelines exist, or are not familiar with their content . Clearly, it is not sufficient to merely disseminate a guideline. Targeted implementation interventions – in which professionals themselves are preferably directly and actively involved – should take place to create awareness. Examples of such targeted interventions may be combinations of (web-based, written or face-to-face) practical recommendations, educational material, and educational meetings (see for instance the Sachs article ).
Characteristics of patients, too, appear to exert influence: for instance, co-morbidity in a patient appears to reduce the chance that guidelines are followed . Professionals, presumably, assume that guidelines are based on a general clinical picture and are insufficiently tailored to the often complex care needs of patients with co-morbidity. For instance, Tinetti et al.  and Durso  therefore argue for greater attention among guideline developers to the specific needs of patients with co-morbidity. To improve guideline implementation, these authors recommend that guidelines should also provide guidance for interventions in patients with multiple conditions as well as information on risks of specific interventions in these patients.
In addition, environmental characteristics influence the implementation of guidelines. For example, support by peers or superiors in following the guidelines, and sufficient staff and time appear to be important for guideline implementation .
However, with respect to environmental characteristics, and also regarding patients' and professionals' characteristics, existing systematic reviews lack methodological rigour, and underlying primary research often focuses on rather heterogeneous guideline subjects and target groups. This hampers evidence-based conclusions. Future sound methodological research regarding these kinds of characteristics is therefore recommended.
More research is already being performed regarding the category "characteristics of the implementation strategies". Almost all included reviews investigated the influence of certain implementation strategies (see Additional file 1), which provides relevant information, particularly for guideline implementers. Most of the reviews indicate that effective implementation strategies often have multiple components and that the use of one single strategy, such as reminders only or one educational intervention, is less effective than a combination of strategies. However, interpretation is hampered by the fact that the high-quality Grimshaw review published in 2004 and 2006 [41, 42] does not show a correlation between the number of components in implementation strategies and their effectiveness.
Explanations for contradictory results
The contradictory results found for "characteristics of the implementation strategies" may be partially explained by limited overlap in target groups and guideline subjects. The Grimshaw review included a large number of primary studies (n = 235) on guidelines with varying target groups and topics, while other relevant reviews involved fewer studies and had a narrower focus, e.g. on guidelines for mental health care , guidelines for pressure ulcers  or guidelines for nursing practice .
Another (partial) explanation may be that Grimshaw et al. [41, 42] were more rigorous in their analysis of primary studies than the other reviewers. Although Grimshaw and colleagues did not conduct a formal meta-analysis (because of the large heterogeneity of studies), they did take effect sizes into account.
However, a methodologically strong review by Wensing et al.  also contradicts the conclusion of Grimshaw et al. [41, 42]. Wensing and colleagues reviewed studies on a rather broad range of professional, educational and quality interventions (which was why the Wensing review did not completely match our inclusion criteria, and was not discussed in previous sections). Wensing et al. concluded that combined implementation strategies with many different aspects are, in general, the most effective. Furthermore, findings of two other methodologically sound meta-reviews [20, 62] are relevant in this regard. These previous meta-reviews were likewise excluded from our meta-review since they focussed on a broad range of professional or educational interventions, and not specifically on clinical guidelines. Yet these meta-reviews equally concluded that successful implementation strategies are often multi-faceted.
Our main conclusion is therefore that multiple strategies for implementing guidelines appear to be more effective than single ones. In the introduction we stated that guidelines are increasingly considered to be part of comprehensive quality systems , often combining guidelines with educational interventions, audits and other actions for improvement. The conclusion, that multiple strategies seem to be most effective, fits with the comprehensive character of today's quality systems.
However, guideline researchers in particular, should be aware that well-constructed empirical research looking into various implementation strategies is still needed in this area . This will enable us to make more definitive statements about the effectiveness of multi-faceted strategies compared to specific single strategies in implementing clinical guidelines.
At the end of this paper, we will discuss some methodological issues concerning our own meta-review. Ten out of the twelve reviews described in this meta-review scored in the low or middle ranges of the Quality Assessment Checklist for Reviews, which indicates "major" or "extensive" flaws (see Additional file 2). Initially, we considered excluding reviews with low or middle range methodological scores, in order to reduce the risk of bias. However, this would have left us with only a very limited number of reviews to be included. We want to offer a complete picture of existing research: this is why we decided after all to include all relevant systematic review studies meeting our inclusion criteria. Regarding this decision, it should also be taken into account that strict criteria for the methods of systematic reviews have only become common practice in recent years, brought about – for instance – by the Cochrane Collaboration. This makes it understandable that less recent review studies in particular do not fully satisfy these criteria or scarcely mention the review methods they have used. Consequently, reviews with a relatively low methodological score do not automatically show distorted results and may sometimes provide valuable insights.
A limitation of this meta-review is that our search for potentially eligible publications ceased in November 2006. Analyzing and synthesizing the results of previous reviews are time-consuming procedures, and therefore a time span between the searches and the submission to a journal cannot be avoided. However, in order to be sure that we did not miss relevant information from very recent papers, we performed an additional search in Pubmed from November 2006 to February 2008, just before submission. This resulted in two additional publications meeting the inclusion criteria; one concerned successful characteristics of implementation strategies for guidelines in obstetrics , while the other focussed on characteristics of strategies for implementing psychiatric guidelines . Neither of these publications presents results that might have altered our conclusions. In line with our above-outlined conclusion on implementation strategies, both recent papers concluded that multi-faceted strategies are generally the most effective.
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312: 71-72.
WHO World Health Organization: Global programme on evidence for health policy. Guidelines for WHO Guidelines. 2003, Geneva: WHO World Health Organization
Schunemann HJ, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 1. Guidelines for guidelines. Health Res Policy Syst. 2006, 4: 13-
Grol R, Grimshaw J: From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003, 362: 1225-1230.
Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J: Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999, 318: 527-530.
Casparie AF, Theuvenet WJ, Klazinga NS, Timmermans AE: Richtlijnontwikkeling als onderdeel van het kwaliteitsysteem [Development of guidelines as part of the quality system]. Evidence-based richtlijnontwikkeling [Evidence-based guideline development]. Edited by: van Everdingen JJE, Burgers JS, Assendelft WJJ, Swinkels JA, van Barneveld TA, van de Klundert JLM. 2004, Houten: Bohn Stafleu Van Loghum
Feder G, Eccles M, Grol R, Griffiths C, Grimshaw J: Clinical guidelines: using clinical guidelines. BMJ. 1999, 318: 728-730.
Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J: Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999, 318: 527-530.
Espeland A, Baerheim A: Factors affecting general practitioners' decisions about plain radiography for back pain: implications for classification of guideline barriers – a qualitative study. BMC Health Serv Res. 2003, 3: 8-
Azevedo JM, Germano CD: Potential of the Executive Order Increasing the Participation of Asian Americans and Pacific Islanders in Federal Programs. Asian Am Pac Isl J Health. 1999, 7: 47-56.
Fretheim A, Schunemann HJ, Oxman AD: Improving the use of research evidence in guideline development: 15. Disseminating and implementing guidelines. Health Res Policy Syst. 2006, 4: 27-
Burgers JS, Grol RP, Zaat JO, Spies TH, Bij van der AK, Mokkink HG: Characteristics of effective clinical guidelines for general practice. Br J Gen Pract. 2003, 53: 15-19.
Grol R, Dalhuijsen J, Thomas S, Veld C, Rutten G, Mokkink H: Attributes of clinical guidelines that influence use of guidelines in general practice: observational study. BMJ. 1998, 317: 858-861.
Bauer MS: A review of quantitative studies of adherence to mental health clinical practice guidelines. Harv Rev Psychiatry. 2002, 10: 138-153.
Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J: Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999, 4: 236-248.
Lomas J: Words without action? The production, dissemination, and impact of consensus recommendations. Annu Rev Public Health. 1991, 12: 41-65.
Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J: Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998, 2: i-88.
Oxman AD, Lavis JN, Fretheim A: Use of evidence in WHO recommendations. World Hosp Health Serv. 2007, 43: 14-20.
Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L: Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001, 39: II2-45.
NHS Centre for Reviews and Dissemination: Getting evidence into practice. Effective Health Care Bulletin. 1999, 5: 1-16.
Oxman AD, Thomson MA, Davis DA, Haynes RB: No magic bullets: a systematic review of 102 trials of interventions to improve professional practice. CMAJ. 1995, 153: 1423-1431.
Solberg LI: Guideline implementation: what the literature doesn't tell us. Jt Comm J Qual Improv. 2000, 26: 525-537.
Wensing M, Weijden van der T, Grol R: Implementing guidelines and innovations in general practice: which interventions are effective?. Br J Gen Pract. 1998, 48: 991-997.
Boyd EA, Bero LA: Improving the use of research evidence in guideline development: 4. Managing conflicts of interests. Health Res Policy Syst. 2006, 4: 16-
Edejer TT: Improving the use of research evidence in guideline development: 11. Incorporating considerations of cost-effectiveness, affordability and resource implications. Health Res Policy Syst. 2006, 4: 23-
Fretheim A, Schunemann HJ, Oxman AD: Improving the use of research evidence in guideline development: 5. Group processes. Health Res Policy Syst. 2006, 4: 17-
Fretheim A, Schunemann HJ, Oxman AD: Improving the use of research evidence in guideline development: 3. Group composition and consultation process. Health Res Policy Syst. 2006, 4: 15-
Oxman AD, Fretheim A, Schunemann HJ: Improving the use of research evidence in guideline development: introduction. Health Res Policy Syst. 2006, 4: 12-
Oxman AD, Schunemann HJ, Fretheim A: Improving the use of research evidence in guideline development: 8. Synthesis and presentation of evidence. Health Res Policy Syst. 2006, 4: 20-
Oxman AD, Schunemann HJ, Fretheim A: Improving the use of research evidence in guideline development: 2. Priority setting. Health Res Policy Syst. 2006, 4: 14-
Oxman AD, Schunemann HJ, Fretheim A: Improving the use of research evidence in guideline development: 7. Deciding what evidence to include. Health Res Policy Syst. 2006, 4: 19-
Oxman AD, Schunemann HJ, Fretheim A: Improving the use of research evidence in guideline development: 12. Incorporating considerations of equity. Health Res Policy Syst. 2006, 4: 24-
Oxman AD, Schunemann HJ, Fretheim A: Improving the use of research evidence in guideline development: 14. Reporting guidelines. Health Res Policy Syst. 2006, 4: 26-
Oxman AD, Schunemann HJ, Fretheim A: Improving the use of research evidence in guideline development: 16. Evaluation. Health Res Policy Syst. 2006, 4: 28-
Schunemann HJ, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 10. Integrating values and consumer involvement. Health Res Policy Syst. 2006, 4: 22-
Schunemann HJ, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 13. Applicability, transferability and adaptation. Health Res Policy Syst. 2006, 4: 25-
Schunemann HJ, Fretheim A, Oxman AD: Improving the use of research evidence in guideline development: 9. Grading evidence and recommendations. Health Res Policy Syst. 2006, 4: 21-
Schunemann HJ, Oxman AD, Fretheim A: Improving the use of research evidence in guideline development: 6. Determining which outcomes are important. Health Res Policy Syst. 2006, 4: 18-
Richens Y, Rycroft-Malone J, Morrell C: Getting guidelines into practice: a literature review. Nurs Stand. 2004, 18 (50): 33-40.
Cheater FM, Closs SJ: The effectiveness of methods of dissemination and implementation of clinical guidelines for nursing practice: a selective review. Clinical Effectiveness in Nursing. 1997, 1: 4-15.
Grimshaw J, Eccles M, Thomas R, Maclennan G, Ramsay C, Fraser C: Toward evidence-based quality improvement. Evidence (and its limitations) of the effectiveness of guideline dissemination and implementation strategies 1966–1998. J Gen Intern Med. 2006, 21 (Suppl 2): S14-S20.
Grimshaw JM, Thomas RE, Maclennan G, Fraser C, Ramsay CR, Vale L: Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004, 8: iii-72.
Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA: Agreement among reviewers of review articles. J Clin Epidemiol. 1991, 44: 91-98.
Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-1278.
Mistiaen P, Francke AL, Poot E: Interventions aimed at reducing problems in adult patients discharged from hospital to home: a systematic meta-review. BMC Health Serv Res. 2007, 7: 47-
Ospina MB, Kelly K, Klassen TP, Rowe BH: Overview of reports on the quality of systematic reviews and meta-analysis in the biomedical literature [conference abstract]. Canadian Journal of Emergency Medicine. 2005, 7: 176-
Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA: A systematic review of the content of critical appraisal tools. BMC Med Res Methodol. 2004, 4: 22-
Shea B, Dubé C, Moher D: Assessing the quality of reports of systematic reviews: the QUORUM statement compared to other tools. Systematic reviews in health care: meta-analysis in context. Edited by: Egger M, Smith GD, Altman DG. 2001, London: BMJ Publishing Group, 122-139.
Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA: Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999, 282: 1458-1465.
Simpson SH, Marrie TJ, Majumdar SR: Do guidelines guide pneumonia practice? A systematic review of interventions and barriers to best practice in the management of community-acquired pneumonia. Respir Care Clin N Am. 2005, 11: 1-13.
Sachs M: [Successful strategies and methods of nursing standards implementation]. Pflege. 2006, 19: 33-44.
Saillour-Glenisson F, Michel P: [Individual and collective facilitators of and barriers to the use of clinical practice guidelines by physicians: a literature review]. Rev Epidemiol Sante Publique. 2003, 51: 65-80.
Grilli R, Lomas J: Evaluating the message: the relationship between compliance rate and the subject of a practice guideline. Med Care. 1994, 32: 202-213.
Gross PA, Pujat D: Implementing practice guidelines for appropriate antimicrobial usage: a systematic review. Med Care. 2001, 39: II55-II69.
Tooher R, Middleton P, Babidge W: Implementation of pressure ulcer guidelines: what constitutes a successful strategy?. J Wound Care. 2003, 12: 373-2.
Thomas L, Cullum N, McColl E, Rousseau N, Soutter J, Steen N: Guidelines in professions allied to medicine. Cochrane Database Syst Rev. 1999, CD000349-2
Davis DA, Taylor-Vaisey A: Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997, 157: 408-416.
Davies J, Freemantle N, Grimshaw J, Hurwitz B, Long A, Russell IT: Implementing clinical practice guidelines: Can guidelines be used to improve clinical practice?. Effective Health Care Bulletin. 1994, 1: 1-12.
The Agree Collaboration: Appraisal of Guidelines for Research & Evaluation (AGREE) Instrument. 2008
Tinetti ME, Bogardus ST, Agostini JV: Potential pitfalls of disease-specific guidelines for patients with multiple conditions. N Engl J Med. 2004, 351: 2870-2874.
Durso SC: Using clinical guidelines designed for older adults with diabetes mellitus and complex health status. JAMA. 2006, 295: 1935-1940.
Grimshaw JM, Shirran L, Thomas R, Mowatt G, Fraser C, Bero L: Changing provider behavior: an overview of systematic reviews of interventions. Med Care. 2001, 39: II2-45.
Chaillet N, Dube E, Dugas M, Audibert F, Tourigny C, Fraser WD: Evidence-based strategies for implementing guidelines in obstetrics: a systematic review. Obstet Gynecol. 2006, 108: 1234-1245.
Weinmann S, Koesters M, Becker T: Effects of implementation of psychiatric guidelines on provider performance and patient outcome: systematic review. Acta Psychiatr Scand. 2007, 115: 420-433.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/8/38/prepub
We thank the Netherlands Organization for Health Research and Development (ZonMw) for its financial support.
Furthermore, we would like to thank Ms Noor Breuning, librarian at the NIVEL, for her advice and support with the execution of the search strategy.
The authors declare that they have no competing interests.
ALF: conception and design of the meta-review; searches and selection of literature; analysis and synthesis of data from the included literature; drafting the manuscript. MCS: conception and design of the meta-review; searches and selection of literature; revising the first draft of the manuscript. AJEdeV: conception and design of the meta-review; analysis and synthesis of data from the included literature; revising the first draft of the manuscript. PM: conception and design of the meta-review; analysis and synthesis of data from the included literature; revising the first draft of the manuscript.