- Research article
- Open Access
Effectiveness of electronic point-of-care reminders versus monthly feedback to improve adherence to 10 clinical recommendations in primary care: a cluster randomized clinical trial
BMC Medical Informatics and Decision Making volume 19, Article number: 245 (2019)
Numerous studies have analyzed the effectiveness of electronic reminder interventions to improve different clinical conditions, and most have reported a small to moderate effect. Few studies, however, have analyzed reminder systems targeting multiple conditions, and fewer still have compared electronic point-of-care reminders systems with other forms of feedback designed to improve delivery of care.
We performed an unblinded cluster randomized clinical trial to compare the effectiveness of an electronic point-of-care reminder system with that of a well-established system providing monthly feedback on adherence to clinical recommendations. The control group received monthly feedback only while the intervention group received monthly feedback in addition to on-screen point-of-care reminders for 10 clinical conditions. The study targeted all physicians and nurses at the 283 primary care centers managed by the Institut Català de la Salut (approximately 6600 professionals).
Following exclusions and randomization, 132 primary care centers (328,728 patients with reminders) were assigned to the intervention group while 137 centers (317,117 patients with reminders) were randomized to the control group. A 20.6% improvement (OR 1.29, 95% CI: 1.25–1.34) in reminder resolution rates was observed in the intervention group. Results varied according to the clinical condition. The most effective reminder was screening for diabetic retinopathy (OR 1.51, 95% CI:1.46–1.57) while the least effective reminders were measurement of glycated hemoglobin (OR: 1.10, 95% CI: 1.07–1.13) and smoking cessation encouragement (OR 1.12, 95% CI: 1.09–1.16).
Electronic point-of-care reminders were more effective than the existing monthly feedback system at resolving the 10 clinical situations. However, more studies are needed to investigate the variations of the effect observed.
Current Controlled Trials ISRCTN42391639, 08/10/2012. Retrospectively registered.
Reminders systems are used to prompt memory or help recall information. Examples of systems used in health care settings are computerized reminders, color code systems, telephone calls, and even letters and postcards . The term computer reminder was used for the first time in 1976 by McDonald  in a clinical trial that found that computer-generated reminders improved clinician behavior. The effects of reminder systems have since been analyzed in many other trials and systematic reviews. Forty years after McDonald’s study, Cheung et al.  reviewed 35 systematic reviews on reminder systems published up to 2009 and assessed their quality using the AMSTAR tool. The results of their analyses, including those restricted to high-quality reviews (AMSTAR score > 5), showed that reminders were an effective means of improving clinician behavior and the quality of care delivery.
Reminders can target patients or clinicians. The most widely studied systems are probably those designed to improve prescribing practices and preventive care delivery. Electronic point-of-care reminders are reminders that are generated from patient data and delivered to the clinician during the clinical encounter. A Cochrane review by Shojania et al.  found that electronic reminders had a modest effect on process adherence (median improvement of 4.2%), while a systematic review and meta-analysis by Holt et al.  found an overall odds ratio of 1.79 (95% CI: 1.56–2.05) for computer-generated on-screen and paper reminders. Most studies of the effectiveness of reminders in primary care settings have analyzed systems targeting a single disorder or clinical condition [6,7,8,9,10,11,12,13]. The few studies of multiple reminder interventions have mostly focused on vaccination, screening, and adverse drug events [14, 15].
In a systematic review aimed at identifying features of successful reminder systems, Kawamoto et al.  found that effective systems 1) provided automatic decision support, 2) used computers to do this, 3) provided a recommendation rather than just an assessment, and 4) provided the support at the time and place of the decision-making.
The Institut Català de la Salut (ICS) is the main primary care service provider in Catalonia and manages approximately 80% of all primary care teams in the Catalan public health system. In 2005, the ICS implemented a universal electronic medical record system for use in primary care known as the ECAP (Estació Clínica d’Atenció Primària). The ECAP is a software system that serves as a repository for structured data on diagnoses (coded according to the International Classification of Diseases 10th revision), clinical variables, prescription data, and laboratory test results. Every month, thanks to a system that has been in place since 2006, primary health care professionals (HCPs) receive a report on their performance through the ECAP. Performance is calculated according to relevant clinical indicators and the report contains feedback and a list of patients with a clinical situation that can be improved [17, 18]. The clinical indicators, which are based on scientific evidence, form what is known as the EQA (Estàndard de Qualitat Assistencial) , which is a quality measure similar to the Quality Outcomes Framework developed by the UK National Health Services. HCPs can access these monthly reports and lists whenever they wish through the ECAP menu. The EQA is also used in the ICS’s pay for performance program.
EQA results have gradually improved over the years, and there is evidence linking improvements to an increasing use of the system by HCPs . To further develop the feedback system in place, we designed an electronic point-of-care reminder system incorporating the success factors identified by Kawamoto et al. . This system is integrated into the ECAP and displays the reminders during the clinical encounter. The hypothesis was that a synchronous decision support system available to HCPs at the point of care would improve adherence to clinical recommendations. To test this hypothesis before deployment of the system, we designed a clinical trial to determine whether immediate, point-of-care, electronic reminders were more effective than the monthly feedback system already in place.
We designed an unblinded cluster randomized trial to compare the effectiveness of an electronic point-of-care reminder system with that of a well-established feedback system to improve adherence to clinical recommendations. The study was originally designed to last one year with a revision after 6 month. The study was, finally, conducted between February 1 and July 31, 2012 and was registered at http://www.controlled-trials.com (ISRCTN42391639). The study protocol was published in 2016 . We adhere to CONSORT guidelines for reporting of results.
The eligible study population comprised all primary care family physicians and nurses (referred to hereafter as HCPs) delivering care to people aged over 14 years at the 283 primary care centers managed by the ICS. HCPs from eight primary care centers participating in another clinical trial on electronic reminders were excluded, as were those from the two centers that had participated in the pilot phase of the trial.
We designed reminders for 10 improvable clinical situations corresponding to a chronic condition or a preventive care measure within the EQA indicator . Computer algorithms were used to analyze the ECAP data weekly and identify patients aged over 14 years with at least one of the 10 clinical situations. The reminders were automatically displayed in a pop-up window that appeared when the HCP accessed the patient’s medical record. Each pop-up window listed the clinical alert(s) generated for the patient, made a recommendation in the form of a call for action (e.g., measure blood pressure) and, where appropriate, showed clinical values and corresponding dates. Each pop-up reminder was accompanied by an orange or red icon indicating the importance of the recommendation (Table 1).
All the participants in the trial, including those in the control group, continued to receive feedback through the usual system. This consisted of a monthly update for each HCP specifying their rate of adherence to the different EQA recommendations together with comparative figures for previous months and their environment (Fig. 1). The report was accompanied by a list of noncompliant patients. An online and previous version of the feedback system can be consulted at http://www.amf-semfyc.com/sisap/ .
The main differences between the well-stablished feedback and the electronic reminders are shown in Table 2.
There were three types of electronic reminders and hence three intervention groups. The reminders were incremental and consisted of 1) a pop-up window, 2) a pop-up window plus a schedule icon, and 3) a pop-up window plus a schedule icon plus a reminder filter settings option.
In the pop-up window group, HCPs accessing the medical record of a patient with an improvable clinical situation were presented with a pop-up window. To consult more information in the patient’s record, the HCP first had to close the window or click on the reminder link.
Pop-up window + schedule icon
In addition to the pop-up window, HCPs in the pop-up window + schedule icon group saw an orange or red icon next to the names of the patients with reminders in the day’s schedule. They were thus immediately able to see how many patients had a reminder on a given day. The icons were orange or red depending on whether the situation was moderately or very important and they also contained a number showing how many reminders each patient had.
Pop-up window + schedule icon + filter settings option
In this third group, in addition to the pop-up window and schedule icon, HCPs were able to apply filters to choose which type of reminders and which patients they wanted to see information for. The default setting for all three groups was “all reminders”.
The units of analysis were the primary care centers and therefore all HCPs from the same center were randomized to the same arm. Randomization was performed using the Stata/SE module to design randomized controlled trials (v. 11.2, StataCorp) . In order to achieve a balanced distribution of the quality of care at the different centers in each arm, these were stratified into quintiles according to EQA results from the month prior to the intervention. The workflow of study population is shown in Fig. 2.
A 10% difference in resolution of clinically improvable situations between groups was considered clinically relevant. Assuming a resolution rate of 59% in the intervention group , an alpha error of 0.01, a power of 95%, and a loss to follow-up of 10%, and correcting for an intraclass correlation of 0.05 in the clusters (primary care centers), it was calculated that 15,950 reminders would be required for each of the four groups (total, 63,800). Nevertheless, in view of the design of the computerized tool, the cluster randomization design, and the distribution of the sample, it was decided not to limit the study to the calculated sample size.
Outcome and variables
The primary outcome measure was the resolution of reminders. A reminder was considered to be resolved when the clinically improvable condition for which it was generated was resolved. Solutions varied according to the type of reminder, and included, for example, prescription of a drug, implementation of a preventive measure, or achievement of a clinical or laboratory target.
The other study variables were the characteristics of the HCPs and the primary care centers to which they belonged. The characteristics analyzed for each center were certification to train postgraduate students, rural or urban location, population assigned and attended, mean number of visits per patient per year, and mean number of physicians and nurses.
Because of the cluster randomization design, we used statistical techniques that took into account the correlations between patients in the different clusters. Intervention effect was estimated using hierarchical (multilevel) models (Wald test), with reminder at level 1 and HCP at level 2. Logistic regression models were used to estimate the effect of the intervention first with adjustment for random HCPs only and then with the addition of EQA results. We finally estimated whether the effect was modified by the introduction of variables at the reminder level. A subgroup analysis by type of reminder was also performed.
Statistical significance was set at 5%. The analyses were performed in Stata/SE, version 11.2 (StataCorp).
The baseline analysis identified 645,845 patients with an improvable clinical situation (13% of the population > 14 years) and generated 968,272 reminders, which is the equivalent of 1.49 reminders per patient.
In the randomization process, 132 primary care centers delivering care to 328,728 patients with 488,811 reminders were assigned to the intervention group, while 137 HCPs delivering care to 317,117 patients with 469,514 reminders were assigned to the control group.
The intervention and control groups were comparable in terms of baseline patient and primary care center characteristics (Table 3).
Almost half of the reminders (47%) were to measure blood pressure (Fig. 3). The next two most common reminders were to encourage smoking cessation (12.8%) and measure glycated hemoglobin in patients with type 2 diabetes mellitus (10.2%). Table 4 shows the number and percentage of reminders by type and clinical condition in the intervention and control groups.
Outcome and estimation
The reminder resolution rate was 27.5% for the intervention group and 22.8% for the control group. Overall, the intervention led to a 20.6% improvement (OR 1.29, 95% CI: 1.25–1.34) in adherence to clinical recommendations. The differences between the intervention groups were nonsignificant, but all three rates were higher than the rate in the control group and they also showed a tendency to increase with the addition of reminder strategies (Table 5).
Statistically significant differences were observed in all reminders. However, results varied according to the clinical condition The two reminders associated with the highest resolution rates (>30%) were the reminders to administer anticoagulant/antiplatelet treatment and measure blood pressure. The lowest rates were seen for hepatitis B vaccination and smoking cessation encouragement (Table 6).
The most effective reminder (when comparing control and intervention groups) was the diabetic retinopathy screening reminder (OR: 1.51, 95% CI: 1.46–1.57) and the two least effective ones were the reminders to measure glycated hemoglobin (OR 1.1, 95% CI: 1.07–1.13) and encourage smoking cessation (OR: 1.12, 95% CI: 1.09–1.16).
On analyzing the reminders by type (treatment, measurement, screening, vaccination, and lifestyle), measurement reminders had the highest resolution frequencies but the most effective ones were for screening and treatment (Table 7).
The multivariate analyses adjusted for age and sex, type of reminder, and EQA results showed no significant differences for the estimated effect of the intervention.
The electronic on-screen point-of-care reminder intervention analyzed in our study had an effectiveness rate of 20.6%, which is higher than most rates reported to date. A systematic review of 61 clinical trials evaluating reminder systems for preventive care measures reported an average improvement of 12 to 14% . However, in a slightly later Cochrane review of the effect on-screen reminders  Shojaina et al. found an improvement of 4.2% (IQR, 0.8–18.8%), suggesting considerable variability across studies. According to the overview of systematic reviews by Cheung et al.  and the updated Cochrane review , reminder systems continue to have a small or moderate effect.
Our study has several distinguishing features that could account in part for the higher effectiveness rate observed.
We analyzed reminders for different types of clinical conditions. Few of the studies published to date have analyzed the use of simultaneous reminders for more than one condition or situation [10, 15, 24, 25]. One of these studies (perhaps the one that is most similar to ours) analyzed the impact of computerized reminders for 13 clinical situations and reported an effectiveness rate of 10% . The relatively high rate observed in our study is even more noteworthy considering that one of the recommendations for increasing the effectiveness of reminder systems is to limit the number of alerts used , although optimal numbers have yet to be investigated. In our case at least, the generation of reminders for 10 improvable clinical situations (with a mean of 1.49 reminders per patient) was not found to be a barrier to uptake.
Our findings are based on an analysis of almost 1 million weekly reminders generated for over 600,000 patients. Most of the studies to date have analyzed much smaller populations .
The vast majority of studies to date have compared reminder or feedback interventions with no interventions. In our study, the HCPs in the control group continued to receive monthly feedback within a widely used system that had been in place since 2006 . One of the research needs identified in the 2012 Cochrane review was the design of studies comparing reminder systems with other types of feedback systems .
The HCPs in our study were very used to receiving and using feedback, with data from 2007 showing that approximately 71% of physicians consulted their feedback reports every month . In other studies, absence of a significant effect has been attributed to the fact that the reminder systems tested were used by only a minority of the target population .
One might expect that some of the distinctive features of our study design, such as a control group used to receiving feedback, would have reduced the likelihood of finding differences between the groups, but this was not the case. One possible explanation for the larger effect observed in our study is that when designing the system we took into account the criteria for success identified by Kawamoto et al.  and others. The 2012 Cochrane review by Ivers et al.  reported that effectiveness also depended on baseline performance and on how feedback is given. In particular, the authors concluded that feedback was more effective when given by a supervisor or a colleague, when given both verbally and in writing, when given more than once, and when given together with clear targets and an action plan for achieving these. Other authors have found reminders to be more effective when feedback is given regularly, when it includes information on the right solution, and when the reminders are integrated into the clinician’s workflow [12, 26, 28].
In our study, recommendations were delivered to the HCPs weekly rather than monthly as in the existing feedback system. In addition, reminders were generated automatically, shown during the clinical encounter, accompanied by additional information and recommendations, and based on indicators that had been in use for years, increasing thus the likelihood of widespread uptake.
We found no significant differences in effect between the three intervention groups. Although one might have expected the reminder system featuring the filter options to have been associated with better performance due to its greater flexibility, the main differences observed were between the simple pop-up window group and the control group. More studies are probably needed to analyze the use of customizable reminder settings.
Generally speaking, the reminders were effective for all the clinical conditions analyzed, although there were some differences. We observed a larger effect for clear, easy-to-resolve reminders, such as prompts to screen for diabetic retinopathy, and for reminders about potentially serious situations, such as antiplatelet or anticoagulant treatment in patients with heart disease. The reminders with the smallest effect were those related to more complicated or difficult-to-resolve situations, such as smoking cessation, which is a difficult goal to achieve in the short term. Our findings in this regard are consistent with reports that feedback on familiar, acceptable tasks or designed to improve prescribing practices are more effective that other types of feedback . One possible explanation could be that when faced with multiple reminders, clinicians prioritize those that are clinically more important or that can be resolved with less time or effort [23, 26, 29]. This hypothesis should be tested in studies analyzing the effect of simultaneous reminders and the reasons behind prioritization.
Our study has certain limitations. While some authors have suggested that pay-for-performance systems might lead more to improvements in documentation than in care delivery [30, 31], a Cochrane review on the topic concluded that there was insufficient evidence for or against financial incentives having a positive impact on the quality of care in primary settings . In our study, as described in the protocol , the financial rewards associated with the EQA indicator were low and were applicable to the HCPs in both the intervention and control groups. We therefore do not believe that they will have influenced the results.
It was not possible to blind the HCPs to the group assignment and this may have resulted in some bias. Nonetheless, we selected cluster randomization because most studies of this type use this design; we decided against any design in which the same HCP would have some patient records with reminders and others without. In a study by Chambers et al.  on the effect of computerized reminders on vaccination rates in primary care, the physicians were randomized to three arms: one that received no reminders, another that received reminders for all their patients, and another that received reminders for half their patients. The worst performance was observed in the group that received sporadic reminders, suggesting that physicians might come to rely on reminders and in their absence assume that a patient does not need a particular intervention.
The duration of our intervention, 6 months, may have limited our findings in two ways. First, the effect of reminders related to situations that require a longer resolution time might have been diminished. One example is the reminder to encourage patients to quit smoking, which was one of the least effective reminders in our study. Second, the study period may have been too short to detect “reminder fatigue”, limiting thus our ability to estimate long-term effects. Failure to analyze long-term effects is one of the main limitations identified in reviews of reminder system studies . Why then did we stop our study after 6 months when the protocol specified 12 months with an interim analysis at 6 months ? On observing the significant differences between the intervention and control groups at the interim point, considering the health implications, and since no other harm were detected, we decided to discontinue the study and implement the intervention across all the primary care centers. More studies are needed to determine whether the effect observed for the intervention is maintained in time, although such studies are difficult to conduct in everyday practice. One of the main advantages of electronic reminder systems and one that is of particular interest in primary care is that even though these systems do not always have a large effect, they are relatively cheap and easy to manage . This advantage, however, is also a potential stumbling block because as the system is relatively easy to apply (technically speaking, it costs the same to produce results for 1 HCP as for 6000), there is a risk that systems with even the least hint of success could be deployed without the necessary guarantees.
The results of this cluster randomized trial show that an electronic point-of-care reminder system featuring simultaneous reminders on multiple clinical conditions can have a positive effect. They also show that immediate reminders are more effective than periodic feedback, an important consideration for future decision support interventions. We observed variations in the effect of the different reminders used and these should be investigated in further studies, alongside other factors such as long-term effects or the effects of multiple reminders and customizable systems.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
angiotensin-converting enzyme inhibitors
angiotensin II receptor blocker
Estació Clínica d’Atenció Primària (primary care electronic medical record system managed by the Institut Català de la Salut)
Ethics and Clinical Research Committee
Estàndard de Qualitat Assistencial (quality of care indicator)
health care professional
Institut Català de la Salut (Catalan Institute of Health)
ischemic heart disease
Sistema de Informació pel Desenvolupament d’Investigació en Atenció Primària (information system for research in primary care)
Sistema d’Informació dels Serveis d’Atenció Primària (primary care information system managed by the Institut Català de la Salut ICS)
MeSH Browser (2011 MeSH). Bethesda (MD): National Library of Medicine (US), Reminder Systems; [Cited 2017 Aug]. [Internet]. Available from: https://www.ncbi.nlm.nih.gov/mesh?Db=mesh&term=Reminder+Systems
Mc Donald CJ. Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med. 1976;84(2):162–7. [Cited 1976 Feb 1] Available from: http://www.ncbi.nlm.nih.gov/pubmed/1252043
Cheung A, Weir M, Mayhew A, Kozloff N, Brown K, Grimshaw J. Overview of systematic reviews of the effectiveness of reminders in improving healthcare professional behavior. Systematic reviews [Internet]. 2012 Aug 16 [cited 2012 Aug 16];1:36. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22898173
Shojania KG, Jennings A, Mayhew A, Ramsay CR, Eccles MP, Grimshaw J. The effects of on-screen, point of care computer reminders on processes and outcomes of care. The Cochrane database of systematic reviews [Internet]. 2009 Jul 8 [cited 2009 Jul 8];(3):CD001096. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19588323
Holt TA, Thorogood M, Griffiths F. Changing clinical practice through patient specific reminders available at the time of the clinical encounter: systematic review and meta-analysis. Journal of general internal medicine [Internet]. 2012 Aug 10 [cited 2012 Aug 10];27(8):974–84. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22407585
Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ (Clinical research ed) [Internet]. 2002 Oct 26 [cited 2002 Oct 26];325(7370):941. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12399345
Filippi A, Sabatini A, Badioli L, Samani F, Mazzaglia G, Catapano A, et al. Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. Diabetes care [Internet]. 2003 May 1 [cited 2003 May 1];26(5):1497–500. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12716811
Hicks LS, Sequist TD, Ayanian JZ, Shaykevich S, Fairchild DG, Orav EJ, et al. Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. Journal of general internal medicine [Internet]. 2008 Apr 1 [cited 2008 Apr 1];23(4):429–41. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18373141
Krall MA, Traunweiser K, Towery W. Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. Studies in health technology and informatics [Internet]. 2004 [cited 2004];107(Pt 1):135–9. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15360790
Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. Journal of the American Medical Informatics Association: JAMIA [Internet]. 2005 Aug 31 [cited 2005 Aug 31];12(4):431–7. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15802479
Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou X-H, Eckert GJ, et al. Effects of computerized guidelines for managing heart disease in primary care. Journal of general internal medicine [Internet]. 2003 Dec 1 [cited 2003 Dec 1];18(12):967–76. Available from: http://www.ncbi.nlm.nih.gov/pubmed/14687254
van Wyk JT, van Wijk MAM, Sturkenboom MCJM, Mosseveld M, Moorman PW, van der Lei J. Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation [Internet]. 2008 Jan 22 [cited 2008 Jan 22];117(3):371–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18172036
Guiriguet C, Muñoz-Ortiz L, Burón A, Rivero I, Grau J, Vela-Vallespín C, et al. Alerts in electronic medical records to promote a colorectal cancer screening programme: a cluster randomised controlled trial in primary care. The British journal of general practice: the journal of the Royal College of General Practitioners [Internet]. 2016 Jul 6 [cited 2016 Jul 6];66(648):e483–90. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27266861
Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, et al. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ: Canadian Medical Association journal = journal de l’Association medicale canadienne [Internet]. 2003 Sep 16 [cited 2003 Sep 16];169(6):549–56. Available from: http://www.ncbi.nlm.nih.gov/pubmed/12975221
Frank O, Litt J, Beilby J. Opportunistic electronic reminders. Improving performance of preventive care in general practice. Australian family physician [Internet]. 2004 Feb 1 [cited 2004 Feb 1];33(1–2):87–90. Available from: http://www.ncbi.nlm.nih.gov/pubmed/14988972
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ (Clinical research ed) [Internet]. 2005 Apr 2 [cited 2005 Apr 2];330(7494):765. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15767266
Coma E, Ferran M, Méndez L, Iglesias B, Fina F, Medina M. Creation of a synthetic indicator of quality of care as a clinical management standard in primary care. SpringerPlus [Internet]. 2013 Dec 13 [cited 2013 Dec 13];2(1):51. Available from: http://www.ncbi.nlm.nih.gov/pubmed/23450738
Coma E, Méndez L. SISAP: 4 años buceando en mares de datos (AMF 2010) Experiencias para compartir. AMF [Internet]. 2010 [cited 2010];(8):473–6. Available from: http://amf-semfyc.com/web/article_ver.php?id=132
Méndez Boo L, Coma E, Medina M, Hermosilla E, Iglesias M, Olmos C, et al. Effectiveness of computerized point-of-care reminders on adherence with multiple clinical recommendations by primary health care providers: protocol for a cluster-randomized controlled trial. SpringerPlus [Internet]. 2016 Sep 7 [cited 2016 Sep 7];5(1):1505. Available from: http://www.ncbi.nlm.nih.gov/pubmed/27652078
1990. Ryan P. RALLOC: Stata module to design randomized controlled trials. [Internet]. Available from: https://ideas.repec.org/c/boc/bocode/s319901.html
Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, et al. Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA [Internet]. 2000 Sep 20 [cited 2000 Sep 20];284(11):1411–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/10989404
Dexheimer JW, Talbot TR, Sanders DL, Rosenbloom ST, Aronsky D. Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. Journal of the American Medical Informatics Association: JAMIA [Internet]. 2008 Jun 28 [cited 2008 Jun 28];15(3):311–20. Available from: http://www.ncbi.nlm.nih.gov/pubmed/18308989
Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, et al. Audit and feedback: effects on professional practice and healthcare outcomes. The Cochrane database of systematic reviews [Internet]. 2012 Jun 13 [cited 2012 Jun 13];(6):CD000259. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22696318
Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Archives of internal medicine [Internet]. 1996 Jul 22 [cited 1996 Jul 22];156(14):1551–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/8687263
Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. The New England journal of medicine [Internet]. 2001 Sep 27 [cited 2001 Sep 27];345(13):965–70. Available from: http://www.ncbi.nlm.nih.gov/pubmed/11575289
Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM. Exploring barriers and facilitators to the use of computerized clinical reminders. Journal of the American Medical Informatics Association: JAMIA [Internet]. 2005 Aug 31 [cited 2005 Aug 31];12(4):438–47. Available from: http://www.ncbi.nlm.nih.gov/pubmed/15802482
El-Kareh RE, Gandhi TK, Poon EG, Newmark LP, Ungar J, Orav EJ, et al. Actionable reminders did not improve performance over passive reminders for overdue tests in the primary care setting. Journal of the American Medical Informatics Association: JAMIA [Internet]. 2011 Apr 27 [cited 2011 Apr 27];18(2):160–3. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21278102
Hysong SJ. Meta-analysis: audit and feedback features impact effectiveness on care quality. Medical care [Internet]. 2009 Mar 1 [cited 2009 Mar 1];47(3):356–63. Available from: http://www.ncbi.nlm.nih.gov/pubmed/19194332
Loo TS, Davis RB, Lipsitz LA, Irish J, Bates CK, Agarwal K, et al. Electronic medical record reminders and panel management to improve primary care of elderly patients. Archives of internal medicine [Internet]. 2011 Sep 26 [cited 2011 Sep 26];171(17):1552–8. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21949163
Bell CM, Levinson W. Pay for performance: learning about quality. CMAJ: Canadian Medical Association journal = journal de l’Association medicale canadienne [Internet]. 2007 Jun 5 [cited 2007 Jun 5];176(12):1717–9. Available from: http://www.ncbi.nlm.nih.gov/pubmed/17548385
Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does pay-for-performance improve the quality of health care? Annals of internal medicine [Internet]. 2006 Aug 15 [cited 2006 Aug 15];145(4):265–72. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16908917
Scott A, Sivey P, Ait Ouakrim D, Willenberg L, Naccarella L, Furler J, et al. The effect of financial incentives on the quality of health care provided by primary care physicians. The Cochrane database of systematic reviews [Internet]. 2011 Sep 7 [cited 2011 Sep 7];(9):CD008451. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21901722
Chambers CV, Balaban DJ, Carlson BL, Grasberger DM. The effect of microcomputer-generated reminders on influenza vaccination rates in a university-based family practice center. The Journal of the American Board of Family Practice [Internet]. 1991 Feb 1 [cited 1991 Feb 1];4(1):19–26. Available from: http://www.ncbi.nlm.nih.gov/pubmed/1996510
To the SISAP team and to Carlos Dominguez and Celina Ribera. We also wish to acknowledge the contributions of the Catalan Health Institute and all the primary care professionals. Translation to english of this paper has been supported by IDIAP Jordi Gol. We thank Anne Murray for her valuable contribution to translation and copy editing of the manuscript.
The authors neither received, nor will receive, individual or group funding for their work.
Ethics approval and consent to participate
This project obtained ethical approval from the Ethics and Clinical Research Committee (ECRC) at Institut Universitari d’Investigació en Atenció Primària Jordi Gol, ref.: P11/62.
Given the nature of the intervention, it was not possible to carry out blinding of the primary care professionals but patients did not receive any information regarding the development of the study. Therefore, the need for consent to participate was waived by the ECRC.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Coma, E., Medina, M., Méndez, L. et al. Effectiveness of electronic point-of-care reminders versus monthly feedback to improve adherence to 10 clinical recommendations in primary care: a cluster randomized clinical trial. BMC Med Inform Decis Mak 19, 245 (2019). https://doi.org/10.1186/s12911-019-0976-8
- Reminder systems
- Quality of health care
- Primary health care
- Electronic health record