Online information retrieval systems have the potential to improve patient care but there are few comparative studies of the impact of online evidence on clinicians' decision-making behaviour in routine clinical work.
A randomized controlled parallel design is employed to assess the effectiveness of an online evidence retrieval system, Quick Clinical (QC) in improving clinical decision-making processes in general practice. Eligible clinicians are randomised either to receive access or not to receive access to QC in their consulting rooms for 12 months. Participants complete pre- and post trial surveys.
Two-hundred general practitioners are recruited. Participants must be registered to practice in Australia, have a computer with Internet access in their consulting room and use electronic prescribing. Clinicians planning to retire or move to another practice within 12 months or participating in any other clinical trial involving electronic extraction of prescriptions data are excluded from the study.
The primary end-points for the study is clinician acceptance and use of QC and the resulting change in decision-making behaviour. The study will examine prescribing patterns related to frequently prescribed medications where there has been a recent significant shift in recommendations regarding their use based upon new evidence. Secondary outcome measures include self-reported changes in diagnosis, patient education, prescriptions written, investigations and referrals.
A trial under experimental conditions is an effective way of examining the impact of using QC in routine general practice consultations.
Several studies have shown that clinicians have many unanswered questions during clinical encounters which may impact on the quality and outcomes of decisions made. Depending on measurement techniques used to estimate the frequency of questions and the clinical setting, clinicians have anywhere between 1 and 6 questions per patient encounter and answers are pursed for only about 30% to 55% of questions [1–4]. Improving access to clinical evidence through the implementation of evidence retrieval systems is one way of supporting information seeking in routine care . Easy access to up-to-date evidence at the point of care may be more effective than other interventions aimed at improving clinicians adoption of evidence-based practice such as CME, distribution of clinical guidelines, clinical detailing, etc.
The effect of implementing information retrieval technology in routine clinical settings and its impact on clinicians' decision-making has not been extensively investigated. A 2005 review found that on average one in three searches for information were associated with a positive impact on decision-making, however in 73% (19 of 26) of the studies examined in this review, the impact was self-reported by clinicians using the system . Assessments of evidence retrieval systems have relied principally on post study surveys. Only a small number have directly measured use in clinical settings and in most cases patterns of searching and clinical usefulness of information retrieved could not be identified.
There are few comparative studies of online evidence retrieval technology in routine clinical settings. A Finnish study found no difference in compliance with recommendations between computerised and paper-based clinical guidelines among newly qualified physicians . In another study involving fourth year medical students, participants reported that access to information on a handheld computer encouraged use of information, improved learning and increased confidence in decision-making . However, the effect of information retrieval technology on clinicians decision-making process and impact upon patient outcomes has not been assessed.
We are conducting a randomised controlled trial (RCT) to investigate the effectiveness of an information retrieval system called Quick Clinical (QC) in general practice . The feasibility of the QC system was demonstrated in a previous clinical trial, detailed results of this study are reported in a separate paper . Overall, 193 clinicians used the online evidence system to conduct on average 8.7 searches per month. The majority of these searches were conducted from consulting rooms (81%) during office hours. The most frequent searches conducted related to questions about diagnosis (40%) and treatment (35%). Search subjects included a broad spectrum of diseases, including common conditions such as asthma, diabetes and hypertension. In this paper we describe the design and proposed analysis of an RCT to evaluate the effectiveness of using QC in routine general practice consultations.
The aim of this trial is to assess the effectiveness of an online evidence retrieval system (QC) in improving clinical decision-making processes in general practice.
The specific hypotheses to be tested in this study are that:
Online evidence systems are clinically acceptable and will be used by clinicians in 'real world' general practice settings.
Online evidence systems are effective in changing clinical decision-making behaviour and result in measurable improvements in evidence-based prescribing decisions.
Study design and setting
A randomized controlled parallel design is used to evaluate the effectiveness of QC.
Participants are randomised to receive access to QC in their practice for 12 months. The system is available from a standard web browser e.g. Microsoft Internet Explorer, Firefox etc. (Figure 1). Each user has a personal username and password and completes an online tutorial about how to use the system. QC is based on the generic use of search filters specifically designed around the information needs of general practitioners (GP). Five types of search filters or "profiles" are available (disease aetiology, diagnosis, treatment, prescribing and patient education). Up to four keywords can be used to describe a clinical question. Clinicians search for information firstly by selecting a search filter to match their question type (e.g. diagnosis, treatment etc.) and then entering keywords to describe the query. For example, a clinician who encounters a 32-year old woman with a fourth presentation of pelvic pain in the last 6 months with ultrasound and swabs for infection all negative, may have a question regarding the social, psychological as well as biological causes of pelvic pain. The clinician could select the 'etiology' profile and enter 'pelvic pain', 'pathology' and 'psychosocial' as keywords (Figure 1). The search filters retrieve evidence from resources including Therapeutic Guidelines (a summarised evidence resource developed by Australian experts); the Merck Manual; Health Insite (a government funded health database); PubMed; MIMS (a pharmaceutical database); and a collection of relevant Australian guidelines identified by a panel of clinical experts. Users can also search each of these resources individually.
The primary outcome measures are clinician acceptance and use of QC and the resulting change in decision-making behaviour. The data from our pilot study of QC involving 227 GPs showed that online evidence was sought in relation to both frequent and rare clinical conditions across a broad range of clinical areas . This suggests that we are likely to see access to evidence having a broad impact across common and rare clinical decisions. The RCT will therefore examine prescribing patterns related to frequently prescribed medications where there has been a recent significant shift in recommendations regarding their use based upon new evidence. Examining the effect on management decisions in clinical areas where there is shifting evidence enables a hypothesis driven approach to determine the impact of online evidence on clinician behaviour.
The RCT will therefore focus on the changes in clinicians' prescribing patterns in Australian National Health priority areas identified by a panel of experts including chief investigators, associate investigators and general practitioners (Table 1). Non-pharmacological clinical management in these priority areas will also be examined e.g. referral for podiatry (diabetes management), spirometry (asthma), HbA1c testing (diabetes), cognitive behavioural therapy (CBT) for depression, dieticians for lipid disorder. In addition to changes in priority areas, prescribing patterns in response to new evidence of the effectiveness of new or existing treatments identified through reviews of literature and expert input will also be assessed. Secondary outcome measures to determine impact on decision-making behaviour include self-reported changes in diagnosis, patient education, prescriptions, investigations and referrals.
Clinical priority areas used to measure primary outcomes.
Expected changes in intervention group
↓ short acting beta-agonists
↑ long-acting beta-agonists
↑ inhaled corticosteroids (decreased dose of steroids over time i.e. back titration or combinations of LABA & corticosteroids)
↑ SSRIs, ↑ SNRIs
↓ other antihypertensives
Upper respiratory tract infection
↓ antibiotics and ↑ appropriate antibiotics
Higher rate of immunization and closer compliance with schedule.
Type 2 Diabetes
↑ glitazones ↑ metformin
Non-inflamatory musculoskeletal including osteoarthritis
↓ Cox-2 inhibitors
Ethical approval for the study was obtained from the University of New South Wales's Human Research Ethics Committee and subsequently ratified by Sydney University and the Royal Australian College of General Practitioners ethics committees.
The study is also recognised by the Royal Australian College of General Practitioners (RACGP) for its continuing medical education (CME) program. Education points are not directly linked to participants' use of QC, but to trial completion.
Identification of eligible general practitioners
GPs who had a computer with Internet access in their consulting rooms and used electronic prescribing are recruited via a call for volunteers advertised in journals, newsletters and a clinician list-server. An overview of the trial process is shown in Figure 3. Volunteers are asked to register via a web page and to complete a short survey to determine their eligibility to participate in the study. The online registration gathers baseline data about the demographics of the sample including practice characteristics and computer use [see Additional file 1]. Eligibility and exclusion criteria are listed in Table 2. To participate in the study clinicians must be registered to practice in Australia; have a computer with Internet connectivity; and prescribe electronically. GPs who are planning to retire or move to another practice within 12 months or participating in other concurrent trials involving electronic extraction of prescription data are excluded from the study.
Study inclusion and exclusion criteria.
• general practitioners registered to practice in Australia
• use a computer with Internet access in consulting rooms
• use electronic prescribing
• concurrent participation in other clinical trials involving electronic extraction of prescription data
• planning to retire or move within the next 12 months
Timing and content of study assessments.
Identifying eligible clinicians
Trial registration: self completed online
Pre-trial survey: self completed online
Prescription data from previous 12 months: electronic extraction
Clinician follow-up procedures
Primary outcome measures: Computer log and electronic prescription data at 6 and12 months
• Physician acceptability focusing on ease of use and usefulness (post-trial survey) and patterns of QC use (computer logs)
• Prescribing patterns in clinical priority areas identified at the start of the study
• Prescribing patterns in response to new evidence of the effectiveness of new or existing treatments
• Patterns of non-pharmacological clinical management
Secondary outcome measures: self-reported in online post-trial survey at 12 months
• Referral patterns
• Management decisions
• Number, timing and types of investigations
Sample size estimation (statistical power: 80% power, at the 5% significance level).
Annual rate of medication use per 100 problem-specific encounter
Sample size (no. of GPs required in each arm)
Asthma- use of preventers
Asthma- SA Bronchodilators
Summary of analyses for the RCT of Quick Clinical.
Analyses to determine impact of Quick Clinical on decision-making
1. Baseline comparisons between control and intervention groups (unadjusted data)
a) proportion of prescriptions by ATC* categories and in priority areas (using 12 months retrospective data)
b) case-mix (age-gender distribution of patients)
c) participants' profile (gender, age, place of graduation, geographic distribution, computer skills)
2. Interim analyses at 6 months
a) analyses of baseline prescribing data. Reports of interim analyses affecting the ongoing conduct of the trial and interpretation of final results
3. Using an intention to treat analysis (i.e. irrespective of patterns of QC use in the intervention group) we will include all participants in the primary analyses using adjusted data to determine:
a) broad differences in proportion of prescriptions by ATC classification
b) specific differences in proportion of prescriptions by priority area (e.g. Antibiotics prescribed for URTI)
c) differences in prescription patterns in response to new evidence of the effectiveness of new or existing treatments
d) differences in non-pharmacological treatments
e) analysis of prescription changes relative to search categories
*Anatomical Therapeutic Chemical classification system
Eligible GPs are randomly allocated to either the control or intervention group, each clinician is given a unique study identification code and the web address of QC is sent to eligible participants one week prior to commencement of the trial. Participants visiting the QC web site are given detailed information about the trial along with the participant information statement and the consent form. A revocation of consent form is also available online. Clinicians willing to participate complete the online consent form and a pre-trial survey which assesses information seeking habits and attitudes to information seeking during consultations [see Additional file 2]. A psychometric instrument used to classify clinicians' response styles to new information as seekers, receptives, traditionalists, or pragmatists is also administered as part of the pre-trial survey [see Additional file 3] .
The timing and content of study assessments are summarised in Table 5. Frequency and purpose of system use of QC are determined from automatically generated computer logs used to record details of each search including the search filter chosen, keywords entered, data sources accessed, user IP (Internet Protocol) address, date, time and duration of searches [see Additional file 4]. Users are asked to identify if the IP address is their home or consulting room. An online feedback facility within the QC user interface is used to record clinicians comments in context of specific searches that generate the feedback (Figure 2).
Participants in both the control and intervention group provide prescribing data in electronic format [see Additional file 5]. The data extraction process excludes all potentially identifying information such as patient name, address, gender, postcode, telephone number, Medicare number, progress notes, images (e.g. X-rays) and any data fields that could identify a patient, such as free text fields. Data for the study period (i.e. 12 months retrospective data plus 12 months of the intervention period) is electronically extracted from prescribing software using secure means without any intrusion on clinicians' work.
The prescription data extraction software and installation instructions are sent by email and also are made available online. The first data extraction (i.e. retrospective prescriptions data from the previous 12 months) is performed after installation of the extraction software at the start of the study. The second extraction is scheduled at 6 months and the final at 12 months. An uninstall program will remove all extraction software, created files and changes to profiles in clinicians' prescribing software. At the end of 12 months all participants complete a post-trial survey [see Additional file 6]. Participants attitudes to information seeking and responses to the psychometric instrument used in the pre-trial survey are repeated post-trial.
Randomisation, blinding and allocation concealment
Eligible GPs are stratified by gender and randomised to the intervention or control group using a computer generated random number sequence in randomly assigned blocks (max. block size 8) and online evidence allocation ratio of 1:1. For each level of strata, a random sequence of study group assignments is generated using a computerized random-number generator . Each level of strata are numbered consecutively. Each doctor receives the next consecutive number in his or her stratum.
Since QC is a behavioural intervention it is not possible to blind participants to the intervention. However investigators are blinded to group assignment. The randomisation sequence is generated and clinicians are randomly assigned to two groups by a researcher from the Evaluation team of the Centre for Health Informatics. This individual is not directly involved in the study nor its analysis.
The group assignment is revealed to clinicians online only after they consent to participate in the study following completion of the pre-trial survey. To minimise contamination of control participants who might work closely with practitioners who are part of the intervention group, only one clinician is eligible to participate in the trial from each practice. In cases where there are multiple registrations from a single practice, only the first registrant is eligible to participate. Participants in the intervention group are asked not to share their QC access details with their colleagues.
Communication with participants
Email will be the primary channel to communicate with participants for information and reminders about:
start of the trial (followed up with 2 fortnightly reminders)
instructions for installing and using prescribing data extraction tool
education points notification and summary of main results
Participants who formally withdraw from the study are contacted by telephone to determine reason for withdrawal. Similarly participants who do not use QC nor provide prescribing data are contacted to determine reasons for not completing the study.
Sample size considerations
Sampling unit is the GP and the unit of analysis is also the GP. We provide the following example to demonstrate that the RCT sample size will be more than sufficient for detecting statistically significant prescription changes in medication areas where there is a shift in evidence. The example used to verify the design is measuring a 10% increase in use of preventer medications (glucocorticoid inhaled medications) in Asthma. Currently on average, 53% of GP encounters for asthma lead to the prescription of a preventer medication . We expect that access to QC will promote the use of preventer medication by GPs who use the intervention. We expect that in the intervention group – at least 58.4% of GP encounters for asthma lead to the prescription of a preventer medication.
As the prescriptions belonging to the same GP will tend to be correlated, the sample size calculations estimated the average number of condition-specific prescriptions per GP and adjustments were made for intracluster correlation factor estimated to be 0.02 .
The number of GPs required in each arm to detect a 10% change in average prescription rate per 100 condition-specific encounters per GP are listed in Table 4. Given that data collection in the trial will run for a minimum of 12 months 100 GPs in each arm will be sufficient to detect a 10% change in prescriptions (p = 0.05, power = 0.80).
All registered participants will have completed the pre-trial survey. The subset of the intervention group analysed for primary outcome will have completed the post-trial survey and provided prescribing data. Participants in intervention group who encounter technical difficulties in using QC will be excluded from this analysis (i.e. those who had the opportunity to use QC but did not do so will be included). All of the control group except those who did not provide prescribing data will be included in the primary analysis. As summarised in Table 5 the intervention and control groups will be compared for baseline differences in prescription patterns, case-mix and profile of participants. Using an intention to treat analysis adjusted data will be analysed to determine differences in prescribing patterns and non-pharmacological management in identified priority areas, and areas in which there have been recent changes in response to new evidence using a one-tailed binomial test for proportions. Self-reported changes in diagnosis, patient education, prescriptions, investigations, referrals will also be examined. Utilisation and outcome measures will be analysed relative to QC usage and patient load (e.g. by number of patients seen).
Time plan for the RCT of Quick Clinical
Recruitment commenced in October 2004, 203 GPs volunteered to participate in the study and the trial formally commenced in May 2005. The trial under experimental conditions will evaluate the effectiveness of using QC in routine general practice consultations.
We wish to thank the clinicians who gave their time to take part in this study. The authors would like to thank Ms. M. Williamson for her contribution to the statistical analyses and invaluable comments that have much improved the Quick Clinical study protocol. Ms. Williamson participated in this research while a Senior Research Fellow at the Centre for Health Informatics, University of New South Wales.
The study is partially funded by an Australian National Health and Medical Research Council (NHMRC) Project Grant. Financial support to develop Quick Clinical was provided by an Australian Research Council SPIRT grant and an NHMRC Development Grant. We thank Health Communication Network (Limited) for providing software tools to collect prescribing data. The Centre for Health Informatics is supported by a New South Wales Health Capacity Building Infrastructure Grant.
Centre for Health Informatics, University of New South Wales
Discipline of General Practice, University of Sydney
Physiology & Pharmacology, School of Medical Sciences, Faculty of Medicine, University New South Wales
St Vincent's Hospital
Covell DG, Uman GC, Manning PR: Information needs in office practice: are they being met?Ann Intern Med 1985,103(4):596–599.PubMed
Ely JW, Osheroff JA, Chambliss ML, Ebell MH, Rosenbaum ME: Answering physicians' clinical questions: obstacles and potential solutions.J Am Med Inform Assoc 2005,12(2):217–224.View ArticlePubMed
Gorman PN, Helfand M: Information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered.Med Decis Making 1995,15(2):113–119.View ArticlePubMed
Smith R: What clinical information do doctors need?BMJ 1996,313(7064):1062–1068.PubMed
Coiera E, Dowton SB: Reinventing ourselves.Med J Aust 2000,173(7):343–344.PubMed
Pluye P, Grad RM, Dunikowski LG, Stephenson R: Impact of clinical information-retrieval technology on physicians: A literature review of quantitative, qualitative and mixed methods studies.Int J Med Inform 2005.
Jousimaa J, Makela M, Kunnamo I, MacLennan G, Grimshaw JM: Primary care guidelines on consultation practices: the effectiveness of computerized versus paper-based versions. A cluster randomized controlled trial among newly qualified primary care physicians.Int J Technol Assess Health Care 2002,18(3):586–596.PubMed
Leung GM, Johnston JM, Tin KY, Wong IO, Ho LM, Lam WW, Lam TH: Randomised controlled trial of clinical decision support tools to improve learning of evidence based medicine in medical students.Bmj 2003,327(7423):1090.View ArticlePubMed
Coiera E, Walther M, Nguyen K, Lovell NH: Architecture for knowledge-based and federated search of online clinical evidence.J Med Internet Res 2005,7(5):e52.View ArticlePubMed
Magrabi F, Coiera EW, Westbrook JI, Gosling AS, Vickland V: General practitioners' use of online evidence during consultations.Int J Med Inform 2005,74(1):1–12.View ArticlePubMed
Green LA, Gorenflo DW, Wyszewianski L: Validating an instrument for selecting interventions to change physician practice patterns: a Michigan Consortium for Family Practice Research study.J Fam Pract 2002,51(11):938–942.PubMed
Britt H, Miller GC, Knox S, Charles J, Pan Y, Henderson J, Bayram C, Valenti L, Ng A, Pan Y, O'Halloran J: General Practice Activity in Australia 2004–05. Canberra , The University of Sydney and the Australian Institute of Health and Welfare 2005.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.