- Research article
- Open Access
Health care providers’ perceptions of use and influence of clinical decision support reminders: qualitative study following a randomized trial to improve HPV vaccination rates
BMC Medical Informatics and Decision Making volume 17, Article number: 119 (2017)
Human Papillomavirus (HPV) leads to serious health issues and remains the most common sexually transmitted infection. Despite availability of effective vaccines, HPV vaccination rates are suboptimal. Furthermore, providers recommend the HPV vaccine less than half the time for eligible patients. Prior informatics research has demonstrated the effectiveness of computer-based clinical decision support (CDS) in changing provider behavior, especially in the area of preventative services.
Following a randomized clinical trial to test the effect of a CDS intervention on HPV vaccination rates, we conducted semi-structured interviews with health care providers to understand whether they noticed the CDS reminders and why providers did or did not respond to the prompts. Eighteen providers, a mix of medical doctors and nurse practitioners, were interviewed from five publicly-funded, urban health clinics. Interview data were qualitatively analyzed by two independent researchers using inductive content analysis.
While most providers recalled seeing the CDS reminders, few of them perceived the intervention as effective in changing their behavior. Providers stated many reasons for why they did not perceive a change in their behavior, yet the results of the trial showed HPV vaccination rates increased as a result of the intervention.
CDS reminders may be effective at changing provider behavior even if providers perceive them to be of little use.
ClinicalTrials.gov Identifier: NCT02551887
Registered on September 15, 2015
Human Papillomavirus (HPV) is the most common sexually transmitted infection in the U.S., with approximately 79 million Americans already infected and 14 million new cases each year . Infection with HPV is a causal factor for serious health issues including cervical cancer, anal cancer, penile cancer, oropharyngeal cancers, genital warts, and recurrent respiratory papillomatosis .
Despite the availability of a 9-valent HPV vaccine (9vHPV) that prevents up to 80–90% of cervical cancers and 90% of genital warts , HPV vaccination rates in the U.S. remain lower than desired to best protect the population against HPV infection . In 2015, only 62.8% of adolescent girls and 49.8% of adolescent boys ages 13 through 17 years received one or more doses of vaccine . The percentages are even lower for series completion (41.9% of girls and 28.1% of boys).
Existing research demonstrates that many physicians do not strongly endorse HPV vaccination or do not deliver timely recommendations [6, 7]. This is of particular concern because one of the strongest predictors of vaccine uptake is healthcare provider recommendation, and a lack of provider recommendation has been reported as a key reason for non-vaccination [8,9,10,11,12].
Computer-based clinical decision support (CDS), which provides appropriate, timely, patient-specific reminders and information to providers, can be effective at changing provider behavior. When implemented effectively, CDS has been shown to improve quality of care [13,14,15,16,17], and can be particularly effective for increasing appropriate use of evidence-based preventive services [14, 16, 18]. The federal “meaningful use” program for electronic health record (EHR) adoption in the U.S. further offers incentives for use of CDS . The first stage of the meaningful use program required providers to adopt at least one CDS rule, and the second stage requires that they implement at least five CDS interventions that promote their institutional quality goals . Later stages of the meaningful use program further encourage submission of information following vaccine administration to public health departments .
Researchers at Indiana University and the Regenstrief Institute possess a long history in using CDS to improve health outcomes and preventive services . Pioneering studies in both inpatient and outpatient settings at Regenstrief provided CDS reminders to providers that encouraged the adoption of preventive services, including influenza and pneumococcal vaccination for eligible patients [23, 24]. In one study , over half of the 6371 patients admitted to a general medicine service during an 18-month period were eligible for one or more of the preventive CDS reminders. Provider ordering rates (intervention vs. control) were 35.8% vs. 0.8% for pneumococcal vaccination and 51.4% vs. 1.0% for influenza vaccination (p < 0.001 in both cases).
Objectives of the study
Given poor HPV vaccination rates in Indiana and prior research on CDS system impact on provider behavior that increased vaccination for influenza and pneumococcal diseases, we designed a randomized clinical trial to test the effect of CDS on HPV vaccination. Like previous CDS efforts, our study offers the opportunity to learn about how providers respond to CDS prompts in the context of health care delivery.
A quantitative analysis of the trial is published elsewhere . Another publication from our study examines provider awareness of the nine-valent form of the HPV vaccine approved for use at the beginning of our study . Our team further published an examination of providers’ general perceptions of HPV vaccination and whether risk compensation influences recommendation of the vaccine to adolescent patients .
In this paper, we summarize the qualitative analysis of interviewers conducted after the trial in order to understand whether providers noticed the CDS reminders, which prompted providers to recommend HPV vaccination. We further examined why providers did or did not respond to the prompts (i.e. changed behavior).
Eskenazi Health is one of the five largest safety net health systems in the United States. The health system contains a 315-bed hospital and nine community health centers located across the metropolitan area of Indianapolis, the eleventh largest city in the United States. There are five pediatric clinics among the nine health centers.
The Child Health Improvement through Computer Automation system (CHICA) is an operational CDS system that has been used to support clinical practice for 12 years  and functions as a front end to an EHR system. When a child is registered in an Eskenazi pediatric clinic, the registration system sends an HL7 ADT (registration) message to CHICA. In response, CHICA queries a copy of the patient’s medical record from the EHR system. CHICA then applies hundreds of Arden Syntax rules to the data to select 20 structured, simple “Yes or No” questions that are displayed on an electronic tablet provided to the patient’s family in the clinic waiting room . Family members answer the questions and return the tablet to a medical assistant who then enters the child’s height, weight and other measurements into the tablet. The information from the family and the medical assistant are then uploaded back into CHICA.
At the same time that CHICA produces the questions for the family to answer, the system sends an HL7 request to CHIRP (Children and Hoosier Immunization Registry Program), Indiana’s immunization information system or IIS [30, 31]. In response, CHICA receives a download of the child’s immunization history. The download includes CHIRP’s “forecast” of the immunizations for which the patient is due. The electronic transfer of immunization information between CHICA and CHIRP is a form of health information exchange (HIE) .
At the end of both processes, CHICA produces several paper documents. The first is the physician worksheet. This worksheet includes up to six alerts and reminders for the provider. The reminders are selected by CHICA, using its Arden Syntax rule set, and are based on the patient’s EHR data as well as the data entered into the tablet. Each alert has up to six check boxes with which the physician can document how s/he responded to the alert. The physician worksheet, when completed, is scanned, and the coded data corresponding to the check boxes are stored along with any provider notes back to the child’s medical record. CHICA may also produce any of a large number of handouts for helping the provider with assessment or patient education. CHICA also produces a summary of the patient’s immunization history along with advice on shots for which the child is due . The technical architecture is summarized in Fig. 1, which depicts the exchange of information between CHICA and the EHR as well as CHIRP, and the output of the process in the context of the randomized study.
To study CDS reminders for HPV vaccination, we conducted a randomized clinical trial (RCT) in which clinical reminders to recommend HPV vaccination were targeted to providers in pediatric outpatient clinics . Randomization occurred at the level of provider. Providers were randomized to one of three arms: 1) usual care or the control arm in which no reminders for HPV were provided but the electronic health record (EHR) system was still used and CDS may prompt the providers regarding other vaccinations; 2) CDS reminders were provided to providers via the EHR system for HPV and other adolescent platform vaccines (prompt group); and 3) providers were provided both CDS reminders for vaccines plus a script to use when recommending the adolescent vaccines (prompt + script group).
The CDS reminders prompted providers to recommend the three adolescent platform vaccines: meningococcal (MCV4), HPV, and tetanus, diphtheria, and pertussis (Tdap) vaccines. The script provided to providers in the third arm read, “Three vaccines are recommended for <patient first name>, meningococcal to prevent meningitis, HPV to prevent cancer, and Tdap to prevent tetanus. All three are recommended at this age”. Eligible patients were children 11–14 years of age who had no previous history of HPV vaccination and no previous history of either MCV4 or Tdap. Eligibility was determined by querying both the EHR system (described below) and the state’s IIS (CHIRP) to establish a lack of prior vaccination history.
As a component of this larger RCT, we conducted semi-structured interviews with providers in each clinic between January and March 2015. The RCT and interview protocols were approved by the Indiana University Institutional Review Board. The primary, quantitative results of the RCT are published elsewhere. This article focuses on a qualitative analysis of interviews conducted with providers.
Participants for this study were pediatric providers working in publicly-funded urban health clinics, had patients 11–14 years of age who were in need of vaccination, and consented to be interviewed. All eligible providers, medical doctors as well as nurse practitioners, were contacted via e-mail. Two additional follow-up e-mails were sent to each participant who did not respond to the initial e-mail. A total of 29 providers were eligible to be interviewed and 18 (62.1%) consented and completed the interview. Participants were recruited until saturation was reached, that is, until we acquired limited new information from the interviews .
Qualitative methodology is ideal when exploring an area where little is known, because it allows the investigators to identify, via in-depth analysis, relevant personal and contextual factors . The majority of the interviews were conducted face-to-face (n = 16), but some were conducted over the phone if the provider could not meet in person (n = 2). All interviews were one-on-one and conducted by one of the co-authors (MLK). Interviews lasted 15–30 min, and participants were compensated with a $50 gift card.
After providing brief information regarding the study, all participants were asked about their general beliefs regarding HPV and HPV vaccination. Providers in the prompt and the prompt + script groups were then asked if they noticed the prompt and whether the prompt influenced their vaccination behavior. Providers in the prompt + script group were also asked if they noticed the script and if the script influenced their vaccination behavior or their conversations with the family. Participants in all three arms were asked if they discussed the CDS reminders and/or scripts with other providers in their clinic in order to assess contamination. Along with these questions, demographic characteristics (including sex, race/ethnicity, and years in practice) as reported during the interview were also collected. Discussions with participants were guided by a semi-structured interview guide [see Additional file 1].
Interviews were audio-recorded, transcribed, and analyzed using inductive content analysis . Content analysis is a qualitative research method that systematically and objectively describes phenomena [37, 38]. Research questions guide the identification of concepts, categories, themes, and conceptual models that are derived from qualitative data via insight and intuition of the researcher(s) . Content analysis can be performed using induction or deduction techniques . We employed inductive content analysis, which involves open coding of the transcripts, iterative generation of the categories or themes through discussion among the researchers, and summarization of the themes through narrative iterations.
Transcripts of the interviews were read by the authors to identify meaningful themes. Two investigators (MLK & SW) then independently coded each interview according to those themes. The codes were reviewed and areas of disagreement were resolved through discussion with the lead author (BED). Using the coded transcripts, we generated descriptive, summary information about each theme.
Here we present information on the respondents and the analysis of respondents’ comments. First, we present demographic information on the respondents. Next we present the key themes that emerged from analysis of the interview data.
Table 1 summarizes the demographic and participation data of the 18 participating providers. Most providers (n = 14) were women. Participants reported practicing medicine for a mean of 13.7 years (M = 10.5 years at their current clinic). The majority (n = 10) of providers were Caucasian, four identified themselves as African American, and four identified with ‘other.’ All arms of the trial were represented with slightly more providers (n = 8) from the CDS reminder plus script arm.
Results of thematic analysis
Five major themes, summarized in Table 2, emerged from respondents’ comments. In response to specific questions during the interview, providers commented on both their awareness and utilization of the CDS reminders. When probed, providers commented on the reasons why they did not utilize the CDS reminders. The providers further described their perceptions of the effectiveness of the CDS reminders given their perceived use or non-use. Finally, providers suggested that their awareness of patient eligibility for a given vaccine is sometimes driven by information derived from nurses and not EHR systems.
Awareness and utilization of CDS reminders
Of the providers who received the CDS reminders (n = 13), those in the second and third arms, the majority (n = 9, 69%) indicated they noticed them. Many of these providers further held a positive view of the reminders, indicating that CDS reminders in general help them do their job. One provider noted, “If CHICA’s going to help me do my job, I think that that should be welcomed.” Yet several providers also stated that, while they held positive views of reminders, they did not believe the interventions changed their behavior, “…but I didn’t change my behavior, like because it’s already drilled into our heads to just do the vaccines.”
Reasons for non-use of CDS reminders
Providers gave several reasons for not using, or ignoring, the clinical reminders. Many stated they were aware that all 11–12 year old patients are due for the adolescent platform vaccines and therefore do not need a reminder. When one provider was asked how she knew her patients were due for the HPV vaccine she responded, “Well, there are a couple ways. One is just realizing that usually around age 11 we vaccinate for [HPV].”
Providers noted that sometimes they do not look at the reminders, because they do not have time during a visit. As one provider stated, “I would say that sometimes we don’t have a chance to get to everything that it’s prompting, though, too, so sometimes—and there’s a list of maybe six, at least, items there, and it may happen in the visit that we ended up discussing other things for whatever reason.” Providers perceive they know what they are supposed to do at routine visits, and sometimes they will look at the reminders as a whole, after the patient leaves. For one particular physician, if they missed a prompt they deemed as “scary”, they will follow-up with a phone call to the patient. However, the prompts the providers deem as worthy of follow-up usually include those indicating imminent danger to the patient such as suicidal thoughts or food insecurities.
One of the most common reasons providers listed for not looking at the reminders was that the prompts are on paper and the patient chart is now on a computer, and switching back and forth between the two interrupted workflow. Many noted that it was easier to review the reminder sheets when the chart was also on paper but, now that they have transitioned to all electronic charts, the papers generally get ignored. One provider stated, “I don’t review them as much as I used to because we are more electronically oriented and carrying my laptop in the room… so the CHICA prompts are probably the last thing I might even look at.” Another said, “I was much better about it when we were charting on the paper because it was like right there but now honestly there are so many things that are printed out and everything that I need is on the computer in terms of the stuff I need to access during the visit.”
Providers also indicated that they have stopped paying close attention to the reminders because they are often incorrect or the parent misunderstood the prompt and answered incorrectly. They indicated that they know these families and have a rapport with them and can judge their risks better than the reminder system. One said, “I appreciate the prompts, but on the other hand, sometimes we know the patient better than the computer does.” Along the same lines, one provider recalled a prompt to discuss gun safety because there may be a gun in the home, but the provider said, “I know the family, especially if I know that they don’t have guns or things or that nature and they print out the gun sheet, then I don’t introduce that information to them.” Additionally, some providers said the parents do not understand the questions as they are presented on the tablet and may answer incorrectly, which gives the provider an inaccurate prompt. For example, one provider said, “I’ll ask about the prompts because it indicates the patient answered a certain way, but when I talk to the parent about it, they misunderstood the question or they didn’t think that that’s what they were answering and it doesn’t apply.” There is also the possibility that the parent is illiterate and cannot read the questions, or the parent may be comfortable answering a question on a tablet/computer but uncomfortable discussing it in-person with the provider and in front of their child. One provider mentioned this and said, “If you’re illiterate, some people may have issue saying, ‘I can’t read this’ or they may not want to say that in front of their kid.” Providers distrust prompts they receive consequently relying on their relationships and common practice to guide their encounters with their patients.
Effect of suggested script for recommending HPV vaccine
Of the group that received both the CDS reminder and suggested script (n = 8), only one provider indicated he noticed the script. However, this provider indicated he did not use the script because, as he said, “I kind of don’t go with the script directly from that script to the parents. I’ve already discussed it in my way so I don’t really…I would say that that script that is in the CHICA form is not routinely used directly to be verbally said to the family.” Other providers said they did not notice it because they already have their “spiel” and know what they are going to say. When asked if they noticed or used the suggested script, one provider said, “Oh, no, I don’t need help with that. I know what I need to say.”
Role of nurses in supporting vaccination processes
A final key theme was the effectiveness of nurses in reminding the provider that a patient is due for vaccination. When asked how he knew his patients were overdue for vaccines, one provider simply stated, “My nurse tells me.” Along those same lines, when asked about reminders for vaccination, 9 of the 18 providers specifically mentioned their nurse. One stated, “My nurse is just, she’s a hawk when it comes to vaccines.”
We interviewed 18 pediatric providers regarding their awareness, utilization and perceived effectiveness of CDS reminders implemented as part of an RCT designed to improve HPV vaccination rates. Among the providers who received the reminders, most reported noticing the CDS reminders or suggested language designed to improve how providers recommend HPV vaccination. Furthermore, respondents perceived the CDS reminders as useful but unlikely to influence vaccination rates given provider self-confidence in current performance and a variety of workflow issues associated with utilization of CDS reminders during a clinical encounter.
The contribution of this study is the detailing of reasons why providers did or did not accept CDS reminder and/or scripts provided to encourage HPV vaccination. These reasons suggest three important lessons for the broader biomedical informatics community, which includes developers, implementers and researchers. First, the use of qualitative methods to solicit feedback from providers is important to understand the impact of a health information technology system. Second, feedback from providers highlights the continued need for biomedical informatics system developers and implementers to focus their efforts on optimizing clinical workflow. Finally, the project demonstrates the potential for HIE, or integration between EHR and IIS systems, to improve population health. These results go beyond our prior examinations of pediatric providers’ general perceptions of the HPV vaccine and influences on whether or not they recommend the vaccine to adolescent patients [26, 27].
Using mixed methods to understand mixed results
The quantitative analysis of the RCT, published elsewhere , found that the third arm of the study had a significantly higher rate of HPV vaccination than the control arm (62% vs 45%, p < 0.05), yet the rates of MCV4 and Tdap vaccination were equivalent across all three arms (MCV4: 81%, 81%, 83%; Tdap: 82%, 83%, 83%; p = 0.45 and 0.36, respectively). Furthermore, there was not a significant difference between the second and third arms (p > 0.1). The quantitative results suggest that the CDS intervention that included the recommendation script impacted HPV vaccination rates.
Methodologically, qualitative interviews with providers who participated in the RCT are useful in understanding attitudes and beliefs with respect to the CDS interventions in light of the mixed quantitative results. From the interview data, we observe only one provider in the third arm recalled noticing the HPV script, whereas many providers in the second arm recalled seeing the CDS reminders. Based on these data, it may be that the providers in the third arm noticed the reminder just not the suggested script, but the script may have nonetheless acted to enhance the salience of the prompt.
Although the HPV vaccination rate in the third arm was higher than in the first, providers perceived that the intervention had no effect on their behavior. While possible that providers in the third arm are better vaccinators than the providers in the first arm, the chance of this statistically is very low given the quantitative analysis. Therefore the discordance between the two sets of results suggest that in some cases CDS reminders may provide subliminal, unconscious messages that actually do affect behavior, even if the behavior change is not perceived by the provider. This conclusion, and the observations in this study, bolster prior research in which the researchers turned off CDS reminders to find that preventive tasks such as vaccination went down in the absence of CDS [22, 40].
Workflow is important for provider adoption and satisfaction
Although CDS reminders can be effective, even when subliminal, informaticians must pay attention to clinical workflow. Respondents in this study commented on multiple aspects in which CHICA and the CDS reminders for HPV vaccination may not have fully supported the busy workflow or cognitive processes of pediatric providers. One issue is that the clinics with the HPV reminder intervention recently implemented EHR systems for clinical documentation, yet CHICA remained reliant on paper-based processes. When CHICA was introduced more than a decade ago, most of the clinical workflow relied on paper-based processes. However, with the recent transition to a greater number of EHR components implemented to comply with ‘meaningful use’ regulations [41, 42], provider workflows increasingly rely on computers for data capture, information retrieval and review of the patient’s chart. This created a cognitive challenge for providers who had to manage both paper and electronic information.
In addition, several providers indicated they often do not have time to review the CDS reminders during a patient encounter. These comments are emblematic of the complexity involved in EHR and CDS systems in a world where computers are integrated into workflow. A study by McDonald et al.  demonstrated that the introduction of EHR systems can increase the amount of time providers work each day due to the complexity of completing documentation in the EHR as opposed to on paper. Providers are also sensitive to the issue of ‘alert fatigue’ where CDS reminders can pose ‘one more thing’ they have to comply with in the age of EHR systems . However, it should be noted this phenomenon was only mentioned by a couple providers so did not appear to be as much of an issue in this study as the disconnect between the EHR system and the paper-based CDS reminders.
A final item related to workflow involves the role of nurses in both information retrieval and the vaccination process. Several providers commented that their awareness of a patient’s vaccination history relies principally on the nurses who support care delivery within the clinic, which may explain why some providers reported not attending to CDS prompts. Nurses are often tasked with retrieving information from the patient’s chart, including tasks that access an IIS to forecast vaccines for which a child is due. Nurses typically then administer the vaccine under direction from a physician following patient (or family) consent. Given a general shift in medicine towards team-based care where each provider works ‘to the top of their license’ , it may be advantageous for CDS systems to target nurses as the recipients of reminders rather than, or in addition to, physicians. Furthermore, prior studies on CDS reminders in the inpatient setting show that standing orders for influenza and pneumococcal vaccines, in which nurses have the clinical authority to not only check on a patient’s eligibility for these vaccines but also offer and administer them without specific direction from a physician, can increase vaccination rates significantly more than CDS reminders presented to physicians . Future studies on CDS reminders for vaccines, including HPV, in pediatric as well as family medicine clinics should explore CDS paradigms in which standing orders for non-physicians are compared with traditional clinical reminders. Additionally, researchers should consider CDS modalities that could provide reminders or prompts directly to patients (or guardians) prior to or during a clinic visit [47, 48].
The role of health information exchange
Although the emphasis of this study focused on provider interaction with the CDS reminders, this study demonstrates the potential for HIE-based CDS systems to improve population health . In the background, between the time the patient was registered into the clinic and when he or she was seen by a provider, the CHICA system integrated data retrieved from the patient’s EHR and the IIS at the state health department. Without HIE between CHICA and the IIS, nurses or other clinic staff would be required to directly access the IIS to retrieve information on the child’s vaccine history and forecast then manually reconcile this information with data in the EHR. This is how it works in most clinics within Indiana and likely the rest of the U.S. Interoperability and HIE  support electronic communication between disparate health information technology systems to support clinical workflow. Recent studies by others show that greater interoperability between EHR systems and IIS are possible as providers continue to adopt EHR systems for ‘meaningful use’ . Future studies should more directly evaluate HIE and its impact on vaccine rates as well as other population health outcomes given greater access to integrated information at the point of care.
No study is without limitations. First, the study was conducted within a single, publicly-funded urban health system. Providers’ comments may not represent what might happen if a similar system was implemented elsewhere. Additionally, participants represent a purposive sample, in which the investigators sought to balance inclusion from all arms of the RCT. Yet respondents may have agreed to participate because they held either strongly positive or strongly negative views with respect to HPV vaccination or the CHICA system. To mitigate the potential for bias, we included opposing viewpoints expressed by respondents in the results where documented.
While our trial of CDS reminders that targeted pediatric providers improved HPV vaccination rates, when asked about the intervention providers reported being aware of the prompts but did not believe they influenced their behavior. Providers believed they generally do a good job of vaccinating adolescents and that awareness of patient eligibility for a vaccine is influenced by other clinical team members’ situational awareness. These perceptions may influence some healthcare organizations to disable CDS reminders for vaccination even in the face of evidence that CDS can improve vaccination rates. Such an action is not recommended and would be counterproductive, even though the exact role of CDS reminders in changing provider behavior remains unclear.
Given this study, more research is necessary to further unpack the role of CDS in vaccination processes within clinics. One potential pathway may be to use CDS prompts that target other members of the clinical team, or there may be better ways to present reminders to providers in the context of clinical workflow. Pathways involving CDS direct to consumers should also be considered. Biomedical informatics researchers and innovators, including those who develop commercial systems, should focus time and effort on understanding clinical workflows and cognitive processes across health systems to design systems that can meet varied clinical practices. While helpful in advancing our understanding CDS, this study should be replicated and analyzed in other contexts to more fully account for the variety of practices and cognitive processes that underlie clinical decisions with respect to vaccination.
Clinical decision support
Child Health Improvement through Computer Automation system
Electronic health record system
Health information exchange
Immunization information system
Randomized controlled trial
Centers for Disease Control and Prevention. Genital HPV infection—CDC Fact Sheet. Atlanta: National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention; Jul 2017. Available at: https://www.cdc.gov/std/hpv/HPV-FS-July-2017.pdf.
National Institutes of Health. HPV and Cancer. Bethesda: National Cancer Institute; 2015.
U.S. Food and Drug Administration. FDA approves Gardasil 9 for prevention of certain cancers caused by five additional types of HPV. Silver Spring: Administration FaD; 2014.
Stokley S, Jeyarajah J, Yankey D, Cano M, Gee J, Roark J, Curtis R, Markowitz L, Immunization Services Division NCfIaRD, CDC. Human papillomavirus vaccination coverage among adolescents, 2007–2013, and postlicensure vaccine safety monitoring, 2006–2014 - United States. MMWR Morb Mortal Wkly Rep. 2014;63(29):620–4.
Reagan-Steiner S, Yankey D, Jeyarajah J, Elam-Evans LD, Curtis CR, MacNeil J, Markowitz LE, Singleton JA. National, regional, state, and selected local area vaccination coverage among adolescents aged 13–17 years - United States, 2015. MMWR Morb Mortal Wkly Rep. 2016;65(33):850–8.
Gilkey MB, Malo TL, Shah PD, Hall ME, Brewer NT. Quality of physician communication about human papillomavirus vaccine: findings from a national survey. Cancer Epidemiol Biomark Prev. 2015 Nov;24(11):1673-9.
Gilkey MB, Calo WA, Moss JL, Shah PD, Marciniak MW, Brewer NT. Provider communication and HPV vaccination: the impact of recommendation quality. Vaccine. 2016;34(9):1187–92.
Bendik M, Mayo RM, Parker VG. Contributing factors to HPV vaccine uptake in college-age women. J Cancer Educ. 2009;24:17.
Centers for Disease Control and Prevention. National and state vaccination coverage among adolescents aged 13–17 years--United States, 2011. MMWR Morb Mortal Wkly Rep. 2012;61(34):671–7.
Brewer NT, Gottlieb SL, Reiter PL, McRee AL, Liddon N, Markowitz L, Smith JS. Longitudinal predictors of human papillomavirus vaccine initiation among adolescent girls in a high-risk geographic area. Sex Transm Dis. 2011;38(3):197–204.
Liddon NC, Hood JE, Leichliter JS. Intent to receive HPV vaccine and reasons for not vaccinating among unvaccinated adolescent and young women: findings from the 2006–2008 National Survey of Family Growth. Vaccine. 2012;30(16):2676–82.
Zimet GD, Weiss TW, Rosenthal SL, Good MB, Vichnin MD. Reasons for non-vaccination against HPV and future vaccination intentions among 19–26 year-old women. BMC Womens Health. 2010;10(1):27.
Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742–52.
Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–38.
Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ (Clin Res Ed). 2005;330(7494):765.
Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux R, Samsa G, Hasselblad V, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess. 2012;203:1–784.
Roshanov PS, Misra S, Gerstein HC, Garg AX, Sebaldt RJ, Mackay JA, Weise-Kelly L, Navarro T, Wilczynski NL, Haynes RB. Computerized clinical decision support systems for chronic disease management: a decision-maker-researcher partnership systematic review. Implement Sci. 2011;6:92.
Sim I, Gorman P, Greenes RA, Haynes RB, Kaplan B, Lehmann H, Tang PC. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001;8(6):527–34.
Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med. 2010;363(6):501–4.
Stage 2 Eligible Professional (EP) Meaningful Use Core and Menu Measures. [http://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/Stage2_MeaningfulUseSpecSheet_TableContents_EPs.pdf]. Accessed 30 June 2017.
Rajamani S, Bieringer A, Muscoplat M. Characterizing the access of clinical decision support offered by immunization information system in Minnesota. Online J Public Health Informa. 2015;7(3):e227.
Biondich PG, Dixon BE, Duke J, Mamlin B, Grannis S, Takesue BY, Downs SM, Tierney WM. Regenstrief medical informatics: experiences with clinical decision support systems. In: Greenes RA, editor. Clinical decision support: the road to broad adoption. 2nd ed. Burlington: Elsevier, Inc.; 2014. p. 165–87.
McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comput. 1992;9(5):304–12.
Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001;345(13):965–70.
Zimet G, Dixon BE, Xiao S, Tu W, Kulkarni A, Dugan T, Sheley M, Downs SM: Physician reminders and recommendation scripts for HPV vaccination: A randomized clinical trial. Acad Pediatr. 2017. (In Press).
Kasting ML, Wilson S, Dixon BE, Downs SM, Kulkarni A, Zimet GD. A qualitative study of healthcare provider awareness and informational needs regarding the nine-valent HPV vaccine. Vaccine. 2016;34(11):1331–4.
Kasting ML, Wilson S, Dixon BE, Downs SM, Kulkarni A, Zimet GD. Healthcare providers’ beliefs and attitudes regarding risk compensation following HPV vaccination. Papillomavirus Res. 2016;2:116–21.
Anand V, Biondich PG, Liu G, Rosenman M, Downs SM. Child Health Improvement through Computer Automation: the CHICA system. Stud Health Technol Inform. 2004;107(Pt 1):187–91.
Biondich PG, Downs SM, Anand V, Carroll AE. Automating the recognition and prioritization of needed preventive services: early results from the CHICA system. AMIA Annu Symp Proc. 2005;2005:51–55.
About Immunization Information Systems. [https://www.cdc.gov/vaccines/programs/iis/about.html]. Accessed 30 June 2017.
CHIRP: Children and Hoosier Immunization Registry Program. [https://chirp.in.gov/]. Accessed 30 June 2017.
Dixon BE. What is Health Information Exchange? In: Dixon BE, editor. Health Information Exchange: Navigating and Managing a Network of Health Information Systems. Waltham: Academic Press; 2016. p. 3–20.
Zhu VJ, Grannis SJ, Rosenman MB, Downs SM. Implementing broad scale childhood immunization decision support as a web service. AMIA Ann Symp Proc. 2009;2009:745–9.
Guest G, Bunce A, Johnson L. How many interviews are enough? An experiment with data saturation and variability. Field Methods. 2006;18(1):59–82.
Patton MQ. Qualitative evaluation and research methods. 4th ed. Thousand Oaks: SAGE Publications, Inc; 2015.
Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107–15.
Babbie E. The Basics of Social Research. 5th ed. Belmont: Wadsworth; 2011.
Downe-Wamboldt B. Content analysis: method, applications, and issues. Health Care Women Int. 1992;13(3):313–21.
Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.
McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med. 1976;295(24):1351–5.
Edmunds M, Peddicord D, Bates DW. Clinical informatics policy and regulations. In: Finnell JT, Dixon BE, editors. Clinical informatics study guide: text and review. Zurich: Springer International Publishing; 2016. p. 47–66.
Xierali IM, Hsiao CJ, Puffer JC, Green LA, Rinaldo JC, Bazemore AW, Burke MT, Phillips RL Jr. The rise of electronic health record adoption among family physicians. Ann Fam Med. 2013;11(1):14–9.
McDonald CJ, Callaghan FM, Weissman A, Goodwin RM, Mundkur M, Kuhn T. Use of internist’s free time by ambulatory care Electronic Medical Record systems. JAMA Intern Med. 2014;174(11):1860–3.
Holden RJ, Voida S, Savoy A, Jones JF, Kulanthaivel A. Human factors engineering and human–computer interaction: supporting user performance and experience. In: Finnell JT, Dixon BE, editors. Clinical informatics study guide: text and review. Zurich: Springer International Publishing; 2016. p. 287–307.
Rosland AM, Nelson K, Sun H, Dolan ED, Maynard C, Bryson C, Stark R, Shear JM, Kerr E, Fihn SD, et al. The patient-centered medical home in the Veterans Health Administration. Am J Manag Care. 2013;19(7):e263–72.
Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. Inpatient computer-based standing orders vs physician reminders to increase influenza and pneumococcal vaccination rates: a randomized trial. JAMA. 2004;292(19):2366–71.
Nazi KM, Hogan TP, Woods SS, Simon SR, Ralston JD. Consumer health informatics: engaging and empowering patients and families. In: Finnell JT, Dixon BE, editors. Clinical informatics study guide: text and review. 1st ed. Zurich: Springer International Publishing; 2016. p. 459–500.
Col N, Correa-de-Araujo R. Consumers and clinical decision support. In: Greenes RA, editor. Clinical decision support (Second edition). Oxford: Academic Press; 2014. p. 741–69.
Haque SN, Dixon BE, Grannis SJ. Public health informatics. In: Finnell JT, Dixon BE, editors. Clinical informatics study guide: text and review. 1st ed. Zurich: Springer International Publishing; 2016. p. 501–20.
The authors wish to acknowledge Abby Church, MPH, of the Regenstrief Institute for her role as project manager for the research study described in the paper.
This study was supported by the Merck-Regenstrief Program in Personalized Health Care Research and Innovation (Project #14). Its contents are the sole responsibility of the authors and do not reflect the official view of Merck & Co., Inc.
This work was further supported, in part, by the Center for HPV Research, funded by the Indiana University–Purdue University Indianapolis Signature Centers Initiative in conjunction with the Indiana University School of Medicine Department of Pediatrics and the Indiana University Melvin and Bren Simon Cancer Center.
MLK is supported by the National Cancer Institute of the National Institutes of Health under Award Number R25CA117865. Its content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health, including the National Cancer Institute or the National Institute of Nursing Research.
Availability of data and materials
The data used in this study were generated from interviews with pediatric providers. The semi-structured interview guides used to collect data are available to other researchers who wish to conduct similar studies. De-identified transcripts from the interviews could be made available to other researchers through a data use agreement with Indiana University. To inquire about access to the interview guides or transcripts, contact the corresponding author.
Ethics approval and consent to participate
This study received ethics approval from the Institutional Review Board (IRB) at Indiana University (Study No. 1408987170). Respondents were provided a study information sheet and asked verbally to consent to be interviewed. The IRB approved a waiver of written consent because the study presented minimal risk to subjects who were practicing healthcare providers.
Consent for publication
The research reported in this article was funded by Merck & Co. through the Merck-Regenstrief Program in Personalized Health Care Research and Innovation, and Gregory Zimet has served as an investigator on research funded by Roche and received travel funds from Merck & Co., Inc. to present research at a scientific conference. Amit Kulkarni is an employee of Merck & Co., Inc.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
About this article
Cite this article
Dixon, B.E., Kasting, M.L., Wilson, S. et al. Health care providers’ perceptions of use and influence of clinical decision support reminders: qualitative study following a randomized trial to improve HPV vaccination rates. BMC Med Inform Decis Mak 17, 119 (2017). https://doi.org/10.1186/s12911-017-0521-6
- Papillomavirus vaccines
- Clinical decision support systems
- Public health informatics