This article has Open Peer Review reports available.
Refining a brief decision aid in stable CAD: cognitive interviews
- Karen Kelly-Blake1Email author,
- Stacie Clark†1,
- Katherine Dontje†2,
- Adesuwa Olomu†1,
- Rebecca C Henry†1,
- David R Rovner†1,
- Marilyn L Rothert†2 and
- Margaret Holmes-Rovner†1
© Kelly-Blake et al.; licensee BioMed Central Ltd. 2014
Received: 12 July 2013
Accepted: 10 February 2014
Published: 13 February 2014
We describe the results of cognitive interviews to refine the “Making Choices©” Decision Aid (DA) for shared decision-making (SDM) about stress testing in patients with stable coronary artery disease (CAD).
We conducted a systematic development process to design a DA consistent with International Patient Decision Aid Standards (IPDAS) focused on Alpha testing criteria. Cognitive interviews were conducted with ten stable CAD patients using the “think aloud” interview technique to assess the clarity, usefulness, and design of each page of the DA.
Participants identified three main messages: 1) patients have multiple options based on stress tests and they should be discussed with a physician, 2) take care of yourself, 3) the stress test is the gold standard for determining the severity of your heart disease. Revisions corrected the inaccurate assumption of item number three.
Cognitive interviews proved critical for engaging patients in the development process and highlighted the necessity of clear message development and use of design principles that make decision materials easy to read and easy to use. Cognitive interviews appear to contribute critical information from the patient perspective to the overall systematic development process for designing decision aids.
Shared Decision-Making (SDM) with patients, informed by decision aids (DAs), is increasingly recommended as a clinical practice reform to improve medical practice and patient involvement[1, 2]. Decision aids are increasingly available to communicate the pros and cons of treatment alternatives. High quality DAs assure balanced presentation of the outcomes that can be achieved by competing alternatives on a population basis, and the risks and benefits of each. Decision aids in the form of patient booklets, DVDs, and websites are largely designed to prepare patients to participate knowledgeably in discussions with their providers about specific clinical decisions. Stacey et al., in a review of randomized trials, report that DAs improve patient knowledge of options, decisional conflict and decision quality, with no detrimental effects on patient anxiety or satisfaction with care or provider. However, they also note that further work is needed to determine patient adherence to chosen options and to evaluate patient-physician communication while using DAs.
A growing consensus about methods to design high quality DAs suggests one key step in the process is the involvement of patients in review and development of the tools. We report here the results of engaging patients in improving a prototype DA after they had used it in their own care. The results of the pilot study using the prototype have been reported elsewhere. The aim of this study was to employ a cognitive interview technique to refine the DA before disseminating for use in community and academic family and internal medicine clinics.
DA for stable CAD patients
Percutaneous coronary intervention (PCI) is commonly used in the initial management of patients with stable coronary artery disease (CAD) in the United States. PCI is a treatment that inserts a small metal or drug-eluting mesh called a stent into the partially blocked area of a heart blood vessel to keep it open. Evidence is clear that PCI improves outcomes in acute myocardial infarction (MI) patients and function and quality of life in patients whose chest pain cannot be controlled by medications. However, numerous randomized trials over the last decade and meta-analyses in patients with stable CAD have shown there is no difference in the risk of death or myocardial infarction (MI) between patients receiving PCI and medical therapy or receiving medical therapy alone[6–8]. Patients, on the other hand, even after discussion with their cardiologists, do not understand this evidence[9, 10]. Whittle et al., in an interview study with 1650 patients and their treating physicians following coronary angiography, report that 83% of patients expected a survival benefit and symptom improvement, even though their cardiologists did not expect a survival benefit. The authors suggest that decision aids might be helpful during decision-making about coronary catheterization for stable CAD if practice-based barriers can be resolved[11, 12].
We conducted a systematic development process to design a DA consistent with International Patient Decision Aid Standards (IPDAS) focused on Alpha testing criteria.
Prototype development: alpha testing 1
A prototype DA was developed by literature review and consultation with nursing, cardiology, internal medicine, literacy, and communication graphics experts. The prototype was used as a patient educational intervention described elsewhere.
Revised DA created
The second was to include the data tables about efficacy of medical therapy with angiography and stent compared with medical therapy alone in the DA. The third was to hire a graphics design firm to make the booklet more attractive. The “Making Choices©” DA provides an evidence summary and decision pages to record choices arrived at in the clinical encounter. Throughout the DA, stress test was explained as the first step in making a decision about subsequent treatment that might include whether to treat with medical therapy alone compared with medical therapy and angiography with stent in stable CAD.
Cognitive interviewing: alpha testing 2
Cognitive interviewing is increasingly used in the development and evaluation of patient education materials. Commonly referred to as “think-aloud” interviewing, cognitive interviews allow researchers to understand how patients perceive and interpret information and to identify any potential problems in the material[14, 15]. The interviewee is encouraged to “think aloud” while reading through the material presented. The interviewer “probes” by asking what is confusing, or unclear, or asking for comments on specific elements in the material. The purpose of the cognitive interviews was to get feedback from stable CAD patients in order to make revisions to “Making Choices©” DA content, design, and language. The “think-aloud” approach was used by the interviewer to ask the participants what they were thinking while reading each page of the decision aid. Scripted probes were included in the interview protocol to provide content standardization across interviews.
A subset of the patients from the Alpha 1 test was recruited for this study. These patients, with stable CAD, had previous exposure to the DA prototype in nurse-led group visits to discuss shared decision-making with their physicians about managing their stable CAD. Thirty potential participants were contacted via phone based on their previous documented interest in participating in further research. Four declined to participate, thirteen were unavailable after three attempted phone calls, and ten patients agreed to participate in the interviews.
Development of the cognitive interview protocol
Cognitive interview questions
“Please look over the first couple of pages. Take as much time as you would like to look over the pages. As you are doing this, tell me out loud any thoughts that go through your mind as you read every 1–2 pages.”
What is new or different about the information presented here from what you already know? Is there any information presented here that you question?
What have you read here that other patients might find confusing?
After reading over each section in the booklet:
What is this section or part of the booklet telling you?
Did it hold your attention? Why or why not?
“What other thoughts came to mind while you were looking over the booklet that you haven’t shared?”
What do you think is missing from the booklet?
What concerns do you think other patients would have that we have not covered?
What concerns do you think other patients would have that we have covered?
What do you think the main message was?
The interview format asked if any of the information presented was confusing, if any information was missing, what concerns patients may have, and finally what they believed was the main message of the DA. Additional probes elicited information about graphics, design, and format.
Informed consent and IRB approval
Participants signed a consent form prior to beginning the interview. The consent form summarized the reasons for the research, protection of confidentiality and data protection procedures, and informed the participants that their participation was voluntary and that they could end the interview at any time. The institutional review board at Michigan State University approved the study protocol (IRB #09-389).
Interviews were conducted by a member of the research team with expertise in qualitative research (KKB). Interviews took place in a location convenient and easily accessible to the participants. The interviews were audio recorded. Each participant was given a copy of the DA. They were invited to read the pages and to “think aloud” about anything that occurred to them while reading. The questions and prompts structured these conversations. When the interviewee and interviewer had discussed all that was relevant on a page, they proceeded to the next page.
Analysis of data
KKB audio recorded the comments of each participant and took handwritten notes of the interview. The recordings were not transcribed. KKB and SC identified common themes and concerns based on recordings and handwritten interview notes, and any differences were resolved by consensus. Our aim was to achieve thematic saturation which was reached when the participants kept repeating the same recommendations and commenting on the same concerns for each page of the DA, and as a result the data were mostly unanimous. Specific recommendations about content, design, and format were collated into a comprehensive document by SC using the audio recordings and handwritten notes. Each of the recommended changes was discussed with the research team, agreement was reached by consensus, and editing ensued. Recommendations that were not enacted were primarily due to cost and time constraints. The data from the cognitive interviews were used to refine a new version of the DA.
Booklet was easy to understand
Medical terms could be confusing
More elaboration on prevention
Talking Points (Figure 2) form helpful and convenient
Easy read, but glossary was helpful
Acronyms (PCI, CATH) and use of “provider” was confusing
Difficulty reading text on colored pages
People looked too happy
How to determine which provider to see for management of heart disease (Figure 3)
Clarification that booklet is a supplement to and not a replacement for physician recommendation
Multiple options to be discussed with physician
Take care of yourself
Stress test is the gold standard**
Overall, the DA was found to be logically organized and presented in a way that would allow patients to have a meaningful and appropriate conversation with their physician about the pros and cons of different treatment options. Participants found the booklet easy to understand based on their personal experience with stress tests and heart attacks. They noted that some of the medical terms could be confusing for newly diagnosed heart disease patients (a glossary page of medical terms was in the DA), but overall, the booklet would be helpful.
Largely, participants felt the DA was an easy read. However, they tended to return to the glossary page multiple times, which they recognized as a helpful resource in the booklet. Acronyms (i.e. PCI, CATH) were confusing and use of the term “provider” was problematic because they felt it could be a reference to an insurance provider. This was easily corrected by using the term “healthcare provider.” It was stressed that because the DA would likely be read during an emotionally charged time, i.e. receiving stress test results, confusing language should be avoided.
Participants did not take issue with the graphics or design, but had difficulty reading the text on colored pages. They also had difficulty distinguishing the information presented in the graphs because of the colored background. Changes were made to the colors used in the booklet based on this feedback. Several participants commented that the pictures used in the booklet showed people who looked happier than most patients would be in the given situation.
How to determine which provider to see regarding their heart disease (in the follow-up form, see Figure 3, they have a checkbox for their PCP, cardiologist, or both).
Clear statements that the booklet is not a replacement for speaking with a physician, but supplements a physician’s recommendation.
Information on what a spouse or family member can expect when a loved one is making treatment decisions about heart disease.
Clarification about the different kind of stress tests a heart disease patient may have.
Three main messages came to light for participants: 1) Patients have multiple options and strategic choices to make based on the results of a stress test, and these choices should be discussed with a physician; 2) take care of yourself; and 3) a stress test is the gold standard for determining the severity of your disease. Item 3 was definitely not the message we sought to convey in the DA because it was not consistent with the evidence. Revisions made based on the results included the following:
Confirmed main message, but revised to correct inaccurate assumption about the stress test as gold standard.
Added a checklist for recording decisions such as scheduling appointments, test results, and future tests presented in a format that could go to the patient, to the clinician, and to the electronic health record. The rationale for this revision, in part, was that even a brief DA booklet is too long for the clinical encounter.
In terms of the feedback received about missing information (see last section), text was added for all but one of the suggestions. No action was taken on suggestion four because the focus was on the exercise treadmill stress test only, and we wanted to avoid confusing the different types such as nuclear, ECHO, and chemical stress testing.
All revisions were reviewed and agreed upon by the research team. The local graphics design firm finalized the design of the “Making Choices©” DA. The DA was copyrighted and printed for distribution.
Our Beta 1 and 2 testing (field testing) is currently underway. We have distributed the brief DA to four primary care practices and one cardiology practice in Michigan. Through this development process, we have focused on a pure primary care strategy. However, next steps in implementation will facilitate collaboration with cardiology in order to improve referral and handoffs between primary care and cardiology.
The study reported here used cognitive interviewing to help guide the development and refinement of the “Making Choices©” decision aid. The DA operates as an evidence summary and decision pages to record choices arrived at in the clinical encounter. The data from the interviews afforded the researchers the opportunity to include new content or modify and expand existing content. Cognitive interviews are increasingly used in the development and evaluation of patient education materials and different protocols are used[13, 19–21]. Commonly referred to as “think-aloud” interviewing, cognitive interviews allow researchers to understand how information is perceived and interpreted and to identify any potential problems in the material[14, 15]. We used this technique to refine the information contained in the “Making Choices©” DA before disseminating to patients and providers for use in routine clinical visits.
Our focus, reported here, was on the Alpha testing element of the IPDAS development process, which is still a work in progress. Cognitive interviews offer an opportunity as a first stage assessment within the Alpha testing process to meet IPDAS criteria. Coulter et al. report that only about 50% of patient decision aids have been field tested with patients and even fewer have been reviewed or tested by clinicians. The prototype was developed in consultation with various medical professionals. The revised DA was then subjected to critique and feedback with patients previously exposed to the prototype. This iterative Alpha testing process, to have patients critique and provide feedback on the comprehensibility and acceptability of the DA, assures the validity and reliability of the development process and the final product. The cognitive interview approach lends itself well to achieving these goals. As we move forward with Beta testing, it will be important to evaluate how the approach might be useful in the field environment.
The cognitive interviews provided valuable input for the revision of the DA, but there are limitations to the approach. First, the interviewees were not a representative sample of all possible stable CAD patients. We interviewed a subset of patients from the larger study that agreed to be contacted for future research. These participants had previously shown a willingness to participate in research and so may be more inclined to participate in such activity. Although we cannot eliminate this limitation, one way to mitigate it is to recruit participants from diverse economic, educational, and cultural backgrounds. Additionally, our sample of participants were all college-educated and above so participants with low literacy and numeracy would be invaluable in helping assure the accessibility and understanding of the information contained in the DA. A further limitation is that the participants in this study had been exposed to a previous version of the DA to facilitate decision making with their physicians, but they were not, at the time of this study facing the decision again. Second, it was relatively easy to make some changes to the DA, especially those that focused on design, appearance, and even some of the content. Some of the feedback highlighted problem areas that were not so easy to address such as including graphs depicting data that extended beyond a five-year survival rate. The graphs included in the DA were part of a package of materials and the data included in the DA were the data available to the research team at a particular point in time. Third, the participants may have not been as critical as they would have been with an unknown interviewer. The participants knew the interviewer (KKB) from the larger study. They may have felt some affinity and did not want to seem overly critical or judgmental of the DA. Along that same vein, people agreeing to provide feedback on the revised DA may have done so because they had more favourable experiences in the original study and were therefore more likely to provide positive feedback towards this DA. Lastly, the results provided too little to fill the knowledge gaps about patient adherence to decision choice or patient-physician communication while using DAs.
In contrast, the strength of the approach was that the same themes and comments appeared in each interview, and by the tenth interview we achieved thematic saturation and there were no new issues raised.
Further evaluation is needed to determine the usefulness of cognitive interviewing as an effective approach for assuring content and message relevance in decision aids to be used by both patients and providers in routine clinical visits.
The cognitive interview approach was instrumental in refining the “Making Choices©” DA. The technique proved to be critical for engaging patients as experts because their perspective cannot be provided by DA developers or clinicians. To our knowledge, this is the first study to report that stable CAD patients have been involved in cognitive interviews to refine a decision aid that addresses stress testing. Participants provided useful feedback on the design, language, and message construction. The feedback received facilitated the dissemination of the new version into community and academic practice settings to be used by both patients and providers in real world clinic settings.
Stable CAD and its concomitant treatment trajectory is a challenging clinical problem for shared decision-making and DA developers invested in producing balanced information. Cognitive interviews appear to contribute critical information from the patient perspective to the overall systematic development process for designing decision aids.
The authors wish to thank Dr. Ibrahim Shah for clarifying cardiology content in the decision aid; Sue Stableford and Darcy Greene, for their assistance in clarifying plain language content and design of the decision aid; and Project Traction for the design and printing of the decision aid. We thank the patients for their time in support of this research. The study was supported in part by a grant from the Michigan State University Clinical and Translational Sciences Institute (CTSI), East Lansing, Michigan and by a grant from Blue Cross and Blue Shield of Michigan.
- Elwyn G, Edwards A, Kinnersley P: Shared decision-making in primary care: the neglected second half of the consultation. Br J Gen Pract. 1999, 49: 477-482.PubMedPubMed CentralGoogle Scholar
- Salzburg Global S: Salzburg statement on shared decision making. BMJ. 2011, 342: d1745-View ArticleGoogle Scholar
- Stacey D, Bennett CL, Barry MJ, Col NF, Eden KB, Holmes-Rovner M, Llewellyn-Thomas H, Lyddiatt A, Légaré F, Thomson R: Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. 2011, Issue 10 Art. No.: CD001431-doi:10.1002/14651858.CD001431.pub3Google Scholar
- Coulter A, Kryworuchko J, Mullen P, CJ N, Stilwell D, Van der Weijden T: Using a systematic development process. 2012 Update of the International Patient Decision Aids Standards (IPDAS) Collaboration’s Background Document. Chapter A. Edited by: Volk R, Llewellyn-Thomas H. 2012,http://ipdas.ohri.ca/IPDAS-Chapter-A.pdf,Google Scholar
- Holmes-Rovner M, Kelly-Blake K, Dwamena F, Dontje K, Henry RC, Olomu A, Rovner DR, Rothert ML: Shared decision making guidance reminders in practice (SDM-GRIP). Patient Educ Couns. 2011, 85 (2): 219-224.View ArticlePubMedGoogle Scholar
- Boden WE, O’Rourke RA, Teo KK, Hartigan PM, Maron DJ, Kostuk WJ, Knudtson M, Dada M, Casperson P, Harris CL, et al: Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med. 2007, 356 (15): 1503-1516.View ArticlePubMedGoogle Scholar
- Boden WE, O’Rourke RA, Teo KK, Maron DJ, Hartigan PM, Sedlis SP, Dada M, Labedi M, Spertus JA, Kostuk WJ, et al: Impact of optimal medical therapy with or without percutaneous coronary intervention on long-term cardiovascular end points in patients with stable coronary artery disease (from the COURAGE Trial). Am J Cardiol. 2009, 104 (1): 1-4.View ArticlePubMedGoogle Scholar
- Weintraub WS, Spertus JA, Kolm P, Maron DJ, Zhang ZF, Jurkovitz C, Zhang W, Hartigan PM, Lewis C, Veledar E, et al: Effect of PCI on quality of life in patients with stable coronary disease. N Engl J Med. 2008, 359 (7): 677-687.View ArticlePubMedGoogle Scholar
- Coylewright M, Shepel K, LeBlanc A, Pencille L, Hess E, Shah N, Montori VM, Ting HH: Shared decision making in patients with stable coronary artery disease: PCI choice. PLoS ONE. 2012, 7 (11): e49827-doi:10.1371/journal.pone.0049827View ArticlePubMedPubMed CentralGoogle Scholar
- Whittle J, Conigliaro J, Good CB, Kelley ME, Skanderson M: Understanding of the benefits of coronary revascularization procedures among patients who are offered such procedures. Am Heart J. 2007, 154: 662-668.View ArticlePubMedGoogle Scholar
- Liao L, Jollis JG, DeLong ER, Peterson ED, Morris KG, Mark DB: Impact of an interactive video on decision making of patients with ischemic heart disease. J Gen Intern Med. 1996, 11 (6): 373-376.View ArticlePubMedGoogle Scholar
- Bernstein SJ, Skarupski KA, Grayson CE, Starling MR, Bates ER, Eagle KA: A randomized controlled trial of information-giving to patients referred for coronary angiography: effects on outcomes of care. Health Expect. 1998, 1 (1): 50-61.View ArticlePubMedGoogle Scholar
- Ahmed N, Bestall J, Payne S, Noble B, Ahmedzai S: The use of cognitive interviewing methodology in the design and testing of a screening tool for supportive and palliative care needs. Support Care Cancer. 2009, 17 (6): 665-673.View ArticlePubMedGoogle Scholar
- Czaja R: Questionnaire pretesting comes of age. Marketing Bulletin. 1998, 9 (Article 5): 52-66.Google Scholar
- Drennan J: Cognitive interviewing: verbal data in the design and pretesting of questionnaires. J Adv Nurs. 2003, 42 (1): 57-63.View ArticlePubMedGoogle Scholar
- Harris-Kojetin L, Fowler F, Brown J, Schnaier J, Sweeney S: The Use of cognitive testing to develop and evaluate CAHPS(TM) 1.0 core survey items. Med Care. 1999, 37 (Suppl 3): MS10-MS21.PubMedGoogle Scholar
- Dontje K, Kelly-Blake K, Olomu A, Rothert M, Dwamena F, Henry R, Rovner D, Holmes-Rovner M: Nurse-led group visits support shared decision-making in stable coronary artery disease. J Cardiovasc Nurs. 2013, 28 (3): 269-276.View ArticlePubMedGoogle Scholar
- Making Choices© Heart Stress Test Decision Guide: [http://www.med-decs.org/images/pdf/MSUStressTestDAFinal07262011.pdf]
- Levine RE, Fowler FJ, Brown JA: Role of cognitive testing in the development of the CAHPS® hospital survey. Health Serv Res. 2005, 40 (6p2): 2037-2056. doi:10.1111/j.1475-6773.2005.00472.xView ArticlePubMedPubMed CentralGoogle Scholar
- Sirken M, Schecheter S: “Interdisciplinary survey methods research”. Cognition and Survey Research. Edited by: Sirken MG, Herrmann DJ, Schechter S, Schwarz N, Tanur JM, Tourangeau R. 1999, New York: WileyGoogle Scholar
- Willis GB: Cognitive Interviewing: A Tool for Improving Questionnaire Design. 2005, Thousand Oaks, CA: SageView ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/14/10/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.