Table 3 Weaknesses of e-learning component of the program

Lack of clarity

Training modality: more instructions needed on what is a decision box and how to access them, exercise is unclear

“At first I wondered what the decision box was. But in reality they’re tools. It could be a good idea to mention what it is, because, in the beginning, I didn’t know what it was. I thought it was something optional, but really it’s not.” (Physician #5, Round #1)

Unclear content: important information should be highlighted or bolded, more precisions required, missing information

[Note from the author: The participant is discussing the critical appraisal exercise] “I don’t know if people will understand it. They aren’t necessarily simple terms. It’s not the kind of thing you see every day. Especially not for clinicians who don’t do much research. For those who’re in journal clubs, it’s usually about interventions and relative risks… that sort of thing.” (Physician #14, Round #3)

Usability

Browsing issues between pages or sections: not intuitive enough, need to clarify how to move to the next section after completing an exercise, access to optional content is unclear

“I didn’t know where to click. I’m not very tech-savvy. I wasn’t sure what you meant by ‘Target Clientele.’ But now I understand that it’s part of the introduction. Maybe you could say ‘Introduction’ instead of ‘Target Clientele?’” (Physician #11, Round #3)

Issues with clickable elements: references within narrated slide shows should be clickable, element dispositioned, some slides should be reordered, confusions between buttons to move to the next page

“Click here to begin reading […]. The arrow is in the wrong place.” (Physician #7, Round #2)

Elements too small: images, narrated slide show

“It would be good to make the button more visible. I’m imagining someone who’s not very comfortable with computers… you know, sometimes, depending on your screen, it can be harder to see.” (Physician #5, Round #1)

Irrelevant or of less value

Content unrealistic: statin example irrelevant for people at low risk of cardiovascular diseases, the film displays a simulation, not reality

“In reality, I’m not sure people would be inclined to talk about taking statins with a patient who’s low risk. I understand; I’ve seen the studies showing there may be some benefits, but at the same time, with figures like that, it’s not necessarily a case where I would use shared decision-making.” (Physician #14, Round #3)

Irrelevant for specific audiences: clinical information not applicable to all professions, information more relevant for older HCPs, information les useful to experienced/less experienced physicians

“This part is a bit more about the medical side, and that one could be more for social workers or others. Sure, there are things that are a bit less relevant for me that I can still explore further, but you know, when I see certain things, I’m more likely to just refer them… “ (Social worker #9, Round #2)

Inappropriate learning strategies

Vignette with avatars displaying patient counseling strategies: not realistic enough, robot-like, childish, information is too theoretical, optional content

“When you’re a clinician and you watch that, you see interview techniques with the patient. The problem with the video is that the avatars don’t have any intonation. When we’re taught how to communicate with our patients, we’re taught what intonations to use. But in this case, there is no intonation. They always speak in exactly the same tone of voice.” (Physician #5, Round #1)

Critical appraisal exercise (Cochrane review on the impacts of patient decision aids): information layout should be improved, boring, reading is less appreciated as learning strategy

“I hate reading studies, especially studies in English! I get confused with the words […]. Even in French, I don’t like it, but in English it’s even worse! I’m just not interested. Especially with the way it’s presented here. There are too many figures and, in any case, it really doesn’t interest me.” (Physician #2, Round #1)

Film: unrealistic. communication of statistic still confusing

“The fact remains that it’s a simulated interview. You know, it’s not reality. It looks nice, but in real life, it’s not often like that.” (Physician #6, Round #2)

Too long: the whole program is too long, some sections are too long, the film is too long, some exercises are too long

“I think it’s a bit long in places. The content on decision aid tools was perhaps a bit long. There were several slides presenting the tools, and I figure people will be able to understand it easily enough. After all, it’s aimed at HCPs.” (Physician #7, Round #2)

Dry: relevant but dry (critical appraisal exercise, and content on evidence appraisal)

“When you get to this part, it shifts from visual mode, which is easier to follow, to a more purely informational mode, with words in English and all that, and lots of text. […]The information is clear enough, but it’s perhaps a little harder to find it. It’s not as simple as with the videos.” (Physician #5, Round #1)

Technical issues: video does not work, difficulties moving to the next slides, feedback does not show, loosing the Internet connection

“The video feedback doesn’t show up, only text.” (Social worker #15, Round #3)

Boring: first part is boring, reading is boring, narrated slide shows are boring, lack of animation

“I’m a very visual person. When there are long stretches of narration only, I just switch off.” (Physician #13, Round #3)

Terminology issues: acronyms should be defined, some terms that are too strong, unclear terminology, what do “option” and “numeracy” mean?, avoid the term “choosing the option of doing nothing”, some terms are difficult to understand

“A list of available health options… I can’t picture exactly what that means.” (Physician #3, Round #1)

Risk communication seems challenging

“I don’t always recall all the probability percentages. In a single day. I might see fifteen patients, all with different problems. I can’t always remember off the top of my head what the probabilities are.” (Physician #2, Round #1)

Optional content

Overall description of the program, program design team,

“You know, we’re so pushed for time that we really want to get straight to the point which, in this case, is the training. I get it that it’s really well done and all that, but I would still try to condense it a bit more.” (Physician #5, Round #1)

GRADE quality of evidence assessment: outside the program’s scope, information that is already known, covered in other programs

“I found it less relevant for shared decision-making, which made it less interesting.. […]. If I wanted to watch a webinar on shared decision-making, I wouldn’t want to watch that.” (Physician #8, Round #2)

Incomplete, missing information: missing sources, specific details missing, missing feedback on some exercises, abbreviations or acronyms are not defined

“I don’t know if it’s feasible, but it would have been fun to have a DB in hand to follow along at the same time as the training session.” (Physician #8, Round #2)

Redundant content: could be shortened, slightly too repetitive

“At one point they give a lot of examples. Maybe there are some things that could be removed.” (Physician #7, Round #2)

Informational or typographic error

“’Are you sure about which choice is best for you?’ Knowing some of my elderly patients, I think this question could destabilize them or make them feel less confident in their decision. I wouldn’t have worded it that way. I would have validated that that is the decision they want to make, but I find it a bit strong to use words like ‘certain’ or ‘best.’ Personally, I would have taken a gentler approach in that sense.” (Physician #11, Round #3)

Patient Preferences section too short: more communication tips should be offered

“Not everyone in the medical field has good patient communication skills, so I think you could expand this section a bit.” (Social worker #15, Round #3)