Table 3 User feedback themes from the group design sessions and changes made to address the feedback

Recommendations, clinical content, and workflow did not always match what was expected by the provider

“Maybe, rather than deferring it for four weeks, maybe defer it at this visit”

“Losartan usually doesn’t cause cough”

“Defer 4 weeks” to “Defer until next BP check” acknowledge reason

Removed “Losartan caused cough” acknowledge reason

Providers found some elements of the design extraneous

“I, particularly, like that you can just click order, e-consult, or a referral with a click, and you don’t have to do anything else”

“I don’t like his photo there. I think the whole thing is dizzying. It’s harder, for some reason, just by visual. I think the other one was easier to process and read”

Remove the order set option

Remove the photo of the department head

Some providers expressed preference for the most efficient method of responding to the alert message

“I think I prefer the previous [single order version]. Seems like it’s less clicks or it takes you—I’m not sure. Once you assign this smart set, you click on a dose of lisinopril and hit the smart set, sign it. It’s gonna go into the orders section, I suppose.”

Removed the Order Set version of BPA 2A

Providers appreciated the visibility and access to important information on the BPA

“I like that the order buttons are already directly there ready to go.”

“I really like the fact that you do put the female, child-bearing age warning. I think that that’s super important to remind people.”

Kept pre-selected order recommendations

Kept female of childbearing age warning

Providers prefer decision support that adds clinical value and useful information at the point of care

“I think this [Minimal Information version of BPA 1A] could fall into the category of the mini warnings and alerts that I end up ignoring because it just doesn’t have useful information in it. “

Removed the minimal information version of BPA 1A

Providers preferred transparency regarding why the alert was firing and how to interact with the BPA

“I guess I could see being in the dark, and being like huh, really, why did this [BPA 1A with a link to labs and BP] fire?”

Decided against the version of BPA 1A with a link to labs and BP in favor of the version with the information included under “Why did this alert fire?”

Providers wanted information to help prevent medication errors

“I guess I would like it to take into account, if possible, if it was like, ‘Patient is not on an ACE,’ and then it would say, ‘but they had something that caused them a rash when they took irbesartan in the past’”

Added a statement regarding cross-reactivity of angioedema between ACE and ARB

Providers wanted consistency between the BPA categories

“It’s not asking for referral to renal. That, to me, seems—it’s not consistent with the previous set of BPAs”

Added both buttons to all BPAs: “Ambulatory BWH Renal E-Consult” and “Ambulatory referral to BWH Renal”