Table 3 Summary of qualitative comments on prototype for each testing round with key corresponding modifications

Format

Navigation

Round 1: Lacked clear directives and instructions on the clickable elements (images, buttons) on each page.

“You must give instructions because just now that one the instructions have, like this one, I wouldn’t know.”—ID01

Added of an example interactive button and standardized navigation buttons across pages.

Round 2: Received positive comments on the intuitiveness of the navigation. However, pop-ups and numerals added to each section was confusing. The example of an interactive button added after round 1 created some confusion.

“I think without the guidance [from interviewer], I won’t know to go to shared care [section by clicking the button].”—ID06

“I see there’s a cross (navigation of popup). Because when you read this way, I suppose I don’t go back there again.”—HCP06

1. Added an orientation section with animation and prompts to orientate users to navigation buttons.

2. For pop-ups, a clear ‘return’ button was added to guide navigation.

3. Changed navigation flow to linear instead of creating possible bidirectionality when users could choose to click into the 'shared care' option before the 'usual care' option.

Round 3: Received affirmative comments on navigation after introducing an orientation guide. Some participants missed clickable buttons on the pages due to colour overuse.

“I think it’s quite clear because you’ve got the click buttons and you got the arrows and everything, it’s so… I think that’s quite clear.”—ID13

“Are the orange words actually clickable?”—HCP07

1. For all clickable portions, a cursor image was added to prompt clicks and this image was included in the orientation navigation guide.

2. Bold/ italics were used for emphasis instead of coloured text.

Design

Round 1: Inconsistencies in colours between buttons and pop-up boxes caused confusion. HCPs raised concerns over the font choice and visual clutter.

“So, you see, the soft tissue cancer here, the colour suddenly becomes the lung cancer colour! So, for me, it’s confusing.”—ID03

1. Standardized the colours of all pop-up boxes with original buttons.

2. Ensured adequate empty space on each page to avoid visual clutter.

Round 2: BCS preferred the use of images and were generally acceptable of the font and size. HCPs provided favourable comments on the aesthetics but suggested more suitable icons in some instances.

“Quite interesting the oncologist is dressed out like the surgeon; it is actually all of us all combined right?”—HCP05

“I think this is a very nice thing and is very pleasantly put together. I think all your graphics.”—HCP06

Amended the graphics used to represent physical effects (replace a flushed woman image) and oncologists to be not suggestive of surgeons exclusively.

Round 3: Received overall positive comments, with suggestions for header font choice and animations.

“Because if you are not a text person, if you see pictures and moving things on the page, it’s easier for you to put your attention to the page rather than just pages.”—ID13

1. Added animations where appropriate for each page.

2. Standardized all fonts to be sans serif and improved colour contrast of headers.

Content

Amount and type of information

Round 1: Requested information.

• BCS: value and roles of HCPs in shared care, usual care description, support group resources.

• HCPs: quantified risk of rare and serious side effects, HCPs involved in shared care, other common treatment side effects.

“You need to paint the current [model]… because… we are trying to get people to consider this new model or embrace this new model. Then you need to paint the current.”—ID03

“Some people may ask what’s the risk [of ovarian cancer] … Small like how small? 1 in 10,000 or 1 in 100,000?”—HCP03

1. Side effects of treatment were reviewed against guidelines and literature, with further quantification of the risks of rare and serious side effects.

2. Separated the discussion of options into the respective dedicated sections to describe usual care and explain differences.

3. Added support group links under additional resources.

Round 2: Requested information.

• BCS: relapse management, frequency of treatment side effects, care provision by pharmacists under shared care.

• HCPs: frequency of treatment side effects, care provision by pharmacists under shared care, cost.

“My suggestion is maybe you want to put some… like this you put statistics but how about this and this?”—ID06

“And why we want the aftercare because we do not want the relapse. But there is nothing mentioned about the relapse. So read already it is as good as not reading.”—ID07

“So, the navigator is free of charge is it when they call them? You mean have to pay for the navigators?”—HCP05

1. Management of relapse was described through surveillance with a need to counterbalance against excessive generalization of the risk of relapse since it depends on a variety of patient factors. The focus of the DA was not to provide a personalized risk score.

2. Provided statistics of side effects, focusing on rare and serious ones.

3. Supplemented information on pharmacists’ role in shared care.

Round 3: Received affirmative comments on the content coverage, with suggested supplement on information related to psychosocial help sources. One survivor requested further for information on Traditional Chinese Medicine usage.

“Counsellor, is it good maybe to show a website, a web link, which organization can be the counsellor, or this information is at the back?”—ID14

“I think this is good enough because for a start. Unless they really need more then they have to read up more.”—ID11

1. Added a new page providing information related to psychosocial services.

2. No Traditional Chinese Medicine-related information were included due to lack of reliable official sources endorsed by the cancer centre.

Clarity and presentation

Round 1:

1. Significant medical jargon used.

2. The concept ‘late and long-term side effects’ was difficult to comprehend. A timeline with specified milestones was suggested to present this concept.

3. Confusion over the naming convention of HCPs.

4. Suggested ordering side effects by frequency.

“Actually, you can have… pop-up to explain what is the long-term effects as opposed to this, and not just have this inside, separate this… We don’t really understand.”—ID04

“For side effects, you can list the order, like the more common ones or more important ones will be on top.”—HCP03

1. Language used was reviewed and simplified to avoid medical jargons.

2. Replaced words with an animated infographic to explain the concept of late and long-term side effects.

3. Clarified on the types of polyclinic doctors (general vs. regular family physician) and contextualized community pharmacists to their respective retail stores.

4. Ordered treatment side effects by frequency and separated side effects that are rare but serious.

Round 2:

1. Medical jargon usage for treatment side effects.

2. The animated infographic explaining late and long-term side effects was too fast..

3. Confusion over the naming convention of HCPs in care options persisted and pharmacists’ role under shared care was unclear.

4. Depiction of the shared care model was suboptimal as it appeared as linear instead of a team-based effort.

“It sounds like they are only restricted to only fever and runny nose… and I think the community pharmacists may protest.”—ID06

“Are you sure you see the same doctor at polyclinic?”—ID10

1. Textual changes made to minimize jargons usage.

2. Separated and slowed animations for the late and long-term side effects, respectively, with their descriptions in words.

3. Created a section to elaborate on each type of HCP before presenting the two care options to improve clarity of roles.

4. To depict shared care, HCPs involved were presented in cyclic and triangular format to illustrate the communication flow, instead of a linear flow. The survivorship care plan was featured as a communication tool to facilitate the information flow.

Round 3: Received affirmative comments with suggested fine-tuning:

1. Emphasize actual cost differences between options.

2. Use a human body representation to contextualize treatment side effects for identification.

3. Contrast differences between care options further.

“The language is simple and clear.”—ID13

“I’m thinking whether it will be better if this thing is designed in a way that maybe like under physical effects, you have a[n] image of a person, of a human, like a human figure.”—HCP08

“The information is ok, just that maybe the points [differences] that you want to bring to the patient is not that clear.”—ID12

1. Emphasized cost difference through contrasting cost savings and cost incurred under each care option.

2. Added a human body figure to represent side effects within each treatment category.

3. Emphasized the differences between usual care and shared care options by isolating out comparison factors and presenting the corresponding information side-by-side.

Accuracy

Round 2: Under usual care, HCPs feedbacked that the list of care providers involved are not comprehensive and there was overgeneralization of poor communication.

“If it is just usual care, we also communicate with each other. Not that distinct. We do right but I think the communication may not be as much as shared care. We don’t communicate, but if there is a problem then we communicate.”—HCP05

Amended the description of HCPs involved in usual care and qualified description of communication to avoid overgeneralization.

Information balance

Round 1: Considerably more information was provided on shared care than usual care.

“I think this one [DA] will get you to accept the polyclinic one [shared care option].”—ID02

A dedicated section was created to describe what happens under usual care option.

Round 2: A consensus that the information presented was slanted towards shared care.

“I found the presentation of information to be slanted towards shared [care].”—ID08

Adopting literature recommendations, sections introducing the two care options were rearranged to side-by-side presentation as much as possible.

Round 3: A consensus that the information presented was balanced.

“I think it’s balanced. The information, you can decide what’s… one way or the other.”—ID13

No corresponding changes.

Preference clarification exercise

Consideration factors

Round 1: Affirmed the relevance of consideration factors, including HCP characteristics (regularity, expertise), oncologist involvement, cost, location/ convenience, ease of appointment arrangement.

“Because you’re seeing so many different people! At least [at cancer centre], you’re seeing the same doctor, the doctor knows your history, you don’t have to keep repeating to every single person.”—ID05

“They[survivors] say they are worried they are getting an inferior surveillance because they’ve gone from a specialist to a primary care physician.”—HCP01

The list of questions was revised according to reported attributes to increase relevance to survivors. To address concerns of perceived inferiority of primary care HCPs, the statement has been qualified to be ‘trained by oncologists from the cancer centre’.

Round 2: Echoed the relevance of factors explored in questions. HCPs suggested mechanisms to allow prioritizing of factors against one another.

“I think cost is important, convenience is important, but I would be willing to pay twelve dollars more, right?”—HCP06

To weigh in the priority of factors, some preference statements were crafted with qualifiers (e.g., I am willing to explore shared care if it cost cheaper than usual care).

Round 3: Echoed the relevance of factors explored in questions. One survivor disregarded all preference statements on shared care due to preconceived disposition while another HCP suggested including pharmacist in preference statements for usual care to maintain balance.

“Yeah, so that’s why I didn't look through the other side, the shared care.”—ID14

“Usual care, can we talk about like something on the pharmacist as well?”—HCP08

To remove undue influence of preconceived dispositions affecting the binary responses to each factor, the exercise was transposed to a questionnaire format, hosted on an external website.

Comprehensibility

Round 1: Significant confusion over questions on patient navigation and favourability of cost-savings.

“Patient navigation… don’t understand.”—ID05

“Again, they may get confused. ‘What is patient navigation again?’ You may have to qualify.”—HCP01

Simplified language for cost-savings and explained the 'care navigation' aspect in more details under newly created section of 'Shared care' for survivors to make associations.

Round 2: Significant confusion over the sample question created after round 1, specifically on its relevance to subsequent questions. Also, BCS reported difficulty in responding to some questions as they lacked concrete experience of shared care.

“So, what’s this supposed to show me? Maybe you should put, prefer usual care, and prefer shared care. Cause now its food in the cafeteria so it’s a bit confusing.”—HCP04

The example question was removed. Acknowledging the hypothetical scenario of shared care, we modified the outcomes of the exercise to ‘I prefer continuing with usual care’ and ‘I am willing to try out shared care’. We transformed this section into a tabular presentation of preference statements where users give a binary response to each attribute, reflecting their slant to either option based on responses.

Round 3: no major concerns raised by survivors, but one HCP was concerned if survivors could interpret the results (i.e., preference indicated by the higher number of statements checked pro-usual vs. pro-shared care).

“I’m not sure if it’s [results interpretation] entirely clear though. Okay. [Be]cause … that may not come across immediately.”—HCP07

A multicriteria decision analysis method was eventually adopted using the external website. For each consideration factor, users could indicate their care preference and whether the factor is important to them. Responses were then scored as ‘pro-usual’ vs. ‘neutral’ vs. ‘pro-shared’. At the end, the site will return a final score (0–100%), with interpretation explained in the DA.

Format

Round 1: Use of sliders on a continuous scale confused participants who suggested for markers as potentially useful guides. There was a general consensus that there should be a 'results' page to help users interpret their eventual preference based on the responses to the listed questions.

“I think its ok to have the ruler to allow people to put on exactly whatever percentage, but you still have the in-between markers to guide them as a frame of mind.”—ID03

“So, unless you want to have a summary page, you want to remind them what they have chosen…”—HCP02

1. Questions on importance of attributes were transformed into Likert scales.

2. Besides asking users to rate the importance of the attributes, responses to each question were mapped to a choice disposition (pro-usual or .pro-shared care) to improve results interpretation. An example question was included.

3. Added a final ‘leaning scale’ for users to indicate their slant to either option after considering the factors.