Developing a web-based information resource for palliative care: an action-research inspired approach
© Street et al; licensee BioMed Central Ltd. 2007
Received: 07 December 2006
Accepted: 14 September 2007
Published: 14 September 2007
General Practitioners and community nurses rely on easily accessible, evidence-based online information to guide practice. To date, the methods that underpin the scoping of user-identified online information needs in palliative care have remained under-explored. This paper describes the benefits and challenges of a collaborative approach involving users and experts that informed the first stage of the development of a palliative care website .
The action research-inspired methodology included a panel assessment of an existing palliative care website based in Victoria, Australia; a pre-development survey (n = 197) scoping potential audiences and palliative care information needs; working parties conducting a needs analysis about necessary information content for a redeveloped website targeting health professionals and caregivers/patients; an iterative evaluation process involving users and experts; as well as a final evaluation survey (n = 166).
Involving users in the identification of content and links for a palliative care website is time-consuming and requires initial resources, strong networking skills and commitment. However, user participation provided crucial information that led to the widened the scope of the website audience and guided the development and testing of the website. The needs analysis underpinning the project suggests that palliative care peak bodies need to address three distinct audiences (clinicians, allied health professionals as well as patients and their caregivers).
Web developers should pay close attention to the content, language, and accessibility needs of these groups. Given the substantial cost associated with the maintenance of authoritative health information sites, the paper proposes a more collaborative development in which users can be engaged in the definition of content to ensure relevance and responsiveness, and to eliminate unnecessary detail. Access to volunteer networks forms an integral part of such an approach.
Several recent reports tabled by government as well as non-governmental health agencies foreground a significant disparity between the information needs of health professionals, patients and caregivers givers and the availability of such information. In response, the reports urge governments to invest in the information, education, and training resources available to health-care professionals, patients and caregivers, as well as the wider community, in order to foster awareness and capacity regarding palliative care [2–10].
Whereas the increasing demand for consumer health information on the Internet has been also the topic of much recent medical, social science and information science literature [11–22], there has been a noticeable absence of studies that focus on how to adequately design such online resources. This paper describes the advantages and challenges of a collaborative, action research-inspired approach involving users (doctors, nurses, and consumer and caregiver representatives) that informed the development of a palliative care website  providing accessible information for the general public, along with evidence-based pain, symptom and psycho-social information for health professionals.
The aim of the government-funded study was to improve the existing website of Palliative Care Victoria (PCV), a state-level palliative care organisation. In light of the large volume of inquiries from community doctors, nurses and members of the general public, the Director of PCV was of the view that the organisation's response to such enquiries could be improved by enhancing the information for the general public, along with evidence-based pain, symptom and psycho-social information for health professionals, in particular general practitioners and community nurses available through its website. The study also took into account more global policy issues. For instance, it took into account the key recommendations outlined by the Improving Supportive and Palliative Care for Adults with Cancer report tabled by the United Kingdom-based National Institute for Clinical Excellence in 2004. Among other the report underscored the value of high quality information for patients and caregivers, the importance of ensuring that the views of patients and caregivers are taken into account in developing and evaluating cancer and palliative care services, and that patients and caregivers have easy access to a range of easy to read, high quality information materials about cancer and palliative care services that are provided free of charge . Yet, the study's main aim was to tackle a range of information gaps highlighted in recent national government reports [3, 4, 6, 7]. In particular, the study responded to the policies outlined in the Australian Government National Palliative Care Priorities, namely to improve professional awareness and commitment of health professionals to the role of palliative care practices, and to provide quality information to patients and their caregivers [23, 24].
Theoretical contributions to the field of educational multi-media design indicate that issues regarding content, language, and 'look and feel' could potentially be resolved through an ongoing consultation with expert user groups [31–33]. In fact, it is widely acknowledged that an ideal design process of educational online resources is based on cybernetic principles of general system theory, which relies on cycles of self-testing, ongoing feedback, and constant adjustments . However, this extremely flexible and responsive approach to web design, although able to accommodate changes in and unanticipated effects of the implementation progresses, is seldom employed due to the high cost of cycling back to stages previously completed . In this project, we employed an iterative, formative development process inspired by these principles.
User feedback was collected in the form of focus group discussions, summary grids, written comments, and 'think aloud protocols', a method where users navigate the prototype site delivering verbal comments throughout the process . An audit trail was designed to keep track of user responses.
Because our aim was to include, as much as possible, the views and expertise of all potential users of the website, it was attempted to include as many representatives of potential stakeholder groups as possible in the PRG and UWG so that they would be adequately represented during the various project cycles.
Stage 1 – literature review, pre-development survey, website search
The thematic literature review was based on an extensive search of electronic databases (Medline, CINAHL, PubMed) as well as web-based searches (Google, Google Scholar) for additional material, such as reports commissioned by government as well as non-governmental agencies. Due to the limited scope of the literature on palliative care, sources dealing with online cancer information were included in this literature review. Permutations of search terms (Palliative, Cancer, Care, Online, Internet, Information, Needs) were used to collect a total of 388 bibliographical entries in an EndNote file. A researcher scanned the abstracts of these articles for relevance, thus eliminating 314 articles. The remaining 74 articles were sourced for inclusion into the literature review. In addition a separate literature review was conducted focusing on the available clinical evidence in subcategories of palliative care such as pain, medication, and psychosocial needs. For the clinical subject areas systematic reviews submitted to the Cochrane database, clinical guidelines, as well as expert consensus panels were included. The research underpinning the clinical information included on the website was evaluated and ranked by clinical palliative care experts. Due to the dearth of research-based evidence in sub-categories such as psycho-social needs, the PRG decided to include also what is considered 'best practice' in those fields.
A short online survey was designed to provide a cursory overview of users and their palliative care information needs. The 10 question survey was posted on the existing PCV website. Participants were asked:
• why they accessed the website
• whether they were health professionals or patients/caregivers/members of the public,
• what kind of information they were looking for,
• whether they found the desired information,
• to indicate areas that should be considered for inclusion in the website,
• whether it was easy to locate the information on the existing site, and
• to comment information gaps in the current site.
Internet users who accessed the site were asked to complete the form online. The survey was posted on the internet for a total of five and a half months beginning from 8 October 2003. Response to the survey was anonymous. The webmaster collected the responses and coded and pre-compiled the data. Subsequently, the responses were screened, duplicates or void forms were eliminated, and the pre-compiled data was analysed.
Furthermore, an extensive search of existing health websites was undertaken to identify the key features of reputable or 'Health on the Net' (HON) accredited websites. The Economic and Social Council of the United Nations accredited HON Foundation is the leading organization promoting and guiding the deployment of useful and reliable online medical and health information, and its appropriate and efficient use.
Stage 2 – gap analysis
Taking into account the preliminary findings from the literature review, the web search, and the results of the pre-development survey, both UWGs drafted a detailed gap analysis framework addressing the information needs of the stakeholders they represented. Moreover, the Project Reference Group as well as key stakeholders and the experts from the Centre for Online Multimedia and Educational Technologies (COMET) at La Trobe University, evaluated the original website of Palliative Care Victoria for content, relevance, accessibility, links, search potential, design features and language level. The original website catered predominantly for palliative care 'insiders' (mainly general practitioners and physicians) and posted a series of pamphlets and press releases that could be handed on to patients. The literature reviews in conjunction with the website search as well as guidelines provided an overall framework for content and design, whereas the online survey gave an initial indication of the kind of subject areas health professionals, patients, and caregivers might expect the future site to cover. In this sense, the 'gap analysis' provided very rough guidelines for the construction of a beta site. However, it is important to remember that, because of the formative, iterative process that underpinned the design of the website, needs and gap analyses were ongoing as stakeholder representatives co-designed and debated the content of the site through the numerous iteration cycles.
Stages 3 and 4 – beta development and testing
An effective Project Reference Group was formed. The Project Reference Group (PRG) brought together nine professionals active in a range of palliative care-related fields and who held senior positions in councils, associations, and academia. Moreover, the PRG included a Project Officer who had extensive knowledge of the palliative care field. The Project Officer moderated most group discussions. The sizeable networks of these individuals were instrumental during the implementation phase of the project as many of the User Working Group members were recruited through these networks. Moreover, these participants brought to the project expertise in clinical palliative care, project management, team leadership, online educational technology, and research/evaluation techniques that could be drawn upon over the course of the project. Several members of the PRG joined an editorial panel that took responsibility for drafting editorial guidelines and monitoring content. The PRG functioned as the central mediation vehicle and most editorial decisions were discussed within this forum.
Two User Working Groups (UWG) were formed to participate in the sourcing and evaluation of content. One UWG drew on the expertise of focus group participants brought together by the Cancer Information & Support Services (CISS) run by The Cancer Council Victoria, as part of their activities directed at identifying unmet needs for people affected by cancer. The working group included 11 cancer patients, caregivers and volunteers. The other UWG was made up of 12 health professionals and included four general practitioners, two palliative care physicians, and six palliative care nurses representing nine organisations. Again the networks associated with these members were very extensive and greatly facilitated the user-based evaluation of the process. It was envisaged that these groups would meet on a monthly basis. UWG participants as well as members of their professional networks were asked to provide feedback in the form of summary grids (see additional file 1) after each design cycle.
In addition to the PRG and UWGs, a substantial number of experts in fields such as palliative care, nursing, community support, or allied health made major contributions to the project. Also, a consumer focus group was conducted by a market research firm. This focus group dealt primarily with the look and feel (colour schemes, navigability) of the website. Because representatives of all major stakeholders were involved in the reference and working groups, it is reasonable to suggest that these stake holder groups had a direct input into the design of the web site. Moreover, bearing in mind that 197 and 166 participants respectively responded to the pre-development and evaluation online surveys, it is likely that over 300 individuals contributed to the design and evaluation of the PCV website.
Stage 5 – evaluation
An evaluation survey was attached to the new website. Not withstanding minor changes, the evaluation survey mirrored the content of the pre-development survey. Based on the experience with the pre-development survey, it was decided to separate the information for the public and health professionals. The pre-development survey did not make this separation. Moreover, the evaluation survey was more structured providing a list of possible responses that participants could tick in addition to an 'other' category followed by an open-ended prompt to specify this response.
Stage 1 – literature review and online survey
The literature review revealed a growing recognition of the existence of unmet information needs regarding palliative care practice [6, 9, 35–44]. Recent evidence suggests that doctors, nurses, patients, and caregivers increasingly turn to the internet to meet some of these information needs [12–14, 16, 18, 20, 22, 45–51]. One of the key concerns that surfaced in the literature focusing on online health is the issue of quality. A large number of studies express concerns with regard to the quality of information provided by web sites that post cancer health information on the Internet [52–58]. Most of these studies alert to the fact that websites commonly display misleading, incomplete, and/or erroneous information and that the confidentiality of data is often not assured [55, 59–64]. Surveys of websites and the way users access them suggest that it is difficult and time consuming to identify good online resources and that the capacity of users to find and access valuable information is limited [65, 66]. For most researchers the key problem faced by users is how to evaluate the quality of online information and how to establish the trustworthiness of sources.
• Lack of editorial control,
• misleading or incomplete content,
• inappropriate design,
• inappropriate cultural bias,
• inappropriate commercial content,
• lack of readability (especially for cultural minorities or people with disabilities),
• inappropriate navigation tools,
• overall disorganisation of the site,
• accessibility of audio and visual resources, and
• problems regarding volume management and maintenance.
In response to these quality concerns, authors propose the use of tools to evaluate the quality of health information web sites [73, 74], and/or to introduce obligatory standards or evidence-based editorial criteria to improve the quality of health information sites [54, 66, 75, 76]. Yet, recent contributions to the field have been critical of evaluation tools, claiming that they are largely ineffective .
Other commentators have recommended the design of designated portal sites or gateway servers that organise and edit web content . To some degree, sites such as OncoLink, the Cancer Net, Cancer Help UK, and others fit into this category. A number of articles give an overview of [79–84] and/or evaluate such sites [85, 86]. Designated portal sites or gateway servers generally enjoy impressive approval ratings among clinicians as well as the general public. Nevertheless, recent articles have raised the issue of sustainability and costs, update, and maintenance work faced by online health information providers . More recent evidence questions the proposition that the design of web portals lastingly improves the quality of online health information .
In summary, the literature suggests a need for a more comprehensive information and education strategy focusing on palliative care. It makes it clear that an understanding of the audiences to be targeted as well as their online information needs is crucial for the development of reliable and effective health information websites . Moreover, it appears that for online health information resources to be efficient, four basic conditions have to be met: content needs to be clinically authoritative, accessible, sustainable, and has to respond to the information needs of potential user groups. Furthermore, the design process benefits from an ongoing involvement of stakeholder groups. However, there is a dearth of evidence regarding the information that should be included in a site providing educational information on palliative care to health professionals as well as to patients and associated caregivers as well as the methodology might be successfully employed to that end.
Responses to Pre- and Post-Development Surveys
Period of Survey online
Oct 2003 – March 2004
May 2006 – March 2007
What information were you looking for on the PCV website?
• To find information that would assist me in my care giving role (15%)
• To find information in support of professional practice (79%)
• To find information that would assist patients in their care-giving role (15)
• Clinical aspects of professional practice (13%)
Public & Health Professionals:
• To find resources on palliative care (66%)
• To understand the health care system better (96%)
• About palliative care services in Victoria (40%)
• About helping someone who has an illness (33%)
• To help the person I care for to receive the best possible treatment (31%)
• To help me deal with my feelings (27%)
• To understand what palliative care is (26%)
• About living with my illness (11%)
• About palliative care services in Australia (11%)
• To help me put my affairs in order (9%)
• To help me receive the best care possible (4%)
• Education available in palliative care (50%)
• Symptom Management (37%)
• Advance Care Planning (31%)
• Loss and Grief (25%)
• Talking with clients and their families (23%)
• Palliative care services in Victoria (22%)
• Palliative Care Victoria (21%)
• Death (21%)
• Services available to my clients (21%)
• A website clients and their families could use as a resource (17%)
• The meaning of palliative care (13%)
• Palliative care services in Australia (4%)
Did you find the information you wanted?
128 yes (65%)
68 no (35%)
137 yes/some but no all (82%)
18 no (11%)
Was it easy to find the information?
130 yes (66%)
57 no (29%)
122 yes (74%)
38 no (23%)
Additional information to be included:
• Professional Practice
• Advance Care Planning
• Home Care
• Contact Details
• List of places to go as a family
• Sings of dying/clinical information
• Financial entitlements
Comments about the website
• the need for better navigation tools,
• the site's limited content,
• the ineffectiveness of the site's search engine, and
• the fact that the site had not been updated.
• Plain, clear and unemotional, but still sensitive – well done.
• Well-mapped out, easy to navigate.
The most important reasons respondents provided for accessing the existing palliative care website were to seek information in support of their role as health professionals (79.26%) and professional practice (13%), to find available resources (66.21%), and to find information that would assist them in their care-giving role (15%). Of the respondents 128 (64.97%) indicated that they had found the information they were looking for and 68 (34.52%) stated that they did not. Respondents to the existing website thought that in-depth information about topics such as pain, grief, volunteering and ethical issues should be included. Also noted was the absence of links for volunteers, a site map, a list of recommended publications, a listing of employment opportunities and short courses, comprehensive information for professionals, course information on professional practice, and contact details.
When asked if it was easy to find the information that they were seeking, 130 (65.99%) of the respondents to the existing site gave an affirmative response, whereas 57 (28.93%) responded negatively. Sixty two (31.47%) participants specified their answer. Of these, 26 (13.20%) gave positive feedback, whereas 20 (10.15%) provided constructive criticism. The issues raised by these respondents were:
• the need for better navigation tools,
• the site's limited content,
• the ineffectiveness of the site's search engine, and
• the fact that the site had not been updated.
Stage 2 – gap analysis of existing site
The expert panel discovered limitations in the organisation, content, language, accessibility, design, navigability, and readability of the existing PCV website. To some degree this finding was expected, given the limited resources available for the ongoing support and development of a site mainly administered by volunteers. Among the most important shortcomings noted by the panel was the very limited scope of the website, the lack of effective search capacity, the complexity of the language used and accessibility issues for people with reading difficulties. It was felt that the website was largely providing information to 'insiders' in the palliative care community but not to general health professionals nor to a substantial segment of caregivers, patients, and the general public.
Framework for health professional portal
The principles & practice of palliative care
What is palliative care – definitions
Communication the corner stone of palliative care
Building a supportive relationship
Communicating bad news
Working and communicating within the multidisciplinary team.
Frequently asked questions
Barriers to pain control
Terminal respiratory congestion
Nausea and vomiting
Weakness and fatigue
Acute confusion and delirium
Advance care planning
What is Advanced Care Planning?
Talking about Prognosis and usual course of disease
Advising the patient for Future Decision Making
The Medical Treatment Act and refusal of Treatment
Loss Anxiety & depression
Normal reactions to loss of good health
Complicated reactions to loss of health
Changing family/relationship roles
Sexuality and Intimacy
The family/support network
Information and Support
Respite and Education Services
What needs to be done when the patient dies
Health Care Professional Care
Grief and bereavement
Normal reaction to loss of another
Resources for helping the bereaved
Complicated grief & bereavement
Medication & grief
Framework for consumer portal
About Palliative Care
What is palliative care?
Terminal illness Precious time Maximising opportunities in life FAQ
Who provides palliative care?
Health care team Eg. pastoral care, family as the key unit provider
How do you access palliative care?
Self referral GP Hospital physician Community services Specialist PC services
When does palliative care begin?
When can I ask about pain relief and symptom control?
When do you discuss palliative care and supportive issues?
Navigating the Health Care System
Palliative care and the health care system
Feeling abandoned? Move from the acute serviced orientation Home or Hospital ?
Talking to your doctor
Hints and strategies
Talking with other health care professionals
Hints and strategies
Types Benefits autions
Unhappy with the health care system?
Patient rights Avenues for mediation
Receiving palliative care
Planning for the time you have
What we know about dealing with a life limiting diagnosis
Managing your symptoms
Fatigue Frustration Pain Difficulty breathing
Managing your life
Living with uncertainty Have your affairs in order
What to expect when you are dying
Fears, pain, reaction and behaviours of others
Creating your memory
Healing old wounds Leaving a message Giving treasures to be treasured
What others have done
Stories & Obituaries
Caring for someone receiving palliative care
Being a caregiver
The role/Rights/Support Resources
Understanding your situation
Physical and psychological demands Personal loss & growth Dealing with different coping strategies Need for a break
Events/Diary/Photos The stories of other caregivers
Obtain important information
Deciding what is important
Physical aspects of caring
Bed comfort/Nutrition Medications/Hygiene
Recognising and managing symptoms
Fatigue/Pain Lack of appetite or urinary control
Caring for someone who is receiving palliative care cont'd
When death occurs
What to expect when the one you are caring for is dying Letting go What to do when someone dies Caring for the body after death
Moreover, working parties in conjunction with the PRG defined the overall scope of the new website delimiting the topics that would be covered by the site. Excluded from the site were topic such as careers and financial entitlements.
Stage 3 and 4 – beta development and testing
Because this phase has been published in detail elsewhere only a short summary of its key elements of this phase is given. The formative user evaluation underpinning the project worked exceptionally well. Perhaps the most surprising element of this phase was the level of support that participants were willing to invest in the project. Whereas the UWGs provided crucial information during the initial stages of the project, the editorial panel in conjunction with key user groups associated with the representatives participating in the UWGs provided valuable input during the evaluation of the beta test site. As anticipated, feedback occurred continuously throughout the project and was circulated through the PRG and UWGs. As a result, the PRG and UWGs were able to respond quickly to user feedback and shape the content and overall design of the website accordingly. Expert evaluators became important mediators of conflicts. Referring to 'good practice' guidelines and by providing their 'expert opinion', they, supported by the PRG, managed to resolve diverging views as well as conflicts of interests that surfaced within UWGs.
Stage 5 – evaluation survey
The online survey posted on the new website resulted in a total of 166 responses (see Table 1 below). A total of 121 (73%) respondents identified themselves as health professionals and 45 (27%) as care givers, patients, or members of the public.
Care givers, patients, and members of the public responding to the new website sought primarily information regarding the health care system (96%), about palliative care services in Victoria (40%), as well as information in order to help others with an illness (33%), help the person they cared for (31%), and help them cope (27%). Health professionals responding to the new website (B) sought information about education (50%), symptom management (37%), advance care planning (31%), loss and grief (25%), talking with clients and families (23%), Palliative Care Victoria (22%), death (21%), services available to their clients (21%), and a resource website for clients (17%).
Of the respondents, 137 (82%) indicated that they had found all or some of the desired information and 18 (11%) stated that they did not. Respondents of the evaluation survey thought that topics such as further education, conferences, scholarships, as well as lists of places to go to as a family, signs of dying/clinical information, volunteering and financial entitlements should be considered.
A total of 122 (74%) respondents claimed that it was easy to access the desired information, whereas 38 (23%) claimed that it was not.
Comments focusing on the new website were positive regarding content and design.
Involving users in the identification of content and links for a palliative care website as well as the evaluation of the site's 'look and feel' is time-consuming and requires initial resources, strong networking skills, and ongoing commitment. However, the benefits of engaging key stakeholders in defining and refining the scope and content of such a website outweigh the disadvantages. Over the course of this study, user involvement led to the widening of the scope of the website and alerted the project group to shortcomings regarding content, language, presentation, and usability.
The original intention of the expert re-developed website was to provide information to the consumers and for health professionals in the community who did not have continual access to palliative care expertise. User involvement significantly augmented these aims. In addition to the original intention, the stakeholder-conducted gap analysis in conjunction with the user online survey identified the need for a site that
• provided reputable expert or evidence-based information,
• addressed information requirements in areas such as professional practice, ethical dilemmas or practice standards, as well as topics related to managing palliative pain, grief, advanced care planning, or home care,
• provided clinical content in accessible lay language for the general public,
• guided consumers through the health care system
• provided tools to enhance medical and consumer decision making,
• supplied printable pamphlets for patients and caregivers,
• regularly monitored and updated links and material,
• provided key points that can be easily visually scanned, and
• does not contain distracting graphics that increase download time.
These points would have been missed if the project had proceeded with a common, expert-based web-design.
To some extent, the success effect of the iterative, participatory design of the website is visible in the evaluation survey posted to the new website. For instance, a comparison of the pre-development survey with the evaluation survey shows a decrease of visitors that did not find the information they were looking for from 35% to 11%. Concurrently, the percentage of participants who thought it easy to find the desired information increased from 66 to 74.
Interestingly, since going live, the new website has become an education resource for nursing staff, students, as well as allied health workers. In fact, 46% of the evaluation survey respondents were nurses. Moreover, the evaluation survey suggests that an, albeit small, but nevertheless important number of care givers and patients is accessing the site. Seventeen percent of evaluation survey respondents were care givers or patients. The overall feedback by health professionals as well as patients and caregivers to the new website was, in contrast to the pre-development survey, exclusively positive. What is more, all of the additional topics that health professional thought might be considered for inclusion were outside the scope of the website as defined by UWGs and the PRG.
Yet, the evaluation survey also demonstrates that the portal for patients, families, and their caregivers might require some finetuning. The evaluation survey suggests that whereas the information needs of health professionals are covered by the new site, this is not entirely the case for patients as well as their families and care givers. Given that the information needs of patients and care givers are likely to be more diverse than those of health professionals, this was expected, however. What is more, only two (1%) of respondents to the evaluation survey were General Practitioners (GPs). The reason for this low share in the overall population of respondents has not been analysed. It has to be born in mind that the evaluation survey reflects only on the first eleven months in the life span of the new website. Future surveys will show whether the site will become a central reference for care givers, patients, and GPs.
What is more, the re-development of the website accompanied by the positive feedback the organisation received has led to significant changes in the way PCA manages information. Whereas the old website functioned mainly as a provider of pamphlets and leaflets on palliative care, the new website has assumed a much more central role in the overall operation of the organisation providing education and information. Moreover, PCA plans to web-cast public relations items on the site. Moreover, encouraged by the re-development of the site, the PCV has applied for additional funding in order to integrate web-based information more fully into its communication strategy. Having become aware of the potential of its online strategy, web-support has assumed a more important role in the organisation. This in conjunction with the additional funding PCV was able to attract has impacted positively on the sustainability of the website.
Several limitations were recorded. Perhaps the most important, and obvious, limitation is the usefulness of the pre-development survey in scoping possible web content for a future potential audience that is not yet accessing the site. Whereas the pre-development survey indicated that more non-health professionals accessed the site than was expected, the pre-development survey gave only limited indication as to the information needs of patients and their care givers. In fact, the development process relied to large extent on the input of the UWG as well as the iterative evaluations to define the portal for non health professionals.
Another limitation that emerged more clearly in connection with subsequent projects is the use of volunteer participants. It is important to bear in mind that not all projects are suited to a development phase based largely on volunteer participation of busy health professionals. Although the overall cost of the project (A$ 184,872) was comparatively low, the study was fairly resource intense. In particular, the study relied on a large number of participants who, over the course of three years, generously donated their time and energy. Clearly, for participants to make such a sustained contribution, they need to be extremely committed. Hence, it is important that web developers bear in mind that the salience of the project outcome is likely to influence the commitment levels and, by extension, the support of participants. Furthermore, because of the need to mobilise all major stakeholders and to involve consumer representatives, this kind of approach requires strong networking skills and only individuals with good links to key user groups will be likely to gain the support of key user representatives.
Also, because of the large number of individuals involved in the project and because of the iterative evaluation cycles, an enormous amount of information will be generated. The administration and processing of this information requires well-honed project management skills. In part this had to do with the ineffectiveness of one of the research tools: the summary grid. The summary grid asked for an assessment of the content of each of the web pages associated with these questions using the following criteria: clarity of information, accuracy of information, usefulness of links, target audience for the page, and layout and further suggestions. This feedback method was time-consuming and only members of the consumer panel and a few members of the health professional panel provided feedback in this manner. However the written feedback was most thorough and provided excellent guidance to adjust language and content, to provide further information, and/or to include further links. However, the majority of participants provided feed-back in an ad-hoc, unstructured manner that did not correspond with the data-collection time points we had set. This generated an extraordinary amount of information that, at times, threatened to overwhelm our project officer. This prompted us to strengthen the methodology of subsequent web design projects involving user participation.
For reasons beyond our control, we were unable to generate a denominator for the responses to the initial and subsequent surveys, as well as for the subsequent utilisation of the website. As a result we are unable to complete the rhetorical loop demonstrating an improved usage of the website. This clearly limits the explanatory power of the study.
A growing body of literature and government reports highlight the existence of a significant information gap with regard to palliative care. As more and more people use the internet to meet their health information needs, the question of what kind of palliative care information internet users are actually looking for becomes central to any online information strategy. However, research that focuses on the palliative care information needs of internet users is scarce. This study illustrates how a user-based methodology involving key stakeholders and potential users can contribute successfully to the development of content, language, and design of the website as well as to its evaluation. Web developers are encouraged to involve potential users in their website development and pay close attention to the content, language, and accessibility needs of their respective audiences.
The project was funded by the Department of Health and Ageing, Canberra, Australia. Moreover, to contribute to this project authors were sponsored by their host organisations: Annette F. Street (Prof.) Clinical Nursing La Trobe University, Kathleen Swift Research Assistant Clinical Nursing La Trobe University, Merilyn Annells (Prof.) Community Nursing La Trobe University, Roger Woodruff (Dr) Medical Oncologist and Director of Palliative Care Austin Health, Terry Gliddon (MA AppSci) Research and Development Manger Royal District Nursing Service, Anne Oakley Manager Melbourne Citymission Palliative Care, and Goetz Ottman (Dr) Research Fellow, Clinical Nursing, La Trobe University.
- Palliative Care Victoria. [http://www.pallcarevic.asn.au]
- Healthcare Management Advisors: Paediatric Palliative Care Service Model Review. 2004, Canberra , The National Palliative Care ProgramGoogle Scholar
- National Palliative Care Strategy: A National Framework for Palliative Care Service Development. 2000, Canberra , Australian Government: Department of Health and Ageing
- Sullivan K, Johnston L, Colyer C, Beale J, Willis J, Harrison J, Welsh K: National Indigenous Palliative Care Needs Study. 2003, Canberra , The National Palliative Care ProgramGoogle Scholar
- Mount Olivet Community Services: Research Study into the Educational, Training and Support Needs of General Practitioners in Palliative Care. 2003, Canberra , The National Palliative Care ProgramGoogle Scholar
- Friedman DB, Hoffman-Goetz L, Arocha JF: Readability of cancer information on the internet. Journal of Cancer Education. 2004, 19 (2): 117-122. 10.1207/s15430154jce1902_13.View ArticlePubMedGoogle Scholar
- Birru M, Steinman RA: Online health information and low-literacy African Americans. Journal of Medical Internet Research. 2004, 6 (3): e26-View ArticlePubMedPubMed CentralGoogle Scholar
- PPR Consulting: Promoting Partnerships in Palliative Care Services. 2004, Melbourne , Department of Human Services VictoriaGoogle Scholar
- Davies E, Higginson IJ: The Solid Facts: Palliative Care. 2004, World Health OrganizationGoogle Scholar
- Thain A, Wales A: Information needs of specialist healthcare professionals: a preliminary study based on the West of Scotland Colorectal Cancer Managed Clinical Network. Health Information & Libraries Journal. 2005, 22 (2): 133-142. 10.1111/j.1471-1842.2005.00570.x.View ArticleGoogle Scholar
- Bleakley A, Merzel CR, VanDevanter NL, Messeri P: Computer access and Internet use among urban youths. American Journal of Public Health. 2004, 94 (5): 744-746.View ArticlePubMedPubMed CentralGoogle Scholar
- Bright MA, Fleisher L, Thomsen C, Morra ME, Marcus A, Gehring W: Exploring e-Health usage and interest among cancer information service users: the need for personalized interactions and multiple channels remains. Journal of Health Communication. 2005, 10 Suppl 1: 35-52. 10.1080/10810730500265609.View ArticlePubMedGoogle Scholar
- Dickerson S, Reinhart AM, Feeley TH, Bidani R, Rich E, Garg VK, Hershey CO: Patient Internet use for health information at three urban primary care clinics. Journal of the American Medical Informatics Association. 2004, 11 (6): 499-504. 10.1197/jamia.M1460.View ArticlePubMedPubMed CentralGoogle Scholar
- Doney L, Barlow H, West J: Use of libraries and electronic information resources by primary care staff: outcomes from a survey. Health Information & Libraries Journal. 2005, 22 (3): 182-188. 10.1111/j.1471-1842.2005.00561.x.View ArticleGoogle Scholar
- Escoffery C, Miner KR, Adame DD, Butler S, McCormick L, Mendell E: Internet use for health information among college students. Journal of American College Health. 2005, 53 (4): 183-188. 10.3200/JACH.53.4.183-188.View ArticlePubMedGoogle Scholar
- Rees T: More Americans going online looking for health information. Profiles in Healthcare Marketing. 2005, 21 (4): 2-Google Scholar
- Balmer C: The information requirements of people with cancer: where to go after the "patient information leaflet"?. Cancer Nursing. 2005, 28 (1): 36-44. 10.1097/00002820-200501000-00005.View ArticlePubMedGoogle Scholar
- Basch EM, Thaler HT, Shi W, Yakren S, Schrag D: Use of information resources by patients with cancer and their companions. Cancer. 2004, 100 (11): 2476-2483. 10.1002/cncr.20261.View ArticlePubMedGoogle Scholar
- Brotherton JM: Internet is indeed useful source for patients with cancer. BMJ. 2004, 328 (7444): 898-10.1136/bmj.328.7444.898.View ArticlePubMedPubMed CentralGoogle Scholar
- Mick J: Patients used the internet to transform their cancer experiences. Evidence Based Nursing. 2004, 7 (4):
- Ziebland S, Chapple A, Dumelow C, Evans J, Prinjha S, Rozmovits L: How the internet affects patients' experience of cancer: a qualitative study. BMJ. 2004, 328 (7439): 564-10.1136/bmj.328.7439.564.View ArticlePubMedPubMed CentralGoogle Scholar
- Pereira J, Bruera E, Quan H: Palliative Care on the Net: An online survey of health care professionals. Journal of Palliative Care. 2001, 17 (1): 41-45.PubMedGoogle Scholar
- Palliative Care Australia: Standards for providing quality palliative care for all Australians. 2005, Yarralumla, ACTGoogle Scholar
- Palliative Care Australia: A Guide to Palliative Care Service Development: A population based approach. 2005, 52-Google Scholar
- Reason P, Bradbury H: Introduction: Inquiry and participation in search of a world worthy of human aspiration. Handbook of action research: participatory inquiry and practices. Edited by: Reason P, Bradbury H. 2001, London , SAGE, 1-14.Google Scholar
- Reason P, Bradbury H: Handbook of action research : participative inquiry and practice. 2001, London , SAGE, 468-Google Scholar
- Street A: Action research. Research methods for nursing and health science. Edited by: Minichiello V, Sullivan G, Greenwood K, Axford R. 2004, Sydney , Pearson Prentice Hall, 278-294.Google Scholar
- Dick B: Utopia made practical? Action Research comes of age. Forum: Qualitative Social Research. 2002, 3 (1):
- Kemmis S, McTaggart R: Participatory action research. Strategies of Qualitative Inquiry. Edited by: Denzin NK, Lincoln YS. 2003, Thousand Oaks , SAGE, 91-102.Google Scholar
- Williamson GR, Prosser S: Action research: Politics, ethics and participation. J Adv Nurs. 2002, 40 (5): 587-593. 10.1046/j.1365-2648.2002.02416.x.View ArticlePubMedGoogle Scholar
- Gabbard JL, Hix D, Swan JE: User-centered design and evaluation of virtual environments. Computer Graphics and Applications, IEEE. 1999, Virginia Polytech. Inst. & State Univ., Blacksburg, VA, 19 (6): 51-69. 10.1109/38.799740.
- Weston CMALB: A Model for Understanding formative evaluation in instructional design. Education Technology Research and Development. 1995, 43 (3): 29-48. 10.1007/BF02300454.View ArticleGoogle Scholar
- Polson PG, Lewis C, Rieman J, Wharton C: Cognitive walkthroughs: a method for theory-based evaluation of user interfaces. Int J Man-Mach Stud. 1992, Academic Press Ltd., 36 (5): 741-773. 10.1016/0020-7373(92)90039-N.
- Wilson B, Jonassen D, Cole P: Cognitive approaches to instructional design. The ASTD handbook of instructional technology. Edited by: Piskurich GM. 1993, New York , McGraw-HillGoogle Scholar
- Families need more Information about Postmortem Examinations. BMJ. 2002, 325: 1121-10.1136/bmj.325.7372.1121.
- Contro N, Larson J, Scofiled S, Sourkes B, Cohen H: Family Perspectives on the Quality of Pediatric Palliative Care. Archives of Pediatric & Adolescent Medicine. 2002, 156 (1): 14-View ArticleGoogle Scholar
- Coolidge C: Death Wish. Forbes. 2001, 206-Google Scholar
- Davies E: What are the Palliative Care Needs of older People and how might they be Met. Health Evidence Network. 2004, World Health OrganisationGoogle Scholar
- Fallowfield L: Palliative Care; Dying cancer patients need more information from doctors. Cancerweekly Plus. 2002Google Scholar
- Foley Pierce S: Improving End-of-Life Care: Gathering Suggestions from Family Members. Nursing Forum. 1999, 34 (2): 5-10.1111/j.1744-6198.1999.tb00981.x.View ArticleGoogle Scholar
- Kirchhoff KT, Walker L, Hutton A, Spuhler V, Vaughan Cole B, Clemmer T: The Vortex: Families' Experiences with Death in the Intensive Care Unit. American Journal of Critical Care. 2002, 11 (3): 200-PubMedGoogle Scholar
- Murray SA, Boyd K, Kendall M, Worth A, Benton FT, Clausen H: Dying of lung cancer or cardiac failure: prospective qualitative interview study of patients and their carers in the community. BMJ. 2002, 325: 929-932. 10.1136/bmj.325.7370.929.View ArticlePubMedPubMed CentralGoogle Scholar
- Kirk P, Kirk I, Kristjanson LJ: What do patients receiving palliative care for cancer and their families want to be told? A Canadian and Australian qualitative study. BMJ. 2004, 328 (7452): 1343-10.1136/bmj.38103.423576.55.View ArticlePubMedPubMed CentralGoogle Scholar
- Wilkes L, White K, O'Riordan L: Empowerment through Information: Supporting rural Families of Oncology Patients in Palliative Care. Australian Journal of Rural Health. 2000, 8 (1): 41-46. 10.1046/j.1440-1584.2000.81258.x.View ArticlePubMedGoogle Scholar
- Pereira J, Bruera E, Macmillan K, Kavanagh S: Palliative Cancer Patients and Their Families on the Internet: Motivation and Impact. Journal of Palliative Care. 2000, 16 (4): 13-PubMedGoogle Scholar
- Wozar JA, Worona PC: The use of online information resources by nurses. Journal of the Medical Library Association. 2003, 91 (2): 216-221.PubMedPubMed CentralGoogle Scholar
- Balmer C: The information requirements of people with cancer: where to go after the "patient information leaflet"?[see comment]. Cancer Nursing. 2005, 28 (1): 36-44. 10.1097/00002820-200501000-00005.View ArticlePubMedGoogle Scholar
- Brotherton JM: Internet is indeed useful source for patients with cancer.[comment]. BMJ. 2004, 328 (7444): 898-10.1136/bmj.328.7444.898.View ArticlePubMedPubMed CentralGoogle Scholar
- Baker L, Wagner TH, Singer S, Bundorf MK: Use of the Internet and E-mail for Health Care Information: Results From a National Survey. JAMA. 2003, 289 (18): 2400-2406. 10.1001/jama.289.18.2400.View ArticlePubMedGoogle Scholar
- Pereira J, Bruera E: The Internet as a resource for palliative care and hospice: A review and proposal. Journal of Pain and Symptom Management. 1998, 16 (1): 59-68. 10.1016/S0885-3924(98)00022-0.View ArticlePubMedGoogle Scholar
- Bath PA, Clark D, Wright M: The International Observatory on End of Life Care (IOELC): An Information Resource for Palliative Care. Health Informatics Journal. 2004, 10 (2): 121-126. 10.1177/1460458204042235.View ArticleGoogle Scholar
- Ekman A, Hall P, Litton JE: Can we trust cancer information on the Internet?--A comparison of interactive cancer risk sites. Cancer Causes & Control. 2005, 16 (6): 765-772. 10.1007/s10552-005-1722-3.View ArticleGoogle Scholar
- Felber SH: Re: bladder cancer facts: accuracy of information on the internet.[comment]. Journal of Urology. 2004, 171 (6 Pt 1): 2383; author reply 2383-PubMedGoogle Scholar
- Ilic D, Risbridger G, Green S: Searching the Internet for information on prostate cancer screening: an assessment of quality. Urology. 2004, 64 (1): 112-116. 10.1016/j.urology.2004.02.027.View ArticlePubMedGoogle Scholar
- Jorgensen KJ, Gotzsche PC: Presentation on websites of possible benefits and harms from screening for breast cancer: cross sectional study.[see comment]. BMJ. 2004, 328 (7432): 148-10.1136/bmj.328.7432.148.View ArticlePubMedPubMed CentralGoogle Scholar
- Molassiotis A, Xu M: Quality and safety issues of web-based information about herbal medicines in the treatment of cancer. Complementary Therapies in Medicine. 2004, 12 (4): 217-227. 10.1016/j.ctim.2004.09.005.View ArticlePubMedGoogle Scholar
- Viswanath K: Science and society: the communications revolution and cancer control. [Review] [62 refs]. Nature Reviews Cancer. 2005, 5 (10): 828-835. 10.1038/nrc1718.View ArticlePubMedGoogle Scholar
- Weissenberger C, Schultze-Seemann W: Re: bladder cancer facts: accuracy of information on the internet. Journal of Urology. 2004, 172: 1544-PubMedGoogle Scholar
- Burkell J, Campbell DG: "What does this mean?" How Web-based consumer health information fails to support information seeking in the pursuit of informed consent for screening test decisions. Journal of the Medical Library Association. 2005, 93 (3): 363-373.PubMedPubMed CentralGoogle Scholar
- Coleman J, Olsen SJ, Sauter PK, Baker D, Hodgin MB, Stanfield C, Emerling A, Hruban RH, Nolan MT: The effect of a Frequently Asked Questions module on a pancreatic cancer Web site patient/family chat room. Cancer Nursing. 2005, 28 (6): 460-468. 10.1097/00002820-200511000-00009.View ArticlePubMedGoogle Scholar
- Fagerlin A, Rovner D, Stableford S, Jentoft C, Wei JT, Holmes-Rovner M: Patient education materials about the treatment of early-stage prostate cancer: a critical review. Annals of Internal Medicine. 2004, 140 (9): 721-728.View ArticlePubMedGoogle Scholar
- Nilsson-Ihrfelt E, Fjallskog ML, Blomqvist C, Ahlgren J, Edlund P, Hansen J, Malmberg L, Villman K, Andersson G: Breast cancer on the Internet: the quality of Swedish breast cancer websites. Breast. 2004, 13 (5): 376-382. 10.1016/j.breast.2004.03.003.View ArticlePubMedGoogle Scholar
- Schmidt K, Ernst E: Assessing websites on complementary and alternative medicine for cancer. Annals of Oncology. 2004, 15 (5): 733-742. 10.1093/annonc/mdh174.View ArticlePubMedGoogle Scholar
- Simon C, Hegedus S: Exploring websites on cancer clinical trials: an empirical review. Contemporary Clinical Trials. 2005, 26 (5): 530-533. 10.1016/j.cct.2005.07.004.View ArticlePubMedGoogle Scholar
- Jones SD: "Information Rx" project launched in Virginia. National Network. 2005, 29 (3): 20-PubMedGoogle Scholar
- Morahan-Martin JM: How internet users find, evaluate, and use online health information: a cross-cultural review. [Review] [61 refs]. Cyberpsychology & Behavior. 2004, 7 (5): 497-510.View ArticleGoogle Scholar
- Nahm ES, Preece J, Resnick B, Mills ME: Usability of health Web sites for older adults: a preliminary study. CIN: Computers, Informatics, Nursing. 2004, 22 (6): 326-34; quiz 335-6.PubMedGoogle Scholar
- Boulos MN: British internet-derived patient information on diabetes mellitus: is it readable?. Diabetes Technology & Therapeutics. 2005, 7 (3): 528-535. 10.1089/dia.2005.7.528.View ArticleGoogle Scholar
- Changrani J, Gany F: Online cancer education and immigrants: effecting culturally appropriate websites. Journal of Cancer Education. 2005, 20 (3): 183-186. 10.1207/s15430154jce2003_14.View ArticlePubMedGoogle Scholar
- Fikar CR, Keith L: Information needs of gay, lesbian, bisexual, and transgendered health care professionals: results of an Internet survey. Journal of the Medical Library Association. 2004, 92 (1): 56-65.PubMedPubMed CentralGoogle Scholar
- Zeng X, Parmanto B: Web content accessibility of consumer health information web sites for people with disabilities: a cross sectional evaluation. Journal of Medical Internet Research. 2004, 6 (2): e19-10.2196/jmir.6.2.e19.View ArticlePubMedPubMed CentralGoogle Scholar
- Weissenberger C, Jonassen S, Beranek-Chiu J, Neumann M, Muller D, Bartelt S, Schulz S, Monting JS, Henne K, Gitsch G, Witucki G: Breast cancer: patient information needs reflected in English and German web sites. British Journal of Cancer. 2004, 91 (8): 1482-1487.PubMedPubMed CentralGoogle Scholar
- Bernstam EV, Sagaram S, Walji M, Johnson CW, Meric-Bernstam F: Usability of quality measures for online health information: Can commonly used technical quality criteria be reliably assessed?. International Journal of Medical Informatics. 2005, 74 (7-8): 675-683. 10.1016/j.ijmedinf.2005.02.002.View ArticlePubMedGoogle Scholar
- Childs S: Developing health website quality assessment guidelines for the voluntary sector: outcomes from the Judge Project. Health Information & Libraries Journal. 2004, 21 Suppl 2: 14-26.View ArticleGoogle Scholar
- Aarimaa M: Telemedicine--contribution of ICT to health. Studies in Health Technology & Informatics. 2004, 100: 111-116.Google Scholar
- Gimenez-Perez G, Caixas A, Gimenez-Palop O, Gonzalez-Clemente JM, Mauricio D: Dissemination of 'patient-oriented evidence that matters' on the Internet: the case of Type 2 diabetes treatment. Diabetic Medicine. 2005, 22 (6): 688-692. 10.1111/j.1464-5491.2005.01522.x.View ArticlePubMedGoogle Scholar
- Bernstam EV, Shelton DM, Walji M, Meric-Bernstam F: Instruments to assess the quality of health information on the World Wide Web: what can our patients actually use?. International Journal of Medical Informatics. 2005, 74 (1): 13-19. 10.1016/j.ijmedinf.2004.10.001.View ArticlePubMedGoogle Scholar
- Darmoni SJ, Dahamna B, Roth-Berghofer TR: Seal of transparency heritage in the CISMeF quality-controlled health gateway. BMC Medical Informatics & Decision Making. 2004, 4: 15-10.1186/1472-6947-4-15.View ArticleGoogle Scholar
- Chambers L, McDowall J, Gelb B: A global children's hospice and palliative care website. International Journal of Palliative Nursing. 2005, 11 (6): 292-293.View ArticlePubMedGoogle Scholar
- Gavrin JR: The web site of the center to advance palliative care. Journal of Pain & Palliative Care Pharmacotherapy. 2004, 18 (4): 77-79. 10.1300/J354v18n04_09.View ArticleGoogle Scholar
- Junghans TB, Sevin IF, Ionin B, Seifried H: Cancer information resources: digital and online sources. Toxicology. 2004, 198 (1-3): 177-193. 10.1016/j.tox.2004.02.021.View ArticlePubMedGoogle Scholar
- Lewis D, Gundwardena S, El Saadawi G: Caring connection: developing an Internet resource for family caregivers of children with cancer. CIN: Computers, Informatics, Nursing. 2005, 23 (5): 265-274. 10.1097/00024665-200509000-00011.PubMedGoogle Scholar
- Packer BR, Yeager M, Burdett L, Welch R, Beerman M, Qi L, Sicotte H, Staats B, Acharya M, Crenshaw A, Eckert A, Puri V, Gerhard DS, Chanock SJ: SNP500Cancer: a public resource for sequence validation, assay development, and frequency analysis for genetic variation in candidate genes. Nucleic Acids Research. 2006, 34 (Database issue): D617-21. 10.1093/nar/gkj151.View ArticlePubMedGoogle Scholar
- Williams M: Palliative care web sites. Home Healthcare Nurse. 2005, 23 (11): 736-737.View ArticlePubMedGoogle Scholar
- Kosinska J, Slowikowski P: Technical aspects of portal technology application for e-health systems. [Review] [18 refs]. Studies in Health Technology & Informatics. 2004, 105: 12-20.Google Scholar
- Winefield HR, Coventry BJ, Lambert V: Setting up a health education website: practical advice for health professionals. Patient Education & Counseling. 2004, 53 (2): 175-182. 10.1016/S0738-3991(03)00149-6.View ArticleGoogle Scholar
- De Moor GJ, Claerhout B, van Maele G, Dupont D: e-Health standardization in Europe: lessons learned. Studies in Health Technology & Informatics. 2004, 100: 233-237.Google Scholar
- Glenton C, Paulsen EJ, Oxman AD: Portals to Wonderland: health portals lead to confusing information about the effects of health care. BMC Medical Informatics & Decision Making. 2005, 5 (1): 7-10.1186/1472-6947-5-7.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/7/26/prepub