Skip to main content

A randomized controlled trial comparing in-person and wiki-inspired nominal group techniques for engaging stakeholders in chronic kidney disease research prioritization



Few studies have evaluated stakeholder engagement in chronic kidney disease (CKD) research prioritization. In this two-arm, parallel group randomized controlled trial, we sought to compare an in-person nominal group technique (NGT) approach with an online wiki-inspired alternative to determining the top 10 CKD research priorities, and to evaluate stakeholder engagement and satisfaction with each process.


Eligible participants included adults ≥18 years with access to a computer and Internet, high health literacy, and from one of the following stakeholder groups: patients with CKD not on dialysis, their caregivers, health care providers who care for patients with CKD, or CKD-related health policymakers. Fifty-six participants were randomized to a wiki-inspired modified NGT that occurred over 3 weeks vs. a 1-day in-person NGT workshop, informed by James Lind Alliance methodology, to determine the top 10 CKD-related research priorities. The primary outcome was the pairwise agreement between the two groups’ final top 10 ranked priorities, evaluated using Spearman’s correlation coefficient. Secondary outcomes included participant engagement and satisfaction and wiki tool usability.


Spearman’s rho for correlation between the two lists was 0.139 (95 % confidence interval −0.543 to 0.703, p = 0.71), suggesting low correlation between the top 10 lists across the two groups. Both groups ranked the same item as the top research priority, with 5 of the top 10 priorities ranked by the wiki group within the top 10 for the in-person group. In comparison to the in-person group, participants from the wiki group were less likely to report: satisfaction with the format (73.7 vs.100 %, p = 0.011); ability to express their views (57.9 vs 96.0 %, p = 0.0003); and perception that they contributed meaningfully to the process (68.4 vs 84.0 %, p = 0.004).


A CKD research prioritization approach using an online wiki-like tool identified low correlation in rankings compared with an in-person approach, with less satisfaction and perceptions of active engagement. Modifications to the wiki-inspired tool are required before it can be considered a potential alternative to an in-person workshop for engaging patients in determining research priorities.

Trial registration


Peer Review reports


Chronic kidney disease (CKD), defined as estimated glomerular filtration rate (eGFR) <60 mL/min/1.73 m2, affects approximately 5–7 % of the adult population [13] and contributes to excess morbidity, mortality and health care costs [4, 5]. Despite the tremendous burden CKD places on health care delivery, care for persons affected by CKD remains suboptimal [6]. These care gaps may result from the lack of research that is relevant to end-users, including patients with CKD, their caregivers, and the clinicians involved in their care [7, 8]. To address these gaps, there is increasing emphasis on the importance of engaging key stakeholders in determining research priorities and establishing user-relevant research objectives [9, 10]. The Strategy for Patient Oriented Research (SPOR) [11] in Canada and the Patient-Centered Outcomes Research Institute (PCORI) in the United States [12] are two initiatives that aim to engage patients in research and knowledge translation and to focus on interventions and outcomes that are important to research end users.

Although few established methods for involving key stakeholders in research prioritization exist, the James Lind Alliance (JLA) priority setting partnership model is widely considered an inclusive and transparent process and has been applied to a number of disease conditions [13]. The final step of the four-step JLA partnership involves an in-person workshop with a nominal group technique (NGT) to use a list of the top 30 research priorities to develop a ranked list of the top 10 research priorities from key stakeholders through small and large group activities [14]. While considered the reference standard, this approach has some clear limitations for participants and coordinators, including time and resource requirements, travel, and logistical consideration [15]. These challenges may be particularly problematic for patients with chronic illness who have difficulty with or are unable to travel, which could limit participant involvement and generalizability of findings. Alternative Internet-based approaches to research priority setting, such as an interactive wiki-like platform, could obviate some of these limitations. A ‘wiki’ is a collaborative web-based platform that allows users to add and edit content online in real time, and in the context of research prioritization can be used to summarize the preferences of multiple stakeholders [16]. Wiki-inspired tools have been successfully applied in similar, multi-stakeholder consensus approaches, including the creation of an asthma action plan [16], but have not been studied in research prioritization exercises that require significant patient involvement.

In this randomized controlled trial we sought to compare the in-person workshop and NGT approach with an online wiki-inspired alternative as the final step in the JLA research prioritization process for key CKD stakeholders, including patients with CKD not on dialysis, their caregivers, and the health care professionals and policymakers involved in their care. We aimed to compare the top 10 research priorities resulting from each process and to evaluate the engagement and satisfaction of participating stakeholders with their assigned intervention.


This trial is reported in accordance with the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement [17] (trial registration:, ISRCTN18248625).

Overview of the JLA CKD priority setting partnership

The JLA priority setting partnership uses established methods to identify and prioritize unanswered disease- or treatment-related questions [14]. Our CKD priority setting process involved: 1) creation of a 12-person steering group of patients with non-dialysis CKD, caregivers, clinicians and researchers; 2) identification of research uncertainties through a national survey of representative CKD stakeholders; 3) refinement and prioritization of the research uncertainties identified in the survey to determine a shortlist of the top 30 research priorities by the steering group; and, 4) participation by stakeholders from across Canada in a 1-day priority setting workshop to determine the top 10 CKD research priorities from a shortlist of 30. Steps 1–3 took place from July 2014 to June 2015; this trial involved step 4 only.

Trial participants

The steering group enlisted the support of nephrology networks, partner organizations, and nephrologists from across Canada to identify potential participants. Eligible participants were ≥18 years of age, English speaking, had access to and were comfortable using high-speed Internet, and had high health literacy (determined by asking the following question: “How confident are you in filling out medical forms by yourself?” [18]). Participants were also required to be a member of a stakeholder group: patients with CKD not on dialysis or with a prior transplant; informal caregivers of persons with CKD (relatives, family members, or friends who help patients manage their illness); health care professionals (primary care physicians, nephrologists, nurses, pharmacists, social workers, or dietitians) who care for patients with CKD; or health policymakers (non-clinicians with the ability to influence or determine policies and practices related to health care delivery for CKD at an international, national, regional, or local level). The following individuals were not eligible to participate: patients receiving dialysis and/or who had received a kidney transplant; persons with an underlying diagnosis of dementia or cognitive impairment; and patients deemed unfit to travel. A research assistant confirmed the eligibility of potential participants, described the trial to them in detail, and obtained their written informed consent.


Participants were randomized 1:1 to an in-person NGT-based workshop or an online wiki-inspired platform. Randomization occurred by random number sequence generation using a central electronic system to ensure allocation concealment. Patients and their caregivers were randomized as a dyad to avoid contamination. Additionally, the 6 patient and caregiver steering group members were allocated to the in-person workshop group only, and not randomized, as they had originally committed to participate in the NGT process of the study and it was therefore deemed important to include them in the NGT arm. Participants in both groups were provided with the top 30 uncertainties shortlist (derived from a national survey and prioritized by the steering group) to review and consider prior to the intervention; participants in the in-person workshop group ranked the 30 uncertainties prior to the workshop, while participants in the wiki-inspired group ranked the top 10 (due to logistical issues of ranking 30 online).

Individuals in the 1-day in-person workshop group participated in a NGT approach [14, 19] in Toronto, Canada, which included a combination of small and large group exercises facilitated by four individuals with experience in the JLA process. The final ranking of the top 10 research priorities was determined by large group consensus, with voting when needed.

Wiki group participants were provided with a 1-h training webinar on use of the wiki and were given unique usernames and passwords to access the secure website before they began the following tasks to determine the top 10 research priorities. The wiki-based intervention took place online over 3 phases: 1) an individual ranking phase (7 days), where participants submitted their preferences for the top 10 research priorities individually, 2) a group ranking phase (12 days), where participants determined the final top 10 research priorities collaboratively, and 3) a sign-off phase (2 days), where participants could review to confirm their agreement, but not re-order the final list. Following the individual phase, individual rankings were aggregated across participants using reverse scoring, such that items ranked first received a score of 10, and those ranked last received a score of 1. These scores were aggregated and presented as a summary rank list to which participants could refer throughout the collaborative ranking phase. During the collaborative ranking phase, participants could re-order the top 10 priorities in real time and discuss their selections using a chat feature. The chat feature consisted of a text box and display of all participant comments in chronological order, organized by unique user identifier (ie identifying information and stakeholder role were not apparent during chat to avoid differential perception of input by provider role). A facilitator monitored the site daily and stimulated discussion through comments using the chat feature. Technical support was available throughout the process. Due to the nature of the interventions, study participants and intervention administrators were not blinded to group assignment; however, statistical analysis was blinded. The in-person workshop took place during the wiki process; participants in both groups were blinded to the ranking results of the other group.


The primary outcome was the pairwise agreement between the two groups’ final top 10 ranked priorities recorded upon conclusion of the workshop and group ranking phase for the in-person and wiki groups, respectively. Secondary outcomes included perceived engagement of participants in their assigned prioritization process and participant satisfaction. We also evaluated usability aspects of the online wiki tool including content, features, and accessibility (see Additional file 1 for details on outcomes and methods of measurement). The electronic questionnaire assessing participant engagement and satisfaction and wiki usability was administered using the FluidSurveys platform (FluidSurveys platform, Fluidware Inc., Ottawa, Ontario)., Key elements of stakeholder engagement were evaluated in the participant engagement section of the questionnaire, including trust, respect, accountability, legitimacy, fairness, and competence [20, 21].

Sample size

The primary outcome was the pair wise agreement between the two groups’ top 10 priorities, therefore the correlation analysis was driven by the number of research priorities considered, which was fixed at 10 for the purposes of this study. As the JLA recommends 20 to 30 participants in the in-person priority setting exercise, with approximately equal representation of patients, clinicians, and caregivers, we estimated that a total of 40–60 participants would be required (we aimed for 50). Inflating this by 10 % to account for potential dropouts, an overall; sample size of 56 was felt to be a conservative estimate for the primary outcome.

Statistical analyses

Baseline demographic information was compared between the in-person workshop and wiki groups using univariate analyses. The unit of analysis for the primary outcome was at the group level, and the pairwise agreement between the two groups’ priorities (provided as ranks) was examined using Spearman’s correlation coefficient and its associated 95 % confidence interval (CI). We also compared the rank order of the top 10 research priorities from each group descriptively, noting the proportion that were within the top 10 across both groups. Our secondary outcomes were obtained by survey for both the in-person and wiki groups, with responses measured using a 5-point Likert scale (where 1 = strongly disagree, 5 = strongly agree), and the proportions of participants who agreed with each statement (4 or 5 on the scale) were compared between groups using chi-square tests. Written comments also obtained from the survey by both groups were examined qualitatively. Two researchers (MJE and BRH) reviewed all written comments and independently coded them for emerging categories and themes. Resulting themes and discrepancies were discussed, and a final coding scheme describing participants’ experiences with each process was established. Participant observation data collected during the in-person workshop was summarized quantitatively (ie frequencies) and qualitatively (ie brief researcher notes). The time requirements to undertake each process were compared descriptively.



Fifty-six individuals were randomized to either the in-person workshop (n = 28) or online wiki-inspired group (n = 28). In addition the 6 patient and caregiver steering group members were allocated to the in-person workshop group only, as established a priori, for a total of 34 participants for the in-person workshop. Overall 25 individuals took part in the 1-day workshop (12 patients, 6 caregivers, 6 health care professionals, and 1 policymaker), and 26 individuals participated in the 3-week online wiki-like process (12 patients, 8 caregivers, 5 health care professionals, and 1 policymaker). Participant enrolment and reasons for withdrawal are outlined in Fig. 1. During the 3-week period of wiki activity, 21 of 26 participants (80.8 %) logged onto the ranking page at least once during either the individual or group phase (3 patients and 2 caregivers did not access the site during the intervention). All participants contributed to the primary analysis of agreement between the two top 10 research priority lists, as this was assessed at the group level. Only participants who completed the post-intervention survey were included in the analysis of participant engagement and satisfaction.

Fig. 1
figure 1

Flow diagram outlining intervention group allocation and withdrawals. *Primary outcome measured at the group level

The majority of study participants were 40–64 years of age, and 47 % were male (Table 1). 12 patients and 6 caregivers participated in the in-person workshop, and 11 patients and 8 caregivers participated in the online wiki-inspired process. Compared to the in-person group, a smaller proportion of patients and caregivers in the wiki-inspired group were < 40 years of age and female (5.6 vs. 21.1 % and 38.9 vs 52.6 % respectively), and they were less likely to work full time (31.6 vs. 44.4 %) or hold a university degree (26.3 vs. 61.1 %).

Table 1 Baseline demographics of participants in the in-person workshop and online wiki groups, n (%)

Comparison of the top 10 CKD research priorities

At the conclusion of the in-person workshop and wiki-inpsired interventions, the top 10 research priorities were determined for each group (Table 2). Overall, Spearman’s rho for correlation between the two groups was 0.139 (95 % CI −0.543 to 0.703, p = 0.71), suggesting a non-significant low level of correlation between the two groups. Both groups independently ranked the same item as the top research priority (What are the most effective new interventions and treatments to prevent the development and progression of kidney disease?). Of the top 10 research priorities ranked by the wiki-inspired group, 5 were ranked within the top 10 by the workshop group, and a further 4 items were ranked within the top 15 by the workshop group. Of note, 5 of the top 10 research priorities in the NGT group were not in the top 10 of the wiki-inspired group, with four of these being health services research questions. The #10 priority in the NGT workshop group was extensively discussed in both small and large group settings, suggesting that greater interaction in the NGT workshop setting may have influenced the final rankings. The research priorities that were common to the top 10 lists of both groups included dietary measures to slow CKD progression, CKD symptoms, and the impact of lifestyle factors and medications on kidney disease. Two wiki group members (both clinicians) had participated previously in an in-person priority setting workshop for patients on or nearing dialysis [22], which may have influenced their perceptions and experiences of the intervention.

Table 2 Top 10 CKD-related research priorities resulting from the in-person workshop vs. online wiki-based groups

Participant satisfaction and engagement

All 25 participants in the NGT-based workshop group and 19 of 26 participants from the online wiki-inspired group completed the post-intervention questionnaire. The non-responders from the wiki group included 4 patients, 2 caregivers and 1 health care professional. All workshop participants felt the in-person NGT format was well-suited (ie noted agreement or strong agreement) to determine CKD research uncertainties, whereas 73.7 % of respondents in the wiki group were satisfied (agree/strongly agree) with the format (p = 0.011) (Fig. 2a). Ninty-six percent of participants in the workshop group felt they could adequately express their views in this format, as compared to 57.9 % in the wiki group (p = 0.003). With respect to participant engagement, workshop participants were significantly more likely than wiki participants to report that the format used by the group respected the ideas of all participants (100 vs. 68.4 %, p = 0.004) and perceived that their views contributed to the final list of research priorities (84.0 vs. 33.3 %, p < 0.001) (Fig. 2b). Respondents from both groups felt they had adequate knowledge (80.0 % in workshop group vs. 83.3 % in wiki group, p = 1.0) and skills (92.0 % in workshop group vs. 89.5 % in wiki group, p = 1.0) to rank the top research priorities, and around one quarter of participants felt that influential participants impacted the group dynamics (28.0 % in workshop vs. 21.1 % in wiki group p = 0.73).

Fig. 2
figure 2

a Satisfaction of workshop and wiki participants on post-intervention questionnaire. *p < 0.05 for between-group difference. b Engagement of workshop and wiki participants on post-intervention questionnaire. *p < 0.05 for between-group difference

Written comments were provided in the survey by 11 participants in the in-person NGT group and 10 participants in the wiki-inspired group. The main themes that emerged were related to opportunities for personal interactions and meaningful contributions, participation rates, format effectiveness and the importance of communication to justify individual rankings (Table 3), with the workshop group generally providing more positive feedback of their experience across these three themes. Written comments from wiki participants highlighted their perceptions of a low level of active participation and inadequacy of the chat feature (which aimed to allow comprehensive discussion of participants’ views, preferences and justification of changes to the their rankings). Wiki participants felt that the top ranked uncertainties were ordered and re-ordered without a clear understanding of the reasons for the changes and with limited group discussion. In contrast, in-person workshop participants felt they were able to contribute equally and effectively among all stakeholders in this format.

Table 3 Main themes identified from written feedback on post-intervention questionnaire

Observation of participants during the in-person workshop revealed a moderate to high degree of verbal and physical engagement for the majority of group members. Three participants (2 patients and 1 caregiver) demonstrated limited verbal and physical interaction with others. Most individuals appeared to readily voice their views, questions and comments spontaneously or with prompting by the facilitator. While the initial small group discussions were reserved and initiated mainly by the facilitator, more spontaneous and animated discussions among the participants arose throughout the day. Some participants contributed more confidently and persuasively than others, but these individuals were not limited to one stakeholder group and did not appear to negatively dominate any of the group discussions. In the wiki-inspired group, chat discussions were similarly reserved and required facilitator prompting initially. However, throughout the process, participants provided spontaneous input and sought each other’s views in a respectful manner.

Wiki usability

During the 3-week period of wiki activity, 21 of 26 participants (80.8 %) logged onto the ranking page at least once during either the individual or group phase. Those who did not log in during the intervention included 3 patients and 2 caregivers. During the group ranking phase, 13 users (50.0 %) were considered active participants (ie they made direct changes to the top 10 uncertainties list [n = 11, 42.3 %] and/or participated in the chat feature [n = 13, 50.0 %]). These active participants included 6 patients, 3 caregivers, and 4 health professionals. One patient who was admitted to hospital during the individual ranking phase accessed the ranking tool, submitted her individual ranking, and participated in the group ranking and discussion from a laptop computer while in hospital.

Over the 12-day group ranking phase, a total of 192 changes were made to the rankings (mean 16 changes/day, range 1–37) and 104 comments were provided in the group chat (mean 8.7 comments/day, range 2–21). The most changes were made on the first day of the group round (37 changes). Twenty-six changes were made to the list on the final day, and the most group discussion (21 comments) occurred on the final day of the exercise. The most active participants included 2 patients and 1 health care provider, each of whom provided between 46 and 58 contributions to the rankings and/or group chat. Half of the wiki-inspired group participants viewed and confirmed they had reviewed the final top 10 list following conclusion of the group ranking phase.

Results from the wiki usability portion of the post-intervention questionnaire are outlined in Additional file 2. Of the 18 respondents, all felt that 12 days were sufficient to undertake the group ranking process, and 61 % used the tool as often as they would have liked. Eighty-three percent of participants experienced no difficulties in changing the rankings, and 61 % found the chat feature useful for discussing the uncertainties as a group. Six participants contacted study coordinators regarding technical issues, all of which related to difficulty logging onto the site. All issues but one were resolved, with this user being able to login after repeated attempts but having ongoing difficulties participating in the ranking process.

Time requirements

The in-person workshop intervention took place over 1 weekday and required travel time for participants from across Canada to attend. The online wiki-based group could access the ranking site at any time of day, from any location, and as often as participants preferred during a 3-week period.


In this randomized trial evaluating two methods of stakeholder engagement to determine CKD research priorities, we found that an online wiki-inspired approach had a low level of correlation compared to the research priorities determined from an in-person NGT approach informed by JLA methodology, with only 5 of the top 10 priorities identified by the wiki-inspired group within the top 10 priorities of the in-person group. There were also important differences between the groups, with fewer health-services research questions ranked by the wiki-inspired group, and differences in perceptions of engagement. In comparison to participants in the in-person workshop, those in the wiki-inspired group were less satisfied with the format of the exercise and felt they were less likely to contribute meaningfully to the process. Whereas all individuals who attended the in-person workshop contributed to the group discussions (either spontaneously or with prompting), 80 % of individuals in the wiki group accessed the tool at least once, and only half were considered ‘active’ participants. We are unable to determine if the differences in research priorities identified were due to the different processes, or differences in the groups themselves (including the patient and caregiver steering group members participating in the in-person workshop), as priorities may vary by group irrespective of the process used to establish them. This will be important to address in future work.

Our study is one of the first to evaluate an online wiki-based tool to engage patients and other stakeholders in health care research prioritization and to undertake a randomized comparison of two distinct research prioritization methods. Although both techniques are based on a NGT approach, the online wiki-inspired option had attractive features including flexibility to contribute at a convenient time during a process that occurred over a longer period of time compared to the 1-day in-person workshop, which the majority of participants felt was adequate. However the longer duration required for participation may be perceived as a disadvantage by some. The standard JLA methodology for stakeholder involvement in health care research prioritization, including a final in-person workshop, has been implemented to date in more than 20 chronic diseases, including prior work by our group to identify research priorities for patients on or nearing dialysis [22]. A recent systematic review on research priority setting in kidney disease identified 16 studies, only 4 of which involved patients in the prioritization process [23]. The studies cited in this review used a variety of methods to elicit research priorities, none of which were described in detail nor used an electronic or online collaborative platform to engage stakeholders, and none of which were evaluated in a randomized trial such as this one.

Wiki-like tools have been applied in other collaborative health care-related activities involving patients and other stakeholders. Gupta et al. included patients and clinicians in a wiki-inspired collaborative design of an asthma action plan (WikiBuild), where groups of 14 participants contributed to the action plan’s content and design over a 1-week period [16]. Other studies examining the application of wiki-based tools in patient-centered research, including the development of clinical practice guidelines [24] and patient information leaflets [25], have found these tools to be promising alternatives to more traditional approaches. In contrast to our study, these other studies generally involved input from a large numbers of patients (>300) and lack the personalized and multi-stakeholder interactions endorsed by the JLA format.

Potential advantages of an online wiki-inspired approach for collaborative research prioritization include flexibility in participation, lack of travel requirements limiting participant eligibility, and lower costs [16]. A particular advantage is the ability to include those whose illnesses may preclude easy travel; for example, one patient in the wiki-inspired group in our study was admitted to hospital during the study period and participated in the process while in hospital. The complex group dynamics often seen with in-person techniques can lead to the development of status hierarchies that favour clinicians over patients and caregivers [19, 26, 27], which could limit the openness of individual contributions to the discussion and influence the final product [28], although our qualitative results would suggest that was not the case in our study. Although neither the wiki group nor the in-person workshop group indicated the presence of overly influential individuals, wiki-inspired participants indicated the ability to contribute honestly and explicitly in this format.

Although 61 % of wiki-inspired participants responding to the post-intervention questionnaire felt that the chat feature was useful for explaining their views and preferences, many expressed concerns that this format did not allow for as comprehensive a discussion and understanding as is possible with face-to-face interactions. Gupta et al. described similar participant concerns for the wiki online chat, as well as the lack of individual accountability for changes made [16]. Other barriers to the adoption of wiki-inspired tools for patient engagement activities described include concerns regarding suitability of the format to its objectives, poor participation rates, and technical challenges, among others [16, 24, 25]. It is unclear whether patients with poor health literacy or cognitive impairment would be able to participate in a wiki-inspired priority setting process, and this is an area for future work. The participant feedback provided during our trial highlights the need to refine the online collaborative tool, and the chat feature in particular, to ensure participants feel engaged and able to contribute meaningfully to the process.

Our study is one of the first to describe the application of a wiki-inspired tool to a research prioritization exercise, with a randomized comparison of methods for eliciting input from a diverse group of stakeholders to determine research priorities. The results, however, should be interpreted in light of the study limitations. The requirement for participants to have regular access to a computer and the Internet may have limited eligible participants and thus the generalizability of findings. However, Internet access is increasingly common among patients with chronic disease (up to 70 % of patients with chronic illness are Internet users) [29], and approximately half of patients with advanced CKD access the Internet to obtain health information [30, 31]. Further, little evidence exists to guide the evaluation of stakeholder engagement with such consensus-driven processes [32]. While quantitative and written responses on the post-intervention survey provide valuable information, additional insights could be gained through participant interviews evaluating individual experiences with the interventions. Finally, 6 of the patient and caregiver steering group members participated in the NGT in-person workshop, and were not part of the randomization process. Although these steering group members were involved in the earlier stages of the CKD priority setting partnership (ie identification of research priorities through national survey and refinement of these priorities into a top 30 shortlist), the final ranking of the top 10 priorities was a collaborative process involving all stakeholders participating in the workshop. All participants were encouraged to contribute to the final ranking process, and the top 10 priorities reflect group consensus rather than the views of any one individual.


Involving patients and other stakeholders in determining research priorities has the potential to enhance the quality and relevance of research for end users. In our randomized controlled trial of patients, caregivers, clinicians and policymakers, we found that an online collaborative wiki-inspired process identified different top ten CKD-related research priorities compared with an in-person workshop, and participants in the wiki group were less likely to feel engaged and satisfied. We are unable to determine if the differences in research priorities identified were due to the different processes or differences in the groups themselves, as priorities may vary by group irrespective of the process used to establish them. This is important to address in future work. Modifications to the wiki tool, including optimization of a communication feature to improve the ability of participants to communicate their views and preferences, need to be designed and evaluated before a wiki process can be considered a feasible alternative to an in-person workshop for engaging patients in determining research priorities.



Confidence interval


Chronic kidney disease


Consolidated Standards of Reporting Trials


Estimated glomerular filtration rate


James Lind Alliance


Nominal group technique


Patient-Centered Outcomes Research Institute


Strategy for patient oriented research


  1. Coresh J, Selvin E, Stevens LA, Manzi J, Kusek JW, Eggers P, et al. Prevalence of chronic kidney disease in the United States. JAMA. 2007;298(17):2038–47.

    Article  CAS  PubMed  Google Scholar 

  2. Arora P, Vasa P, Brenner D, Iglar K, McFarlane P, Morrison H, et al. Prevalence estimates of chronic kidney disease in Canada: results of a nationally representative survey. CMAJ. 2013;185(9):E417–23.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Aitken GR, Roderick PJ, Fraser S, Mindell JS, O'Donoghue D, Day J, et al. Change in prevalence of chronic kidney disease in England over time: comparison of nationally representative cross-sectional surveys from 2003 to 2010. BMJ Open. 2014;4(9), e005480.

    Article  PubMed  PubMed Central  Google Scholar 

  4. Go AS, Chertow GM, Fan D, McCulloch CE, Hsu CY. Chronic kidney disease and the risks of death, cardiovascular events, and hospitalization. N Engl J Med. 2004;351(13):1296–305.

    Article  CAS  PubMed  Google Scholar 

  5. Jha V, Garcia-Garcia G, Iseki K, Li Z, Naicker S, Plattner B, et al. Chronic kidney disease: global dimension and perspectives. Lancet. 2013;382(9888):260–72.

    Article  PubMed  Google Scholar 

  6. Ronksley PE, Hemmelgarn BR. Optimizing care for patients with CKD. Am J Kidney Dis. 2012;60(1):133–8.

    Article  PubMed  Google Scholar 

  7. Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci. 2012;7:50.

    Article  PubMed  PubMed Central  Google Scholar 

  8. Liberati A. Need to realign patient-oriented and commercial and academic research. Lancet. 2011;378(9805):1777–8.

    Article  PubMed  Google Scholar 

  9. Brett J, Staniszewska S, Mockford C, Herron-Marx S, Hughes J, Tysall C, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect. 2014;17(5):637–50.

    Article  PubMed  Google Scholar 

  10. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65.

    Article  PubMed  Google Scholar 

  11. Canadian Institutes of Health Research. Strategy for Patient-Oriented Research (SPOR) Accessed 15 Aug 2015.

  12. Patient-Centered Outcomes Research Institute (PCORI) Accessed 15 Aug 2015.

  13. Cowan K. The James Lind alliance: tackling treatment uncertainties together. J Ambul Care Manage. 2010;33(3):241–8.

    Article  PubMed  Google Scholar 

  14. Cowan K. The James Lind Alliance Guidebook. Oxford: James Lind Alliance; 2013.

    Google Scholar 

  15. Keown K, Van Eerd D, Irvin E. Stakeholder engagement opportunities in systematic reviews: knowledge transfer for policy and practice. J Contin Educ Health Prof. 2008;28(2):67–72.

    Article  PubMed  Google Scholar 

  16. Gupta S, Wan FT, Newton D, Bhattacharyya OK, Chignell MH, Straus SE. WikiBuild: a new online collaboration process for multistakeholder tool development and consensus building. J Med Internet Res. 2011;13(4), e108.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Schulz KF, Altman DG, Moher D. CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. Ann Intern Med. 2010;152(11):726–32.

    Article  PubMed  Google Scholar 

  18. Powers BJ, Trinh JV, Bosworth HB. Can this patient read and understand written health information? JAMA. 2010;304(1):76–84.

    Article  CAS  PubMed  Google Scholar 

  19. Murphy MK, Black NA, Lamping DL, McKee CM, Sanderson CF, Askham J, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess. 1998;2(3):i–iv. 1–88.

    CAS  PubMed  Google Scholar 

  20. Deverka PA, Lavallee DC, Desai PJ, Esmail LC, Ramsey SD, Veenstra DL, et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J Comp Eff Res. 2012;1(2):181–94.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Lavallee DC, Williams CJ, Tambor ES, Deverka PA. Stakeholder engagement in comparative effectiveness research: how will we measure success? J Comp Eff Res. 2012;1(5):397–407.

    Article  PubMed  Google Scholar 

  22. Manns B, Hemmelgarn B, Lillie E, Dip SC, Cyr A, Gladish M, et al. Setting research priorities for patients on or nearing dialysis. Clin J Am Soc Nephrol. 2014;9(10):1813–21.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Tong A, Chando S, Crowe S, Manns B, Winkelmayer WC, Hemmelgarn B, et al. Research priority setting in kidney disease: a systematic review. Am J Kidney Dis. 2015;65(5):674–83.

    Article  PubMed  Google Scholar 

  24. den Breejen EM, Nelen WL, Knijnenburg JM, Burgers JS, Hermens RP, Kremer JA. Feasibility of a wiki as a participatory tool for patients in clinical guideline development. J Med Internet Res. 2012;14(5), e138.

    Article  Google Scholar 

  25. van de Belt TH, Faber MJ, Knijnenburg JM, van Duijnhoven NT, Nelen WL, Kremer JA. Wikis to facilitate patient participation in developing information leaflets: first experiences. Inform Health Soc Care. 2014;39(2):124–39.

    Article  PubMed  Google Scholar 

  26. Pagliari C, Grimshaw J. Impact of group structure and process on multidisciplinary evidence-based guideline development: an observational study. J Eval Clin Pract. 2002;8(2):145–53.

    Article  PubMed  Google Scholar 

  27. Cantrill JA, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract. 1996;4:67–74.

    Article  Google Scholar 

  28. Mansell I, Bennett G, Northway R, Mead D, Moseley L. The learning curve: the advantages and disadvantages in the use of focus groups as a method of data collection. Nurse Res. 2004;11(4):79–88.

    Article  PubMed  Google Scholar 

  29. Kruse RL, Koopman RJ, Wakefield BJ, Wakefield DS, Keplinger LE, Canfield SM, et al. Internet use by primary care patients: where is the digital divide? Fam Med. 2012;44(5):342–7.

    PubMed  Google Scholar 

  30. Schatell D, Wise M, Klicko K, Becker BN. In-center hemodialysis patients’ use of the internet in the United States: a national survey. Am J Kidney Dis. 2006;48(2):285–91.

    Article  PubMed  Google Scholar 

  31. Ong SW, Jassal SV, Porter E, Logan AG, Miller JA. Using an electronic self-management tool to support patients with chronic kidney disease (CKD): a CKD clinic self-care model. Semin Dial. 2013;26(2):195–202.

    Article  PubMed  Google Scholar 

  32. Esmail L, Moore E, Rein A. Evaluating patient and stakeholder engagement in research: moving from theory to practice. J Comp Eff Res. 2015;4(2):133–45.

    Article  PubMed  Google Scholar 

Download references


We would like to thank Mr. David Newton for his assistance in the development and administration of the online collaborative ranking tool. We also thank Ms. Sarah Gil for her regular communication with and technical assistance to all participants in the wiki group, and Ms. Sarah Gillis for assisting with the coordination of the in-person workshop.


MJE is supported by an Alberta Innovates-Health Solutions Clinician Fellowship and Kidney Research Scientist Core Education and National Training Program Post-Doctoral Fellowship. BRH, NP, SBA and BJM are supported by an alternative funding plan from the Government of Alberta and the Universities of Alberta and Calgary. SBA is supported by an Alberta Innovates – Health Solutions Clinical Investigator award. HT is supported by an Alberta Innovates – Health Solutions Graduate Studentship. AL is supported by a Canada Research Chair in Health Policy and Citizen Engagement, SES is supported by a Tier 1 Canada Research Chair in Knowledge Translation, and BRH is supported by the Roy and Vi Baay Chair in Kidney Research. This research was supported by the Canadian Institutes of Health Research and an interdisciplinary team grant from Alberta Innovates – Health Solutions, the Interdisciplinary Chronic Disease Collaboration. The sponsors of the study had no role in study design, data collection, data analysis, data interpretation or writing of the report.

Availability of data and material

The data supporting the main conclusions of this article are included within Table 2 (top 10 CKD-related research priorities resulting from the in-person workshop vs. online wiki-based groups). All other data is stored in a secure data holding at the University of Calgary and may be made available from the corresponding author on reasonable request.

Authors’ contributions

MJE, BRH, and SES were responsible for the conception and design of this project. MJE, NP, SBA, AL, GCC, DRH, KTH, ACL, BVV, DMY and BRH were members of the JLA steering committee who contributed substantially to the design and implementation of the workshop and wiki interventions. HT, MD, EL, and BJM were additional non-steering committee individuals who provided methodological expertise and significant contributions in the planning and administration of the in-person workshop. MJE conducted and is responsible for the data analysis. All authors drafted or revised the article critically for important intellectual content, and all authors gave final approval of the version to be published.

Competing interests

The authors declare that they have no competing interests.

Consent for publication

Not applicable.

Ethics approval and consent to participate

Ethics approval for this study was obtained from the Institutional Review Board of the University of Calgary. Written informed consent was obtained from all participants.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Meghan J. Elliott.

Additional files

Additional file 1:

Table summarizing methods used for assessing primary and secondary outcomes. (DOCX 104 kb)

Additional file 2:

Figure demonstrating usability responses from the wiki-based group on the post-intervention questionnaire. (DOCX 120 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Elliott, M.J., Straus, S.E., Pannu, N. et al. A randomized controlled trial comparing in-person and wiki-inspired nominal group techniques for engaging stakeholders in chronic kidney disease research prioritization. BMC Med Inform Decis Mak 16, 113 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: