The CRISP colorectal cancer risk prediction tool: an exploratory study using simulated consultations in Australian primary care
© The Author(s). 2017
Received: 15 June 2016
Accepted: 6 January 2017
Published: 19 January 2017
In Australia, screening for colorectal cancer (CRC) with colonoscopy is meant to be reserved for people at increased risk, however, currently there is a mismatch between individuals’ risk of CRC and the type of CRC screening they receive.
This paper describes the development and optimisation of a Colorectal cancer RISk Prediction tool (‘CRISP’) for use in primary care. The aim of the CRISP tool is to increase risk-appropriate CRC screening.
CRISP development was informed by previous experience with developing risk tools for use in primary care and a systematic review of the evidence. A CRISP prototype was used in simulated consultations by general practitioners (GPs) with actors as patients. GPs were interviewed to explore their experience of using CRISP, and practice nurses (PNs) and practice managers (PMs) were interviewed after a demonstration of CRISP. Transcribed interviews and video footage of the ‘consultations’ were qualitatively analyzed. Themes arising from the data were mapped onto Normalization Process Theory (NPT).
Fourteen GPs, nine PNs and six PMs were recruited from 12 clinics. Results were described using the four constructs of NPT: 1) Coherence: Clinicians understood the rationale behind CRISP, particularly since they were familiar with using risk tools for other conditions; 2) Cognitive participation: GPs welcomed the opportunity CRISP provided to discuss healthy and unhealthy behaviors with their patients, but many GPs challenged the screening recommendation generated by CRISP; 3) Collective Action: CRISP disrupted clinician-patient flow if the GP was less comfortable with computers. GP consultation time was a major implementation barrier and overall consensus was that PNs have more capacity and time to use CRISP effectively; 4) Reflexive monitoring: Limited systematic monitoring of new interventions is a potential barrier to the sustainable embedding of CRISP.
CRISP has the potential to improve risk-appropriate CRC screening in primary care but was considered more likely to be successfully implemented as a nurse-led intervention.
KeywordsColorectal cancer Colorectal cancer risk assessment Colorectal cancer risk prediction Primary care Cancer screening General practitioners Bowel cancer screening
Australia and New Zealand have the highest rates of colorectal cancer (CRC) globally, with estimated age standardized rates of 44.8 and 32.2 per 100,000 in men and women respectively . Risk-stratified screening is increasingly recognized as an approach to maximize the benefits and reduce the potential harms of cancer screening, especially in the era of personalized/precision medicine . In Australia, the National Health and Medical Research Council (NHMRC) guidelines recommend the non-invasive and inexpensive fecal occult blood test (FOBT), using the fecal immunochemical test (FIT), for screening in people at average risk of CRC, and more invasive and expensive colonoscopy only for those who are at increased risk, with risk-stratification based on age and family history criteria . However, it is estimated that for every 1,000,000 Australians aged 50 years and older, 80 000 people at average risk are being over-screened with colonoscopy, 29 000 at increased risk are not having the colonoscopy they need [4, 5], and only 37% participate in the National Bowel Cancer Screening Program (NBCSP) . Over-screening potentially results in excess costs and excess risks to the patient due to the invasive nature of colonoscopy, and under-screening potentially results in missing early-stage cancer or precancerous lesions.
Risk-stratified screening requires valid risk prediction models that accurately discriminate people at increased risk from those who are not. These risk models require easy-to-use risk assessment tools for implementation into clinical practice . A recent systematic review of trials of cancer risk assessment tools in primary care found that few trials have been conducted with risk tools in primary care, and of those trials, many have been limited by methodological flaws including low recruitment rates and a ‘healthy volunteers’ effect . Nevertheless, the review showed that such tools can improve patient risk perception, cancer knowledge and screening intentions, although the effect on actual screening behaviours is less clear. Few risk assessment tools in these trials implemented validated cancer risk models or applied the latest evidence on risk communication methods. Furthermore, several had undergone limited clinical evaluation prior to testing in a full scale randomized controlled trial (RCT), contrary to recommendations about the development and evaluation of complex interventions [9, 10].
Our NHMRC funded ‘Centre for Research Excellence to optimise screening for colorectal cancer’ is undertaking a program of research to develop an Australian CRC risk prediction model, and to implement it into a risk assessment tool for use in primary care (‘general practice’). This paper describes an initial exploratory study based on the UK’s Medical Research Council Framework for the Development and Evaluation of Complex Interventions to improve healthcare [9, 10]. This qualitative Phase I study aimed to examine the useability and acceptability of a prototype tool ‘CRISP’ (Colorectal cancer RISk Prediction tool), identify barriers and enablers to implementing CRISP in Australian general practice, and optimize the design of CRISP as an important step in the development of an intervention prior to an efficacy (Phase II) RCT [10, 11].
We recruited general practices from a broad range of clinics including some from the Victorian Primary Care Practice-Based Research Network (VicReN). Practice managers (PMs) and practice nurses (PNs) were recruited from the same practices to explore the context for practice implementation. Participants were reimbursed for their time.
The CRISP tool was programmed as Java software leveraging the Apache Wicket open source framework for web application development. This software tool was designed to implement any epidemiological risk model with simple modifications to data entry fields. Based on a review of previously validated risk models , the ‘Freedman’ CRC risk model (with minor modifications) was chosen to be the epidemiological model underpinning the prototype CRISP tool .
This study used an ‘action design’ method to allow for changes in CRISP to explore the potential utility of the different iterations of the tool as respondents’ data were analyzed.
CRC risk assessment
The CRISP risk assessment tool was designed to assist clinicians to collect risk information from patients and provide screening recommendations during a consultation. The CRISP interface was based on the results of our systematic review of cancer risk assessment tools in primary care  and previous experience in developing cancer risk tools for use in primary care . Risk variables captured by CRISP included: age, gender, BMI, smoking, exercise, diet, previous colonoscopy, previous polyp, use of non-steroidal anti-inflammatory drugs and family history and age of onset of CRC, ovarian or endometrial cancer, for first and second degree relatives.
no screening for people less than 50 years old with a five-year absolute CRC risk less than 1.00%;
FOBT for people 50 years or older with a five-year absolute CRC risk less than 1.00%;
colonoscopy for people with a five-year absolute CRC risk greater than or equal to 1.00%; or
referral to a familial cancer centre for further risk assessment for people with a strong family history of CRC or family history suggestive of Lynch syndrome.
GPs were familiarized with CRISP using paper-based cases and then conducted two simulated consultations with actors professionally trained to be patients [full descriptions of the case vignettes available in Appendix A]. The actors’ performances were monitored to ensure standardization. The simulated consultations were conducted in the GPs’ consulting rooms and video recorded.
CRISP was demonstrated and explained to the PNs and practice managers PMs.
Semi-structured interviews were conducted by the same interviewer (JGW) after the simulated consultations or CRISP demonstration. Interviews were audio-recorded and transcribed.
The interview guide was developed using Normalization Process Theory (NPT). NPT is a well-established sociological theory which proposes a working model for implementing, embedding and sustaining practices and/or interventions into clinical practice, in particular, in primary care . Acceptability, usability and implementation strategies were explored at an individual level (GP, PN and PM) and organizational level (the practice) using the four domains of NPT: ‘Coherence’, ‘Cognitive participation’, ‘Collective action’ and ‘Reflexive monitoring’ .
Data saturation was reached when no new themes were occurring after three consecutive interviews for any of the three groups.
Analysis commenced soon after data collection started. Video footage of the simulated consultations was analyzed to explore important non-verbal observations such as body language of the GP and ‘patient’ and any impact using CRISP had on doctor-patient interaction. The time spent per screen and time to complete a CRISP session as well as any user error were captured electronically. Video footage and audio transcripts were coded in dedoose 6.0.24® . A coding framework was developed inductively involving discussions between the authors (JGW, JDE, FMW). The themes that arose from the analysis were then mapped onto the four constructs of NPT.  CRISP was modified during the study in response to observations and early data analysis. The revised version of CRISP was reviewed by two GPs who had been previously involved in the study and two GPs unfamiliar with CRISP.
Ethics approval and consent to participate
All participants were recruited following an informed consent process involving approved plain language statements and consent forms (HREC reference number: 1340491).
Characteristics of general practitioners, practice nurses and practice managers participating in the study
50 (29, 62)
Practice location, metropolitan
Number of years in general practiceb
22 (1, 34)
Number of GPs working in practice:b
2 (0, 7)
5 (0, 14)
Hours worked in an average weekb
30 (12, 50)
Postgraduate qualifications, yes
More than one qualification
55 (48, 68)
Practice location, metropolitan
Number of years in general practiceb
35 (9, 40)
Hours worked in an average weekb
36 (30, 40)
Specialist qualifications, yes
More than one specialization
50 (29, 58)
Practice location, metropolitan
Number of years in general practiceb
2 (1, 21)
Hours worked in an average weekb
40 (31, 46)
Specialist qualifications, yes
Results of the analysis using the four constructs of NPT
So we use quite a few assessment tools in the practice and … obviously find them valuable. So, from that point of view, we’re not unfamiliar with using them and certainly wouldn’t be averse to using them across the board. [PM: 49 years, female]
…but don’t forget that little person; okay let’s make sure that’s not you. [GP: 50 years, female]
…there’s a whole lot of people and this is your chance out of them of cancer… [GP: 31 years, female]
The graph was more challenging to explain. In the first iteration of CRISP the y axis (CRC risk) ranged from 0% to 20% which generated a steep curve at times, and caused concern that this might mislead patients about their risk (Fig. 3). All staff preferred the natural frequency icon array to the graph depicting comparative CRC risk over time.
…that’s just the tricky bit of saying [to the patient] that this is what’s [sic] the recommendation but we don’t always follow what recommendations are… [GP: 50 years, female]
Because once you go beyond that first degree relative, really the risk diminishes so significantly that getting all that extra data isn’t going to be that significant. [GP: 50 years, female]
Anything that gets people to have a think about their health is a good thing, anything at all. [PN: 48 years, female]
GP: Do you smoke?Patient: No.GP: Good man! [GP 53 years, male]
On the other hand, there was a general distrust of some of the recommendations, particularly for FOBT. Colonoscopy was commonly referred to as the ‘gold standard’ for CRC screening. GPs were more likely to recommend colonoscopy if the patient wanted one or they had previously been recommended one by a gastroenterologist, even if the patient was at average risk.
… nurses … see the patient first a lot of the time and so they could go through these sorts of things… [PM: 48 years, female]
The existing health checks for 45 – 49 year olds  which are funded by the Australian government were suggested as an optimum time for PNs to introduce CRISP.
… I’ve got a 26-year-old patient that has …metastases everywhere …of course, everybody said it was her endometriosis… [GP: 50 years, female]
…Yeah. I just don’t think it will work in the waiting room…We might start getting asked questions at the front desk… That’s the thing, how do we answer? You know what I mean? [PM: 29 years, female]
…the doctors are very open to suggestion. If the practice nurses, management, or another doctor feels that this could be a really good idea for the doctors, it’s taken to the doctors’ meeting and overall feeling of shall we do this, shall we not is taken place…[PM: 42 years, female]
So I’m in a really big group practice… and … people would be really excited… [But it] would vary because some people don’t change their practice much at all… [GP: 33 years, female]
…this is a labyrinth!… especially having 3 pages for the 2nd degree relatives…[GP: 50 years, female]
GPs wanted CRISP integrated into existing clinical software so data could be auto-populated and then updated into the patient record but acknowledged that the variety of software used in different practices would make it difficult to integrate CRISP universally.
There were challenges using CRISP within a GP consultation. GPs felt uncomfortable taking their attention away from the patient by looking at the computer. Despite this, GPs often re-engaged with the patient when explaining the risk output and screening recommendations. Younger GPs, particularly those who could touch type, were able to move more seamlessly between the computer and the patient. When GPs asked questions of their patients prior to opening CRISP, which was not uncommon, they felt the tool was unnecessarily repetitive.
… the capacity’s there. Yeah, and we certainly have things that we can do like that, but often we do nothing. … - it’s like ships in the night, so we can be here and not even say hi to each other all day sometimes…[PM: 55 years, female]
Consequently, there were limited systems within the practices to monitor or evaluate the effectiveness of CRISP over time.
As a preliminary qualitative study in a program of research to develop and trial a novel CRC risk prediction tool, we identified clear barriers and facilitators to the implementation of our prototype tool (CRISP) into general practice. We found that GPs were capable of using CRISP in simulated consultations, and were generally used to applying risk tools as part of their clinical decision making. Despite this, GPs did not always adhere to the CRISP screening advice, with many favouring colonoscopic screening even for people at average risk as it was considered a ‘better’ screening method. Barriers to using CRISP included limited consultation time, long-winded family history collection, and for some GPs, lack of confidence with computers. All clinic staff thought the questions about lifestyle were very helpful for facilitating a discussion about how to change unhealthy behaviors and could see how CRISP might fit into a health-check conducted by the PN. For CRISP to be implemented into practice, it was obvious that it would need to be more intuitive and shorter and would be easier if it was integrated into the patient clinical medical record.
Iterative modifications to the CRISP tool during the study
- 1.development of an interactive family history collection page on a single web-page to collect family history; this was more intuitive and user friendly, and substantially reduced the time to complete CRISP (Fig. 4);
- 2.modifications to the natural frequency icon array to include a larger denominator population and therefore present smaller risks more clearly (Fig. 5);
- 3.modifications to the graph to reduce potential overestimation of risk over time (Fig. 6); and
- 4.as a result of our findings on GPs’ perceptions about the relative benefits and harms of FOBT and colonoscopy, we developed additional risk communication output in CRISP to highlight the potential harms of colonoscopy and the sensitivity of FOBT to detect CRC using ‘expected frequency trees’ (Fig. 7) .
Our results are consistent with findings about the few available cancer risk assessment tools used in primary care . The use of a dedicated clinician such as a PN has the potential to increase uptake of the risk tool [22, 23] and, similar to findings in the Family Healthware Impact Trial, health promotion within a risk tool increased clinician ‘buy in’ . In comparison with other risk tools, GPs did not experience a sense of ‘loss of control’ of the consultation when using the tool, even if the screening advice contradicted their expectations [23, 25].
The CRISP research team previously conducted an exploratory study of a diagnostic risk assessment tool (QCancer) using similar methods, but the CRISP study involved GPs, PNs and PMs. CRISP was more acceptable to GPs than QCancer, although both were difficult to introduce into a GP consultation. The advantage of interviewing all staff in the CRISP study was the discovery that clinic staff were in agreement that CRISP would be better suited to use in a nurse consultation. Another significant difference between QCancer and CRISP was the user interface. A major limitation to the accuracy of QCancer was input error . This is a common limitation to the accuracy of medical technology in clinical care , but was not found to be a factor in this study. Perhaps more significantly though was the nature of the clinical task being supported by the risk assessment tool. GPs in this study were less confronted by discussing future risks of colorectal cancer than they were using the QCancer tool which presented risks of current undiagnosed cancer for multiple tumour types.
Our study demonstrated that primary care in Australia has the capacity to implement risk assessment tools using PNs and potentially contribute to more risk-appropriate CRC screening. Practice staff felt CRISP would fit into the context of a preventive health consultation conducted by a PN, in particular as part of ‘45 – 49 year old’ health checks already funded by the Australian government to increase screening for chronic diseases . This provides the opportunity to prime patients to participate in the NBCSP prior to receiving their NBCSP FOBT after turning 50, and identify patients at higher risk of CRC for whom colonoscopy and/or genetic counselling may be indicated. The variety of risk communication formats employed in the CRISP tool is designed specifically to support discussion and shared decision making where there is a choice of screening test and different harms and benefits to consider. The expected frequency trees are expected to challenge GPs’ beliefs about the superiority of colonoscopic screening. Implementing CRISP as a PN-led intervention has the advantage of nesting CRISP into an existing model of care, which frequently involves shared decision making about disease prevention. This will also minimize the impact on GPs’ limited consultation time.
Strengths and limitations
Although some of the participants were recruited from a practice-based research network, we recruited a range of GPs, PNs and PMs including: 1) rural and metropolitan practice staff, 2) solo and group practice staff, and 3) staff from a range of ages and experience level in general practice. The sample of PNs was mainly female and of similar ages, however, this was not atypical for this population and is likely to be representative . This applied to PMs as well.
We attempted to recruit GPs, PNs and PMs from the same clinics, to assess congruency of ideas between staff at the same clinic. However, this was not always possible with some clinic staff unavailable for an interview and one clinic did not have a PN.
Another limitation was the potential for social desirability bias as participants might not have felt comfortable criticizing CRISP to the researchers during the interviews. This was offset by the data gained from the simulated consultations which have been demonstrated to predict actual clinical performance .
This study has provided important contextual information on some of the potential barriers and facilitators to implementing CRISP and risk-stratified CRC screening in primary care. It has informed further development of CRISP so that it is more likely to meet the needs of users and provide risk information that could support improved screening decision making. We have identified a preferred model of implementation in practice involving PNs. This approach will now be tested in an RCT to assess the effects of implementing CRISP on GP and patient CRC screening behaviours.
Colorectal cancer RISk prediction tool
Fecal immunochemical test
Fecal occult blood test
National health and medical research council (Australia)
Normalization process theory
Randomized controlled trial
The authors would like to acknowledge the clinic staff who participated in this study. Thanks also to Professor Elizabeth Murray, whose discussions about NPT informed very valuable additions to the paper.
This study was funded by a National Health and Medical Research Council (NHMRC) Centre for Research Excellence grant (APP1042021). M Pirotta is supported by an NHMRC Career Development Fellowship. JD Emery is supported by an NHMRC Practitioner Fellowship. FM Walter is supported by a NIHR Clinician Scientist award.
Availability of data and materials
Consent for publication of raw data was not obtained because the videos and audiotapes identify individuals. In the interest of collecting candid qualitative data, it was important for participants to know that they were not going to be identified publicly. All data in this manuscript have been de-identified or aggregated to ensure privacy complying with the University of Melbourne HREC.
The idea for the manuscript was conceived by JDE, MP, MJ, JGW, and CRECRC. The CRISP tool was designed and developed by AB, SM, NH and JD. JGW wrote the first draft of the manuscript. JDE, MP and FW reviewed the manuscript and were involved in its critical revision prior to submission. All authors read, contributed to and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
All participants consented for publication of the results of the study. The Department of General Practice Human Ethics Advisory Group, University of Melbourne have been consulted for approval of publication.
Ethics approval and consent to participate
Ethics approval for this study was obtained from the University of Melbourne Human Research Ethics (HREC) Committee (HREC reference number: 1340491). All participants were recruited following an informed consent process involving approved plain language statements and consent forms.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Ferlay J, Soerjomataram I, Ervik M, Dikshit R, Eser S, Mathers C. al e: GLOBOCAN 2012 v1.0, Cancer Incidence and Mortality Worldwide: IARC CancerBase No. 11 [online]. In. Lyon, France; 2013.
- Burton H, Chowdhury S, Dent T, Hall A, Pashayan N, Pharoah P. Public health implications from COGS and potential for risk stratification and screening. Nat Genet. 2013;45(4):349–51.View ArticlePubMedGoogle Scholar
- Australian Cancer Network Colorectal Cancer Guidelines Revision Committee: Guidelines for the Prevention, Early Detection and Management of Colorectal Cancer. In: The Cancer Council Australia and Australian Cancer Network,. Sydney Australia; 2005.
- Ait Ouakrim D, Lockett T, Boussioutas A, Keogh L, Flander LB, Winship I, Giles GG, Hopper JL, Jenkins MA. Screening practices of Australian men and women categorized as “at or slightly above average risk” of colorectal cancer. CCC. 2012;23(11):1853–64.PubMedGoogle Scholar
- Ait Ouakrim D, Boussioutas A, Lockett T, Winship I, Giles GG, Flander LB, Keogh L, Hopper JL, Jenkins MA. Screening practices of unaffected people at familial risk of colorectal cancer. Cancer Prev Res. 2012;5(2):240–7.View ArticleGoogle Scholar
- Australian Institute of Health and Welfare: National Bowel Cancer Screening Program: monitoring report 2012–2013. In., vol. 2014, Cancer series 84. Cat. no. CAN 81. edn. Canberra, Australia; 2014.
- Usher-Smith J, Emery J, Hamilton W, Griffin S, Walter F. Risk prediction tools for cancer in primary care. Br J Cancer. 2015;113:1645–50.View ArticlePubMedPubMed CentralGoogle Scholar
- Walker J, Licqurish S, Chiang P, Pirotta M, Emery J. Cancer risk assessment tools in primary care: a systematic review of randomized controlled trials. Ann Fam Med. 2015;13(5):480–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M: Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008, 337(a1655).
- Campbell M, Fitzpatrick R, Haines A, Kinmonth AL, Sandercock P, Spiegelhalter D, Tyrer P. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321(7262):694–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Zuiderent-Jerak T, Forland F, Macbeth F. Guidelines should reflect all knowledge, not just clinical trials. BMJ. 2012;345(e6702).
- Win A, MacInnis R, Hopper J, Jenkins M. Risk prediction models for colorectal cancer: a review. Cancer Epidemiol Biomarkers Prev. 2012;21(3):398–410.View ArticlePubMedGoogle Scholar
- Freedman A, Slattery M, Ballard-Barbash R, Willis G, Cann B, Pee D, Gail M, Pfeiffer R. Colorectal cancer risk prediction tool for white men and women without known susceptibility. J Clin Oncol. 2009;27(5):686–93.View ArticlePubMedGoogle Scholar
- Emery J, Morris H, Goodchild R, Fanshawe T, Prevost A, Bobrow M, Kinmonth A-L. The GRAIDS trial: a cluster randomised controlled trial of computer decision support for the management of familial cancer risk in primary care. Br J Cancer. 2007;97(4):486–93.View ArticlePubMedPubMed CentralGoogle Scholar
- Trevena L, Zikmund-Fisher B, Edwards A, Gaissmaier W, Galesic M, Han P, King J, Lawson M, Linder S, Lipkus I. Presenting quantitative information about decision outcomes: a risk communication primer for patient decision aid developers. BMC Med Inform Decis Mak. 2013;13 Suppl 2:S7.View ArticlePubMedPubMed CentralGoogle Scholar
- Spiegelhalter D, Pearson M, Short I. Visualizing uncertainty about the future. Science. 2011;333(6048):1393–400.View ArticlePubMedGoogle Scholar
- Gigerenzer G, Edwards A. Simple tools for understanding risks: from innumeracy to insight. BMJ. 2003;741–744.
- May C, Finch T. Implementing, embedding, and integrating practices: an outline of normalization process theory. Sociology. 2009;43(3):535–54.View ArticleGoogle Scholar
- Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, Finch T, Kennedy A, Mair F, O’Donnell C. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med. 2010;8(1):63.View ArticlePubMedPubMed CentralGoogle Scholar
- Sociocultural Research Consultants ('SCRC') 2014: dedoose: Great Research Made Easy. Available from: http://www.dedoose.com [Accessed 10 Sept 2014].
- Australian Government Department of Health. Health assessment for people aged 45 to 49 years who are at risk of developing chronic disease. In: Note A27,. Edited by Medical Benefits Schedule. Australia; 2014.
- Holloway R, Wilkinson C, Peters T, Russell I, Cohen D, Hale J, Rogers C H. Cluster-randomised trial of risk communication to enhance informed uptake of cervical screening. Br J Cancer. 2003;53(493):620–5.
- Emery J, Morris H, Goodchild R, Fanshawe T, Prevost A, Bobrow M, Kinmonth A-L: The GRAIDS Trial: a cluster randomised controlled trial of computer decision support for the management of familial cancer risk in primary care. Br J Cancer 2007, 97(4):486-493
- Ruffin M, Nease D, Sen A, Pace W, Wang C, Acheson L, Rubinstein W, O’Neill S, Gramling R. Effect of preventive messages tailored to family history on health behaviors: the family healthware impact trial. Ann Fam Med. 2011;9(1):3–11.
- Chiang PP, Glance D, Walker J, Walter F, Emery J. Implementing a risk tool into general practice consultations: an exploratory study using simulated consultations with Australian general practitioners. B J Cancer. 2015;112:S77–83.
- Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. J Am Med Inform Assoc. 2004;11(2):104–12.
- Australian Institute of Health and Welfare 2013. Nursing and midwifery workforce 2012. National Health Workforce Series no. 6. Cat. no. HWL 52. Canberra: AIHW, Australia
- O’Hagan J, Davies L, Pears R. The use of simulated patients in the assessment of actual clinical performance in general practice. N Z Med J. 1986;99(815):948–51.