- Research article
- Open Access
Envisioning the future of clinical analytics: a modified Delphi process in New South Wales, Australia
BMC Medical Informatics and Decision Making volume 20, Article number: 210 (2020)
Clinical analytics is a rapidly developing area of informatics and knowledge mobilisation which has huge potential to improve healthcare in the future. It is widely acknowledged to be a powerful mediator of clinical decision making, patient-centred care and organisational learning. As a result, healthcare systems require a strategic foundation for clinical analytics that is sufficiently directional to support meaningful change while flexible enough to allow for iteration and responsiveness to context as change occurs.
In New South Wales, the most populous state in Australia, the Clinical Analytics Working Group was charged with developing a five-year vision for the public health system. A modified Delphi process was undertaken to elicit expert views and to reach a consensus. The process included a combination of face-to-face workshops, traditional Delphi voting via email, and innovative, real-time iteration between text re-formulation and voting until consensus was reached. The six stage process engaged 35 experts — practising clinicians, patients and consumers, managers, policymakers, data scientists and academics.
The process resulted in the production of 135 ideas that were subsequently synthesised into 23 agreed statements and encapsulated in a single page (456 word) narrative.
The visioning process highlighted three key perspectives (clinicians, patients and managers) and the need for synchronous (during the clinical encounter) and asynchronous (outside the clinical encounter) clinical decision support and reflective practice tools; the use of new and multiple data sources and communication formats; and the role of research and education.
New approaches to clinical analytics have the potential to transform healthcare delivery by reshaping clinical decision-making practices, influencing patient-provider interactions, altering patient trajectories and outcomes, and driving improvements in quality of care [1,2,3]. Realising this potential is not straightforward however, and requires strategic decisions about feasibility; preferred approaches to clinical and organisational transformation; and appropriate investment in information collection, digital technology and change management processes. In complex adaptive systems such as health where there is a vast array of options, priorities and perspectives, these types of strategic decisions are best guided by a shared and well informed vision of the future .
Many developed healthcare systems are seeking to capitalise on the promise of digital healthcare [4,5,6,7] and more specifically clinical analytics . In New South Wales (NSW), the most populous state in Australia, eHealth and data analytics are a key strategic priority .
The Clinical Analytics Working Group is a sub-committee of the NSW Health Analytics Steering Committee. The working group’s role is to advise on how analytics can be harnessed to support clinical and organisational decision making, leading to improved patient safety, quality and outcomes. It seeks to find solutions that will provide timely access to electronically captured data from clinical systems and other relevant sources; and transform it into meaningful information to better inform service delivery planning for effective and efficient clinical care delivery. One of the key tasks assigned to the working group is to develop and articulate a five-year vision for clinical analytics for NSW Health. This paper describes how, through a series of workshops and a modified Delphi process, this has been achieved.
What is meant by clinical analytics?
There is considerable ambiguity in the terminology used regarding clinical analytics – with terms such as analyses, analytics, health analytics, data analytics - used interchangeably but often with different meaning.
In this paper and in the Clinical Analytics Working Group, the term, clinical is used to denote the examination and treatment of patients – it refers to patient-provider interactions and spans preventive, diagnostic, therapeutic and supportive care. Data analytics relates to automated processes that produce information from raw data. As a concept, it is distinct from data analysis which is purposeful and sometimes iterative interrogation of data to produce information . Clinical analytics relates to the automated processes that produce clinically-relevant information to support clinician-mediated decisions, patient-mediated decisions and shared decisions . Clinical analytics utilises data analytics and data analyses.
Clinical analytics and clinical decision support
Clinical decision support represents one of the most powerful applications of clinical analytics, and aims to make data about a patient easier to access or more apparent; and to foster problem solving and guide action by users. Defined as “the use of information and communication technologies to bring relevant knowledge to bear on the healthcare and wellbeing of a patient,” users of clinical decision support include doctors, nurses, scientists and technical staff, allied health professionals, pharmacists, patients and carers (: p8).
Synchronous clinical decision support most frequently provides standardised information, triggered by a particular parameter such as a pharmacy order or a set of laboratory results and in the form of real-time pop ups, dialogue boxes and advice. When provided in an alert format, this type of decision support can be disruptive of workflow and experience to date has shown that a high proportion of alerts are ignored or overridden [13, 14].
Asynchronous clinical decision support provides feedback after an index decision is made. It provides opportunities for aggregation and comparisons with other providers, as well as time series analyses. It can provide a trigger for reflective practice and opportunities to consider and possibly reset heuristics.
Why a five-year vision?
The articulation of a five-year vision is needed to harness the potential of clinical analytics in improving healthcare - supporting clinical decision making, patient-centred care and organisational learning. The objectives for the process are to build a shared understanding of the potential opportunities of clinical analytics; unearth underlying values and understanding held by stakeholders which can be leveraged to coordinate and guide improvement efforts in an integrated way; and highlight common barriers and levers for change.
The Working Group agreed on two key parameters for the vision at the outset of the process. First, that a tripartite perspective was needed - giving consideration to the potential benefits and implications for clinicians, patients and managers. Second, that the five-year time horizon should encapsulate projects or initiatives that could be completed in that time frame; as well as more ambitious projects that will be developed and piloted in the 5 year period, but require an extended implementation period.
Method - building consensus through a modified Delphi process
A Delphi method is an iterative process in which a group of expert stakeholders come to a structured consensus view on a particular topic . Delphi processes have been successfully conducted to establish research priorities for a range of different topic areas such as identifying challenges for coordination in healthcare services , determining the future organisation of thrombectomy services in England , agreeing research priorities for patient safety in mental health  and for emergency medicine physicians in Australia and New Zealand .
The Delphi method, originally developed in the 1950s, entails engaging a group of experts - generally through a number of rounds where statements or options for the issue under consideration are circulated; feedback is elicited, collated and analysed; and the resulting syntheses or views are re-circulated to the group for further refinement and ratification . The goal is to reduce the range of responses and arrive at something close to expert consensus.
In the classic Delphi process there is no interaction between experts and researchers. We undertook a modified Delphi which incorporated a face to face workshop, an online survey and feedback, and real-time iteration between anonymised redrafting of text and voting using Sli.do software. Such modifications are frequently used to more fully explore context, enablers and barriers for change [21,22,23]. These processes sought first to elicit the breadth of experts’ views of the future; and second to establish a convergence of opinion of a likely five-year vision. As a secondary outcome, the process also highlighted consistently identified barriers and levers for change.
The process used to develop the five-year vision for clinical analytics is summarised in Fig. 2.
Six stage process to develop the vision
Stage 1: Identify expert panel
The Working Group agreed that experts would be eligible for inclusion in the Delphi process if they were: active in NSW as: a practising clinician, with an interest in data analytics; a data scientist with 2+ years’ experience using health data; a patient and / or consumer representative with an interest in health data; a senior policymaker or manager with responsibility for data analytics or eHealth strategy or; an academic with strong research record in data analytics in health.
In August 2018, invitations were sent by email to 35 experts; and 32 agreed to participate in the visioning exercise (91% response rate). The characteristics of the groups are summarised in Table 1.
Stage 2: Pre-meeting identification of thematic areas
An initial set of thematic areas was generated from Working Group deliberations, and supplemented by a rapid review of peer reviewed and grey literature. Search terms were “clinical analytics”; “horizon scanning”; “data analytics”. Results from PubMed searches were:
Search ((“strategic vision” AND “last five years”[PDat))) OR “horizon scan” 153 items found.
Search ((“data analytics” [Title/Abstract]) OR (“clinical analy*”[Title/Abstract]) OR “strategic vision” OR “horizon scan” AND “last five years”[PDat])) 117 items found.
Articles were screened for key topics to be used as catalysts for discussion and consideration. Data were extracted into a thematic table.
Six key areas of healthcare where clinical analytics could have a pivotal role in the future were identified. They were:
The synchronous provision of data analytics to support point of care decisions
Asynchronous provision of data analytics to support reflective practice
Clinical analytics for the purpose of guiding and supporting a system response
A system-wide and systematic approach to assessing and investing in data analytics
Integration of analytics across data sources (administrative, patient reported measures, clinical registry and primary care) to provide a better assessment of outcomes from a multimorbidity perspective
Using clinical analytics for knowledge generation and transfer through research and education
Stage 3: Face to face workshop
A workshop in September 2018 brought together 32 experts, and was run by expert facilitators from the implementation team of the NSW Agency for Clinical Innovation. Data were collected through physical artefacts; video; and contemporaneous note taking.
Participants were asked to perform three key tasks. First, to articulate what they considered to be key elements of the 5 year vision in each of the six thematic areas. They were asked to consider what could be accomplished within three time periods: achieved in 5 years’ time; pilots in 5 years’ time; either initiatives in train OR strategy / plans in place in 5 years’ time.
The second key task was to consider a set of ‘I statements’ that addressed clinical analytics from a patient, consumer and carer perspective (Appendix 1). These statements were generated by England’s National Voices organisation – which draws on 160 health and social care charities – covering a diverse range of health conditions and communities, and connecting with the experiences of millions of people.
For the third key task, participants were asked to identify key barriers and enablers to achieving the vision for clinical analytics.
Stage 4: Delphi round 1
Two researchers collated and independently clustered the themes. Results were compared and any discrepancies resolved by discussion. This resulted in 23 statements, structured into clinician, patient and manager perspectives. These vision statements were circulated via email to experts, 35 responded. Experts were asked to select from the following options:
I accept this statement as formulated
I do not accept this statement
I propose a modification to the statement – change it to _____.
For 12 statements, > 80% of respondents accepted the text as formulated. These statements were incorporated into the final vision narrative. For the remaining 13 statements, suggestions for modification were collected.
Stage 5: Delphi round 2
A combined in-person and online meeting used real-time voting and redrafting processes via the Sli.do software. For each of the 13 statements to be considered, the original text was shown, the suggested modifications and a draft reformulation was reviewed. A round of anonymised voting ascertained whether consensus had been reached. If not, further redrafting occurred followed by another voting round. This iterative process continued until > 80% agreement was achieved.
Stage 6: Circulate final narrative statement
The final agreed narrative was circulated to all participants for final endorsement.
The workshop generated 135 statements across the six thematic areas. These statements were parsed into vision narratives within each of the stakeholder perspectives: clinician (C1-C6); patients (P1 –P3); and managers (M1 – M3) (see Appendix 2 for the statements, mapped to thematic areas and vision narratives).
During the face to face workshop, a sticker voting system was used to prioritise the statements and those with more than 10 votes are shown in Table 2.
One session of the face to face workshop elicited barriers and enablers. The results are tabulated in Appendix 3.
In the initial Delphi round conducted by email, the percentage of respondents who accepted the statements as formulated ranged from 55 to 100%. Of the 23 statements circulated 12 achieved > 80% agreement (Appendix 4). The remaining 13 statements proceeded to the next Delphi round.
The second Delphi round was a meeting with in-person and online participants who had their responses collected anonymously. The 90 min session saw all statements reach agreement, taking between 1 to 5 iterations (Appendix 5). No statements were rejected outright, and all were modified until consensus was reached. The final result of the visioning exercise was synthesised into a one-page narrative (Table 3).
Predicting the future is a fraught task. Doing so in areas technology plays a pivotal role is particularly challenging. In 1994, the view of an official in the UK’s Whitehall was that “email will never catch on”  – highlighting the danger of relying on a singular perspective in envisioning the future.
Our intention in developing a shared view of the future for clinical analytics is not to formulate a precise prediction of the future. Our goal is to draw on a range of perspectives and expertise so that NSW Health can be positioned for a range of future scenarios and enhance its ability to respond to changing needs, policy priorities and circumstances.
The modified Delphi process resulted in a concise picture of what we expect the clinical analytics landscape to look like in 2024. It also revealed a number of enablers and barriers that will affect the realisation of the vision.
Enablers and barriers
Deliberations about how to achieve the five-year vision focused on the need for frameworks, collaborative working and a high regard for patients’ perspectives. Experts emphasised the importance of collaboration in clinical analytics approaches between clinicians, and analysts and data specialists; the breaking down of silos. At the same time, they acknowledged the importance of delivering on patients’ expectation that their data are used to drive improvement. A key enabler was seen to be a framework that captures the principles and real world practice of clinical decision making – to act as a guide for the type of analyses, alerts; and reporting that will secure behaviour change and quality and safety improvement; and will minimise cognitive load on clinicians.
There was a shared imperative to be ‘realistic’ in expectations, enthusiasm for the Pareto principle or ‘80–20’ rule – emphasising that the development of clinical analytics should be guided by a ‘satisficing’ approach  hat does not seek perfection nor complete accuracy. Sandbox initiatives and capacity for experimentation and experiential learning about potential solutions and ‘what works’ were regarded to be highly desirable.
Reflecting on clinical decision support
The modified Delphi process placed significant focus on how clinical analytics can support clinical decision making and the deliberations of the group prompted us to develop a conceptual model that considers four key areas (Table 4) .
There are several limitations with this study. First, in terms of expert recruitment, our Delphi panel was founded on a pre-existing group that had been convened to advise NSW Health on clinical analytics. While we supplemented this group, patients were under-represented. Second, the expert panel varied in size and composition over the three rounds of activity. For the idea generation phase, the core group was supplemented with targeted invitations to bridge potential gaps in coverage of key stakeholders, interest groups and knowledge. For the Delphi rounds, participation rates varied with 53% of the core group voting in all rounds and several substitutions of organisational representatives between the rounds. Third, for many of the vision statements we did not reach 100% consensus across the group. Given the nature of the question and the call for speculation about the future inherent in the process, we agreed that 80% agreement would provide sufficient precision to allow us to conclude that this was a shared vision. Future work aims to overcome the limitation of under-representation of some groups – through broader engagement through crowdsourcing techniques.
Clinically led improvement, enabled by new technology and analytic capacity, is transforming the delivery of healthcare and our management of population health. Yet strategic decisions about the scale of clinical transformation and associated investment in information, analytics, and digital technology are often reactive, destined to play catch up and make do .
Perhaps this reticence is based on the assumption that predicting the future is something of a fool’s errand. Yet forethought and foresight are absolutely essential if we are to harness to potential of innovation in healthcare. The trick is to be sufficiently prescient to be prepared for both expected and unexpected gains and consequences of profound change.
Availability of data and materials
All data are provided in Appendices.
Electronic Medical Record
New South Wales
Patient Reported Measures
Commonwealth Scientific and Industrial Research Organisation. Future of Health. Canberra: CSIRO; 2018.
National Institutes of Health. All of Us – Precision Medicine Initiative. Washington DC: NIH; 2017.
Song TM, Ryu S. Big data analysis framework for healthcare and social sectors in Korea. Healthc Inform Res. 2015;21(1):3–9.
Imison C, Castle-Clarke S, Watson R, Edwards N. Delivering the Benefits of digital healthcare. London: The Nuffield Trust; 2016.
Australian Digital Health Agency. Framework for Action: How Australia will deliver the benefits of digitally enabled health and care. ADHA: Canberra; 2017.
NHS Digital [online], Innovative uses of data and data science. https://digital.nhs.uk/data-and-information/data-insights-and-statistics/innovative-uses-of-data-and-data-science Accessed 2 Jan 2019.
Canada Health Infoway, https://www.infoway-inforoute.ca/en/ Accessed 2 Jan 2019.
Australian Medical association. Call for a clinical analytics unit to drive policy and evaluation [online] https://ama.com.au/sa/focus-clinical-analytics Accessed 2 Dec 2019.
NSW Health. NSW Health Analytics Framework: transformed health through data and insights. Sydney: NSW Ministry of Health; 2017.
Womack DM, Kennedy R, Bria B. Current practices in clinical analytics: a hospital survey report. NI 2012 (2012). 2012;2012:458.
An JY. Healthcare analytics for quality and performance [book review]. Healthc Inform Res. 2013;19(4):324–5.
Sanders D. Data driven decision-support https://www.slideshare.net/dalesanders1/data-driven-clinical-quality-and-decision-support Accessed 16 Sept 2019.
Santucci W, Day R, Baysari M. Evaluation of hospital-wide computerised decision support in an intensive care unit: an observational study. Anaesth Intensive Care. 2016;44(4):507–12.
The Lancet Oncology. Clinical decision making: more than just an algorithm. Lancet Oncol. 2017;18(12):1553.
Hsu CC, Sandford BA. The Delphi technique: making sense of consensus. Pract Assess Res Eval. 2007;12(10):1–8.
Holen-Rabbersvik E, Eikebrokk TR, Fensli RW, Thygesen E, Slettbo A. Important challenges for coordination and inter-municipal cooperation in health care services: a Delphi study. BMC Health Serv Res. 2013;13(1):451.
Halvorsrud K, Flynn D, Ford G, et al. A Delphi study and ranking exercise to support commissioning services: future delivery of thrombectomy services in England. BMC Health Serv Res. 2018;18:135.
Dewa L, Murray K, Thibaut B, Ramtale S, Adam S, Darzi A, Archer S. Identifying research priorities for patient safety in mental health: an international expert Delphi study. BMJ Open. 2018;8:3.
Deane H, Wilson C, Babl F, et al. PREDICT prioritisation study: establishing the research priorities of paediatric emergency medicine physicians in Australia and New Zealand. Emerg Med J. 2018;35(1):39–45.
Helmer-Hirschberg O. Analysis of the Future: The Delphi Method. California: RAND Corporation; 1967.
Lim M, Thompson B, D’Silva C, et al. Development and reliability of an appropriateness and prioritization instrument for eye care practice: a modified Delphi process. Ophthalmic Epidemiol. 20019;27(1):19–28.
Eubank BH, Mohtadi NG, Lafave MR, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol. 2016;16:56.
Schneider P, Evaniew N, Rendon JS, on behalf of The PARITY Investigators, et al. Moving forward through consensus: protocol for a modified Delphi approach to determine the top research priorities in the field of orthopaedic oncology. BMJ Open. 2016;6:e011780.
Davies C, Bowcott O, Greenfield P. Files show John Major’s aides viewed emails as passing fad. The Guardian https://www.theguardian.com/uk-news/2018/dec/28/national-archives-john-majors-aides-emails-were-passing-fad Accessed 16 Sept 2019.
Simon H. Rational Choice and the Structure of the Environment. Psychol Rev. 1956;63(2):129–38.
Greenes R. Definition, Scope and Challenges IN: Clinical Decision Support: the road to broad adoption 2nd ed. London: Academic Press; 2014.
No external funding.
Ethics approval and consent to participate
This work was realised as part of clinical engagement processes related to the development of a clinical analytics strategy in New South Wales. It was not part of an externally-funded research project and was done under the auspice of the regularly audited programmes of the Agency for Clinical Innovation. Therefore, no research ethics approval was sought for this project conducted as part of system management activities.
Consent for publication
The authors are all full-time employees of NSW Health. They have no other financial interests to declare.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Reflecting on I Statement. Appendix 2. Clinical analytics fiver-year statements (original thematic areas and parsed to final narrative one-pager). Appendix 3. Barriers and enabler identified in workshop. Appendix 4. Online Survey. Appendix 5. Final Delphi
About this article
Cite this article
Sutherland, K., Yeung, W., Mak, Y. et al. Envisioning the future of clinical analytics: a modified Delphi process in New South Wales, Australia. BMC Med Inform Decis Mak 20, 210 (2020). https://doi.org/10.1186/s12911-020-01226-7
- Clinical analytics