Skip to main content
Download PDF

An informatics consult approach for generating clinical evidence for treatment decisions

BMC Medical Informatics and Decision Making volume 21, Article number: 281 (2021) Cite this article

Abstract

Background

An Informatics Consult has been proposed in which clinicians request novel evidence from large scale health data resources, tailored to the treatment of a specific patient. However, the availability of such consultations is lacking. We seek to provide an Informatics Consult for a situation where a treatment indication and contraindication coexist in the same patient, i.e., anti-coagulation use for stroke prevention in a patient with both atrial fibrillation (AF) and liver cirrhosis.

Methods

We examined four sources of evidence for the effect of warfarin on stroke risk or all-cause mortality from: (1) randomised controlled trials (RCTs), (2) meta-analysis of prior observational studies, (3) trial emulation (using population electronic health records (N = 3,854,710) and (4) genetic evidence (Mendelian randomisation). We developed prototype forms to request an Informatics Consult and return of results in electronic health record systems.

Results

We found 0 RCT reports and 0 trials recruiting for patients with AF and cirrhosis. We found broad concordance across the three new sources of evidence we generated. Meta-analysis of prior observational studies showed that warfarin use was associated with lower stroke risk (hazard ratio [HR] = 0.71, CI 0.39–1.29). In a target trial emulation, warfarin was associated with lower all-cause mortality (HR = 0.61, CI 0.49–0.76) and ischaemic stroke (HR = 0.27, CI 0.08–0.91). Mendelian randomisation served as a drug target validation where we found that lower levels of vitamin K1 (warfarin is a vitamin K1 antagonist) are associated with lower stroke risk. A pilot survey with an independent sample of 34 clinicians revealed that 85% of clinicians found information on prognosis useful and that 79% thought that they should have access to the Informatics Consult as a service within their healthcare systems. We identified candidate steps for automation to scale evidence generation and to accelerate the return of results.

Conclusion

We performed a proof-of-concept Informatics Consult for evidence generation, which may inform treatment decisions in situations where there is dearth of randomised trials. Patients are surprised to know that their clinicians are currently not able to learn in clinic from data on ‘patients like me’. We identify the key challenges in offering such an Informatics Consult as a service.

Peer Review reports

Background

Evidence informing treatment decisions traditionally takes years to generate and leaves many clinical uncertainties unaddressed [1]. Especially in patients with two or more conditions (multimorbidity), it has been hard to generate evidence tailored to ‘patients like me’ and embed this evidence in clinical decision making using electronic health records. There are thousands of clinical practice recommendations, but only a small proportion (15–20%) of these recommendations are supported by level A (trial) evidence [1,2,3,4,5,6]. A systematic review of trial registration records found that 79% of randomised controlled trials (RCTs) excluded patients with concomitant chronic conditions [6]. The United States Food and Drug Administration, Medicines Healthcare Regulatory Authority and European Medicines Agency are increasingly recognising the role of real-world evidence [7] but guidance thus far has not considered its near real-time generation.

The Informatics Consult concept has been proposed [8,9,10,11] to produce on-demand evidence in which clinicians request novel evidence based on the care of prior patients to inform the treatment of a specific patient, with return of results in decision-relevant clinical timescales. For example, a hepatologist seeing a patient with cirrhosis in the clinic learns that they have developed atrial fibrillation. What evidence is, or could rapidly be, available to inform a decision on anti-coagulation for stroke prevention? This is an example of a treatment indication and a treatment contra-indication coexisting in the same patient. Initial experience of the Informatics Consult from Stanford University, has focussed on questions of prognosis (rather than treatment decisions) and highlighted analytical and scaling challenges in returning results [11]. However, demonstrators of the Informatics Consult for treatment decisions are lacking.

Our objective was to demonstrate proof of concept of the Informatics Consult using the atrial fibrillation-cirrhosis-warfarin example. Specifically, we sought to (1) develop prototype electronic health record forms for requesting an Informatics Consult and return of results, (2) generate four sources of evidence for an Informatics Consult (evaluate available RCT evidence, meta-analyse prior observational studies, emulate a target trial using electronic health records, and Mendelian randomisation [12, 13]), (3) for each form of evidence, to identify steps necessary for automation to accelerate evidence generation and return of results to clinicians, and scale across multiple exemplars (Additional file 1-1) and (4) explore clinician acceptability of the Informatics Consult.

Methods

Developing prototype electronic health record (EHR) request form and report form

An Informatics Consult is triggered by a request made by a clinician from the electronic health records (EHR) for multiple novel sources of evidence. We developed prototypes for the request form and the report form in consultation with the Chief Clinical Research Informatics Officer and clinicians (cardiologists, hepatologists and clinical pharmacologists). We illustrate possible forms based on an EHR platform, but similar design principles are relevant in EHRs from other vendors, and in other settings, including primary care.

Retrieving information on currently recruiting trials

We searched for previously reported and currently recruiting trials on anticoagulants in patients with atrial fibrillation, with stroke or mortality in the primary outcome using the ClinicalTrials.gov registry. We then used the 8-digit National Clinical Trial numbers to retrieve detailed information on inclusion and exclusion criteria for each trial to identify if any trials included patients with both atrial fibrillation and cirrhosis.

Meta-analysis of prior observational studies

Study identification We searched PubMed for peer-reviewed articles using the keywords “antithrombotic”, “anticoagulant”, “warfarin”, “cirrhosis” and “atrial fibrillation”. We considered eligible studies as those reporting the effects of anticoagulation therapy in patients with both liver cirrhosis and atrial fibrillation. We excluded reviews, single case reports, editorials and small case series (< 10 cases). Data extraction We extracted the following variables: author, setting, eligibility criteria, number of patients with atrial fibrillation and cirrhosis, number of patients in treated and untreated groups and summary measures. Analyses were performed following PRISMA guidelines. Outcomes of interest were mortality and ischaemic stroke. Statistical analysis A meta-analysis of associations was performed by pooling hazard ratios (HRs) or odds ratios (ORs) depending on data availability from observational studies using DerSimonian and Laird random-effects models. We also performed leave-one-out sensitivity analyses.

Target trial emulation

We used population-based EHRs to perform a target trial emulation, which is the application of design principles from RCTs to inform analyses on observational data [14, 15]. We obtained informational governance approval from the Medicines Healthcare Regulatory Authority (UK) Independent Scientific Advisory Committee (20_078R) to analyse the Clinical Practice Research Datalink (CPRD) linked to secondary care Hospital Episode Statistics and the Office for National Statistic death registration. The study population was 3,854,710 adults aged ≥ 30 years. Phenotype definitions for atrial fibrillation, cirrhosis and other conditions included as baseline covariates as well as definitions for prescriptions are available at https://caliberresearch.org/portal and have previously been validated [16, 17]. Phenotypes for primary care records were generated using Read clinical terminology (version 2). Phenotypes for secondary care records were generated using ICD-10 terms.

We developed a target trial protocol where eligibility criteria, treatment assignment, treatment strategy, follow-up period, causal contrast and statistical analyses were specified (Additional file 1-2). Each component of the trial protocol is matched as closely as possible to the design of a randomised trial with minor modifications to accommodate the use of observational data. We employed the intention-to-treat effect as a causal contrast, which was warfarin initiation versus no initiation at baseline. To emulate a target trial, we ensured that individuals are classified as warfarin initiators versus non-initiators at baseline (i.e., using baseline information to assign baseline treatment status). The baseline is defined as the latest date by which a patient has both cirrhosis and atrial fibrillation given that all eligibility criteria are met. As we were interested in assessing the effects of warfarin use on stroke, we have also excluded prevalent cases of ischaemic stroke. Individuals were followed until the development of an outcome of interest, which were all-cause mortality and incident ischaemic stroke. Propensity score matching (PSM) analyses were performed by matching the warfarin initiator and non-initiator groups. PSM was performed using the nearest-neighbour matching method (a 1:3 match was performed where possible) with a calliper width of 0.2 of the standard deviation of the logit of the propensity score. The PSM cohort was subjected to analyses of all-cause mortality and incident stroke using the Kaplan–Meier and logrank test method and the Cox proportional hazard regression model. As patients and clinicians would be interested in understanding risk in specific demographic categories, we performed subgroup analyses for all-cause mortality in patients aged ≤ 65, aged > 65, men, women and in patients with normal international normalise ratio (INR) measurements.

Genetic evidence: two-sample Mendelian randomisation (MR)

As an example of drug target validation in the general population, we performed MR to investigate the causal relationship between warfarin use and stroke risk. Vitamin K1 (phylloquinone) is a central component in the production of blood coagulation factors. Warfarin (a vitamin K antagonist) inhibits the activity of vitamin K epoxide reductase to interfere with the recycling of vitamin K and to reduce blood clotting. We considered four single nucleotide polymorphisms (SNPs) that predict circulating phylloquinone (vitamin K1) selected from a genome-wide meta-analysis study on Europeans [18]. Four SNPs, all on separate chromosomes, were selected as they had the strongest association with circulating phylloquinone: rs2108622 (chromosome 19), rs2192574 (chromosome 2), rs4645543 (chromosome 8) and rs6862071 (chromosome 5). We retrieved genome wide association study (GWAS) summarised data for stroke outcomes from the MEGASTROKE study [19]. MR was performed using the “MendelianRandomisation” package in R [20]. We explored four methods for MR; inverse-variance weighted (IVW), MR-Egger, simple median and weighted median.

Potential for automating the informatics consult

We tasked a review panel with expertise in health informatics, epidemiology, evidence synthesis and computer science (drawn from co-authors) with two questions: (1) What are the most time-consuming tasks for each of the 4 streams of evidence identification and generation? (2) What automation opportunities might be important for acceleration and scaling.

Clinician acceptability of the informatics consult

After the initial development of the Consult, to gain insights into the acceptability and feasibility of the Informatics Consult, we conducted a pilot survey with an independent sample of 34 clinicians who had not taken part in the research.

Results

Prototype EHR request form

Clinicians wanted the request form (Fig. 1) to include a succinct free text statement of the clinical question, auto propagation of the diagnosis combination from the patient’s EHR, and suggested structured treatment, efficacy and safety outcomes, based on inputs from the survey. Some clinicians wanted to further specify eligibility criteria for target trial emulation. Interestingly clinicians wanted not only evidence directly related to treatment effectiveness, but also requested national prevalence estimates of cirrhosis and atrial fibrillation, prognosis (1-year mortality), and current treatment variation (proportion of warfarin initiators).

Fig. 1
figure 1

Informatics Consult Electronic health record request form prototype

Full size image

Prototype EHR report

We provide an overall summary report based on new evidence on warfarin use (lower all-cause mortality and lower stroke risk) in patients with atrial fibrillation and cirrhosis (Fig. 2). Through the Consult report, clinicians will have the opportunity to queue patients for RCTs where relevant. We summarise evidence on prevalence, 1-year background mortality risk, meta-analysis of observational studies, target trial emulation results and genetic evidence. The detailed results included in the report for each form of evidence were provided below.

Fig. 2
figure 2

Informatics Consult Electronic health record report prototype

Full size image

Reported and currently recruiting randomised trials

We did not identify any previous reported RCTs on anticoagulants relevant to patients with cirrhosis. We identified four currently recruiting anticoagulant trials. All four trials reported exclusion criteria related to liver cirrhosis such as contraindication to anticoagulation, i.e., hepatic impairment and elevated liver function tests (Fig. 3).

Fig. 3
figure 3

Trial evidence and currently recruiting trials of anticoagulation in patients with atrial fibrillation and cirrhosis to reduce stroke risk. A Clinical question and summary of trial evidence. B Previously completed and currently recruiting randomised trials evaluating anticoagulants and stroke outcomes have exclusion criteria related to cirrhosis

Full size image

Meta-analysis of observational studies on warfarin use and ischaemic stroke

We identified 142 articles from PubMed, of which only 4 observational studies remained eligible on full-text review, and were included in the meta-analysis [21,22,23,24] (Fig. 4A). The pooled hazard ratio (HR) of warfarin use in patients with atrial fibrillation and cirrhosis on stroke was 0.71, 95% confidence interval [CI] (CI = 0.39–1.29), with high heterogeneity between studies I2 = 73% (Fig. 4B). We also performed the leave-one-out sensitivity analysis with HRs ranging from 0.54 (CI = 0.30–1.00) to 0.92 (CI = 0.55–1.54). Only 1 study reported warfarin use and all-cause mortality and found a lower mortality risk (HR = 0.65; = 0.55–0.76) [21].

Fig. 4
figure 4

New synthesis of prior observational evidence. Meta-analysis of the association between warfarin use and the risk of ischaemic stroke in observational studies including approaches for automation. A Characteristics of observational studies included in the meta-analysis. B Forest plot depicting the hazard ratios calculated with the DerSimonian and Laird random-effects models. HR = hazard ratio; CI = confidence interval; SE = standard error

Full size image

Target trial emulation using population-based EHRs

Per the target trial protocol (Additional file 1-2), a cohort encompassing 1022 individuals fulfilling all eligibility criteria was created (initiators = 443; non-initiators = 579). We performed PSM on 22 baseline covariates and generated a matched cohort involving 235 initiators and 526 non-initiators (Fig. 5A), baseline patient characteristics before and after PSM are shown in Additional file 1-3. We estimated an intention-to-treat HR for all-cause mortality of 0.61 (CI = 0.49–0.76; p < 0.0001) comparing warfarin initiators with non-initiators (Fig. 5B). Warfarin used was associated with lower risk of ischaemic stroke: HR = 0.27 (CI = 0.08–0.91, p = 0.034) (Fig. 5B). The 1022 eligible participants for the target trial were categorised into the five subgroups (aged ≤ 65, aged > 65, men, women and patients with normal INR), followed by PSM (Additional file 1-4; Additional file 1-5; Additional file 1-6; Additional file 1-7; Additional file 1-8; Additional file 1-9). Intention-to-treat HRs for all-cause mortality comparing warfarin initiators versus non-initiators were as follow: aged ≤ 65: HR = 0.62 (0.45–0.86, p = 0.0041); aged > 65: HR = 0.61 (0.46–0.83, p = 0.0015); men: HR = 0.64 (0.49–0.84, p = 0.0014); women: HR = 0.53 (0.37–0.77, p = 0.00087) and normal INR of < 1.7: HR = 0.62 (0.50–0.78, p < 0.0001), where warfarin therapy is associated with lower mortality risk (Additional file 1-4).

Fig. 5
figure 5

New observational evidence through target trial emulation (intention-to-treat analysis) where eligibility and treatment assignment were aligned with time zero of follow-up, as is done in randomised controlled trials. A CONSORT diagram showing the selection of eligible individuals for the target trial emulation of anticoagulation therapy in patients with atrial fibrillation and cirrhosis. B Kaplan–Meier plots of the propensity-matched cohort for all-cause mortality and ischaemic stroke. Flow diagram depicts analysis design. P values from logrank tests were indicated. Hazard ratios from Cox proportional hazards regression analyses were indicated. Numbers in parentheses indicate the 95% confidence intervals

Full size image

Genetic evidence

We found no GWAS summary data for vitamin K1 in patients with atrial fibrillation and cirrhosis, but we did find genetic evidence to indicate that warfarin use is associated with reduced stroke risk. For two-sample MR, we used four methods (inverse-variance weighted (IVW), MR-Egger, simple median and weighted median). The IVW analyses, which assumes no pleiotropy, revealed that higher genetically predicted levels of vitamin K1 were associated with a higher risk of any stroke with an odds ratio (OR) of 1.06 (95% CI 1.00–1.11) per Ln-nmol/L increase in vitamin K1 (Fig. 6). However, these results were not replicated using methods (simple median, weighted median and MR-Egger) which allow for genetic pleiotropy. When considering stroke subtypes, we observed that higher genetically predicted levels of vitamin K1 were associated with a higher risk of large artery atherosclerotic stroke for 3 out of 4 MR methods: simple median (OR = 1.25 [1.03–1.51]); weighted median (OR = 1.25 [1.03–1.52]) and IVW (OR = 1.29 [1.11–1.50]) (Fig. 6).

Fig. 6
figure 6

Genetic evidence. Two-sample Mendelian randomisation on circulating vitamin K1 levels and risk of stroke. *Indicate significant results

Full size image

Informatics consult report generation

Table 1 shows additional opportunities for pipelining each of the four streams of evidence identification and generation to return the Consult report. Computable EHR phenotypes and computable clinical trial protocols can be used to automate the process of trial identification and trial recruitment [25, 26]. For meta-analyses of observational studies, approaches for semi-automated systematic reviews [27, 28], batch extraction of data from articles [29, 30] and mapping of SNOMED-CT terms to MeSH descriptors in PubMed [31] can be used in the pipelining process. The DExtER tool can be used for automatic extraction of EHR databases and automated cohort creation for the target trial emulation process [32]. Recent addition of an analytical module to DExtER enables cohort creation, statistical analyses and results visualisation within short time scales for matched cohort studies. Two-sample MR to generate genetic evidence can be automated using the MR-Base platform [33], which returns results within minutes.

Table 1 Traditional versus automated approaches for evidence synthesis
Full size table

Clinician survey on the acceptability and feasibility of the Informatics Consult

We surveyed an independent sample of 34 clinicians from eight specialties with results shown in (Additional file 1-10, Additional file 1-11). Results indicated that 79% of clinicians thought that they should have access to the Informatics Consult as a service within their healthcare systems (21% responded ‘maybe). Clinicians found each section of the report useful (or ‘maybe useful’) as follows: prognosis [85% useful (12% maybe useful)], a summary of evidence of efficacy and safety [79% useful (21% maybe useful)] and disease prevalence [68% useful (18% maybe useful)]. Only 18% of clinicians thought that they needed to have access to the details of the evidence in the clinic with one clinician stating: “Multi-disciplinary team meetings might benefit from Informatics Consults, this is where difficult cases are discussed and there is time to review newly generated evidence”. When asked whether clinicians found detailed reports on the four sources of evidence useful (or ‘maybe useful’), responses were as follow: randomised trials: 82% useful (15% maybe useful); meta-analysis of prior observational evidence: 76% useful (18% maybe useful); target trial emulation: 62% useful (18% maybe useful) and genetic evidence: 26% useful (24% maybe useful). 74% of clinicians would discuss the Informatics Consult report with their patients (26% answered maybe). Clinicians offered 15 further clinical questions where the Informatics Consult might be of value and made additional comments including: “Ultimately, we practise defensive medicine—would my decision stand up in court based on available data—the Informatics Consult should help with that.” and “Regulators and guideline developers will require replication and quality assurance of evidence generated in clinical timescales”.

Discussion

We demonstrate that the Informatics Consult offers a novel paradigm to generate new clinical evidence. We found that in patients with atrial fibrillation and cirrhosis, initiation of warfarin was common, may be associated with lower all-cause mortality and may be effective in lowering stroke risk. Given the ubiquity of clinical uncertainty where there is little or no evidence, and that current modes of generating new evidence may never be initiated or, if initiated, take years to report, it is imperative to accelerate learning from extant data.

Informatics consult versus traditional approaches for evidence generation and delivery

The Informatics Consult puts the treating clinician and the patient at the centre of evidence generation. Indeed, in seeking to address a range of questions from the clinician and patient, the Informatics Consult, if automation is plausible, could enable the simultaneous delivery of evidence from different sources, rather than employing a one-study-one-design-at-a-time approach. The Consult is embedded within EHR system—making it a form of an electronic consult, which is increasingly being adopted to seek specialist input [34, 35]. Additional information on how the Informatics Consult differs from traditional approaches to evidence generation and use is summarised in Additional file 1-1.

Concordance across sources of evidence identified from the Consult

A primary motivation for requesting an Informatics Consult is to understand how a particular treatment influences an outcome to help guide decision making. We show a degree of concordance across four sources of evidence. RCTs demonstrate the effectiveness of oral anticoagulation in stroke prevention in patients with atrial fibrillation: but as we demonstrate reported and currently recruiting RCTs exclude patients with cirrhosis (i.e., patients having ‘contraindications’ for anticoagulants). This suggests that the prospect of ever mounting, or successfully recruiting to, an RCT in patients with AF and cirrhosis is low. Meta-analysis of observational studies and target trial emulation suggest evidence on the potential benefits of warfarin for stroke reduction, suggesting the significant impact on strokes and deaths averted if these patients are treated with anticoagulant therapy. Although we did not find any relevant GWAS summary data, evidence suggests that lower levels of vitamin K1 (target of warfarin) are associated with lower stroke risk which is corroborated by another study [36].

Returning results on prevalence, prognosis and treatment variation

Interestingly, the clinicians involved in this study recommended that information beyond efficacy and safety should be included as options for the clinician to request in the Informatics Consult. By providing information on the prevalence of cirrhosis and atrial fibrillation, we demonstrate that this pair of conditions is not highly prevalent (although not considered rare). Knowing that the health system has data on diagnosis, treatment and outcomes in an estimated 35,000 individuals (scaled up to the population of England) with both conditions, highlights the importance of several areas. First, the importance of accessing and learning from nationwide data at scale; any one clinician may have clinical experience of only a handful of such cases, and it is not feasible or scalable to create registries. Second, the clinician likely has never had access before to population-based, contemporary prognostic information on such patients (1-year mortality: 15%), nor the knowledge that in practice 43% of patients were started on warfarin. We anticipate that providing clinicians and patients with this information may stimulate further questions on generating new Consults on a wider range of prognostic outcomes, and predictors of prognosis.

Need and demand for informatics consult

Nearly all clinicians are faced with treatment decisions where limited evidence exists where insights might be gained from analysis of large-scale data on ‘patients like me’. Previous studies have employed EHR data for diagnostic consultations [37], and others have created clinical informatics consult services based on the medical literature [38]. But currently, few if any clinicians can request such insights. A clinician from the survey mentioned that evidence from the Informatics Consult is important to help their decision in practicing defensive medicine given that the majority of clinical practice recommendations in professional society guidelines are not supported by RCT evidence. There is limited information on the system-wide frequency of treatment 'clashes' where indication and contraindications coexist in the same patient. This is especially relevant for patients with multimorbidity where evidence from RCTs are limited [6], which may result in individuals being subjected to low-quality recommendations. For example, certain targeted cancer therapy such as angiogenesis inhibitors may cause an increase the prevalence of hypertension during treatment, which highlights the importance of considering the impact of cancer therapy on adverse side effects and cardiotoxicity [39]. Our preliminary analyses demonstrate that 26 in 10,000 women have breast cancer and hypertension; these individuals might benefit from the Informatics Consult.

Developing the informatics consult as a service

The majority of clinicians in our survey thought that they should have access to the Informatics Consult as a service as their healthcare systems seek to learn from existing data [40]. We have demonstrated the feasibility with this one example. Scaling to other clinical examples, and a service, requires five inter-related challenges to be addressed in clinical standards setting, implementation and evaluation, access to data, informatics, and knowledge management. First, bodies that define standards of care—including clinical practice guideline developers and regulatory and technology assessment bodies—are already considering real-world evidence generated in conventional timescales [7, 41]. In the context of a more rapid generation of evidence, the Informatics Consult raises new questions about replication, open peer review and quality assurance pipelines. Second, development, implementation and evaluation of the impact of the Informatics Consult on clinical decision making in practice are required. Feedback from our survey suggests that multidisciplinary team (MDT) meetings, which concentrate on ‘difficult’ clinical cases may present an opportunity [42]. Third, an Informatics Consult service requires approved, immediate access to large scale, clinically detailed updated data, most likely in a trusted research environment. The coronavirus pandemic offers a new precedent for such information governance approvals, which can not foresee the nature of the next clinician's question. Fourth, biomedical knowledge (in EHRs, trial protocols, clinical guidelines) needs to become more computable and interoperable. Design and analysis methods need to be improved, standardized and widely distributed. For example, the Observational Health Data Science and Informatics community offers open-source software implementing many routine analyses methods [43]. Fifth, there needs to be an open process of making the knowledge available in an Informatics Consult library. As the library builds, Clinicians might make a request for which a previous consult already provides an answer. The Library might also serve as a platform for connecting ‘patients like me’, registering the frequency of therapeutic dilemmas and potential treatment uncertainties, identifying the need for new RCTs, and informing their design and facilitate targeted recruitment into trials.

Strengths of the study

To the best of our knowledge, this is the first demonstration of Informatics Consult for a treatment decision, triangulation of evidence from meta-analyses of observational studies, target trial emulation using EHR data and MR. The breadth, depth and longevity (long follow-up period) of the EHR data, which links primary care, secondary care and the death registry is an advantage. Another significant strength is engagement with an independent sample of clinicians from multiple specialties to gauge the feasibility and acceptability of the Informatics Consult.

Limitations of the study

Our study has important limitations. First, although we have prototyped EHR request and report forms based on feedback from clinicians, these have not yet been implemented in live clinical systems. Second, as initial proof of concept, we have not assessed bleeding outcomes and newer non-vitamin K oral anticoagulants due to current data access limitations. Third, it remains difficult to assess whether there unmeasured confounding despite employing causal inference methods. It is also possible that some prescription data is missing, which means that patients are incorrectly assigned to the control arms. Future analyses on a more recent and larger dataset involving both CPRD Gold and Aurum [44] would be beneficial. Fourth, a limitation of the MR analyses is that access to summarised GWAS data for stroke outcomes in patients with cirrhosis is limited. The advantages of using publicly available GWAS summarised data are speed and transparency, both of which are essential to the Informatics Consult. Although individual-level data would allow more flexibility to conduct analyses in specific patient subgroups and to select which variables to generate the summarised data for, such analyses could not be returned within a clinical timescale that is not scalable and cannot be fully automated. Fifth, although we have identified potential ways to automate the analytic process, these have not been implemented here.

Conclusion

We proposed an Informatics Consult framework to summarize evidence from four sources and have developed a report prototype for answering a treatment question to enable new ways of data-informed decision making. The Informatics Consult may stimulate a conversation among public, professionals and policymakers about more rapidly realising the benefits of health system learning from ‘patients like me’.

Availability of data and materials

Codelists for the conditions included in our study are available on the CALIBER portal (https://www.caliberresearch.org/portal) and can be downloaded in a machine-readable format. The data used in this study are available on successful ethics application to the Clinical Practice Research Datalink (CPRD). All summarised data and results are made available as supplementary materials.

Abbreviations

AF:

Atrial fibrillation

RCT:

Randomised controlled trials

HR:

Hazard ratio

EHR:

Electronic health record

OR:

Odds ratio

CPRD:

Clinical Practice Research Datalink

PSM:

Propensity score matching

INR:

International normalise ratio

SNP:

Single nucleotide polymorphism

GWAS:

Genome wide association study

IVW:

Inverse-variance weighted

CI:

Confidence interval

MR:

Mendelian randomisation

MDT:

Multidisciplinary team

References

  1. van Dijk WB, Grobbee DE, de Vries MC, Groenwold RHH, van der Graaf R, Schuit E. A systematic breakdown of the levels of evidence supporting the European Society of Cardiology guidelines. Eur J Prev Cardiol. 2019;26:1944–52.

    Article  Google Scholar 

  2. Meyer C, Bowers A, Wayant C, Checketts J, Scott J, Musuvathy S, et al. Scientific evidence underlying the American College of Gastroenterology’s clinical practice guidelines. PLoS ONE. 2018;13:e0204720.

    Article  Google Scholar 

  3. Koh C, Zhao X, Samala N, Sakiani S, Liang TJ, Talwalkar JA. AASLD clinical practice guidelines: a critical review of scientific evidence and evolving recommendations. Hepatology. 2013;58:2142–52.

    Article  Google Scholar 

  4. Fanaroff AC, Califf RM, Windecker S, Smith SC, Lopes RD. Levels of evidence supporting American College of Cardiology/American Heart Association and European Society of Cardiology Guidelines, 2008–2018. JAMA. 2019;321:1069–80.

    Article  Google Scholar 

  5. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC. Scientific evidence underlying the ACC/AHA clinical practice guidelines. JAMA. 2009;301:831–41.

    Article  CAS  Google Scholar 

  6. Du Vaure CB, Dechartres A, Battin C, Ravaud P, Boutron I. Exclusion of patients with concomitant chronic conditions in ongoing randomised controlled trials targeting 10 common chronic conditions and registered at ClinicalTrials. gov: a systematic review of registration details. BMJ Open. 2016;6:e012265.

    Article  Google Scholar 

  7. U.S. Food and Drug Administration. Framework for FDA’S real-world evidence program. https://www.fda.gov/science-research/science-and-research-special-topics/real-world-evidence. Accessed 16 Nov 2020.

  8. Gombar S, Callahan A, Califf R, Harrington R, Shah NH. It is time to learn from patients like mine. npj Digit Med. 2019;2:2018–20. https://doi.org/10.1038/s41746-019-0091-3.

    Article  Google Scholar 

  9. Longhurst CA, Harrington RA, Shah NH. A “green button” for using aggregate patient data at the point of care. Health Aff. 2014;33:1229–35.

    Article  Google Scholar 

  10. Callahan A, Gombar S, Jung K, Steinberg E, Harrington R, Shah NH. Delivering on-demand evidence via an informatics consultation service, pp. 3–5.

  11. Schuler A, Callahan A, Jung K, Shah NH. Performing an informatics consult: methods and challenges. J Am Coll Radiol. 2018;15:563–8.

    Article  Google Scholar 

  12. Davies NM, Holmes M V., Davey Smith G. Reading Mendelian randomisation studies: a guide, glossary, and checklist for clinicians. BMJ. 2018;362.

  13. Burgess S, Davey Smith G, Davies NM, Dudbridge F, Gill D, Glymour MM, et al. Guidelines for performing Mendelian randomization investigations. Wellcome Open Res. 2020;4:186.

    Article  Google Scholar 

  14. Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183:758–64.

    Article  Google Scholar 

  15. Dickerman BA, Hernán MA. Avoidable flaws in observational analyses: an application to statins and cancer. Nat Med. 2019. https://doi.org/10.1038/s41591-019-0597-x.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Kuan V, Denaxas S, Gonzalez-Izquierdo A, Direk K, Bhatti O, Husain S, et al. A chronological map of 308 physical and mental health conditions from 4 million individuals in the English National Health Service. Lancet Digit Heal. 2019;1:e63-77.

    Article  Google Scholar 

  17. Denaxas S, Gonzalez-Izquierdo A, Direk K, Fitzpatrick NK, Fatemifar G, Banerjee A, et al. UK phenomics platform for developing and validating electronic health record phenotypes: CALIBER. J Am Med Informatics Assoc. 2019.

  18. Dashti HS, Shea MK, Smith CE, Tanaka T, Hruby A, Richardson K, et al. Meta-analysis of genome-wide association studies for circulating phylloquinone concentrations. Am J Clin Nutr. 2014;100:1462–9.

    Article  CAS  Google Scholar 

  19. Malik R, Chauhan G, Traylor M, Sargurupremraj M, Okada Y, Mishra A, et al. Multiancestry genome-wide association study of 520,000 subjects identifies 32 loci associated with stroke and stroke subtypes. Nat Genet. 2018;50:524–37.

    Article  CAS  Google Scholar 

  20. Yavorska OO, Burgess S. MendelianRandomization: an R package for performing Mendelian randomization analyses using summarized data. Int J Epidemiol. 2017;46:1734–9.

    Article  Google Scholar 

  21. Serper M, Weinberg EM, Cohen JB, Reese PP, Taddei TH, Kaplan DE. Mortality and hepatic decompensation in patients with cirrhosis and atrial fibrillation treated with anticoagulation. Hepatology. 2020;0–3.

  22. Choi J, Kim J, Shim JH, Kim M, Nam GB. Risks versus benefits of anticoagulation for atrial fibrillation in cirrhotic patients. J Cardiovasc Pharmacol. 2017;70:255–62.

    Article  CAS  Google Scholar 

  23. Kuo L, Chao TF, Liu CJ, Lin YJ, Chang SL, Lo LW, et al. Liver cirrhosis in patients with atrial fibrillation: would oral anticoagulation have a net clinical benefit for stroke prevention? J Am Heart Assoc. 2017;6.

  24. Lee SJ, Uhm JS, Kim JY, Pak HN, Lee MH, Joung B. The safety and efficacy of vitamin K antagonist in patients with atrial fibrillation and liver cirrhosis. Int J Cardiol. 2015;180:185–91.

    Article  Google Scholar 

  25. Ahmad FS, Ricket IM, Hammill BG, Eskenazi L, Robertson HR, Curtis LH, et al. Computable phenotype implementation for a national, multicenter pragmatic clinical trial: lessons learned from ADAPTABLE. Circ Cardiovasc Qual Outcomes. 2020;CIRCOUTCOMES–119.

  26. Willoughby C, Fridsma D, Chatterjee L, Speakman J, Evans J, Kush R. A standard computable clinical trial protocol: the role of the BRIDG model. Drug Inf J DIJ/Drug Inf Assoc. 2007;41:383–92.

    Article  Google Scholar 

  27. Marshall IJ, Wallace BC. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst Rev. 2019;8:163.

    Article  Google Scholar 

  28. Kiritchenko S, De Bruijn B, Carini S, Martin J, Sim I. ExaCT: automatic extraction of clinical trial characteristics from journal publications. BMC Med Inform Decis Mak. 2010;10:56.

    Article  Google Scholar 

  29. Constantin A, Pettifer S, Voronkov A. PDFX: fully-automated PDF-to-XML conversion of scientific literature. In: Proceedings of the 2013 ACM symposium on Document engineering. 2013. pp. 177–80.

  30. Parizi RM, Guo L, Bian Y, Azmoodeh A, Dehghantanha A, Choo K-KR. CyberPDF: smart and secure coordinate-based automated health PDF data batch extraction. In: 2018 IEEE/ACM international conference on connected health: applications, systems and engineering technologies (CHASE). 2018. pp. 106–11.

  31. Jacobs AK, Quinn TA, Nelson SJ. Mapping SNOMED-CT concepts to MeSH concepts. In: AMIA annual symposium proceedings. 2006. p. 965.

  32. Gokhale KM, Chandan JS, Toulis K, Gkoutos G, Tino P, Nirantharakumar K. Data extraction for epidemiological research (DExtER): a novel tool for automated clinical epidemiology studies. Eur J Epidemiol. 2020;36:1–14.

    Google Scholar 

  33. Hemani G, Zheng J, Elsworth B, Wade KH, Haberland V, Baird D, et al. The MR-Base platform supports systematic causal inference across the human phenome. Elife. 2018;7:e34408.

    Article  Google Scholar 

  34. Vimalananda VG, Gupte G, Seraj SM, Orlander J, Berlowitz D, Fincke BG, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015;21:323–30.

    Article  Google Scholar 

  35. Informatics Consultation Service at Stanford. http://greenbutton.stanford.edu. Accessed 17 Dec 2020.

  36. Larsson SC, Traylor M, Markus HS. Circulating vitamin K1 levels in relation to ischemic stroke and its subtypes: a Mendelian randomization study. Nutrients. 2018;10:1–7.

    Google Scholar 

  37. Li Y-C, Haug PJ, Lincoln MJ, Turner CW, Pryor TA, Warner HH. Assessing the behavioral impact of a diagnostic decision support system. In: Proceedings of the annual symposium on computer application in medical care. 1995. p. 805.

  38. Plante DA, Kassirer JP, Zarin DA, Pauker SG. Clinical decision consultation service. Am J Med. 1986;80:1169–76.

    Article  CAS  Google Scholar 

  39. Mouhayar E, Salahudeen A. Hypertension in cancer patients. Texas Hear Inst J. 2011;38:263.

    Google Scholar 

  40. Budrionis A, Bellika JG. The learning healthcare system: Where are we now? A systematic review. J Biomed Inform. 2016;64:87–92.

    Article  Google Scholar 

  41. European Medicines Agency. Guideline on registry-based studies. https://www.ema.europa.eu/en/guideline-registry-based-studies. Accessed 15 Dec 2020.

  42. National Guideline Centre (UK). Emergency and acute medical care in over 16s: service delivery and organisation. London: National Institute for Health and Care Excellence (UK); 2018 Mar. (NICE Guideline, No. 94.) Chapter 29, Multidisciplinary team meeting.

  43. Observational Health Data Sciences and Informatics Methods Library. https://www.ohdsi.org/methods-library/. Accessed 07 Dec 2020.

  44. Wolf A, Dedman D, Campbell J, Booth H, Lunn D, Chapman J, et al. Data resource profile: Clinical Practice Research Datalink (CPRD) aurum. Int J Epidemiol. 2019;48:1740–1740g.

    Article  Google Scholar 

Download references

Acknowledgements

We thank clinical colleagues who have contributed to the pilot survey and who have consented for their names to be acknowledged: Christopher Tomlinson, Upkar Gill, Paul Harrow, Harry Martin, Omer Ahmad, Louise China, Vanessa Taylor, Sai Ambati, Amy Prideaux, Alex May, Ruth Gilbert, Hamish Miller, Bilal Mateen, Rhys Davies, Bu'Hussain Hayee, David Cronin, Katharine Pollock and Philip Oppong.

Funding

AGL is supported by funding from the Wellcome Trust (204841/Z/16/Z), National Institute for Health Research (NIHR) University College London Hospitals Biomedical Research Centre (BRC714/HI/RW/101440), NIHR Great Ormond Street Hospital Biomedical Research Centre (19RX02) and the Health Data Research UK Better Care Catalyst Award (CFC0125). MK is funded by the British Heart Foundation (FS/18/5/33319). RMB is supported by a UKRI Innovation Fellowship funded by the Medical Research Council (Grant No: MR/S003797/1). ADS is supported by a postdoctoral fellowship from THIS Institute. RJBD is supported by the NIHR Biomedical Research Centres at South London and Maudsley NHS Foundation Trust (SLAM; IS-BRC1215-20018); Health Data Research UK; UK Research and Innovation (UKRI) London Medical Imaging & Artificial Intelligence Centre for Value Based Healthcare; the BigData@Heart Consortium (Grant No. 116074 of the European Union Horizon 2020 programme); the NIHR BRC and Research Informatics Unit at University College London Hospitals; and the NIHR Applied Research Collaboration South London at KCHFT. GDS and TG are funded by the Medical Research Council Integrative Epidemiology Unit at the University of Bristol MC_UU_00011/1&4. HH is an NIHR Senior Investigator and is funded by the NIHR University College London Hospitals Biomedical Research Centre, supported by Health Data Research UK (LOND1). The funding bodies do not play any roles in the design of the study and collection, analysis, and interpretation of data in writing the manuscript.

Author information

Authors and Affiliations

  1. Institute of Health Informatics, University College London, London, UK

    Alvina G. Lai, Wai Hoong Chang, Constantinos A. Parisinos, Michail Katsoulis, Ruth M. Blackburn, Anoop D. Shah, Vincent Nguyen, Spiros Denaxas, Folkert W. Asselbergs, Reecha Sofat, Richard J. B. Dobson, Amitava Banerjee & Harry Hemingway

  2. Health Data Research UK, London, UK

    Alvina G. Lai, Wai Hoong Chang, Ruth M. Blackburn, Spiros Denaxas, Krishnarajah Nirantharakumar, Folkert W. Asselbergs, Richard J. B. Dobson, Aroon Hingorani, Alastair K. Denniston, Simon Ball & Harry Hemingway

  3. University College London Hospitals NIHR Biomedical Research Centre, London, UK

    Anoop D. Shah, Spiros Denaxas, Folkert W. Asselbergs, Reecha Sofat & Bryan Williams

  4. The Alan Turing Institute, London, UK

    Spiros Denaxas

  5. Medical Research Council Integrative Epidemiology Unit, University of Bristol, Bristol, UK

    George Davey Smith & Tom R. Gaunt

  6. Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK

    George Davey Smith, Tom R. Gaunt & Jonathan Sterne

  7. Institute of Applies Health Research, University of Birmingham, Birmingham, UK

    Krishnarajah Nirantharakumar

  8. Statistics and Bioinformatics Group, School of Fundamental Sciences, Massey University, Palmerston North, New Zealand

    Murray P. Cox

  9. Public Health Wales, University Hospital of Wales, Cardiff, UK

    Donall Forde

  10. Department of Cardiology, University Medical Center Utrecht, Utrecht University, Utrecht, Netherlands

    Folkert W. Asselbergs

  11. Institute of Cardiovascular Science, University College London, London, UK

    Folkert W. Asselbergs, Aroon Hingorani, Riyaz Patel & Bryan Williams

  12. University College London Hospitals NHS Trust, London, UK

    Anoop D. Shah, Steve Harris & Bryan Williams

  13. Medical Research Council Biostatistics Unit, University of Cambridge, Cambridge, UK

    Sylvia Richardson

  14. Department of Biostatistics and Health Informatics, King’s College London, London, UK

    Richard J. B. Dobson

  15. Barts Health NHS Trust, The Royal London Hospital, Whitechapel Rd, London, UK

    Amitava Banerjee

  16. University Hospitals Birmingham NHSFT, Birmingham, UK

    Alastair K. Denniston & Simon Ball

  17. UCL Great Ormond Street Institute of Child Health, London, UK

    Neil J. Sebire

  18. NIHR Great Ormond Street Hospital Biomedical Research Centre, London, UK

    Neil J. Sebire

  19. Department of Medicine, School of Medicine, Stanford University, Stanford, CA, USA

    Nigam H. Shah

  20. Barts Liver Centre, Blizard Institute, Queen Mary University of London, London, UK

    Graham R. Foster

Authors
  1. Alvina G. Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wai Hoong Chang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Constantinos A. Parisinos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michail Katsoulis
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ruth M. Blackburn
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Anoop D. Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Vincent Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Spiros Denaxas
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. George Davey Smith
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tom R. Gaunt
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Krishnarajah Nirantharakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Murray P. Cox
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Donall Forde
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Folkert W. Asselbergs
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Steve Harris
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Sylvia Richardson
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Reecha Sofat
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Richard J. B. Dobson
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Aroon Hingorani
    View author publications

    You can also search for this author in PubMed Google Scholar

  20. Riyaz Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  21. Jonathan Sterne
    View author publications

    You can also search for this author in PubMed Google Scholar

  22. Amitava Banerjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  23. Alastair K. Denniston
    View author publications

    You can also search for this author in PubMed Google Scholar

  24. Simon Ball
    View author publications

    You can also search for this author in PubMed Google Scholar

  25. Neil J. Sebire
    View author publications

    You can also search for this author in PubMed Google Scholar

  26. Nigam H. Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  27. Graham R. Foster
    View author publications

    You can also search for this author in PubMed Google Scholar

  28. Bryan Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  29. Harry Hemingway
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Research question: AGL, GRF, HH. Funding: AGL, HH. Study design and analysis plan: AGL, WHC, CP, HH. Preparation of data, including electronic health record phenotyping in the CALIBER portal: AGL, WHC, CP, SD. Statistical analysis: AGL, WHC, MK. Drafting initial versions of the manuscript: AGL, HH. Critical review of early and final versions of the manuscript: AGL, WHC, CAP, MK, RMB, ADS, VN, SD, GDS, TRG, KN, MPC, DF, FWA, SH, SR, RS, RJBD, AH, RP, JS, AB, AKD, SB, NJS, NHS, GRF, BW, HH. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Alvina G. Lai.

Ethics declarations

Ethics approval and consent to participate

We obtained informational governance approval from the Medicines Healthcare Regulatory Authority (UK) Independent Scientific Advisory Committee (20_078R) to analyse the Clinical Practice Research Datalink (CPRD) linked to secondary care Hospital Episode Statistics and the Office for National Statistic death registration. Ethics approval was exempted for the survey as it is classified under service evaluation. Survey participants gave informed consent to participate by returning the completed survey.

Consent to publish

Not applicable.

Competing interests

AB has received research funding from AstraZeneca for work unrelated to this research. GRF receives funding from companies that manufacture drugs for hepatitis C virus (AbbVie, Gilead, MSD) and consults for GSK, Arbutus and Shionogi in areas unrelated to this research. TRG and GDS have received research funding from GlaxoSmithKline and Biogen for work unrelated to this research.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

Supplementary data 1 to 11.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lai, A.G., Chang, W.H., Parisinos, C.A. et al. An informatics consult approach for generating clinical evidence for treatment decisions. BMC Med Inform Decis Mak 21, 281 (2021). https://doi.org/10.1186/s12911-021-01638-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12911-021-01638-z

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us