Replicating medication trend studies using ad hoc information extraction in a clinical data warehouse

Abstract

Background

Medication trend studies show the changes of medication over the years and may be replicated using a clinical Data Warehouse (CDW). Even nowadays, a lot of the patient information, like medication data, in the EHR is stored in the format of free text. As the conventional approach of information extraction (IE) demands a high developmental effort, we used ad hoc IE instead. This technique queries information and extracts it on the fly from texts contained in the CDW.

Methods

We present a generalizable approach of ad hoc IE for pharmacotherapy (medications and their daily dosage) presented in hospital discharge letters. We added import and query features to the CDW system, like error tolerant queries to deal with misspellings and proximity search for the extraction of the daily dosage. During the data integration process in the CDW, negated, historical and non-patient context data are filtered. For the replication studies, we used a drug list grouped by ATC (Anatomical Therapeutic Chemical Classification System) codes as input for queries to the CDW.

Results

We achieve an F1 score of 0.983 (precision 0.997, recall 0.970) for extracting medication from discharge letters and an F1 score of 0.974 (precision 0.977, recall 0.972) for extracting the dosage. We replicated three published medical trend studies for hypertension, atrial fibrillation and chronic kidney disease. Overall, 93% of the main findings could be replicated, 68% of sub-findings, and 75% of all findings. One study could be completely replicated with all main and sub-findings.

Conclusion

A novel approach for ad hoc IE is presented. It is very suitable for basic medical texts like discharge letters and finding reports. Ad hoc IE is by definition more limited than conventional IE and does not claim to replace it, but it substantially exceeds the search capabilities of many CDWs and it is convenient to conduct replication studies fast and with high quality.

Background

Reliable information on the use of medication in a hospital and its changes over time is of great importance for many acute and chronic diseases – from a hospital, patient and payor perspective. This is reflected by many studies reporting medication trends: e.g. attention deficit hyperactivity disorder (ADHD) [1], atrial fibrillation (AF) (US [2], Denmark [3, 4]), chronic kidney disease (CKD) [5, 6], rheumatoid disease [7] or hypertension (HT) [8] (England [9], France [10], Germany [11], Sweden [12], US [13, 14]).

However, medical research (like many other disciplines) is affected by the so called replication crisis, addressed in an article in 2012 reporting that only 11% of the pre-clinical cancer studies could be replicated [15]. The Nature Journal conducted a survey of 1500 scientists in 2016, in which 70% of them stated that they had failed to reproduce another scientist’s experiment [16].

The ability to reproduce findings reported in a clinical study is a cornerstone of scientific progress. Replication of medication trend studies can be performed using a CDW, which is an important, albeit little exploited and published use case.

CDWs can deal with structured data very well. Unfortunately, a lot of the patient information in the electronic health record (EHR) is still stored in free text. E.g. Jensen et al. retrieved on average 146 unstructured text documents for each patient from EHR of their hospital for their study [17]. Medication, too, is usually documented as free text within the discharge letter. As a solution, advanced CDW systems offer a query language that can extract data from free text (e.g. in [18]).

The conventional approach is to perform information extraction (IE) in the ETL¹ process. A well-known system for IE of medication is MedEx [19]. Beside other rule based-systems like [20], hybrid systems exist using machine learning techniques [21]. A good overview on IE from free text is given by Wang et al. [22].

Rule based systems require a high volume of handcrafted rules and learning systems need a large amount of manually labeled training data. Either way, a lot of expert work is necessary. Besides high developmental efforts, another disadvantage of conventional IE is its slow promptness and non-adaptability by users [18].

A novel way to retrieve information from plain text is ad hoc IE. Ad hoc IE is described as extracting the existence of any concepts (e.g. chronic kidney disease) or any numbers, like the left ventricular ejection fraction (LVEF) value, from textual sources in real-time. The Boolean ad hoc IE queries the existence (yes/no) of a medical concept. A medical concept is a named entity that may have a feature/property or a numeric value. Examples of Boolean concepts are single findings or assessments (e.g. moderate mitral insufficiency, severe aortic stenosis), drugs (e.g. Aspirin, beta blocker) or diagnoses (e.g. appendicitis, myocardial infarction). Numeric IE extracts the value as number of a numerical concept. That could be for example the value of a laboratory finding (e.g. cholesterol, glucose, LEVF) or a derived values/indexes (e.g. BMI, age). A numerical condition can be defined optionally, like LVEF <45, matching all mentions of LVEF with a value lower than 45. In some finding reports, the exact value of a concept is not given but there is a formulation indicating an interval or an inequality of a value (e.g. “LVEF lower than 45"). These statements can be queried in conjunction with numeric ad hoc IE exploiting both qualitative and quantitative information from textual reports e.g. for checking inclusion or exclusion criteria of studies. In addition to count queries, which only asses the presence of a concept or the validity of constraints (e.g. BMI >25), the actual values can also be returned for further processing.

This technique showed good results and requires little developmental effort, since the text is indexed efficiently and can be queried with powerful features [18].

Objectives

This work introduces ad hoc IE for medication and their daily dosage from hospital discharge letters. We present and evaluate query features for a CDW. As an example of use, we show medication trend estimations. Therefore we replicate existing studies from the literature in a large CDW of the University Hospital of Würzburg using ad hoc IE. The results will be compared with the corresponding published data describing similarities and differences.

Methods

The developmental steps included extensions and features for the data integration process and the development of new data query tools. For study replication, the drug names had to be acquired and transformed.

CDW system design

We implemented our features in the PaDaWaN CDW [23], which uses the full-text-search engine Apache Solr² as storage engine, based on the index library Apache Lucene³. The PaDaWaN-CDW contains both, unstructured text data and structured data, including core data (e.g. age, sex etc.), coded data (e.g. ICD10 and OPS etc.) and numerous other types of information of the clinical information system (CIS) (e.g. lab data) [18]. The data integration process of the PaDaWaN-system contains analyzers for the respective data types. At the end of the pipeline, all values are stored in the Lucene index and can be queried from physicians in the PaDaWaN Web GUI [23]. We modified and extended generic tools for text analysis in the import pipeline (see below). We also added new query features to the framework, which can be used in the front end GUI during runtime.

Data integration development

Lexical analysis

The text analysis tool for discharge letters splits the text into sections like diagnoses, medications, and laboratory values. Figure 1 shows an example for a medication section. We added a sentence splitter for medication extraction that separates the individual medication instructions from each other. Furthermore, we deactivated the stemmer because the word endings of the medications should not be touched. Finally, a custom tokenizer ensures that the quantity, strength and dosage information of the medication instructions are correctly decomposed. Table 1 shows an example of the lexical analysis.

Fig. 1

Example of a medication section of a hospital discharge letter. a German. b English

Table 1 Lexical analysis of the medication section in the discharge letter

Context of information

The context of information in a discharge letter is an important topic. Many pieces of information are negated [24] (e.g. “no fever”, “dizziness is denied”) or they relate to other persons (e.g. within the context of family history). Some information like medications within in the discharge letter have a temporal context and may not be valid any longer (e.g. medication might have been stopped at hospital entry or during hospitalization, like Ramipril in Fig. 1). Depending on the application or evaluation, different types of information are relevant or must be excluded. In most cases, physicians are interested in the confirmed and current findings of a patient.

The PaDaWaN data integration process already identifies negations in the texts with an extended version of the NegEx-algorithm [25]. These negations can be excluded in the GUI for certain queries like medication extraction [18]. We extended this NegEx-version to a ConText [26] implementation. This algorithm handles not only negations but also the context of an information. It is implemented using Apache UIMA⁴. Furthermore, we added several trigger tokens for the patient history.⁵ Using these modifications, the non-currently used drugs are excluded from the text. The remaining, relevant medications remain retrievable at runtime by user queries.

Text query features

Spelling error tolerant query

PaDaWaN already contains several text query features like token, phrase and regular expression queries. Since medical reports are often manually entered, some names of medications are misspelled. For such typos we added a spelling error tolerant query feature that makes use of the Damerau-Levenshtein distance. It is a string metric for measuring the edit distance between two sequences and can thus be employed to assess how much two medication names differ. The distance measures includes a transposition operation (transposition of two adjacent characters) in addition to three edit operations, i.e. insertion, deletion, and substitution [27]. Table 2 shows selected examples of misspellings and their Damerau–Levenshtein distance to the product name.

Table 2 Examples of misspelled medication names and their Damerau–Levenshtein distance

Dose extraction with proximity search

Although most medication trend studies only consider the use of a drug, we also strived to extract the daily dosage of the medication. This requires two pieces of information: the strength and the cumulative daily amount of the drug. The strength is given in digits with a standard unit (usually milligrams or micrograms) with the drug name. The dosing interval is usually coded by a number-hyphen notation like 1/2-0-1/2. The numbers represent the units that must be taken in the morning, at noon and in the evening. A optional fourth digit refers to the number before going to bed. The daily dose is obtained by adding these three or four numbers and then multiplying by the strength. We added a feature that makes it easier to query the daily dose. The proximity query searches the given tokens next to each other. The order of these tokens is irrelevant. Proximity queries do not match across sentence boundaries. Since each medication instruction is provided in a segmented fashion as a single sentence during the import, proximity queries do not match dosage information of other medications. Table 3 shows an example of how a daily dose can be extracted. The corresponding request is displayed as well as matching and not matching text snippets. With this technique, queries can be made for the different drug strengths and daily dosages.

Table 3 Example for promximity searches to query the daily dose of a medication instruction

Query token generation

The Anatomical Therapeutic Chemical (ATC) Classification System is an international classification of active ingredients of drugs⁶. In the literature, ATC codes are used to encode drugs and active agents groups. In order to get all brand, drug and agent group names of an ATC-group like C07 Beta Blocking Agents, we use the ABDA-DB⁷, which contains all names in English and German. Since medical reports rarely contain the full name of a drug, we processed the names from the ABDA-DB in various ways: a) names were simplified by omitting the names of the manufacturers and the strength of the drug; b) other unnecessary words were removed; that includes modifiers concerning the effect like forte and the administration form like oral; c) abbreviations and alternative spellings were considered. Table 4 shows examples of the processing of drug names. The resulting tokens were used for the queries. Hyphens do not need to be treated because they are removed by the tokenizing procedure.

Table 4 Example for the processing of the drug names

Evaluation

We performed tests to evaluate our development and conducted case studies aiming to replicate findings reported in selected medication trends studies.

Medication extraction

Since medication studies only consider the use of drugs, the replication requires just Boolean IE. Therefore we carried out a comprehensive test. We further evaluated the requests for the daily dosage using ad hoc IE. To protect privacy, these texts were de-identified and in addition they must not leave the clinical network.

Extraction of drugs. For the evaluation of the medication extraction 600 documents were randomly selected from the disease domains hypertension, atrial fibrillation and chronic kidney disease. From each domain, 100 medication sections from 2005 and 100 sections from 2015 were sampled, resulting in a total of 600 documents. A manually annotated gold standard was created for these documents. All medications, brands, drug and substance names were annotated using the Apache UIMA CAS type system. In order to save time, the text was first automatically pre-announced using the medication tokens gained in “Query token generation” section. Then, the texts were manually corrected to obtain the gold standard. The ATHEN environment⁸ was used to perform this work [28]. Afterwards the original texts were imported into the PaDaWaN-CDW with the data integration pipeline. Then queries were made with all drug names and the hits detected were annotated. At the end, all hits found by the system were compared to the gold standard.

Daily dosage. The extraction of the daily medication dosage was evaluated with several drugs: Antihypertensive drugs: EsidrixⓇ (Thiazide-Diuretika, ATC: C03A), ConcorⓇ (β-blocker, C07A), DelixⓇ (ACE inhibitor C09A) and novel oral anticoagulants (NOAC) used for atrial fibrillation: EliquisⓇ, PradaxaⓇ, XareltoⓇ. For each drug, 100 medication sections containing this drug from 2015 were selected. For the antihypertensive drugs another 100 units were selected for the year 2005. This was not possible for the NOACs, since they did not exist at that time. Queries were made in the PaDaWaN system and evaluated manually. For the evaluation, all dose strengths were extracted. The proximity query feature was used to extract the dose.

Study replication

To evaluate the quality of the study replication, we chose five studies from the literature covering three domains (hypertension, atrial fibrillation, chronic kidney disease) and compared the major and sub-findings with the results of the University Hospital of Würzburg in total, respectively restricted to its Department of Internal Medicine I (Med1) using the ad hoc query feature with of the CDW. The drugs were extracted from the medication section of the discharge letter. That contains in almost every case the medication at discharge representing the recommended / prescribed medication. Additionally the medication at admission is described in 18% (Med1: 13%) of all cases. At discharge from hospital, patients receive 8% (Med1: 19%) more medication than at admission, while nearly all medications from admission were continued at discharge. (Tested for the main drug agent groups for hypertension.) We used the whole medication section with all medication descriptions as data source to identify weather a drug is taken or not.

This was conducted with the PaDaWaN-CDW including about 1 million patients with 5 million patient cases and more than 600 million pieces of single information. We applied the same in- and exclusion criteria as in the respective publications. However, we did not compute age-adjusted values. Not every single evaluation in the publications was reproduced; we rather focused on the main statements and central result tables of the studies or took the most interesting parts of the publications to show the power of our approach.

Hypertension We chose [13] as first drug trend study, because it is a highly cited study addressing a large population. The analyzed data was acquired during the National Health and Nutrition Examination Survey (NHANES) [29]. We further aimed to replicate the results of Shah and Stafford [14] concerning the findings on systolic blood pressure. These authors used data from the National Disease an Therapeutic Index (NDTI), a nationally representative physician survey. We extracted this information from the discharge letter via numeric ad hoc IE [18].

Atrial Fibrillation. In the replication of the study for atrial fibrillation [3] the ad hoc IE from unstructured texts was combined with structured data from the CDW and differentiated according to these. Subgroups such as comorbidity and age groups were investigated by Gadsbøll et al. [4]. The data sources of these studies were the Danish National Patient Registry, the (Danish) National Prescription Registry and the (Danish) Civil Registration System, containing various information on all prescriptions dispensed in Danish pharmacies since 1995.

Chronic Kidney Disease. We also selected a study to examine temporal trends and treatment patterns by patients with CKD and type 2 diabetes mellitus (T2DM) [5]. In this work, medication groups are evaluated. In a more detailed analysis, CKD was broken down into different severity levels (stages), and the medicative effect of the medication groups was considered [5]. This study also used the data from NHANES.

Tables 5 and 6 map all drug and diagnostic group designations used in respective publications to ATC and ICD10 codes, respectively. These codes were used for the replication of these studies. Table 7 summarizes the replicated studies and shows their inclusion and exclusion criteria.

Table 5 Mapping between diagnostic group designations used in the literature and ICD10 codes used for the replication

Table 6 Mapping between drug group designations used in the literature and ATC codes used for the replication

Table 7 Overiew of replicated studies and their inclusion and exclusion criteria

Results

Ad hoc IE evaluation

Extraction of drugs

Table 8 shows the performance of the ad hoc extraction of medications with an overall F1-score of 0.983 (precision 0.997 and recall 0.970).

Table 8 Performance of the ad hoc extraction of medications

Most errors were caused by abbreviations. The misspelling based errors could be significantly reduced by the error tolerant query feature. Table 9 shows the error analysis of the ad hoc extraction of medications. The most common occurrences of the error groups are shown below. Abbreviation Fraxi (20), Tiotropium (6), Mg Verla (4), Dreisavit (3), Dabigatran (2), Insuman (2), Isosorbid (2) Not in DB Eunerpan (9), Polybion (4), Aclidinium (2), Calcetat (2), Natriumperchlorat (2), Cranoc (2), Calcetat (2) Alternative notation Glycopyrronium (2), Dikalium Clorazepat (2), Humaninsulin (1), Diuretikum (1), Ca Carbonat (1) Misspelling Ferrosanol (4), Eins alpha (2), Amphomoronal (2), Beclometasondipropionat (2), Klazid (2), Rehnagel (2), Cardular (2), Calciumdiacetat (2) Search to fuzzy diabetes ≈ diabetex (4), diagnostik ≈ diagnostika (1), antihypertensiven ≈ antihypertensives (1) Incorrect extracted medication thrombozyten (1), cholesterin (1), albumin (1), kalium (1), natrium (1)

Table 9 Error analysis of the ad hoc extraction of medications

Extraction of daily drug dose

An analysis on the data set for the daily dose, that contains 900 mentions of selected drugs, revealed that 5% of the mentioned drugs were discontinued or reduced. 90% had an indicated strength, 92% an instruction and 89% a strength and an instruction. See Table 10.

Table 10 Presence of strength and instruction application of medication in the evaluation set

The most common daily taken dose was one unit (57%) followed by two units (31%), see Table 11.

Table 11 Summed daily dose of the medication units in the evaluation set

The overall F1-score for the extraction of the daily medication dose was 0.974. The precision was the same or slightly higher than the recall in all tests. The extraction results were slightly better on the antihypertensive drug set (F1: 0.982) than on the NOACs drug set (F1: 0.958). The documents from 2015 also showed slightly better results than those of 2005 (F1: 0.977 vs 0.968). The complete results can be found in Table 12.

Table 12 Performance of the ad hoc extraction of the daily medications dose

Most errors were caused by an unusual notation. See Table 13 and listing below. Other error sources were supplements, which contained numbers, incorrect splitting of the tokenizer, double mentions in same document, segmentation faults, and a too wide gap between the drug name and the instructions. Notation Esidrix 1x1, Pradaxa 150-0-150 mg Supplement Pradaxa 110 mg 1-0-1 (bitte 1 Tag vor stationären Aufnahmetermin pausieren); Tokenizer EuthyroxⓇDouble mention Medikation bei Entlassung: Esidrix 12,5 mg 1-0-0; Medikamente bei Entlassung: Esidrix 25 pausiert Segmentation Gap Concor 5 mg (bei Bedarf) 1 – 0 – 0 – 1

Table 13 Error analysis of the ad hoc extraction of the daily medications dose

Study replication

The presented results for the University Hospital of Würzburg (UKW) and the Department of Internal Medicine I (Med1) were computed via ad hoc IE (see “Study replication” section). Since the ad hoc IE had an F1 score of 0.974, there may be small deviations from the exact values.

Hypertension

Study: Trends in antihypertensive medication use and blood pressure control among United States adults with hypertension Table 14 shows the results of the replication of the medication trend study to hypertension for the years 2000 to 2010. The findings of the referenced paper and their reproducibility by our results are listed in Table 15. The computation time to query the data for Table 14 from the CDW was 2 min 26 s.

Table 14 Replication of the medication group trend study for hypertension [13]

Table 15 Findings of the replicated studies compared to our results

Current trends of hypertension treatment in the United States.

Table 16 shows the grouped systolic blood pressure of hypertensive patients and Table 18 lists their the use of drug agent groups. The findings of the referenced paper and their reproducibility by our results are listed in Table 17. The computation time to query the data for Tables 16 and 18 from the CDW was aggregated 49 min 55 s.

Table 16 Systolic blood pressure (SBP) in mm Hg of hypertensive patients compared to [14]

Table 17 Findings of the replicated studies compared to our results

Table 18 Use of drug agent groups and systolic blood pressure (SBP, measured in mm Hg) groups of hypertensive patients compared to [14]

Chronic kidney disease

Study: Understanding CKD among patients with T2DM: prevalence, temporal trends, and treatment patterns – NHANES 2007-2012 Figure 2 is an additional evaluation showing all severity levels of CKD over time. The computation time to query the data from the CDW was 14 s.

Fig. 2

Temporal trend of CKD stages in the UKW. The severity degrees of CKD-patients are shown over time

Figure 3 shows the hypertension medication agent groups by degrees of severity of CKD for all patients with hypertension and CKD for the years 2013-2016. The computation time to query the data from the CDW for Fig. 3 was 1 min 3 s.

Fig. 3

Medication agent groups by degrees of severity of CKD in the UKW of CKD patients with hypertension

Tables 19 and 21 compare the findings of Wu et al. [5] to our findings for the UKW and the Med1 concerning medication and agent groups for patients with CKD and T2DM. It shows the medication for diabetes as well as the hypertension. The findings of the referenced paper and their reproducibility by our results are listed in Table 20. The computation time to query the data from the CDW was 3 min 16 s for Table 19 and 5 min 9 s for Table 21.

Table 19 Medication and agent groups for CKD with T2DM compared to [5]

Table 20 Findings of the replicated studies compared to our results

Table 21 Medication and agent groups for CKD with T2DM compared to [5]

Atrial fibrillation

The studies on atrial fibrillation (AF) investigate the characteristics and the temporal trend of the use of oral anticoagulants (OAC).

Study: Increased use of oral anticoagulants in patients with atrial fibrillation: temporal trends from 2005 to 2015 in DenmarkGadsbøll et al. investigate the increased use of oral anticoagulants in patients with atrial fibrillation [3]. Figure 4 shows the temporal trend of VKA and OACs compared to [4]. The findings of the referenced paper and their reproducibility by our results are listed in Table 22. The computation time to query the data from the CDW for Fig. 4 was 25 s.

Fig. 4

Temporal trend of VKA and OACs compared to [4]. a UKW. b Paper

Table 22 Findings of the replicated studies compared to our results

Figure 5 shows the temporal trend for AF patient age groups using OACs like in [4]. The computation time to query the data from the CDW for Fig. 5 was 55 s.

Fig. 5

Temporal trend of OAC clustered by age groups compared to [4]. a UKW. b Paper

Study: Non-vitamin K antagonist oral anticoagulation usage according to age among patients with atrial fibrillation: Temporal trends 2011–2015 in Denmark Staerk et al. made a detailed research for the years 2011 and 2015, since NOAC became relevant [4]. Figures 6 and 7 is a detailed analyses of the temporal trend OACs listing its representatives: Dabigatran, Rivaroxaban, Apixaban. The computation time to query the data from the CDW was 36 sec for Fig. 6 and 29 sec for Fig. 7.

Fig. 6

Temporal trend of VKA and OAC usage of all AF patients compared to [4]. a UKW. b Paper

Fig. 7

Temporal trend of VKA and NOACs of AF patients aged ≥85 compared to [4]. a UKW. b Paper

Table 24 shows the distribution among sex and age groups. Table 25 analyses the comorbidities and Table 26 lists the concomitant medication. The values in the referenced paper refer to the time period between 22.8.2011 and 1.1.2016. We computed the values for the same period (named UKW_11) and for the period 1.1.2016 - 1.1.2018 (named UKW_16). The computation time to query the data from the CDW was 1 min 10 s for Table 24, 1 min 40 s for Table 25 and 2 min 10 s for Table 26. The findings of the referenced paper and their reproducibility by our results are listed in Table 23.

Table 23 Findings of the replicated studies compared to our results

Table 24 Characteristics of patients with atrial fibrillation using VKAs or OAC medications compared to [4]

Table 25 Comorbidities of patients with atrial fibrillation using VKAs or OAC. (Continuation of Table 24)

Table 26 Concomitant medication of patients with atrial fibrillation using VKAs or OAC. (Continuation of Table 24)

Table 27 summarizes the results of the study replication. Main findings were replicated and confirmed by us to 93%, sub-findings to 68% and overall to 75%.

Table 27 Summary of the of the study replication results, including main, sub and overall findings

Daily medication dose extraction.

As an additional evaluation, we extracted the daily dose of patients with AF using ad hoc IE. All three OACs agent groups with their drugs where analyzed: Xarelto (Rivaroxaban) (see Table 28), Eliquis (Apixaban) (see Table 29) and Pradaxa (Dabigatran) (see Table 30).

Table 28 Extraction of the daily medication dose of Xarelto for patients with AF

Table 29 Extraction of the daily medication dose of Eliquis for patients with AF

Table 30 Extraction of the daily medication dose of Pradaxa for patients with AF

The average daily dose was 19,31 mg of Xarelto, 7,4 mg of Eliquis and 232,3 mg of Pradaxa.

Discussion

First, the results of the replication studies are discussed, and second, the ad hoc IE tests and the system itself are compared to other approaches.

Study replication

Major result & comparison. One study (AF Trend from 2005 to 2015 [3]) could be completely replicated, i.e., all main findings and sub-findings were confirmed by us. Overall, 93% of the main findings, 68% of other detailed findings and 75% of all findings could be replicated. Table 27 lists the results of the individual replications. As mentioned in “Background” section, many researchers have tried to reproduce other researchers work, but 70% failed. 24% researchers reporting a successful replication of experiments were able to publish their work. In case of unsuccessful reproduction this proportion was only 13% [16]. Of course, when conducting replication experiments, some deviations have to be expected. Concerning the sources of variation, not only the exact reproduction of the study design is important, but also the population under study and time trends observed regarding diagnosis and therapy matter. E.g., Gu et al. reported that the control of blood pressure (BP) levels “varied greatly between recent publications” [13]. Staerk et al. mentioned that the most frequently used NOAC agent in their study was different to a previous study owing to changes in prescription patterns over time [4].

Study details. The distribution among the groups of active substances for hypertension in the UKW was slightly different compared to the paper [13]. In Med1, patients got substantially more drugs, probably indicating treatment preferences of a certain clinic.

In the CKD study, 75% of all findings agreed with our results, but there were also some deviations. Some observations differed only in stage 5 of CKD. This could be explained with different sizes of population of the subgroups with level 1, 4 and 5. These were caused by the basic population (population-based sample vs. hospital patients). The trends in the studies of atrial fibrillation could be replicated by us, however with a surprisingly small temporal shift. The comorbidities and the concomitant medication differed slightly, but many agreed.

Data acquisition & study population. The studies differed regarding the data acquisition approach: The hypertension [13] and CKD [5] studies were based on NHANES, the AF studies [3, 4] on the Danish National Prescription Registry and the hypertensive study with SBP used a physician survey. The medication in NHANES was "self-reported data (via a patient survey questionnaire)" [5]. We took the medication information from the discharge letter written by physician, which should be reflected in higher accuracy. NHANES is a representative sample of the U.S., i.e. both healthy and sick people, whereas a CDW collects information on hospitalized or ambulatory patients. There are even differences within a hospital. The medication use was found higher in almost all cases at the Med1 compared to the entire clinic. This is comprehensible, because hypertension, atrial fibrillation and chronic kidney diseases are usually treated there. The studies also differed regarding the number of analyzed cases. The AF studies used a nation-wide data source, i.e. three to four times more patients than which were present in the local CDW. For the hypertension study, we analyzed eight times more cases, in the CKD even 25 times more cases.

Analysis duration. While our queries took only a few minutes, it probably took a few weeks or months to conduct the studies for the referenced papers.

Ad hoc IE

Ad hoc IE possesses features of a conventional IE and query functions of CDWs. Therefore, the evaluation results and the system itself are compared with other approaches.

Comparison of evaluation results

According to [22] MedEx is the most widespread used tool for extracting medication information from clinical texts. In their original paper they achieved an F1-score of 93,2% for extracting drug names, a score of 94,6% for the strength and 96,0% for the frequency [19]. Two years later they published a case study around the medication warfarin and pushed the F1 score to 95% (recall 99,7%, precision 90,8%) for extracting the daily dosage [30]. In another study, they tried to calculate the daily dosage for the drug tacrolimus with an extended MedEx version and reported precisions of 90-100% and recalls of 81-100%. For discharge summaries they achieved F1 measures of 96% for strength and 88% for daily dosage [31].

Some papers mention, that they had to deal with more complex medication instructions like dosing in 2 h intervals [19, 30–32]. This may complicate the calculation of the dosage and explain the inferior results compared to ours (F1 97,4%, precision 97,7%, recall 97,2%).

The results of the extraction of the drug names alone were only partially comparable with ours. First, no lists of medications were used in the literature, and second, these are all conventional IEs. We applied ad hoc IE, which extracts the information on the fly during runtime.

Conventional versus ad hoc IE

Conventional IE. IE turns unstructured information embedded in texts into structured data [33]. More precisely, it is the automatic extraction of concepts, entities and events, as well as their relations and associated attributes [22]. It consists of subtasks, i.e. entity recognition, relation extraction, event extraction (including time and date), and template filling [33]. In a conventional IE application information are computed by many expensive processing steps [34]. Therefore, each text is annotated several times, e.g. with parts of speech tagging, syntactic or dependency parsing or word list labeling. The output of a tagging process is the input for the next step. Thereafter rule-based systems apply rules on these annotations to extract information. Machine learning approaches use additional features and a trained model for the extraction step.

Ad hoc IE. In ad hoc IE, a segmentation separates non-related concepts. On these segments, a one-step annotation can be made effectively. But this step is quite fast, due to the index, and in contrast to the conventional IE, there are not “many of expensive processing steps” [34]. Thus, ad hoc IE is suitable for domains that can be handled with a one-step annotation. A survey revealed that 65% of clinical information extraction systems are rule-based and often use a regular expression as a search pattern [22]. Hence, they are interesting for ad hoc IE and could possibly be implemented with it. Ad hoc IE shifts the time of extraction from the data-integration phase to runtime, enabling a flexible IE at runtime for all users.

Ad hoc IE does not address all sub-tasks of a conventional IE application. However, the tasks important to the medical domain are supported: Named entity recognition is ensured by the query functions, relation extraction for medical concepts is accomplished by segmentation and for patient identification by context detection.

Comparison In summary, the ad hoc IE was found to be very well suited for this task. It yielded as good results as the conventional IE but was characterized by a much lower developmental effort, promptness of results and intuitive adaptability by users. In domains with complicated structure, conventional IE might be superior in terms of confidence and accuracy [18]. However, ad hoc IE does not claim to replace conventional IE, it rather should be considered a supplement for quick analysis to get a good and detailed overview for further investigations. An additional advantage of ad hoc IE is its ability not only to return the number of hits, but also to retrieve hit snippets from texts. This addresses two points: 1) Queries can be refined iteratively and 2) the system can also be used as an evaluation environment.

Query Features of other CDWs

Text query features are poorly supported in CDWs [18]. Most of them, like the well known i2b2, store their data in SQL-DBs and just support the like-operator⁹ a SQL full text index. Other CDW index their textual data with index libraries as Apache Solr (e.g. tranSMART [35] or Roogle [36]) or with SQL full text index (e.g. STRIDE [37]). Dr. Warehouse performs an negation detection as well and excludes negated findings from the search [38]. However, no system has query features that exceed a token search.

Comparison to SQL Many CDWs use a SQL-Server as storage engine. Texts can be queried via the like-operator, which is used to perform wildcard queries. However, this is limited in many ways: Error tolerant queries, which deal with misspellings, are not supported. Drug names that consist of several words are difficult or cumbersome to find with SQL methods. Especially, if these words are not next to each other and, e.g., separated by a brand name.

Extracting dose information reliably using SQL is next to impossible. Several words can be between the drug name and the instruction, e.g. additional information about the application. A segmentation of the drugs would be necessary in any case. Additionally, an SQL-based approach is much slower than a text index based system.

Limitations

Limitations for conducting medication trend studies in a CDW relate to complex inclusion and exclusion criteria that can not appropriately be mapped, like complex temporal constraints. Some techniques frequently used in clinical analyses are more difficult to apply like adjustment for important confounders, e.g. sex and age. This is not a technical limitation, but it would require a laborious recalculation.

The feasibility of replication studies depends as well on the data embedded in the CDW. Only integrated concepts or texts can be queried. The populations of studies are always different, so the population of a specific hospital department does not correspond to the overall population.

Conclusion

With the presented approach of the ad hoc IE for medications, which provides equally good results for this task as the conventional approach, it is possible to quickly carry out analyses like the study replications shown here. We combined ad hoc IE with additional filters based on structured and unstructured data: We stratified the data by year and severity of the respective condition, and analyzed subgroups like age, comorbidities and concomitant medication. Furthermore, we used ad hoc IE to transform unstructured data from the discharge letters to structured data (e.g. systolic blood pressure groups) and extracted the daily dosage per drug on the fly.

To calculate daily medication dosages, each strength unit combination must still be queried individually. It is intended to calculate this automatically, e.g. with the use of function queries.