Deficiencies in the transfer and availability of clinical trials evidence: a review of existing systems and standards
© Van Valkenhoef et al.; licensee BioMed Central Ltd. 2012
Received: 27 September 2011
Accepted: 24 August 2012
Published: 4 September 2012
Decisions concerning drug safety and efficacy are generally based on pivotal evidence provided by clinical trials. Unfortunately, finding the relevant clinical trials is difficult and their results are only available in text-based reports. Systematic reviews aim to provide a comprehensive overview of the evidence in a specific area, but may not provide the data required for decision making.
We review and analyze the existing information systems and standards for aggregate level clinical trials information from the perspective of systematic review and evidence-based decision making.
The technology currently used has major shortcomings, which cause deficiencies in the transfer, traceability and availability of clinical trials information. Specifically, data available to decision makers is insufficiently structured, and consequently the decisions cannot be properly traced back to the underlying evidence. Regulatory submission, trial publication, trial registration, and systematic review produce unstructured datasets that are insufficient for supporting evidence-based decision making.
The current situation is a hindrance to policy decision makers as it prevents fully transparent decision making and the development of more advanced decision support systems. Addressing the identified deficiencies would enable more efficient, informed, and transparent evidence-based medical decision making.
Health care policy decision makers such as drug regulatory authorities, reimbursement policy makers and guideline committees routinely evaluate the efficacy and safety of medicines, as well as other factors such as costs. Clinical trials provide the pivotal evidence for drug efficacy and safety. The ability to efficiently identify and make use of the results of existing clinical trials is critical to evidence-based policy decision making.
Until recently, journal publications were the only generally available source of trial designs and results. Thus, systematically reviewing the medical literature for the clinical trials that address a specific topic is of central importance to evidence-based health care policy [1, 2]. This provides decision makers with a coherent overview of the current evidence, and also helps to set the agenda for future clinical research [3, 4]. However, systematic reviewing is currently not feasible for most decision makers, because it is time consuming and expensive.
Therefore, most decision makers will have to rely on published systematic reviews. However, this is problematic because the review may not match the needs of the decision maker. Thus, even when a relevant systematic review is available, there may be a need to go back to the underlying trial data, especially for quantitative decision modeling. It may additionally be necessary to update or extend the review, or to combine several of them. Doing so also requires access to the underlying trial data, but these are not commonly reported. This is a serious limitation to the efficiency of both evidence-based decision making and systematic reviewing.
Thus, the quality of health care policy could be improved if systematic reviews could be performed for whatever decision is currently at hand, ideally even on demand. This would require enormous improvements to the manner in which clinical trials evidence is made available. Although efforts to standardize the information systems for the management and regulatory submission of clinical trials have been successful [5–7], this has not so far resulted in similar improvements in the dissemination of clinical trial evidence. A comprehensive overview of the various information systems that store and process clinical trials information could identify the gaps in information transfer that limit the efficiency of systematic reviews and consequently health care policy decision making.
The need to identify and summarize the evidence for decision makers is evident from the sheer scale of the available information: PubMed alone indexes nearly 20 million publications from over 5,500 journals, and this is only a selected subset of the biomedical literature . Systematic review addresses this need and consists of three steps: literature screening, data extraction, and reporting. In the first step, literature databases are searched, yielding a set of potentially relevant publications. These are screened for suitability, which results in them being included in, or excluded from, the review. Because literature searches are often inaccurate, thousands of publications may need to be screened. Moreover, to ensure comprehensiveness and avoid bias, multiple databases have to be searched  and multiple publications of a single trial have to be identified as such. Once the relevant trials have been identified and the corresponding reports retrieved, the data have to be extracted from the reports. Finally, the collected data are summarized and combined (e.g. using meta-analysis), and reported in a journal article or a technical report. Typically only this final product is made available, even though making the results of the screening step and the extracted data available would greatly enhance the efficiency of future systematic reviews and decision making. Thus, to assess the efficiency of clinical trials results dissemination, systematic review should not be ignored.
The difficulty of performing a systematic review also impacts the quality of systematic reviews themselves: it leads to reviews that focus on a single treatment or a pair of treatments. Consequently, for one particular therapeutic indication many competing reviews may be available, that each provide only a small part of the overall picture . This has led to ‘overviews of reviews’ or ‘umbrella reviews’ summarizing the results of several existing reviews . Umbrella reviews generally merely repeat the pooled summaries of treatment effects from the original reviews, but it has been argued that they may lead to misleading and inconsistent conclusions . An approach based on the individual studies is therefore preferable but labor-intensive if the data are not available in a structured format.
Scope and objectives
The aim of this paper is to identify opportunities to enhance the efficiency of systematic review and evidence-based decision making, supported by a broad and useful overview of the current state of the art in the transfer and availability of clinical trial evidence. To these ends, we provide a critical overview of existing systems and standards that support the dissemination of clinical trial results.
Because publicly available clinical trial results are nearly always aggregated (at the population level) rather than reported per patient, and because most decision makers base their decisions on such data, we limit the scope of this paper to systems and standards for the aggregate level.
We included academic publications and websites of manufacturers or standardization bodies that describe information systems or standards that deal with the transfer and availability of aggregate-level results of clinical trials. We also considered review articles and peer-reviewed position papers related to such information systems.
We identified relevant publications through key word searches using Google, Google Scholar, ISI Web of Science and PubMed (last searched May 2011). We also screened the reference lists of included publications. In addition, through our participation in the Escher project of the Dutch Top Institute Pharma (TI Pharma), we were able to engage in discussions with many experts from the pharmaceutical industry, regulatory authorities, and academia.
Publications (both peer-reviewed articles and web pages published by companies or standardization bodies) were screened for eligibility using titles and abstracts (if applicable). Potentially relevant publications were read in full. If a peer-reviewed article and a web page conveyed (nearly) identical information, only the peer-reviewed work was included. Moreover, web pages were excluded if the source was not considered authoritative for the subject matter. Included publications were summarized using keywords, and especially important sections were highlighted for later reference. For each system and standard we collected the context in which it is used, its purpose, its defining features, the types of data it handles and/or produces, its connection to other systems or standards, and expected future developments.
Pharmaceutical industry and other investigators may choose to summarize selected results of clinical trials in manuscripts submitted to peer-reviewed scientific journals. A clinical trial may result in any number of publications, from none to dozens. Unfortunately, such publications frequently lack sufficient information to allow the reader to judge whether the trial was rigorously conducted. The CONSORT statement [11, 12] aims to improve the situation by providing guidance on the proper reporting of clinical trials. Nevertheless, trial reporting is often still inadequate  and selective outcome reporting is common .
Coverage of abstract databases PubMed and EMBASE, and the Cochrane Library
PubMed/MEDLINE (1 Jan 2011)
Identified clinical trial
Identified randomized controlled trial
(2) 17 May 2011
EMBASE (17 May 2011)
∼ 24 M
Cochrane library (17 May 2011)
Clinical trials (CENTRAL)
(4) issue 2 of 4, Apr 2011
(4) issue 5 of 12, May 2011
(4) issue 2 of 4, Apr 2011
All reviews (Cochrane + Other)
(4) issue 5 of 12, May 2011
When clinical trial results were only published in scientific journals, the decision whether to publish the results was completely left to the investigator, which led to incomplete trial reporting and publication bias . For example, over half of the clinical trials that supported successful new drug applications made to the FDA had still not been published 5 years after the medicines’ market approval . This is a serious problem that can lead to incorrect conclusions from a systematic review.
As early as 1986 the registration of trials in advance was proposed as a solution to publication bias . The trial bank concept proposes to take the registration of trials even further by recording not only the existence of a trial, but also the study protocol (in advance) and the results (after completion) in a “machine readable” way [15, 24]. For trial banks to be successful, all trials must be entered in a way that conforms to a single machine-readable data model [24, 25]. The Global Trial Bank project was set up in 2005 to create a practically usable trial bank , but in 2008 it was put on hold due to lack of funding .
The FDA Modernization Act of 1997 made the US the first country to make trial registration a legal requirement. To implement this legislation, the ClinicalTrials.gov registry was launched in February 2000 . The initial installment focused on providing a record of trials for enabling patient recruitment and investigator accountability. In 2004, both the World Health Organization (WHO) and the International Committee of Medical Journal Editors (ICMJE) released statements in support of the prospective registration of clinical trials. This policy has been widely adopted and now assures that the existence of most recent trials is known . Subsequently, various organizations, including the WHO, have called for a full disclosure of the trial protocol (including amendments) and results [30–36]. In the US, recent legislation  has required protocol registration since December 2007, basic results reporting since September 2008, and Adverse Drug Events (ADEs) reporting since September 2009 . Other governments with policies requiring prospective registration include the EU, India, Argentina, Brasil, Israel and South Africa .
To register a trial in ClinicalTrials.gov , researchers enter summary protocol information  when their studies are initiated, and subsequently create the results section  when the data collection for at least one primary outcome measure is complete. The ClinicalTrials.gov staff will review the results data after their submission. The data are reported in a structured tabular format and some meta-data, such as units of measurement or the use of standard vocabularies, can also be provided. Limited support for reporting statistical analyses is offered; these analyses are tied to specific results tables. Study protocols have long been available in XML format, and the retrieval of results in XML format was added in December 2011 .
Other countries have set up their own registries. Since 2004, the European Medicines Agency (EMA) has established clinical trial registration in accordance with the EU Directive 2001/20/EC through the EudraCT system. EudraCT was opened to the public only recently as the EU Clinical Trials Register, on 22 March 2011 , and the records are being released in a staggered fashion. Currently 17,102  of over 28,150 registered trials  are available. Another international registry is the Current Controlled Trials Ltd.’s ISRCTN registry, which has been in operation since 1998 . It provides a semi-structured textual representation of the trial protocol, but no results. A number of countries have open national registries that generally record information in a way similar to ISRCTN. All of these registries are less sophisticated than ClinicalTrials.gov.
ClinicalTrials.gov and the 14 WHO primary registries, with the number of registered trials per 19 March 2012 (‘studies’), whether the register is indexed by the WHO search portal (‘indexed’) and whether the registry also enables results publication (‘results’)
ClinicalTrials.gov (United States)
European Union Clinical Trials Register
ISRCTN register (international)
Japan Primary Registries Network
Australian New Zealand Clinical Trials Registry
The Netherlands National Trial Register
Clinical Trials Registry - India
Iranian Registry of Clinical Trials
Chinese Clinical Trial Register
German Clinical Trials Register
Cuban Public Registry of Clinical Trials
South Korea Clinical Research Information Service
Brazilian Clinical Trials Registry
Pan African Clinical Trial Registry
Sri Lanka Clinical Trials Registry
Although a decentralized system of federated registries (both national and multi-national) seems cumbersome and may cause duplicate registration, there are important reasons why this method is to be preferred to a centralized approach : national registries, for example, are in the position to ensure complete registration in their region of influence and are perfectly aligned with the local political situation. As long as the different registries are sufficiently interoperable, an overarching organization such as the ICTRP can aggregate their databases.
The increased transparency enabled by trial registration offers new opportunities for evidence-based medicine and will likely lead to an increase in the number of systematic reviews that are undertaken . However, the current registries contain only text-based or semi-structured information and lack a common coding system, for example for labeling interventions. The amount of protocol information registered is often insufficient to judge the validity of reported results and the problem of identifying all relevant studies has not yet been solved . In addition, the publicly available information may be incomplete or even “largely incomprehensible” . The call for federated, open access, mandatory results databases continues [32–36], and it is likely that the trend toward open and complete registration of results will continue.
The process of systematic review produces data and meta-data that is potentially useful for future reviewers and decision makers (see Background). The following assesses whether the existing information systems enable the dissemination of this information.
The Cochrane Collaboration provides several databases to support reviewers. Besides the CENTRAL database of clinical trials (see Section Scientific literature) the most relevant ones are the Cochrane Database of Systematic Reviews, in which Cochrane reviews are published , and the DARE database of other reviews. Table 1 provides statistics regarding the scope of the library. In contrast to the traditional journal publications of systematic reviews, which usually provide data in tables or figures, the Cochrane Reviews incorporate descriptions and results of the original studies. However, the published dataset has many data elements removed and the use of the data is restricted by license.
There are several software programs to aid in systematic review and meta-analysis, such as Comprehensive Meta-Analysis, MetaWin, MetaStat and MetaAnalyst. Moreover, many general-purpose statistical programs, such as SPSS Statistics, SAS Statistics, Stata, and R, have meta-analysis functionality. In general, dedicated meta-analysis software will provide easier data entry and management, while statistical programs will offer more powerful tools for analysis. The Cochrane Collaboration also provides the Review Manager software for performing systematic reviews (http://ims.cochrane.org/revman). Review Manager is unique in that it provides not only data analysis and management features, but includes functionality to write the full systematic review report. Indeed, the Review Manager file itself is submitted to the Cochrane library for review and eventual publication. Unfortunately, all of these systems lack sufficient meta-data to enable automated processing.
Finally, published systematic reviews are usually presented in a textual format without the underlying dataset, making it difficult to perform additional analyses that may be required for decision making. Thus, the inclusion of data on a new trial or new compound, as would be required for regulatory decision making, is also impossible. In conclusion, systematic review currently represents a missed opportunity to introduce additional structure to the available clinical trials information.
After a pharmaceutical company develops a drug, it compiles the evidence collected from the discovery and development processes into a dossier that is submitted to the regulators who decide upon its market authorization. Submissions to the EMA and the national medicines boards in Europe are mainly text-based, containing aggregate-level results of clinical trials based on the applicant’s statistical analysis. The Food and Drug Administration (FDA) requests the submission of patient-level data, which is unlikely to become publicly available and as such is out of scope for this paper. The dossier forms the basis on which regulators assess the benefit-risk profile of a new drug. Although the clinical trial results are pivotal in this assessment, the decision is only indirectly based on them, as the decision making process is based on informal discussion between experts. While the decision may still be of high quality, this informal framework does not allow pharmaceutical companies or patients to discern how different pieces of the evidence weigh in on it.
The EMA publishes the European Public Assessment Reports (EPARs) of all centrally approved or refused medicines on its website. Note that this does not include all applications submitted to the EMA, as they can be withdrawn before a decision is reached . The EPAR contains information on all trials, but is completely textual without a semantic structure. Moreover, its information is directly derived from the submission by the applicant, while there is no standardization concerning what information should be provided, or in which format. Trials submitted to the EMA are required to be registered in EudraCT and will thus also be made known to the public through the EU Clinical Trials Register.
Standards and data models
A common standard of how clinical trials are performed and how their results are presented would make the process of systematic review more reliable and less laborous . Some progress has been made by ClinicalTrials.gov, which currently registers and displays aggregate results. To do so, ClinicalTrials.gov has developed their own model, the Data Element Definitions (DED) [41, 42]. This model allows the reporting of aggregated outcome data and statistical analyses to some extent, but the information cannot be processed automatically because most fields are free text. This also means that finding all trials that are relevant to a specific patient condition is inaccurate, and thus requires overly broad search terms . This lack of standardization and interoperability among registries and other databases should be addressed in the near future.
Several projects aim to enable general purpose re-use of clinical trials information, e.g. for cross-study analyses. We identified three such projects. The first is the Biomedical Research Integrated Domain Group (BRIDG) project, a collaboration between the Clinical Data Interchange Standards Consortium (CDISC), Health Level 7 (HL7), the National Cancer Institute (NCI) and the FDA that aims to bring together the common elements of their various standards into a complete data model for clinical trials . The BRIDG model is implementation-independent in the sense that it specifies the problem domain, not a specific solution. For example, unlike some other CDISC standards it does not specify the format in which to submit data to the FDA. BRIDG is subdivided between the protocol representation, study conduct, adverse event and regulatory perspectives. Unfortunately, a data analysis perspective is currently missing as there is no adequate standard for the modelling of statistical analyses. In short, the BRIDG model is accurate as regards the management of a single clinical trial, but not as regards cross-study analysis . For example, the study population and eligibility criteria, outcomes and the measures used to assess outcomes do not have a sufficiently deep semantic structure.
To enable cross-study analyses and efficiently finding relevant trials, the Human Studies Database (HSDB) project aims to share fully machine understandable representations of study design information between institutions . HSDB is developing the Ontology of Clinical Research (OCRe), which defines the concepts that should be accessible across the individual institutions’ databases. At the time of writing, OCRe included a study design representation derived from BRIDG , a study design typology , the ERGO formal machine readable representation of eligibility criteria , and a model of study outcomes that separates the phenomena of interest from the variables that encode them . While OCRe is a promising effort, its representation of study design is far from comprehensive, and it completely lacks a model for trial results.
Finally, the Ontology Based eXtensible conceptual model (OBX) is another ontology for representing clinical trials [57, 58]. Its aim is to make the results of immunology studies available for data re-use and re-analysis. The OBX also incorporates study design representation ideas from BRIDG and the ClinicalTrials.gov DED . While it appears successful in developing a broadly applicable data model for biomedical studies, and also allows the inclusion of trial results, it would appear that OBX suffers from similar shortcomings as regards the depth of modelling as BRIDG does.
All of the discussed models rely on an external coding system for their clinical content. Such coding systems, known as controlled terminologies of clinical terms, are an important first step in the application of information technology to medicine . There are many controlled terminologies for medicine, often developed for specific applications, but unfortunately there is as yet no standardization of which ones should be used, and there is no accurate mapping between them [60, 61]. For example, in clinical research the Medical Dictionary for Regulatory Activities (MedDRA) is used to code ADEs, while the healthcare area prefers the Systematized Nomenclature of Medicine, Clinical Terms (SNOMED CT) dictionary. This hinders the interoperability of the various information systems being used.
Having reviewed the information systems and standards dealing with the information from clinical trials, we will now summarize their deficiencies concerning the integration of clinical trials information from different resources, discuss how the status quo could be improved and identify directions for future research.
Systems and standards oriented towards the management of single studies are relatively mature, but this is not the case for cross-study analysis. There are no known large, successful, and publicly available data warehouses, nor any standards that would enable cross-study analyses of aggregate level results. Considerable effort is required to harmonize the current clinical research standards. Important areas that require standardization are the representation of statistical analyses and aggregate results, as well as complex semantic structures such as patient eligibility criteria. None of the general purpose data models being developed are yet in widespread use, and from the perspective of capturing the designs and results of clinical trials in a reusable way, none of them are close to completion.
Although much effort is spent to publish the results of clinical trials, the current systems do not facilitate optimal use of the information. The journals and abstract databases that publish the trial results do not preserve the results’ structure and thus require manual data extraction. Moreover, relevant articles are hard to identify and the retrieval of all available studies cannot be guaranteed. Public registries are meant to improve the efficiency and reliability of the identification of relevant studies, but the available data is not sufficiently structured to realize this. Moreover, the systems that currently deal with clinical trials results are not interlinked nor do they use interoperable standards. In short, there is not yet a comprehensive system of structured machine-understandable databases that contains descriptions of the design, execution, and summary-level results of individual trials. This situation hinders systematic review and makes cross-study analyses and data-mining prohibitively difficult. Thus, current infrastructure is focused on text-based reports of single studies, whereas efficient evidence-based medicine requires the automated integration of multiple clinical trials from different information resources.
Moreover, while systematic review collects and appraises the available evidence that is relevant to a certain question, the results are published in an unstructured format. This makes it hard to use the underlying data to inform evidence-based decisions, to verify the analyses, to update the review or to perform a combined analysis of several reviews for an umbrella review. The effort spent on literature screening and data extraction does not result in availability of this information for future reviewers, leading to duplication of effort.
Importantly, the perceived lack of transparency in regulatory decision making may erode public trust in drug regulation and the pharmaceutical industry. More explicit quantitative decision models would enable a more transparent and reproducible regulatory process, as well as a clearer communication of the requirements to the industry. However, for most real-world decisions it is currently too expensive to include all the evidence. This difficulty of accessing existing data is not only relevant to regulators and the industry, but also to reimbursement organizations, prescribing physicians and patients.
Proposed future situation
Now, we consider how these deficiencies might be addressed. Let us assume for a moment that a comprehensive machine-understandable standard were available for the design and aggregate level results of clinical trials. Then, it would be better for those submitting the data if both regulators and registries used this format, rather than a number of disparate formats. In addition, journal publications could easily be supplemented with data in this format.
Availability of a standard alone, however, is not enough to enable efficient access to the evidence. The data sets also need to be collected and made available in such a way that relevant clinical trials are easily identified. For this a collaborative (federated) system of databases should be established to capture all clinical trials data. Some of the stakeholders (e.g. regulators and registries) may require that data be submitted to a database that they control so that they can ensure the integrity of the data. This is fine as long as (1) the databases are interoperable and enable access to the information in the same format, (2) there is a single point of access through which the different databases can be identified and located, and (3) duplicate entries can be easily identified. It seems more likely for such a combination of databases to emerge from the current registries than a single centralized system.
However, a suitable general-purpose data model is not likely to become available in the near future. Further, the usefulness of any data model should be demonstrated to the industry and other stakeholders before putting it into practice. We argue that the requirements for a general purpose data model for cross-study analysis and decision making are not sufficiently well known at the moment. Therefore, analysis tools and decision support systems that ensure data extraction done for the analysis can be shared with other researchers should be developed first, to illuminate these requirements.
In order to attain the desired future system for aggregate-level clinical trial results dissemination, we identify concrete research directions for medical informatics, decision making and statistics researchers. Progress on each of these topics can be made in parallel.
policy decision making based on clinical trials – to enable a direct and explicit link between the decision and the supporting evidence in drug regulation, reimbursement policy and guideline formulation
Development of a platform to share structured systematic review data sets
Discovery or creation of incentives for systematic reviewers to share the results of literature screening and data extraction
Identification of the core data elements and modeling that are needed to increase the accuracy of literature searches
Automated tagging and data extraction to facilitate transition to more structured data sets
Development of search tools to integrate querying of abstract databases and registries
Development of methods to identify duplicate trial publications and registrations
Development of a comprehensive data model for clinical trials and their aggregate level results
As with any review paper, there is a risk that relevant publications have not been identified, either because the search terms were not broad enough, or because relevant studies were not identified as such based on their title and abstract. We acknowledge that the broad scope of this particular review increases that risk.
The nature of the collected information necessitates a qualitative synthesis, and the identified deficiencies are at least partially subjective. The future that we propose is based on the premise that a standard for aggregate clinical trial data will become available. Unfortunately, it is unclear how and when this could be realized. Finally, the list of proposed research directions is sure to be incomplete, and we hope the present paper will ignite discussions on this topic.
We reviewed the existing systems and standards dealing with aggregate level results of clinical trials. The transfer of evidence to scientific journals, public registries, and regulators is a largely ad hoc and text-based affair. In part, this is because there are currently no data standards that enable cross-study analyses. We have argued that the lack of a standardized, federated system for results dissemination leads to gaps in the transfer and availability of evidence to the relevant decision makers. As long as such a system does not exist, systematic review will remain an incredibly inefficient ad-hoc process, and evidence-based decision making will remain unnecessarily difficult. We believe that these difficulties lead to a lack of transparency in health care policy decision making, which threatens public trust in the decision makers.
In the future, results registries and regulatory systems should be harmonized and federated to create a system of databases that forms the core of a more automated and efficient process of systematic review and evidence-based decision making. In addition, systematic reviews are currently a missed opportunity to introduce additional structure to the domain of clinical trials information, which should be addressed by more complete dissemination of their results. Although this vision is still far from realized, current trends seem to support this direction. Future work should not only focus on developing the ‘ideal’ data model for all of clinical research (justly called a monumental task) but start by creating useful tools to support decision makers and systematic reviewers. Availability of such tools will lead to increased demand for an accessible evidence base and to a better understanding of its requirements.
GvV is a PhD student for the Escher project of Top Institute Pharma, working on evidence-based decision support for medicines regulation. He has an MSc in Artificial Intelligence.
TT is an assistant professor of Business Intelligence Systems at the Econometric Institute of the Erasmus University Rotterdam. Formerly, he was a post-doc researcher for the Escher project. He has a double PhD degree in Computer Science (University of Turku, Finland) and Management Science (University of Coimbra, Portugal). His main expertise is in Multi-Criteria Decision Analysis (MCDA).
BdB is a professor of Business Information Modelling at the University of Groningen. He is interested in databases and information modelling with interdisciplinary applications to bioinformatics and medicine.
HH is a professor of Cardiology at the University Medical Center Groningen. There, he is also director of the Trial Coordination Center and head of the Data-Management Project. Moreover, he is a clinical assessor for the Dutch Medicines Evaluation Board and clinical expert for the European Medicines Agency (EMA).
This study was performed in the context of the Escher project (T6-202) of Top Institute Pharma.
- Chalmers I: The lethal consequences of failing to make use of all relevant evidence about the effects of medical treatments: the need for systematic reviews. Treating individuals: from randomised trials to personalised medicine. Edited by: Rothwell P. 2007, Edinburgh: Elsevier, 37-58.Google Scholar
- Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn’t. BMJ. 1996, 312 (7023): 71-72. 10.1136/bmj.312.7023.71.View ArticlePubMedPubMed CentralGoogle Scholar
- Mulrow CD: Rationale for systematic reviews. BMJ. 1994, 309 (6954): 597-599. 10.1136/bmj.309.6954.597.View ArticlePubMedPubMed CentralGoogle Scholar
- Sutton AJ, Cooper NJ, Jones DR: Evidence synthesis as the key to more coherent and efficient research. 2009, 9 (29): e-publication.Google Scholar
- Bleicher P: Clinical trial technology: at the inflection point. Biosilico. 2003, 1 (5): 163-168. 10.1016/S1478-5382(03)02373-4.View ArticleGoogle Scholar
- El Emam K, Jonker E, Sampson M, Krleza-Jeric K, Neisa A: The Use of Electronic Data Capture Tools in Clinical Trials: Web-Survey of 259 Canadian Trials. J Med Internet Res. 2009, 11: e8-10.2196/jmir.1120.View ArticlePubMedPubMed CentralGoogle Scholar
- van Valkenhoef G, Tervonen T, de Brock, Hillege H: Clinical trials evidence in drug development and regulation: a survey of existing systems and standards. SOM Research Report 12003-Other, School of Management, Faculty of Economics and Business, University of Groningen, Groningen, The Netherlands 2012, [http://irs.ub.rug.nl/dbi/4fcf224db9977]
- Crumley ET, Wiebe N, Cramer K, Klassen TP, Hartling L: Which resources should be used to identify RCT/CCTs for systematic reviews: a systematic review. BMC Med Res Methodol. 2005, 5: 24-10.1186/1471-2288-5-24.View ArticlePubMedPubMed CentralGoogle Scholar
- Caldwell DM, Welton NJ, Ades AE: Mixed treatment comparison analysis provides internally coherent treatment effect estimates based on overviews of reviews and can reveal inconsistency. J Clin Epidemiol. 2010, 63 (8): 875-882. 10.1016/j.jclinepi.2009.08.025.View ArticlePubMedGoogle Scholar
- Ioannidis JPA: Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. Can Med Assoc J. 2009, 181 (8): 488-493. 10.1503/cmaj.081086.View ArticleGoogle Scholar
- Begg C, Cho M, Eastwood S, Horton R, Moher D, Olkin I, Pitkin R, Rennie D, Schulz KF, Simel D, Stroup DF: Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA. 1996, 276 (8): 637-639. 10.1001/jama.1996.03540080059030.View ArticlePubMedGoogle Scholar
- Schulz KF, Altman DG, Moher D, for the CONSORT group: CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. PLoS Med. 2010, 7 (3): e1000251-10.1371/journal.pmed.1000251.View ArticlePubMedPubMed CentralGoogle Scholar
- Chan AW, Altman DG: Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005, 365 (9465): 1159-1162. 10.1016/S0140-6736(05)71879-1.View ArticlePubMedGoogle Scholar
- Chan A, Hrobjarttson A, Haahr M, Gotzsche P, Altman D: Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. J Am Med Assoc. 2004, 291 (20): 2457-2465. 10.1001/jama.291.20.2457.View ArticleGoogle Scholar
- Sim I, Owens DK, Lavori PW, Rennels GD: Electronic Trial Banks: A complementary method for reporting randomized trials. Med Decis Making. 2000, 20 (4): 440-450. 10.1177/0272989X0002000408.View ArticlePubMedGoogle Scholar
- Haynes RB, McKibbon KA, Wilczynski NL, Walter SD, Werre SR: Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey. BMJ. 2005, 330 (7501): 1179-10.1136/bmj.38446.498542.8F.View ArticlePubMedPubMed CentralGoogle Scholar
- Boudin F, Nie JY, Bartlett JC, Grad R, Pluye P, Dawes M: Combining classifiers for robust PICO element detection. BMC Med Inform Decis Mak. 2010, 10: 29-10.1186/1472-6947-10-29.View ArticlePubMedPubMed CentralGoogle Scholar
- Karimi S, Pohl S, Scholer F, Cavedon L, Zobel J: Boolean versus ranked querying for biomedical systematic reviews. BMC Med Inform Decis Mak. 2010, 10: 58-10.1186/1472-6947-10-58.View ArticlePubMedPubMed CentralGoogle Scholar
- Dickersin K, Manheimer E, Wieland S, Robinson KA, Lefebvre C, McDonald S: Development of the Cochrane Collaboration’s central register of controlled clinical trials. Eval Health Prof. 2002, 25 (38): 38-64.View ArticlePubMedGoogle Scholar
- The Cochrane Foundation: About The Cochrane Library,. 2010 [http://www.thecochranelibrary.com/view/0/AboutTheCochraneLibrary.html]. [http://www.webcitation.org/5tEEbQuNz]
- Dickersin K, Rennie D: Registering clinical trials. J Am Med Inform Assoc. 2003, 290 (4): 516-523.View ArticleGoogle Scholar
- Lee K, Bacchetti P, Sim I: Publication of Clinical Trials Supporting Successful New Drug Applications: A Literature Analysis. PLoS Med. 2008, 5 (9): e191-10.1371/journal.pmed.0050191.View ArticlePubMedPubMed CentralGoogle Scholar
- Simes RJ: Publication bias - the case for an international registry of clinical-trials. J Clin Oncol. 1986, 4 (10): 1529-1541.PubMedGoogle Scholar
- Sim I: Trial Banks: An Informatics Foundation for Evidence-Based Medicine. PhD thesis, Stanford University: Stanford, CA, USA 1997
- Sim I, Olasov B, Carini S: An ontology of randomized controlled trials for evidence-based practice: content specification and evaluation using the competency decomposition method. J Biomed Inform. 2004, 37 (2): 108-119. 10.1016/j.jbi.2004.03.001.View ArticlePubMedGoogle Scholar
- Sim I, Detmer D: Beyond trial registration: A global trial bank for clinical trial reporting. PLoS Med. 2005, 2 (11): 1090-1092.View ArticleGoogle Scholar
- AMIA: AMIA Global Trial Bank. 2010, [https://www.amia.org/global-trial-bank]. [http://www.webcitation.org/5ma1fjB1A]
- McCray AT, Ide NC: Design and implementation of a national clinical trials registry. JAMIA. 2000, 7 (3): 313-323.PubMedPubMed CentralGoogle Scholar
- Zarin DA, Ide NC, Tse T, Harlan WR, West JC, Lindberg DAB: Issues in the registration of clinical trials. JAMA. 2007, 297 (19): 2112-2120. 10.1001/jama.297.19.2112.View ArticlePubMedGoogle Scholar
- Krleza-Jeric K, Chan AW, Dickersin K, Sim I, Grimshaw J, Gluud C, the Ottawa Group: Principles for international registration of protocol information and results from human trials of health related interventions: Ottawa statement (part 1). BMJ. 2005, 330 (7497): 956-958. 10.1136/bmj.330.7497.956.View ArticlePubMedPubMed CentralGoogle Scholar
- Sim I, Chan AW, Gulmezoglu AM, Evans T, Pang T: Clinical trial registration: transparency is the watchword. Lancet. 2006, 367 (9523): 1631-1633. 10.1016/S0140-6736(06)68708-4.View ArticlePubMedGoogle Scholar
- Kaiser J: Making Clinical Data Widely Available. Science. 2008, 322 (5899): 217-218. 10.1126/science.322.5899.217.View ArticlePubMedGoogle Scholar
- Ghersi D, Clarke M, Berlin J, Guelmezoglu AM, Kush R, Lumbiganon P, Moher D, Rockhold F, Sim I, Wager E: Reporting the findings of clinical trials: a discussion paper. Bull of the World Health Organization. 2008, 86 (6): 492-493.View ArticleGoogle Scholar
- Zarin DA, Tse T: Medicine - Moving toward transparency of clinical trials. Science. 2008, 319 (5868): 1340-1342. 10.1126/science.1153632.View ArticlePubMedPubMed CentralGoogle Scholar
- Chan AW: Bias, Spin, and Misreporting: Time for Full Access to Trial Protocols and Results. PLoS Med. 2008, 5 (11): e230-10.1371/journal.pmed.0050230.View ArticlePubMedPubMed CentralGoogle Scholar
- Sim I, Chute CG, Lehmann H, Nagarajan R, Nahm M, Scheuermann RH: Keeping Raw Data in Context. Science. 2009, 323 (5915): 713a-10.1126/science.323.5915.713a.View ArticleGoogle Scholar
- FDA: Food and Drug Administration Amendments Act of 2007,. Public Law 110-85 (September 27, 2007), Section 801 [From the U.S. Government Printing Office]
- Wood AJJ: Progress and Deficiencies in the Registration of Clinical Trials. N Engl J Med. 2009, 360 (8): 824-830. 10.1056/NEJMsr0806582.View ArticleGoogle Scholar
- ICTRP: About Trial Registration: Organizations with Policies. 2010, [http://www.who.int/ictrp/trial_reg/en/index2.html]. [http://www.webcitation.org/5mZpckdBf]
- Tse T, Williams RJ, Zarin DA: Reporting “Basic Results” in ClinicalTrials.gov. Chest. 2009, 136: 295-303. 10.1378/chest.08-3022.View ArticlePubMedPubMed CentralGoogle Scholar
- ClinicalTrialsgov: ClinicalTrials.gov Protocol Data Element Definitions (DRAFT). 2009, [http://prsinfo.clinicaltrials.gov/definitions.html]. [http://www.webcitation.org/5mYe6OxvP]
- ClinicalTrialsgov: ClinicalTrials.gov “Basic Results” Data Element Definitions (DRAFT). 2009, [http://prsinfo.clinicaltrials.gov/results_definitions.html]
- ClinicalTrialsgov: Linking to ClinicalTrials.gov. 2011, [http://clinicaltrials.gov/ct2/info/linking]. [http://www.webcitation.org/66HHPj8RI]
- European Medicines Agency: About EU Clinical Trials Register. 2011, [https://www.clinicaltrialsregister.eu/about.html]. [http://www.webcitation.org/5yNkGPZZX]
- European Medicines Agency (EMA): EU Clinical Trials Register. 2012, [https://www.clinicaltrialsregister.eu/]. [http://www.webcitation.org/66HHLkoV1]
- European Medicines Agency: EudraCT Public Web Report for February 2011. 2011, [https://eudract.ema.europa.eu/docs/statistics/EudraCT_Statistics_February.pdf]
- Current Controlled Trials Ltd: Current Controlled Trials: Frequently asked questions. 2010, [http://www.controlled-trials.com/information/faqs]. [http://www.webcitation.org/5mbV14caj]
- The Ministerial summit on health research: The Mexico statement on health research. 2004, [http://www.who.int/rpc/summit/agenda/en/mexico_statement_on_health_research.pdf]. [Mexico City, November 16-20]
- Grobler L, Siegfried N, Askie L, Hooft L, Tharyan P, Antes G: National and multinational prospective trial registers. Lancet. 2008, 372 (9645): 1201-1202. 10.1016/S0140-6736(08)61498-1.View ArticlePubMedGoogle Scholar
- Honig PK: Systematic Reviews and Meta-Analyses in the New Age of Transparency. Clin Pharmacol Ther. 2010, 88 (2): 155-158. 10.1038/clpt.2010.124.View ArticlePubMedGoogle Scholar
- Starr M, Chalmers I: The evolution of The Cochrane Library, 1988–2003. 2003, [http://www.update-software.com/history/clibhist.htm]
- Eichler HG, Aronsson B, Abadie E, Salmonson T: New drug approval success rate in Europe in 2009. Nat Rev Drug Discov. 2010, 9: 355-356. 10.1038/nrd3169.View ArticlePubMedGoogle Scholar
- Tu SW, Peleg M, Carini S, Bobak M, Ross J, Rubin D, Sim I: A practical method for transforming free-text eligibility criteria into computable criteria. J Biomed Inform. 2011, 44 (2): 239-250. 10.1016/j.jbi.2010.09.007.View ArticlePubMedGoogle Scholar
- Biomedical Research Integrated Domain Group (BRIDG): BRIDG Model Release 3.0.3 User’s Guide. 2010, [http://gforge.nci.nih.gov/frs/?group_id=342]
- Sim I, Carini S, Tu S, Wynden R, BH P, Mollah S, Gabriel D, Hagler H, Scheuermann R, Lehmann H, Wittkowski K, Nahm M, Bakken S: The Human Studies Database Project: Federating Human Studies Design Data Using the Ontology of Clinical Research. Proceedings of the AMIA CRI Summit. 2010, AMIA, 2010. http://proceedings.amia.org/ Google Scholar
- Carini S, Pollock BH, Lehmann HP, Bakken S, Barbour EM, Gabriel D, Hagler HK, Harper CR, Mollah SA, Nahm M, Nguyen HH, Scheuermann RH, Sim I: Development and evaluation of a study design typology for human research. AMIA Annu Symp Proc. 2009, 2009: 81-85.PubMedPubMed CentralGoogle Scholar
- Kong YM, Dahlke C, Xiang Q, Qian Y, Karp D, Scheuermann RH: Toward an ontology-based framework for clinical research databases. J Biomed Inform. 2011, 44: 48-58. 10.1016/j.jbi.2010.05.001.View ArticlePubMedGoogle Scholar
- Scheuermann RH: Ontology-Based eXtensible Data Model. 2010, [http://pathcuric1.swmed.edu/Research/scheuermann/OBX.html]. [http://www.webcitation.org/5y3tihqvB]
- Cimino JJ: Review paper: coding systems in health care. Methods Inf Med. 1996, 35 (4–5): 273-284.PubMedGoogle Scholar
- Nadkarni PM, Darer JD: Determining correspondences between high-frequency MedDRA concepts and SNOMED: a case study. BMC Med Inform Decis Mak. 2010, 10: 66-10.1186/1472-6947-10-66.View ArticlePubMedPubMed CentralGoogle Scholar
- Vikstrom A, Skaner Y, Strender LE, Nilsson GH: Mapping the categories of the Swedish primary health care version of ICD-10 to SNOMED CT concepts: rule development and intercoder reliability in a mapping trial. BMC Med Inform Decis Mak. 2007, 7: 9-10.1186/1472-6947-7-9.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/12/95/prepub