Relemed: sentence-level search engine with relevance score for the MEDLINE database of biomedical articles
© Siadaty et al; licensee BioMed Central Ltd. 2007
Received: 12 April 2006
Accepted: 10 January 2007
Published: 10 January 2007
Receiving extraneous articles in response to a query submitted to MEDLINE/PubMed is common. When submitting a multi-word query (which is the majority of queries submitted), the presence of all query words within each article may be a necessary condition for retrieving relevant articles, but not sufficient. Ideally a relationship between the query words in the article is also required. We propose that if two words occur within an article, the probability that a relation between them is explained is higher when the words occur within adjacent sentences versus remote sentences. Therefore, sentence-level concurrence can be used as a surrogate for existence of the relationship between the words.
In order to avoid the irrelevant articles, one solution would be to increase the search specificity. Another solution is to estimate a relevance score to sort the retrieved articles. However among the >30 retrieval services available for MEDLINE, only a few estimate a relevance score, and none detects and incorporates the relation between the query words as part of the relevance score.
We have developed "Relemed", a search engine for MEDLINE. Relemed increases specificity and precision of retrieval by searching for query words within sentences rather than the whole article. It uses sentence-level concurrence as a statistical surrogate for the existence of relationship between the words. It also estimates a relevance score and sorts the results on this basis, thus shifting irrelevant articles lower down the list.
In two case studies, we demonstrate that the most relevant articles appear at the top of the Relemed results, while this is not necessarily the case with a PubMed search. We have also shown that a Relemed search includes not only all the articles retrieved by PubMed, but potentially additional relevant articles, due to the extended 'automatic term mapping' and text-word searching features implemented in Relemed.
By using sentence-level matching, Relemed can deliver higher specificity, thus eliminating more false-positive articles. By introducing an appropriate relevance metric, the most relevant articles on which the user wishes to focus are listed first. Relemed also shrinks the displayed text, and hence the time spent scanning the articles.
MEDLINE® is the U.S. National Library of Medicine's (NLM) primary literature database, indexing >15 million citations in the fields of medicine, nursing, dentistry, veterinary medicine, the health care system, and the preclinical sciences . Encountering extraneous articles in response to a query submitted to MEDLINE/PubMed is not uncommon. However, every one of the articles retrieved contains all of the query words. This leads us to the conclusion that the presence of query words in an article is not a sufficient condition for the article to be relevant to user's query, although it is a necessary.
About 83% of queries sent to PubMed®, NLM's search engine for MEDLINE , are multi-word queries (see Additional File 1). When submitting a query with multiple words, the user is usually interested in some type of relationship  between the words, such that the "presence of relationship" between the query words in the article also becomes a necessary condition for relevance.
Examples of retrieval services for MEDLINE
NLM's search engine for MEDLINE
alternative search interface using slider controllers to implement search limits, methodology filters, and MeSH terminologies
free-text, natural language query tool for PubMed
inputs an entire paragraph and returns articles that are similar to it
a search engine to MEDLINE
shows first the articles that contain the search terms most frequently in the title and/or abstract
biologist-friendly interface for enhanced PubMed search
gives ranked list of relevant specialties that relate to topics discussed in each article
classifies the retrieved articles using Gene Ontology terms
A tool for summarizing the results of a PubMed query
A tool for identifying links between two sets of Medline articles
finds PDFs for PubMed citations
Tuning a search engine to attain two different scenarios of retrieval.
Scenario 1. Query with specificity of 99.99% is insufficient for a database of 16 million records.
records returned to user
Scenario 2. The price for a very high specificity: Missing a large number of relevant records.
records returned to user
In addition to trying to prevent irrelevant articles from appearing in the retrieved articles, one may also locate and isolate less relevant articles that have been retrieved. This can be done by estimating a relevance score for each retrieved article, and then sorting the articles by the score. Irrelevant retrieved articles will be shifted to the end of the list, effectively hidden from the user. Among the implemented information retrieval systems for MEDLINE, some do define relevance scores. However, these relevance scores are mainly based on place of occurrence and frequency of keywords extracted from the user's query. They do not incorporate the presence of a relationship between the query words.
We propose that if two words occur within an article, the probability that a relation between them is explained is clearly higher when the words occur within the same sentence (or adjacent sentences) versus remote sentences. This is a probabilistic expression of linguistic common sense. Therefore, sentence-level concurrence (co-occurrence) can be used as a surrogate for existence of the relationship between the words.
We have designed and implemented a publicly accessible search engine for MEDLINE. Our search engine, Relemed , retrieves relevant articles by detecting sentence-level concurrence of search terms. The search engine estimates a relevance score where presence of the relationship between the words is an important component of the score. To maintain high sensitivity while increasing specificity, the search engine utilizes article-level concurrence as the last level of relevance. In this paper we explain the Relemed retrieval system and its relevance score, and compare it to PubMed.
Through a lease contract with NLM, we obtained MEDLINE data in extensible markup language (XML) format. We designed and implemented algorithms to extract title, abstract, and citation information from each XML article record, then scanned through the abstract text to detect and separate sentences. To detect a sentence we used '.', '?', and '!' as delimiters. We then joined back consecutive sentences where the period was sandwiched by single capital letters, some specific words such as 'etc.' and 'et al.', or by digits such as '0.05'.
Database tables, and their fields
Database table 3a
PubMed ID number
sentence ID number
text of the sentence
Database table 3b
PubMed ID number
Citation information for the article
We designed and implemented a software application to receive a user's query, prepare the query in SQL (structured query language), interrogate the database, format the database results in HTML language (HyperText Markup Language), and post it back to the user's browser.
Queries submitted to Relemed can simply be composed of one or a few words, separated by space. By default, the system uses Boolean 'and' operator to connect the words. Also, Boolean operators 'or' and 'not' are supported. One can use asterisk * for truncation, parentheses () for grouping, and quotes "" for exact phrase matching. These are in accordance with PubMed query language.
We used the Unified Medical Language System  to implement 'automatic term mapping'. When a query is submitted to Relemed, synonyms for query words are found and added automatically to the query, using 'or' as the operator, thus improving the sensitivity of the search.
We used freely available open source software to build the search engine, including Perl to pre-process data and write the query application , MySQL to implement the database , and Apache to serve the user's HTTP requests (HyperText Transfer Protocol) . The server was installed with a Fedora operating system , hence the so-called LAMP architecture (Linux Apache MySQL Perl). XHTML (eXtensible HyperText Markup Language) was used to produce the user interface and the reports .
Given an article record, with title (one sentence), a few abstract sentences, and MeSH terms  (concatenated together and treated as one sentence), one can assign importance weights to each of the three sentence types (title, abstract, MeSH). Then one can combine the types to define several levels of 'relevance'. Thus one can try to measure how closely an article answers the user's query. Then one can sort the returned results by the relevance metric. This pushes the most relevant articles to the top of the result list, where the user would see the most relevant results first.
The eight relevance levels defined by Relemed.
Query must match
T and A and M
T and A
T and M
A and M
We conducted two case studies to evaluate the Relemed search engine, and compare it to PubMed. The topics were chosen from real cases encountered in our daily practice. To decrease evaluation bias we concealed the source of each article (Relemed or PubMed) from the raters (who evaluated the biomedical relevance of the articles). This was accomplished by presenting the articles in a unified format to the raters. The two questions addressed were: Q1. Given a query, is the collection of articles returned by Relemed the same as PubMed? Q2.Are the most relevant articles listed at the top of the Relemed results?
Starting with a query, we chose a pre-defined article count n, like 10. We queried Relemed with the query, and saved PMIDs of the first n articles within each relevance level, hence giving a total of 8n PMIDs. Likewise we presented PubMed with the same query, and saved the first 8n PMIDs. Then we wrote a program into which we fed the two lists of 8n PMIDs. The program made a unique list of PMIDs. Then the program queried the database for each PMID, and wrote an HTML report where the article contents (all fields available under the 'MEDLINE' format, including title, abstract, and MeSH) are included. Keywords were highlighted in the HTML report, to facilitate evaluation process. Nothing in the report indicated which search engine (Relemed or PubMed) retrieved each article. Two raters inspected the articles independently, and assigned true positive (TP) or false positive (FP) labels to each, thus defining the 'gold standard'. To resolve potential discordance between the two raters, a discussion was made on each of the discordant articles to reach a consensus. Then the program transferred the TP and FP assignments back to the query results of each of the PubMed and Relemed, thus 'breaking the blind'. Finally we estimated the precision (= positive predictive value, which is percentage of retrieved articles that are relevant) for each of the relevance levels of Relemed, and consecutive bins of size n in PubMed.
To analyze the precision data, and to attach statistical significance (by constructing 95% confidence bands for the precision curves), we used 'local regression' implemented in package 'locfit' of R statistical language [25, 26]. Also, to measure inter-rater agreement, we used Cohen's kappa, which measures the agreement between the evaluations of two raters when both are rating the same object.
In the Additional File #2 we present three more examples, further evaluating Relemed and comparing it to PubMed as benchmark.
Case study 1: role of 'infection' in 'sudden infant death syndrome' (SIDS)
SIDS is death of an infant less than one year old that cannot be explained after thorough medical investigation . Despite years of research, no definitive cause has been found, but there are many potential factors proposed by investigators, such as the position of baby during sleep, the use of a pacifier, history of parents' smoking, recent infection, change in temperature, etc. In this example the user wants to retrieve articles on SIDS that link infection as a potential cause of death in SIDS (or explains absence of such a relationship).
We used the query 'sids (infection or infect*)' in both PubMed and Relemed. We included the truncated word 'infect*' to automatically include all the variations of the word 'infect', such as infectious, infections, infective, etc. To include all other synonymous phrases (that do not necessarily contain the word 'infect'), we included the word 'infection'. This is necessary since the 'automatic term mapping' of the search engines only add synonyms for non-truncated words. We added the phrase '1900/1/1:2006/3/10 [dp]' to the query submitted to PubMed, to make the corpus of articles searched in the two search engines similar. This phrase limits "date of publication" to the range specified (March 10th was the last date we updated Relemed database for the purpose of this study).
Both the engines searched all articles in MEDLINE from the earliest available publication dates to 3/10/2006. PubMed returned 608 articles, whereas Relemed returned 927. Twenty nine out of 608 articles of PubMed were not included in the Relemed results. These 29 articles were of two groups. Group one was articles with a publication date of 3/10/2006 or earlier, but added to the MEDLINE after March 10, 2006. Since this was the last date Relemed database was updated (for the purpose of this study), these articles did not exist in Relemed. The second group was articles where no variation or synonym for 'infection' existed in any field, but since PubMed 'explodes' a term to all of the narrower terms in the MeSH hierarchy tree under it, terms like 'septicemia' and 'septic abortion', as well as 'corneal ulcer' and 'trachoma', were included in the PubMed search but not Relemed. Of 927 articles returned by Relemed, 338 were not found by PubMed, for two reasons: 1. some synonyms for SIDS are not recognized by PubMed. An example is 'cot death'. This term was more common during 70's and 80's. 2. The acronym 'sids' in the submitted query is mapped to 'sudden infant death'. However in PubMed this longer phrase is only used to match to MeSH terms and not to abstract or title, thus missing some articles.
Count of articles in each Relemed relevance level for the two case studies
Count of retrieved articles
Case study #1
Case study #2
A false positive article for query of case study #1, where query words do concur, both in text and in MeSH (but not in the same sentence).
DiFranza JR, Aligne CA, Weitzman M. Prenatal and postnatal environmental tobacco smoke exposure and children's health. Pediatrics. 2004 Apr;113(4 Suppl):1007-15. (PMID 15060193) ... A large literature links both prenatal maternal smoking and children's ETS exposure to decreased lung growth and increased rates of respiratory tract infection s, otitis media, and childhood asthma, with the severity of these problems increasing with increased exposure. Sudden infant death syndrome, behavioral problems, neurocognitive decrements, and increased rates of adolescent smoking also are associated with such exposures. ... [MeSH] drug effects. etiology. adverse effects. Animals. Asthma. etiology. Child. Child Behavior. drug effects. Embryonic and Fetal Development. Female. Humans. Infant. Intelligence. drug effects. Otitis Media. etiology. Pregnancy. Respiratory Tract Infection s. Smoking. adverse effects. Sudden Infant Death. etiology. Tobacco Smoke Pollution. analysis
Case study 2: finding 'questionnaires' for measuring 'health literacy'
Health literacy is the degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions . In this example, the user has a research project in which he wants to measure health literacy of the participants. He is interested in finding publications that give clues about existing questionnaires/instruments for health literacy.
We used the query
"health literacy" and (instrument* or question* or measur* or scale* or assessment* or index* or test*)
and PubMed returned 157 articles, whereas Relemed returned 158 of which 153 were shared with PubMed (a 96.8% overlap). There were 4 articles in PubMed that were absent from Relemed. All the four were articles with publication dates within the studied range (from the earliest publication date to 3/10/2006), but that have been added to the MEDLINE after March 10, 2006 (the last update for Relemed database). The five articles found by Relemed but not by PubMed contained the term 'health literacy' and 'test' in abstract or title, but still could not be retrieved by PubMed. These seem to be false negatives for PubMed.
The precision in PubMed for the first 28 articles was 39.3%, while precision for the first 28 articles of Relemed was estimated at 68.9%. The Kappa measure of inter-rater agreement was 0.496, which was significantly higher than chance (P-value < 0.001).
Comparison of information retrieval systems of MEDLINE
There are more than 30 retrieval services that use MEDLINE as their data source , some of which are shown in Table 1. Some use MEDLINE as the main or the only data source, such as PubMed, OVID, SLIM, askMEDLINE, and eTBLAST. Others use multiple databases, e.g. MedMiner. Some return articles as their main results (PubMed), while others return some digested form, such as a graph (Chilibot and ConceptLink). Some focus on data-mining, MedBlast and HAPI (High-density Array Pattern Interpreter); others on genomics or proteomics (GoPubMed and iHOP). Some are designed for "literature-based discovery", finding relationships between biomedical concepts from MEDLINE that are not expressed in any article directly, e.g. Arrowsmith and BITOLA. Some are specialized in the classification of articles, e.g. AnneOTate, CISMeF, and MedMOLE.
The majority of these services do not estimate relevance scores. None of them incorporate any relationship between the words in computing the relevance score.
OVID supports a 'proximity operator' where the user can ask for the two keywords to be within some specified distance (measured by the number of words separating them). However, this feature does not recognize sentence boundaries. For example, a word at end of a sentence is considered adjacent to the word in the beginning of the next sentence, and is treated the same way as when the two words were adjacent within the same sentence. Moreover, there is no automatic feature to utilize the adjacency operator, for sorting the resulting articles by increasing distance between the keywords matched per article. The user has to manually submit multiple queries with increasing proximity distances to be able to have a gradient of distances. Also note that word-proximity has less obvious cut-off values, compared to 'sentence' which is a more clear-cut linguistic unit.
PubMed has a feature called "Related Articles". After a search retrieves some articles, each article has a link that displays 'related articles' to it. These related articles in turn are sorted by a relevance score . However, this score does not incorporate the original query that the user submitted. In other words, given that many biomedical concepts can be expressed in an article, the article can be retrieved by very different queries sent by different users. However, in all these instances the related articles of the original article are exactly the same, irrespective of what concept the user was originally interested in. PubMed also gives the options to sort the search results by one of the four criteria: 1.Pub Date 2.First Author 3.Last Author 4.Journal. These do not necessarily reflect the relevance of an article to the user's query.
One may try to use some of the PubMed features to detect 'relation' between words for a multi-word query. Three methods come to mind: 1.One can limit the search to the titles only. Then if the (two) words appear in the title, it has a high probability that some sort of relation is declared between them in the article. Although this method could attain fairly high specificity, it may miss relevant articles because it does not utilize any of the sentences of the abstract, i.e. it is potentially of low sensitivity. 2. If the two or more words the user is asking have hierarchical relation in the MeSH, then MeSH can show high specificity. For example, when the user is interested in adverse effects of antidepressant therapy, the MeSH subheading 'adverse effects' to the MeSH heading 'antidepressive agents' is a good query. A similar case is when all the query words map to a single MeSH term. For example, query 'two dimensional gel electrophoresis' maps to "electrophoresis, gel, two-dimensional" [MeSH Terms]. In such cases many of the retrieved articles can be relevant. 3.If the query words are mainly used consecutively in the article text, one may be able to use quoting (the operator ""), in order to instruct PubMed to retrieve articles where the words appear exactly (in the same proximity and order) as they are in the quoted phrase. However, these are not common cases.
We emphasized that the majority of queries sent to MEDLINE/PubMed are multi-word queries, where two or more words are included in the query. For these queries, the user can be looking for articles that are about 1.each word, and 2.some relationship between the words. Currently, the retrieval systems of MEDLINE (including PubMed) identify articles with the requested words but not their relationship. Drawing on linguistics, the chance of the article claiming some relation between the two words is higher when they concur within a sentence than an article (or abstract). This was the basis for creating the Relemed search engine.
There is a limit to the amount of text a user is willing or able to scan. By using a sentence level matching, Relemed is able to deliver higher specificity, thus reducing false positive (FP) articles. Also, by introducing relevance metric, the most useful articles are shown first, where the user focuses most. By composing the matching sentences and highlighting the keywords, Relemed shrinks the text and the time the user spends for the 'scan & eliminate' process (where the user reads the titles or quickly scans the abstracts, and decides whether to eliminate the article or leave it for the next round of more in-depth screening). The two examples used in the paper demonstrated that the higher precision attained at the start of results in Relemed facilitates this type of screening.
Recognizing, however, that Relemed returns almost identical collection of articles as PubMed, one question is the location of the false positive articles in the Relemed results? We believe that Relemed's relevance levels 7 and 8 would contain majority of FPs.
There are limitations in the current implementation of Relemed. First, we note that since Relemed matches a query against each single sentence, having too many words in the query might return no article in the first few relevance levels. And second, if the total number of articles returned from MEDLINE is small, sorting them according to a relevance metric may not improve the retrieval process significantly.
There are also additional features that can be added to Relemed to improve its usefulness. For example, it would be helpful if Relemed showed the total number of matched articles per relevance level in the first page of results. The software currently used to implement Relemed does not support a fast response time for such a feature. It would also be useful to add search capability for fields like author names, and publication dates. We have also considered making the matched sentences and the article contents collapsible/expandable (via JavaScrips for example), rather than showing all the material at once. Finally, it may be possible to refine the relevance score by utilizing natural language processing algorithms in ascertaining the relation between words.
Additional evaluation and comparison of Relemed with PubMed and other search engines is essential. But we believe these initial results hold promise for improving the precision and efficiency of search using the sentence-level search capabilities and relevance sorting.
Availability and requirements
Project name: Relemed
Project home page: http://www.relemed.com
Operating systems: Platform independent
Programming language: Perl
Other requirements: None
License: Free access
Any restrictions to use by non-academics: Incorporation into commercial products restricted
HyperText Markup Language
HyperText Transfer Protocol
Linux Apache MySQL Perl
Medical Subject Headings
Sentence-level search Engine with Relevance score For MEDline
Sudden Infant Death Syndrome
Structured Query Language
eXtensible HyperText Markup Language
eXtensible Markup Language
We thank the two reviewers for their constructive suggestions.
- MEDLINE. [http://www.nlm.nih.gov/databases/databases_medline.html]
- Entrez PubMed. [http://www.ncbi.nlm.nih.gov/entrez/query.fcgi]
- Current Relations in the Semantic Network. [http://www.nlm.nih.gov/research/umls/META3_current_relations.html]
- Mani I, Maybury MT, (Eds): Advances in Automatic Text Summarization. 1999, Cambridge: MIT PressGoogle Scholar
- SLIM: Slider Interface for MEDLINE/PubMed searches – BETA. [http://pmi.nlm.nih.gov/slide/]
- askMEDLINE. [http://askmedline.nlm.nih.gov/ask/ask.php]
- eTBlast > Search. [http://invention.swmed.edu/etblast/etblast.shtml]
- Ovid's MEDLINE. [http://www.ovid.com/site/catalog/DataBase/901.jsp]
- HubMed. [http://www.hubmed.org]
- Accueil CISMeF. [http://www.chu-rouen.fr/cismef/]
- GoPubMed – Ontology based literature search (Biotec/TU-Dresden). [http://www.gopubmed.org]
- Anne O'Tate. [http://126.96.36.199/cgi-bin/arrowsmith_uic/AnneOTate.cgi]
- Start ARROWSMITH. [http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/start.cgi]
- PubMed Gold Free Full Text Search -Neurotransmitter.net. [http://www.neurotransmitter.net/ftsearch.html]
- Peterson WW, Birdsall TG, Fox WC: The theory of signal detectability. Transactions of the IRE professional group on information theory. 1954, 4: 171-212. 10.1109/TIT.1954.1057460.View ArticleGoogle Scholar
- Tanner WP, Swets JA: A decision-making theory of visual detection. Psychol Rev. 1954, 61 (6): 401-409. 10.1037/h0058700.View ArticlePubMedGoogle Scholar
- ReleMed search engine. [http://www.relemed.com]
- Unified Medical Language system (UMLS). [http://www.nlm.nih.gov/research/umls/]
- Comprehensive Perl Archive Network. [http://www.cpan.org]
- MySQL AB. [http://www.mysql.com/]
- The Apache HTTP Server Project. [http://httpd.apache.org/]
- Fedora Project, sponsored by Red Hat. [http://fedora.redhat.com/]
- The Extensible HyperText Markup Language. [http://www.w3.org/TR/xhtml1/]
- Medical Subject Headings. [http://www.nlm.nih.gov/mesh/meshhome.html]
- R Development Core Team: R: A language and environment for statistical computing. 2004, Vienna, Austria: R Foundation for Statistical ComputingGoogle Scholar
- Loader C: Local Regression and Likelihood. 1999, New York: SpringerGoogle Scholar
- Willinger M, James LS, Catz C: Defining the Sudden Infant Death Syndrome (SIDS): Deliberations of an Expert Panel Convened by the National Institute of Child Health and Human Development. Pediatric Pathology. 1991, 11: 677-684.View ArticlePubMedGoogle Scholar
- U.S. Department of Health and Human Services: Healthy People 2010: Understanding and Improving Health. 2000, Washington, DC: U.S. Government Printing Office, 2Google Scholar
- MetaDB: MEDLINE Interfaces List. [http://www.neurotransmitter.net/metadb/index.php?catid=65]
- Computation of Related Articles. [http://www.ncbi.nlm.nih.gov/entrez/query/static/computation.html]
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/7/1/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.