Who can you trust? A review of free online sources of “trustworthy” information about treatment effects for patients and the public

Abstract

Background

Information about effects of treatments based on unsystematic reviews of research evidence may be misleading. However, finding trustworthy information about the effects of treatments based on systematic reviews, which is accessible to patients and the public can be difficult. The objectives of this study were to identify and evaluate free sources of health information for patients and the public that provide information about effects of treatments based on systematic reviews.

Methods

We reviewed websites that we and our colleagues knew of, searched for government sponsored health information websites, and searched for online sources of health information that provide evidence-based information. To be included in our review, a website had to be available in English, freely accessible, and intended for patients and the public. In addition, it had to have a broad scope, not limited to specific conditions or types of treatments. It had to include a description of how the information is prepared and the description had to include a statement about using systematic reviews. We compared the included websites by searching for information about the effects of eight treatments.

Results

Three websites met our inclusion criteria: Cochrane Evidence, Informed Health, and PubMed Health. The first two websites produce content, whereas PubMed Health aggregated content. A fourth website that met our inclusion criteria, CureFacts, was under development. Cochrane Evidence provides plain language summaries of Cochrane Reviews (i.e. summaries that are intended for patients and the public). They are translated to several other languages. No information besides treatment effects is provided. Informed Health provides information about treatment effects together with other information for a wide range of topics. PubMed Health was discontinued in October 2018. It included a large number of systematic reviews of treatment effects with plain language summaries for Cochrane Reviews and some other reviews. None of the three websites included links to ongoing trials, and information about treatment effects was not reported consistently on any of the websites.

Conclusion

It is possible for patients and the public to access trustworthy information about the effects of treatments using the two of the websites included in this review.

Background

Patients and the public must make choices among different treatment options. We define “treatment” broadly, as any preventive, therapeutic, rehabilitative, or palliative action intended to improve the health or wellbeing of individuals or communities [1]. This includes, for example, drugs, surgery and other types of “modern medicine”; lifestyle changes, such as changes to what you eat or how you exercise; herbal remedies and other types of “traditional” or “alternative medicine”, and public health interventions. Few people would prefer that decisions about what they should and should not do for their health should be uninformed. Yet, if a decision is going to be well informed rather than misinformed, they need information that is relevant, trustworthy, and accessible. They also need to be able to distinguish between claims about the effects of treatments that are trustworthy and those that are not [2].

Often the problem is too much information rather than too little. For example, a Google search for “back pain” yields over 60 million hits [3]. PubMed, a free search engine for accessing MEDLINE and other databases maintained by the United States National Library of Medicine, includes over 27 million citations [4], and this represents only a fraction of the biomedical literature. The Cochrane Central Register of Controlled Trials, a bibliographic database that is restricted to controlled trials of treatments, contains over a million citations [5]. It is not practical for people making decisions about treatments to use search engines or databases such as these to find relevant information, critically appraise the studies they find, synthesize them, and interpret the results.

Systematic reviews reduce the risk of being misled by bias (systematic errors) and the play of chance (random errors), by using systematic and explicit methods to identify, select, and critically appraise relevant studies, and to collect and analyse data from them [6]. For information about treatment effects to be trustworthy, it should be based on systematic reviews. For it to be accessible to patients and the public, it should be easy to find and should be clearly communicated in plain language [7].

Unfortunately, a large amount of information about treatment effects is not based on systematic reviews and is not trustworthy [8,9,10,11,12,13,14,15,16,17,18,19]. This includes handouts for patients [8, 9], internet-based information [10, 11], information in social and mass media [12,13,14,15,16,17,18], information produced by patient organisations [8, 9, 12]. press releases [18], and advertisements [19]. Studies of the trustworthiness of health information have used a variety of criteria, but have consistently found important limitations [8,9,10,11,12,13,14,15,16,17,18,19]. Although trustworthy information about treatment effects can be found, evidence-based information is frequently written for health professionals or researchers, rather than for patients and the public [7].

There is an abundance of health information on the internet, which has become an important source of health information over the past two decades [10, 11, 20,21,22,23,24], but patients and the public find it difficult to search the internet for trustworthy information [21,22,23], and are unlikely to critically appraise the information that they do find [22, 23].

There are a number of websites that aim to improve access to trustworthy health information for patients and the public. The objectives of this study were to identify free sources of health information for patients and the public which provide information about the effects of treatments based on systematic reviews, and to evaluate those websites.

Our motivation for undertaking this review grew out of a desire to respond to people who were looking for trustworthy information about the effects of specific treatments and landed on Testing Treatments international [25], a website for promoting critical thinking about treatment claims. Rather than simply noting that the Testing Treatments website does not provide the information they were seeking, we wanted to help them by directing them to sources that do provide this information. Given this motivation, we restricted our review of websites to ones with a broad scope. There were two reasons for this. First, websites with a broad scope can meet the needs of most people seeking trustworthy information about treatment effects. Furthermore, although disease-specific websites can be useful, it would be impractical to assess any more than a small sample of websites for specific conditions or types of treatments. Second, it is easier to become familiar with one or a small number of websites than it is to use multiple websites for questions about different conditions or types of treatments.

Methods

We considered any website that defined itself as providing “health information”, which included information about treatments. To be included in this review a website needed to be:

• Available in English

• Freely accessible (i.e. non-commercial with no cost to users or membership fees)

• That described itself as being intended for patients and the public

• Broad in scope (not limited to specific conditions or types of treatments)

• Explicitly based on systematic reviews (i.e. there had to be a description of how the information is prepared and the description had to include a statement about using systematic reviews)

We identified websites that potentially met those criteria by considering websites that we and our colleagues (see Acknowledgements) knew of. The first author (AO) searched for government sponsored websites in English speaking countries (including Australia, Canada, Ireland, New Zealand, the UK, and the USA); searched Google for “health information” and “patient information” to identify websites that are frequently accessed for health information; and checked links to other websites on the websites that were identified. On 29 January 2018, AO conducted a final set of searches using the following terms: “health information”, “patient information”, “evidence-based health information”, and “evidence-based patient information”; and these search engines: Google [3], Bing [26], DuckDuckGo [27], and HONsearch patients [28]. Google and Bing are the two most popular search engines, DuckDuckGo is not affected by your previous search history, and HONsearch searches “trustworthy” health websites. The first 20 hits for each search were screened, and any websites that looked like they might meet our inclusion criteria were checked.

AO assessed each identified website for inclusion and the second author (EP) checked those judgements using information provided on the websites. In addition, we emailed each excluded website to confirm that our reason for excluding it was correct.

AO collected the following information for each included website:

• The stated purpose

• A statement that information about treatment effects is based on systematic reviews

• Availability of links to the systematic reviews

• Reporting size of effects

• Reporting certainty of the evidence; i.e. a judgement using GRADE (Grading of Recommendations Assessment, Development and Evaluation) [29,30,31] or another formal approach or an informal judgement about how sure we can be about the reported effects

• Availability of links to ongoing trials

• Information about how up-to-date information about treatment effects is

• What other information is provided

• What tools there are for searching, sorting, and filtering information

• Use of plain language (i.e. summaries written for patients and the public) and the availability of a glossary

EP checked all of the information that was recorded and the judgements that were made. To inform these judgements, both authors independently searched each included website for eight questions about treatments to assess the ease of finding information (AO on 22 December 2017 and EP on 9 January 2018). We selected the eight questions by searching Google for “common health questions” and selecting the first relevant list that we found (25 Questions About Your Health Answered - Oprah.com). Many of the questions in that list were not about treatment effects and we modified some of the questions with the intention of having a variety of questions for different types of conditions and treatments. Table 1 shows the original question from that list, our question, the conditions, the treatments, and the initial search terms that we used to find information about treatment effects on each website.

Table 1 Questions about treatments used to assess the included websites

We then independently assessed what was reported about treatment effects, the consistency of reporting, and the advantages and disadvantages of each website. Disagreements were resolved by discussion. Based on these assessments and the information we had collected for each website we suggested how the websites could be improved and provided tips for website users.

For each question, we searched for information using plain language terms without Boolean logic (using the first terms shown for each question in the last column of Table 1). We recorded the number of hits for each search and each relevant summary that we found. We assessed the search as easy if we found relevant information using plain language terms without Boolean logic and the relevant information was one of the first few hits. We assessed searches as hard if we had to use technical terms or Boolean logic, or if we could not find relevant information; and as moderate if finding relevant information required some minor fiddling with the search terms or screening more than a few hits.

For each relevant summary that we found, we recorded whether any information was provided about benefits of the treatment and harms of the treatment, whether quantitative information was provided for at least one outcome, and whether a formal or informal assessment of the certainty of the evidence was provided. We then ranked the three websites for each question based on an overall assessment of how hard it was to find relevant information and the completeness of the information about the effects of the treatments.

Results

We considered 35 websites for inclusion. Of these, 26 were excluded because information about treatment effects was not explicitly based on systematic reviews (Table 2), five were excluded because they were not intended for patients and the public (Table 3), and one was under development (Table 4). Three of the 34 websites met our inclusion criteria: Cochrane Evidence, Informed Health, and PubMed Health (Table 5). Cochrane Evidence and Informed Health produce content, whereas PubMed Health, which was discontinued in October 2018, aggregated content, including content from the first two websites.

Table 2 Websites excluded because they are not explicitly based on systematic reviews^a

Table 3 Websites excluded because they are not intended for patients and the public^a

Table 4 Websites under development^a

Table 5 Included websites

Cochrane Evidence provides plain language summaries of over 7500 Cochrane Reviews, most of which are systematic reviews of the effects of treatments. The systematic reviews and the plain language summaries are prepared and updated by Cochrane review groups. Cochrane is a global independent network of researchers, professionals, patients, carers, and people interested in health, with over 37,000 contributors from more than 130 countries.

The plain language summaries include links to the full reviews. The full reviews are available in The Cochrane Library, which can be accessed for free in countries that have a national subscription or if the review or an update was published more than one year previously. The headings and content of the plain language summaries are inconsistent. The summaries include some background information, the authors’ conclusions, and links to other summaries that may be of interest. There is variability in the quality of the summaries. Some summaries include pop-up definitions (but not links to longer explanations) for some research and medical terms, and there is a glossary of terms relevant for Cochrane Reviews available on the Cochrane website. The summaries are translated into Chinese, Croatian, Czech, French, German, Japanese, Korean, Malay, Polish, Portuguese, Romanian, Russian, Spanish, Tamil, and Thai. The glossary is only in English.

No other information regarding treatments is provided in Cochrane Evidence, besides the plain language summaries of Cochrane Reviews. Cochrane website, where Cochrane Evidence is found has other information about the Cochrane Colaboration. Navigation tools for Cochrane Evidence are limited to a simple search for the entire Cochrane website. It is possible to sort findings by relevance, alphabetically, or by date of publication; and to filter the summaries by broad health topics and whether the reviews are new or updated.

Informed Health is the English-language version of the German website Gesundheitsinformation.de. The website is prepared by the Institute for Quality and Efficiency in Health Care (IQWiG) in Germany. IQWiG is a professionally-independent, scientific institute established under the Health Care Reform 2004.

The Informed Health website provides information about treatment effects together with other information for a wide range of topics. The website includes “research summaries” for some but not all treatments. “These are objective, brief summaries of the latest findings on a research question described in the title. They usually summarize the results of studies, for instance the results of one or (rarely) several systematic reviews or IQWiG reports. They also describe the study/studies in more detail and explain how the researchers came to their conclusions.” The website states that they “mainly use systematic reviews of studies to answer questions about the benefits and harms of medical interventions.” Links to systematic reviews are provided when these are used, but the reviews may not be freely available.

All of the research summaries that we examined (Additional file 1) included quantitative information about the size of the benefits, and they included frequencies for at least one outcome, but most often only for one outcome. The certainty of the evidence is not reported. All of the information is in plain language, written for patients and the public. There are hyperlinks to background information (but not pop-up definitions). There is a glossary of “medical and scientific” terms that includes primarily medical terms and few research terms.

In addition to information about treatments, Informed Health includes information about symptoms, causes, outlook, diagnosis, everyday life, where to learn more, and explanations (“Extras”) of topics such as how the body works, how treatments work, and types of treatments. Navigation tools for Informed Health include browsing by broad topic areas, an index (A to Z list) and a simple search. Search results can be sorted by relevance, the date information on the website was created, or the date it was updated.

PubMed Health specialized in systematic reviews of clinical effectiveness research. It included plain language summaries and abstracts of Cochrane Reviews; abstracts (technical summaries) of systematic reviews in the Database of Abstracts of Reviews of Effects (DARE) up to 31 March 2015; full texts of reviews from public agencies; information developed by public agencies for consumers and clinicians based on systematic reviews; and methods resources about the best research and statistical techniques for systematic reviews and clinical effectiveness research. PubMed Health was a service provided by the National Center for Biotechnology Information at the U.S. National Library of Medicine. It was discontinued October 31, 2018 “in an effort to consolidate similar resources and make information easier to find”. It included information from over 40,000 systematic reviews from a variety of sources, but plain language summaries were not available for most of those reviews. Links to the systematic reviews were provided, but not all of the reviews were freely available.

The reporting was inconsistent. Headings, reporting of effects, and reporting of the certainty of the evidence were inconsistent. PubMed Health had an extensive glossary (Health A – Z) and background information on drugs. Navigation tools included a simple search. Search results could be sorted by date of publication and filtered by Article types (including “Consumer information”); when information was added to PubMed Health, Content providers (including Cochrane and IQWiG); and Reviews with a quality assessment.

None of the three included websites includes links to ongoing trials and adverse effects are not consistently reported on any of the websites. All three include information about how up-to-date the information about treatment effects is.

PubMed Health was the easiest website to search, despite the large number of records that it includes. However, we had difficulties searching all three websites. We found information easily in Cochrane Evidence and Informed Health for one of the eight questions in Table 1, and for three of the questions in PubMed Health (Additional file 1). Conversely, it was hard to find information (or we did not find any information) for the five questions in Cochrane Evidence, six questions in Informed Health, and three questions in PubMed Health. It was not possible to use Boolean logic when searching Informed Health. This was possible on the other two websites, but none of the three provided any instructions or help for searching.

When we found information, it was consistently available about benefits, but only Informed Health consistently reported this information quantitatively in the plain language summaries. Quantitative information was provided in the linked scientific abstracts. None of the websites consistently reported information about harms or the certainty of the evidence, although Cochrane plain language summaries in Cochrane Evidence and PubMed Health frequently reported the certainty of the evidence. When the certainty of the evidence was reported using GRADE or another systematic approach, there was not a link to an explanation of what the grade means.

Overall we were most satisfied with Cochrane Evidence for 2 questions, with Informed Health for one question, and with PubMed Health for 3 of our questions. We did not find information about treatment effects on any of the three websites for two questions: “Should I stop using phone, tablet, computer, and TV screens before going to bed (for insomnia)?” and “Should I get my osteoarthritic knee replaced?” Informed Health provided advice for the first question (“For instance, it might help to only listen to relaxing music before going to bed and keep from talking on the phone or playing computer or mobile phone games”), but no reference to research evidence for that advice. We easily found relevant systematic reviews for both of these questions in Epistemonikos (Additional file 1).

Discussion

We identified three websites for patients and the public that provide free information about treatment effects based on systematic reviews. A fourth, promising website, CureFacts, was under development (Table 4), and is still under development as of February 2019. Twenty-two other websites that provide free information for patients and the public claim to provide trustworthy, evidence-based information. However, it is not possible to know the extent to which the information they provide about treatment effects is based on systematic reviews, so is therefore less likely to be trustworthy. We considered four websites that provide access to systematic reviews, but none of these is intended for patients and the public (Table 3). Nonetheless, some people may find these useful, particularly Epistemonikos. It includes over 100,000 systematic reviews with the abstracts translated to Arabic, Chinese, Dutch, French, German, Italian, Portuguese, and Spanish. It is aimed for health professionals, researchers and policymakers but plain language summaries are not available for most of the reviews. Although it is not intended for patients and the public, it “has been used by well-informed lay people and journalists successfully” (Table 3).

We did not consider databases that are not free, such as Trip Pro, which includes access to over 100,000 systematic reviews; or patient information from web-based medical compendia for clinicians, such as Best Practice, Dynamed, and UptoDate. We also did not consider websites that provide information for patients and the public based on guidelines, such as the UK National Institute for Health and Care Excellence (NICE) guidance for patients; or websites that are limited to specific conditions or types of treatments.

The three websites for patients and the public that explicitly provided information about treatment effects based on systematic reviews were likely to appeal to different people and their appeal may vary depending on the question being asked. We found that we preferred each of the websites for at least one of the eight questions we used as test cases (Table 1). We found PubMed Health somewhat easier to search, despite the large number of records it includes, and we found both Cochrane plain language summaries and Health Information research summaries when searching PubMed Health. Simple instructions regarding the use of Boolean logic and the use of quotations to limit searches would help improve the ease of use for all three websites. For example, the default for Cochrane Evidence appears to be to insert OR between words, resulting in large numbers of irrelevant hits.

All of the websites could be improved by more consistent use of headings and consistent reporting of both benefits and (especially) harms; inclusion of quantitative information about the size of the effects; and information about the certainty of the evidence based on the use of a consistent set of criteria, such as GRADE [29,30,31], and links to explanations of what the grades mean. Because many systematic reviews, including Cochrane Reviews, do not consistently provide this information, plain language summaries based on systematic reviews cannot always provide this information. However, they can alert users to the absence of trustworthy information about adverse effects, when this is the case, and it is possible to provide an assessment of the certainty of the evidence even when review authors have not done this [32, 33].

All three websites provided plain language summaries of systematic reviews and all three had glossaries. However, none of the websites included both pop-up short definitions (which can be quickly accessed and read as scroll overs without having to go to another webpage) and links to longer explanations (that can be easily accessed when needed).

None of the websites included links to ongoing trials. This is something that, for example, NHS Choices does [34]. This is important because when there is important uncertainty about the effects of treatments, participating in a randomised trial may be the best option for patients [35, 36].

We are not aware of any other studies that have attempted to systematically identify and evaluate websites that provide free access to information about the effects of treatments for patients and the public which is based on systematic reviews. There are thousands of websites that provide health information and we did not systematically screen all of these. Although we believe it is unlikely that there are other websites that meet our inclusion criteria, we did not consider websites for specific conditions or types of interventions, non-English language websites, or websites that were not freely accessible. Others might want to assess these and other sources of information about treatment effects in future studies.

"The evaluation criteria that we used were based on our judgement about what information is important and what is needed to make that information accessible. For example, providing a link to the systematic review enables people to go to the source of information about treatment effects for more information, if they desire. It also makes the basis of the information clear. Information about the size of effects and the certainty of the evidence is essential for making well-informed decisions. Basic search tools are necessary to make it easy to find information on the websites, and summaries that are written in plain language for patients and the public are more likely to be understandable than abstracts written for researchers or health professionals. Consistent headings, content, and use of language make it easier for users to become familiar with the websites and to find and understand information.

Our evaluation was based in part on searching for answers for eight treatment questions (Table 1). The criteria that we used to assess what we found for each question did not require a great deal of judgement. Consequently, there were only minor disagreements in our assessments (Additional file 1), and those were easily resolved. It is uncertain how representative what we found for those questions is for what would be found for other treatment questions, but we believe they provided a fair basis for assessing the websites. Moreover, we sent full drafts of this report to people responsible for each website and their corrections did not substantially alter our assessments or conclusions.

We did not evaluate the readability of the plain language summaries and, although we described other information that each website provides, we did not evaluate whether the websites provided other information that patients and the public want or need to make informed decisions; for example, information about other treatment alternatives, costs, and people’s experiences with the treatment [37, 38]. We also did not evaluate how users of the websites experience them [39]. All of these are potential areas for future research."

Conclusions

It is possible for patients and the public to access trustworthy information about the effects of treatments based on systematic reviews using two of the three websites included in this review. However, all three of these websites could be improved and made more useful and easier to use by consistently reporting information about the size of both the benefits and harms of treatments and the certainty of the evidence, and by making it easier to find relevant information.

Searching the three websites frequently yielded much irrelevant information. Users can limit searches by using Boolean logic - inserting AND between terms (e.g. for the condition and for the treatment) and quotation marks to indicate that words need to be next to each other; e.g. “back pain”. However, this is unlikely to be obvious to novice users. Some users may want to use sources that are not intended for patients and the public, such as Epistemonikos, if they are unable to find information on one of these websites. They also might want to consider searching for ongoing trials, if there is important uncertainty about the effects of relevant treatments.

There are many other websites that claim to provide evidence-based or reliable information about treatments, but it is difficult to assess the reliability of the information about treatment effects provided on those websites since they do not explicitly base that information on systematic reviews.