Seal of transparency heritage in the CISMeF quality-controlled health gateway

The availability of Internet health tools and services has been increasing at a phenomenal rate in recent years making the Internet a major source of knowledge for healthcare professionals, medical students and also patients and the general public. This increase has made it an absolute necessity to continually assess the quality of health information on the Internet. Indeed, creating a Web site is relatively easy, therefore uncontrolled health information can be launched by virtually anyone with access to the Internet. Peer review is often absent throughout this media as opposed to scientific journals. There have been numerous debates about the variable quality of health information on the Internet and its impact on public health [1]. There is no other field in which inaccurate, incomplete, or biased information can be potentially more damaging [2].

In the past five years, a lot of authors have scrutinized the quality of the health content available on the Internet. These studies, assessing the quality of health information, have been extensively reviewed by Eysenbach et al. [1]. In the meantime, several worldwide initiatives have been undertaken to define criteria for assessing the quality of health information on the Internet. These initiatives have been reviewed by Risk and Dzenowagis [3]. However, no consensus has been reached by healthcare professionals or consumers on how to assess the quality of health information on the Internet.

As of today, access to accurate and trustworthy health information on the Internet is not an easy task; there are a great number of directories and search engines available in this new media [4]. But, there is also a need to develop reliable and quality-controlled health subject gateways to disseminate relevant trustworthy health information. Koch [5] defined quality-controlled subject gateways as Internet services which apply a rich set of quality measures to support systematic resource discovery. Considerable manual effort is used to process a selection of resources, which meet quality criteria, and to display a rich description and indexing of these resources with standards-based metadata. Regular checking and updating ensure optimal collection management. The main goal is to provide a high-quality of subject access through resource indexing using controlled vocabularies and via a practical classification structure for advanced searching and browsing.

The objective of CISMeF (French acronym for Catalogue and Index of health resources in French) [6, 7] is to describe and index the main health resources in French in order to assist health professionals, students and consumers in their search for electronic information available on the Internet.

CISMeF is a quality-controlled subject gateway initiated by the Rouen University Hospital (RUH). Each of the following phases proposed by Koch [5], which characterise a typical quality-controlled subject gateway, are implemented in CISMeF: (a) selection and collection development, based on the Net Scoring, a list of 49 criteria to assess quality of health information (URL: http://www.chu-rouen.fr/netscoring) [4], (b) collection management, (c) creation of metadata (performed by experts), (d) resource description (an extensive and documented metadata set), and (e) resource indexing (using a controlled vocabulary system).

CISMeF is manually maintained. In CISMeF, a resource is defined as 1) a Web site or 2) high-quality documents from this Web site. CISMeF describes and indexes the most important sources of institutional health information in French, in order to allow them to be searched quickly and precisely. A great variety of resources are indexed, in terms of resource type (clinical guidelines, teaching material, patients information, etc.), and resource format (html, pdf, etc.). Its Universal Resource Locator (URL) is http://www.chu-rouen.fr/cismeforhttp://www.cismef.org The CISMeF Web site opened in February 1995. In December 2003, 13,012 resources had been indexed, with an average of 55 new resources indexed each week. CISMeF is considered by most professionals as the reference health institutional Web site in France with as many as 25,000 unique machines visiting this Web site by working day. Doc'CISMeF is the search tool of the CISMeF gateway [7].

In 1997, because the quality of health information became an important issue for the building and maintenance of a trustworthy health gateway, the CISMeF team participated in the development of a user guide named Net Scoring [8]. The goal of the Net Scoring project was to provide a set of criteria that can be consistently used to assess the quality of health information on the Internet. Between 1997 – 2002 the CISMeF gateway selected health resources using the main criteria established by the Net Scoring initiative (source of information, disclosure, editorial review process, date of last update, and feedback mechanism) in view of the fact that the selection process is mandatory to create a trustworthy health Internet directory. Resources that are not compliant with basic ethical criteria are not included in the CISMeF database.

In 2002, CISMeF became a contributor to the MedCIRCLE project (URL: http://www.medcircle.org) [9]. The aim of this project is to establish a global Web of transparency for networked health information and to increase the accessibility and findability of trustworthy health websites using "Semantic Web" approaches, which essentially means to make "narrative" information on the Web accessible in a machine processable format by using RDF (Resource Description Framework) expressed in XML (eXtended Markup Language) [10]. MedCIRCLE is a collaboration of trustworthy European health subject gateways, medical associations, accreditation, certification, or rating services, which share the common goal of evaluating, describing, or indexing health information. MedCIRCLE began in March 2002 and lasted till December 2003. Whereas CISMeF initially addressed quality on a more finely grained level, i.e., the quality of documents or single web pages, MedCIRCLE focused on whole websites including information about the respective publisher.

The main deliverable of this project for the CISMeF team was the evaluation and rating of 270 health information providers (or publishers) which who release health resources in French on a regular basis. Because CISMeF catalogues and indexes not only Web sites but also and mainly quality-controlled documents from those health publishers, we introduced the concept of "transparency transitivity" or " transparency heritage". It consists in applying to these documents the evaluation performed for their publishers, thus passing on the transparency label from the publishers to their documents.

The CISMeF team in the MedCIRCLE consortium has evaluated and annotated the main health information providers (or publishers) included in the CISMeF database: national agencies, medical societies, universities and hospitals. CISMeF first checks the publishers' information without asking the health information provider to self-declare any metadata as described in the MedCERTAIN quality management process.

CISMeF used HIDDEL to select and evaluate the 270 health publishers most represented in CISMeF and made the results of their evaluations explicit and accessible using RDF metadata. These were exported into the searchable MedCIRCLE Open Directory.

Each site rated in CISMeF has a record in the CISMeF database that generates an RDF into HTML file. The search tool Doc'CISMeF (URL: http://doccismef.chu-rouen.fr) displays the information originating from each of the publishers that were evaluated with a specific MedCIRCLE button, which is linked to the MedCIRCLE central repository where HIDDEL metadata elements are displayed (see Figure 3).

Seal of trust, such as the one developed by HON, is a "quality seal" or a "seal of approval": i.e. the HON logo provides an accreditation, whereas the MedCIRCLE seal is not a "quality seal" but a "transparency seal": it is a button allowing health professionals and consumers to access metainformation. The presence of a MedCIRCLE button on a health Web site does not imply, in any way, that the site meets minimum required standards. This decision is left up to the user. In contrast, a seal of trust is a quality seal: i.e. every Web site with a seal of approval (such as the HON seal) has been previously accredited a third party.

Nonetheless, every resource included in CISMeF and those evaluated by the MedCIRCLE process are quality-controlled. The MedCIRCLE consortium takes a very neutral approach and does not impose but strongly recommends certain procedures or minimum metadata, taking into account that collaborating gateways, accreditors, certifiers, raters may approach from very different angles.

CISMeF has applied full heritage from the evaluated publishers: each document from a MedCIRCLE rated publisher, indexed in CISMeF, will also receive the MedCIRCLE button of the publisher with the same link to MedCIRCLE central repository. The idea is to keep the common Infoprovider elements for every document, and to use CISMeF metadata to disclose Sitespecific elements specific to each document, in particular the indexing with the MeSH thesaurus. At the end of the project in December 2003, starting from 270 websites, the translation from CISMeF metadata to HIDDEL, led to 6,480 evaluated resources in CISMeF in September 2003 (49.8% of the 13,012 resources included in CISMeF). All CISMeF selected HIDDEL metadata (70 out of 305) are displayed in the Doc'CISMeF record in RDF into HTML. The top five publishers indexed in CISMeF, which produced trustworthy documents in French are: Grenoble Medical School (N = 435), Health Canada (n = 275), Strasbourg Medical School (N = 263), and French Ministry of Health (N = 248).

Every new document (e.g. clinical guideline or teaching material) included in CISMeF from one of the 270 main publishers evaluated through the MedCIRCLE process inherits automatically: (a) a MedCIRCLE button linking to the repository and (b) HIDDEL metadata included in CISMeF database and displayed in the Doc'CISMeF record in RDF into HTML. On the other hand, a document that comes from one of the 270 publishers evaluated within the MedCIRCLE project but not included in the CISMeF database will not receive a MedCIRCLE button in the CISMeF gateway. However, this more global transitivity could be applied in a more generic search tool such as Google.

Since February 2003, when the MedCIRCLE button became operational in the CISMeF gateway, the CISMeF team has decided to go on (keep) applying the MedCIRCLE transparency process after the end of the project with the following rules: (a) check every month if there is a new publisher with five documents already included in CISMeF and (b) if so, begin the MedCIRCLE process for these publishers and apply transparency heritage for their respective documents. We applied these rules after the end of the EU-funding project. In March 2004, the CISMeF database contained a complete evaluation through the MedCIRCLE process for 346 publishers (+76, as compared to the EU-grant proposal) and 7,053 documents from these publishers (53.3% of the 13,227 resources included in CISMeF).

In the MedCIRCLE repository, end-users can access an aggregate view of what people say about a certain Web site by clicking: The CISMeF gateway is one of many possible producers of trustworthy metadata regarding a health information provider. Metadata from the Open Directory can also be fed into search engines and other gateways.

As the number of health related Web sites worldwide has been estimated as being around 100,000, complete coverage by a single third party evaluation body is impossible. Instead, a collaborative approach as shown in this paper has to be promoted, whereby different rating services or organizations use comparable standards and a common metadata language. More recently, the Health on the Net Foundation has developed a HON tool bar in the course of the EU-funded Active Health (Active Environment for Health Promotion and Disease Prevention, URL: http://www.hon.ch/Project/ActiveHealth.html) consortium (URL: http://www.active-health.info/): this HON tool bar is indicating if the site is accredited directly by HON or indirectly by one specific accredited Web sites as health gateways such as CISMeF or MedlinePlus. In this context, HON is creating a seal of quality trust heritage where MedCIRCLE is creating a seal of transparency heritage. A formal evaluation of these two examples of heritage (quality seal vs. transparency seal) is mandatory to check their respective hypothetical added value.

One of the main findings of the MedCIRCLE consortium in the course of the project has been that there is no absolute objective quality of a Web site. Quality is to a certain degree subjective, may vary in time and also according to the eye of the beholder. A Web site that a consumer looking for health information finds acceptable one day may be unacceptable another day. For example, a consumer may search general information for one drug (e.g. after a TV show) and finds advice on a patient information Web site. Later, the same consumer searches for the same drug for his/her child. But this time, he/she checks if the respective Web site is sponsored by a pharmaceutical company. He/she comes to the same Web site as before, but this time this Web site will be unacceptable. The context of the search changed, making a general advice acceptable and a specific advice unacceptable. By providing metadata about health related websites, MedCIRCLE allows the health information consumers to decide themselves if a website is of good quality. Trust is improved by enhancing transparency. More globally, the MedCIRCLE consortium will lead to a safer Internet by providing a seal of transparency to over 7,000 resources on the Internet via the CISMeF search engine. The precaution principle is now widely accepted in the European Union, which develops Action Plans, such as the Action Plan for Safer Use of the Internet (URL: http://www.saferinternet.org), which partially granted the MedCIRCLE consortium.

The MedCIRCLE button does not directly fulfill the objective to only identify reliable health information. This objective is in fact fulfilled by th HON initiative. In contrast, the MedCIRCLE consortium indirectly fulfill this objective (i.e. to identify only reliable health information) by providing a seal of transparency. The Netizen (citizen on the Internet) has the active role to evaluate the reliability when reading the HIDDEL metadata after clicking the MedCIRCLE button. With a seal of trust, the role of the Netizen is more passive but leads to a faster trust information access. Nevertheless, a drawback of the MedCIRCLE button is a possible misunderstanding by end-users, who might confuse it with a quality seal and therefore may forget to actively press the MedCIRCLE button, which may hinder their valid judgement. Here again, a formal evaluation of theses two approaches (transparency vs. trust) is necessary.

While using Net Scoring, CISMeF was acting as an implicit third party. One of the main results of the MedCIRCLE project, from the CISMeF team's point of view, is that CISMeF proceeded from being an implicit to being an explicit third party thanks to the creation of the MedCIRCLE button now used for 346 publishers (2.6% of the resources included in CISMeF) and 7,053 resources (53.3% of the resources included in CISMeF), which multiplies by 20 the number of resources with a seal of trust.

The HIDDEL language is totally embedded in the CISMeF metadata element sets. This allows a very easy interoperability with MedCIRCLE tools, and more specifically Archer. Semantic Web approaches already used in the MedCIRCLE project could open up new ways for educating health professionals and consumers and reaching less savvy health professionals and consumers, because part of the intelligence and knowledge currently required to critically appraise information on the health professional or consumer Web site could be built into the search tools. The feasibility of this approach has been already demonstrated by CISMeF but also by the German (AQuMed) and Spanish partners (COMB). The impact on health professionals and consumers is subject to ongoing investigation within the MedCIRCLE project. The Semantic Web may provide the health professional or the consumer with a greater capacity to determine the reliability of a given health information provider or service than the Web in its current form.

With the MedCIRCLE project and transparency heritage, CISMeF became an explicit third party.