This article has Open Peer Review reports available.
Comparison of different approaches applied in Analytic Hierarchy Process – an example of information needs of patients with rare diseases
© The Author(s). 2016
Received: 28 January 2016
Accepted: 3 August 2016
Published: 9 September 2016
The Analytic Hierarchy Process (AHP) is increasingly used to measure patient priorities. Studies have shown that there are several different approaches to data acquisition and data aggregation. The aim of this study was to measure the information needs of patients having a rare disease and to analyze the effects of these different AHP approaches. The ranking of information needs is then used to display information categories on a web-based information portal about rare diseases according to the patient’s priorities.
The information needs of patients suffering from rare diseases were identified by an Internet research study and a preliminary qualitative study. Hence, we designed a three-level hierarchy containing 13 criteria. For data acquisition, the differences in outcomes were investigated using individual versus group judgements separately. Furthermore, we analyzed the different effects when using the median and arithmetic and geometric means for data aggregation. A consistency ratio ≤0.2 was determined to represent an acceptable consistency level.
Forty individual and three group judgements were collected from patients suffering from a rare disease and their close relatives. The consistency ratio of 31 individual and three group judgements was acceptable and thus these judgements were included in the study. To a large extent, the local ranks for individual and group judgements were similar. Interestingly, group judgements were in a significantly smaller range than individual judgements. According to our data, the ranks of the criteria differed slightly according to the data aggregation method used.
It is important to explain and justify the choice of an appropriate method for data acquisition because response behaviors differ according to the method. We conclude that researchers should select a suitable method based on the thematic perspective or investigated topics in the study. Because the arithmetic mean is very vulnerable to outliers, the geometric mean and the median seem to be acceptable alternatives for data aggregation. Overall, using the AHP to identify patient priorities and enhance the user-friendliness of information websites offers an important contribution to medical informatics.
The number of studies measuring patient priorities by using the Analytic Hierarchy Process (AHP) has increased significantly in the last few years . The AHP was developed by Thomas L. Saaty in the 1970s to solve complex problems of multiple criteria decision-making , based on the idea that it is more reliable to judge the relative importance of several criteria with the help of respective pairwise comparison in a hierarchical structure than to judge their absolute importance . The method was originally applied in the marketing sector and later in healthcare research. In addition, the AHP can be used to relate subjective criteria, which can be both quantitative and qualitative. As implied, it has been demonstrated that the AHP is a useful method for healthcare delivery as well as medical informatics decision-making [1, 4–7]. In this study, we ranked the information needs of people having a rare disease and their relatives using different AHP methods. This ranking of information needs is then transferred accordingly to display information categories on a web-based information portal about rare diseases in Germany. Because the available space on a user-friendly website homepage is restricted, the most important categories should be more accessible than less important categories. To present information categories on this website according to the user’s priorities, this paper consulted both experts in medical informatics and patient-reported outcomes.
Today, approximately 4 million people in Germany suffer from rare diseases. The level in the United States is similar to that in Europe, with approximately 30 million people living with rare diseases. It is estimated that 400 million people worldwide suffer from a rare disease. Currently, international definitions of rare diseases vary greatly. For example in the EU, a disease is considered rare if it affects fewer than one in 2000 citizens, whereas in the United States a disease is considered rare if it affects fewer than 200,000 people, or about one in 1500 people [8, 9]. To improve patients’ well-being, a national action plan for people with rare diseases was adopted by the Federal Government in Germany in 2013 that is supposed to coordinate national efforts invested in rare diseases. The establishment of a rare diseases information portal is one component of a broader set of planned measures, which includes 52 policy proposals . Although conditions may differ significantly, patients having rare diseases and their relatives frequently face similar challenges [10, 11], which include protracted diagnosis processes as well as a deficient information base. To address these deficiencies, both medical experts and experts on medical informatics consider it relevant to assess the priorities of the (potential) patients and relatives.
As part of the development of an information portal for rare diseases, we used the AHP to identify the importance of several information types, e.g., information about therapy and social-legal advice. However, there are no best practices or a common gold standard available for applying the methods . More precisely, it is noticeable that there are several methodological differences in the published studies concerning data acquisition and aggregation . In some studies, single participants were interviewed (e.g. [12–14]), whereas in others, group discussions were used to analyze the priorities (e.g. [15, 16]). It therefore remains unknown which data acquisition method is more suitable for the AHP. To determine whether two methods (individual and group decisions) yield the same outcomes, we implemented them separately. The goals of this study were on the one hand to analyze the different influences of individual and group judgements on data acquisition, and on the other hand, to examine the different effects on the AHP results of using the arithmetic and geometric mean as well as the median for the data aggregation. We also discuss the degree to which the results of this study can be transferred to other disciplines. Finally, we fulfill our objective of providing a recommendation on choosing appropriate methods for further studies using the AHP.
Patients suffering from a rare disease were eligible to participate in the study. In addition, the relatives of these patients, for example, the parents of a child suffering from such a disease, were eligible to participate. The inclusion of both patient and relatives is necessary because many patients suffering from a rare disease are diagnosed as children, and the information priorities of the parents appear as a proxy for the children’s priorities. Moreover, both patients and relatives will use the information portal. Patients were excluded if they were unable to concentrate continuously on the questionnaire or did not adequately understand the German language. Participants were recruited by the Freiburg Centre for Rare Diseases (Medical Center of the University Freiburg, Germany) and through rare disease self-help groups.
The AHP is a stepwise problem-solving procedure. First, the decision-makers have to construct a hierarchical structure of the criteria. To achieve this, the multiple criteria decision problem must be broken down into its component parts . The information needs of people suffering from a rare disease were identified by an Internet research study, including a review of already existing websites providing information on rare diseases. Furthermore, a preliminary qualitative study, the subjects of which were patients suffering from a rare disease, yielded important findings about the wording of the identified items that were regarded as the defined targets. We designed a three-level hierarchy by grouping these items into information fields and information types.
The final operation was consistency verification, which is listed as one of the key benefits of the AHP . Saaty demonstrated that the consistency ratio (CR) can be calculated using the consistency index and the random index . The CR value of a perfectly cardinal consistency matrix is 0. The CR value reflects the internal consistency of an observed set of judgements, and CR ≤ 0.2 has been determined to be an acceptable level of consistency [20, 21]. The results of participants who answered consistently were included in the analyses. Finally, the priorities of individual participants were aggregated to analyze the priorities of all the participants. The different data acquisition and aggregation methods are described in the following section.
The same AHP questionnaire was used for the face-to-face group discussions. The group meetings were held at the Universities of Hannover, Frankfurt am Main, and Freiburg im Breisgau. After the interviewer presented a description of the structure and method of the AHP, each group member judged the relative priorities of each comparison. Then, the individual judgements (on a 9-point scale) were gathered and displayed anonymously on a screen. The group members discussed each pairwise comparison, as well as the rationales behind the individual judgements. Finally, for each pairwise comparison, a common group decision (consensus) was reached. The calculated group priorities were aggregated with all the other group priorities (Fig. 2) when the answers were consistent, as described above. The distribution of the priorities of individual and group weights was analyzed in separate box plots for each category using the statistics software R.
Priorities can be aggregated using the arithmetic mean. According to a frequently used method for aggregating the priorities of individuals into a consensus rating, we also used the geometric mean [21–23]. In addition, we used the median to calculate the mean value of the priorities. The median divides the data set into two equal parts and indicates the mean value. The individual priorities were aggregated using each of these methods independently to consider the different distributions resulting from the different methods. These results are presented in the “Data aggregation” subsection of the Results section.
Demographic statistics of the study population
x < 30
30 ≤ x < 50
50 ≤ x <70
x > 70
Estimated severity of the disorder
Aggregating single priorities is required to generate a summary of the study results. Depending on the data aggregation method, the ranks of the information criteria and the corresponding weights differ slightly. An advantage of using different methods separately is that the different distributions of the data sets can be considered and results can be compared between the methods.
We have demonstrated that the AHP can be used to identify patient priorities with regard to the information needs of people having rare diseases. For this purpose, group decisions were as suitable as individual decisions. Although the local rank of the information types resulted in a similar order of individual and group decisions, their global weights varied slightly. Interestingly, we found another important aspect: group judgements were in a significantly smaller range than individual judgements. This result may be correlated with the fact that group judgements are more frequently consistent. Hence, it could conceivably be hypothesized that using smaller ranges, e.g., a 7- or 5-point scale, would lead to more consistent answers. Unfortunately, we cannot compare the response behavior with that reported in other published studies, because such an analysis was not conducted in these studies . Furthermore, it can be argued that group decisions frequently represent the compromise solution of the group participants, and therefore, the group judgements are a mean of the individual judgements and consequently the group’s priorities have a more limited range. We attempted to avoid a situation in which the group participants gave only the mean of their individual judgements as their answer. Frequently, the group participants discussed the rationales behind the individual judgements and decided on a common group priority that was not the mean of the individual judgements. Sometimes, the group judgement was even outside the range of the individual minimum and maximum judgements. There are, however, other possible explanations that should be investigated in further studies.
The findings of this study suggest that there is no “gold standard” method for data acquisition. According to our data, both the individual and group methods lead to very similar results. Moreover, there is no right or wrong ranking of the priorities of information needs. Researchers should select the most suitable method using other criteria, such as the thematic perspective of the study or the properties of the goods or topics that are addressed. It can be argued that, on the one hand, for free or non-rival goods, methods that involve individual decision-making are more suitable, because there is no need for the participants to be prepared to compromise; other people will not face disadvantages or advantages because of one individual’s decision. On the other hand, group decisions are suitable for scarce or rival goods. Another aspect that should be considered is the peer pressure exerted in group discussions. The group situation can lead to particular disadvantages when intimate insights should be given in the interview, in which case, individual participants do not dare to answer truthfully or do not state their personal opinions. With regard to the implementation of the rare disease information portal or other websites, the order of information categories should not be influenced by other users. Therefore, an individual user’s priorities shall be used to identify which information categories are more important and should be more accessible on the website than less important categories. In summary, the use of patient priorities to expand the user-friendliness of information websites using the AHP offers an important contribution for medical informatics.
According to our data, aggregations by median, arithmetic mean, and geometric mean lead to very similar rankings of information criteria. Because the arithmetic mean is very vulnerable to outliers, the median and the geometric mean appear to be acceptable alternatives for data aggregation, although the differences between the two methods depend on additional factors, such as the number of criteria in the hierarchy and the number of participants. Nevertheless, comparing the analyses using different methods offers the advantage of enabling consideration of the different distributions of the data sets.
The AHP method can lead to judgements that do not meet the defined CR requirement. We determined that the use of ranking cards prior to pairwise comparison of each category may help participants answer more consistently. Furthermore, we noticed that a comparison of four aspects of a category (such as the comparison of four information fields) is more challenging for participants than a comparison of three aspects of a category (such as the comparison of three information types) in terms of cardinal consistency. This fact was used to confirm the conditions for participation in this study: patients who were unable to concentrate on the questionnaire continuously were excluded, as well as children. This participation bias may lead to a non-representative ranking of the information needs of people suffering from a rare disease. Further applications of the AHP should consider restricting the number of pairwise comparisons in each category. Moreover, by setting a CR at ≤ 0.2, we could include a sufficient number of judgements in our analysis. If we had set a lower CR value, the number of included judgements would have been lower, and consequently, the informative value of this study would have been more limited.
Assumptions and limitations
The number of patients living with any one rare disease is limited. For this reason, we pooled patients with heterogeneous rare diseases, who frequently face similar challenges and have similar information needs. However, because of the relatively low number of participants interviewed in this study, the results may not be representative. Furthermore, a bias exists regarding the information criteria current events, because no information types were grouped in this information field. In addition, we attempted to minimize the interviewer bias, as well as the bias between telephone and face-to-face interviews.
To the best of our knowledge, this is the first study to investigate the differences in individual and group judgements when conducting an AHP. Our study demonstrated the need for better strategies for choosing an appropriate method. Both methods led to similar outcomes; however, the response behavior differed. In brief, we demonstrated that the AHP can be used to identify the importance of several information types to people having a rare disease, and to order these information types on a website that presents information on rare diseases. Using the results of the AHP, we could rank the information needs of people suffering from a rare disease and their relatives according to their priorities. These priorities can be used to constitute information categories that are more important and should be more accessible on the website than less important categories. Overall, the use of an AHP to identify patient priorities and expand the user-friendliness of information websites offers an important contribution to medical informatics. According to our data, the use of different methods for data aggregation had no distinct influence on the ranking of the information criteria.
The strength of our study is in the transparent comparison of the different approaches applied in the AHP. The study indicates appropriate methods for conducting an AHP in other healthcare settings and in the field of medical informatics. Even if the results of the data acquisition methods do not differ, as was shown in our data, it is important that the researcher explain and justify the choice of method. We suggest that researchers select a suitable method based on the thematic perspective of the study or the properties of the goods or topics they are addressing. For example, it can be argued that group judgements should be used for studies addressing goods with limited availability. This investigation yielded important findings for subsequent studies that use the AHP method as a tool for medical decision-making and identifying patients’ priorities.
AHP, analytic hierarchy process; CHERH, center for health economics research hannover; CR, consistency ratio; Ind, individual.
The authors wish to thank the patients and their families for participating in this study.
The Center for Health Economics Research Hannover (CHERH) is funded by the Federal Ministry of Education and Research. This study was funded in part by the Federal Ministry of Health Germany. The study sample consisted of randomly selected participants from the qualitative main study of the ZIPSE project.
Availability of data and materials
The datasets analyzed during the current study are available from the corresponding author on reasonable request.
FP carried out the analyses and drafted the manuscript. FP and KS prepared the data adequately. FP and AB collected the data. KS revised the manuscript. KD and MF made substantial contributions to the conception of the article. JMS revised the manuscript for important intellectual content. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Ethical approval was issued by the Ethics Committee of the Albert-Ludwigs-Universität Freiburg (53/14). Informed consent was obtained from all participants prior to the survey and interviews.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Schmidt K, Aumann I, Hollander I, Damm K, Schulenburg JM. Applying the Analytic Hierarchy Process in healthcare research: A systematic review and evaluation of reporting. BMC Med Inform Decis Mak. 2015;15:115.View ArticleGoogle Scholar
- Saaty TL. A Scaling Method for Priorities in Hierarchical Structures. J Math Psychol. 1977;15:234–81.View ArticleGoogle Scholar
- Pecchia L, Melillo P. Analytic Hierarchy Process for Health Technology Assessment: A Case Study for Selecting a Maintenance Service Contract. In: Ventre AGS, Maturo A, Hošková-Mayerová S, Kacprzyk J, editors. Multicriteria and Multiagent Decision Making with Applications to Economics and Social Sciences Studies in Fuzziness and Soft Computing. Heidelberg: Springer; 2013. p. 275–88.View ArticleGoogle Scholar
- Dolan JG. Are Patients Capable of using the Analytic Hierarchy Process and Willing to Use It to Help Make Clinical Decisions? Med Decis Mak. 1995;15:76–80.View ArticleGoogle Scholar
- Pecchia L, Bath PA, Pendleton N, Bracale M. Analytic Hierarchy Process (AHP) for Examining Healthcare Professionals’ Assessments of Risk Factors. Methods Inf Med. 2011;50:435–44.View ArticlePubMedGoogle Scholar
- Liberatore MJ, Nydick RL. The Analytic Hierarchy Process in Medical and Health Care Decision Making: A Literature Review. Eur J Oper Res. 2008;189(1):194–207. doi:10.1016/j.ejor.2007.05.001.View ArticleGoogle Scholar
- Hajrahimi N, Dehaghani SM, Hajrahimi N, Sarmadi S. Quality assessment of Isfahan Medical Faculty web site electronic services and prioritizing solutions using analytic hierarchy process approach. J Educ Health Promot. 2014;3:117. doi:10.4103/2277-9531.145920.View ArticlePubMedPubMed CentralGoogle Scholar
- Kaplan W, Wirtz VJ, Mantel-Teeusisse A, Stolk P, Duthey B, Laing R. Priority Medicines for Europe and the World 2013 Update, World Health Organization. 2013. http://www.who.int/medicines/areas/priority_medicines/MasterDocJune28_FINAL_Web.pdf. Accessed 21 Dec 2015.
- Regulation (EC) No 141/2000. Regulation (EC) No 141/2000 of the European Parliament and of the Council of 16 Dec 1999 on Orphan Medicinal Products. Off J Europ Com. 1999. http://ec.europa.eu/health/files/eudralex/vol-1/reg_2000_141/reg_2000_141_en.pdf. Accessed 21 Dec 2015.
- National Action Alliance for People with Rare Diseases. National Action Plan for People with Rare Diseases. 2013. http://www.namse.de/images/stories/Dokumente/Aktionsplan/national%20plan%20of%20action.pdf. Accessed 21 Dec 2015.
- Eidt D, Frank M, Reimann A, Wagner TOF, Mittendorf T, Schulenburg JM. Maßnahmen zur Verbesserung der gesundheitlichen Situation von Menschen mit seltenen Erkrankungen in Deutschland. 2009. http://www.bmg.bund.de/uploads/publications/BMG-G-09050-Bericht-Massnahmen-seltene-Krankheiten_200908.pdf. Accessed 21 Dec 2015.
- Chung KP, Chen LJ, Chang YJ, Chang YJ, Lai MS. Application of the Analytic Hierarchy Process in the Performance Measurement of Colorectal Cancer Care for the Design of a Pay-for-performance Program in Taiwan. Int J Qual Health Care. 2013;25:81–91.View ArticlePubMedGoogle Scholar
- Dolan JG, Boohaker MD, Allison J, Imperiale TF. Patients Preferences and Priorities regarding Colorectal Cancer Screening. Med Decis Mak. 2013;33:59–70.View ArticleGoogle Scholar
- Suner A, Celikoglu CC, Dicle O, Sökmen S. Sequential Decision Tree using the Analytic Hierarchy Process for Decision Support in Rectal Cancer. Artif Intell Med. 2012;56:59–68.View ArticlePubMedGoogle Scholar
- Danner M, Hummel JM, Volz F, van Manen JG, Wiegard B, Dintsios CH, Bastian H, Gerber A, Ijzerman AJ. Integrating Patients’ Views into Health Technology Assessment: Analytic Hierarchy Process (AHP) as a Method to Elicit Patient Preferences. Int J Technol Assess Health Care. 2011;27:369–75.View ArticlePubMedGoogle Scholar
- Taghipour H, Mohammadyarei T, Jafarabadi MA, Hashemi AA. On-Site or Off-side Treatment of Medical Waste: A Challenge. J Environ Health Sci Eng. 2014;12:86.View ArticleGoogle Scholar
- Dolan JG, Isselhardt BJ, Cappuccio JD. The Analytic Hierarchy Process in Medical Decision Making: A Tutorial. Med Decis Mak. 1989;9:40–50.View ArticleGoogle Scholar
- Saaty TL. The Analytic Hierarchy Process. New York: McGraw Hill; 1980.Google Scholar
- Ho W. Integrated Analytic Hierarchy Process and its Applications – A Literature Review. Eur J Oper Res. 2008;186:211–28.View ArticleGoogle Scholar
- Dolan JG. Shared Decision-making - Transferring Research into Practice: The Analytic Hierarchy Process (AHP). Patient Educ Couns. 2008;73:418–25.View ArticlePubMedPubMed CentralGoogle Scholar
- Brinkmeyer D, Müller R. Entscheidungsunterstützung mit dem AHP. Z Agrarinformatik. 1994;2:82–92.Google Scholar
- Saaty TL, Shang JS. Group Decision-making: Head-count versus Intensity of Preference. Socioecon Plann Sci. 2007;41:22–37.View ArticleGoogle Scholar
- Aczel TL, Saaty T. Procedures for Synthesizing Ratio Judgement. J Math Psychol. 1983;27:93–102.View ArticleGoogle Scholar