- Research article
- Open Access
- Open Peer Review
A quantitative analysis of qualitative studies in clinical journals for the 2000 publishing year
BMC Medical Informatics and Decision Makingvolume 4, Article number: 11 (2004)
Quantitative studies are becoming more recognized as important to understanding health care with all of its richness and complexities. The purpose of this descriptive survey was to provide a quantitative evaluation of the qualitative studies published in 170 core clinical journals for 2000.
All identified studies that used qualitative methods were reviewed to ascertain which clinical journals publish qualitative studies and to extract research methods, content (persons and health care issues studied), and whether mixed methods (quantitative and qualitative methods) were used.
60 330 articles were reviewed. 355 reports of original qualitative studies and 12 systematic review articles were identified in 48 journals. Most of the journals were in the discipline of nursing. Only 4 of the most highly cited health care journals, based on ISI Science Citation Index (SCI) Impact Factors, published qualitative studies. 37 of the 355 original reports used both qualitative and quantitative (mixed) methods. Patients and non-health care settings were the most common groups of people studied. Diseases and conditions were cancer, mental health, pregnancy and childbirth, and cerebrovascular disease with many other diseases and conditions represented. Phenomenology and grounded theory were commonly used; substantial ethnography was also present. No substantial differences were noted for content or methods when articles published in all disciplines were compared with articles published in nursing titles or when studies with mixed methods were compared with studies that included only qualitative methods.
The clinical literature includes many qualitative studies although they are often published in nursing journals or journals with low SCI Impact Factor journals. Many qualitative studies incorporate both qualitative and quantitative methods.
Quantitative studies provide answers or insights for many important questions or issues in health care and clinical research. Other important questions dealing with why, how, contexts, and experiences of individuals or groups, can be best addressed using qualitative methods. Other issues benefit from interleaving or integration of both research traditions. Miller and Crabtree , describe their experiences working in family medicine, a clinical domain where balancing qualitative and quantitative research styles benefits both patients and their families and health care professionals. They embrace holding "quantitative objectivism in one hand and qualitative revelations in another" and encourage others to use findings from both paradigms in understanding and practicing effective health care. Creswell and colleagues expand on this theme by stating that "When used in combination, both quantitative and qualitative data yield a more complete analysis, and they complement each other" .
Most studies in the major clinical journals have been quantitative studies. Very few qualitative studies and even fewer that combine both qualitative and quantitative approaches are published. An example of the breadth of qualitative studies and how findings and results can be combined across paradigms is a study by Jolly and Wiles [3, 4] who used mixed methods to study a nurse-led intervention for 422 adults after myocardial infarction and 175 adults with new-onset angina in 67 general practices in the United Kingdom. Their study showed statistically insignificant results at 1 month for eating healthy food, participating in exercise programs, and successful smoking cessation. Although patients in the nurse-led group were more likely to attend a rehabilitation program (37% vs. 22%, P = 0.001) attendance was disappointingly low. The researchers interviewed a group of patients using qualitative methods and found that people felt survival after a myocardial infarction indicated that the event had not been all that serious. Health care professionals often communicated simplified data about recurrence and being "back to normal" in 6 weeks. Because of these two issues, patients felt that their cardiac problems had probably been mild and therefore were not sufficiently motivated to implement major lifestyle changes.
Another example of the use of mixed methods was research done by Willms and Wilson and their colleagues [5–7] on smoking cessation. They found the meanings that patients who smoked attributed to their cigarettes (peer acceptance, coping during a time of stress and feeling out of control, feeling more like an adult, and smoking as more glamorous, tough, and rebellious) had more influence on cessation than did such external conditions as nicotine gum or counseling. Until the complex issues of why individuals smoke were dealt with, few were motivated to change their attitudes towards smoking and thus stop smoking.
Another effective example of integrated qualitative (ethnography) and quantitative (epidemiology) methods was a study done by Borkan and colleagues  to determine predictors of recovery after hip fracture in elderly patients. Traditional predictors such as age, type of break, and comorbidity, were collected by using standard questionnaires. In-depth interviews were used to collect injury narratives focusing on internal explanations of the fracture, sense of disability, and view of the future after hip fracture. None of the epidemiology factors predicted successful outcomes but those who perceived their fracture as more external or mechanical as opposed to an internal or organic problem (e.g., related to chronic disease) were more likely to have good recovery. Persons who perceived their disability in the context of autonomy, independence, and connection with the outside world also showed better ambulation at 3 and 6 months than persons with a more narrow and confined view of the fracture and its resulting disability.
Donovan and colleagues  used mixed methods to study prostate cancer screening and treatment choices to determine why study recruitment was lower than expected. Rousseau and Eccles and their colleagues [10, 11] used qualitative methods (case interviews) to explain the limited use of computerized guidelines for asthma and angina in a primary care study done in the United Kingdom. Many other examples exist; Creswell and colleagues describe 5 additional mixed methods studies in primary care as well as provide criteria for evaluating mixed methods studies .
We postulate that qualitative studies, either stand-alone reports or studies with mixed methods, are occurring more frequently in health care. This paper was done to describe the publishing of qualitative studies in 1 year of clinical literature, document and present the range of content and techniques in these studies, and establish a baseline for subsequent studies. We defined our sample to include all articles published in a set of major general medical, mental health, or nursing journals during 2000. We determined how many qualitative studies were published and in which journals, and extracted design methods and healthcare content, and how often studies used mixed methods and analyses. Because the nursing literature published a higher proportion of qualitative studies in our sample we also compared studies published in nursing journals with other journals to ascertain if quantitative differences exist across disciplines in the use of qualitative methods. Our analysis is a quantitative review of qualitative studies in health care in 2000.
The Health Information Research Unit of the Department of Clinical Epidemiology and Biostatistics, Faculty of Health Sciences at McMaster University in Hamilton, Ontario, Canada was the editorial office for four evidence-based summary journals in 2000: ACP Journal Club (internal medicine content), Evidence-Based Medicine (family/general practice content), Evidence-Based Nursing (general care nursing content), and Evidence-Based Mental Health (mental health care content). Their purpose is to provide enhanced abstracts and commentaries on important high-quality original studies and review articles for their respective clinical audiences. To identify these studies and review articles, 6 research staff read major clinical journals to ascertain if articles were in 1 or more categories of therapy, diagnosis, prognosis, etiology, economics, clinical prediction guides, differential diagnosis, and qualitative studies and if so, did each meet predefined methodology criteria for study quality. For 2000 we intensified our data collection to provide data to update and develop new clinical retrieval searching hedges for MEDLINE, PsycINFO, CINAHL, and EMBASE using methods described by Haynes and colleagues. One hundred and seventy journals provided data for this article.
The staff of the Health Information Research Unit has established quality criteria for the 8 categories of clinical literature that must be met before articles are judged appropriate for clinical application and publication in an abstract journal. Qualitative studies have 3 criteria:
• content relates to how people feel or experience certain situations, specifically those that relate to health care
• data collection methods and analyses are appropriate (primary analytical mode is inductive rather than deductive)
• units of collection and analysis are ideas, thoughts, concepts, phrases, incidents, or stories that become categories or themes.
The reading methods have been developed during the past 13 years and inter-rater reliability kappa (chance adjusted agreement) for identifying categories and applying criteria is consistently > 80%.
For this paper, KAM, one of the readers, analyzed the qualitative studies. Qualitative systematic reviews were excluded leaving only reports of original studies. These were assessed to extract journal title, qualitative study type, data collection methods, research question, persons studied, setting, and disease or health condition considered. In addition, studies with mixed methods were further analyzed although we did not use stringent criteria for assessing the quality  of the combination of methods. We identified mixed methods articles using a loose criterion of "some numerical or statistical analysis of quantitative data or qualitative data that had been turned into quantitative data". (An example of quantifying qualitative data is the study done by Borkan and colleagues  on hip fracture.) The analysis had to be fairly substantial–for example, a simple descriptive analysis of baseline demographics of the participants was not sufficient to be included as a mixed methods article.
In addition, Giacomini and Cook [14, 15], as part of the Evidence-Based Working Group in the Users' Guides to the Medical Literature, describe attributes that they have identified as belonging to high-quality qualitative studies: participant selection, data collection, and analysis methods. These aspects were also extracted for analysis in this report.
Data were taken from article abstracts and if needed, the full text was reviewed. Methodologies assessed were phenomenology, grounded theory, ethnography, case studies, narrative analysis, participant action, critical incident techniques, and discourse analysis. Author descriptions were used and if an additional methodology was found it was added to the list of types using definitions and descriptions from the Handbook of Qualitative Analysis, 2nd edition by Denzin and Lincoln . Data collection and sampling procedures were also extracted. Multiple designations were allowed. To assess the reproducibility a random 10% (n = 35) sample of citations was reviewed using predefined decision rules by another researcher trained in research methods.
The 170 journals included 60 330 articles of which 31 496 (52%) contained original data or were review articles. 3830 of these (6%) passed criteria for being high-quality and clinically relevant in 1 of the 8 categories. 367 articles met quality criteria for original studies or reviews of qualitative studies. Table 1 lists the journals that published at least 1 qualitative study. Twelve systematic reviews were excluded leaving 355 qualitative studies for assessment. Approximately 0.6% of all articles in the 170 journals and 9% of all high-quality, clinically relevant studies were qualitative studies.
The reproducibility of the categorization was measured by kappa (chance adjusted agreement): 0.92 for disease/condition, 0.83 for groups studied, 0.81 for setting, 0.73 for data collection, and 0.63 for data analysis type. The agreement for data analysis type was disappointing but not surprising in that 20% of the studies did not label their analyses necessitating assignment of analysis type by the data extractors. Agreement was low for participant selection methods (kappa 0.5) and therefore data on participant selection methods are not reported.
The 355 qualitative studies appeared in 48 journals (mean 7.4 articles per journal, range 1 to 86). These 48 journals were only 28% of the 170 clinical journals being read. Most of the qualitative studies were published in nursing journals: The 17 nursing titles included 214 qualitative studies (61% of all of the qualitative studies).
Few qualitative studies were published in the high-circulation, general healthcare journals. Using SCI Impact Factor ranking for 2000, only 4 of the top 20 journals (Table 2) published qualitative studies. These 4 journals published 15 qualitative studies with BMJ publishing 12. The highest-ranking journal with qualitative studies was Annals of Internal Medicine, ranked number 6. JAMA, ranked number 2, published articles about qualitative studies in 2000 [14, 15] but did not publish any qualitative studies.
Mixed qualitative and quantitative studies
37 qualitative studies (11%) included qualitative and quantitative methods and analyses. These were published in 17 journals with only 1 article in BMJ from the top 20 journal titles in Table 1. Social Science and Medicine published 10 of these mixed methodology studies–the most of any title studied.
Content of the studies is shown in Table 3. Many studies dealt with a range of participants and settings. Patients (56%), family (22%), and other non-health care professionals (14%) were studied more often than health professionals (nurses (21%), physicians (11%), and others (5%)). Non health care settings occurred more often with home or similar settings being studied in 44% of studies and other community settings in 16%. Health care settings were the hospital (25%), clinic (17%), nursing home (5%), and the emergency department (2%).
Disease/condition breakdowns represented common health care situations: cancer (11%), mental health (10%), pregnancy and childbirth (9%), cerebrovascular disease (10%), general issues such as vaccinations or Internet use, and nonspecific spectrum of diseases (e.g., all patients in a clinic) (12%). Many uncommon issues were also assessed. For example, Tongprateep  reports a phenomenology study designed to help nurses better understand essential elements of spirituality and health among rural Thai elders.
Analysis of the 37 articles with mixed methods showed similar patterns for settings, persons studied, and disease/condition evaluated except that more physicians were studied (P < 0.025) and more situations dealing with injury (P < 0.001) were evaluated. For the 211 articles in Nursing journals, very little difference was also seen except that fewer physicians were studied (P < 0.001) and more studies were done outside clinical settings (P < 0.001).
Phenomenology (37%), grounded theory (35%), and ethnography (18%) were used most often with some case studies (7%), narrative analysis (6%), participant action (3%) research, critical incident techniques (1%), and discourse analysis (1%) (Table 4). More than one qualitative method was used in 8% of studies. This pattern of methodology choice was similar for the 37 mixed methods studies and the 211 Nursing articles except that mixed studies methods used relatively more case studies and the Nursing studies used fewer of them (P < 0.025). The mixed methods studies did not included participatory action research, critical incident technique, or discourse analysis studies, methods that could be difficult to combine with quantitative studies.
Semi-structured interviews were used (77%) with some focus groups (18%) and observation (14%). These methods are major data gathering techniques in qualitative studies. Questionnaires (7%), document analysis (6%), and structured (4%) and unstructured interviews (1%) were used less often. For mixed methods studies, patterns were similar although questionnaires were used more frequently (24% vs. 7%, P < 0.01). Nursing studies did not differ for data gathering techniques.
Sampling is important in all studies–often no single right way exists for a study question. Purposive, snowball, and theoretical sampling are often used in qualitative studies and random and consecutive sampling for quantitative studies. All methods were represented in this analysis but the breakdowns are not reported because of low inter-rater agreements for categorization and missing author information.
In 2000 the major clinical journals published many qualitative studies–approximately 9% of all high-quality, clinically relevant articles. Most of the qualitative studies were reports of original research although 12 (3%) were systematic reviews. Most of the qualitative studies were in nursing journals although some medical journals such as BMJ and Annals of Internal Medicine also published several. Three of the high circulation medical journals (New England Journal of Medicine, Lancet, and JAMA) and 16 of the top 20 clinical journals, based on SCI Impact Factors, did not publish any qualitative studies. This is likely a reflection on the emphasis on a positivist, numerical approach that many of these journals embrace. The difference in proportion of qualitative studiers published in nursing journals is probably because of two historical, but linked factors. Qualitative studies have roots in women's studies and the nursing profession has always dealt with the patient as much more of a whole person rather than basic sciences facts and numbers. Both of these factors lead to more emphasis on understanding and embracing qualitative methods for research and practice. This view is substantiated by the fact that MEDLINE indexes most of the qualitative studies under the term Nursing Methodology Research until 2003.
A substantial proportion of the qualitative studies (11%) included both qualitative and quantitative (mixed) data. In general, these mixed methods studies were similar to the single methodology studies except they did more assessments of physicians and relied more on questionnaires to gather data for analysis. The presence of these mixed methods or multipardigmatic studies as described by Miller and Crabtree  and Creswell  is encouraging for those who espouse harnessing methodologies appropriate for exploring, explaining, and interpreting the complexities and ranges of issues in health care practice and research.
It is also interesting comparing qualitative studies in Nursing and non-Nursing journals. Regardless of the differences in proportion of qualitative studies published, from a content point of view few differences exist between the Nursing and non-Nursing journals except that more physicians were studied in the non-Nursing journals and fewer studies were done in clinical settings–not unsurprising findings. This indicates that the content and methods of qualitative studies seem to be similar across disciplines or if the methods are combined with quantitative methods.
This review of the publication of qualitative studies is limited in several ways. The proportion of journals studied was very low in relation to the total number of journals published. MEDLINE indexes over 4000 journals and this number is still a relatively small proportion of all journals that deal with health care. In addition, all of the journals searched were published in English so we do not know about qualitative studies in other languages. Although our criteria were relatively strict for including qualitative studies, our criteria for mixed methods studies could certainly have been stronger. We did not count the number of high-quality quantitative studies that could have included some qualitative analyses. We studied only 1 year of publishing; much could have changed since 2002.
Qualitative studies provide insight into social, emotional, and experiential aspects of health and health care and as such, they have an important place in understanding health and health care. Hopefully more studies will be published and more will be published in the high impact (high circulation) journals. This paper provides a basis for measuring increases.
Qualitative studies are being done and are published in a wide range of healthcare journals. These journals however are not the highest impact journals. It is encouraging to see that the number of qualitative studies that were published in 2000 and also the number of studies that combined qualitative and quantitative methods. More can be done however to complete and publish qualitative studies, and where appropriate, integrate the best of both methodologies. Both qualitative and quantitative researchers and clinicians need to work together to make this happen. Journal editors can also encourage submission of qualitative and mixed methods studies and facilitate publication of those they do receive.
This work was done in partial fulfillment of PhD requirements for KAM. Both authors have supplied intellectual input in designing and implanting the survey. KAM has collected and analyzed the data and both authors have contributed to writing the paper and agree on its content.
ISI Science Citation Index
Miller WL, Crabtree BF: Clinical research. Handbook of Qualitative Research. Edited by: Denzin N K and Lincoln Y S. 2001, Thousand Oaks, CA, Sage Publications, 607-631. 2nd
Creswell JW, Fetters MD, Ivankova NY: Designing a mixed methods study in primary care. Annals of Family Medicine. 2004, 2: 7-12. 10.1370/afm.104.
Jolly K, Bradley F, Sharp S, Smith H, Mant D: Follow-up care in general practice of patients with myocardial infarction or angina pectoris: initial results of the SHIP trial. Southampton Heart Integrated Care Project. Fam Pract. 1998, 15: 548-555. 10.1093/fampra/15.6.548.
Wiles R: Patients' perceptions of their heart attack and recovery: the influence of epidemiological "evidence" and personal experience. Soc Sci Med. 1998, 46: 1477-1486. 10.1016/S0277-9536(97)10140-X.
Willms DG: A new stage, a new life: individual success in quitting smoking. Soc Sci Med. 1991, 33: 1365-1371. 10.1016/0277-9536(91)90280-P.
Gilbert JR, Wilson DM, Best JA, Taylor DW, Lindsay EA, Singer J, Willms DG: Smoking cessation in primary care. A randomized controlled trial of nicotine-bearing chewing gum. J Fam Pract. 1989, 28: 49-55.
Willms GG, Best JA, W Taylor D: A systematic approach for using qualitative methods in primary prevention research. Medical Anthropology Quarterly. 1990, 4: 391-409.
Borkan JM, Quirk M, Sullivan M: Finding meaning after the fall: injury narratives from elderly hip fracture patients. Soc Sci Med. 1991, 33: 947-957. 10.1016/0277-9536(91)90265-E.
Donovan J, Mills N, Smith M, Brindle L, Jacoby A, Peters T, Frankel S, Neal D, Hamdy F: Quality improvement report: Improving design and conduct of randomised trials by embedding them in qualitative research: ProtecT (prostate testing for cancer and treatment) study. Commentary: presenting unbiased information to patients can be difficult.[see comment]. BMJ. 2002, 325: 766-770. 10.1136/bmj.325.7367.766.
Rousseau N, McColl E, Newton J, Grimshaw J, Eccles M: Practice based, longitudinal, qualitative interview study of computerised evidence based guidelines in primary care.[see comment]. BMJ. 2003, 326: 314-10.1136/bmj.326.7384.314.
Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I: Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial.[see comment]. BMJ. 2002, 325: 941-10.1136/bmj.325.7370.941.
Haynes RB: ACP Journal Club: Purpose and procedure. [http://www/acpjc.org/shared/purpose_and_procedure.htm]
Haynes RB, Wilczynski N, McKibbon KA, Walker CJ, Sinclair JC: Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Med Inform Assoc. 1994, 1: 447-458.
Giacomini MK, Cook DJ: Users' guides to the medical literature: XXIII. Qualitative research in health care B. What are the results and how do they help me care for my patients? Evidence-Based Medicine Working Group. JAMA. 2000, 284: 478-482. 10.1001/jama.284.4.478.
Giacomini MK, Cook DJ: Users' guides to the medical literature: XXIII. Qualitative research in health care A. Are the results of the study valid? Evidence-Based Medicine Working Group. JAMA. 2000, 284: 357-362. 10.1001/jama.284.3.357.
Denzin NK, Lincoln YS: Handbook of Qualitative Research. 2001, Thousand Oaks, CA, Sage Publications, 2nd
Tongprateep T: The essential elements of spirituality among rural Thai elders. J Adv Nurs. 2000, 31: 197-203. 10.1046/j.1365-2648.2000.01212.x.
Creswell JW: Research Design: Qualitative, Quantative and Mixed Methods Approaches. 2003, Thousand Oaks, CA, Sage Publications, 2nd
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/4/11/prepub
This research was funded by the U.S. National Library of Medicine.
The Hedges Team who did the data collection, entry, and verification included Nancy Bordignon, Angela Eady, Brian Haynes, Susan Marks, Ann McKibbon, Doug Morgan, Cindy Walker Dilks, Stephen Walter, Nancy Wilczynski, and Sharon Wong. Catherine Arnot Smith provided the second review methods for inter-rater agreement.