- Research article
- Open Access
- Open Peer Review
ACSC Indicator: testing reliability for hypertension
BMC Medical Informatics and Decision Makingvolume 17, Article number: 90 (2017)
With high-quality community-based primary care, hospitalizations for ambulatory care sensitive conditions (ACSC) are considered avoidable. The purpose of this study was to test the inter-physician reliability of judgments of avoidable hospitalizations for one ACSC, uncomplicated hypertension, derived from medical chart review.
We applied the Canadian Institute for Health Information’s case definition to obtain a random sample of patients who had an ACSC hospitalization for uncomplicated hypertension in Calgary, Alberta. Medical chart review was conducted by three experienced internal medicine specialists. Implicit methods were used to judge avoidability of hospitalization using a validated 5-point scale.
There was poor agreement among three physicians raters when judging the avoidability of 82 ACSC hospitalizations for uncomplicated hypertension (κ = 0.092). The κ also remained low when assessing agreement between raters 1 and 3 (κ = 0.092), but the κ was lower (less than chance agreement) for raters 1 and 2 (κ = -0.119) and raters 2 and 3 (κ = -0.008). When the 5-point scale was dichotomized, there was fair agreement among three raters (κ = 0.217). The proportion of ACSC hospitalizations for uncomplicated hypertension that were rated as avoidable was 32.9%, 6.1% and 26.8% for raters 1, 2, and 3, respectively.
This study found a low proportion of ACSC hospitalization were rated as avoidable, with poor to fair agreement of judgment between physician raters. This suggests that the validity and utility of this health indicator is questionable. It points to a need to abandon the use of ACSC entirely; or alternatively to work on the development of explicit criteria for judging avoidability of hospitalization for ACSC such as hypertension.
Internationally, there is widespread interest in the promise of ambulatory care sensitive conditions (ACSC). ACSC are chronic conditions that should be effectively managed in the community with appropriate medical screening, monitoring and follow-up.
Health care leaders are driven to reduce waste and inefficiency through eliminating unnecessary hospital admissions. Reducing avoidable hospitalizations is a leading healthcare reform in many provinces across Canada and internationally [1, 2]. Avoidable hospitalizations are a common and costly occurrence and policymakers have identified them as a quick and easy way to curb healthcare costs. In Canada, initiatives reducing ACSC are often referred to as "right care in the right place at the right time” [3,4,5,6].
Hospitalization rates for ACSC have been used in many jurisdictions as a proxy for the presence or absence of appropriate primary and preventive care; more physician visits within a community should result in fewer hospitalizations for ACSC . A lower rate of ACSC is often interpreted as being indicative of better access to primary care, or better quality of primary care services, resulting in prevention of hospitalizations. As a healthcare system performance indicator, ACSC have been used to inform health policy by assessing the primary health care system (including funding models), improving our understanding of the factors that influence health and by identifying gaps in health status and outcomes for specific populations . In Canada, individuals who experience an ACSC hospitalization represent 6.0% of all hospitalized Canadians, and account for nearly 11.0% of all hospital days . However, the variation in ACSC rates across provinces or regions could be caused by true quality of care differences, differences in data quality or other social factors.
Initially ACSC were developed using hospitalization data to identify hospitalizations in the general United States population that could potentially have been avoided with adequate access to appropriate ambulatory or primary care in the community [8, 10]. Since their inception in 1993, hospitalization rates for ACSC have provided the basis for measuring adequacy of ambulatory and primary health care performance in many regions and countries [11,12,13,14,15]. In its national health indicators report, the Canadian Institute for Health Information (CIHI) reports age-standardized hospitalization rates for seven ACSC: angina, asthma, chronic obstructive pulmonary disease, diabetes, epilepsy, heart failure, and hypertension.
Administrative hospital discharge abstract database (DAD) is the only data source for producing and reporting national and provincial rates of ACSC in Canada. Reasons for the widespread use of ACSC are that the DAD is ready to be analyzed and has wide geographic coverage. The DAD is generated by trained medical coders and includes information about all patients admitted to hospital. Each discharge record contains a unique identification number for each admission, location of residence, physician specialty, up to 25 coded diagnoses recorded using the International Classification of Disease 10th Revision, Canadian Modification(ICD-10-CA) coding system, and an indicator flagging the occurrence of death during a hospitalization. To identify ACSC hospitalizations for each of the seven ACSC conditions, CIHI searches the most responsible diagnosis coding field in the DAD using specific ICD codes (e.g. ICD-10-CA codes I10.0, I10.1, I11 for uncomplicated hypertension) . Exclusion criteria include those who died before discharge, individuals aged 75 years and older and admissions recorded as newborn or stillbirth.
Despite widespread use, it is unclear if ACSC is a reliable and valid indicator that accurately measures the performance of the primary health care system. ACSC hospitalization rates can potentially be used as a marker of access to and quality of primary care, however, only if a significant proportion of them are deemed avoidable and judgments of avoidability have high inter-rater reliability (i.e. judgments of avoidability are reproducible). Thus, the reliability of physician judgment of avoidable hospitalizations for ACSC derived from medical chart review needs to be evaluated. If the ratings of avoidable hospitalizations cannot be reproduced reliably between physicians, the validity and utility of this health indicator are questionable. If assessments of avoidable hospitalizations for ACSC derived from medical chart review are (reproducible) reliable between-physician, it will provide a methodological foundation for determining the proportion of ACSC hospitalizations that are truly avoidable based on medical charts. Thus, the purpose of this research was to test the inter-physician reliability (reproducibility) of judgments of avoidable hospitalizations for one ACSC, uncomplicated hypertension, derived from medical chart review. We chose hypertension as it is a highly prevalent condition affecting more than 20% of Canadians , the number one reason for primary care physician visits in Canada , and the management of hypertension and related complications is estimated to consume 10% of all health care spending [19, 20].
We used administrative health data, specifically the DAD from the province of Alberta, Canada from fiscal years 2012/13 to 2014/15. The DAD is generated by trained coders and includes information about all patients admitted to hospital primarily for monitoring Canadian national health services utilization. Each DAD record contains a unique identification number for each admission, a patient chart number, date of admission, date of discharge, location of residence, physician specialty, diagnoses (up to 25 coded diagnoses are recorded using the International Disease Classification 10th Revision, Canadian Modification (ICD-10-CA) coding system), procedures (up to 20), and an indicator flagging the occurrence of death during a hospitalization. The DAD also has a ‘diagnosis-type’ indicator. The coders assign a one digit ‘diagnosis-type’ code to specify the timing of diagnosis. Type M is the most responsible diagnosis, which is defined in Canada as the condition responsible for the greatest resource use during the hospital stay .
Identifying ACSC hospitalization for uncomplicated hypertension in the major diagnosis field
Following CIHI methods  (see Additional file 1) we identified ACSC hospitalizations for uncomplicated hypertension (i.e. ICD-10-CA code I10.0 benign hypertension, I10.1 malignant hypertension, and I11 hypertensive heart disease excluding cases with cardiac procedures) between April 1, 2012 and Aug 1, 2014 (fiscal years 2012 to 2014). In the DAD, specifically, we identified ACSC hospitalizations by searching the most responsible diagnosis coding field using ICD-10 codes I10.0x, I10.1x, I11.x and then excluded patients who underwent cardiac procedure, died in hospital, or those younger than age 20 or older than age 75 at the time of hospitalization. If a patient experienced more than one ACSC hospitalization in the study period only the first admission was selected.
Expert medical chart review
In the period of April 1, 2012 and Aug 1, 2014, a total of 238 patients with an ACSC hospitalization for uncomplicated hypertension were identified in the DAD at the Foothills Medical Centre in Calgary, Alberta. A random number was assigned to each patient and the number shorted decently. We selected the first 82 patient medical charts as the sample. To reach an inter-rater agreement of 0.75 among the three raters, 75 charts needed to be reviewed. To ensure stability of the kappa statistic, we selected 82 charts to be reviewed by 3 raters.
Chart review was conducted by three experienced physicians. We chose general internal medicine specialists as they typically have more experience with hypertension-related hospital admissions compared to other specialties (i.e. surgeons). The three physicians routinely worked at hospitals and ambulatory care clinics, and had significant experience admitting patients. All physicians underwent training, including a discussion of data extraction, the definition of complications and avoidability (defined as a hospitalization that might ordinarily have been prevented ), and the use of implicit methods for judging avoidability . We did not provide specific diseases for defining vascular complications given that the nature of physician’s judgement is implicit. During training the physician raters reviewed 10 charts together.
The physicians used a questionnaire for the medical chart review that included items to confirm the inclusion and exclusion criteria (see Additional file 2). Once the physicians had gone over all aspects of the chart (including cover page, emergency department notes, narrative summaries, admission notes, consultation reports, surgery/operative reports, physician daily progress notes, physician orders and discharge summaries) they used their professional judgment to implicitly judge the avoidability of the hospitalization using a validated 5-point scale [25,26,27] (see Table 1). Experts commonly use implicit methods for clinical judgment, which rely on the subjective opinion of the individual judge; no predetermined criteria or factors are used . Implicit methods have traditionally been used to rate avoidability of hospital readmissions  and are also commonly used during medical panel ratings or Delphi processes [29,30,31,32].
ACSC hospitalization for uncomplicated hypertension in coding fields 2 to 25
In addition to searching the major diagnosis field for uncomplicated hypertension as reason for admission as per CIHI’s methodology, we search the secondary diagnosis coding fields (2 to 25) in the DAD using ICD-10 codes I10.0x, I10.1x, I11.x between April 1, 2012 and Aug 1, 2014. We did this to assess coding misclassification of uncomplicated hypertension (i.e. non-coding of uncomplicated hypertension in the major diagnosis code). In Canada, the most responsible diagnosis coding field is defined as the condition responsible for the greatest resource use during the hospital stay whereas in the United States it is defined as the reason for hospital admission . Thus, searching secondary diagnosis coding fields allows us to determine if patients were admitted to hospital for hypertension and had a subsequent condition such as pneumonia that resulted in long length of stay at hospital. For those patients the pneumonia was likely to be coded as the most responsible condition and uncomplicated hypertension is likely to be coded as secondary diagnosis.
A total of 12,917 patients with a secondary diagnosis of uncomplicated hypertension were identified in the DAD at the Foothills Medical Centre in Calgary, Alberta. A random sample of 320 medical charts were selected and reviewed by 2 trained nurses. The nurses reviewed the discharge summaries. If uncomplicated hypertension was mentioned or listed as a most responsible diagnosis, the chart was flagged. All of the flagged charts were then reviewed by the same three physicians as per the methodology described above.
Other study variables
The DAD was used to determine all other study variables: age, sex, income quintile, region, and Charlson Comorbidity. A measure of socioeconomic status and income quintile was assigned using the 2001 Statistics Canada Census data. Specifically, median household income for each Census dissemination area, the smallest geographic unit for which Census data are released, was linked to each patient’s postal code assigned using the Statistics Canada Postal Code Conversion file  Rural and urban status was also defined using postal code. Charlson comorbidities  were derived from the DAD and three categories of comorbidity were considered: (1) no comorbidity; (2) vascular risk-related comorbidity (myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, diabetes with and without complications and renal disease); and (3) unrelated comorbidity (ie, unrelated to vascular risk including dementia, chronic pulmonary obstructive disease, connective tissue disease, rheumatic disease, peptic ulcer disease, paraplegia/hemiplegia, liver disease, cancer/metastatic carcinoma, and AIDS/HIV).
Reproducibility on the 5-point scale across raters was assessed. The reason for using the scale is that avoidable hospitalization is determined by multiple factors, including disease severity (such as acuity of disease), family support to patients, self-management of disease, healthcare system or continuity of care planning (from primary care to tertiary care and vice versa), geographic location, and socio-economic status . There are no objective clinical criteria to determine appropriateness of admission. Then, the scale was dichotomized and assessed using the following cut point: not-avoidable points 1-3 and avoidable points 4-5 for easy interpretation. Fleiss’s kappa statistics for the 5-point scale and Cohen’s kappa statistic (κ) for the dichotomized variable and their p-value were used to assess agreement among the physician raters . The κ was interpreted as follows: a near perfect agreement (0.81 ≤ κ < 1.0), substantial agreement (0.61 ≤ κ < 0.80), moderate agreement (0.41 ≤ κ < 0.60), fair agreement (0.21 ≤ κ < 0.40), slight agreement (0.01 ≤ κ < 0.20), and less than chance agreement (κ < 0) . Analyses were performed with SAS statistical software version 9.3 (SAS Institute Inc, Cary, North Carolina). The institutional ethics review board for the University of Calgary approved the study.
ACSC hospitalization for uncomplicated hypertension in the major diagnosis field
In total, 238 ACSC hospital admissions for uncomplicated hypertension were identified in the most responsible diagnosis coding field at the Foothills Medical Centre in fiscal years 2012 to 2014. Among the 82 patients whose medical charts were reviewed, 34.2% were younger than 50, 36.6% were aged 50 to 64 years, and 46.3% were male. Most patients resided in an urban setting (95.1%), and 45.1% had no Charlson comorbidities (Table 2). The average length of stay for an ACSC hospitalization for uncomplicated hypertension was 4.5 days.
Using the validated 5-point scale, there was poor agreement among the three physician raters when judging the avoidability of ACSC hospitalization (κ = 0.092, Table 3). The κ remained low level when assessing agreement between raters 1 and 3 (κ = 0.092), but the κ was lower (less than chance agreement) for raters 1 and 2 (κ = -0.119) and raters 2 and 3 (κ = -0.008). When the 5-point scale was dichotomized, there was fair agreement among three raters (κ = 0.217) and among rater 1 and 3 (κ = 0.388). The agreement was very low for raters 2 and 3 (κ = 0.157), and raters 1 and 2 (κ = -0.009).
The proportion of ACSC hospitalizations for uncomplicated hypertension that were rated as avoidable was 32.9%, 6.1% and 26.8% for raters 1, 2, and 3, respectively (Table 4). The Venn diagram (Fig. 1) describes the agreement between raters, showing that 3 cases (3.7%) were rated as avoidable by all 3 raters, 12 cases (14.6%) by only 2 raters, and 21 cases (25.6%) by only 1 rater. The number of cases rated as not avoidable by all 3 reviewers, by only 2 and by only 1 rater was 46 (56.1%), 21 (25.6%) and 12 (14.6%), respectively.
ACSC hospitalization for uncomplicated hypertension in coding fields 2 to 25
There were 12,917 hospital admissions with uncomplicated hypertension listed as a secondary diagnosis (coding fields 2 to 25 in the DAD) at the Foothills Medical Centre and the Peter Lougheed Centre in Calgary, Alberta during the study period. Among the randomly chosen 320 medical charts reviewed by 2 nurses, 14 (4.4%) were flagged as admissions where uncomplicated hypertension was potentially the most responsible diagnosis. After reviewing the medical charts the three physicians agreed that 6 of the 14 (42.8%) admissions were actually ACSC hospital admissions for uncomplicated hypertension. When the 5-point scale was dichotomized, one physician rated one chart as an avoidable hospitalization. All other charts were rated as not avoidable by all three physician raters.
Internationally, ACSC hospitalization rates are being used as a quality-of-care measure. Indeed ACSC admissions may indicate challenges with access to or signal sub-optimal primary care quality, but only if these ACSC admissions are truly avoidable. This study assessed the reliability of specialist ratings of avoidable ACSC hospitalizations for uncomplicated hypertension derived from medical chart review. Importantly, this study found a low proportion of hospitalizations rated as avoidable, and poor to fair agreement of judgment between physician raters. Given that the ratings of avoidability were low and were poorly reproduced, the study findings suggest that the validity (both face and construct) and utility of this health indicator is questionable. Therefore it may be unlikely that ACSC rates would significantly decrease with improved primary care management of uncomplicated hypertension as this health indicator may be too “soft” to measure performance of the primary healthcare system.
Although our study findings question the premise of the ACSC indicator, it may only be true at the level of implicit judgment of medical charts. In implicit review, sometimes termed peer review, the reviewer judges what is being studied (i.e. avoidability of hospitalization) against his/her own clinical knowledge, opinions and beliefs. Thus, implicit review may be biased by reviewers’ experience, consistency, attention to detail, and harshness of judgment . This begs the question: Can multiple physician reviewers judge avoidability of ACSC hospitalizations in the same way based on their own clinical experience? Our findings clearly demonstrate that this is difficult to do, which reflects findings from other studies assessing avoidability of hospital readmissions and quality of care [24, 39,40,41].
One potential reason for the low ratings of avoidability and the poor to fair agreement of judgment between reviewers may be due to the definition of ACSC; the concept of ACSC is clear (i.e. no need to be hospitalized if the patient receives appropriate out-patient care), but it is hard to operationalize in practice. Another reason is that a medical chart may be insufficient to judge avoidability, which requires comprehensive information such as primary care visits, hypertension management, diagnoses/investigations, drug adherence and patient self-management. When this information is not documented in the medical chart, judging the avoidability of a hospital admission poses a great challenge. In the future, seamless health data exchange between in-patient and out-patient providers will likely provide a more comprehensive resource to assess these important factors. Another potential reason is differing views of the definition of avoidability. For example, one rater could view lack of drug adherence as a patient issue and judge the hospital admission as unavoidable whereas another rater may believe it is amenable to intervention by the physician and hence judge the admission as avoidable. Lastly, there are no clear clinical guidelines for hospital admissions for patients with uncomplicated hypertension, making judgments of avoidability challenging.
One way to improve agreement of judgment of avoidability of ACSC hospitalizations for uncomplicated hypertension would be to develop explicit criteria. Explicit judgments rely on predetermined criteria set by group agreement. Explicit review has a certain “boilerplate” character because the same standards are applied to every case  and the burden of accuracy falls on the criteria, not on the reviewer. Unfortunately, such criteria for judging avoidable hospitalization due to ACSC are not available. To increase reproducibility of ratings explicit methods to guide judgment of avoidable hospitalizations based on medical charts (i.e. tools such as a checklist or decision tree) need to be developed.
Currently the DAD is the only data source for producing and reporting on ACSC statistics. In Canada, the most responsible diagnosis coding field is defined as the condition responsible for the greatest resource use during the hospital stay whereas in the United States it is defined as the reason for hospital admission . Thus, we completed a medical chart review where uncomplicated hypertension was listed in the secondary diagnosis field to determine if patients were admitted to hospital for hypertension and had a subsequent complication such as stroke or congestive heart failure. We did this because for those patients the complication was likely to be coded as the most responsible condition. Our findings show that only a small proportion (1.6%) of charts were such cases. Further, it has been shown that coding hypertension in the DAD is done reasonably well. Quan et al. assessed hypertension coding accuracy using ICD-10 and reported sensitivity 68.3%, specificity 97.8%, positive predictive value 93.1%, and negative predictive value 87.7%. Overall it appears that coding misclassification does not significantly affect ACSC hospitalization rates.
Our study has limitations. First, medical chart review was conducted using only hospital medical records as linked hospital and outpatient medical records were not available. Second, we assessed only one of the seven ACSC reported by CIHI. However the ACSC that was chosen, hypertension, is a highly prevalent and high impact chronic condition commonly encountered within all clinical settings and is of major interest from a public health perspective. It is unclear if the results of this study are also applicable to the other ACSC conditions. Third, we reviewed medical charts in Calgary and did not assess for potential differences between tertiary care and community hospitals. Fourth, Canada has very high treatment and control rates for hypertension relative to other countries [17, 42]. Thus, these finding may only relate to Canada. In countries with low control rates of hypertension, the proportion of hospitalizations rated as avoidable may be higher.
In conclusion, this study found a low proportion of ACSC hospitalizations for uncomplicated hypertension were rated as avoidable. Assessments of avoidability were not reliably reproduced between physician raters, with only poor to fair agreement. These findings either points to a need to abandon the use of the ACSC entirely (since the notion of avoidable hospital stays is so central to the concept of ACSC); or alternatively a need to work on the development of more explicit criteria for judging avoidability (i.e. a Q&A checklist for physician reviewers, or a decision-tree logic model to aid decisions). As it stands currently, the validity and reliability of this health indicator are questionable and it is potentially dangerous to link ACSC hospitalization rates to primary care funding models. As ACSC is a potential tool to trigger positive cases for further exploration, we recommend future research to develop explicit criteria for judging avoidability of ACSC hospitalizations.
Ambulatory care sensitive conditions
Canadian Institute for Health Information
Discharge abstract database
International Classification of Disease 10th Revision, Canadian Modification
Ajmera M, Wilkins TL, Sambamoorthi U. Dual Medicare and Veteran Health Administration use and ambulatory care sensitive hospitalizations. J Gen Intern Med. 2011;26 Suppl 2:669–75.
Tian Y, Dixon A, Gao H. Emergency hospital admissions for ambulatory care-sensitive conditions: identifying the potential for reductions. The King's Fund. 2012;1-13.
Saskatoon Health Region. Complex Continuing Care Strategy 2010-2021.2010. https://www.saskatoonhealthregion.ca/locations_services/Services/Senior-Health. Accessed 12 Nov 2014.
Meloche J. Primary Health Care and Health System Design. Ontario Local Health Integration Network; 2012. http://www.lhins.on.ca. Accessed 3 Nov 2015.
Government of Saskatchewan. Ministry of Health and Health System Plan for 2015/16. 2014. http://www.finance.gov.sk.ca/PlanningAndReporting/2015-16/HealthPlan1516PRINT.pdf. Accessed 3 Jan 2015.
Alberta Government. Alberta's Primary Health Care Strategy. 2014. 2012. http://www.health.alberta.ca/documents/Primary-Health-Care-Strategy-2014.pdf. Accessed 2 Feb 2015.
Cloutier-Fisher D, Penning MJ, Zheng C, Druyts EB. The devil is in the details: trends in avoidable hospitalization rates by geography in British Columbia, 1990-2000. BMC Health Serv Res. 2006;6:104.
Brown AD, Goldacre MJ, Hicks N, Rourke JT, McMurtry RY, Brown JD, et al. Hospitalization for ambulatory care-sensitive conditions: a method for comparative access and quality studies using routinely collected statistics. Can J Public Health. 2001;92:155–9.
Sanmartin C, Khan S. Hospitalizations for Ambulatory Care Sensitive Conditions: The factors that matter. Ottawa: Statistics Canada; 2011. Report No.: 82-622-X.
Pappas G, Hadden WC, Kozak LJ, Fisher GF. Potentially avoidable hospitalizations: inequalities in rates between US socioeconomic groups. Am J Public Health. 1997;87:811–6.
Nyweide DJ, Weeks WB, Gottlieb DJ, Casalino LP, Fisher ES. Relationship of primary care physicians' patient caseload with measurement of quality and cost performance. JAMA. 2009;302:2444–50.
Rizza P, Bianco A, Pavia M, Angelillo IF. Preventable hospitalization and access to primary health care in an area of Southern Italy. BMC Health Serv Res. 2007;7:134.
Paramore LC, Elixhauser A. Assessment of quality of care for managed care and fee-for-service patients based on analysis of avoidable hospitalizations. Value Health. 1999;2:258–68.
Backus L, Moron M, Bacchetti P, Baker LC, Bindman AB. Effect of managed care on preventable hospitalization rates in California. Med Care. 2002;40:315–24.
Laditka SB, Laditka JN. Geographic variation in preventable hospitalization of older women and men: implications for access to primary health care. J Women Aging. 1999;11:43–56.
Canadian Institute for Health Information. Health Indicators 2013. Ottawa: Canadian Institute for Health Information; 2013.
McAlister FA, Wilkins K, Joffres M, Leenen FH, Fodor G, Gee M, et al. Changes in the rates of awareness, treatment and control of hypertension in Canada over the past two decades. CMAJ. 2011;183:1007–13.
Hemmelgarn BR, Chen G, Walker R, McAlister FA, Quan H, Tu K, et al. Trends in antihypertensive drug prescriptions and physician visits in Canada between 1996 and 2006. Can J Cardiol. 2008;24:507–12.
World Health Organization. World Health Organization 2008-2013 action plan for global strategy for the prevention and control of noncommunicable diseases: prevent and control cardiovascular diseases, cancers, chronic respiratory diseases and diabetes. Geneva: World Health Organization; 2008.
Cecchini M, Sassi F, Lauer JA, Lee YY, Guajardo-Barron V, Chisholm D. Tackling of unhealthy diets, physical inactivity, and obesity: health effects and cost-effectiveness. Lancet. 2010;376:1775–84.
Quan H, Parsons GA, Ghali WA. Assessing accuracy of diagnosis-type indicators for flagging complications in administrative data. J Clin Epidemiol. 2004;57:366–72.
Canadian Institute for Health Information. Health Indicators 2011. Ottawa; Canadian Institute for Health Information; 2011.
Guo L, MacDowell M, Levin L, Hornung RW, Linn S. How are age and payors related to avoidable hospitalization conditions? Manag Care Q. 2001;9:33–42.
Brook RH, Appel FA. Quality-of-care assessment: choosing a method for peer review. N Engl J Med. 1973;288:1323–9.
Forster AJ, Murff HJ, Peterson JF, Gandhi TK, Bates DW. Adverse drug events occurring following hospital discharge. J Gen Intern Med. 2005;20:317–23.
Barbara O, Jose SM, Jayna HL, Ward F, Maeve O, Deborah W, et al. A framework to assess patient-reported adverse outcoms stemming from hospitalization. BMC Health Serv Res. 2016;16:357.
Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170:1678–86.
Yam CH, Wong EL, Chan FW, Wong FY, Leung MC, Yeoh EK. Measuring and preventing potentially avoidable hospital readmissions: a review of the literature. Hong Kong Med J. 2010;16:383–9.
Jette N, Quan H, Tellez-Zenteno JF, Macrodimitris S, Hader WJ, Sherman EM, et al. Development of an online tool to determine appropriateness for an epilepsy surgery evaluation. Neurology. 2012;79:1084–93.
Quan H, Drosler S, Sundararajan V, Wen E, Burnand B, Couris CM, et al. Adaptation of AHRQ Patient Safety Indicators for Use in ICD-10 Administrative Data by an International Consortium. Advances in Patient Safety: New Directions and Alternative Approaches. Rockville, MD: Agency for Health Care Research and Quality. 2008. http://www.ahrq.gov/qual/advances2/. Accessed 2 May 2014.
Kennedy J, Quan H, Feasby TE, Ghali WA. An audit tool for assessing the appropriateness of carotid endarterectomy. BMC Health Serv Res. 2004;4:17.
Feasby TE, Quan H, Tubman M, Pi D, Tinmouth A, So L, et al. Appropriateness of the use of intravenous immune globulin before and after the introduction of a utilization control program. Open Med. 2012;6:e28–34.
Statistics Canada. Postal code converstion file (PCCF), Reference guide. 2006. https://www.google.ca/?gws_rd=ssl#q=Statistics+Canada.+Postal+code+conversion+file+(PCCF),+Reference+guide.+2006. Accessed 29 Jan 2015.
Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi JC, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. 2005;43:1130–9.
Muenchberger H, Kendall E. Predictors of preventable hospitalization in chronic disease: priorities for change. J Public Health Policy. 2010;31:150–63.
Chen B, Zaebst D, Seel L. A macro to calculate kappa statistics for categorizations by multiple raters. Philadelphia: Sugi, editor. 2005. SAS. 11-5-2014.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–74.
Weingart SN, Davis RB, Palmer RH, Cahalane M, Hamel MB, Mukamal K, et al. Discrepancies between explicit and implicit review: physician and nurse assessments of complications and quality. Health Serv Res. 2002;37:483–98.
Goldfield N. How important is it to identify avoidable hospital readmissions with certainty? CMAJ. 2011;183:E368–9.
van Walraven C, Bennett C, Jennings A, Austin PC, Forster AJ. Proportion of hospital readmissions deemed avoidable: a systematic review. CMAJ. 2011;183:E391–402.
Ashton CM, Kuykendall DH, Johnson ML, Wray NP. An empirical assessment of the validity of explicit and implicit process-of-care criteria for quality assessment. Med Care. 1999;37:798–808.
Statistics Canada. Blood pressure of adults, 2012 to 2013. Health Fact Sheets. Statistics Canada Catelogue no. 82-625-X. 2014. http://www.statcan.gc.ca/pub/82-625-x/2016001/article/14657-eng.htm. Accessed 8 Mar 2015.
None to declare.
Availability of data and materials
Data will be available upon request from the corresponding author.
RW, WG, GC, ED, DR, NJ and HQ designed the study. TK, BM, NC, and RD collected the data. RW, GC and HQ analyzed the findings. RW and HQ drafted the paper. All authors critically revised the paper and approved of the final version of the manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
This study was approved by the University of Calgary’s Conjoint Health Research Ethics Board (REB15-1180).
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.