This article has Open Peer Review reports available.
Accuracy of automatic syndromic classification of coded emergency department diagnoses in identifying mental health-related presentations for public health surveillance
© Liljeqvist et al.; licensee BioMed Central Ltd. 2014
Received: 28 May 2014
Accepted: 11 September 2014
Published: 23 September 2014
Syndromic surveillance in emergency departments (EDs) may be used to deliver early warnings of increases in disease activity, to provide situational awareness during events of public health significance, to supplement other information on trends in acute disease and injury, and to support the development and monitoring of prevention or response strategies. Changes in mental health related ED presentations may be relevant to these goals, provided they can be identified accurately and efficiently. This study aimed to measure the accuracy of using diagnostic codes in electronic ED presentation records to identify mental health-related visits.
We selected a random sample of 500 records from a total of 1,815,588 ED electronic presentation records from 59 NSW public hospitals during 2010. ED diagnoses were recorded using any of ICD-9, ICD-10 or SNOMED CT classifications. Three clinicians, blinded to the automatically generated syndromic grouping and each other’s classification, reviewed the triage notes and classified each of the 500 visits as mental health-related or not. A “mental health problem presentation” for the purposes of this study was defined as any ED presentation where either a mental disorder or a mental health problem was the reason for the ED visit. The combined clinicians’ assessment of the records was used as reference standard to measure the sensitivity, specificity, and positive and negative predictive values of the automatic classification of coded emergency department diagnoses. Agreement between the reference standard and the automated coded classification was estimated using the Kappa statistic.
Agreement between clinician’s classification and automated coded classification was substantial (Kappa = 0.73. 95% CI: 0.58 - 0.87). The automatic syndromic grouping of coded ED diagnoses for mental health-related visits was found to be moderately sensitive (68% 95% CI: 46%-84%) and highly specific at 99% (95% CI: 98%-99.7%) when compared with the reference standard in identifying mental health related ED visits. Positive predictive value was 81% (95% CI: 0.57 – 0.94) and negative predictive value was 98% (95% CI: 0.97-0.99).
Mental health presentations identified using diagnoses coded with various classifications in electronic ED presentation records offers sufficient accuracy for application in near real-time syndromic surveillance.
Syndromic surveillance in Emergency Departments (EDs) endeavours to achieve near real-time monitoring of routinely collected health-related data to signal uncommonly high levels of presentations of particular syndromes [1–3]. Syndromes may be collections of symptoms or characteristics which are grouped into categories, often focussed on early pre-diagnostic signs and symptoms, or prodromes. Syndromic surveillance has proven useful in the early detection of trends in emergency department presentations for a variety of syndromes [4–6]. One of the challenges in designing a surveillance system is to achieve the best balance between sensitivity and specificity. An over-sensitive system is likely to produce false positives and false alarms, and to be less specific than is useful, while the opposite may result in missing important signals. Good stewardship of human and other resources demands efforts to achieve the optimal balance based on available resources and technology.
Measurements of the accuracy of ED data streams to identify particular syndromes using International Classification of Diseases (ICD) and Systematized Nomenclature of Medicine - Clinical Terms (SNOMED CT) concepts [7, 8] have been carried out before [9, 10], but the syndrome of mental health-related ED presentations lacks such attention.
Inclusion of “Mental Health” as a syndrome in routine real-time surveillance is considered appropriate for a number of reasons. Mental illness accounts for a greater disability burden in developed countries than any other disease group, including cancer and heart disease . Monitoring ED presentations related to mental health problems is important as it provides information relevant to policy, planning and guidance for interventions that may be required locally or on a broader scale .
In addition, increases in mental health-related ED presentations may occur due to emergencies or disasters causing collective anxiety. Natural and man-made disasters may disrupt access to mental health services and medications, which could result in an increase in both the number and severity of ED presentations for mental health problems. One of the threats that led to the relatively recent introduction of, and surge in, syndromic surveillance was the potential use of chemical, biological and radiological (CBR) weapons by governments or terrorists [13–15]. Some of these agents can cause alterations in mental health state on a mass scale. Potentially, industrial accidents may result in release of toxins or chemicals with mind altering effects.
Development of methods to rapidly collect disease data for this purpose has accelerated in recent years. Greater use of electronic medical records and patient management information systems has improved the completeness and uniformity of data collection systems [16, 17]. These developments may have improved the accuracy of ED syndromic surveillance.
Mental health codes in ICD 9 and ICD 10 and SNOMED CT concepts
290, 293.2-302, 306–307.80, 307.82-316.99, 799.2, V71.0
F03-F04, F06-F09, F20-F54, F59-F69, F84-F99, R44-R46, Z03.2, Z04.6, Z86.5
SNOMED CT concepts that map to any of the ICD-9 or ICD-10 codes (For complete list of concepts, see Additional file 1)
Syndromic surveillance relies on accurate recording of information in the data sources that are used. The aim of this study was to estimate the accuracy of using emergency department coding to identify mental health related visits. To our knowledge, there have been no studies investigating the accuracy of mental health codes compared with written information from triage nurse notes.
A random sample of n = 500 electronic records was selected from a database of all recorded ED visits (N = 1,815,588) in calendar year 2010 for the 59 hospitals participating in ED syndromic surveillance. In choosing the sample size, we balanced the burden of manual classification by the clinicians and the need for reasonably constrained confidence intervals.
Identifying mental health related ED visits
For the study, the sample of 500 records was classified in two ways. First using the automated syndrome grouping based on the diagnosis code for “Mental health problems” and second using a reference standard. Bertens, et al.  described criteria for external validation of reference standards when using expert panels in diagnostic research. Their recommendations include four considerations: constitution of the expert panel; information presented to experts; decision process (e.g. classification or case definition); and validity of the expert’s diagnoses (e.g. agreement testing). We followed these criteria when establishing the reference standard in this study.
The reference standard was obtained using the triage nurse notes included on the surveillance records. The notes were examined separately by three clinicians: a psychiatrist (Psych), an ED physician (EDP) and an Intensive Care Paramedic (pre-hospital emergency clinician (PHEC)). These three practice areas were selected in order to represent three phases of the patient journey and hence three different clinical perspectives. Each clinician was blinded to the diagnostic code recorded in the electronic record and to the assessment of the other two clinicians.
Mental disorders are defined according to the clinical diagnostic criteria. For this study, a mental disorder is defined as a condition which affects a person’s cognitive, emotional or social abilities and attracts a diagnosis of psychiatric illness.
Mental health problems
This includes, but is not restricted to, such things as, stress, anxiety or depression. Individuals with mental health problems may never meet the diagnostic threshold for a mental disorder.
-A mental disorder or mental health problem is the main reason for the ED visit.
-A mental disorder or mental health problem is a major contributing factor leading to the condition which is the main reason for the ED visit.
The clinicians’ results were assessed for agreement with each other using the Kappa statistic for inter-rater reliability in SAS . There are several ways of measuring agreement, but the Kappa statistic is used most frequently in the medical literature . Viera and Garrett described a scale for interpreting the Kappa statistic: <0 = Less than chance agreement; 0.01-0.20 = Slight agreement; 0.21-0.40 = Fair agreement; 0.41-0.60 = Moderate agreement; 0.61-0.80 = Substantial agreement; and 0.81-0.99 = Almost perfect agreement .
This was done to assess whether the clinicians’ opinions were in agreement as a measure of reliability and to provide a valid reference standard against which to compare the automatic classification of coded ED diagnoses. A reference standard called ‘AllClin’ was created which included the records about which all three clinicians agreed.
After establishment of the ‘AllClin’ reference standard, the automated coding from the ED syndromic data extraction was compared with the reference standard using the Kappa statistic to assess agreement. Finally, the ‘AllClin’ reference standard was used to estimate the sensitivity, specificity and positive predictive value (PPV) and negative predictive value (NPV) of the ED syndrome-coded method of identifying mental health-related visits to EDs. As an additional level of investigation, we examined the triage notes and provisional diagnoses of the false positive and false negative cases.
From the random sample of 500 records of any-cause ED visits selected from the NSW ED syndromic surveillance database, the code group for mental health identified 21 (4.2% (95% CI 2.4% - 6%)) mental health related presentations.
Agreement between the clinicians
Clinician’s agreement with each other in identifying mental health related ED visits in the 500 sample records
Psychiatrist compared with emergency department physician
P = < 0.001
95% CI (0.70 - 0.92)
Pre-hospital emergency clinician compared with psychiatrist
P = < 0.001
95% CI (0.77 - 0.96)
Emergency department physician compared with pre-hospital emergency clinician
P = < 0.001
95% CI (0.79- 0.97)
Agreement between automatic syndromic classification of coded ED data and the reference standard
Automatic syndromic classification of coded emergency department diagnoses measured against the reference standard of AllClin*
P = < 0.001
95% CI (0.58 - 0.87)
95% CI (46% - 84%)
95% CI (98% - 99.7%)
95% CI (57% - 94%)
95% CI (97%- 99%)
We found that the accuracy of using emergency department coding with ICD 9, ICD 10 and SNOMED CT concepts to identify mental health-related visits in EDs was satisfactory. Compared with clinician review of corresponding triage notes the routine system performed with high specificity and lower but still acceptable sensitivity as it identified over two thirds of mental health related visits to EDs.
It is reasonable to assume that specificity must be high since a record must be made of the reason for the visit being within the mental health set of parameters. On the other hand, as suggested by examination of the triage notes of the false negative cases, a mental health condition may be a major contributing reason for an ED visit while not being the immediate condition treated. This would not be possible to identify by means of ED coding, and thus would reduce sensitivity. Despite this, the ED surveillance system assessed in this study provided sensitive results. Due to the accuracy of automatic syndromic classification of coded emergency department diagnoses in identifying mental health-related presentations, it is justified to continue to include it in routine public health surveillance.
This study was confined to the use of ED triage notes and ED diagnostic codes partly for logistical reasons, but also for the purpose of assessing this early, often pre-definitive diagnosis phase of the patient journey in an attempt to promote early detection of changes in patterns in mental health-related visits to EDs. Our study provides a foundation for future studies of trends and time series analyses in mental health presentations to EDs.
The current study examined the internal consistency of diagnosis within a clinical record and did not have an external diagnostic reference standard. However, as recommended by Bertens, et al.  we used three separate areas of clinical expertise, presented the information in a uniform manner with each clinician blinded to the result of the others, provided a clear case definition of mental health-related problems, and finally tested the agreement between the clinicians. Therefore, we believe that the reference standard was relevant to the aim of the study, which was to assess the utility of diagnostic codes for routine syndromic surveillance of mental health.
Another potential limitation of this study, but at the same time one of the motivations for it, is that clinical staff, sometimes experiencing competing priorities, must enter the coded data in patient records. This also occurs in different settings, e.g. rural, city, day-time, night-time, busy and less busy times and at different facilities between which there may be variations. This study cannot quantify the impact of such potential inconsistencies. The presence of the false positive cases, however, indicates that there may be mistakes in the data entry process. The inconsistencies we identified were due to provisional diagnoses labelling a mental health problem, such as anxiety when further examination of the record demonstrated that the problem was somatic, for example cardiac.
Syndromic surveillance by automatic syndrome grouping of coded ED diagnoses has been assessed previously by Liljeqvist et al.  in a study of influenza-like illness (ILI), which found that grouped coding was both sensitive and specific for the identification of ILI for ED visits. The mental health syndrome grouping is considerably broader in scope than that of ILI and thus may offer greater challenges in syndrome grouping. On the other hand, accuracy might decline if we were attempting to identify more specific mental health syndromes such as “anxiety”.
Skovgaard et al.  measured the accuracy of ICD-10 codes in the diagnostic classification of mental health problems in children aged 1½ years. Their study involved a smaller sample than our study and included a detailed examination by clinical specialists. They found that ICD-10 codes offer a sufficient frame for classification of mental health conditions in 1½ year old children if used by highly specialised clinicians in controlled settings.
Inconsistencies between triage notes and discharge records may demonstrate that a person’s initial complaint was not the actual cause of the problem leading to the ED visit. Fleischauer and others  used Kappa statistics to assess agreement between syndromic grouping of the chief complaint on presentation in EDs with discharge diagnoses in patient records. They found moderate agreement overall and recommended that the validity of automated surveillance coding could be improved by including discharge diagnoses. The clinical examination of triage notes performed in our study demonstrates an option which may offer more timely results and more sensitive identification than automated syndromic groupings alone and thus could identify cases which would otherwise be missed by automated methods.
Some challenges in ED surveillance have been described by authors such as Gorelick et al.  who discussed the problems inherent with coding within EDs and suggested that further study is required into coding practices in the ED setting. The high degree of unpredictability in the ED environment and varying degrees of competence among staff in coding, they suggest, threaten validity of coded data automatically extracted from EDs. The same argument is provided by O’Malley and others  who explored actual coding practices. They found that errors were made due to issues such as patient-clinician communication problems, lack of training in coding among coders, large amounts of information with lack of attention to detail by clinicians and both unintentional and intentional miscoding.
In 2000, Hirshon  described how limitations to ED data collection occurred due to lack of uniformity in data collection and in data entry. While some of these issues have been addressed through increased use of electronic systems and increased standardisation of medical records internationally, they are still improving and require further assessment.
Deliberations about any syndromic surveillance system that collects data from EDs should consider the large number of diagnoses that can be recorded and the lack of coding rules in ED triage compared with, for example, the coding in formal post-hospital admission records, such as discharge or separation records, which would offer more careful and accurate diagnostic data than the ED setting. There must be a trade-off between accuracy and timeliness however. This study identifies that there is a level of accuracy in ED mental health syndromic coding which makes it a useful addition to comprehensive surveillance despite the complexities within which it operates and while providing near real-time results.
Considering that increases in, or changes to, mental health-related ED presentation patterns can be triggered by a large range of events and community needs, achieving comprehensive surveillance that can identify changes in patterns as early as practicable is a high priority.
Another reason for including mental health problems in coded syndromic ED surveillance is the high burden mental health-related visits place on EDs. While the state of mental health of the population may not traditionally have been something for which early warning was a key priority, rather monitoring of trends over time was the goal, mental health has increasingly become a focus of emergency response organisations internationally and in Australia [12, 25]. Continuing to monitor mental health as a syndrome as is currently conducted in NSW offers a sufficiently sensitive gauge to detect mental health related visits in EDs. Increased sensitivity may be desired under some circumstances and then a practical solution, at least on a local level, would be to examine the text in triage notes as we demonstrated.
Syndromic surveillance cannot in itself offer a comprehensive epidemiological picture of any disease, but provides an important component of the most comprehensive surveillance programs. One of the weaknesses of syndromic surveillance is its lack of sensitivity and specificity in identifying specific diseases. As it monitors syndromes such as influenza like illness, chest pain or fever, it can only be an indicator of disease trends. A broad “mental health” syndrome is possibly the most challenging to define as it covers a vast range of conditions, some diagnosable and others not, some presenting as the actual reason for an ED visit and some hidden behind other conditions such as physical trauma or cardiac disease.
This study demonstrates that grouping of diagnosis codes from ED information systems for syndromic surveillance using ICD 9, ICD 10 and SNOMED CT concepts can identify mental health visits in EDs, and that it can identify these visits with adequate sensitivity and specificity to be useful in routine rapid syndromic surveillance.
Under circumstances where either automated signals indicate an anomaly or known events warrant deeper investigation, it may be useful to use clinicians to examine the ED records in more detail. Further development of free text extraction such as explored by Travers, et al.  may also be useful in increasing sensitivity of syndromic surveillance of mental health related visits in EDs. Ongoing analyses of the accuracy of syndromic ED coding of mental health related visits would be useful in increasing the depth of understanding of the available data and of the epidemiology of mental health conditions.
Ethical approval was provided by New South Wales Population Health and Health Services Research Ethics Committee: LNR/11/CIPHS/71.
The source data for this study were from patient information systems used routinely in New South Wales public hospitals and reported to the NSW Ministry of Health for public health surveillance purposes. The source data available for the study did not include database fields for patient name, date of birth or address so obtaining informed consent was not feasible. The study protocol was approved by the New South Wales Population Health and Health Services Research Ethics Committee.
We would like to acknowledge the New South Wales Public Health Officer Training Program which provided the opportunity for this research to be conducted.
- Cashmore AW, Muscatello DJ, Merrifield A, Spokes P, Macartney K, Jalaludin BB: Relationship between the population incidence of pertussis in children in New South Wales, Australia and emergency department visits with cough: a time series analysis. BMC Med Inform Decis Mak. 2013, 13: 40-10.1186/1472-6947-13-40.View ArticlePubMedPubMed CentralGoogle Scholar
- Freeman EJ, Colpe LJ, Strine TW, Dhingra S, McGuire LC, Elam-Evans LD, Geraldine SP: Public health surveillance for mental health. Prev Chronic Dis. 2014, 7 (1): Cited May 20, 2014 at URL: http://www.cdc.gov/pcd/issues/2010/jan/09_0126.htm Google Scholar
- Muscatello D, Churches T, Kaldor J, Zheng W, Chiu C, Correll P, Jorm L: An automated, broad-based, near real-time public health surveillance system using presentations to hospital Emergency Departments in New South Wales, Australia. BMC Public Health. 2005, 5: 141-152. 10.1186/1471-2458-5-141.View ArticlePubMedPubMed CentralGoogle Scholar
- Lazarus R, Kleinman KP, Dashevsky I, DeMaria A, Platt R: Using automated medical records for rapid identification of illness syndromes (syndromic surveillance): the example of lower respiratory infection. BMC Public Health. 2001, 1: 9-10.1186/1471-2458-1-9. doi:10.1186/1471-2458-1-9View ArticlePubMedPubMed CentralGoogle Scholar
- Zheng W, Aitken R, Muscatello D, Churches T: Potential for early warning of viral influenza activity in the community by monitoring clinical diagnoses of influenza in hospital emergency departments. BMC Public Health. 2007, 7: 250-10.1186/1471-2458-7-250.View ArticlePubMedPubMed CentralGoogle Scholar
- Hope K, Merritt T, Eastwood K, Main K, Durrheim DN, Muscatello D, Todd K, Zheng W: The public health value of emergency department syndromic surveillance following a natural disaster. Dep Health Commonw Aust Commun Diseases Intell. 2014, 32: Communicable Diseases Intelligence Volume 32, Number 1 - March 2008. Cited Aug 2014 at URL: http://www.health.gov.au/internet/main/publishing.nsf/content/cda-cdi3201m.htm Google Scholar
- World Health Organization, (n.d.): Classifications, International Classification of Diseases (ICD). 2014, Cited April 19, 2014 at URL: http://www.who.int/classifications/icd/en/ Google Scholar
- International Health Terminology Standards Development Organisation (2007). 2014, Cited April 19, 2014 at URL: http://www.ihtsdo.org/snomed-ct/snomed-ct0/
- Liljeqvist GTH, Staff M, Puech M, Blom H, Torvaldsen S: Automated data extraction from general practice records in an Australian setting: trends in influenza-like illness in sentinel general practices and emergency departments. BMC Public Health. 2011, 13: 435-doi:10.1186/1471-2458-11-435View ArticleGoogle Scholar
- Skovgaard AM, Houmann T, Christiansen E, Andreasen AH: The reliability of the ICD-10 and the DC 0–3 in an epidemiological sample of children 1½ years of age. Infant Ment Health J. 2005, 26 (5): 470-480. 10.1002/imhj.20065.View ArticleGoogle Scholar
- World health Organization: Department of Mental Health and Substance Abuse in collaboration with the Victorian Health Promotion Foundation and The University of Melbourne. 2004, Geneva: World health Organization, Cited April 21, 2014 at URL: http://www.who.int/mental_health/evidence/en/promoting_mhh.pdf Google Scholar
- NSW Ministry of Health: Emergency Response Coordination Unit, (2013). Mental Health Impacts. 2014, from URL: http://www.health.nsw.gov.au/emergency_preparedness/mental/Pages/default.aspx Google Scholar
- Bravata DM, McDonald KM, Smith WM, Rydzak C, Szeto H, Buckeridge DL, Haberland C, Owens DK: Systematic review: surveillance systems for early detection of bioterrorism-related diseases. Ann Intern Med. 2004, 140 (11): 910-922. 10.7326/0003-4819-140-11-200406010-00013.View ArticlePubMedGoogle Scholar
- Henning KJ: Overview of syndromic surveillance: what is syndromic surveillance?. CDC MMWR. 2004, 53: 5-11. Cited April 21, 2014 at URL: http://www.cdc.gov/mmwr/preview/mmwrhtml/su5301a3.htm Google Scholar
- Reis BY, Mandi KD: Time series modelling for syndromic surveillance. BMC Med Inform Decis Mak. 2003, 3: 2-10.1186/1472-6947-3-2.View ArticlePubMedPubMed CentralGoogle Scholar
- Hiller KM, Stoneking L, Min A, Rhodes SM: Syndromic Surveillance for Influenza in the Emergency Department–A Systematic Review. PLoS One. 2013, 8 (9): e73832-10.1371/journal.pone.0073832. doi:10.1371/journal.pone.0073832View ArticlePubMedPubMed CentralGoogle Scholar
- Smith PF, Hadler JL, Stanbury M, Rolfs RT, Hopkins RS: “Blueprint version 2.0”: updating public health surveillance for the 21st century. J Public Health Manag Pract. 2013, 19: 231-239. 10.1097/PHH.0b013e318262906e. doi:10.1097/PHH.0b013e318262906eView ArticlePubMedGoogle Scholar
- Bertens LCM, Broekhuizen BDL, Naaktgeboren CA, Rutten FH, Hoes AW, van Mourik Y, Moons KGM, Reitsma JB: Use of expert panels to define the reference standard in diagnostic research: a systematic review of published methods and reporting. PLoS Med. 2013, 10 (10): e1001531-10.1371/journal.pmed.1001531. doi:10.1371/journal.pmed.1001531View ArticlePubMedPubMed CentralGoogle Scholar
- SAS Institute Inc: SAS Campus Drive, Cary, North Carolina 27513, USA. All rights reserved. 2014Google Scholar
- Viera AJ, Garret JM: Understanding Interobserver Agreement. Kappa Stat Fam Med. 2005, 37 (5): 360-363. Cited April 21, 2014 at URL: http://www1.cs.columbia.edu/~julia/courses/CS6998/Interrater_agreement.Kappa_statistic.pdf Google Scholar
- Fleischauer AT, Silk BJ, Schumacher M, Komatsu K, Santana S, Vaz V, Wolfe M, Hutwagner L, Cono J, Berkelman R, Treadwell T: The validity of chief complaint and discharge diagnosis in emergency department–based syndromic surveillance. Acad Emerg Med. 2004, 11: 1262-1267. doi:10.1197/j.aem.2004.07.013PubMedGoogle Scholar
- Gorelick M, Knight S, Alessandrini EA, Stanley RM, Chamberlain JM, Kupperman N, Alpern ER: Lack of agreement in pediatric emergency department discharge diagnoses from clinical and administrative data sources. Acad Emerg Med. 2007, 14: 646-652. 10.1111/j.1553-2712.2007.tb01852.x.View ArticlePubMedGoogle Scholar
- O’Malley KJ, Cook KF, Price MD, Raidford Wildes K, Hurdle JF, Ashton CM: Measuring diagnoses: ICD code accuracy. Health Serv Res. 2005, 40: 1620-1639. 10.1111/j.1475-6773.2005.00444.x.View ArticlePubMedPubMed CentralGoogle Scholar
- Hirshon JM: The rationale for developing public health surveillance systems based on emergency department data. Acad Emerg Med. 2000, 7: 1428-1432. 10.1111/j.1553-2712.2000.tb00503.x.View ArticlePubMedGoogle Scholar
- World Health Organization: Building Back Better Sustainable Mental Health Care after Emergencies. 2013, Retrieved March 2014 from URL: http://apps.who.int/iris/bitstream/10665/85377/1/9789241564571_eng.pdf?ua=1 Google Scholar
- Travers D, Haas SW, Waller AE, Schwartz TA, Mostafa J, Best NC, Crouch J: Implementation of emergency medical text classifier for syndromic surveillance. AMIA Annu Symp Proc. 2013, 2013: 1365-1374.PubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/14/84/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.