A software tool for creating simulated outbreaks to benchmark surveillance systems
© Cassa et al; licensee BioMed Central Ltd. 2005
Received: 07 November 2004
Accepted: 14 July 2005
Published: 14 July 2005
Evaluating surveillance systems for the early detection of bioterrorism is particularly challenging when systems are designed to detect events for which there are few or no historical examples. One approach to benchmarking outbreak detection performance is to create semi-synthetic datasets containing authentic baseline patient data (noise) and injected artificial patient clusters, as signal.
We describe a software tool, the AEGIS Cluster Creation Tool (AEGIS-CCT), that enables users to create simulated clusters with controlled feature sets, varying the desired cluster radius, density, distance, relative location from a reference point, and temporal epidemiological growth pattern. AEGIS-CCT does not require the use of an external geographical information system program for cluster creation. The cluster creation tool is an open source program, implemented in Java and is freely available under the Lesser GNU Public License at its Sourceforge website. Cluster data are written to files or can be appended to existing files so that the resulting file will include both existing baseline and artificially added cases. Multiple cluster file creation is an automated process in which multiple cluster files are created by varying a single parameter within a user-specified range. To evaluate the output of this software tool, sets of test clusters were created and graphically rendered.
Based on user-specified parameters describing the location, properties, and temporal pattern of simulated clusters, AEGIS-CCT created clusters accurately and uniformly.
AEGIS-CCT enables the ready creation of datasets for benchmarking outbreak detection systems. It may be useful for automating the testing and validation of spatial and temporal cluster detection algorithms.
The public health information infrastructure is yielding real-time access to health data, enabling new approaches to surveillance for infectious outbreaks. Prior to the laboratory confirmation or physician diagnosis of an infectious disease, ill persons may exhibit behavioral patterns, symptoms, signs, or laboratory findings that can be tracked through a variety of data sources. The process of monitoring these data is often referred to as syndromic surveillance [1–3].
Real time outbreak detection algorithms tend to focus on the temporal and spatial patterns of cases. Some detection routines, such as Cusum , look at just one type of pattern, while others, such as the Space-Time Scan Statistic  incorporate both. What these algorithms have in common is an underlying model of typical, or baseline patterns, and the goal of detection is to recognize perturbations from the baseline . The outbreak detection performance of a surveillance system can be measured in terms of its ability to distill "signal" (a cluster of cases in time and/or space) from noisy baseline.
Benchmarking the performance of detection algorithms requires training and validation data. When real data are not available, simulated data are often used [6, 7]. Simulated outbreaks must reflect the diversity of threats that a surveillance system is expected to encounter and detect, whether these outbreaks occur naturally or are man-made.
Our approach to validating detection algorithms is to use semi-synthetic data, that is, authentic baseline data injected with artificial signals . These signals are defined by a controlled feature set of variable parameters such as the size, location, shape, and duration of simulated outbreaks. Here we describe a software tool, the AEGIS Cluster Creation Tool (AEGIS-CCT), that enables users to create simulated clusters with controlled feature sets, varying the desired cluster radius, density, distance, and relative location away from a central point. AEGIS-CCT does not require the use of an external geographic information system (GIS) program for cluster creation.
Parameters that can be altered when creating a single cluster.
Cluster ID Number
User specified reference or identification number for each cluster
Number of Points in the cluster
Number of patients or points in the generated cluster.
"Reference Point" GIS Location
The latitude-longitude coordinates of a reference point, which could be a hospital or a primary care facility, for example.
Maximum cluster radius
The distance between the outermost point in the cluster and the center of the cluster.
"Angle" from the reference point
The angle of the cluster measured counter-clockwise from due east of the reference point as zero degrees, using unit circle convention.
Distance from the reference point
The distance between the center point of the cluster and the reference point.
Numbers of Days the Cluster should span
The number of days from when the first person shows symptoms to when the last person does.
This specifies which of the three models of temporal progression to use. Additional models can be incorporated into the software.
Description and output filenames
The user can specify where the cluster data and user-specified cluster description will be written.
The AEGIS-CCT is a Java package including a geospatial engine and a user interface created using the Swing toolkit. The source for the entire package is provided under the Lesser GNU Public License  on a sourceforge.net development site . Full details and updates to AEGIS-CCT can be obtained online .
Geocoding and precision of location
Programmatic methods were implemented to assign latitude-longitude coordinates to simulated cluster points, taking into account physical earth surface distances and not relying on external GIS software. Inside an AEGIS-CCT GIS class, there are three primary methods to handle these conversions. The first is a method to find the distance between two locations, which uses the specific latitude-longitude of the reference point to create a ratio of degrees per meter for north-south latitude and east-west longitude. Artificial data points are created 0.05 degrees to the north or east, and the corresponding physical distances (x, y) are calculated using the Haversine Formula, described below. The ratio is then computed, dividing the artificial data point distance by the calculated physical distance in meters on the Earth's surface. A second method finds a point that is a specific physical distance, measured at a specified angle, from a reference point. The angle is measured from the Euclidian x-axis and increased in a counter-clockwise form. The output is a second GIS data point related to the reference point by the angle and distance specified by the user. The third method finds the number of degrees of latitude and longitude per unit of physical distance in each respective direction.
Placing patients on a map requires consideration of earth curvature and precise latitudes and longitudes. Spherical equations break down significantly at small distances, but the Haversine formula  provides computationally exact results in almost all circumstances. For this calculation, Earth has radius R, and the locations of two points in spherical coordinates (latitude and longitude) have names [lon1, lat1] and [lon2, lat2]. The Haversine Formula is calculated using the following code:
dlon = lon2 - lon1;
dlat = lat2 - lat1;
a = (sin(dlat/2)) ^2 + cos(lat1) * cos(lat2) * (sin(dlon/2)) ^2;
c = 2 * atan2(sqrt(a), sqrt(1-a));
d = R * c;
This implementation was quality tested for accuracy using a series of latitude-longitude pairs from a sample dataset, measuring distances between two data points. Those results were identical with distances calculated by commercially-available GIS software.
Outbreaks vary in their temporal progressions or epidemic curves. Three such progressions were implemented in AEGIS-CCT as date algorithms to model the ways in which a disease might manifest in a population over time: a random, a linear, and an exponential growth spread. Additional epidemiological date algorithms can be added by other users to the AEGIS-CCT by implementing a method to assign a specific temporal distribution within an array in Java.
Analysis of accuracy and uniformity of patient cluster data
Semi-synthetic datasets created by the cluster generator fall within a specified set of parameter-based boundaries. Cluster data points are created randomly within the domain defined by those parameters, so it is important to verify that the clusters are accurately created and are close to uniformly generated.
To measure the uniformity of generated clusters, 10 test clusters were created with 100 points in each cluster. The centroid of each set of cluster points was then calculated and compared to the specified center point of that cluster. In every case, the cluster centroid was within five percent of the specified cluster radius, in distance, from the specified center point. This result demonstrates that the datasets are uniform, within a small threshold, when they contain a sufficient number of points, as would be expected with a random distribution.
To measure the accuracy of the geocoding engine, 360 clusters were made, forming a circle, around a single center point, varying the angle evenly (one degree added per cluster,) and they each had a cluster center point that fell precisely along the circle defined by all points at the same radial distance from the original counterpoint. This same test was conducted at five randomly selected latitude-longitude locations and the same results were obtained.
Sample cluster parameters and output
Example of single cluster parameters and multiple cluster parameters.
Type of Cluster
Set of Clusters Varying Angle
Number of Points in Cluster
Reference Point Latitude
Reference Point Longitude
Angle from reference point
[varies, see below]
Distance from reference point
Number of Days
Linear time-growth cluster north of center point
Varied angle around center point and created 4 clusters.
Number of Clusters
Sample output to a comma separated value file from AEGIS-CCT. Note: Values are point identification number, longitude, latitude and day number.
... (to point with ClusterID 29)
The CCT can create new data files or it can append cluster data to an existing file. The artificial portion of a semi-synthetic dataset can then be automatically combined with baseline patient distributions by specifying a file that contains the baseline data. In future versions, the CCT may provide an xml schema to which input baseline data files should conform.
AEGIS-CCT does not yet have built-in procedures to generate more complicated time-series or spatial distributions. Extensibility was taken very seriously when creating AEGIS-CCT, and sufficient abstractions were made to allow for ease of adding additional models. The generator is also extensible in other ways so that additional parameters can be added if they are easily computable. Parameters can be added or deleted by updating the GUI and modifying the GenerateCluster method in the main geospatial class. Potential areas for expansion in future versions include log-linear and logarithmic time-series models as well as Gaussian spatial distributions. It will be necessary to determine what the most pertinent and physically realistic models of syndromic spread are. Once the most realistic scenarios are assessed and modeled, they can be programmatically implemented and incorporated into the cluster creation tool. As long as these distributions can be implemented using Java methods, it is possible to quickly add another temporal distribution to AEGIS-CCT.
Evaluation of surveillance systems for the early detection of outbreaks is particularly challenging  when the systems are designed to detect events for which there are a few or no historical examples. Some real-time surveillance systems are designed to provide early warning of a biological attack. Fortunately, few people have been infected with biological warfare agents, although there are notable exceptions. For example, residents of Sverdlovsk were exposed in 1979 during an accidental release of anthrax from a weapons plant  and there were eleven infections, resulting in five deaths in the Florida, New York and Washington DC mailed-anthrax attacks in 2001 . In the absence of sufficient real outbreak data, measuring the detection performance of a system requires simulation. AEGIS-CCT enables the ready creation of datasets for benchmarking outbreak detection systems.
Availability and requirements
Lists the following
* Project name: AEGIS Cluster Creation Tool
* Project home page: http://sourceforge.net/projects/chipcluster/
* Operating system(s): Platform independent
* Programming language: Java
* Other requirements: Java 1.3.1 or higher
* License: e.g. GNU LGPL
* Any restrictions to use by non-academics: none
List of abbreviations
Geographical Information Systems
Graphical User Interface
This work was supported by the National Institutes of Health through R01LM007970-01 from the National Library of Medicine, and by grant 2002-12-1 from the Alfred P. Sloan Foundation. This software was presented in part at the 2003 National Syndromic Surveillance Conference in New York, NY.
- Mandl KD, Overhage JM, Wagner MM, Lober WB, Sebastiani P, Mostashari F, Pavlin J, Gesteland PH, Treadwell T, Koski E, Hutwagner L, Buckeridge DL, Aller R, Grannis S: Implementing syndromic surveillance: a practical guide informed by the early experience. J Am Med Inform Assoc. 2004, 11: 141-150 [PrePrint published Nov 21, 2003; as doi:10.1197/jamia.M1356]. 10.1197/jamia.M1356.View ArticlePubMedPubMed CentralGoogle Scholar
- Tsui FC, Espino JU, Dato VM, Gesteland PH, Hutman J, Wagner MM: Technical description of RODS: a real-time public health surveillance system. J Am Med Inform Assoc. 2003, 10: 399-408. 10.1197/jamia.M1345.View ArticlePubMedPubMed CentralGoogle Scholar
- Lombardo J, Burkom H, Elbert E, Magruder S, Lewis SH, Loschen W, Sari J, Sniegoski C, Wojcik R, Pavlin J: A Systems Overview of the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE II). J Urban Health. 2003, 80: I32-I42..PubMedPubMed CentralGoogle Scholar
- Hutwagner LC, Maloney EK, Bean NH, Slutsker L, Martin SM: Using laboratory-based surveillance data for prevention: an algorithm for detecting Salmonella outbreaks. Emerging Infectious Diseases. 1997, 3: 395-400.View ArticlePubMedPubMed CentralGoogle Scholar
- Kulldorff M, Athas WF, Feurer EJ, Miller BA, Key CR: Evaluating cluster alarms: a space-time scan statistic and brain cancer in Los Alamos, New Mexico. Am J Public Health. 1998, 88: 1377-1380.View ArticlePubMedPubMed CentralGoogle Scholar
- Reis BY, Pagano M, Mandl KD: Using temporal context to improve biosurveillance. Proc Natl Acad Sci U S A. 2003, 100: 1961-1965. 10.1073/pnas.0335026100.View ArticlePubMedPubMed CentralGoogle Scholar
- Goldenberg A, Shmueli G, Caruana RA, Fienberg SE: Early statistical detection of anthrax outbreaks by tracking over-the-counter medication sales. Proc Natl Acad Sci U S A. 2002, 99: 5237-5240. 10.1073/pnas.042117499.View ArticlePubMedPubMed CentralGoogle Scholar
- Mandl KD, Reis BY , Cassa C: Measuring outbreak detection performance using controlled feature set simulations. MMWR. 2004, National Syndromic Surveillance Conference Proceedings:Google Scholar
- Open Source Initiative: GNU Lesser General Public License. [http://www.opensource.org/licenses/lgpl-license.php]
- SourceForge: AEGIS-CCT. [http://www.sourceforge.net/projects/chipcluster/]
- Sinnott RW: Virtues of the Haversine. Sky and Telescope. 1984, 68: 159-Google Scholar
- Buehler JW, Hopkins RS, Overhage JM, Sosin DM, Tong V: Framework for evaluating public health surveillance systems for early detection of outbreaks: recommendations from the CDC Working Group. MMWR Recomm Rep. 2004, 53: 1-11.PubMedGoogle Scholar
- Meselson M, Guillemin J, Hugh-Jones M, Langmuir A, Popova I, Shelokov A, Yampolskaya O: The Sverdlovsk anthrax outbreak of 1979. Science. 1994, 266: 1202-1208.View ArticlePubMedGoogle Scholar
- Jernigan JA, Stephens DS, Ashford DA, Omenaca C, Topiel MS, Galbraith M, Tapper M, Fisk TL, Zaki S, Popovic T, Meyer RF, Quinn CP, Harper SA, Fridkin SK, Sejvar JJ, Shepard CW, McConnell M, Guarner J, Shieh WJ, Malecki JM, Gerberding JL, Hughes JM, Perkins BA, Anthrax Bioterrorism Investigation T: Bioterrorism-related inhalational anthrax: the first 10 cases reported in the United States. Emerging Infectious Diseases. 2001, 7: 933-944.View ArticlePubMedPubMed CentralGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/5/22/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.