Combining Numerical and Spatial Data for Community Health Assessment

Modern-day CHA professionals in developed countries frequently analyze public health data in order to identify health priorities. The steps in the process might be the:

• Identification of the spatial location of a geographic community using GIS or a paper map;

• Identification of health factors within the community using numerical data such as death counts, disease incidence or prevalence rates;

• Identification of the spatial location of bordering communities of interest using GIS or a paper map;

• Identification of health factors within bordering communities using numerical data such as death counts, disease incidence, or prevalence rates;

• Comparison of factors within the community against factors of the bordering community using statistical methods for adjustment and calculations such as relative risk and odds ratios;

• Viewing of results using tables, graphs, or spatial visualization.

The first step (of identification of the location of a geographic community) is a spatial component. This step represents the act of merely locating the area or region of interest on a map. The second step, identifying the health factors within the community, is purely numerical. For example, the ranking of top 5 diseases per 100,000 for a particular age category aggregated at the community level is a numerical process. However, the next step, identifying the bordering communities of interest is purely spatial. Like the first step, this can be done by using a map. The identification of health factors in these counties is purely numerical as in step 2. Statistical measures and adjustments are performed in order to determine health priorities.

Many community health experts use Information Technology (IT) for this type of data analysis. We conducted a survey of CHA professionals and found that many of them use software such as databases, statistical packages, and even GIS [15]. The potential for OLAP-GIS in community health data analysis is not well understood. We thus decided to conduct an evaluation comparing OLAP-GIS to information technology (IT) that is commonly used, including GIS and traditional analytical/statistical tools. We hypothesized that using an OLAP-GIS system instead of the combined use of a SPSS and GIS would greatly facilitate CHA data analysis when considering efficiency, accuracy and user satisfaction.

SOVAT

At the University of Pittsburgh, we have developed an OLAP-GIS system called the Spatial OLAP Visualization and Analysis Tool (SOVAT) [16, 17]. SOVAT is intended to support community health assessment data analysis. The system combines large amounts of health and population data and displays the information through a graphical user interface. The interface, developed using an iterative design approach [18], supports direct user manipulation as well as analysis of numerical and spatial components (Figure 1).

The SOVAT interface contains the ability to navigate through large public health data sets by using OLAP functions such as: drill-down (view more detailed data), drill-up (view more aggregated data), and slice and dice (view specific variables of data). In addition to these functions, SOVAT contains unique functions that are not standard in OLAP but were believed to enhance community health assessment. One such feature is called drill-out, which enables the user to click on a map object such as a county, and submit a query that contains both numerical and spatial aggregation. For example, to perform drill out on a 'region A', SOVAT would first identify the regions that border region A. This would be done through spatial analysis of the coordinates. Then the system would aggregate the numerical measures (such as an incidence rate) for each bordering region. This function enables the user to quickly perform comparisons of different geographical areas across different numerical public health measures.

We evaluated SOVAT against technology that we previously determined to be commonly used by CHA professionals, namely the combined use of SPSS statistical software and GIS software (referred to here as SPSS-GIS) [15], in order to understand its potential as a data analysis tool during community health assessments.

Recruitment and Setting

The participants were recruited via fliers that were posted around the University of Pittsburgh campus. The essential inclusion criterion was that the participants had experience using SPSS. Interested participants replying that they had never heard of SPSS or had used it a couple of times, were not enrolled in the study.

The study took place in a testing room within the Department of Biomedical Informatics at the University of Pittsburgh. The room contained a desk and chair for the participant, a laptop computer, and an overhead screen and projector.

Software used in the Study

The current technology (SPSS-GIS) comprised two separate software applications: SPSS statistical software 13.0 and ArcView 9.1. SPSS, GIS, and SOVAT applications were all run locally off of the laptop during the evaluation.

Study Procedures

Before entering the testing room, participants were asked to complete the informed consent form. Each session lasted approximately two and a half hours and was divided into two parts: training and evaluation. Once in the room, the participants were shown a pre-recorded instructional video that served as the introductory script for using the system being evaluated. They were allowed to take notes during this time. The content of the video, including the facets of the interface and the methodologies for producing queries, was deemed appropriate for use in the study by one of the co-investigators (VM) who is an expert in Human-Computer Interaction (HCI). After watching the video, the participants were given two practice tasks to solve using the system. After completing each task, they were shown a video solution for that task.

Once the participants completed the five tasks, they were asked to complete the IBM Post-Study System Usability Questionnaire (PSSUQ) [20]. This is mainly a close-ended questionnaire that has been found to be both a reliable and valid instrument for measuring user system satisfaction [20]. The PSSUQ is oriented in a 7-point Likert Scale format with lower numbers indicating higher levels of satisfaction. In addition to measuring overall system satisfaction, the questionnaire can be divided across three categorical areas: system usefulness, information quality, and interface quality.

After completing the second session, participants also completed a short one-on-one interview that lasted less than five minutes. The purpose of this interview was to go beyond the numeric responses from the satisfaction questionnaire and obtain more qualitative feedback regarding their attitudes towards both systems.

Objective Measurements

The researchers believed that two essential criteria for determining system potential were efficiency and accuracy. Efficiency is a well-defined usability metric [19] and is often represented in the literature as time to task completion. Accuracy is an especially important criterion in community health assessment. Both efficiency and accuracy likely lead to a greater sense of confidence (i.e. positive feeling when using the system) and ultimately greater system use. In addition, the allocation of both financial and human resources for community health improvement is often based on conclusions drawn from data analysis. Software applications that lead to erroneous results and conclusions, could lead an inappropriate use of resources (of both time and money). The variables are described as:

• Time to complete each task (Efficiency) – This measure was defined by the time between when a participant finished reading the question to when the participant indicated he/she was done. The use of screen capture software allows one to measure the participant's time for each task. This screen capture method is also non-intrusive.

• Answer to Problem (Accuracy) – An answer was defined as the action of the participant verbalizing an answer to all the questions in the task followed by saying that they were 'done'. The answer did not have to be the same as what was currently being shown on the screen at the time. The participant had to answer all parts of the question correctly to successfully answer the task.

Subjective Measurements

As mentioned, the PSSUQ was used for subjective measurement analysis of user satisfaction. A brief post-study interview was also conducted immediately following the completion of the second session. The question posted to every participant was "Which software system did you like better and why"? User preference was identified from the responses.

Statistical Analysis

Both descriptive and inferential statistics were calculated for analysis purposes. Descriptive statistics were used for time, answer, satisfaction, and user preference. Inferential statistics were calculated by conducting mixed model analysis. This method enabled for design, period, and intervention effects to be identified across the variables time and user satisfaction. Statistical analysis was conducted using SPSS 13.0 for Windows.

Time

Figure 2 shows the mean and 99% confidence interval for time rounded to the nearest minute. The results are shown by task by period. The five tasks are named based on their most distinguishable characteristic and are: boundary detection, community creation, state-wide comparison, ranking analysis, and municipality-level analysis.

Success Rate

Figure 3 shows the success rates for the study. The success rate is equal to the number of tasks answered correctly divided by the number of tasks attempted. For SOVAT, all tasks were attempted. For SPSS-GIS, two participants attempted only three of the five tasks.

The bars indicate that the participants were more accurate using SOVAT than SPSS-GIS, yet the overlapping 99% confidence intervals suggests that the differences are not significant. Examining the specific tasks, the community creation task and the state-wide comparison were the most difficult tasks to perform using SPSS-GIS.

User Satisfaction

Figure 4 shows the mean and 99% confidence intervals for the PSSUQ, with overall, as well as the satisfaction categories, by period. As mentioned, lower scores indicate higher levels of satisfaction.

The subjective data shows that SOVAT is perceived as more satisfactory across all periods and satisfaction categories than SPSS-GIS. Analyzing the three specific categories (not shown), system usefulness showed the greatest mean difference, while interface quality had the smallest mean difference.

User Preference

Before completing the study, participants were asked additional questions such as, "What system did you like better and why?" In total, twelve of the thirteen participants (92%) preferred SOVAT, while one of the thirteen participants (8%) preferred the combination of SPSS and GIS.

The majority of the positive responses towards SOVAT were in relation to its ease of use. The participants also liked the layout and design of the SOVAT interface better than SPSS-GIS. The most popular response in relation to this theme was that they like "1 program vs. 2." Hence, having to go back and forth between numerical and spatial data displays was not as popular as the combined interface of SOVAT.

Mixed Model Analysis: Time

At the .01 level, there is no group effect for any of the tasks. This indicates that the participants were sufficiently randomized to each group in relation to the variable time. The p-values for period indicate that there is no period effect for time. This indicates that the period (1 or 2) does not effect the time to complete the tasks. The intervention effect is significant at the .01 level. This indicates that the type of system used (SOVAT or SPSS-GIS) impacted the time to complete the tasks. As supported by the descriptive results, the participants completed the tasks in much shorter time with SOVAT than when they used a combination of SPSS and GIS.

Mixed Model Analysis: User Satisfaction

The group effect corresponds to the treatment* period interaction which is an alias for a carryover effect [22]. As can be seen, the group effect is significant at the .01 level for overall satisfaction, system usefulness, and interface quality. It is not significant for information quality (p = .108). This indicates that a carryover effect was present in relation to participant responses on the satisfaction questionnaire.

In relation to period effect, there is no significant effect at the .01 level, suggesting that the period does not influence the satisfaction level of the participant. The intervention effect is significant at the .01 level for all satisfaction categories. This supports the mean results from the satisfaction questionnaire (Figure 4) that suggest that the participants are more satisfied with SOVAT than with SPSS-GIS.

OLAP-GIS Use in Community Health

A survey involving community health professionals in Canada showed that 70% of the respondents felt that GIS could enhance their community health decision making [23]. Despite its growing popularity, a significant limitation of GIS is that it is not designed to support numerical and multidimensional data exploration. Combining OLAP with GIS can enhance this process. Examples of using OLAP for public health decision making [24–29] show that members in the field are beginning to recognize it. Commercial products such as ESRI's OLAP add-on for ArcGIS and Microsft's OLAP add-on for MapPoint offer widespread availability and utilization of OLAP-GIS within the public health community.