- Research article
- Open Access
- Open Peer Review
Using eye tracking and gaze pattern analysis to test a “dirty bomb” decision aid in a pilot RCT in urban adults with limited literacy
BMC Medical Informatics and Decision Makingvolume 16, Article number: 67 (2016)
Eye tracking is commonly used in marketing to understand complex responses to materials, but has not been used to understand how low-literacy adults access health information or its relationship to decision making.
This study assessed how participants use a literacy appropriate “dirty bomb” decision aid. Participants were randomized to receive a CDC “factsheet” (n = 21) or literacy-appropriate aid (n = 29) shown on a computer screen. Using 7 content similar slides, gaze patterns, mean pupil fixation time and mean overall time reading and looking at slides were compared. Groups were also compared by literacy level and effect on 'confidence of knowledge' and intended behavior.
Results revealed differing abilities to read densely written material. Intervention participants more precisely followed text on 4 of 7 content-similar slides compared to control participants whose gaze patterns indicated unread text, or repeated attempts at reading the same text, suggesting difficulty in understanding key preparedness messages. Controls had significantly longer pupil fixations on 5 of 7 slides and spent more overall time on every slide. In those with very low literacy, intervention participants were more likely than controls to say they understood what a “dirty bomb” is and how to respond if one should occur.
Results indicate limited- literacy adults, especially those with very low literacy, may not be able to understand how to respond during a “dirty bomb” using available materials, making them vulnerable to negative health events. This study provides insights into how individuals perceive and process risk communication messages, illustrating a rich and nuanced understanding of the qualitative experience of a limited literacy population with written materials. It also demonstrates the feasibility of using these methods on a wider scale to develop more effective health and risk communication messages designed to increase knowledge of and compliance with general health guidelines, and enhance decision making. This has application for those with learning disabilities, those with limited media-literacy skills, and those needing to access the diverse array of assistive technologies now available. Eye tracking is thus a practical approach to understanding these diverse needs to ensure the development of cogent and salient communication.
Eye tracking technology assesses how individuals view visual content by monitoring and recording eye movements as a subject processes the stimulus in real time. All systems use cameras, with some mounted to a flat stable surface and others embedded in a computer screen or mounted on lightweight glasses. Eye tracking is commonly used in marketing research , training systems involving hand-eye coordination (e.g., pilot flight training, surgery training) [2, 3], academic applications such as reading instruction or studies of cognition and information processing , and digital design work (e.g., websites, smart phones and tablet applications) [5, 6]. Eye movements, and the time spent on any one passage or image, indicate an individual’s interest or attention and provide an index of the input of information to more complex processing and/or reasoning [7, 8]. Eye tracking output can also clearly differentiate whether an individual is reading text as expected [9–12] or if a subject runs their eyes repeatedly over the same passages, suggesting difficulty in processing. The relationship between eye-tracking and comprehension has been studied since the early 1990’s, throughout which eye-tracking methodologies have made valuable contributions to our understanding of how to integrate information from texts, graphics and images to maximize comprehension. Current studies indicate brain structural correlates to text and sentence construction with comprehension can be validated with eye tracking methods . Eye tracking can also be used to pinpoint specific pieces of content that pose comprehension difficulties, and these measures can be related to other outcome measures, such as knowledge gained, decision making, and intended or actual behavior. As such, eye movement measures can provide valuable information for the design or refinement of health communication materials, particularly when these combine language and visual information processing [14, 15], to enhance decision-making.
While eye tracking has been used in a number of health-related studies [15–19] few have used it to understand the needs of low-literacy groups [9, 20], and none have used eye tracking to test a decision aid for readability and comprehension by intended users with low literacy, comparing text that is written in an accessible format to text that is typical of the type of health information available to the public on the Internet. The present study fills this gap by testing a decision aid designed to provide information about what to do in the event of a “dirty bomb” explosion in an urban area.
The importance of examining low-literacy populations’ perceptions of and responses to Radiological Terror Events (RTEs) such as a “dirty bomb” is evident. The 2003 National Assessment of Adult Literacy, the most recent national-level assessment of literacy, revealed that 14 % of adults in the United States have below basic literacy, 22 % have basic literacy, 53 % have intermediate literacy, while only 12 % are proficient . Overall, the United States ranks 17th of 24 developed nations in literacy ability . The National Center for Education Statistics (NCES) found that having only basic health literacy was greater in urban areas (16.3 %) compared to non-urban areas (13.1 %), with minorities and those over age 65 being most at risk of having literacy issues . This prevalence has significant implications for communicating threats about terror, especially terror events that may be difficult to conceptualize, such as an explosion that includes radiation exposure.
The needs of low-literacy populations in the event of a terror or bioterror event are not well understood. This is especially critical because as we move farther away from the September 11, 2001 attacks, a decrease in resources directed toward emergency preparedness has occurred. RTEs, such as the explosion of a “dirty bomb”, present a threat that requires national, state, and local preparedness , but little research has been conducted on the public’s likely response. Studies that have been done indicate that potential threats from nuclear technologies and radiation contamination are increasing exponentially [25, 26], while at the same time these types of terror attacks are not well understood, and thus trigger exceptional feelings of helplessness, confusion, fear, and distress [27, 28].
An added area of concern is the public's trust of government authorities in times of crisis. While “sheltering in place” in an intact structure is the recommended way to stay safe in the event of a “dirty bomb” explosion, research shows that barriers to complying with government directives include distrust and lack of confidence in government or public health authorities, fear of subsequent attacks, and beliefs that needed supplies or help will not be made available [29–31]. These are serious factors that are likely to impede the acceptance of authority directives and decrease informed decision making. Several studies have noted a distrust of the government, particularly among minority populations and those with low education levels, due to the fear of discrimination or unequal treatment during such emergencies [30–32]. Most of these studies, however, did not focus on the unique challenges of those with low literacy, who may not only be less trusting of information, but also not able to access or understand information if presented. In a study that specifically examined how people would behave during RTEs, only 60 % of the participants reported that they would comply with instructions given by emergency preparedness planners . In addition, little is currently known about the perceptions and understanding of low-literacy individuals concerning dirty bombs. A study conducted by Wray et al. found that participants preferred messaging that provided clear and accurate information and concrete action steps that were simple and consistently presented; since individuals were often confused by terms and messages utilized in current governmental communication efforts (e.g., “shelter in place”) . Additionally, non-English speakers had difficulty understanding messages and feared they would miss critical information . Currently, however, no studies have developed or tested literacy appropriate health communication materials or interventions for urban adults with limited literacy, but there is evidence from other domains, such as healthcare, that designing materials for literacy has effects not only on understanding of materials but on behavior as well [34–36].
The present study sought to address this gap by using eye tracking to assess how a systematically-developed, literacy appropriate decision aid, designed to foster understanding about how to respond to a “dirty bomb” event, addressed the needs of adults with limited literacy compared to a higher-literacy sample. The content of the decision aid was based on “Frequently Asked Questions” information developed by the Centers for Disease Control and Prevention (CDC), one of the few materials on RTEs publically available. Results showed clear patterns of differential text use in a small randomized pilot and demonstrated that eye tracking is a viable and useful tool in the quest to design materials for low-literacy groups to enhance decision-making.
This study included a multi-phase protocol. Phase I included qualitative focus groups and quantitative surveys. Results of this phase were used in Phase II to develop a low-literacy risk communication decision aid about “dirty bomb” RTEs. Phase III involved testing the developed decision aid in a pilot randomized controlled trial. This paper discusses the results of Phase III, and focuses on the eye-tracking results.
Phase I included both qualitative and quantitative data collection. This involved three focus groups with thirty-seven urban 18 to 65-year-olds, representing both men and women recruited from areas with low income and education levels in Philadelphia, PA . From existing literature on perceptions of radiation and radiological terror, a moderator’s guide was developed and tailored for the target sample . These questions helped to identify the range of cognitive and affective issues thought to be important for understanding perceptions of dirty bombs, including: (1) perceptions of emergency situations, (2) knowledge of a dirty bomb, (3) reaction to dirty bomb scenarios, and (4) comprehension of information provided. Results were analyzed using the Krueger method of analyzing narrative text, including familiarization, identifying a thematic framework, indexing, charting, mapping, and interpretation [39, 40].
Focus group results were then used to develop a comprehensive survey administered to 50 low-literacy adults, which assessed perception and understanding of a “dirty bomb,” trust of information sources, and intention to comply with government directives about “sheltering in place.” Implemented as a cross-sectional intercept survey, recruitment of respondents occurred in a variety of community-based sites in North Philadelphia, including local hospitals, neighborhood youth and senior community centers, and supermarkets. Four groupings of conceptually-related statements were included in the survey, including: 1. Statements on knowledge of a dirty bomb, trust of information sources if a dirty bomb occurred, and need for information about who was responsible for the event; 2. Statements on action intentions (staying home vs. leaving; getting children/family), belief in general preparedness, and preparedness activities such as having an emergency plan, food, and water; 3. Statements on what respondents would be worried about in the event of a dirty bomb, such as safety of food and water, risk of illness, breathing radiation, and its effects and risks to children and pets; and, 4. Statements that assessed participants’ perspectives of how likely a dirty bomb is to occur; likelihood of this event occurring compared to other threats (such as violence, flooding, or a car accident); beliefs about the trustworthiness of local, state, and federal authorities; and, whether their neighborhood would be treated fairly in response/recovery efforts.
Phase II consisted of analyzing survey results using a variety of methods, including perceptual mapping and message vector modeling, and a cluster analysis to identify potential differences within the sample group that might need to be addressed in the design of the resulting decision aid . Perceptual Mapping uses multidimensional scaling analysis to produce a three-dimensional graphic display of how participants perceive the relationships among the set of elements, by modeling the similarities and dissimilarities as distances between points in a multidimensional space. The resulting maps display the risk/benefit elements relative to each other and to “Self”, which can be an individual or group average. Perceptual mapping thus provides a graphical representation of how respondents conceptualize the decision or situation being evaluated. These methods are used extensively in marketing and advertising, and have been used to evaluate a number of public health decisions by the authors [42–45]. Message vector modeling can then be used to identify optimum message concept combinations for changing the positioning of elements in the mapping display (i.e. changing the perceptions or attitudes of the target segment). After the perceptual map is produced to display the relationships among concepts and between the concepts and “self”, vector modeling is used to determine which concept or concepts should be emphasized in a message or intervention to “move” the group toward the desired behavior, in this case to “shelter in place” in the event of a “dirty bomb”. This desired behavior establishes the target vector. (For more information on perceptual mapping and message vector modeling techniques, see http://sites.temple.edu/turiskcommlab/).
Based on these results, we developed a literacy-appropriate decision aid, optimizing the fit between the subjects’ perceptual maps and the content and format of literacy-appropriate information about a “dirty bomb”. The resulting decision aid included overall information about what a “dirty bomb” is and how to respond should an explosion occur, as well as specific message concepts determined to be important through the perceptual mapping. These concepts included: (1) Emphasizing trust in the information and recommended responses of authorities/experts, (2) Complying with recommended actions, and (3) Getting prepared for an emergency before one occurs. We also emphasized that in such a crisis, it is less important to dwell on who was responsible for the attack (a common need felt by victims of attacks that can impede taking immediate action) than it is to determine how to immediately protect yourself and your family. As well, the decision aid presented straight-forward information about what a dirty bomb is and how to respond if one occurred.
The control material was the CDC’s printed document, “Frequently Asked Questions” for a “dirty bomb”, which is available to the public on the CDC website (http://emergency.cdc.gov/radiation/dirtybombs.asp) and presented in a question/answer format. The reading level of this material, assessed using the Flesch-Kincaid test , was at an 8.2 grade level. Since the text on CDC’s website involved scrolling down a page and we wanted to be able to compare how participants read unique text and compare it across groups, both control and intervention materials were presented through a PowerPoint slide format, with each “question and answer” on a different slide. To focus on literacy relevant variables, the layout, graphics, and basic format of the low-literacy decision aid was made consistent, but text was written at a 5th grade reading level and utilized graphics to represent words. The CDC FAQs decision aid was 19 slides, compared to the intervention (low-literacy) decision aid, which was 35 slides. Many slides were content similar, but the intervention decision aid was longer to account for special message content derived from the perceptual mapping but not addressed in the CDC's FAQs. Figure 1 illustrates differences between the Control decision aid (CDC's content) and the Intervention decision aid (designed for low-literacy individuals) on three content-similar slides.
During Phase III of the study, a randomized pilot-test comparing the developed literacy appropriate “dirty bomb” decision aid (intervention) and the existing CDC “dirty bomb” FAQs (control) was conducted with a convenience sample of 50 adults (29 in the intervention group, 21 in the control group) in the Risk Communication Laboratory at Temple University in Philadelphia. The testing occurred over six months, between January and June, 2012. We conducted this pilot study to assess the feasibility of conducting a fully powered randomized trial on whether a systematically developed, literacy appropriate decision aid would help urban residents understand the risks of a “dirty bomb” and how to respond if one occurred, as well as to determine if eye tracking was a useful and valid method of determining how those with low-literacy can or cannot access text.
Participants were recruited for the pilot through community-based methods, including placing flyers in the Temple University Hospital’s General Internal Medicine Clinic, community organizations surrounding Temple University, as well as local pharmacies and supermarkets. Flyers included a “tear tab” with a phone number so those interested could call about participating in the study. We also focused on users of a neighborhood food bank, a senior services organization, and a federal services office. In these cases, research assistants visited the sites, talked with people about the study and screened them for eligibility. The goal was to have participants who closely represented people who matched the socio-demographic groups represented in the focus group and survey analysis.
Screening included demographic information (age, gender, ethnicity and education level) and health literacy level. In-person literacy screening utilized the Rapid Estimate of Adult Literacy in Medicine – Revised (REALM-R) test, which is a word recognition test in which participants are asked to pronounce eleven medical words of increasing difficulty . The first three words are not scored, giving the test a scored range of 0–8. Participants unable to pronounce the first three words have very low literacy skills and those unable to pronounce more than six are at risk of having inadequate literacy. In this study, those scoring a 6 or below were eligible. Over the phone literacy screening utilized the Single Item Literacy Screening tool (SILS)  and participants qualified if they indicated they always, often or sometimes “need help reading instructions, pamphlets or other written material from a doctor or pharmacy”. Other eligibility criteria included being 18 or over, being able to read and speak English, and the absence of dyslexia or other learning problems that interfere with reading/tracking text. If eligible, appointments to come into the Risk Communication Laboratory were made and participants were randomized to the control and intervention groups using a random numbers table. Participants were provided incentives to take part in the study, including a $20 gift card and two transit tokens. The university's Institutional Review Board approved all procedures and consent materials.
When participants arrived at the Risk Communication Laboratory a trained research assistant orally administered all consent forms and instruments, which were tailored for people with low literacy. The eye tracking data collection procedures were explained to participants using pictures of the equipment and samples. Participants were then escorted to the testing lab, which is a small private room, and they were connected to the eye tracking system in preparation for viewing their assigned decision aid. The input from the physiological monitors was fed directly into a computer database for storage and analysis. Eye tracking was performed using an Applied Science Laboratory stationary eye tracker (Eye Trac 6000). This equipment requires a participant to sit in front of a screen with a Velcro band around his/her head, which automatically corrects for any head movement during eye tracking data collection. The control and intervention messages were presented on a computer monitor for each participant to view. Once the system acquired a pupil lock and calibrated the subject’s eye tracking, he/she was told to look at the screen and read the slide presented, and then indicate to the technician when he/she was done. The technician then advanced the slide for the participant until he/she was done looking at all slides. Once the participant was finished, he/she was directed back to another room where the post-education survey protocol was conducted. The full study took approximately 45 min to administer. Results of eye tracking analyses are discussed here.
The two groups (Intervention v. Control) are profiled in Table 1 on demographic factors. In addition, all participants were English speaking, with none reporting English as a second language. While all participants had low literacy, we also dichotomized this variable to assess whether differences were seen in those with very low literacy (defined as those who scored a 0–2 on the REALM-R or said they “always” or “frequently” have trouble filling out medical forms on the SILS), compared to those with low literacy (defined as those who scored a 3–6 on the REALM-R or said they “sometimes” have trouble filling out medical forms of the SILS). Demographic characteristics of these groups are provided in Table 2. For continuous variables (Tables 3, 4 and 5), means, sample sizes and standard deviations are provided. Using SPSS Version 23, T-Tests were computed to examine differences between groups as well as sub-groups by literacy level. Four groups were used: 1. Intervention, Very Low Literacy; 2. Intervention, Low literacy; 3. Control, Very Low Literacy; and, 4. Control, Low Literacy. Group differences were examined on mean qualitative ratings of ability to track text, average pupil fixation, and average time to complete reading a slide, a composite average of these ratings, as well as outcome measures of confidence in knowledge of what a “dirty bomb” is and intended behavior in the likelihood of a “dirty bomb” exploding.
To extract pupil fixation and average time to complete a slide, as well as extracting the gaze patterns, the Eyenal software program (designed for analysis of eye-tracking data) was used (Applied Science Labs). In addition, a 5-point subjective rating scale was developed and tested by the authors to rate the consistency of a participant's gaze patterns, including the ease with which the participant appeared to follow the text. To establish inter-coder reliability in the use of this scale, three study coders first came to agreement about what each scale point meant. The five-point tracking consistency scale was defined as follows:
=“Very chaotic tracking with very limited ability to follow text”,
= "Mostly chaotic tracking and quite limited ability to follow text,"
= “Mixed tracking, where tracking of some areas of the text/slide were precise, and the rest chaotic and inconsistent,"
= "Mostly precise tracking and relatively consistent following of text,"
= "Very precise tracking and highly consistent following of text."
Coder training involved practicing use of the scale to rate eye-tracking consistency until a high level of inter-coder reliability was achieved (Pearson r = .898, Spearman R = .989). After training, coders independently rated each participant's eye-tracking patterns on each slide. In the few cases where rater discrepancies occurred, the coders reviewed the gaze pattern in question together, and discussed their ratings to arrive at a consensus scale value.
Eye-tracking comparisons were only performed on seven content-similar slides to control for content. These seven slides included similar questions and responses and allowed us to compare eye tracking patterns across similar content. The slides included:
What is a dirty bomb?
Dirty bomb is not the same as an atomic bomb.
What to do if you are inside and close to the explosion?
What to do if you are outside and close to the explosion?
What should I do about my kids and family?
Will my food and water be safe?
How do I know if I’ve been exposed to radiation?
As illustrated in Table 1, there were 29 participants in the intervention group and 21 in the control group. No significant differences between the groups were found on demographic variables and literacy levels, indicating randomization was successful. Sub-analysis of the group by literacy level (Table 2) showed similar characteristics in those who had very low literacy compared with those with low literacy. No significant differences by literacy group were found. For all groups, the dominant age category was 36 to 59 and they were primarily African American. The majority also reported graduating from high school. Literacy statistics indicated a very low level of literacy. For the 31 participants who did the REALM-R screening, the mean score was 2.58, with a range of 0 to 5 (SD 1.42) indicating very low literacy. Of the 19 participants screened over the phone with the SILS, the mean score was a 3.37 (range 3 to 5, SD = .684).
Table 3 summarizes independent- samples t-Tests, comparing the intervention and control groups across the seven content similar slides on gaze pattern and consistency of eye tracking. The intervention group was rated significantly higher in ability track on four of the seven slides, including the “A dirty bomb is not an atomic bomb” (M = 3.65, SD = 1.49; t(29) = 2.26, p = .032), “what to do if you are outside and close to the explosion” (M = 3.81, SD = 1.17; t(26) = 2.48, p = .02), “what to do if you are inside and close to the explosion” (M = 3.88, SD = .96; t(27) = 3.61, p = .001), and “what should I do about my children and family” (M = 3.53, SD = 1.06; t(26) = 2.70, p = .012) slides. Intervention participants had higher means on all but one of the slides (“What is a dirty bomb”), which in both groups was very simple and contained only one statement describing what a dirty bomb is.
Figure 2 shows summary slides of selected participant eye tracking to illustrate the differences evident in tracking patterns; each represents a different participant. The blue line indicates the actual pattern of tracking and the green dots indicate pupil fixation (the larger the dot, the longer the participant spent looking at that location). These patterns illustrate the differences observed between participants presented with higher vs. lower literacy information. For example, in Slide 1 – Control, the participant looked all over the page but did not have a specific gaze pattern that indicated he/she was actually reading the text. Slide 2 – Intervention, however, shows a clear gaze pattern indicating the participant was following the text as would be expected if he/she was reading the text. It is also clear that he/she looked at the photograph on the slide after reading the text. Similarly, in slides 3 and 5 – control, it appears that the participants made an attempt to read the title and the first sentence of text but then did not finish, not looking at the rest of the text. Slides 4 and 6 – intervention, however, clearly show the participants not only followed the text, but also spent time looking at the photographs. (Additional files 1 and 2: Videos to Fig. 2 are provided, which show real-time eye tracking for selected intervention and control participants.)
Mean number of pupil fixations on the seven selected slides indicated that the control group spent significantly longer looking at specific words or phrases on three of the slides (Table 4). These include “A dirty bomb is not an atomic bomb” (M = 38.14, SD = 30.05; t(43) = 3.28, p = .002), “What to do if you’re inside” (M = 47.14, SD = 38.22; t(43) = 2.03, p = .048), “How do I know about radiation exposure” (M = 54.20, SD = 43.89; t(42) = 2.46, p = .02). A fourth slide, “What should I do about my children and family” approaches significance, with the control group having more mean fixations (M = 50, SD = 36.46; t(42) = 1.86, p = .069). This is indicative of the more densely worded material in the control slides and their low-literacy level. In some cases, control participants had twice the number of pupil fixations compared to the intervention participants, most likely because of the density of the text. This correlates with the observations of real time tracking, where control participants can be seen repeating lines of text or skipping back and forth on the slide.
Table 5 illustrates the mean time it took participants to read/look at each slide. Participants were allowed to take as much time as they wanted on each slide and were told to tell the technician when they were done and were ready to move to the next. As illustrated, control participants took significantly longer to read all seven compared slides, sometimes taking twice as long as intervention participants. Even on the simple “What is a dirty bomb” slide, control participants spent on average 21.94 s (SD = 17.32) looking at the slide compared to just 12.66 s (SD = 4.53) spent by intervention participants (t(43) = 2.54, p = .015). Similar patterns are seen in the other slides: “An dirty bomb is not an atomic bomb” (M = 32.35 s, SD = 16.81; t(43) = 5.79, p = .000), “What to do if you’re outside” (M = 27.29 s, SD = 17.16; t(43) = 2.46, p = .018), “What to do if you’re inside” (M = 31.61 s, SD = 24.02; t(43) = 2.91, p = .006), “What should I do about my children and family” (M = 35.72 s, SD = 24.06; t(42) = 3.51, p = .001), “What to do about food and water” (M = 34.85 s, SD = 14.30; t(42) = 5.02, p = .000), and “How do I know about radiation exposure” (M = 31.57 s, SD = 32.05; t(41) = 2.84, p = .007). This correlates with both the eye tracking ratings and the pupil fixation analysis and strengthens the argument that the control participants had a difficult time reading and processing the text-heavy control slides.
In addition to looking at individual slides, composite scores for overall ability to track slides, fixations and time were calculated and then correlated to the dichotomized literacy variable. Pearson Correlations indicates that literacy level was associated with four of the 7 slides on the eye tracking rating (Dirty Bomb is not an Atomic Bomb, .404, p = .024; What do with family, .409, p = .031; What to do about food/water, .473, p = .006; Radiation Exposure, .406, p = .024). In addition, comparing average time, fixation and tracking ratings by intervention group, the Intervention, Very Low Literacy group had lower time and fixation composite scores (time = 32.08 s vs. 40.20 for Control, Very Low Literacy; fixation = 39.97 vs.49.37) and higher tracking scores (3.32 vs. 2.79). These differences, however, were not significant.
Finally, Independent Samples T-Tests were run comparing effects of the intervention on those with the lowest literacy levels on outcome measures, specifically confidence in knowledge of what a “dirty bomb” is, and intended behavior (i.e. sheltering in place) if one occurred (participants indicated how much they agreed or disagreed with a statement on a 0-10 scale). Table 6 illustrates that significant differences were seen in three post-intervention knowledge or behavior survey questions: Know what a dirty bomb is (M = 8.76, SD = 2.64; t(28) = 2.78, p = .01); Know what to do if a bomb blows up (M = 8.95, SD = 1.50; t(28) = 2.49, p = .02); My ability to follow instructions if a “dirty bomb” happens (M = 8.80, SD = .62; t(27) = 2.87, p = .008). These differences were not seen in the Low Literacy by intervention groups.
Monitoring of eye tracking in real time and watching video of each participant’s session clearly revealed that in this pilot study the control group participants had a difficult time getting through the dense text. For many of the low-literacy participants, it was obvious that they either chose to randomly glance around the slide text and not systematically work through it, or they began to work through the text, often re-reading given lines several times before giving up and resorting to random looking patterns. In some instances, it seemed that control participants would look off the slide for a period of time before deciding enough time had passed to tell the technician to advance to the next slide. Observing these patterns could be difficult to watch, as the apparent frustration many participants had with the text was evident. These observations were supported by the data, which indicated significant differences in not only the gaze patterns tracked across time, but was also in the measures of time spent at each fixation point, and the total number of fixation points per slide. These differences also remained once tracking fixations and time were combined as a composite score and compared, and when the sample was compared by their literacy level and intervention group. It was clear that the dense text provided in the control condition caused participants to spend more time with trying to get through the text and attempting to read lines more than once. Despite the small sample size of this study, the differences we found in tracking and outcome measures (confidence in knowledge and ability to follow instructions) indicate a large effect size (cohen’s d = .94). However, future studies using eye tracking measures will have to take into account the intent of the trial (eye tracking as outcomes vs. changes in knowledge and behavior) and whether larger sample sizes are required to test the effect of these types of materials are larger audiences.
Clearly, having access to literacy appropriate materials is important. In this study, significant effects on confidence in knowing what a dirty bomb was, how to respond if one occurred and ability to carry out instructions was seen in those with the very lowest literacy. This is a significant finding, indicating that even those with very low literacy can benefit from having information presented in a manner that addresses their needs. Because of the serious consequences of exposure to a “dirty bomb’, it is important that disaster planning and management teams address the accessibility of risk communication materials designed for limited-literacy groups, so that everyone can have equal access to potentially lifesaving information and make informed decisions for themselves and their families. Health literacy studies suggest that a third-to-a-half of the US population has some difficulty using written materials , such as those available on the CDC website -- materials that have been measured at an 8th grade reading level or higher [50, 51]. If large numbers of US adults cannot use complex prose, it is important public health communication specialists develop materials that are accessible and presented in a developmentally appropriate form. If these materials are not available, people with limited-literacy will be unable to adequately prepare for or respond to an emergency such as a “dirty bomb”, thus decreasing their ability to make informed decisions.
Public trust and confidence can help mitigate and improve the ability to manage a terror threat , but this is severely compromised if a large group is unable to access or process emergency communications. Risk communication is essential to containing the public’s fear and fatalism, and for ensuring public cooperation during crisis events . However, risk communication is especially challenging when applying the messages to high risk, high outrage situations  such as a “dirty bomb” due to their capacity to produce widespread fear, which reduces complexity of information processing and inhibits memory abilities . These limitations are further accentuated in low literacy populations, as distrust may be exacerbated by use of complicated terminology that assumes the public has more information and understanding than they do and making them most at risk for negative outcomes.
On a conceptual level, the goal of effectively communicating a health/risk message to a particular target audience must take into account not only their literacy skills, but also their level of education and experience with the topic being addressed. As such, the design of effective messages strategies requires attention to the task of informing and educating the receiver of the message, and the task of creating an information processing experience that effectively and efficiently optimizes the formatting, graphics, and visual/auditory aspects of message structure. These latter structural elements are critical to optimizing overall processing, learning, motivation, decision making, and behavior. Eye tracking analyses are particularly well suited to understand both the content/educational aspects of message design and the structural information processing aspects, and to understand how difficult it may be to access material that is not developed for those with low literacy. In this pilot study it was clear that low-literacy participants were more comfortable with the decision aid written at their literacy level and were more receptive to its messages; this is especially true in those with the lowest literacy. Eye tracking was thus a viable and useful method for understanding how low literacy adults may respond to and understand emergency communications, informing the development of better, more accessible materials.
Limitations and future research
The limitations of this research include the fact that as a pilot study the sample size is, by design, small. Although this is the case, care was taken to accurately assess the participant's literacy level and to randomly assign participants to the control and intervention test-groups. Future studies should expand the sample size and also expand the geographic scope of the study to explore urban v. suburban differences as well as potential regional differences across the country.
There are also other types of eye tracking data that could have been used in the analysis of this study, including saccades and smooth pursuits. These could be useful in larger studies to assess ease of tracking and reading material, especially when video or other interactive activities are incorporated into learning materials. These were not used here. In addition, the subjective eye tracking rating, despite rigorous training of raters and good reliability, is still a source of bias. Perhaps in a larger study this could be compared through a more objective mathematical evaluation of areas of interest and/or machine learning techniques. These measures were not available for this study. In addition, screening participants for visual acuity and vision quality would be advised.
Another limitation involves the current focus on the topic of radiological terror events (Dirty Bomb). Individual responses, in terms of information processing, physiological reactions, and knowledge gain will likely vary with different types of content (e.g., what to do in natural v. man-made disasters; active-shooter v. kidnapping/ransom events; bombs at a public event v. bombing a building, etc.). Public Health officials will have to develop "preparedness" materials for these types of differences as the frequency of such events continues to increase. This may also affect eye tracking results, as information more familiar to an audience may hinder the ability to differentiate as easily between tested materials.
A third important limitation of the present study is that it focuses on a particular aspect (literacy) of the audience's ability to access, interpret, and apply critical information about what to do in a particular emergency situation. The eye tracking and message design methodology used in this research can be directly applied to areas other than audience literacy levels. For example, it can be used to design more effective materials for individuals with learning disabilities, or those with limited media-literacy skills, or those needing to access the diverse array of assistive technologies now available.
The use of eye tracking is an effective approach to understand how health communication materials are accessed and processed by individuals with limited-literacy skills. We believe that this pilot work clearly demonstrates the feasibility of using these methods on a wider scale to develop more effective health and risk communication messages designed to increase knowledge of and compliance with general health guidelines on the one hand and specific health-based emergency directives on the other. It is also clear that the message design and testing approach has application not only for those with low-literacy, but for those with learning disabilities, those with limited media-literacy skills, and those needing to access the diverse array of assistive technologies now available. The long-term goal is to evolve and refine powerful message design tools that will contribute to effective decision making and the elimination of health disparities evident in educational, social, and personal contexts.
CDC, Centers for Disease Control and Prevention; REALM-R, Rapid Estimate of Adult Literacy in Medicine-Revised; RTE, Radiological Terror Event; SILS, Single Item Literacy Screener
Shi SW, Wedel M, Pieters FGM. Information acquisition during online decision making: A model-based exploration using eye-tracking data. Manage Sci. 2013;59:1009–26.
Sarter NB, Mumaw RJ, Wickens CD. Pilots’ monitoring strategies and performance on automated flight decks: An empirical study combining behavioral and eye-tracking data. J Hum Fac Erg Soc. 2007;49:347–57.
Chetwood AS, Kwok K, Sun L, Mylonas GP, Clark J, Darzi A, Yang G. Collaborative eye tracking: A potential training tool in laparoscopic surgery. Surg Endosc. 2012;26:2003–9.
Mason L, Pluchino P, Tornatora MC. Using eye-tracking technology as an indirect instruction tool to improve text and picture processing and learning. Brit J Educ Technol. 2015. doi:10.1111/bjet.12271.
Wang Q, Yang S, Liu M, Cao Z, Ma Q. An eye-tracking study of website complexity from cognitive load perspective. Decis Support Syst. 2014;62:1–10.
Molina AI, Redondo MA, Lacave C, Ortega M. Assessing the effectiveness of new devices for accessing learning materials: An empirical analysis based on eye tracking and learner subjective perception. Comput Hum Behav. 2014;31:475–90.
Beatty J. Pupillometric signs of selective attention in man. In: Galbraith GC, Kietzman ML, Donchin E, editors. Neurophysiology and psychophysiology: Experimental and clinical applications. Hillsdaly: Lawrence Erlbaum Associates, Inc.; 1988. p. 138–43.
Steinhauer S. Pupillary Responses, Cognitive Psychophysiology and Psychopathology. 2006. http://www.wpic.pitt.edu/research/biometrics/Publications/PupilWeb.htm. Accessed 4 Feb 2016.
Mackert M, Champlin SE, Pasch KE, Weiss BD. Understanding health literacy measurement through eye tracking. J Health Commun. 2013;18:185–96.
Miller B, O’Donnell C. Opening a window into reading development: eye movements’ role within a broader literacy research framework. School Psychol Rev. 2013;42:123–39.
Sereno SC, Rayner K. Measuring word recognition in reading: eye movements and event-related potentials. Trends Cogn Sci. 2003;7:489–93.
Smith AC, Monaghan P, Huettig F. Literacy effects on language and vision: emergent effects from an amodal shared resources (ASR) computational model. Cogn Psychol. 2014;75:28–54.
Fengler A, Meyer L, Friederici AD. Brain structural correlates of complex sentence comprehension in children. Dev Cog Neuro. 2015;15:48–57.
Krugman DM, Fox RJ, Fletcher JE, Fischer PM, Rojas TH. Do adolescents attend to warnings in cigarette advertising? An eye-tracking approach. J Advertising Res. 1994;34:39–52.
Thomsen SR, Fulton K. Adolescents’ attention to responsibility messages in magazine alcohol advertisements: an eye-tracking approach. J Adolesc Health. 2007;41:27–34.
Turner MM, Skubisz C, Pandya SP, Silverman M, Austin L. Predicting visual attention to nutrition information on food products: the influence of motivation and ability. J Health Commun. 2014;19:1017–29.
Berenbaum E, Latimer-Cheung AE. Examining the link between framed physical activity ads and behavior among women. J Sport Exercise Psy. 2014;36:271–80.
Bialkova S, Grunert KG, Juhl HJ, Wasowicz-Kirylo G, Stysko-Kunkowska M, van Trijp HC. Attention mediates the effect of nutrition label information on consumers’ choice. Evidence from a choice experiment involving eye-tracking. Appetite. 2014;76:66–75.
Brown SL, Richardson M. The effect of distressing imagery on attention to and persuasiveness of an anti-alcohol message: a gaze-tracking approach. Health Educ Behav. 2012;39:8–17.
Summers K, Langford J, Wu J, Abela C, Souza R. Designing web-based forms for users with lower literacy skills. P Am Soc Inform Sci Tech. 2007;43:1–12.
Rudd R. Health literacy skills of U.S. Adults. Am J Health Behav. 2007;31:S8–S18.
Goodman M, Finnegan R, Mohadjer L, Krenzke T, Hogan, J. Literacy, numeracy, and problem solving in technology-risk environments among U.S. adults: Results from the program for international assessment of adult competencies 2012: First Look (NCES 2014-008). 2013. Washington, DC.
Mohadjer L, Kalton G, Krenzke T, Liu B, de Kerckhove W, Li L, et al. National assessment of adult literacy: Indirect county and state estimates of the percentage of adults at the lowest literacy level for 1992 and 2003. Research and Development Report. NCES 2009-482. 2009. Washington, D.C.
Centers for Disease Control and Prevention. Strengthening CDC’s emergency response. 2009. http://www.cdc.gov/phpr/publications/2009/phprep_report_2009.pdf. Accessed 4 Feb 2016.
Slovic P. Perception of risk from radiation. In: Slovic P, editor. The perception of risk. London: Earthscan Publications Limited; 2001. p. 264–74.
Acton JM, Brooke Rogers M, Zimmerman PD. Beyond the Dirty Bomb: Re-thinking Radiological Terror. Survival: Global Politics and Strategy. 2007;49(3):151–68.
Rosa EA, Freudenberg WR. The historical development of public reactions in nuclear power: implications for nuclear waste policy. In: Dunlap RE, Kraft ME, Rosa EA, editors. Public reactions to nuclear waste: Citizen’s views of repository sitting. Durham: Duke University Press; 1993. p. 32–63.
Nutbeam D. Health literacy as a public health goal: a challenge for contemporary health education and communication strategies into the 21st century. Health Promot Int. 2006;15:259–67.
Meredith LS, Eisenman DP, Rhodes H, Ryan G, Long A. Trust influences response to public health messages during a bioterrorist event. J Health Commun. 2007;12:217–32.
Wray R, Rivers J, Whitworth A, Jupka K, Clements B. Public perceptions about trust in emergency risk communication: qualitative research findings. Int J Mass Emerg Disasters. 2006;24:45–75.
Wray RJ, Becker SM, Henderson N, Gilk D, Jupka K, Middleton S, et al. Communicating with the public about emerging health threats: lessons from the Pre-Event Message Development Project. Am J Public Health. 2008;98:2214–22.
Marist College Institute for Public Opinion. How Americans feel about terrorism and security two years after 9/11. Survey conducted on behalf of the National Center for Disaster Preparedness and the Children’s Health Fund. 2003. http://maristpoll.marist.edu/tag/terrorism/. Accessed 4 Feb 2016.
Lasker RD. Redefining readiness: Terrorism planning through the eyes of the public. New York: New York Academy of Medicine; 2004.
Pignone M, DeWalt D, Sheridan S, Berkman N, Lohr K. Interventions to improve health outcomes for patients with low literacy. J Gen Int Med. 2005;20:185–92.
Sheridan S, Halpern D, Viera A, Berkman N, Donahue K, Crotty K. Interventions for individuals with low health literacy: a systematic review. J Health Comm. 2011;16(Supp 3):30–54.
Taggart J, Williams A, Dennis S, Newall A, Shortus T, Zwar N, Denney-Wilson E, Harris M. A systematic review of interventions in primary care to improve health literacy for chronic disease behavioral risk factors. BMC Fam Pract. 2012; Doi:10.1186/1471-2296-13-49.
Bass SB, Greener JR, Ruggieri D, Parvanta C, Mora G, Wolak C, et al. Attitudes and perceptions of urban African Americans of a "dirty bomb" radiological terror event: Results of a qualitative study and implications for effective risk communication. Disaster Med Public Health Prep. 2015;9:9–18. doi:10.1017/dmp.2014.158.
Becker S. Emergency communication and information issues in terrorist events involving radioactive materials. Biosecur Bioterror. 2004;2(3):195–207.
Krueger RA. Focus Groups: A Practical Guide for Applied Research. CA: Sage Publications; 1994.
Rabiee F. Focus-group interview and data analysis. Proc Nutr Soc. 2004;63:655–60.
Bass SB, Mora G, Ruggieri D, Wolak C, Gordon TF, Rovito MJ, et al. Understanding of and willingness to comply with recommendations in the event of a ‘dirty bomb’: Demographic differences in low-literacy urban residents. Washington, DC: American Public Health Association; 2001.
Bass SB, Gordon TF, Ruzek SB, Hausman AJ. Mapping perceptions related to acceptance of smallpox vaccination by hospital emergency room personnel. Biosecur Bioterror. 2008;6(2):179–90.
Ruggieri DG, Bass SB, Rovito MJ, Ward S, Gordon TF, Paranjape A, et al. Perceived colonoscopy barriers and facilitators among urban African American patients and their medical residents. J Health Comm. 2013;18(4):372–90.
Bass SB, Gordon TF, Ruzek SB, Wolak C, Ruggieri D, Mora G, et al. Developing a computer touch-screen interactive colorectal screening decision aid for a low-literacy African American population: lessons learned. Health Promot Pract. 2013;14(4):589–98.
Bass SB, Wolak C, Greener J, Tedaldi E, Nanavati A, Ruppert K, et al. Using perceptual mapping methods to understand gender differences in perceived barriers and benefits of clinical research participation in urban minority HIV+ patients. AIDS Care. 2016;28(4):528–36.
Kincaid JP, Fishburne RP, Rogers RL, Chissom BS. Derivation of new readability formulas (Automated Readability Index, Fog Count, and Flesch Reading Ease Formula) for Navy enlisted personnel. Research Branch Report 8-75. Millington, TN. 1975.
Bass PF, Wilson JF, Griffith CH. A shortened instrument for literacy screening. J Gen Intern Med. 2003;18:1036–8.
Morris NS, MacLean CD, Chew LD, Littenberg B. The single item literacy screener: evaluation of a brief instrument to identify limited reading ability. BMC Fam Pract. 2006;7:21. doi:10.1186/1471-2296-7-2.
Kutner M, Greenberg E, Jin Y, Paulsen C. The health literacy of America’s adults: Results from the 2003 National Assessment of Adult Literacy (NCES 2006–483). U.S. Department of Education, National Center for Education Statistics: Washington, DC. 2006. http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2006483 2006. Accessed 1 Apr 2015.
Vanderford M, Nastoff T, Telfer J, Bonzo SE. Emergency communication challenges in response to Hurricane Katrina: Lessons from the Centers for Disease Control and Prevention. J Appl Commun Res. 2007;35:9–25.
Lagassé L, Rimal R, Smith K, Storey D, Rhoades E, Barnett D, et al. How accessible was information about H1N1flu? Literacy assessments of CDC guidance documents for different audiences. PLoS One. 2011;6:11. doi:10.1371.journal.pone.0023583
Eisenman DP, Wold C, Setodji C, Hickey S, Lee B, Stein B, Long A. Will public health’s response to terrorism be fair? Racial/ethnic variations in perceived fairness during a bioterrorist event. Biosecur Bioterror. 2004;2:146–56.
National Research Council. Improving risk communication. Washington, DC: National Academy Press. 1989.
Sandman PM. Hazard versus outrage in the public perception of risk. In: Covello VT, McCallum DB, Pavlova MT, editors. Effective Risk Communication: The Role and Responsibility of Government and Nongovernment Organizations. New York: Plenum Press; 1989. p. 45–9.
The authors would like to thank Kaitlin Kordusky, MPH, who contributed to the study by recruiting participants and collecting data.
This study was funded by the National Institute of Biomedical Imaging and Bioengineering, 5RO3EB9561-2 (Health Literacy mechanism): “Developing Radiological Risk Communication Materials for Low-Literacy Populations,” Bass, SB – Principal Investigator
Availability of data and materials
Educational materials are available at http://sites.temple.edu/turiskcommlab/. Those interested in data files can contact the lead author, Sarah Bauerle Bass, Ph.D., MPH at firstname.lastname@example.org
SBB was Principal Investigator for the study and conceived the study and all protocol. She also was primary author of the manuscript. TFG was a co-investigator of the study and contributed by analyzing eye tracking data and conceiving of the developed eye tracking scale reported in this manuscript. He also contributed to the writing of the discussion section of the manuscript. RG collected all eye tracking data and assisted with data analysis, including gaze pattern analysis. CP was a co-investigator of the study and contributed to the writing of the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
All study materials and protocols were approved by the Temple University Institutional Review Board, study number 12434.
a. Eye tracking video of Intervention condition – what about my kids and family. b. Eye tracking video of Control condition – what about my kids and family. (ZIP 32338 kb)
a. Eye tracking video of Intervention condition – what about my food. b. Eye tracking video of Control condition – what about my food. (ZIP 48292 kb)