Recruitment of participants
Participants were recruited and data collected between March and August 2013. Potential participants were identified at locations offering social services (e.g., soliciting door-to-door in a low-income housing community, social services offices, and community resource fairs) and locations not offering social services (e.g., University listservs, student/family housing, and flyers in local coffee shops). All individuals interested in participating completed a brief, online screening questionnaire that included basic demographic questions, including level of education and insurance status, as well as health care access questions [2]. A detailed description of the online screening questionnaire has been provided elsewhere [12].
Eligible participants were 21 to 35 years of age, had searched the Internet for health information within the past 12 months, and reported at least one barrier to health care services access, including inability to get an appointment in a timely manner, challenges with transportation to see a health care provider, no consistent primary care provider, or inability to pay for services [2]. Study participants received $20 compensation for their time.
Data collection
Participants were randomly assigned to one of two clinical symptom scenarios to prompt their Internet search: (1) fever, mild headache, dry cough, and myalgia (suggestive of influenza); or (2) fever, severe headache, and stiff neck (suggestive of meningitis). Symptom scenarios were developed based on Centers for Disease Control (CDC) guidelines and input from the clinical co-author (RLK) and study consultants. Both symptom scenarios were pilot-tested for face validity and understanding in a small sample of adults (n = 8) who fit inclusion criteria for study participation. Following randomization to one of the two scenarios, the lead author (SLP) asked participants, “What do you think you are experiencing?” – for judgment about etiology – and then instructed them to “Search the Internet, as though [you were] experiencing this situation.” The lead author also trained all participants how to narrate their processes of searching and decision-making. All study participants were instructed to “think aloud” while conducting their search for information related to the clinical scenario. At the end of their search, participants were again asked to identify their perceived symptom etiology (which was stated in terms of their response to the question “What do you think you are experiencing?”).
Participants had a choice of web browser to conduct their search: Firefox, Internet Explorer, or Google Chrome. Web browsers opened to a blank page. Internet searches and participants’ “think-aloud” vocalizations were digitally recorded using screen capture video-recording software [13], which also captured mouse clicks and keystrokes. Between each search session, web browser search history and cookies were deleted. Audio recordings were transcribed verbatim for content analysis.
Coding of internet search behaviors
Search-related mouse clicks and keystrokes making up the step-by-step process of Internet searching and decision-making processes were examined and coded as unique components of the search process. Each mouse click and combination of keystrokes (Internet searching) was coded as related to assessing one of the following “search pattern components”: cause of the symptoms (etiological assessment), defining the meaning of symptoms (symptom exploration), or characterizing a pattern of action based on the symptoms (treatment seeking). Etiological assessment describes testing a diagnostic hypothesis (i.e., entering the search term “meningitis” or clicking on a hyperlink titled “Flu”). Symptom exploration describes searching that involves using symptoms to guide the search (e.g., “achy, high temperatire [sic], sore muscles” or clicking on a link “cough, muscle pain”). Treatment seeking describes searching for remedies, recommended actions or alerts such as recommendations for seeking immediate care from a health care provider, looking for a cure, or searching for health care services (i.e., entering the search term “flu remedies” or selecting the link “when to seek Medical Care”). These coding patterns were grouped into broad thematic categories for analysis (characterizing search-related motivation and decision making) and analyzed for the number of times participants switched (switching) between search pattern components. Perez et al. (2015) provide examples and detailed explanation of this methodology.
Searches that consisted of entering at least two different search terms and then clicking or selecting at least one link beyond one search were labeled ‘branching.’ Searches that consisted of entering less than two searches and did not select a link beyond a search were labeled as ‘pruning.’
Analyses of participants and search strategies for understanding symptoms
Study participants who had no college degree and who were recruited from sites offering public services (e.g., employment, housing, welfare services, and food stamps) were identified as lower socioeconomic status (lower-SES); those with a Bachelor’s degree or any post-graduate educational experience, regardless of recruitment site, were classified as higher socioeconomic status (higher-SES). Education is a reflection of a range of noneconomic social characteristics, such as general and health-related knowledge, literacy, and problem-solving skills, with important health effects [14]. In addition to education, recruitment site was used as a proxy for resource utilization, access, and dependency.
Participants’ etiological assessments consisted of their “best guess at the diagnosis.” Assessments were coded as “correct,” if the “best guess” matched the diagnosis of the assigned scenario; “incorrect,” if the “best guess” did not match the assigned scenario; and “unsure,” if a study participant stated that they “didn’t know,” were “confused,” or “unsure” about what they were experiencing.
Next, we classified the differences between pre- and post-search etiological assessments. Etiological assessments that did not change as a result of the Internet search were coded as “no influence”; for etiological assessments where there was a change in decision from the initial decision (i.e., from correct to incorrect, incorrect to correct, unsure to either correct or incorrect, or correct/incorrect to unsure) were coded as “any influence.”
Finally, we examined the relationship between individual socioeconomic status (lower-SES versus higher-SES) and influence of the Internet search on decision (“no influence” vs. “any influence”) using Pearson’s χ
2 test of significance. Building on previous research that concluded that there are two types of Internet search behaviors for processing information – intuitive (i.e., unconscious, rapid, automatic, and high capacity thin) and deliberative (i.e., conscious, slow, and deliberative processing) [12], we examined the relationship between socioeconomic status (lower-SES versus higher-SES) and information processing (intuitive vs. deliberative), again using Pearson’s χ
2 test of significance.
All statistical analyses were performed using SAS(r) software version 9.3 (SAS Institute, Cary, NC).
Qualitative analysis of “think aloud” processes
“Think aloud” transcripts for the 78 study participants were systematically reviewed for important information searching components by two of the authors (SLP and DAP) using iterative content analysis. First, transcripts were examined for participant expression of perceived etiology. Next, “think aloud” data were assessed for the following: a) use of search terms, b) selection of websites, c) articulation of rationale for information selection or search strategy, d) perception of website credibility, e) mention of previous experiences and knowledge, f) attention to information formatting, including illustrations, g) articulation of frustration and confusion, and h) assessment of symptoms during search. Coded information was grouped into discrete categories, and coherent themes and patterns that described decision-making processes. Authors DAP and SLP then reviewed thematic content and patterns and agreed on a final categorization of themes that reflected Internet decision-making processes and search-related motivation.