Skip to main content

Readability, understandability and language accessibility of Swedish websites about the coronavirus disease 2019: a cross-sectional study

Abstract

Background

The COVID-19 pandemic has caused significant morbidity and mortality. To mitigate its spread, members in the general population were prompted to apply significant behavioral changes. This required an effective dissemination of understandable information accessible for people with a wide range of literacy backgrounds. The aim of this study was to investigate the readability, understandability and language accessibility of Swedish consumer-oriented websites containing information about COVID-19.

Methods

Websites were identified through systematic searches in Google.se (n = 76), and were collected in May 2020 when the pandemic spread started in Sweden. Readability and understandability were assessed with the Readability Index, the Ensuring Quality Information for Patients (EQIP) tool, and the Patient Education Materials Assessment Tool Understandability subscale (PEMAT-PU).

Results

The median total sample score for Readability Index was 42.0, with the majority of scores being classified as moderate (n = 30, 39%) or difficult (n = 43, 57%). Median total sample scores were for EQIP 54.0% (IQR = 17.0, Range = 8–75) and for PEMAT-PU 60.0% (IQR = 14.75, Range = 12–87). The majority of the websites did not have any texts or links containing information in an alternative language (n = 58, 76%).

Conclusions

Swedish websites contained information of difficult readability and understandability at the beginning of the coronavirus disease 2019 pandemic, with few providing information available in alternative languages. It is possible that these deficits contributed to the spread and impact of the virus. There is a need for studies investigating methods aiming to enhance the readability, understandability and language accessibility of web-based information at the beginning of an epidemic or pandemic.

Peer Review reports

Background

The Web is an enormous and highly popular resource for health-related information, highly utilized by a wide range of individuals within the general population [1, 2]. The Web has a great potential to empower consumers by offering high-quality information that is accessible and tailored according to their needs [3, 4]. It introduces an opportunity for people to access a great volume of health-related information when it is convenient for them, with the option to remain anonymous if they so desire [5]. When used correctly and appropriately, the Internet has the potential to improve health-related information uptake, enhance the patient-professional relationship, promote knowledge, and support decision-making processes [6]. All of these aspects are undoubtedly worth considering when implementing strategies to mitigate the spread of communicable diseases.

No universal mechanisms have been implemented to ensure high-quality websites providing information about health and wellbeing [3]. This illustrates a risk that online consumers encounter low-quality information that will fail to empower them towards informed decisions and healthy behaviors. For health-related information to be adequately understood and accurately interpreted, the recipient needs to have sufficient health literacy. Personal health literacy is defined as “the degree to which individuals have the ability to find, understand, and use information and services to inform health-related decisions and actions for themselves and others”, while organizational health literacy concerns the extent to which organizations enable these decisions and actions [7]. Low health literacy in the general population is an acknowledged challenge, with studies indicating high prevalence of low levels, and conversely, a low prevalence of proficient levels [8]. Repeatedly, low health literacy has been shown to be associated with increased hospitalization and emergency care use, and among older populations, worsened health status and higher mortality [9]. One way to address low health literacy is to disseminate content written in a plain, easy-to-read, clear, and accessible language [10].

In early 2020, the coronavirus COVID-19 caused a pandemic with health-related consequences across the globe. The pandemic caused significant morbidity and mortality in Europe, including Sweden [11]. As the disease spread, members of the general population were faced with unfamiliar challenges related to disease prevention in their daily life, which required health-related behavioral changes [12]. Thus, the public had a considerable demand for high-quality information about aspects such as disease prevention, symptoms and treatment [13]. Indeed, timely and adequate communication is essential in order to empower individuals to make autonomous and informed health-related decisions [14]. Considering the prevalent utilization of the Web as a resource for health-related information, some studies investigating the readability of such materials have been conducted. These empirical studies have utilized a range of various automated readability formulas [15,16,17,18,19,20,21], all concluding that web-based information about COVID-19, written in the English language, is difficult to read [15, 17, 21,22,23]. Previous results uniformly show that only a small minority of websites meet the recommended 6th grade readability level [15, 16, 18,19,20,21]. However, little is known about the readability, understandability, and accessibility of websites containing information in non-English languages. There are a great number of languages represented in Europe, calling attention to the considerable challenge of disseminating high-quality information so that it reaches the diverse population. An important finding in one previous study was the lack of readily translated information and limited utilization of graphic-based material about COVID-19 [17], raising questions about the accessibility of translated information in non-English speaking settings. A further problem with the previously conducted studies is the general reliance on automated readability formulas, which do not capture intrinsic design elements and various multi-dimensional aspects related to the presentation of the web-based information, often referred to in the literature as understandability [24].

Taken together, there is a need to systematically assess the quality of the information disseminated at the beginning of the pandemic in non-English speaking settings, utilizing both automated readability tests as well as assessments of understandability. The overall aim of this study was to add to the current knowledge about the quality of web-based information about COVID-19 by providing a systematic assessment of information written in the Swedish language. Specifically, our objectives were to investigate: (1) the readability of information in the Swedish language through an automated readability formula; (2) the understandability by systematically assessing aspects of language and presentation not captured with an automated formula; and (3) determine the accessibility of readily translated versions of original web-based information written in the Swedish language.

Methods

Design

This was a descriptive cross-sectional study utilizing quantitative variables including automated readability calculations and systematic assessments based on established instruments. This study is an in-depth analysis of a data set previously analyzed in regard to quality variables other than those analyzed herein [25]. This study is reported according to the STROBE checklist for cross-sectional studies (Additional file 1).

Data collection

In May 2020, a set of 17 searches were performed in the Swedish version of Google (Google.se), the most popular search engine in Sweden [26]. Based on previous research reporting search patterns in the general population, we used several search strings, screened the first 20 links in each hit list, and retrieved the information presented in the first web page of each link [27, 28]. The search strings were inspired by popular and rising COVID-19 related search terms in Google Trends. No quotation marks or other search engine operators were used. The search strings are presented in detail in Additional file 2. The terms involved common Swedish terms for COVID-19, the coronavirus, disease prevention, self-care, and symptoms. A total of 340 hits were screened for inclusion using the Web browser Google Chrome, set to incognito mode in order to limit the impact of previous searches. To be included, websites needed to contain text-based information about COVID-19 intended for the general population, be written in Swedish, and be accessible without any password or payment requirements. Websites providing information specifically developed for health professionals were excluded, based on the assumption that we aimed to investigate the quality of information intended for the general population.

In the initial screening, 97 websites were excluded because they were irrelevant (n = 74), not written in Swedish (n = 11), and were inaccessible (n = 12). Among the remaining websites, 35 were excluded because they were written for health professionals (n = 33) or did not contain any text-based content (n = 2). After correcting for duplicate hits (n = 132), 76 unique websites were included in the final sample. All included websites were captured with NCapture in May 2020, to save the content as it was published at the time of data collection.

Data analysis

The data were analyzed with an automated readability formula, two tools for systematic assessment of understandability, and by recording readily available translated versions of the included websites. The last author, a specialist intensive care nurse and midwife who is a researcher and associate professor, performed all assessments. Additional file 3 presents the dataset of readability and understandability scores for the included websites.

Readability

The readability of the text-based content in all included websites were calculated with Readability Index (Swedish: Läsbarhetsindex, LIX), an automated formula used to determine the readability of Swedish texts. Scores less than 25 indicate easy readability, while scores over 60 indicate difficult readability. A score over 40 signals that the readability is too difficult for an average person to fully comprehend [29]. The corresponding grade levels of LIX scores are: less than 28 represent elementary school (grades 1–5), 28–43 represent junior high school (grades 6–9), 44–55 represent senior high school (grades 10–12) and more than 55 represent college or university [30].

Understandability

Understandability was assessed with The Ensuring Quality Information for Patients (EQIP) tool and the understandability subscale in the Patient Education Materials Assessment Tool for printable materials (PEMAT-PU). Both instruments have been shown to have adequate validity and reliability [31,32,33]. EQIP assesses a set of quality criteria related to language, visual aids, tone, and design/layout. Thirteen questions are rated as yes (1), partially (0.5), no (0), or not applicable [32]. To determine the overall score, the sum of the EQIP ratings are divided with the total number of applicable items and multiplied with 100, generating a percentage score between 0 and 100%. Higher scores indicate better quality. An EQIP score above 75% is considered high quality, 51–75% is considered good quality with minor problems, and scores below 51% is considered serious or severe problems in quality [32].

PEMAT-PU assesses a set of quality criteria related to content, word choice and style, use of numbers, organization, layout and design, and use of visual aids. Seventeen questions are rated as agree (1), disagree (0), or not applicable [33]. To determine the overall score, the sum of the ratings in the PEMAT-PU are divided with the total number of applicable items and multiplied with 100, generating a percentage score between 0 and 100%. Higher scores indicate better quality. A PEMAT-PU score of 70% has been suggested as a cut-off value, indicating that websites scoring less than 71% would be considered poorly understandable [33].

Language accessibility

All websites were thoroughly read through repeatedly and all translated sections or links to translations in an alternative language found within were recorded. Links leading to automated translations utilizing external services such as Google Translate were not considered readily available translations.

Results

Website affiliations

Within the total sample of 76 unique included websites, they were affiliated with the government (n = 19, 25%), health care services (n = 17, 22%), newspapers (n = 17, 22%), information websites produced by independent companies with the sole purpose to provide web-based information (n = 9, 12%), pharmacies (n = 5, 7%), and nine websites (12%) were categorized as having other affiliation (humanitarian organizations, n = 2; universities, n = 2; insurance company, n = 1; medical products company, n = 1; online health food store, n = 1; patient organization, n = 1; wiki page, n = 1).

Readability

The median score for Readability Index (LIX) was 42.0 for the total sample (Table 1), with the majority of the readability scores being classified as moderate (n = 30, 39%) or difficult (n = 43, 57%), Table 2. The highest median LIX scores, illustrating the most difficult readability within the sample, were found for government-affiliated websites (Med = 44.0, IQR = 8.5). The lowest median LIX scores, illustrating the easiest readability within the sample, were found for websites affiliated with health care (Med = 39.0, IQR = 6.0). However, none of the included websites in the total sample were classified as having very easy readability, and only one website affiliated with news was classified as having easy readability. As depicted in Additional file 4, the lowest LIX scores, indicating the easiest readability within the sample, were found in websites affiliated with news (LIX = 29 and 33) and health care (LIX = 31). Conversely, the highest LIX scores, indicating most difficult readability within the sample, were found in websites affiliated with the government (LIX = 50, 53, and 54).

Table 1 Automated readability scores and assessment scores for included websites (n = 76)
Table 2 Readability levels for the included websites (n = 76)

Understandability

The median EQIP score was 54.0% (IQR = 17.0, Range = 8–75), approaching a score classified as serious or severe problems in quality (Table 1). In total, seven questions in EQIP had more than half of the included websites not or only partly adhering to the criteria (Fig. 1). While all median EQIP scores indicated quality deficits, sources affiliated with information websites (Med = 50.0, IQR = 16.0) and pharmacies (Med = 42.0, IQR = 12.0) had median scores below 51%, indicating serious or severe problems. As depicted in Additional file 4, the highest EQIP scores, indicating highest quality, were found in websites affiliated with news (75%), health care (75%), and the government (73%). Conversely, the lowest EQIP scores, indicating low quality, were found in websites affiliated with the government (8% and 29%) and an information website (19%).

Fig. 1
figure 1

Distributions of understandability assessment scores for the included websites (n = 76)

The median PEMAT-PU score was 60.0% (IQR = 14.75, Range = 12–87), indicating poorly understandable content (Table 1). In total, four of the questions in PEMAT-PU had more than half of the websites not adhering to the criteria (Fig. 1). Regardless of website affiliation, all median PEMAT-PU scores indicated poorly understandable information, with lowest median PEMAT-PU scores among sources affiliated with newspapers (Med = 56.0%, IQR = 10.0) and information websites (Med = 47.0%, IQR = 13.0). As depicted in Additional file 4, the highest PEMAT-PU scores, indicating the most understandable content, were found in websites affiliated with the government (87% and 80%) and a pharmacy (80%). Conversely, the lowest PEMAT-PU scores, indicating least understandable content, were found in websites affiliated with news (31%), the government (17%), and an information website (12%).

Language accessibility

The most common translated sections/links to information in an alternative language in the included websites were English (n = 12, 16%), sign language (n = 8, 11%), and Arabic (n = 6, 8%). The least common were Farsi (n = 1, 1%), Romani kelderash (n = 1, 1%), and Syrian (n = 1, 1%), Table 3. The median number of websites with links to information in alternative languages was 0 (Range 0–30), with the majority of the websites not containing any link to information in an alternative language other than Swedish (n = 58, 76%). Eleven (14%) websites contained links to one alternative language, two (3%) contained links to two alternative languages, one (1%) contained links to three alternative languages, and four (5%) contained links to more than six alternative languages.

Table 3 Websites with links to information in other languages

Discussion

The aim of this study was to investigate the readability, understandability and language accessibility of consumer-oriented websites about COVID-19 written in the Swedish language, revealing moderate to difficult readability levels, poor understandability, and limited language accessibility. The COVID-19 pandemic has illustrated the importance of strategies implemented to ensure the dissemination of high-quality information about disease prevention. To the extent of our knowledge, this is the first study to investigate the readability of Swedish consumer-oriented websites about COVID-19 at the beginning of the pandemic. The findings resonate what has been reported regarding difficult readability determined with automated readability calculations for websites written in the English language [15, 17, 21,22,23]. Established recommendations state that web-based sources should be written on a sixth-grade reading level [34], and our findings as well as other studies investigating readability in other languages confirm that the readability of web-based information about COVID-19 far exceeds this goal [15, 17, 21,22,23]. The median LIX score was 42, corresponding to approximately 9th to 10th grade readability level. Similar grade levels (ranging from 8.7 to 14.3) have been reported in previous studies investigating websites written in the English language, confirming our findings and giving further weight to the problematic situation on the Web [15, 17, 18, 20, 21]. In our study, difficult readability was found regardless of website affiliation, including websites originating from sources patients traditionally rely on, such as health care services and the government. Interestingly, one previous study observed easier readability of sources affiliated with the government and public health services, albeit still more difficult than the recommended readability grade level [21]. Other studies, on the other hand, report very small or insignificant differences between website affiliations [15, 17, 20]. This calls attention to the need for measures aiming to enhance the readability and understandability of online sources in general.

Understandability is acknowledged in the literature as a core quality criteria for web-based information, capturing nuances of presentation, writing, and language not represented within automated readability formulas [24]. As of yet, understandability of web-based sources about COVID-19 has not been extensively investigated in research. One study indicate good understandability with a PEMAT score of 83% [18], illustrating results conflicting with our findings and a need for further investigation in order to draw firm conclusions. Research has shown that complementing standard information routes with non-textual media, such as relevant and clear illustrations, has the potential to increase information uptake and combat low health literacy [35]. However, only 7 (9%) of the included websites in our study utilized visual aids when possible to make content more easily understood. Another study report very similar findings, with 7% of the websites therein providing graphical information [17]. Taken together, the results of our study highlights a high probability that members in the general population are faced with texts containing information of difficult readability and low understandability when they turn to the Web for information about COVID-19. We acknowledge an urgent need to improve quality standards on the Web and highly encourage future research that addresses this challenge.

The COVID-19 pandemic placed substantial challenges and responsibilities on all persons within the general population, involving health-related behavioral changes related to preventive measures applied in their daily lives [36]. As a response to these unfamiliar and crucial circumstances, a high demand on high-quality and continuously updated information was seen [13, 37]. High COVID-19 hospitalization and mortality has been reported in Sweden, which has been shown to be associated with the proportion of migrants living in certain geographical areas [38]. Moreover, high risk to contract COVID-19 when not speaking the native language in a country has been reported [39], calling further attention to the need for reliable information available in a range of different languages. A review of government produced risk communication about COVID-19 revealed that a considerable proportion of countries in Europe lack translated information, concluding substantial and important gaps in the availability translated information available for non-native speakers [40]. According to another study investigating online information in the UK, there has been a lack of resources and appropriate COVID-19 online educational material available to minority groups, with substandard readability and a significant lack of translated information [17]. Our results give further weight to this problem, showing that the majority of the included websites only included Swedish information. Sweden has a diverse population, with many ethnic backgrounds represented and a significant proportion not being native speakers. Our findings indicate a high possibility that non-Swedish speaking persons seeking high-quality information about COVID-19 grounded in a Swedish context experienced difficulties finding online information written in an alternative language.

Understanding information about preventive measures, including when and how to apply these in daily life, is necessary in order to mitigate the spread of infectious diseases leading to epidemics and pandemics [12, 41]. The difficult readability, poor understandability and lack of readily translated material could have affected their comprehension and retention of the information, which may have affected the spread of the disease, and this should be addressed in future studies. We acknowledge that some persons may use automated translation services such as Google translate when accessing information about COVID-19. We investigated readily available translated versions of Swedish information and cannot make any conclusions about the reliability of using these automated services. Our findings call attention to the need for systematic approaches among developers of consumer-oriented health-related information on the Web to provide information that is both readable and understandable. Utilizing easy-to-understand language translated in alternative languages and accompanied by appropriate visual aids has the potential to enhance knowledge within the general population, but more research about this is needed.

Methodological considerations

There are methodological limitations that need to be considered when interpreting the results of this study. The searches were designed with the intent to mimic search patterns in the general population, but we cannot dismiss the risk of not identifying some online sources used by the general population. Information consumers may utilize search terms in other languages, other search engines, or other language versions of Google. This would affect which websites that would be accessed and would thus affect the generalizability. Our findings need to be interpreted together with other studies investigating websites in other languages and settings. Readability was determined with Readability Index (LIX), which is a popular and established method used to produce a quantified score based on the number of words in sentences and the proportion of long words [29]. Automated readability formulas have been criticized for producing a simplified understanding of readability determined through formal text properties, not taking into account other complex aspects such as word familiarity and medical jargon [42]. Therefore, we also assessed each included website with two systematic instruments focusing on interaction aspects such as language, understandability, visual aids, tone, and presentation. Readability and literacy are complex concepts that are not easily captured in full. We acknowledge that the utilized formulas and systematic tools in our study have intrinsic limitations and do not take into account the health literacy of the recipient. Thus, we encourage further studies within this field of research.

The assessments confirmed substandard quality, performed by a researcher and associate professor. We acknowledge a need to conduct research that utilize laypersons as assessors of website quality, particularly those with low health literacy and reading texts in their non-native language. One assessor, who is a health professional and a researcher, scored the included websites. While other studies have shown adequate interrater reliability when utilizing EQIP [31] and PEMAT [43], this is nevertheless a limitation that needs to be considered when interpreting the findings. The large majority of studies investigating readability of online information about COVID-19 have been based on English content, and therefore, we argue that our study brings novel and important findings. Nevertheless, the generalizability of the results needs to be taken into consideration when transferring it to other contexts.

Conclusion

According to our findings, Swedish websites about the coronavirus disease 2019 contain information of difficult readability and poor understandability. Links to information available in alternative languages was very scarce. It is possible that these deficits contributed to the spread and impact of the virus. We encourage more studies investigating methods to increase the readability, understandability and language accessibility of web-based information at the beginning of an epidemic or pandemic.

Availability of data and materials

All data generated or analysed during this study are included in this published article and its supplementary information files.

Abbreviations

COVID-19:

The Coronavirus Disease 2019

LIX:

Readability Index [Läsbarhetsindex]

EQIP:

The Ensuring Quality Information for Patients tool

PEMAT-PU:

The Patient Education Materials Assessment Tool for printable

References

  1. Kummervold PE, Chronaki CE, Lausen B, Prokosch H-U, Rasmussen J, Santana S, et al. eHealth trends in Europe 2005–2007: a population-based survey. J Med Internet Res. 2008;10:e42.

    Article  Google Scholar 

  2. Quittschalle J, Stein J, Luppa M, Pabst A, Löbner M, Koenig H-H, et al. Internet use in old age: results of a german population-representative survey. J Med Internet Res. 2020;22:e15543.

    Article  Google Scholar 

  3. Eysenbach G, Jadad AR. Evidence-based patient choice and consumer health informatics in the Internet age. J Med Internet Res. 2001;3:E19.

    CAS  Article  Google Scholar 

  4. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res. 2001;16:671–92.

    CAS  Article  Google Scholar 

  5. Powell J, Inglis N, Ronnie J, Large S. The characteristics and motivations of online health information seekers: cross-sectional survey and qualitative interview study. J Med Internet Res. 2011;13:e20.

    Article  Google Scholar 

  6. Langford AT, Roberts T, Gupta J, Orellana KT, Loeb S. Impact of the internet on patient-physician communication. Eur Urol Focus. 2020;6:440–4.

    Article  Google Scholar 

  7. U.S. Department of Health and Human Services. Health literacy in healthy people 2030. 2021. https://health.gov/our-work/national-health-initiatives/healthy-people/healthy-people-2030/health-literacy-healthy-people-2030.

  8. Kutner M, Greenberg E, Paulsen C, White S. The health literacy of America’s adults: results from the 2003 national assessment of adult literacy. 2006.

  9. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011;155:97–107.

    Article  Google Scholar 

  10. Alexandria CQ. Plain language for patient education. J Consum Health Internet. 2017;21:209–15.

    Article  Google Scholar 

  11. Juul FE, Jodal HC, Barua I, Refsum E, Olsvik Ø, Helsingen LM, et al. Mortality in Norway and Sweden during the COVID-19 pandemic. Scand J Public Health. 2021. https://doi.org/10.1177/14034948211047137.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Arefi MF, Poursadeqiyan M. A review of studies on the COVID-19 epidemic crisis disease with a preventive approach. Work. 2020;66:717–29.

    Article  Google Scholar 

  13. Le HT, Nguyen DN, Beydoun AS, Le XTT, Nguyen TT, Pham QT, et al. Demand for Health Information on COVID-19 among Vietnamese. Int J Environ Res Public Health. 2020;17:4377.

    CAS  Article  Google Scholar 

  14. Schulz PJ, Nakamoto K. Health literacy and patient empowerment in health communication: the importance of separating conjoined twins. Patient Educ Couns. 2013;90:4–11.

    Article  Google Scholar 

  15. Basch CH, Mohlman J, Hillyer GC, Garcia P. Public health communication in time of crisis: readability of on-line COVID-19 information. Disaster Med Public Health Prep. 2020;14:635–7.

    Article  Google Scholar 

  16. Jayasinghe R, Ranasinghe S, Jayarajah U, Seneviratne S. Quality of online information for the general public on COVID-19. Patient Educ Couns. 2020. https://doi.org/10.1016/j.pec.2020.08.001.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Khan S, Asif A, Jaffery AE. Language in a time of COVID-19: literacy bias ethnic minorities face during COVID-19 from online information in the UK. J Racial Ethnic Health Disparities. 2020. https://doi.org/10.1007/s40615-020-00883-8.

    Article  Google Scholar 

  18. Kruse J, Toledo P, Belton TB, Testani EJ, Evans CT, Grobman WA, et al. Readability, content, and quality of COVID-19 patient education materials from academic medical centers in the United States. Am J Infect Control. 2020. https://doi.org/10.1016/j.ajic.2020.11.023.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Mishra V, Dexter JP. Comparison of readability of official public health information about COVID-19 on websites of international agencies and the governments of 15 countries. JAMA Netw Open. 2020;3:e2018033.

    Article  Google Scholar 

  20. Szmuda T, Özdemir C, Ali S, Singh A, Syed MT, Słoniewski P. Readability of online patient education material for the novel coronavirus disease (COVID-19): a cross-sectional health literacy study. Public Health. 2020;185:21–5.

    CAS  Article  Google Scholar 

  21. Worrall AP, Connolly MJ, O’Neill A, O’Doherty M, Thornton KP, McNally C, et al. Readability of online COVID-19 health information: a comparison between four English speaking countries. BMC Public Health. 2020;20:1635.

    CAS  Article  Google Scholar 

  22. Valizadeh-Haghi S, Khazaal Y, Rahmatizadeh S. Health websites on COVID-19: are they readable and credible enough to help public self-care? J Med Libr Assoc. 2021;109:75–83.

    Article  Google Scholar 

  23. Wrigley Kelly NE, Murray KE, McCarthy C, O’Shea DB. An objective analysis of quality and readability of online information on COVID-19. Health Technol (Berl). 2021;11:1093–9.

    CAS  Article  Google Scholar 

  24. Sun Y, Zhang Y, Gwizdka J, Trace CB. Consumer evaluation of the quality of online health information: systematic literature review of relevant criteria and indicators. J Med Internet Res. 2019;21:e12522.

    Article  Google Scholar 

  25. Stern J, Georgsson S, Carlsson T. Quality of web-based information at the beginning of a global pandemic: a cross-sectional infodemiology study investigating preventive measures and self care methods of the coronavirus disease 2019. BMC Public Health. 2021;21:1141.

    CAS  Article  Google Scholar 

  26. The Internet Foundation in Sweden. Svenskarna och internet 2018 [Swedes and the internet 2018]. 2018. https://www.iis.se/docs/Svenskarna_och_internet_2018.pdf. Accessed 28 Dec 2018.

  27. Fiksdal AS, Kumbamu A, Jadhav AS, Cocos C, Nelsen LA, Pathak J, et al. Evaluating the process of online health information searching: a qualitative approach to exploring consumer perspectives. J Med Internet Res. 2014;16:e224.

    Article  Google Scholar 

  28. Feufel MA, Stahl SF. What do web-use skill differences imply for online health information searches? J Med Internet Res. 2012;14:e87.

    Article  Google Scholar 

  29. Björnsson CH. Läsbarhet [Readability]. Stockholm: Liber; 1968.

  30. Anderson J. Lix and Rix: variations on a little-knownreadability index. J Read. 1983;26:490–6.

    Google Scholar 

  31. McCool ME, Wahl J, Schlecht I, Apfelbacher C. Evaluating written patient information for eczema in German: comparing the reliability of two instruments, DISCERN and EQIP. PLoS ONE. 2015;10:e0139895.

    Article  Google Scholar 

  32. Moult B, Franck LS, Brady H. Ensuring quality information for patients: development and preliminary validation of a new instrument to improve the quality of written health care information. Health Expect. 2004;7:165–75.

    Article  Google Scholar 

  33. Shoemaker SJ, Wolf MS, Brach C. Development of the Patient Education Materials Assessment Tool (PEMAT): a new measure of understandability and actionability for print and audiovisual patient information. Patient Educ Couns. 2014;96:395–403.

    Article  Google Scholar 

  34. The Joint Commission. Advancing effective communication, cultural competence, and patient- and family-centered care: a roadmap for hospitals. Oakbrook Terrace: The Joint Commission; 2010.

  35. Houts PS, Doak CC, Doak LG, Loscalzo MJ. The role of pictures in improving health communication: a review of research on attention, comprehension, recall, and adherence. Patient Educ Couns. 2006;61:173–90.

    Article  Google Scholar 

  36. Helsingen LM, Refsum E, Gjøstein DK, Løberg M, Bretthauer M, Kalager M, et al. The COVID-19 pandemic in Norway and Sweden - threats, trust, and impact on daily life: a comparative survey. BMC Public Health. 2020;20:1597.

    CAS  Article  Google Scholar 

  37. Rovetta A, Bhagavathula AS. COVID-19-related web search behaviors and infodemic attitudes in Italy: infodemiological study. JMIR Public Health Surveill. 2020;6:e19374.

    Article  Google Scholar 

  38. Fonseca-Rodríguez O, Gustafsson PE, San Sebastián M, Connolly A-MF. Spatial clustering and contextual factors associated with hospitalisation and deaths due to COVID-19 in Sweden: a geospatial nationwide ecological study. BMJ Glob Health. 2021;6:e006247.

    Article  Google Scholar 

  39. Kim HN, Lan KF, Nkyekyer E, Neme S, Pierre-Louis M, Chew L, et al. Assessment of disparities in COVID-19 testing and infection across language groups in Seattle. Washington JAMA Netw Open. 2020;3:e2021213.

    Article  Google Scholar 

  40. Nezafat Maldonado BM, Collins J, Blundell HJ, Singh L. Engaging the vulnerable: A rapid review of public health communication aimed at migrants during the COVID-19 pandemic in Europe. J Migr Health. 2020;1:100004.

    Article  Google Scholar 

  41. Anwar A, Malik M, Raees V, Anwar A. Role of mass media and public health communications in the COVID-19 pandemic. Cureus. 2020;12:e10453.

    PubMed  PubMed Central  Google Scholar 

  42. Bailin A, Grafstein A. Readability: text and context. London: Palgrave Macmillan; 2016.

    Book  Google Scholar 

  43. Vishnevetsky J, Walters CB, Tan KS. Interrater reliability of the Patient Education Materials Assessment Tool (PEMAT). Patient Educ Couns. 2018;101:490–6.

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

Open access funding provided by Uppsala University.

Author information

Affiliations

Authors

Contributions

SG conceived and designed the study, and critically reviewed the manuscript. TC conceived and designed the study, collected the data, analyzed the data and wrote the manuscript. Both authors have read and approved the final version of this manuscript.

Authors’ information

Information about the assessor of the data; TC is a registered nurse, a specialist intensive care nurse with additional formal education in infectious diseases, is a registered midwife, has a master’s degree in caring sciences, has a Ph.D. in medical sciences, is a researcher at the Swedish Red Cross University College and Uppsala University, and has a clinical background in geriatrics, hematology and intensive care. TC has previous experience of conducting systematic quality assessment of websites about various health-related topics.

Corresponding author

Correspondence to Tommy Carlsson.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1

. STROBE checklist for cross-sectional studies.

Additional file 2

. Searches in Google.se.

Additional file 3

. Data set.

Additional file 4

. Quality scores of the top and bottom three websites in regard to the investigated variables.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Georgsson, S., Carlsson, T. Readability, understandability and language accessibility of Swedish websites about the coronavirus disease 2019: a cross-sectional study. BMC Med Inform Decis Mak 22, 131 (2022). https://doi.org/10.1186/s12911-022-01873-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12911-022-01873-y

Keywords

  • Consumer health information
  • COVID-19
  • Readability
  • Severe acute respiratory syndrome coronavirus 2
  • World wide web
  • Quality