Skip to main content
Download PDF

Development and usability evaluation of an electronic health report form to assess health in young people: a mixed-methods approach

BMC Medical Informatics and Decision Making volume 23, Article number: 91 (2023) Cite this article

Abstract

Background

Electronic Patient-Reported Outcomes (ePROs) have potential to improve health outcomes and healthcare. The development of health-technology applications, such as ePROs, should include the potential users and be theoretically grounded. Swedish Youth Health Clinics (YHCs) offer primarily sexual and psychological healthcare for young people aged 12 to 25 years old. Young people in healthcare settings are considered a vulnerable group. The development of a collection of Patient-Reported Outcomes (PROs) in an Electronic Health Report Form (eHRF) for identifying health and health-related problems in young people, was preceded by a qualitative interview study, exploring young people’s views on using an eHRF at YHCs and which questions about health an eHRF should contain. The aim of the current study was to develop and evaluate the usability of an eHRF prototype for identifying health and health-related problems in young people visiting YHCs.

Methods

This study used a participatory design. During the development, an expert panel consisting of eight researchers and one Information Technology worker, participated. A wide literature search was performed to find PROs to construct an eHRF prototype to cover health areas. A mixed methods usability evaluation included 14 participants (young people, healthcare professionals, and an expert panel).

Results

The development resulted in an eHRF prototype, containing ten reliable and valid health questionnaires addressing mental-, physical-, and sexual health and social support, a self-efficacy question, and background questions, in total 74 items. The interviews in the usability evaluation resulted in three categories describing the usability of the eHRF: ‘Captures the overall health of young people but needs clarification’, ‘Fun, easy, and optional and will keep young people’s interest’, and ‘Potential contribution to improve the health consultation’. The quantitative results support the usability of the eHRF for YHCs.

Conclusions

The participatory approach contributed to development of the eHRF prototype to cover health areas adapted for the target population. The usability evaluation showed that the eHRF was usable and had the potential for self-reflection and contributions to cooperation between young people and healthcare professionals during the health consultation.

Peer Review reports

Introduction

Health technology, such as Electronic Patient-Reported Outcomes (ePRO), have advantages over paper-based Patient-Reported Outcomes (PRO), for example decrease costs, improve data quality, and facilitate clinical symptom management and decision making [1, 2]. The use of ePROs has the potential to improve health outcomes [3], and influence patients and organizations, such as healthcare services, to work toward personalized health-related needs [4]. Both ePROs and analog PROs provide evidence-based [3], self-reported patient health status [5], and help to understand the treatment impact on functioning and well-being [6]. The development of health-technology applications, such as ePROs, demands an extensive process characterized by the participation of technology developers, clinicians, and patients to ensure that relevant health outcomes are captured [7,8,9], the technology fits within the healthcare systems, and is adequate and durable for the users [3].Validation of ePROs may be important, depending on the degree of modifications made when converting from PROs to ePROs [2].

Participatory research is used to develop applications for assessing health and health-related problems in young populations. In New Zealand, the ePRO YouthCHAT [10], is used for assessing psychosocial health (mental health and risky behaviors) in young people from 13 to 25 years of age [10, 11], long-term physical conditions at outpatient clinics [12] and Maori youth [13], as well as investigating the motivation for accepting help [14]. In the context of a young Swedish population (12 to 25 years) visiting Youth Health Clinics (YHCs), the SEXual health Identification Tool (SEXIT), is used to identify young people at increased risk of, or with existing poor sexual health [15], and the young person’s will to address health-related issues. Staff at YHCs have found SEXIT useful in YHC health assessments [16]. YHCs in Sweden focus on strengthening sexual and reproductive health and the rights of youth and young adults, but also address young people’s physical and mental well-being. Young people are a vulnerable group, as they often lack of autonomy [17] in the healthcare setting. For example, young people may find it hard to ask about health concerns and vulnerable feelings to healthcare professionals. This can lead to health risks not being detected by healthcare, which contributes to health inequity [18].

Human health is complex and depends on an interaction of biological, psychological, and social factors, as described by the biopsychosocial theory [19]. This is also reflected in the World Health Organization’s (WHO) 1948 definition of health as “a state of complete physical, mental, and social well-being, not merely the absence of disease or infirmity” (page 1) [20]. Thus, the assessment with ePROs used at YHCs needs to consider not only sexual health, but also other aspects of young people´s health. A previous interview study with fifteen young people, aged 17 to 22 years, visiting five YHCs in two regions in central Sweden showed that an electronic assessment for young people should include questions about mental-, physical-, sexual health and social support, and have the potential for self-reflection and increased self-awareness [21].

With the previous study as a starting point, the aim of this study was twofold. First, to develop a collection of ePRO questionnaires; an Electronic Health Report Form (eHRF) prototype. The eHRF prototype should contain ePRO questionnaires in four health areas, to identify health and health-related problems in young people visiting YHCs. Second, the aim was to evaluate the usability of the eHRF prototype.

Materials and methods

The development of the eHRF prototype was based on the theoretical foundation [22] from a participatory research approach [23]. For the current study, participation means the right of young people and healthcare professionals to influence decisions for the content and layout of an eHRF for the YHC [24]. The development of an eHRF prototype was guided by biopsychosocial theory [19] and a holistic view of health. The concept of self-efficacy [25] was used to clarify the state of engagement for possible behavior change.

The study was approved by the Regional Ethics Committee, Uppsala, Sweden (dnr 2020 − 01921) and was performed according to the principles of the Declaration of Helsinki [26].

Study design

This study was conducted during autumn 2020 and consisted of two parts: (I) the development and (II) the usability evaluation of the eHRF prototype.

  1. I)

    To develop an eHRF prototype for identifying health and health-related problems in young people visiting YHCs, a wide literature search was performed, based on a previous interview study [21].

  2. II)

    The usability evaluation of the eHRF prototype was applied with a mixed-methods convergent design, using qualitative and quantitative data collection in a side-by-side approach and merging of data by discussion [27], as shown in Fig. 1.

Fig. 1
figure 1

Mixed-methods convergent design in the usability evaluation of the Electronic Health Report Form (eHRF) prototype

Full size image

Participants

Overall, a diverse group of participant characteristics was wanted, to include as many aspects of knowledge, expertise, and experience as possible, which is important in participation research [28]. Young people had previously participated in identifying the different health areas to include in an eHRF for YHC [21], allowing the researchers’ perspective to assert itself in the development phase. The target number of participants for the development phase was an expert panel with eight researchers. For the usability evaluation, the participant target number was 12 young people and three YHC healthcare professionals.

eHRF prototype development participants

Participants in the development of the eHRF prototype were an expert panel and an Information Technology (IT) company (Fig. 2).

Fig. 2
figure 2

The participants in the Electronic Health Report Form (eHRF) prototype development

Full size image

The expert panel members were purposively selected based on their professional and academic experience. The participating members had multi-professional clinical experience in the development of health technology and working with young people in mental healthcare, primary healthcare, and YHCs. Their academic degrees ranged from Master of Science to professor.

A collaboration with an IT-company contact person was included. The IT-company had experience in collaboration with several universities and healthcare regions in Sweden, offering solutions for a medical documentation system [29].

eHRF prototype usability evaluation participants

Participants in the eHRF prototype usability evaluation were young people, healthcare professionals and an expert panel (Fig. 3). A convenient sample of healthcare professionals and young people were included from the same selected YHC, situated in a small municipality in Central Sweden, taking approximately 25 appointments per week. The expert panel were affiliated with five regions in Central Sweden.

Fig. 3
figure 3

The participants in the Electronic Health Report Form (eHRF) prototype usability evaluation

Full size image

Inclusion criteria were: 16–23 years old, visiting the YHC, and being fully fluent in Swedish. Heterogeneity was desired in terms of gender identity, place of birth, living conditions, sexual orientation, and level of education. The young people were informed about the study by healthcare professionals during a planned visit to the YHC. They were asked about participation and to be contacted by the first author (PVL) for information on voluntary participation and confidentiality, registration in the IT system, and time for data collection.

All healthcare professionals working at the selected YHC participated in the study and received information about the study in an e-mail that included information on voluntary participation and confidentiality, and registration in the IT system. Appointments were booked for the data collection.

The purposively selected expert panel members were chosen using the same criteria used for the eHRF prototype development expert panel members. The experts were e-mailed information about the study and invited to a digital group meeting.

eHRF prototype development

Data collection

The selection of PRO questionnaires for the eHRF prototype was in line with the COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) guidelines for selecting outcome measurements [30], presented in Fig. 4. A stepwise process started with determining the domains to be measured. Consensus for the health areas to focus on was reached within the research group, based on the previous interview study with young people [21]; physical health (including lifestyle habits), mental health (including questions about self-harm and suicide risk), sexual health and sexual experience and relationships/social support. The next step involved finding the PROs to include. This was based on a broad PubMed scope and internet search, performed by PVL in 2019. The search involved keywords and phrases, for instance, “health”, “health questionnaire”, eHealth, and “young people/youth/adolescent”. The Swedish website https://www.fbanken.se, a resource targeted at healthcare professionals and constructed to collect valid health questionnaires, primarily concerning mental health and social support, was also consulted for appropriate questionnaires. The references in all identified studies were explored. The expert panel’s clinical work and research experience contributed with suggestions for questionnaires to, and ideas for searching for new questionnaires to consider for the eHRF prototype.

Fig. 4
figure 4

Development process with COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) guidelines

Full size image

Analysis

An iterative analysis process took place amongst members of the expert panel. The panel discussed appropriate questionnaires for the eHRF prototype, based on validity and reliability. The PRO questionnaires were evaluated for reliability and validity, questionnaire length, target group and if translated into Swedish.

The iterative process helped to limit the selection of included PRO questionnaires and ended when consensus had been reached among the expert panel.

eHRF prototype construction

The selected PRO questionnaires were compiled in a paper version. The researchers PVL and AS constructed one question of self-efficacy for change in accordance with Bandura’s [31] work, with the recommended 100-poing Numerical Rating Scale (NRS) for responses.

To prepare the questionnaires for digitalization, PVL, ETA and MA coded the questionnaire items and response options in Microsoft Excel. The IT-company digitalized the questionnaires in close collaboration with PVL and ETA. The eHRF prototype was divided into sections based on content: physical health (including lifestyle habits), mental health, sexual health, and social support. Each PRO questionnaire started with the visual aid of an emoji, portraying its’ content. The emojis were all free to use from https://pixabay.com/sv/images/search/emojies/.

The digitalization process and the eHRF prototype functionality were tested by the expert panel in an iterative process and changes were made continuously until coherence was reached.

eHRF prototype usability evaluation

Data collection

Interviews

All participants were informed of the study conditions and gave consent before the interviews. The interviews were either in person at the YHC or digital and were performed individually except for the expert panel who participated in a digital group interview. Participants in digital interviews gained access to the eHRF through an SMS to their smartphones. Participants in face-to-face interviews gained access to a tablet, provided by PVL. The interviews followed a semi-structured interview guide (Table 1). PVL performed the interviews, which lasted 50–90 min. All interviews were recorded with an external voice recorder and transcribed verbatim. For details in qualitative data collection procedure, see Supplementary file 1.

Table 1 The interview guide for the usability aspects of the Electronic Health Report Form (eHRF) prototype
Full size table
System Usability Scale

The System Usability Scale (SUS) [32] is a 10-item scale (Table 2) applied to assess the users’ perceived usability of a product [33]. A five-point Likert scale, ranging from Totally disagree [1] to Totally agree [5], was used for each item. The Swedish version SUS 1,4_sv [34] was used. The scale’s original phrase “system” has been changed to “eHRF prototype”.

The SUS scores can be ranged and converted to different grades of usability [33], to help interpretation of the scores. Values from 71.4 to 100 are viewed as “good usability” up to “best imaginable usability”. Values of 80 or more are considered above the average usability mean score [33].

Table 2 The statements of the System Usability Scale questionnaire [34] with indications of negative or positive statements
Full size table

Data analysis

Inductive qualitative content analysis was performed for the interviews [35]. All transcribed interviews were read repeatedly for familiarization with the data. Throughout the process, dialog occurred between the researchers (PVL, ÅR, MM, ETA, AS). First, two transcripts were read and meaning units were coded separately by two of the authors (ÅR and PVL) with the purpose of ensuring agreement on the process and content of important aspects of the data. Comparison was made to strengthen credibility before proceeding with the analysis. Data describing the participants’ views on usability and ideas for improvements to the eHRF prototype were coded. The development of the categories was discussed and elaborated several times between the authors to establish equal content within the category and orthogonality between the categories (Table 3).

Table 3 Examples of the schematic analysis process
Full size table

The SUS Likert scale scores for each item and participant will be presented and converted into a total value for each participant according to instructions [33].

Results

Participants

Totally participated, in the eHRF prototype development, eight researchers with variety of academic experiences in the expert panel (Table 4) and one participant represented the IT-company.

Table 4 Characteristics of the expert panel participants
Full size table

In total 14 people participated in the eHRF prototype usability evaluation. Young people (n = 4) and healthcare professionals (n = 3) participated in both the qualitative and quantitative usability evaluation. Demographic information is presented in (Tables 5 and 6). The members of the expert panel, except the doctoral student (n = 7), participated in the qualitative usability evaluation.

Table 5 Demographic information of the young people
Full size table
Table 6 Demographic information of the healthcare professionals from the Youth Health Clinic (YHC).
Full size table

The participants in the development and usability evaluation of the eHRF prototype are displayed in Fig. 5.

Fig. 5
figure 5

Process and participants in developing and evaluating the Electronic Health Report Form (eHRF) prototype [21]

Full size image

The eHRF prototype development

A total of 43 health questionnaires (17 physical health, 19 mental health, 3 sexual health, 4 social support) were identified for possible inclusion in the eHRF prototype. The iterative analysis process contributed to the selection of 10 questionnaires: consisting of 66 items, and seven background questions. The questionnaires the SEXual health Identification Tool (SEXIT) [15], the questions for the National Guidelines: Living Habits [36], and the questions for the National Guidelines: Living Habits, indicator questions for physical activity [36, 37] and the study-specific self-efficacy question were developed in Swedish [31] (for English versions, see Supplementary file 2. The questionnaires Alcohol Use Disorders Identification Test – Consumption (Audit-C) [38], Health Behavior in School-aged Children (HBSC) [39, 40], Generalized Anxiety Disorder 7-item scale (GAD-7) [41], Patient Health Questionnaire-9 (PHQ-9) [42], The SCOFF Questionnaire (acronym from the questions) [43], and Berlin Social Support Scales (BSSS) subscales “Need for support” and “Support seeking” [44] were available in Swedish (for English version see Supplementary file 2). The selected questionnaires had at least acceptable reliability and validity in its original language. However, some questionnaires were not tested for the YHC age group 12 to 23.

In total, the eHRF prototype consisted of 74 items (Table 7).

Table 7 The Electronic Health Report Form (eHRF) prototype with background questions, reliability, and validity
Full size table

One PRO questionnaire, The Berlin Social Support Scale (BSSS) [44] was not previously translated to Swedish. Therefore, a translation process was performed. PVL and AS translated the English version of BSSS. The Swedish translation was sent to an American psychologist/researcher, fluent and living in Sweden, who translated the BSSS back to English. PVL and AS reviewed the translation and only made adjustments to a few items to improve comprehension in Swedish.

The eHRF prototype usability evaluation

Interviews

The qualitative content analysis of the transcribed interviews resulted in three categories and seven subcategories. The categories are described below and displayed in Fig. 6. Throughout the results section the term “participants” is used when young participants, healthcare professionals, and the expert panel are included. A more detailed description of the categories and subcategories including citations is available (Supplementary file 3).

Fig. 6
figure 6

Overview of the categories and subcategories for the Electronic Health Report Form (eHRF) prototype usability

Full size image

Captures overall health of young people but needs clarification

This category confirmed that the eHRF prototype health areas (mental-, physical-, and sexual health and social support) captured the overall health of young people. The young participants believed that the eHRF prototype had the potential to make young people aware of their health and possible need for behavior change. However, for young people who were not willing to change, the self-efficacy question of behavior change could be difficult to answer.

There were suggestions for improvements, for example remove questions on hight and weight and neutralize gender descriptions. Also, they suggested to focus more on family relations. The participants wanted a definition for “social support” and the term “behavioral change” changed to “lifestyle change” and a need for another word for “anxious”. The healthcare professionals found the self-efficacy behavior change question important and connected it to their experience of the challenge of having conversations about behavior change.

Fun, easy and optional will keep young people’s interest

This category stated that an easy-going and neutral eHRF prototype design, good layout, and structure could help young people answer the health questions. Although they liked the emojies in the eHRF, they suggested that the emojis could possibly be exchanged for other pictures or even colors or fonts. The expert panel and healthcare professionals, on the other hand, were hesitant or critical toward details in the layout, suggesting that the emojis used in the eHRF prototype could affect young people by portraying an emotion about the health area and questions it represented; they wanted the emojis to be changed to something more neutral, commenting that “less is more.” The participants found it essential that young people could choose to answer the health questions before the meeting. Both young participants and healthcare professionals wanted young people to be able to disregard questions that they were uncomfortable answering.

Potential contribution to improving the health consultation

The third category summarized that the eHRF prototype was found to bring something valuable to the health assessment conversation. The young participants thought that the eHRF would help them to answer honestly to sensitive questions. The healthcare professionals, on the other hand, hesitated on the questions’ sensitive topics and suspected that young people may not even complete the eHRF prototype questions. They also suspected that answering questions without knowing who they would meet at the YHC could make young people unwilling to use the eHRF prototype.

The young participants believed that answering the health questions could make them more focused on their health and help them prioritize which health areas that were more important. The healthcare professionals agreed with the young participants and said that the eHRF prototype questions and the young people´s answers could form solid ground for talking with young people about health and contributing to their professional evaluation.

The satisfaction usability scale

The individual and the sum of participants’ scores for each item of the SUS are shown in Tables 8 and 9. The participants claimed high agreement with all statements. The participants rated highly, the positive statements “I think I would like to use the eHRF prototype” and I would imagine that most people would learn to use this eHRF prototype very quickly”. They disagreed most with the negative statement “I think that I would need the support of a technical person to be able to use this eHRF prototype”. There were no missing items in the SUS responses.

Table 8 The participants individual scores and sum for each positive statement of the System Usability Scale
Full size table
Table 9 The participants individual scores and sum for each negative statement of the System Usability Scale
Full size table

The SUS scores were converted to grades of usability [33]. The converted scores show that five participants’ scores concurred with the grade “best imaginable” (participants 3, 4, 5, 6, 7) and two participants’ scores were concurrent with the grade “good” (participants 1 and 2), as shown in Fig. 7.

Fig. 7
figure 7

Participants’ (n = 7) individually converted usability scores and corresponding grades.

Full size image

Discussion

This study had a participatory research approach to develop and evaluate an eHRF prototype with credible content and usability at YHCs. The current study was based on a previous interview study with young people at YHCs [21] that was the start of the development process, exploring of content and design of an eHRF for YHCs and hence, strengthening the young person’s perspective in this study. Due to ethical considerations, the youngest YHC visitors (12–15 years) were excluded. However, the participants in the eHRF evaluation represent the clinical YHC young- and healthcare professional population. Together with the expert panel participants, the group shows diversity in ages, knowledge, and experiences.

The search for and selection of PRO questionnaires in this development of the eHRF were mostly in line with the COSMIN guidelines [30]. However, the literature search had limitations in structure and search terms, reducing the possibility to reproduce the search. The construction was strengthened by the previous interview study of young people [21] and the expert panel’s experience of health questionnaires. Including several stakeholders [45], and performing the development process in steps [30], are ways to assess content validity, described by Almanasreh, Moles and Chen [46]. The researchers developed the eHRF prototype from ten questionnaires [28, 33,34,35,36,37,38,39,40,41], in total 74 items including background questions and a self-efficacy question. Self-efficacy questions have potential to increase the understanding of young individuals’ ability to change behavior to improve health [25], and occur in PROs/ePROs for young people.

The evaluation of the eHRF prototype usability consisted of qualitative interviews and a quantitative usability questionnaire. The combination of the qualitative and quantitative data was assumed to provide additional perspectives and a more complete understanding [27] of the usability of the eHRF prototype. Overall, the results indicated that the eHRF prototype was usable for YHCs. In summary, there were positive correlations between qualitative and quantitative data, indicating that the eHRF prototype was easy to understand and found easy to use. The qualitative data showed that the young participants were also positive toward the eHRF prototype possibility to increase their understanding of health. This is supported by the previous interview study, stating that answering meaningful health questions in an eHRF prototype could potentially lead to self-reflection and increased self-awareness [21]. Kutcher et al. [47] has highlighted the importance of increasing mental health literacy, i.e., how well individuals can understand, and communicate about health-related information for making informed health decisions (page 16) [48]. Same-level health conversations between young person and healthcare professional may act as the starting point for behavior change to improve health because it involves self-reflection as well as information for the consultation with the healthcare provider that will help determine suitable interventions [49]. In young people, health technology, has been used to identify psychosocial issues [11,12,13] and improving health equity [50]. Additionally, young people have pointed out the importance of electronic health questionnaires for time-efficient health assessments, directed toward what is essential to the young person [51].

The young participants thought that responding to health questions before meeting with a healthcare professional could make it easier to honestly answer sensitive questions. This has found also, by Thabrew et al. [52]. However, the healthcare professionals in the current study were concerned that the questions on sensitive topics may make it hard for young people to answer honestly. This was also found in a qualitative study that interviewed healthcare professionals at YHCs for their experiences of using SEXIT [16]. This may implicate a need for training healthcare professionals to talk about sensitive topics.

The qualitative findings highlight the importance of an appealing design, to keep young people’s interest and help them complete the eHRF prototype health questions, also supported by the previous interview study [21]. In the current study the layout and design of the eHRF prototype was appreciated by the young people, and the emojis viewed as refreshing and fun. This has been found important to improve usability and user satisfaction in guiding clinical decisions [53]. However, healthcare professionals and expert panel disagreed to some of the emojies.

The study had several limitations. The target number of participants for the usability evaluation was not reached. There were in total fourteen participants in the usability evaluation (four young people, three healthcare professionals, seven members of the expert panel). The few young people included reflects that the estimated number of appointments were even lower, due to the COVID-19 pandemic restrictions, and the water leak causing poor environment at the YHC. Another study limitation was that only seven participants responded to the SUS questionnaire. For a significant outcome, the SUS questionnaire requires at least eight participants [33]. However, for early usability evaluations, five participants have been found sufficient to identify usability issues [54]. An additional weakness is that all participants were female. Although a study limitation, this is reflective of the YHC patient population, supported by surveys, showing that almost 90% of YHC visitors are female [55]. All considered, population in the current study should be satisfactory for the usability evaluation in the YHC setting. Another limitation was that inconsistent records were kept of the young people who declined to participate, due to the clinical pre-requisites at the YHC. Finally, the ongoing COVID-19 pandemic during the study period demanded adaptions to the study procedure, for example digital interviews. This may have affected the candor of the participants’ responses to the SUS.

This study acknowledges that research needs to serve the society and its current context [56]. Hence, it was pragmatically designed to allow clinical and research solutions and to involve the future users, i.e., young people and healthcare professionals. The eHRF developed in this study, consisting of several PROs (traditionally non-digital), has not been validated. However, no changes were made to the PROs when converted digitally, indicating that the need for validation is lower [2]. Also, ePROs provide better data quality, decrease costs, and facilitate clinical symptom management and decision-making compared to PROs [1]. Still, if implementing the eHRF prototype for clinical use at YHCs, future research is needed to validate the eHRF prototype for ages 12–15 and for other healthcare settings. This eHRF has potential to provide a structured and unified assessment of young people’s health. This may reduce the risk for health inequities among young people [57].

The next research step is to further develop the eHRF prototype in considering the improvement suggestions from the participants, for example removing some questions and exchanging emojies. After improving the eHRF, a feasibility study (protocol ISRCTN23855544) is planned to evaluate feasibility aspects (process, resources and management), [58], before performing a fourth-coming Stepped Wedge Cluster Randomized Trial (SW-CRT).

Conclusions

This study presents the results from the development and early usability testing of an eHRF using a participatory approach. The collaboration of an expert panel and an IT company resulted in the development of an eHRF prototype. It was based prior findings of young people’s opinions on important health areas and valid and reliable PRO questionnaires. The eHRF reflects the biopsychosocial perspective, including self-efficacy for behavior change as a shared base for a health conversation between healthcare professional and young people at YHCs. The usability evaluation showed that the eHRF prototype was usable, could lead to self-reflection and cooperation between young people and healthcare professionals during the health consultation. One limitation to consider was the potential effect of adjusting face-to-face interviews to digital. Suggested improvements need to be considered for further eHRF development. Implication of research are on implementation barriers and facilitators within a feasibility study.

Data Availability

Deidentified data may be available from the corresponding author upon request and subject to General Data Protection Regulation (GDPR) and the Swedish Ethical Review Authority requirements.

Abbreviations

ePRO:

Electronic Patient-Reported outcome

PRO:

Patient-Reported outcome

YHC:

Youth Health Clinic

SEXIT:

Sexual health Identification Tool

WHO:

World Health Organization

eHRF:

Electronic Health Report Form

IT- company:

Information Technology company

NRS:

Numerical Rating Scale

SUS:

Satisfaction Usability Scale

SMS:

short message service

Audit:

Alcohol Use Disorder Identification Test

HBSC:

Health Behavior in School-aged Children

GAD-7:

General Anxiety Disorder 7-item scale

PHQ-9:

Patient Health Questionnaire-9

SCOFF:

acronym from the five questions in the SCOFF questionnaire

BSSS:

Berlin Social Support Scale

GDPR:

General Data Protection Regulation

References

  1. Meirte J, Hellemans N, Anthonissen M, Denteneer L, Maertens K, Moortgat P, et al. Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med. 2020;3(1):e15588.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, et al. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report. Value Health. 2009;12(4):419–29.

    Article  PubMed  Google Scholar 

  3. Sharma A, Harrington RA, McClellan MB, Turakhia MP, Eapen ZJ, Steinhubl S, et al. Using Digital Health Technology to Better Generate evidence and deliver evidence-based care. J Am Coll Cardiol. 2018;71(23):2680–90.

    Article  PubMed  Google Scholar 

  4. Riis CL, Stie M, Bechmann T, Jensen PT, Coulter A, Möller S et al. ePRO-based individual follow-up care for women treated for early breast cancer: impact on service use and workflows. J Cancer Surviv. 2021.

  5. U.S.Food and Drug Administration. Guidance for industry. Patient reported outcome meaures: use in medical product development to support labelling claims. 2009 updated 10/17/2019. Fed Regist 2009;74:65132-3.:[Available from: https://www.fda.gov/media/77832/download.

  6. Acquadro C, Berzon R, Dubois D, Leidy NK, Marquis P, Revicki D, et al. Incorporating the patient’s perspective into Drug Development and Communication: an ad Hoc Task Force Report of the patient-reported outcomes (PRO) Harmonization Group Meeting at the Food and Drug Administration, February 16, 2001. Value in Health. 2003;6(5):522–31.

    Article  PubMed  Google Scholar 

  7. Staniszewska S, Adebajo A, Barber R, Beresford P, Brady L-M, Brett J, et al. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int J consumer Stud. 2011;35(6):628–32.

    Article  Google Scholar 

  8. Haywood KL. Patient-reported outcome II: selecting appropriate measures for musculoskeletal care. Musculoskelet Care. 2007;5(2):72–90.

    Article  Google Scholar 

  9. Trujols J, Portella MJ, Iraurgi I, Campins MJ, Siñol N, de Los Cobos JP. Patient-reported outcome measures: are they patient-generated, patient-centred or patient-valued? J Ment Health. 2013;22(6):555–62.

    Article  PubMed  Google Scholar 

  10. Goodyear-Smith F, Corter A, Suh H. Electronic screening for lifestyle issues and mental health in youth: a community-based participatory research approach. BMC Med Inform Decis Mak. 2016;16(1):140.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Thabrew H, Kumar H, Goldfinch M, Cavadino A, Goodyear-Smith F. Repeated psychosocial screening of High School Students using YouthCHAT: Cohort Study. JMIR Pediatr Parent. 2020;3(2):e20976.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Thabrew H, Goodyear-Smith F. Youth version, case-finding and help Assessment Tool (YouthCHAT): an electronic, psychosocial screener for young people with long-term physical conditions. J Paediatr Child Health. 2019;56(1):102–6.

    Article  PubMed  Google Scholar 

  13. Martel RM, Darragh ML, Lawrence AJ, Shepherd MJ, Wihongi T, Goodyear-Smith FA. YouthCHAT as a primary Care E-Screening Tool for Mental Health Issues among Te Tai Tokerau Youth: protocol for a Co-Design Study. JMIR Res protocols. 2019;8(1):e12108.

    Article  Google Scholar 

  14. Goodyear-Smith F, Arroll B, Coupe N. Asking for help is helpful: validation of a brief lifestyle and mood assessment tool in primary health care. Ann Fam Med. 2009;7(3):239–44.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Hammarstrom S, Nilsen P, Lindroth M, Stenqvist K, Bernhardsson S. Identifying young people exposed to or at risk of sexual ill health: pilot implementation of an evidence-informed toolkit (SEXIT) at swedish youth clinics. Eur J Contracept reproductive health care: official J Eur Soc Contracept. 2019;24(1):45–53.

    Article  Google Scholar 

  16. Hammarström S, Lindroth M, Nilsen P, Nolskog P, Bernhardsson S. Staff’s experiences of a pilot implementation of the SEXual health identification Tool for assessing sexual ill health among visitors to swedish youth clinics: a focus group study. Sex Reprod Healthc. 2021;29:100643.

    Article  PubMed  Google Scholar 

  17. Higginbottom G, Liamputtong P, Participatory Qualitative Research Methodologies in Health. 2015 2023/02/07. 55 City Road 55 City Road, London: SAGE Publications Ltd. Available from: https://methods.sagepub.com/book/participatory-qualitative-research-methodologies-in-health.

  18. Robards F, Kang M, Luscombe G, Sanci L, Steinbeck K, Jan S, et al. Predictors of young people’s healthcare access in the digital age. Aust N Z J Public Health. 2019;43(6):582–8.

    Article  PubMed  Google Scholar 

  19. Engel GL. The need for a new medical model: a challenge for biomedicine. Science. 1977;196(4286):129–36.

    Article  CAS  PubMed  Google Scholar 

  20. World Health Organization. Basic documents of the World Health Organization 49 ed. New York, NY.: World Health Organization; 2020. (n.d.).

    Google Scholar 

  21. Lostelius P, Mattebo M, Söderlund A, Revenäs Ã, Thors Adolfsson E. An electronic patient-reported outcome created based on my needs is worth using: an explorative qualitative study investigating young people’s opinions for a health assessment tool. J Patient-Reported Outcomes. 2022;6(1):29.

    Article  Google Scholar 

  22. DeVellis RF. Scale development: theory and applications. United States pf America. SAGE Publications, Inc.; 2017.

  23. Higginbottom G, Liangputtom P, Participatory Qualitative Research Methodologies in Health. 2015 2022/05/17. 55 City Road, London: SAGE Publications Ltd. Available from: https://methods.sagepub.com/book/participatory-qualitative-research-methodologies-in-health.

  24. Sverker A, Thyberg I, Valtersson E, Björk M, Hjalmarsson S, Östlund G. Time to update the ICF by including socioemotional qualities of participation? The development of a “patient ladder of participation” based on interview data of people with early rheumatoid arthritis (the swedish TIRA study). Disabil Rehabil. 2020;42(9):1212–9.

    Article  PubMed  Google Scholar 

  25. Bandura A. The primacy of self-regulation in Health Promotion. Appl Psychol. 2005;54(2):245–54.

    Article  Google Scholar 

  26. The World Medical Association I. WMA Declaration of Helsinki - Ethical principles for medical research involving humab subjects 1964 updated 2021. Available from: https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/.

  27. Creswell JW, Creswell JD. Research Design. Qualitative, quantitative & mixed methods approaches. 5th ed. London, United Kingdom: SAGE Publications Ltd.; 2018.

    Google Scholar 

  28. Page K. Ethics and the co-production of knowledge. Public Health Res Pract. 2022;32(2).

  29. Entermedic. Available from: https://www.entermedic.com/.

  30. Prinsen CAC, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, et al. How to select outcome measurement instruments for outcomes included in a “Core Outcome Set” – a practical guideline. Trials. 2016;17(1):449.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Bandura A, Guide for constructing Self-Efficacy Scales. Self-efficacy beliefs of adolescents [Internet]. Greenwich, CT, USA.: Information Age Publishing INC; 2005.

    Google Scholar 

  32. Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClelland, editors. I, redaktörer. Usability evaluation in industry. Volume I. London: Taylor & Francis; 1996. pp. 189–94.

    Google Scholar 

  33. Lewis JR. The System Usability Scale: past, Present, and Future. Int J Hum Comput Interact. 2018;34(7):577–90.

    Article  Google Scholar 

  34. Göransson B. SUS Swedish. Rosenfeld media; 2001–2011. Available from: https://rosenfeldmedia.com/wp-content/uploads/2020/09/SUS-svensk.pdf.

  35. Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24(2):105–12.

    Article  CAS  PubMed  Google Scholar 

  36. Socialstyrelsen. Nationella riktlinjer levnadsvanor. Available from: https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/dokument-webb/nationella-riktlinjer/levnadsvanor-fragor-om-levnadsvanor.pdf.

  37. Kallings L. Validering av Socialstyrelsens screeningfrågor om fysisk aktivitet. Stockholm: Gymnastik- och idrottshögskolan Available from: https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/dokument-webb/nationella-riktlinjer/levnadsvanor-validering-av-indikatorfragor-till-patienter-om-fysisk-aktivitet.pdf.

  38. Liskola J, Haravuori H, Lindberg N, Niemelä S, Karlsson L, Kiviruusu O, et al. AUDIT and AUDIT-C as screening instruments for alcohol problem use in adolescents. Drug Alcohol Depend. 2018;188:266–73.

    Article  PubMed  Google Scholar 

  39. King A, Wold B, Tudor-Smith C, Harel Y. The health of youth. A cross-national survey. WHO Reg Publ Eur Ser. 1996;69:1–222.

    CAS  PubMed  Google Scholar 

  40. Haugland S, Wold B. Subjective health complaints in adolescence–reliability and validity of survey methods. J Adolesc. 2001;24(5):611–24.

    Article  CAS  PubMed  Google Scholar 

  41. Spitzer RL, Kroenke K, Williams JB, Lowe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7.

    Article  PubMed  Google Scholar 

  42. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–13.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  43. Morgan JF, Reid F, Lacey JH. The SCOFF questionnaire: assessment of a new screening tool for eating disorders. BMJ (Clinical research ed). 1999;319(7223):1467–8.

    Article  CAS  PubMed  Google Scholar 

  44. Schulz U, Schwarzer R. Soziale Unterstützung bei der Krankheitsbewältigung. Die Berliner Social Support Skalen (BSSS) [Social support in coping with illness: the Berlin Social Support Scales (BSSS)]. Diagnostica. 2003;49:73–82.

    Article  Google Scholar 

  45. Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: conducting a content validity study in social work research. Social Work Research. 2003;27(2):94–104.

    Article  Google Scholar 

  46. Almanasreh E, Moles R, Chen TF. Evaluation of methods used for estimating content validity. Res Social Administrative Pharm. 2019;15(2):214–21.

    Article  Google Scholar 

  47. Kutcher S, Wei Y, Costa S, Gusmão R, Skokauskas N, Sourander A. Enhancing mental health literacy in young people. Eur Child Adolesc Psychiatry. 2016;25(6):567–9.

    Article  PubMed  Google Scholar 

  48. Berkman ND, Davis TC, McCormack L. Health literacy: what is it? J Health Communication. 2010;15(sup2):9–19.

    Article  PubMed  Google Scholar 

  49. Glanz K, Rimer BK, Viswanath K. Health behavior and health education: theory, research and practice. San Francisco: Jossey-Bass; 2015.

    Google Scholar 

  50. Graham GN, Ostrowski M, Sabina AB. Population health-based approaches to utilizing digital technology: a strategy for equity. J Public Health Policy. 2016;37(Suppl 2):154–66.

    Article  PubMed  Google Scholar 

  51. Bradford S, Rickwood D. Acceptability and utility of an electronic psychosocial assessment (myAssessment) to increase self-disclosure in youth mental healthcare: a quasi-experimental study. BMC Psychiatry. 2015;15:305.

    Article  PubMed  PubMed Central  Google Scholar 

  52. Thabrew H, D’Silva S, Darragh M, Goldfinch M, Meads J, Goodyear-Smith F. Comparison of YouthCHAT, an electronic Composite Psychosocial Screener, with a clinician interview Assessment for Young People: Randomized Controlled Trial. J Med Internet Res. 2019;21(12):e13911.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Payne TH, Hines LE, Chan RC, Hartman S, Kapusnik-Uner J, Russ AL, et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc. 2015;22(6):1243–50.

    Article  PubMed  Google Scholar 

  54. Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. Conference on Human Factors in Computing Systems; Amsterdam, The Netherlands: Association for Computing Machinery; 1993. p. 206–13.

  55. Sveriges Kommuner och Regioner (SKR). Ungdomsmottagningar och ungas sexuella hälsa. Nuläge och vägen framåt. ; 2020. Report No.: ISBN 978-91-7585-880-7.

  56. Alvesson M, Sköldberg K. Reflexive methodology: new vistas for qualitative research. 3rd ed. London: Sage publications; 2017.

    Google Scholar 

  57. Marmot M. Achieving health equity: from root causes to fair outcomes. Lancet (London England). 2007;370(9593):1153–63.

    Article  PubMed  Google Scholar 

  58. Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.

    Article  PubMed  PubMed Central  Google Scholar 

  59. Selin KH. Alcohol Use Disorder Identification Test (AUDIT): what does it screen? Performance of the AUDIT against four different criteria in a swedish population sample. Subst Use Misuse. 2006;41(14):1881–99.

    Article  PubMed  Google Scholar 

  60. Hansson E, Daukantaité D, Johnsson P. SCOFF in a general swedish adolescent population. J Eat Disord. 2015;3:48.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We thank all participants in this study for their contributions to the project: young people, healthcare providers, and experts, as well as our contact at the IT company.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Region Västmanland (LTV 939219) and the Regional Research Council (RFR 939311).

Swedish Research Council for Health, Working Life and Welfare (FORTE) dnr 2018 − 01127, Regional Research Council Mid Sweden (RFR 939665).

Author information

Authors and Affiliations

  1. Clinic for Pain Rehabilitation Västmanland, Region Västmanland, Västerås, Sweden

    Petra V Lostelius

  2. School of Health, Care and Social Welfare, Mälardalen University, Västerås, Sweden

    Petra V Lostelius, Magdalena Mattebo, Anne Söderlund & Åsa Revenäs

  3. Centre for Innovation, Research and Education, Region Västmanland, Vastmanland Hospital, Vasteras, Sweden

    Petra V Lostelius

  4. Centre for Clinical Research, Region Västmanland – Uppsala University, Region Vastmanland, Vasteras, Sweden

    Eva Thors Adolfsson, Sofia Vadlin & Åsa Revenäs

  5. Department of Women’s and Children’s Health, Uppsala University, Uppsala, Sweden

    Mikael Andersén

  6. Orthopedic Clinic, Västerås hospital Region Västmanland, Västerås, Sweden

    Åsa Revenäs

Authors
  1. Petra V Lostelius
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Magdalena Mattebo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eva Thors Adolfsson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anne Söderlund
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mikael Andersén
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sofia Vadlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Åsa Revenäs
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

The first and corresponding author, PVL, was involved in all aspects of the study’s two phases, including planning, data collection and analysis, and manuscript writing. The authors MM, ETA, AS and ÅR took part in the planning of the study and the data analysis. The authors MM, AS, SV and MA were involved in the manuscript writing, where authors ETA and ÅR were major contributors. All authors read and approved the manuscript.

Corresponding author

Correspondence to Petra V Lostelius.

Ethics declarations

Competing interests

The authors declare that they have no competing interests.

Ethics approval and consent to participate

The study was approved by the Regional Ethics Committee, Uppsala, Sweden (dnr 2020 − 01921) and carried out according to the Declaration of Helsinki [26]. The participants were provided with verbal and written study information and granted informed consent.

Consent for publication

The study does not contain any detailed personal data from participants.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Supplementary Material 3

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lostelius, P.V., Mattebo, M., Adolfsson, E.T. et al. Development and usability evaluation of an electronic health report form to assess health in young people: a mixed-methods approach. BMC Med Inform Decis Mak 23, 91 (2023). https://doi.org/10.1186/s12911-023-02191-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12911-023-02191-7

Keywords

  • Developmental study
  • Electronic patient-reported outcome
  • Mixed methods research
  • Usability study
  • Medical informatics
  • Participatory research
  • Young people
  • Youth health clinic

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us