- Open Access
Development and usability evaluation of an electronic health report form to assess health in young people: a mixed-methods approach
BMC Medical Informatics and Decision Making volume 23, Article number: 91 (2023)
Electronic Patient-Reported Outcomes (ePROs) have potential to improve health outcomes and healthcare. The development of health-technology applications, such as ePROs, should include the potential users and be theoretically grounded. Swedish Youth Health Clinics (YHCs) offer primarily sexual and psychological healthcare for young people aged 12 to 25 years old. Young people in healthcare settings are considered a vulnerable group. The development of a collection of Patient-Reported Outcomes (PROs) in an Electronic Health Report Form (eHRF) for identifying health and health-related problems in young people, was preceded by a qualitative interview study, exploring young people’s views on using an eHRF at YHCs and which questions about health an eHRF should contain. The aim of the current study was to develop and evaluate the usability of an eHRF prototype for identifying health and health-related problems in young people visiting YHCs.
This study used a participatory design. During the development, an expert panel consisting of eight researchers and one Information Technology worker, participated. A wide literature search was performed to find PROs to construct an eHRF prototype to cover health areas. A mixed methods usability evaluation included 14 participants (young people, healthcare professionals, and an expert panel).
The development resulted in an eHRF prototype, containing ten reliable and valid health questionnaires addressing mental-, physical-, and sexual health and social support, a self-efficacy question, and background questions, in total 74 items. The interviews in the usability evaluation resulted in three categories describing the usability of the eHRF: ‘Captures the overall health of young people but needs clarification’, ‘Fun, easy, and optional and will keep young people’s interest’, and ‘Potential contribution to improve the health consultation’. The quantitative results support the usability of the eHRF for YHCs.
The participatory approach contributed to development of the eHRF prototype to cover health areas adapted for the target population. The usability evaluation showed that the eHRF was usable and had the potential for self-reflection and contributions to cooperation between young people and healthcare professionals during the health consultation.
Health technology, such as Electronic Patient-Reported Outcomes (ePRO), have advantages over paper-based Patient-Reported Outcomes (PRO), for example decrease costs, improve data quality, and facilitate clinical symptom management and decision making [1, 2]. The use of ePROs has the potential to improve health outcomes , and influence patients and organizations, such as healthcare services, to work toward personalized health-related needs . Both ePROs and analog PROs provide evidence-based , self-reported patient health status , and help to understand the treatment impact on functioning and well-being . The development of health-technology applications, such as ePROs, demands an extensive process characterized by the participation of technology developers, clinicians, and patients to ensure that relevant health outcomes are captured [7,8,9], the technology fits within the healthcare systems, and is adequate and durable for the users .Validation of ePROs may be important, depending on the degree of modifications made when converting from PROs to ePROs .
Participatory research is used to develop applications for assessing health and health-related problems in young populations. In New Zealand, the ePRO YouthCHAT , is used for assessing psychosocial health (mental health and risky behaviors) in young people from 13 to 25 years of age [10, 11], long-term physical conditions at outpatient clinics  and Maori youth , as well as investigating the motivation for accepting help . In the context of a young Swedish population (12 to 25 years) visiting Youth Health Clinics (YHCs), the SEXual health Identification Tool (SEXIT), is used to identify young people at increased risk of, or with existing poor sexual health , and the young person’s will to address health-related issues. Staff at YHCs have found SEXIT useful in YHC health assessments . YHCs in Sweden focus on strengthening sexual and reproductive health and the rights of youth and young adults, but also address young people’s physical and mental well-being. Young people are a vulnerable group, as they often lack of autonomy  in the healthcare setting. For example, young people may find it hard to ask about health concerns and vulnerable feelings to healthcare professionals. This can lead to health risks not being detected by healthcare, which contributes to health inequity .
Human health is complex and depends on an interaction of biological, psychological, and social factors, as described by the biopsychosocial theory . This is also reflected in the World Health Organization’s (WHO) 1948 definition of health as “a state of complete physical, mental, and social well-being, not merely the absence of disease or infirmity” (page 1) . Thus, the assessment with ePROs used at YHCs needs to consider not only sexual health, but also other aspects of young people´s health. A previous interview study with fifteen young people, aged 17 to 22 years, visiting five YHCs in two regions in central Sweden showed that an electronic assessment for young people should include questions about mental-, physical-, sexual health and social support, and have the potential for self-reflection and increased self-awareness .
With the previous study as a starting point, the aim of this study was twofold. First, to develop a collection of ePRO questionnaires; an Electronic Health Report Form (eHRF) prototype. The eHRF prototype should contain ePRO questionnaires in four health areas, to identify health and health-related problems in young people visiting YHCs. Second, the aim was to evaluate the usability of the eHRF prototype.
Materials and methods
The development of the eHRF prototype was based on the theoretical foundation  from a participatory research approach . For the current study, participation means the right of young people and healthcare professionals to influence decisions for the content and layout of an eHRF for the YHC . The development of an eHRF prototype was guided by biopsychosocial theory  and a holistic view of health. The concept of self-efficacy  was used to clarify the state of engagement for possible behavior change.
The study was approved by the Regional Ethics Committee, Uppsala, Sweden (dnr 2020 − 01921) and was performed according to the principles of the Declaration of Helsinki .
This study was conducted during autumn 2020 and consisted of two parts: (I) the development and (II) the usability evaluation of the eHRF prototype.
To develop an eHRF prototype for identifying health and health-related problems in young people visiting YHCs, a wide literature search was performed, based on a previous interview study .
The usability evaluation of the eHRF prototype was applied with a mixed-methods convergent design, using qualitative and quantitative data collection in a side-by-side approach and merging of data by discussion , as shown in Fig. 1.
Overall, a diverse group of participant characteristics was wanted, to include as many aspects of knowledge, expertise, and experience as possible, which is important in participation research . Young people had previously participated in identifying the different health areas to include in an eHRF for YHC , allowing the researchers’ perspective to assert itself in the development phase. The target number of participants for the development phase was an expert panel with eight researchers. For the usability evaluation, the participant target number was 12 young people and three YHC healthcare professionals.
eHRF prototype development participants
Participants in the development of the eHRF prototype were an expert panel and an Information Technology (IT) company (Fig. 2).
The expert panel members were purposively selected based on their professional and academic experience. The participating members had multi-professional clinical experience in the development of health technology and working with young people in mental healthcare, primary healthcare, and YHCs. Their academic degrees ranged from Master of Science to professor.
A collaboration with an IT-company contact person was included. The IT-company had experience in collaboration with several universities and healthcare regions in Sweden, offering solutions for a medical documentation system .
eHRF prototype usability evaluation participants
Participants in the eHRF prototype usability evaluation were young people, healthcare professionals and an expert panel (Fig. 3). A convenient sample of healthcare professionals and young people were included from the same selected YHC, situated in a small municipality in Central Sweden, taking approximately 25 appointments per week. The expert panel were affiliated with five regions in Central Sweden.
Inclusion criteria were: 16–23 years old, visiting the YHC, and being fully fluent in Swedish. Heterogeneity was desired in terms of gender identity, place of birth, living conditions, sexual orientation, and level of education. The young people were informed about the study by healthcare professionals during a planned visit to the YHC. They were asked about participation and to be contacted by the first author (PVL) for information on voluntary participation and confidentiality, registration in the IT system, and time for data collection.
All healthcare professionals working at the selected YHC participated in the study and received information about the study in an e-mail that included information on voluntary participation and confidentiality, and registration in the IT system. Appointments were booked for the data collection.
The purposively selected expert panel members were chosen using the same criteria used for the eHRF prototype development expert panel members. The experts were e-mailed information about the study and invited to a digital group meeting.
eHRF prototype development
The selection of PRO questionnaires for the eHRF prototype was in line with the COnsensus-based Standards for the selection of health Measurement Instruments (COSMIN) guidelines for selecting outcome measurements , presented in Fig. 4. A stepwise process started with determining the domains to be measured. Consensus for the health areas to focus on was reached within the research group, based on the previous interview study with young people ; physical health (including lifestyle habits), mental health (including questions about self-harm and suicide risk), sexual health and sexual experience and relationships/social support. The next step involved finding the PROs to include. This was based on a broad PubMed scope and internet search, performed by PVL in 2019. The search involved keywords and phrases, for instance, “health”, “health questionnaire”, eHealth, and “young people/youth/adolescent”. The Swedish website https://www.fbanken.se, a resource targeted at healthcare professionals and constructed to collect valid health questionnaires, primarily concerning mental health and social support, was also consulted for appropriate questionnaires. The references in all identified studies were explored. The expert panel’s clinical work and research experience contributed with suggestions for questionnaires to, and ideas for searching for new questionnaires to consider for the eHRF prototype.
An iterative analysis process took place amongst members of the expert panel. The panel discussed appropriate questionnaires for the eHRF prototype, based on validity and reliability. The PRO questionnaires were evaluated for reliability and validity, questionnaire length, target group and if translated into Swedish.
The iterative process helped to limit the selection of included PRO questionnaires and ended when consensus had been reached among the expert panel.
eHRF prototype construction
The selected PRO questionnaires were compiled in a paper version. The researchers PVL and AS constructed one question of self-efficacy for change in accordance with Bandura’s  work, with the recommended 100-poing Numerical Rating Scale (NRS) for responses.
To prepare the questionnaires for digitalization, PVL, ETA and MA coded the questionnaire items and response options in Microsoft Excel. The IT-company digitalized the questionnaires in close collaboration with PVL and ETA. The eHRF prototype was divided into sections based on content: physical health (including lifestyle habits), mental health, sexual health, and social support. Each PRO questionnaire started with the visual aid of an emoji, portraying its’ content. The emojis were all free to use from https://pixabay.com/sv/images/search/emojies/.
The digitalization process and the eHRF prototype functionality were tested by the expert panel in an iterative process and changes were made continuously until coherence was reached.
eHRF prototype usability evaluation
All participants were informed of the study conditions and gave consent before the interviews. The interviews were either in person at the YHC or digital and were performed individually except for the expert panel who participated in a digital group interview. Participants in digital interviews gained access to the eHRF through an SMS to their smartphones. Participants in face-to-face interviews gained access to a tablet, provided by PVL. The interviews followed a semi-structured interview guide (Table 1). PVL performed the interviews, which lasted 50–90 min. All interviews were recorded with an external voice recorder and transcribed verbatim. For details in qualitative data collection procedure, see Supplementary file 1.
System Usability Scale
The System Usability Scale (SUS)  is a 10-item scale (Table 2) applied to assess the users’ perceived usability of a product . A five-point Likert scale, ranging from Totally disagree  to Totally agree , was used for each item. The Swedish version SUS 1,4_sv  was used. The scale’s original phrase “system” has been changed to “eHRF prototype”.
The SUS scores can be ranged and converted to different grades of usability , to help interpretation of the scores. Values from 71.4 to 100 are viewed as “good usability” up to “best imaginable usability”. Values of 80 or more are considered above the average usability mean score .
Inductive qualitative content analysis was performed for the interviews . All transcribed interviews were read repeatedly for familiarization with the data. Throughout the process, dialog occurred between the researchers (PVL, ÅR, MM, ETA, AS). First, two transcripts were read and meaning units were coded separately by two of the authors (ÅR and PVL) with the purpose of ensuring agreement on the process and content of important aspects of the data. Comparison was made to strengthen credibility before proceeding with the analysis. Data describing the participants’ views on usability and ideas for improvements to the eHRF prototype were coded. The development of the categories was discussed and elaborated several times between the authors to establish equal content within the category and orthogonality between the categories (Table 3).
The SUS Likert scale scores for each item and participant will be presented and converted into a total value for each participant according to instructions .
Totally participated, in the eHRF prototype development, eight researchers with variety of academic experiences in the expert panel (Table 4) and one participant represented the IT-company.
In total 14 people participated in the eHRF prototype usability evaluation. Young people (n = 4) and healthcare professionals (n = 3) participated in both the qualitative and quantitative usability evaluation. Demographic information is presented in (Tables 5 and 6). The members of the expert panel, except the doctoral student (n = 7), participated in the qualitative usability evaluation.
The participants in the development and usability evaluation of the eHRF prototype are displayed in Fig. 5.
The eHRF prototype development
A total of 43 health questionnaires (17 physical health, 19 mental health, 3 sexual health, 4 social support) were identified for possible inclusion in the eHRF prototype. The iterative analysis process contributed to the selection of 10 questionnaires: consisting of 66 items, and seven background questions. The questionnaires the SEXual health Identification Tool (SEXIT) , the questions for the National Guidelines: Living Habits , and the questions for the National Guidelines: Living Habits, indicator questions for physical activity [36, 37] and the study-specific self-efficacy question were developed in Swedish  (for English versions, see Supplementary file 2. The questionnaires Alcohol Use Disorders Identification Test – Consumption (Audit-C) , Health Behavior in School-aged Children (HBSC) [39, 40], Generalized Anxiety Disorder 7-item scale (GAD-7) , Patient Health Questionnaire-9 (PHQ-9) , The SCOFF Questionnaire (acronym from the questions) , and Berlin Social Support Scales (BSSS) subscales “Need for support” and “Support seeking”  were available in Swedish (for English version see Supplementary file 2). The selected questionnaires had at least acceptable reliability and validity in its original language. However, some questionnaires were not tested for the YHC age group 12 to 23.
In total, the eHRF prototype consisted of 74 items (Table 7).
One PRO questionnaire, The Berlin Social Support Scale (BSSS)  was not previously translated to Swedish. Therefore, a translation process was performed. PVL and AS translated the English version of BSSS. The Swedish translation was sent to an American psychologist/researcher, fluent and living in Sweden, who translated the BSSS back to English. PVL and AS reviewed the translation and only made adjustments to a few items to improve comprehension in Swedish.
The eHRF prototype usability evaluation
The qualitative content analysis of the transcribed interviews resulted in three categories and seven subcategories. The categories are described below and displayed in Fig. 6. Throughout the results section the term “participants” is used when young participants, healthcare professionals, and the expert panel are included. A more detailed description of the categories and subcategories including citations is available (Supplementary file 3).
Captures overall health of young people but needs clarification
This category confirmed that the eHRF prototype health areas (mental-, physical-, and sexual health and social support) captured the overall health of young people. The young participants believed that the eHRF prototype had the potential to make young people aware of their health and possible need for behavior change. However, for young people who were not willing to change, the self-efficacy question of behavior change could be difficult to answer.
There were suggestions for improvements, for example remove questions on hight and weight and neutralize gender descriptions. Also, they suggested to focus more on family relations. The participants wanted a definition for “social support” and the term “behavioral change” changed to “lifestyle change” and a need for another word for “anxious”. The healthcare professionals found the self-efficacy behavior change question important and connected it to their experience of the challenge of having conversations about behavior change.
Fun, easy and optional will keep young people’s interest
This category stated that an easy-going and neutral eHRF prototype design, good layout, and structure could help young people answer the health questions. Although they liked the emojies in the eHRF, they suggested that the emojis could possibly be exchanged for other pictures or even colors or fonts. The expert panel and healthcare professionals, on the other hand, were hesitant or critical toward details in the layout, suggesting that the emojis used in the eHRF prototype could affect young people by portraying an emotion about the health area and questions it represented; they wanted the emojis to be changed to something more neutral, commenting that “less is more.” The participants found it essential that young people could choose to answer the health questions before the meeting. Both young participants and healthcare professionals wanted young people to be able to disregard questions that they were uncomfortable answering.
Potential contribution to improving the health consultation
The third category summarized that the eHRF prototype was found to bring something valuable to the health assessment conversation. The young participants thought that the eHRF would help them to answer honestly to sensitive questions. The healthcare professionals, on the other hand, hesitated on the questions’ sensitive topics and suspected that young people may not even complete the eHRF prototype questions. They also suspected that answering questions without knowing who they would meet at the YHC could make young people unwilling to use the eHRF prototype.
The young participants believed that answering the health questions could make them more focused on their health and help them prioritize which health areas that were more important. The healthcare professionals agreed with the young participants and said that the eHRF prototype questions and the young people´s answers could form solid ground for talking with young people about health and contributing to their professional evaluation.
The satisfaction usability scale
The individual and the sum of participants’ scores for each item of the SUS are shown in Tables 8 and 9. The participants claimed high agreement with all statements. The participants rated highly, the positive statements “I think I would like to use the eHRF prototype” and I would imagine that most people would learn to use this eHRF prototype very quickly”. They disagreed most with the negative statement “I think that I would need the support of a technical person to be able to use this eHRF prototype”. There were no missing items in the SUS responses.
The SUS scores were converted to grades of usability . The converted scores show that five participants’ scores concurred with the grade “best imaginable” (participants 3, 4, 5, 6, 7) and two participants’ scores were concurrent with the grade “good” (participants 1 and 2), as shown in Fig. 7.
This study had a participatory research approach to develop and evaluate an eHRF prototype with credible content and usability at YHCs. The current study was based on a previous interview study with young people at YHCs  that was the start of the development process, exploring of content and design of an eHRF for YHCs and hence, strengthening the young person’s perspective in this study. Due to ethical considerations, the youngest YHC visitors (12–15 years) were excluded. However, the participants in the eHRF evaluation represent the clinical YHC young- and healthcare professional population. Together with the expert panel participants, the group shows diversity in ages, knowledge, and experiences.
The search for and selection of PRO questionnaires in this development of the eHRF were mostly in line with the COSMIN guidelines . However, the literature search had limitations in structure and search terms, reducing the possibility to reproduce the search. The construction was strengthened by the previous interview study of young people  and the expert panel’s experience of health questionnaires. Including several stakeholders , and performing the development process in steps , are ways to assess content validity, described by Almanasreh, Moles and Chen . The researchers developed the eHRF prototype from ten questionnaires [28, 33,34,35,36,37,38,39,40,41], in total 74 items including background questions and a self-efficacy question. Self-efficacy questions have potential to increase the understanding of young individuals’ ability to change behavior to improve health , and occur in PROs/ePROs for young people.
The evaluation of the eHRF prototype usability consisted of qualitative interviews and a quantitative usability questionnaire. The combination of the qualitative and quantitative data was assumed to provide additional perspectives and a more complete understanding  of the usability of the eHRF prototype. Overall, the results indicated that the eHRF prototype was usable for YHCs. In summary, there were positive correlations between qualitative and quantitative data, indicating that the eHRF prototype was easy to understand and found easy to use. The qualitative data showed that the young participants were also positive toward the eHRF prototype possibility to increase their understanding of health. This is supported by the previous interview study, stating that answering meaningful health questions in an eHRF prototype could potentially lead to self-reflection and increased self-awareness . Kutcher et al.  has highlighted the importance of increasing mental health literacy, i.e., how well individuals can understand, and communicate about health-related information for making informed health decisions (page 16) . Same-level health conversations between young person and healthcare professional may act as the starting point for behavior change to improve health because it involves self-reflection as well as information for the consultation with the healthcare provider that will help determine suitable interventions . In young people, health technology, has been used to identify psychosocial issues [11,12,13] and improving health equity . Additionally, young people have pointed out the importance of electronic health questionnaires for time-efficient health assessments, directed toward what is essential to the young person .
The young participants thought that responding to health questions before meeting with a healthcare professional could make it easier to honestly answer sensitive questions. This has found also, by Thabrew et al. . However, the healthcare professionals in the current study were concerned that the questions on sensitive topics may make it hard for young people to answer honestly. This was also found in a qualitative study that interviewed healthcare professionals at YHCs for their experiences of using SEXIT . This may implicate a need for training healthcare professionals to talk about sensitive topics.
The qualitative findings highlight the importance of an appealing design, to keep young people’s interest and help them complete the eHRF prototype health questions, also supported by the previous interview study . In the current study the layout and design of the eHRF prototype was appreciated by the young people, and the emojis viewed as refreshing and fun. This has been found important to improve usability and user satisfaction in guiding clinical decisions . However, healthcare professionals and expert panel disagreed to some of the emojies.
The study had several limitations. The target number of participants for the usability evaluation was not reached. There were in total fourteen participants in the usability evaluation (four young people, three healthcare professionals, seven members of the expert panel). The few young people included reflects that the estimated number of appointments were even lower, due to the COVID-19 pandemic restrictions, and the water leak causing poor environment at the YHC. Another study limitation was that only seven participants responded to the SUS questionnaire. For a significant outcome, the SUS questionnaire requires at least eight participants . However, for early usability evaluations, five participants have been found sufficient to identify usability issues . An additional weakness is that all participants were female. Although a study limitation, this is reflective of the YHC patient population, supported by surveys, showing that almost 90% of YHC visitors are female . All considered, population in the current study should be satisfactory for the usability evaluation in the YHC setting. Another limitation was that inconsistent records were kept of the young people who declined to participate, due to the clinical pre-requisites at the YHC. Finally, the ongoing COVID-19 pandemic during the study period demanded adaptions to the study procedure, for example digital interviews. This may have affected the candor of the participants’ responses to the SUS.
This study acknowledges that research needs to serve the society and its current context . Hence, it was pragmatically designed to allow clinical and research solutions and to involve the future users, i.e., young people and healthcare professionals. The eHRF developed in this study, consisting of several PROs (traditionally non-digital), has not been validated. However, no changes were made to the PROs when converted digitally, indicating that the need for validation is lower . Also, ePROs provide better data quality, decrease costs, and facilitate clinical symptom management and decision-making compared to PROs . Still, if implementing the eHRF prototype for clinical use at YHCs, future research is needed to validate the eHRF prototype for ages 12–15 and for other healthcare settings. This eHRF has potential to provide a structured and unified assessment of young people’s health. This may reduce the risk for health inequities among young people .
The next research step is to further develop the eHRF prototype in considering the improvement suggestions from the participants, for example removing some questions and exchanging emojies. After improving the eHRF, a feasibility study (protocol ISRCTN23855544) is planned to evaluate feasibility aspects (process, resources and management), , before performing a fourth-coming Stepped Wedge Cluster Randomized Trial (SW-CRT).
This study presents the results from the development and early usability testing of an eHRF using a participatory approach. The collaboration of an expert panel and an IT company resulted in the development of an eHRF prototype. It was based prior findings of young people’s opinions on important health areas and valid and reliable PRO questionnaires. The eHRF reflects the biopsychosocial perspective, including self-efficacy for behavior change as a shared base for a health conversation between healthcare professional and young people at YHCs. The usability evaluation showed that the eHRF prototype was usable, could lead to self-reflection and cooperation between young people and healthcare professionals during the health consultation. One limitation to consider was the potential effect of adjusting face-to-face interviews to digital. Suggested improvements need to be considered for further eHRF development. Implication of research are on implementation barriers and facilitators within a feasibility study.
Deidentified data may be available from the corresponding author upon request and subject to General Data Protection Regulation (GDPR) and the Swedish Ethical Review Authority requirements.
Electronic Patient-Reported outcome
Youth Health Clinic
Sexual health Identification Tool
World Health Organization
Electronic Health Report Form
- IT- company:
Information Technology company
Numerical Rating Scale
Satisfaction Usability Scale
short message service
Alcohol Use Disorder Identification Test
Health Behavior in School-aged Children
General Anxiety Disorder 7-item scale
Patient Health Questionnaire-9
acronym from the five questions in the SCOFF questionnaire
Berlin Social Support Scale
General Data Protection Regulation
Meirte J, Hellemans N, Anthonissen M, Denteneer L, Maertens K, Moortgat P, et al. Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med. 2020;3(1):e15588.
Coons SJ, Gwaltney CJ, Hays RD, Lundy JJ, Sloan JA, Revicki DA, et al. Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO Good Research Practices Task Force report. Value Health. 2009;12(4):419–29.
Sharma A, Harrington RA, McClellan MB, Turakhia MP, Eapen ZJ, Steinhubl S, et al. Using Digital Health Technology to Better Generate evidence and deliver evidence-based care. J Am Coll Cardiol. 2018;71(23):2680–90.
Riis CL, Stie M, Bechmann T, Jensen PT, Coulter A, Möller S et al. ePRO-based individual follow-up care for women treated for early breast cancer: impact on service use and workflows. J Cancer Surviv. 2021.
U.S.Food and Drug Administration. Guidance for industry. Patient reported outcome meaures: use in medical product development to support labelling claims. 2009 updated 10/17/2019. Fed Regist 2009;74:65132-3.:[Available from: https://www.fda.gov/media/77832/download.
Acquadro C, Berzon R, Dubois D, Leidy NK, Marquis P, Revicki D, et al. Incorporating the patient’s perspective into Drug Development and Communication: an ad Hoc Task Force Report of the patient-reported outcomes (PRO) Harmonization Group Meeting at the Food and Drug Administration, February 16, 2001. Value in Health. 2003;6(5):522–31.
Staniszewska S, Adebajo A, Barber R, Beresford P, Brady L-M, Brett J, et al. Developing the evidence base of patient and public involvement in health and social care research: the case for measuring impact. Int J consumer Stud. 2011;35(6):628–32.
Haywood KL. Patient-reported outcome II: selecting appropriate measures for musculoskeletal care. Musculoskelet Care. 2007;5(2):72–90.
Trujols J, Portella MJ, Iraurgi I, Campins MJ, Siñol N, de Los Cobos JP. Patient-reported outcome measures: are they patient-generated, patient-centred or patient-valued? J Ment Health. 2013;22(6):555–62.
Goodyear-Smith F, Corter A, Suh H. Electronic screening for lifestyle issues and mental health in youth: a community-based participatory research approach. BMC Med Inform Decis Mak. 2016;16(1):140.
Thabrew H, Kumar H, Goldfinch M, Cavadino A, Goodyear-Smith F. Repeated psychosocial screening of High School Students using YouthCHAT: Cohort Study. JMIR Pediatr Parent. 2020;3(2):e20976.
Thabrew H, Goodyear-Smith F. Youth version, case-finding and help Assessment Tool (YouthCHAT): an electronic, psychosocial screener for young people with long-term physical conditions. J Paediatr Child Health. 2019;56(1):102–6.
Martel RM, Darragh ML, Lawrence AJ, Shepherd MJ, Wihongi T, Goodyear-Smith FA. YouthCHAT as a primary Care E-Screening Tool for Mental Health Issues among Te Tai Tokerau Youth: protocol for a Co-Design Study. JMIR Res protocols. 2019;8(1):e12108.
Goodyear-Smith F, Arroll B, Coupe N. Asking for help is helpful: validation of a brief lifestyle and mood assessment tool in primary health care. Ann Fam Med. 2009;7(3):239–44.
Hammarstrom S, Nilsen P, Lindroth M, Stenqvist K, Bernhardsson S. Identifying young people exposed to or at risk of sexual ill health: pilot implementation of an evidence-informed toolkit (SEXIT) at swedish youth clinics. Eur J Contracept reproductive health care: official J Eur Soc Contracept. 2019;24(1):45–53.
Hammarström S, Lindroth M, Nilsen P, Nolskog P, Bernhardsson S. Staff’s experiences of a pilot implementation of the SEXual health identification Tool for assessing sexual ill health among visitors to swedish youth clinics: a focus group study. Sex Reprod Healthc. 2021;29:100643.
Higginbottom G, Liamputtong P, Participatory Qualitative Research Methodologies in Health. 2015 2023/02/07. 55 City Road 55 City Road, London: SAGE Publications Ltd. Available from: https://methods.sagepub.com/book/participatory-qualitative-research-methodologies-in-health.
Robards F, Kang M, Luscombe G, Sanci L, Steinbeck K, Jan S, et al. Predictors of young people’s healthcare access in the digital age. Aust N Z J Public Health. 2019;43(6):582–8.
Engel GL. The need for a new medical model: a challenge for biomedicine. Science. 1977;196(4286):129–36.
World Health Organization. Basic documents of the World Health Organization 49 ed. New York, NY.: World Health Organization; 2020. (n.d.).
Lostelius P, Mattebo M, Söderlund A, Revenäs Ã, Thors Adolfsson E. An electronic patient-reported outcome created based on my needs is worth using: an explorative qualitative study investigating young people’s opinions for a health assessment tool. J Patient-Reported Outcomes. 2022;6(1):29.
DeVellis RF. Scale development: theory and applications. United States pf America. SAGE Publications, Inc.; 2017.
Higginbottom G, Liangputtom P, Participatory Qualitative Research Methodologies in Health. 2015 2022/05/17. 55 City Road, London: SAGE Publications Ltd. Available from: https://methods.sagepub.com/book/participatory-qualitative-research-methodologies-in-health.
Sverker A, Thyberg I, Valtersson E, Björk M, Hjalmarsson S, Östlund G. Time to update the ICF by including socioemotional qualities of participation? The development of a “patient ladder of participation” based on interview data of people with early rheumatoid arthritis (the swedish TIRA study). Disabil Rehabil. 2020;42(9):1212–9.
Bandura A. The primacy of self-regulation in Health Promotion. Appl Psychol. 2005;54(2):245–54.
The World Medical Association I. WMA Declaration of Helsinki - Ethical principles for medical research involving humab subjects 1964 updated 2021. Available from: https://www.wma.net/policies-post/wma-declaration-of-helsinki-ethical-principles-for-medical-research-involving-human-subjects/.
Creswell JW, Creswell JD. Research Design. Qualitative, quantitative & mixed methods approaches. 5th ed. London, United Kingdom: SAGE Publications Ltd.; 2018.
Page K. Ethics and the co-production of knowledge. Public Health Res Pract. 2022;32(2).
Entermedic. Available from: https://www.entermedic.com/.
Prinsen CAC, Vohra S, Rose MR, Boers M, Tugwell P, Clarke M, et al. How to select outcome measurement instruments for outcomes included in a “Core Outcome Set” – a practical guideline. Trials. 2016;17(1):449.
Bandura A, Guide for constructing Self-Efficacy Scales. Self-efficacy beliefs of adolescents [Internet]. Greenwich, CT, USA.: Information Age Publishing INC; 2005.
Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan P, Thomas B, Weerdmeester B, McClelland, editors. I, redaktörer. Usability evaluation in industry. Volume I. London: Taylor & Francis; 1996. pp. 189–94.
Lewis JR. The System Usability Scale: past, Present, and Future. Int J Hum Comput Interact. 2018;34(7):577–90.
Göransson B. SUS Swedish. Rosenfeld media; 2001–2011. Available from: https://rosenfeldmedia.com/wp-content/uploads/2020/09/SUS-svensk.pdf.
Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24(2):105–12.
Socialstyrelsen. Nationella riktlinjer levnadsvanor. Available from: https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/dokument-webb/nationella-riktlinjer/levnadsvanor-fragor-om-levnadsvanor.pdf.
Kallings L. Validering av Socialstyrelsens screeningfrågor om fysisk aktivitet. Stockholm: Gymnastik- och idrottshögskolan Available from: https://www.socialstyrelsen.se/globalassets/sharepoint-dokument/dokument-webb/nationella-riktlinjer/levnadsvanor-validering-av-indikatorfragor-till-patienter-om-fysisk-aktivitet.pdf.
Liskola J, Haravuori H, Lindberg N, Niemelä S, Karlsson L, Kiviruusu O, et al. AUDIT and AUDIT-C as screening instruments for alcohol problem use in adolescents. Drug Alcohol Depend. 2018;188:266–73.
King A, Wold B, Tudor-Smith C, Harel Y. The health of youth. A cross-national survey. WHO Reg Publ Eur Ser. 1996;69:1–222.
Haugland S, Wold B. Subjective health complaints in adolescence–reliability and validity of survey methods. J Adolesc. 2001;24(5):611–24.
Spitzer RL, Kroenke K, Williams JB, Lowe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. 2006;166(10):1092–7.
Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606–13.
Morgan JF, Reid F, Lacey JH. The SCOFF questionnaire: assessment of a new screening tool for eating disorders. BMJ (Clinical research ed). 1999;319(7223):1467–8.
Schulz U, Schwarzer R. Soziale Unterstützung bei der Krankheitsbewältigung. Die Berliner Social Support Skalen (BSSS) [Social support in coping with illness: the Berlin Social Support Scales (BSSS)]. Diagnostica. 2003;49:73–82.
Rubio DM, Berg-Weger M, Tebb SS, Lee ES, Rauch S. Objectifying content validity: conducting a content validity study in social work research. Social Work Research. 2003;27(2):94–104.
Almanasreh E, Moles R, Chen TF. Evaluation of methods used for estimating content validity. Res Social Administrative Pharm. 2019;15(2):214–21.
Kutcher S, Wei Y, Costa S, Gusmão R, Skokauskas N, Sourander A. Enhancing mental health literacy in young people. Eur Child Adolesc Psychiatry. 2016;25(6):567–9.
Berkman ND, Davis TC, McCormack L. Health literacy: what is it? J Health Communication. 2010;15(sup2):9–19.
Glanz K, Rimer BK, Viswanath K. Health behavior and health education: theory, research and practice. San Francisco: Jossey-Bass; 2015.
Graham GN, Ostrowski M, Sabina AB. Population health-based approaches to utilizing digital technology: a strategy for equity. J Public Health Policy. 2016;37(Suppl 2):154–66.
Bradford S, Rickwood D. Acceptability and utility of an electronic psychosocial assessment (myAssessment) to increase self-disclosure in youth mental healthcare: a quasi-experimental study. BMC Psychiatry. 2015;15:305.
Thabrew H, D’Silva S, Darragh M, Goldfinch M, Meads J, Goodyear-Smith F. Comparison of YouthCHAT, an electronic Composite Psychosocial Screener, with a clinician interview Assessment for Young People: Randomized Controlled Trial. J Med Internet Res. 2019;21(12):e13911.
Payne TH, Hines LE, Chan RC, Hartman S, Kapusnik-Uner J, Russ AL, et al. Recommendations to improve the usability of drug-drug interaction clinical decision support alerts. J Am Med Inform Assoc. 2015;22(6):1243–50.
Nielsen J, Landauer TK. A mathematical model of the finding of usability problems. Conference on Human Factors in Computing Systems; Amsterdam, The Netherlands: Association for Computing Machinery; 1993. p. 206–13.
Sveriges Kommuner och Regioner (SKR). Ungdomsmottagningar och ungas sexuella hälsa. Nuläge och vägen framåt. ; 2020. Report No.: ISBN 978-91-7585-880-7.
Alvesson M, Sköldberg K. Reflexive methodology: new vistas for qualitative research. 3rd ed. London: Sage publications; 2017.
Marmot M. Achieving health equity: from root causes to fair outcomes. Lancet (London England). 2007;370(9593):1153–63.
Thabane L, Ma J, Chu R, Cheng J, Ismaila A, Rios LP, et al. A tutorial on pilot studies: the what, why and how. BMC Med Res Methodol. 2010;10:1.
Selin KH. Alcohol Use Disorder Identification Test (AUDIT): what does it screen? Performance of the AUDIT against four different criteria in a swedish population sample. Subst Use Misuse. 2006;41(14):1881–99.
Hansson E, Daukantaité D, Johnsson P. SCOFF in a general swedish adolescent population. J Eat Disord. 2015;3:48.
We thank all participants in this study for their contributions to the project: young people, healthcare providers, and experts, as well as our contact at the IT company.
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Region Västmanland (LTV 939219) and the Regional Research Council (RFR 939311).
Swedish Research Council for Health, Working Life and Welfare (FORTE) dnr 2018 − 01127, Regional Research Council Mid Sweden (RFR 939665).
The authors declare that they have no competing interests.
Ethics approval and consent to participate
The study was approved by the Regional Ethics Committee, Uppsala, Sweden (dnr 2020 − 01921) and carried out according to the Declaration of Helsinki . The participants were provided with verbal and written study information and granted informed consent.
Consent for publication
The study does not contain any detailed personal data from participants.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Lostelius, P.V., Mattebo, M., Adolfsson, E.T. et al. Development and usability evaluation of an electronic health report form to assess health in young people: a mixed-methods approach. BMC Med Inform Decis Mak 23, 91 (2023). https://doi.org/10.1186/s12911-023-02191-7
- Developmental study
- Electronic patient-reported outcome
- Mixed methods research
- Usability study
- Medical informatics
- Participatory research
- Young people
- Youth health clinic