Development an extended-information success system model (ISSM) based on nurses’ point of view for hospital EHRs: a combined framework and questionnaire

Background Understanding the hospital EHR success rate has great benefits for hospitals. The present study aimed to 1-Propose an extended-ISSM framework and a questionnaire in a systematic manner for EHR evaluation based on nurses’ perspectives, 2-Determine the EHR success rate, and 3-Explore the effective factors contributing to EHR success. Methods The proposed framework was developed using ISSM, TAM3, TTF, HOT-FIT, and literature review in seven steps. A self-administrated structured 65-items questionnaire was developed with CVI: 90.27% and CVR: 94.34%. Construct validity was conducted using EFA and CFA. Eleven factors were identified, collectively accounting for 71.4% of the total variance. In the EFA step, 15 questions and two questions in EFA were excluded. Finally, 48 items remained in the framework including dimensions of technology, human, organization, ease of use, usefulness, and net benefits. The overall Cronbach’s alpha value was 93.4%. In addition, the hospital EHR success rate was determined and categorized. In addition, effective factors on EHR success were explored. Results In total, 86 nurses participated in the study. On average, the “total hospital EHR success rate” was moderate. The total EHR success rates was ranging from 47.09 to 74.96%. The results of the Kruskal–Wallis test showed that there was a significant relationship between “gender” and “self-efficacy” (p-value: 0.042). A reverse relation between “years of experience using computers” and “training” (p-value: 0.012) was observed. “Years of experience using EHR” as well as “education level” (p-value: 0.001) and “ease of use” had a reverse relationship (p-value: 0.034). Conclusions Our findings underscore the EHR success based on nurses’ viewpoint in a developing country. Our results provide an instrument for comparison of EHR success rates in various hospitals. Supplementary Information The online version contains supplementary material available at 10.1186/s12911-022-01800-1.

decades, the e-health including EHR has become more popular among hospitals in Iran. SEPAS is the national hospital-based EHR for all Iranian citizens. The aim of this system is to connect hospitals and medical centers and to aggregate medical information at all levels of the healthcare system. All hospitals in Iran should adopt an EHR system and send inpatient and outpatient data to SEPAS [3]. Howerver, hospital EHRs have particular complexities [4]. Many studies suggest that EHR implementation alone is insufficient to effectively improve healthcare quality [5]. As such, various factors contribute to the successful implementation and use of EHR [6]. For example, human factors such as resistance to change [1,7], low perceived uselessness [1], time and resource constraints, suboptimal clinic workflows, information access limitations, and insufficient clinician training [8] were the most frequent barriers regarding adoption of EHR in several studies. For the EHR systems to be successfully implemented and their potential impacts realized, it is essential that human and organizational processes are understood by users and involved in motivating change and adoption [9]. The interactive dimensions of the context, content, and process can shape organizational changes in hospitals and achieve the success of EHR implementation in hospitals depending on the effective interaction of the aforementioned dimensions [4]. The findings of the study by Ojo and Popoola revealed a highly significant relationship between technical, social, organizational, financial, and political factors and EHR success in hospitals [10]. These results were supported by the results of the Ciccomoello et.al study. Users and technology can collectively form the system and influence its acceptance and adoption. They believed that the expressed needs, the involvement of different users, and assessing system impacts on user's point of view were key factors affecting EHR adoption [9]. Moreover, the results of the study by Handayani et al. showed that non-technological dimensions, such as human and organizational dimensions, were more effective on EHR success than technological dimensions [11]. Therefore, user characteristics and business environment are important factors in adoption of EHRs.
Nurses are the largest user group of EHRs and provide health services in every clinical environment including hospitals [12]. EHR has great advantages for nurses in hospitals such as avoiding missing data, making nurses aware of the importance of documentation as a data source, adopting the use of evidence-based tools, avoiding customization, and seeking ways to use nursing documentation for research and quality measurement [13]. Rudin et al. suggested four EHR benefits for nurses including better clinical decisions, better triage decisions, better collaboration, and automation of tasks [14]. However, despite the expected benefits of EHR systems, the adoption of these system is different among nurses [15]. Krick and Tobias believed that the reasons for the lack or low acceptance of EHRs by nurses can be due to low usability of the system, no visible benefits for real work practice, as well as privacy issues. Scientific evaluations that offer the implementation status of the systems from different users' perspectives could help to understand the bigger picture of digital nursing technologies' success and provide important insights on specific impact factors. Comprehensive evaluation frameworks clearly show vital aspects of evaluation and play a significant role in supporting researchers, decision-makers, and developers. Evaluation frameworks can be used to provide a structure for the evaluation of nursing systems as well as information and definitions of technology success, evaluation areas, methods, and tools. Accordingly, an evaluation framework can facilitate a systematic approach in nursing systems evaluation [16]. In addition, the results of a literature review suggested that a modification of the existing frameworks may provide a better explanation of nurses' acceptance and success of EHRs [12]. Hence, developing an EHR evaluation framework and understanding of the hospital EHR success rate based on nurses' points of view has great benefits for hospitals about EHR adoption. The first objective of this research was to propose an extended-ISSM framework and a questionnaire in a systematic manner for EHR evaluation based on nurses' perspectives. The second purpose was to determine the EHR success rate. The final goal was to explore the effective factors on EHR success in a hospital as a case study.

Research framework
The research framework was developed in the following seven steps (Fig. 1).

Step 1: Development primary version of EHR framework and duplication screening
The proposed framework was developed mainly based on the Information Systems Success Model (ISSM) [17], Technology Acceptance Model (TAM2) [18], Task Technology Fit (TTF) [19], Human Organization Technology-Fit (HOT-FIT) [20], and our published literature review [21].
The ISSM is a well-validated evaluation framework and has seven dimensions. ISSM was developed by DeLone and Mclean [22] and was subsequently updated [17]. ISSM encompassed seven dimensions: system quality, information quality, service quality, system use, intention to use, satisfaction, and net benefits. System success depends on the interaction between its dimensions [17]. TAM was developed by Davis et al. in 1989 to predict users' adaptation and use of new technology.
It proved that individuals' behavioral intention to use new technology is determined by two beliefs: perceived usefulness and perceived ease of use [23]. Venkatesh and Davis [18] proposed an extension of TAM known as TAM2 . The dimensions of TAM3 were as follows: perceived usefulness, perceived ease of use, computer self-efficacy, perceptions of external control computer, playfulness, computer anxiety, perceived enjoyment, subjective norm, voluntariness, image, job relevance, output quality, result demonstrability, behavioral intention, and use. Goodhue and Thompson suggested TTF in 1995. They believed that consistency between technology and tasks can increase users' performance. TTF encompasses four dimensions including task characteristics, technology characteristics, task-technology fit, and performance/utilization [19]. HOT-FIT was developed by Yusof et al. in 2008 based on ISSM and IT-Organization Fit Model [24]. Our literature review was conducted to identify EHR evaluation frameworks that identified 110 evaluation measures to assess EHR from 62 eligible studies. In addition, 53 evaluation measures were identified from the selected frameworks including ISSM, TAM 3, TTF, and HOT-FIT. In the first step, the total identified evaluation measures (n = 163) were saved in an excel file. In this step, duplicate evaluation measures in each dimension were identified (n = 43). Subsequently, 120 evaluation measures were retrieved in six scopes in the following dimensions. Additional file 1 (the list, relative frequency, and the final status of the evaluation measures) shows the list, relative frequency, and the final status of each of the evaluation measures. The categorization of dimensions was based on our literature review.
• Technology: System quality, information quality, and service quality. • Human: Satisfaction, system use, computer knowledge and self-efficacy, users' characteristics and personality, and positive or negative feeling about EHR.  • Net benefits: Effects on outcome quality of care, effects on workflow and organization, and effects on the environment.

Step 2: Frequency analysis
Some evaluation measures were extracted from only one, two, or three studies. It is not possible for all evaluation measures to be included in an evaluation framework, because it can lead to low integrity and practicality of the framework. Hence, in the second step, frequency analysis was conducted. The evaluation measures with frequency = < 5% (n < = 3) from the literature review were excluded from the framework (n = 59).

Step 3: Face validity
Since the evaluation measures were collected from various resources, an expert panel team assessed the 61 remaining evaluation measures to merge them with the common concept. In the third step, 26 evaluation measures were merged. For example, the evaluation measure of "accessibility" in the dimension of "system quality" was merged with "sufficient resources". The evaluation measures including "usability" and "confusion" in system quality dimension were merged with dimensions of "ease of use". The list of merged evaluation measures are shown in Additional file 1.

Step 4: Content validity
In the fourth step, a self-administrated structured 65-items questionnaire that covered the 35 remaining evaluation measures were developed as follows (Additional file 2: Extended ISSM with a 65-items questionnaire for hospital EHRs based on nurses' point of view validated by expert panel before Confirmatory Factor Analysis (CFA)): • Technology (number of questions = 24): • System quality: sufficient resources, reliability, availability, system interoperability and integration with other information systems, response time. • Information quality: privacy and security, up-todate, sufficiently, format, locatability, accuracy right level of detail, authorization, and timeliness. • Service quality: empathy, responsiveness, assurance, responsiveness and training.
• Organization (number of questions = 9): Management support, social support, and Task Technology Fit (TTF) and environment. • Human (number of questions = 14): Self-efficacy, positive or negative feeling about EHR (including computer anxiety, result demonstrability, perceived enjoyment), users' satisfaction, system use, voluntariness, image, and job relevance. Eleven experts participated in the expert panels including: three nurses, three medical informatics specialists, two health information management specialists, and three hospital HIS managers. Each expert had a long experience in the use of hospital EHR. Content validity ratio (CVR) and Content Validity Ratio (CVR) were calculated for each item by an expert panel. A CVR of zero or greater indicates that at least half of the experts deemed the item as "Essential" for the construct assessment [25]. An expert panel validated the questionnaire with CVI: 90.27% and CVR: 94.36%. No evaluation measure was excluded in this step.

Step 5: Construct validity with exploratory factor analysis (EFA)
In the fifth step, the validated 65-items questionnaire by the expert panel was used to construct validity, which was carried out using Exploratory Factor Analysis (EFA). A 5-point Likert scale ranging from "completely disagree" to "completely satisfy" was mainly used.
Data gathering was conducted in the largest psychiatric hospital and education center in eastern Iran. The hospital EHR has implemented in the case hospital ten years ago. Given the fact that human participants were involved in the current study, all methods were conducted based on the ethical standards of the Ethical Committee of Mashhad University of Medical Sciences. Scale development in factor analysis (EFA, CFA) are large sample size method because sample size affects precision and replicability of the results [26]. Therefore, all registered nurses in the EHR database who worked in the case hospital were invited to participate in the study (n = 112). The purpose of the study was explained and participants were assured that their confidentiality would be maintained. Participation in this study was voluntary, and the participants could withdraw from the study at any time. Questionnaires were provided to users who agreed to participate in this study. The written consent was obtained from all participants. The EFA was performed using Principal Component Analysis (PCA). The Kaiser-Meyer-Olkin (KMO) was conducted to measure sampling adequacy, and Bartlett's Test of Sphericity was applied to assess whether factor analysis is appropriate. Varimax rotation was performed to identify the uncorrelated factors. The factor's extraction was consistent with the eigenvalue > 1 rule. Since only items with a loading factor ≥ 0.3 are acceptable for a specific factor, the threshold of factor loading was considered as 0.3 or greater [27,28]. In this step, fifteen questions with KMO equal to or less than 0.3 and multiple factor loadings were excluded that covered nine evaluation measures, and 50 questions that belonged to 26 evaluation measures remained in the framework and were categorized based on results of EFA. Three evaluation measures of system quality including "reliability", "availability", and "system interoperability and integration with other information systems", as well as two evaluation measures of authorization, and timeliness in information quality dimension were excluded from the framework. "Training" with two questions created a factor and was separated from "service quality". The evaluation measures including "system use", "voluntariness", "image", and "job relevance" were categorized with "performance expectancy" named usefulness dimension. In addition, evaluation measures of "positive or negative feeling about EHR" dimension including computer anxiety, result demonstrability, and perceived enjoyment as well as users' satisfaction were excluded from the human scope. One question in dimension of ease of use had multiple factor loading and was excluded.
The evaluation measure of 'environment" in organization dimension was excluded from the framework and evaluation measure of "task equivocality", "task interdependence", and "compatibility and fitness with the work process" were grouped together and created the factor of TTF. Three evaluation measures encompass "effects on outcome quality of care", "effects on workflow and organization", and "privacy and security" were grouped as a factor that was named "net benefit".

Step 6: Construct validity with confirmatory factor analysis (CFA)
In the sixth step, after conducting the EFA and excluding the questions with KMO equal or less than 0.3 and multiple factors, a Confirmatory Factor Analysis (CFA) was performed on the entire set of remaining items simultaneously (26 evaluating measures and 50 questions) to confirm the framework. CFA represents a powerful statistical technique used to determine whether the number of factors and pattern of item-factor loadings is consistent with what would be expected by a priori theory [29]. The method of CFA was the same as EFA. The results of the CFA showed that eleven factors with eigenvalues greater than 1.00 were identified, jointly accounting for 71.4% of the total variance. The value of KMO was 0.774, indicating sampling adequacy for the factor analysis. Bartlett's test of Sphericity was statistically significant (p-value < 0.000). Out of 50 questions, the factor loading of one question was lower than 0.3, which was not excluded from the questionnaire by the research team. Two questions in information quality and social support dimensions had multiple factor loading removed from the questionnaire. Finally, 48 items remained in the study. According to the results of CFA, eleven factors were identified in the extended-ISSM questionnaire. Four factors including computer resource, information quality, service quality, and net benefits completely confirmed the original ISSM dimensions. Seven dimensions including training, task technology fit, social support, top management support, self-efficacy, ease of use, and usefulness were identified and considered as influencing factors on EHR success. Dimension of the "satisfaction" that is one of the ISSM dimensions was excluded in EFA. The 48-item questionnaire included six scopes as follows (Additional file 3: Extended ISSM with a 50-items questionnaire for hospital EHRs based on nurses' point of view validated by Exploratory Factor Analysis (EFA)): • Technology • System quality: sufficient resources (factor 1, n = 3). • Information quality (factor 2, n = 7): up-to-date, sufficiently, format, locatability, accuracy, and right level of detail. • Service quality (factor 3, n = 5): empathy, responsiveness, assurance, responsiveness. • Training (factor 4, n = 2) • Organization • Management support (factor 5, n = 2) • social support (factor 6, n = 2) • Human: Self-efficacy (factor 8, n = 2). • Ease of use (factor 9, n = 3) • Usefulness (factor number 10, including evaluation measures of "performance expectancy", "system use", "voluntariness", "image, and "job relevance", n = 8). • Net benefits (factor number 11 including evaluation measures of "effects on outcome quality of care", "effects on workflow and organization", and "privacy and security", n = 11). Step 7: Reliability of the framework In the seventh Step, the reliability of the 48-item questionnaire was measured by Cronbach's alpha. Table 1 shows the dimensions and questions of the proposed questionnaire. The overall Cronbach's alpha value of the instrument was determined as 93.40%, demonstrating high reliability. The value of Cronbach's alpha was very high among the six dimensions of computer resource, information quality, service quality, training, ease of use, usefulness, and net benefits. The range of values of Cronbach's alpha in all dimensions was from 40 to 91.7%. The low value of Cronbach's alpha belonged to the selfefficacy dimension, and its value was 40.0%. Table 1 and Additional file 3 show all items of the proposed framework including remaining and excluded items.

Statistics
Participants' responses were tabulated and scores for invert statements were reversed. No statistical imputation was performed for missing data. Descriptive summary statistics were calculated as frequencies and percentages for demographics. Means rate and 95% Confidence Interval (CI) of hospital EHR success for the dimensions of extended-ISSM were calculated based on nurses' perspectives. Hospital EHR success rate in the case hospital was calculated in three steps. First, the rate of hospital EHR success based on nurses' point of view was determined for each of the evaluation measures and then total dimensions of the extended-ISSM questionnaire by all nurses. Finally, the mean of the hospital EHR success rate was categorized as follows: 1-Appropriate (75% ≤ hospital EHR success rate), 2-Moderate (50% ≤ hospital EHR success rate < 75%), 3-Low (25% ≤ hospital EHR success rate < 50%), and 4-Poor (coverage rate < 25%). For more details about the hospital EHR success rate measurement, see our previous papers [30,31].
The data distribution of EHR success dimensions for different levels of effective factors on each dimension was determined using the Shapiro-Wilk test of normality and Kolmogorov-Smirnov. The dependent variables were dimensions of extended-ISSM and independent variables were age (30 > / 30-40/ 40 <), gender, educational level (bachelor/master), ICDL certification (yes/no), nursing status (nurses/ head nurses), number of shifts per day (1/2), years of work experience (< 5/ 6-10/ > 10), years of experience using computers (< 3/ ≥ 3), and years of experience using EHR (< 6/ ≥ 6). As in all dimensions of extended-ISSM were found to have a nonnormal distribution, comparison tests were conducted using Mann-Whitney U tests for two-level variables and Kruskal-Wallis for three or more level variables. A p-value of less than 0.05 was considered statistically significant. Non-normally distributed data were reported as medians and Interquartile Ranges (IQR) in the current study. Data analysis was performed using SPSS, version 26 statistical software.

Participants
A total of 112 questionnaires were distributed among all invited nurses. Finally, 86 valid questionnaires were collected (Response rate 76.7%). The participants were 68 nurses and 18 head nurses. Fifty-one of the participants were female, and 35 were male. Most of the cases (n = 49, 57.0%) were aged 31-40. Eighteen cases were 30 years old, and nineteen were over 40 years old. In addition, 80.2% of nurses had Bachelor's degree and 18.8% had a Master's degree.
Most of the participants (n = 35, 40.7%) had more than ten years of work experience. Moreover, 38.4% and 20.9% of the participants had 5-10 and less than 5 years of work experience, respectively. Fifty-five participants had one shift per day. The majority of the participants (n = 71, 82.6%) had more than three years of experience to work with computer, and also, had over six years of EHR experience (n = 65, 75.6%).

Mean rate of hospital EHR success
On average, the "total hospital EHR success rate" was 66.81% (95% CI: 64.69%, 68.93%) that were categorized in the "moderate" group. The range of "total EHR success" rate was from 45.35% (95% CI 40.96, 49.73) to 74.96% (95% CI: 70.61, 79.30). The dimensions of "computer resource", "ease of use" and "social support" acquired the highest hospital EHR success rate, respectively. In addition, the dimensions of "training" and "top management support" had the lowest rates of hospital EHR success rate. Low managerial support and training may lead to none of the dimensions being categorized in the "appropriate" category. It seems that due to the shortage of training courses, the users relied on self-efficacy and social support. As shown in Fig. 2, the overall dimension of "ease of use" acquired the highest success rate, followed by "usefulness" and "net benefit". The case HIS was used in the hospital for more than 10 years. As such, most users realized the benefits and applications of the EHR in the workflow. Table 2 presents the mean success rates for all dimensions of hospital EHR evaluation based on the nurses' point of view. Figure 2 shows the total hospital EHR success rate in departments of the case hospital.

Effective factors on hospital EHR success
The results of the Mann-Whitney U tests and Kruskal-Wallis test showed that there was a significant statistical relationship between the variables of "gender" and "selfefficacy" (p-value: 0.042) ( Table 3). The median "selfefficacy" in females (Median: 80%, Q1-Q3: 60%-90%) was higher than males (Median: 60%, Q1-Q3: 50-80%) ( Table 4). There was not a significant statistical relationship between the variables of age, ICDL certification, and nursing status. Currently, as part of the employment process, new nurses are required to have ICDL certification. In addition, in recent years, computers and use of     internet have become more common in Iran, particularly among the youth. All nurses and head nurses use EHRs as their daily routine in the case hospital. Hence, no significant relationship was observed between the variables of age, nursing status as well as ICDL certificate and different dimensions. Three significant relationships were observed as follows: A reverse relation between the variable of "years of experience using computers" and the dimension of "training" (p-value: 0.012) as well as "years of experience using EHR" and the dimension of "ease of use" was observed (p-value: 0.034).
There was a significant difference in the "ease of use" dimension and education level groups (p-value: 0.001). The median of the "ease of use" dimension in the Master group (Median: 80%, Q1-Q3: 80-100%) was higher than in the Bachler group (Median: 73.33%, Q1-Q3: 60-80%). According to our results, a significant statistical relation between other dimensions and the independent variables was not observed. There was a significant relationship between experience in the use of computers and HIS and ease of use as well as training. The user with more level of computer and EHR felt the need to continue with the training courses.

Discussion
In this study, evaluation measures were extracted from a comprehensive literature review and were combined with the most famous evaluation frameworks for EHR. Using a systematic method, more frequent and applicable evaluation measures to assess EHR were identified and nurses participated in the face, content, and construct validity. Finally, an extended-ISSM based on nurses' points of view for hospital EHRs was developed. In addition, the mean of success rate based on nurses' perspectives was determined. The proposed evaluation framework and a questionnaire as well as the proposed method of calculating the success rate can be applied in future studies.
According to the results of CFA, eleven factors were identified in the extended-ISSM questionnaire. Four factors completely confirmed the original ISSM dimensions. Seven dimensions were identified and considered as influential factors on EHR success. The added factors can strengthen the ISSM framework for determining EHR success. The importance of 11 identified factors in determining the success rate in previous studies has been confirmed. For example, Lu et al. proposed a combined model of ISSM and TAM based on nurses' points of view. The factor of "ease of use" was added to the model [32]. In the study by Hsiao et al. "top management support", "user self-efficacy", "perceived ease of use" factors were added to ISSM to better understand the factors affecting acceptance of hospital information systems in nurses [33]. Otieno et al. developed a questionnaire for EHR success based on nurse's point of view that included all dimensions of ISSM [34]. The "Training" factor was included in the modified TAM in a study by Aggelidis et al. Majority of the participants in this study were nurses [35]. Our proposed framework travels beyond previous extended or modified models for nurses. The framework composed of 11 factors in an extended-ISSM. Also, we provide a questionnaire for determination of the EHR success rate with a good degree of validity and reliability.
In line with previous studies, our findings showed that the lowest EHR success rate belonged to the factor of "training". Habibi-Koolaee et al. reported that the mean of computer skills, knowledge, and nurses' attitude towards EHR was 43.4%. They believed that holding related courses in health information systems should be considered in the nursing curriculum [36]. Poor or insufficient training courses, poor literacy and skills in technology were the frequently identified barriers to adoption and use of EHR in the study by Tsai et al. [37]. In our findings, we observed that a reverse relation between the "years of experience using computers" and "training" as well as "years of experience using EHR" and "ease of use". Participants in the group of "over three years of experience in the computer" needed more training courses than the other groups. On the other hand, the rate of "ease of use" in participants with over six years of experience using EHR was lower than in other groups, and it might be due to the short training courses in which new EHR users participated. However, continuous training programs were not held regularly for the users. Therefore, the users after passing six years may need training. In addition, according to our results and the results of the study by Zaman et al., the variables of ICDL as general computer skills were not an effective factor in EHR success [38].
Our results showed that the mean rate of the factors of "information quality" and "service quality" were moderate. Overall, our findings are in accordance with the results of previous studies. Insufficient resource, inadequate training and technical support for users, poor literacy and skills in technology were identified barriers to EHR adoption in a scoping review [37]. Furthermore, in one study, poor quality of nursing documentation was affirmed [39].