- Research article
- Open Access
- Open Peer Review
Improving performance in medical practices through the extended use of electronic medical record systems: a survey of Canadian family physicians
BMC Medical Informatics and Decision Makingvolume 15, Article number: 27 (2015)
Numerous calls have been made for greater assimilation of information technology in healthcare organizations in general, and in primary care settings in particular. Considering the levels of IT investment and adoption in primary care medical practices, a deeper understanding is needed of the factors leading to greater performance outcomes from EMR systems in primary care. To address this issue, we developed and tested a research model centered on the concept of Extended EMR Use.
An online survey was conducted of 331 family physicians in Canadian private medical practices to empirically test seven research hypotheses using a component-based structural equation modeling approach.
Five hypotheses were partially or fully supported by our data. Family physicians in our sample used 67% of the clinical and 41% of the communicational functionalities available in their EMR systems, compared to 90% of the administrative features. As expected, extended use was associated with significant improvements in perceived performance benefits. Interestingly, the benefits derived from system use were mainly tied to the clinical support provided by an EMR system. The extent to which physicians were using their EMR systems was influenced by two system design characteristics: functional coverage and ease of use. The more functionalities that are available in an EMR system and the easier they are to use, the greater the potential for exploration, assimilation and appropriation by family physicians.
Our study has contributed to the extant literature by proposing a new concept: Extended EMR Use. In terms of its practical implications, our study reveals that family physicians must use as many of the capabilities supported by their EMR system as possible, especially those which support clinical tasks, if they are to maximize its performance benefits. To ensure extended use of their software, vendors must develop EMR systems that satisfy two important design characteristics: functional coverage and system ease of use.
The need for the study
In the face of rapidly increasing healthcare costs, associated with an aging population and the concomitant rise in chronic illnesses , governments in developed countries such as the United Kingdom and Canada have felt obliged to improve the efficiency and effectiveness with which primary medical care is provided to their citizens. This has led, for instance, to a greater emphasis on the prevention and monitoring roles played by family physicians and other healthcare professionals . Another important effect has been the deployment of information technology (IT) in support of the incremental and radical changes made by these governments to their healthcare systems. Of all the IT-based applications currently in use in primary care settings, electronic medical records (EMR) have the most wide-ranging capabilities  and thus the greatest potential for enhancing health care services. An EMR can be defined as a computerized system where physicians record relevant information such as patient demographics, medical histories, consultation notes, lists of problems, allergies, vaccinations, vital signs, and prescriptions/renewals . Some EMR systems may also contain other functionalities such as automated alerts, medical appointments and reminders. In short, an EMR system is designed to support the needs of individual physicians who are directly caring for patients in their medical practicesa.
A recent survey conducted by the Commonwealth Funds revealed high adoption rates of EMRs by family physicians in New Zealand, Australia and several European countries, including the Netherlands, the United Kingdom, Sweden and Germany . So the potential value of these systems seems to be widely recognized, yet prior research shows that significant challenges remain before such benefits can be reaped. Recent systematic reviews on this topic found either inconclusive or mixed effects of EMR systems on performance outcomes. For instance, Lau et al.  concluded that, based on prior empirical studies, there is a 51% chance that an EMR system will improve office practices, a 30% likelihood that it will have no effect, and a 19% chance that it will lead to negative outcomes. Similarly, Holroyd-Leduc et al.  found that while EMR systems appear to have clear advantages over traditional paper-based records in terms of legibility and accessibility, effects on work processes (e.g., quality of care, individual efficiency and productivity) and clinical outcomes (e.g., blood pressure control, glycemic control) have not yet been found.
Previous research reveals that the lack of perceived benefits from EMRs may be at least partially attributed to underutilization of these systems by physicians, which may be related to users’ attitudes toward EMRs. For instance, Miller and Sim , who conducted interviews with several EMR users, observed that attaining benefits depends heavily on physicians’ use of EMR functionalities. More recently, Price et al.’s  qualitative analysis showed there are “ceiling effects” to EMR use in primary care practices owing to numerous factors, including a lack of awareness or availability of EMR functionalities to support clinical work as well as the poor usability of these systems. Similarly, Bouamrane and Mair , who also carried out in-depth interviews with general practitioners, concluded that there remains substantial scope for improving family physicians’ interactions and overall satisfaction with these systems.
In light of these findings, the present study examines how EMR systems are actually being used by family physicians, which system design characteristics influence physicians’ usage patterns, and the extent to which EMR use is associated (or not) with particular performance benefits. More specifically, we sought answers to the following research questions: 1) How can the use of EMR systems by family physicians in medical practices best be characterized? 2) To what extent does the use of an EMR system by family physicians, as well as their satisfaction with the system’s functionalities, influence performance benefits? and 3) Do the design characteristics of a specific EMR system determine the extent to which family physicians use the system and are satisfied with it?
Research model and hypotheses
Prior EMR acceptance research has mainly focused on the basic dichotomous adoption decision (yes/no) or amount of system usage (i.e. frequency, time and extent) (e.g., [10,11]). We posit that such limited theoretical attention to EMR usage may explain our lack of knowledge about its impacts on performance outcomes in primary care settings. To overcome such limitations, our research model (see Figure 1) is built around the concept of “Extended Use,” which we have borrowed from the field of information systems (IS).
Saga and Zmud  first coined the term “extended use”, broadly describing it as individuals using more of the technology’s features in order to carry out a more comprehensive set of work tasks. More recently, Hsieh and Wang  defined extended system usage as “the use behaviour that goes beyond typical usage and can potentially lead to better results and returns” (p.217). Prior research in the medical informatics field has shown that the use of systems such as PACS (picture archiving and communication system), clinical decision support systems and CPOE (computerized physician order entry) can have positive effects on performance outcomes (e.g., [14-16]). However, the main implication of these findings is that such benefits are not usually associated with the mere adoption of these systems, but rather with their extensive use [3,17]. Hence, the first research hypothesis is formulated as follows:
Hypothesis 1 – Extended use of an EMR system is positively and significantly associated with performance benefits.
One of the premises of this study is that the value created through system usage must be measured from the user’s perspective . In this regard, the user’s attitude toward a system, defined as “user satisfaction”, has been found to predict performance or beneficial impacts better than his or her usage behaviours . The argument here is that while user satisfaction may not “cause” performance per se, it indicates that the new system has met individual and organizational expectations  and, in turn, that the performance improvements expected from its use should ensue . Furthermore, previous studies of inpatient clinical information systems have also confirmed that user satisfaction is a pre-requisite to the attainment of performance benefits (e.g., [22,23]). Hence, our second hypothesis is formulated as follows:
Hypothesis 2 - EMR user satisfaction is positively and significantly associated with performance benefits.
Prior empirical findings demonstrate that user satisfaction increases as training, experiential learning or improved facilitating conditions renders IS use more effective over time (e.g., ). The empirical evidence supporting a direct influence of system usage on user satisfaction, however, has been rather weak . Still, a more frequent and intensive use of clinical information systems has been previously associated with increased user satisfaction. For instance, a recent study by Maillet et al.  reveals that the extent to which registered nurses use an EHR (electronic health record) system to support their work in acute care settings influences their level of satisfaction with such a system. Hence, our third hypothesis is formulated as follows:
Hypothesis 3 – Extended use of an EMR system is positively and significantly associated with user satisfaction.
Prior research clearly indicates that user behaviours and attitudes toward a system are predicated upon the system’s ease of use, i.e. the extent to which users perceive system use to be free of effort [27-31]. For one thing, the system’s ease of use initially facilitates exploratory usage behaviours on the part of users, leading them to experiment with and eventually use a wider range of the system’s functionalities . Furthermore, previous research has also found a system’s ease of use positively influences user satisfaction. For instance, a study by Jaspers et al.  found that physicians’ satisfaction with a redesigned EMR system was closely related to their perceptions not only of its enhanced functionality but also of its improved usability. In line with these studies, our next two hypotheses can be formulated as follows:
Hypothesis 4 - An EMR system’s ease of use is positively and significantly associated with extended use of the system.
Hypothesis 5 - An EMR system’s ease of use is positively and significantly associated with user satisfaction.
Users’ behaviours and attitudes are also predicated upon the system’s functional coverage, i.e. the extent to which the system “includes the features that the organization needs to operate and that users need to do their work” [34:746]. It stands to reason that the greater the number of functionalities available in a system, the greater the potential for their exploration, assimilation and appropriation by physicians [34,35]. For example, Sicotte et al.  found that physicians were more satisfied with a PACS system when they perceived it to be more useful, i.e. when it provided “a complete range of functionalities” in support of their work, was “well harmonized” with their clinical practices, and was “compatible” with all aspects of their tasks. These findings lead to the formulation of the following hypotheses:
Hypothesis 6 - Functional coverage of an EMR system is positively and significantly associated with its extended use.
Hypothesis 7 - Functional coverage of an EMR system is positively and significantly associated with user satisfaction.
As shown in Figure 1, our research model includes two control variables: individual and organizational experience with EMRs. We posit that individual experience may influence physicians’ behaviours and attitudes toward EMR systems through a process of experiential “learning by doing,”  while organizational experience may exercise such influence through a process of infusion or institutionalization .
An online survey was conducted of family physicians in private medical practices in the province of Québec, Canada to empirically test the research model. The questionnaire instrument, which was developed in both French and English, was initially validated by representatives of the study’s sponsors, Canada Health Infoway and the Quebec Federation of General Practitioners (QFGP).
Data collection and sample
An invitation to participate in the study was emailed to 4,845 members of the QFGP who were practising in primary care settings and could be reached by email. The invitation letter, co-signed by a high-ranking officer of the QFGP, contained a hyperlink directing the family physicians to the questionnaire through a secure Web site. The online questionnaire was developed on the Qualtrics online survey platform . Seven days after the first invitation, a reminder letter was emailed to all members of the survey’s target population. Ethical approvals were obtained from HEC Montréal in April 2013.
Data was thus collected from 780 family physicians, representing a 16% response rate. Low response rates raise the possibility of non-response bias. As suggested by Hikmet and Chen , the potential for such a bias was first assessed by comparing the 156 “late” respondents (i.e. those responding after having received the reminder) with the 624 “early” respondents. Our analyses indicate no statistically significant differences between these two sets of respondents. While this must be interpreted as an encouraging sign, researchers must remain vigilant about response rates if they are uncertain about sample representativeness , so as a second step we compared the demographic characteristics of our respondents with those of the target population, i.e. QFGP members. Our sample was found to be statistically representative of the population in terms of physician age, gender, region and length of professional experience. In short, these two assessments indicate that sampling error is unlikely, and thus give confidence that our sample approximates the characteristics of the target population .
Whereas the survey initially targeted two different types of family physicians – those working in primary care practices with or without an EMR system – this article only analyzes responses from the 331 family physicians who were actually using such a system. The results from the survey of doctors working in medical clinics without an EMR can be found elsewhere . Of the 331 EMR users, 48% were women. As for age, 38% of the users were in their 50s, 28% in their 40s, and 19% in their 30s. Respondents had 22 years’ experience in the medical profession on average, with a minimum of 2 years and a maximum of 45 years. They also had an average of 4 years’ experience using their medical practice’s EMR system, and 56% of the sampled physicians had 3 years’ experience or less.
In order to measure the functional coverage of an EMR system and the extent of its use based on a list of twenty-four EMR functionalities (see Table 1), respondents were asked to indicate whether each functionality was available or not in their system, and if it was available, whether or not they actually used it. They were also asked to indicate their level of satisfaction with each of these functionalities, on a scale of 1 (very dissatisfied) to 5 (very satisfied). The list of functionalities was developed from previous empirical studies on the actual use of EMR systems in primary care settings [10,44-48].
A principal components exploratory factor analysis (EFA) was first performed to empirically – rather than theoretically – group the EMR functionalities used by the surveyed family physicians into eight functional categories. As shown in Table 1, each of these categories was in turn determined to be part of an EMR system’s clinical, communicational or administrative capabilities. This grouping of EMR functionalities is based on descriptions in the literature of activities performed by primary care physicians – such as prescribing medication, ordering laboratory tests or billing patients – as being either part of a clinical process (e.g., ), a communication process (e.g.,) or an administrative process (e.g., ).
Using a set of 24 potential outcomes of EMR system usage (see Table 2), also extracted from the extant literature [52-65], respondents were asked to what extent they agreed (where 1 = strongly disagree and 5 = strongly agree) with statements about the impacts of EMR usage on their individual performance (e.g. “has facilitated my application of clinical care guidelines for my patients”) and on their clinic’s performance (e.g. “has improved teamwork and the continuity of care provided to patients”). A confirmatory factor analysis (CFA) was performed to ascertain the reliability and validity of the perceived performance benefits construct. As presented in Table 2, the assumed penta-dimensional structure of this construct was confirmed, as well as the five measures’ composite reliability and convergent validity.
Similarly, a CFA of the EMR User Satisfaction construct was conducted to verify the tri-dimensional structure assumed, i.e. satisfaction with the EMR system’s clinical, communicational, and administrative functionalities. The results of this analysis, presented in Table 3, also confirm this to be true, as well as the three measures’ composite reliability and convergent validity.
The instrument measuring the Perceived Ease of EMR Use construct was taken from the Commonwealth Fund’s international survey of family physicians . It is composed of eight 5-point scales assessing the ease with which a physician can use an EMR system to gather information for clinical reporting and decision making purposes (1: unable to generate this information, 2: difficult, 3: neutral, 4: easy, 5: very easy to generate this information). As shown in Table 4, an EFA of this construct produced a bi-factorial structure, with one measure of ease of use with regard to patients and another with regard to care providers.
Finally, our respondents were asked questions aimed at contextualizing their use of EMRs in terms of their individual and medical practice demographics, including the physician’s and the practice’s experience with EMR systems, which represent the research model’s two control variables.
Results and discussion
As shown in Figure 1, one exogenous construct in the research model, EMR Functional Coverage, and one endogenous construct, Extended EMR Use, have been modeled as being “formative” rather than “reflective”, given their composite and multidimensional nature. Since each formative indicator captures a different aspect of the model, changes in these indicators bring about or “cause” change in their underlying construct . Component-based structural equation modeling (SEM) was used to validate our research model; Partial Least Squares (PLS) was employed because it is better suited to measurement models that include both exogenous and endogenous formative constructs .b As implemented in SmartPLS software , this approach was also chosen for its robustness with regard to the distribution of residuals c.
Test of the measurement model
Table 5 presents descriptive statistics, reliability coefficients and the correlation matrix of the research variables. The first step in the data analysis consisted of simultaneously estimating the measurement and structural models using PLS. Psychometric properties of both the formative and reflective construct indicators (measures) were thus assessed within the context of our research model.
Given that the usual reliability and validity criteria do not apply to a formative construct, it must first be verified that there is no multicollinearity among the indicators forming this construct . This was verified with the variance inflation factor (VIF), based on the guideline that this statistic should not be greater than 3.3 for any formative indicator .d As shown in Table 5, this condition held for all six indicators.
Once the validity of the formative constructs has been assessed, the unidimensionality and reliability of the reflective constructs must then be evaluated. As depicted in Figure 2, the fact that all indicator loadings (λ) on these constructs were greater than the 0.7 threshold confirmed their unidimensionality. As shown in Table 6, composite reliability coefficient values above the 0.7 threshold also provide strong evidence of the reliability of the three reflective constructs. Evidence of the convergent validity of the reflective constructs was also found, as their average variance extracted (AVE) values are all above the 0.5 threshold.
The last property to be verified is discriminant validity, which shows the extent to which each construct in the research model is unique and different from the others. Here, the shared variance between a reflective construct and other constructs must be less than the AVE by a construct from its indicators. The results in Table 6 show this to be the case for all three reflective constructs included in the model. For the two formative constructs, discriminant validity is demonstrated by a correlation with any other construct that is significantly different from unity (at p < 0.001) .
Test of the structural model
The research hypotheses were tested by assessing the path coefficients (β) estimated by PLS (see Figure 2). While PLS does not provide model fit indices, the performance of the structural model can be ascertained by the strength and significance of the path coefficients and by the proportion of construct variance (R2) that it explains . Moreover, only path coefficients greater than 0.2 should be considers “truly” significant, since PLS tends to underestimate structural paths when compared with covariance structure-based approaches, such as EQS or LISREL .
Hypothesis 1 (confirmed)
As shown in Figure 2, a positive and significant path coefficient (ß = 0.38, p < 0.001) confirms the hypothesis that extended use of an EMR system by family physicians leads to improvements in performance benefits. Further examining the weights of the three indicators that “form” the “extended EMR use” construct, one finds that the “use of clinical functionalities” indicator is by far the most determinant (γ = 0.82, p < 0.001). The concept of extended use and the performance benefits to be obtained from such use are thus mainly tied to the clinical support provided by an EMR system through patient care-oriented functionalities such as e-prescriptions and the monitoring of chronically-ill patients.
Hypothesis 2 (confirmed)
The hypothesis that increasing physicians’ satisfaction with an EMR system leads to increased performance benefits for the physicians and their practices is affirmed by a positive and significant path coefficient (ß = 0.30, p < 0.001). This implies that governments and medical associations should not only encourage more extended use of EMRs by family physicians, but that they should also encourage EMR software designers and vendors to offer systems whose clinical, communicational and administrative functionalities meet user requirements in their contexts and, thus, better satisfy physicians. Since these requirements vary by physician and by medical practice, the preceding results allow one to associate specific user requirements with specific individual and organizational performance benefits, thus providing an empirical basis for greater flexibility and customization of EMR systems, whether through proprietary or open-source software .
Hypothesis 3 (unconfirmed)
Given a non-significant path coefficient (ß = 0.08, p > 0.05), the hypothesis that more extended use of EMRs by family physicians increases user satisfaction could not be confirmed. As mentioned above in the formulation of the hypothesis, prior empirical results in this regard have been mixed. Here, the intended consequences of EMR usage (performance improvements) are seen to have both a behavioural determinant (extended use) and an attitudinal determinant (user satisfaction), and these determinants act independently.
Hypothesis 4 (partly confirmed)
Returning to Figure 2, a positive and significant – though not very strong – path coefficient (ß = 0.12, p < 0.05) partially confirms the hypothesis that physicians use EMRs more extensively when they perceive such usage to be free of effort. This result is aligned with the findings of Price et al.  that family physicians tend not to use those EMR features that they find difficult to use or disruptive to patient care work flow. In further analyzing this relationship, one finds that it is strongest between the system’s ease of use with regard to patients and the physician’s use of clinical functionalities (see the correlations in Table 5). This latter result again emphasizes the patient care- and clinical process-centered view of the extended use of EMRs.
Hypothesis 5 (confirmed)
The positive and significant path coefficient (ß = 0.26, p < 0.01) also confirms the hypothesized relationship between an EMR system’s ease of use and physicians’ satisfaction with using it. While this is hardly surprising given previously-cited IT user satisfaction surveys, this result nonetheless adds validity and generalizability to the nomological network linking the five research constructs in our research model.
Hypothesis 6 (confirmed)
The hypothesis that a wider functional coverage of an EMR system leads physicians to use the system in a more extensive way is fully confirmed, as the corresponding path coefficient is shown to be strongly positive and significant (ß = 0.80, p < 0.001). Examining the weights of the three formative indicators of the “EMR functional coverage” construct, one finds the “availability of clinical functionalities” indicator to be by far the most determinant (γ = 0.80, p < 0.001). Notwithstanding such a positive finding, much like Price et al.  we observed an important “ceiling effect” in terms of EMR usage by family physicians in private medical practices. Indeed, the results in Table 5 indicate that surveyed physicians only use, on average, 67% of the clinical functionalities and 47% of the communication functionalities in their EMRs. This compares with their use of 90% of its administrative features. A lack of awareness among physicians of EMR capabilities may explain, at least in part, this gap between what is available and what is actually being used (see Table 1). To minimize the “ceiling effect” problem, it is recommended that training and user-group sessions be organized on a regular basis (rather than only when the EMR system is first implemented). Such sessions or meetings would appear to be particularly important when new versions of an EMR product are released to the user community. Importantly, EMR vendors must carefully manage their clients’ expectations and ensure that clients have a clear understanding of the capabilities and limitations of their products, to avoid unnecessary dissatisfaction.
Hypothesis 7 (unconfirmed)
Given a non-significant path coefficient (ß = 0.05, p > 0.05), the last research hypothesis could not be confirmed. Thus, the functional coverage of an EMR system seems to have no bearing on physicians’ satisfaction with its various functionalities. In light of the previous finding that extended use also has no direct bearing on user satisfaction, one may surmise that family physicians generally have much higher expectations regarding the ease of use of an EMR system than its functional coverage.
Returning to Figure 2, the “extended EMR use” and “EMR user satisfaction” constructs were found to jointly explain 33% of the variance in the dependent variable. This represents a “strong” effect size, in that it is very close to the 35% threshold . Lastly, no significant effect could be observed for the two control variables: the physician’s and the practice’s experience with EMRs.
Test of the adjusted research model
Further analysis was performed by making two adjustments to the initial model. First, in line with a position taken previously by certain IS researchers [13,76], the third hypothesis was reformulated by reversing the causal direction of the relationship posited between extended use and user satisfaction. Second, in line with previous IS studies [45,77], the EMR artefact’s characteristics (perceived ease of use and functional coverage) were posited to have direct effects on the attainment of performance benefits, i.e. irrespective of the extent to which physicians use the system and are satisfied with it. Figure 3 presents the test results from the adjusted structural model with PLS.
First, given a non-significant path coefficient (ß = 0.03, p > 0.05) estimated from the second PLS structural analysis, the following alternative to the third hypothesis could not be confirmed:
Hypothesis 3 alt (unconfirmed) - EMR user satisfaction is positively and significantly associated with extended EMR use.
In light of the previously-cited studies, this additional result provides even more reason to clearly distinguish EMR user satisfaction and extended EMR use in terms of their components and determinants, since the mutual dependence of these two constructs was found to be low.
Second, a positive and significant path coefficient (ß = 0.30, p < 0.01) confirms the following hypothesis:
Hypothesis 8 (confirmed) - An EMR system’s ease of use is positively and significantly associated with performance benefits.
The intended consequences of EMR adoption and assimilation in primary care settings are thus found to have an added artefactual determinant (the EMR system), i.e. irrespective of the behavioural (extended use) and attitudinal (user satisfaction) determinants previously demonstrated. This result again underscores the need to account for EMR design characteristics in explanations of the benefits to be obtained from EMR use.
As IT artefacts, prior research has often “blackboxed” EMR systems or “reduced them to surrogate measures” . The preceding results provide an empirical basis for EMR design and evaluation that is coherent with the particular individual (family physician) and organizational (primary care clinic) contexts of EMR use. EMR designers and vendors may therefore find it useful to employ the empirical description of EMR systems provided in this study to pinpoint the functional features that most and least satisfy users, and whose usage is most or least conducive to performance. To this end, Tables 1, 2, 3 may provide an initial reference framework. These results may also guide the evaluation and improvement of existing systems, such as highlighting their missing features, since family physicians and other primary care professionals generally have no benchmarks for EMR functionality and effectiveness to support their decision making on these issues .
Overall, the adjusted model explained 41% of the variance in the performance benefits perceived by family physicians. This represents a significant increase from the initial model. The revised model thus performed well in its nomological integration of the EMR artefact, the primary care physicians who use this artefact extensively, and the performance outcomes of such use.
By developing and empirically testing a research model around the concept of “Extended EMR Use,” this study has produced original and novel results that may eventually apply to other types of clinical information systems deployed in other settings. Initial findings include the determination of what actually constitutes extended use of an EMR system, and the confirmation that extensive use of an EMR, and especially its clinical functionalities, is positively and significantly associated with performance benefits. The results of this study also confirm that the design characteristics of an EMR system, in terms of functional coverage and ease of use, determine the extent to which family physicians will make extensive use of the system.
As mentioned above, previous studies on this topic have focused either on identifying the benefits ensuing from physicians’ adoption of this technology, or on determining their “intention to use” EMR systems. In answering the call for more assimilation of IT in healthcare organizations, our study concentrated instead on physicians’ actual usage behaviours and on the characteristics of the EMR systems available to them , as we contend that a deeper understanding of EMRs as they are actually being implemented and used in primary care settings is a prerequisite to fully explaining whether or not and why performance improvements are achieved. Moreover, previous studies have generally taken a “generic” or theoretical approach to characterizing the functionalities of EMR systems, and have focused on specific features, such as electronic communication, or on individual functionalities, such as the monitoring of chronically-ill patients. Our study complements previous research since we founded our conceptualization on the reality of physicians’ actual use of EMRs in primary care medical practices.
Lastly, our results must be considered in light of the study’s low response rate and the inherent limitations of survey research, as there may yet be biases linked to the perceptual nature of the data. In particular, measuring all variables through a self-administered questionnaire and using single respondents may pose a risk of common method bias (CMB) and lead the relationships between constructs to be overestimated, such that basic precautions must be taken to minimize this risk . Thus, the questionnaire was designed to be anonymous, giving the respondents all the latitude needed to express their true perceptions, attitudes and behaviours. Robust measurement scales were used, and the independent and dependent variables were placed in different sections of the questionnaire. Different question formats and scale types were also used for the five sets of variables measuring the theoretical constructs. Moreover, given that the potential for CMB is very rarely acknowledged in medical informatics survey research, with a few exceptions [81,82] we used the recommended “latent method factor” approach to further examine this issue , allowing us to conclude that CMB is not a major threat in this studye.
aWhile the concepts of EMR and EHR (electronic health record) are sometimes used interchangeably, like other authors we consider them to be two distinct health information systems. An EHR system is an “aggregated, longitudinal system of systems which assembles health information about a patient over a wide area network from, potentially, many geographically dispersed data sources” [4:24]. It usually draws from sources such as EMRs, hospital information systems, drug repositories, centralized laboratory data systems, and national immunization databases .
bWhile covariance structure analysis can be applied to formative constructs, this requires stringent model identification conditions that are theoretically problematic or empirically inapplicable in a research model such as ours. For an endogenous construct, the identification condition requires specifying the construct’s formative indicators as endogenous variables .
cThe three “EMR functional coverage” variables and the three “extended EMR use” variables are operationalized through “index” rather than “scale” measures . An index variable tends to follow a Poisson-type distribution rather than a normal distribution, i.e. it is right-skewed if the mean is small. Moreover, an index comprises elements that are not expected to be highly intercorrelated, hence the inappropriateness of using Cronbach’s alpha coefficient to test its reliability .
dVIF i = 1/(1-R i 2) where R i 2 is the unadjusted R2 obtained when component i is regressed against all other components of the formative construct.
eWe conducted a CFA in which an “unmeasured latent method” construct was added to the measurement model, with the measures being allowed to load on this construct as well as on their theoretical construct. The CFA allowed us to break down the variance of the measures into theoretical, random error and method components . The results show that 66% of the variance was explained by the five theoretical constructs, 8% by random errors, and 26% by the method construct. Moreover, this model did not fit the data any better than a second CFA model in which the latent method construct was removed.
Average Variance Extracted
Confirmatory Factor Analysis
Computerized Physician Order Entry
Clinical Information Systems
Common Method Bias
Exploratory Factor Analysis
Electronic Health Records
Electronic Medical Records
Picture Archiving and Communication System
Quebec Federation of General Practitioners
Structural Equation Modeling
Variance Inflation Factor.
Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness. J Am Med Assoc. 2002;288(14):1775–9.
Broemeling AM, Watson DE, Prebtani F. Population patterns of chronic health conditions, co-morbidity and healthcare use in Canada: implications for policy and practice. Healthcare Quart. 2008;11(3):70–6.
Miller RH, Sim I. Physicians’ use of electronic medical records: barriers and solutions. Health Affair. 2004;23(2):116–26.
Ludwick DA, Doucette J. Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. Int J Med Inform. 2009;78:22–31.
Schoen C, Osborn R, Squires D, Doty M, Rasmussen P, Pierson R, et al. A survey of primary care doctors in ten countries shows progress in use of health information technology, less in other areas. Health Affair. 2012;31(12):2805–16.
Lau F, Price M, Boyd J, Partridge C, Bell H, Raworth R. Impact of electronic medical record on physician practice in office settings: a systematic review. BMC Med Inform Decis. 2012;12(10):1–10.
Holroyd-Leduc JM, Lorenzetti D, Straus SE, Sykes L, Quan H. The impact of the electronic medical record on structure, process, and outcomes within primary care: a systematic review of the evidence. J Am Med Inform Assoc. 2011;18:732–7.
Price M, Singer A, Kim J. Adopting electronic medical records: are they just electronic paper records? Can Fam Physician. 2013;59:e322–9.
Bouamrane MM, Mair FS. A study of general practitioners’ perspectives on electronic medical records systems in NHSScotland. BMC Med Inform Decis. 2013;13:58.
Gans D, Kralewski J, Hammons T, Dowd B. Medical groups’ adoption of electronic health records and information systems. Health Affair. 2005;24(5):1323–33.
Menachemi N. Barriers to ambulatory EHR: who are ‘imminent adopters’ and how do they differ from other physicians? Inform Prim Care. 2006;14(2):101–8.
Saga VL, Zmud RW. The nature and determinants of IT acceptance, routinization and infusion. In: Levine L, editor. Diffusion, transfer and implementation of information technology. Amsterdam: North-Holland; 1994. p. 67–86.
Hsieh JJPA, Wang W. Explaining employees’ extended use of complex information systems. Eur J Inform Syst. 2007;16(3):216–27.
Lepanto L, Paré G, Aubry D, Robillard P, Lesage J. Impacts of PACS on dictation turnaround time and productivity. J Digit Imaging. 2006;19(1):92–7.
Wells S, Furness S, Rafter N, Horn E, Whittaker R, Stewart A. Integrated electronic decision support increases cardiovascular disease risk assessment four fold in routine primary care practice. Eur J Cardiov Prev R. 2008;15:173–8.
Spaulding TJ, Raghu TS. Impact of CPOE usage on medication management process costs and quality outcomes. Inquiry. 2013;50(3):229–47.
Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med. 2014;160(1):48–54.
Petter S, DeLone W, McLean ER. The past, present, and future of “IS success”. J Assoc Inform Syst. 2012;43:341–62.
Gelderman M. The relation between user satisfaction, usage of information systems and performance. Inform Manage. 1998;34(1):11–9.
Paré G, Trudel MC, Forget P. Adoption, use and impact of e-booking in private medical practices: mixed-methods evaluation of a two-year showcase project in Canada. JMIR Med Inform. 2014;2(2), e24.
Iivari J. An empirical test of the DeLone-McLean model of information system success. Data Base Adv Inform Syst. 2005;26(2):8–27.
Sicotte C, Paré G, Moreault MP, Lemay A, Valiquette L, Barkun J. Replacing an inpatient electronic medical record: lessons learned from user satisfaction with the previous system. Method Inform Med. 2009;48(1):92–100.
van der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc. 2003;10(3):235–43.
DeLone WH, McLean ER. The DeLone and McLean model of information systems success: a ten-year update. J Manage Inform Syst. 2003;19(4):9–30.
Petter S, McLean ER. A meta-analytic assessment of the DeLone and McLean IS success model: an examination of IS success at the individual level. Inform Manage. 2009;46:159–66.
Maillet E, Mathieu L, Sicotte C. Modeling factors explaining the acceptance, actual use and satisfaction of nurses using an electronic patient record in acute care settings: an extension of the UTAUT. Int J Med Inform. 2015;84:36–47.
Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Quart. 2003;27(3):425–78.
Sabherwal R, Jeyaraj A, Chowa C. Information system success: individual and organizational determinants. Manage Sci. 2006;52(12):1849–64.
Myers JJ, Koester KA, Chakravarty D, Perason C, Maiorana A, Shade SB, et al. Perceptions regarding the ease of use and usefulness of health information exchange systems among medical providers, case managers and non-clinical staff members working in HIV care and community settings. Int J Med Inform. 2012;81:e21–9.
Bossen C, Jensen LG, Udsen FW. Evaluation of a comprehensive EHR based on the DeLone and McLean model for IS success: approach, results, and success factors. Int J Med Inform. 2013;82(10):940–53.
Saeed KA, Abdinnour S. Understanding post-adoption IS usage stages: an empirical assessment of self-service information systems. Inform Syst J. 2013;23:219–44.
Jain V, Kanungo S. Beyond perceptions and usage: impact of nature of information systems use on information system-enabled productivity. Int J of Hum-Comput Int. 2005;19(1):113–36.
Jaspers MWM, Peute LWP, Lauteslager A, Bakker PJM. Pre-post evaluation of physicians’ satisfaction with a redesigned electronic medical record system. Stud Health Tech Inform. 2008;136:303.
Igbaria M, Guimaraes T, Davis GB. Testing the determinants of microcomputer usage via a structural equation model. J Manage Inform Syst. 1995;11(4):87–114.
Joos D, Chen Q, Jirjis J, Johnson KB. An electronic medical record in primary care: impact on satisfaction, work efficiency and clinic processes. In AMIA Annual Symposium Proceedings (Vol. 2006, p. 394–8). Bethesda, Maryland, United States: American Medical Informatics Association
Sicotte C, Paré G, Kra Bini K, Moreault MP, Laverdure G. Virtual organization of hospital medical imaging: a user satisfaction survey. J Digit Imaging. 2010;23(6):689–700.
Adler PS, Clark KB. Behind the learning curve: a sketch of the learning process. Manage Sci. 1991;37(3):267–81.
Rogers EM. Diffusion of Innovations. New York: Free Press; 1995. p. 519.
Snow J. Qualtrics survey software: handbook for research professionals. Provo, Utah: Qualtrics Labs, Inc. 2012. http://www.qualtrics.com/university/researchsuite/research-resources/other-resources/ebook-downloads/. Accessed Feb 3 2015.
Hikmet N, Chen SK. An investigation into low mail survey response rates of information technology users in health care organizations. Int J Med Inform. 2003;72:29–34.
Cook C, Heath F, Thompson RL. A meta-analysis of response rates in web- or internet-based surveys. Educ Psychol Meas. 2000;60(6):821–36.
Tabachnick BG, Fidell LS. Using multivariate statistics. Boston, MA: Pearson Education Inc.; 2007.
Paré G, Raymond L, Ortiz de Guinea A, Poba-Nzaou P, Trudel MC, Marsan J, et al. Barriers to organizational adoption of EMR systems in family physician practices: a mixed-methods study in Canada. Int J Med Inform. 2014;83(8):548–58.
Miller RH, Sim I, Newman J. Electronic medical records in solo/small groups: a qualitative study of physician user types. Stud Health Tech Inform. 2004;107(Pt 1):658–62.
O’Malley AS, Grossman JM, Cohen GR, Kemper NM, Pham HH. Are electronic medical records helpful for care coordination? Experiences of physician practices. J Gen Intern Med. 2010;25(3):177–85.
Rao S, DesRoches C, Doelan K, Campbell E, Miralles P, Jha A. Electronic health records in small physician practices: availability, use, and perceived benefits. J Am Med Inform Assoc. 2011;18:271–7.
Pfoh ER, Abramson E, Zandieh S, Edwards A, Kaushal R. Satisfaction after the transition between electronic health record systems at six ambulatory practices. J Eval Clin Pract. 2012;18:1133–9.
Ramaiah M, Subrahmanian E, Sriram RD, Lide BB. Workflow and electronic health records in small medical practices. Perspectives in health information management. Am Health Inform Manage Assoc. 2012;9:1d.
Barnsley J, Berta W, Cockerill R, MacPhail J, Vayda E. Identifying performance indicators for family practice. Can Fam Physician. 2005;51(5):700–8.
Wellner SJ, Barnet B, Cheng TL, Daaleman TP. Processes for effective communication in primary care. Ann Intern Med. 2005;142:709–14.
Graves P. Quantifying quality in primary care. Oxford, England: Radcliffe Medical Press; 2002.
Wager KA, Lee FW, White AW, Ward DM, Ornstein SM. Impact of an electronic medical record system on community-based primary care practices. J Am Board Fam Pract. 2000;13:338–48.
Kinn JW, O’Toole MF, Rowley SM, Marek JC, Bufalino VJ, Brown AS. Effectiveness of the electronic medical record in cholesterol management in patients with coronary artery disease (Virtual LipidClinic). Am J Cardiol. 2001;88:163–5.
Mitchell E, McConnahie A, Sullivan F. Consultation computer use to improve management of chronic disease in general practice: a before and after study. Inform Prim Care. 2003;11:61–8.
Singh R, Servoss T, Kalsman M. Estimating impacts on safety caused by the introduction of electronic medical records in primary care. Inform Prim Care. 2004;12:235–41.
Hsu J, Huang J, Fung V, Robertson N, Jimison H, Frankel R. Health information technology and physician–patient interactions: impact of computers on communication during outpatient primary care visits. J Am Med Inform Assoc. 2005;12(4):474–80.
Schade CP, Sullivan FM, van der Lusignan S, Madeley J. E-prescribing, efficiency, quality: lessons from the computerization of UK family practice.J Am Med Inform Assoc. 2006;13:470–5.
Cauldwell M, Beattie C, Cox B, Denby W, Ede-Golightly J, Linton F. The impact of electronic patient records on workflow in general practice. Health Inform J. 2007;13:155–60.
Crosson JC, Ohman-Strickland PA, Hahn KA, DiCicco-Bloom B, Shaw E, Orzano AJ. Electronic medical records and diabetes quality of care: results from a sample of family medicine practices. Annal Fam Med. 2007;5:209–15.
Jen WY, Chao CC. Measuring mobile patient safety information system success: an empirical study. Int J Med Inform. 2008;77(10):689–97.
Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grand R. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15:430–8.
Shachak A, Reis S. The impact of electronic medical records on patient-doctor communication during consultation: a narrative literature review. J Eval Clin Pract. 2009;15:641–9.
Zhou L, Soran CS, Jenter CA. The relationship between electronic health record use and quality of care over time. J Am Med Inform Assoc. 2009;16:457–64.
Beasly JW, Wetterneck TB, Temte J, Lapin JA, Smith P, Rivera-Rodriguez J, et al. Information chaos in primary care: implications for physician performance and patient safety. J Am Board Fam Pract. 2011;24(6):745–51.
Ryan A, Bishop T, Shih S, Casolino L. Small physician practices in New York needed sustained help to realize gains in quality from use of electronic health records. Health Affair. 2013;32(1):53–62.
MacKenzie SB, Podsakoff PM, Jarvis CB. The problem of measurement model misspecification in behavioral and organizational research and some recommended solutions. J Appl Psychol. 2005;90(4):710–30.
Roberts N, Thatcher JB. Conceptualizing and testing formative constructs: tutorial and annotated example. Data Base Adv Inform Syst. 2009;40(3):9–39.
Ringle CM, Wende S, Will S. SmartPLS 2.0. Hamburg: (M3) Beta; 2005.
Gefen D, Straub DW, Boudreau MC. Structural equation modeling and regression: guidelines for research practice. Commun Assoc Inform Syst. 2000;4(7):1–76.
Petter S, Straub DW, Rai A. Specifying formative constructs in information systems research. MIS Quart. 2007;31(4):623–56.
Diamantopoulos A, Siguaw JA. Formative versus reflective indicators in organizational measure development: a comparison and empirical illustration. Brit J Manage. 2006;17(4):263–82.
Anderson JC, Gerbing DW. Structural equation modeling in practice: a review and recommended two-step procedure. Psychol Bull. 1988;103:411–23.
Chin WW. The Partial Least Squares approach for structural equation modeling. In: Marcoulides GA, editor. Modern Methods for Business Research. Hillsdale, New Jersey: Lawrence Erlbaum Associates; 1998. p. 295–336.
Millard PS, Bru J, Berger CA. Open-source point-of-care electronic medical records for use in resource-limited settings: systematic review and questionnaire surveys. BMJ Open. 2012;2(4):e000690.
Cohen J. Statistical Power Analysis for the Behavioral Sciences. Hillsdale, New Jersey: Lawrence Erlbaum Associates; 1988.
Bergeron F, Raymond L, Rivard S, Gara MF. Determinants of EIS use: testing a behavioral model. Decis Support Syst. 1995;14(2):131–46.
Byrd TA, Thrasher EH, Lang T, Davidson NW. A process-oriented perspective of IS success: examining the impact of IS on operational cost. Omega. 2006;34(5):448–60.
Ortiz de Guinea A, Webster J. An investigation of information systems use patterns: technological events as triggers, the effects of time, and consequences for performance. MIS Quart. 2013;37(4):1165–88.
Otieno GO, Hinako T, Motohiro A, Daisuke K, Keiko N. Measuring effectiveness of electronic medical records systems: toward building a composite index for benchmarking hospitals. Int J Med Inform. 2008;77(10):657–69.
Strong DM, Volkoff O. Understanding organization-enterprise system fit: a path for theorizing the information technology artifact. MIS Quart. 2010;34(4):731–56.
Chiang KM, Liu CF, Ma CC. Motivating medical information system performance by system quality, service quality, and job satisfaction for evidence-based practice. BMC Med Inform Decis. 2012;12:135.
Kuo KM, Liu CF, Ma CC. An investigation of the effect of nurses’ technology readiness on the acceptance of mobile electronic medical record systems. BMC Med Inform Decis. 2013;13:88.
Podsakoff PM, MacKenzie SB, Lee JY, Podsakoff NP. Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol. 2003;88(5):879–903.
Garets D, Davis M. Electronic medical records vs. electronic health records: yes, there is a difference. HIMSS Analytics White Paper, 2006. http://www.healthit.gov/buzz-blog/electronic-health-and-medical-records/emr-vs-ehr-difference. Accessed Feb 3 2015.
Temme D, Hildebrandt L. Formative measurement models in covariance structure analysis: Specification and identification. SFB 649 Discussion paper 2006–083, Institute of Marketing, Humboldt-Universität zu Berlin, Germany, 2006. https://ideas.repec.org/p/hum/wpaper/sfb649dp2006-083.html. Accessed Feb 3 2015.
Diamantopoulos A, Winklhofer HM. Index construction with formative indicators: an alternative to scale development. J Marketing Res. 2001;38(2):269–77.
Bollen K, Lennox R. Conventional wisdom on measurement: a structural equation perspective. Psychol Bull. 1991;110:305–14.
Williams LJ, Cote JA, Buckley MR. Lack of method variance in self-reported affect and perceptions at work: reality or artifact? J Appl Psychol. 1989;74(3):462–8.
The authors would like to acknowledge the assistance of Dr. Serge Dulude and Ms. Isabelle Paré, both from the Quebec Federation of General Practitioners (FMOQ in French), for their participation in the development of the questionnaire instrument and for their assistance during the data collection phase. The financial support of Canada Health Infoway is also acknowledged.
The authors declare that they have no competing interests.
All authors actively participated in the development of the study design. Data collection and data analysis were performed by LR, GP and AOG. All authors were also involved in interpreting the main results. LR, GP and AOG wrote and revised the manuscript. All authors read and approved the final manuscript.
About this article
- Electronic medical records
- Primary care
- Family physicians
- Extended use
- User satisfaction
- Ease of use
- Functional coverage
- Survey research
- Structural equation modeling