A Socio-technical assessment of the success of picture archiving and communication systems: the radiology technologist’s perspective
BMC Medical Informatics and Decision Making volume 13, Article number: 109 (2013)
With the increasing prevalence of Picture Archiving and Communication Systems (PACS) in healthcare institutions, there is a growing need to measure their success. However, there is a lack of published literature emphasizing the technical and social factors underlying a successful PACS.
An updated Information Systems Success Model was utilized by radiology technologists (RTs) to evaluate the success of PACS at a large medical center in Taiwan. A survey, consisting of 109 questionnaires, was analyzed by Structural Equation Modeling.
Socio-technical factors (including system quality, information quality, service quality, perceived usefulness, user satisfaction, and PACS dependence) were proven to be effective measures of PACS success. Although the relationship between service quality and perceived usefulness was not significant, other proposed relationships amongst the six measurement parameters of success were all confirmed.
Managers have an obligation to improve the attributes of PACS. At the onset of its deployment, RTs will have formed their own subjective opinions with regards to its quality (system quality, information quality, and service quality). As these personal concepts are either refuted or reinforced based on personal experiences, RTs will become either satisfied or dissatisfied with PACS, based on their perception of its usefulness or lack of usefulness. A satisfied RT may play a pivotal role in the implementation of PACS in the future.
Information Systems (ISs) are adopted to improve service, quality of information, and organizational efficiency . Since ISs usually involve considerable financial investment, the measurement of their success has become a critical issue in healthcare and non-healthcare industries [2–4].
The definition of a successful IS has become a topic of substantial research and debate [3–7]. Most researchers [8–10] agree that evaluation of an IS should focus on the organizational influences surrounding the IS. These influences may be viewed from numerous perspectives including the usefulness of an IS [11, 12], its usage , the information quality produced by an IS , the value of the IS to the organization , and its return on investments . Although a variety of measures are available to evaluate the success of an IS, no single measure is superior to another . Often, the choice of measures is dependent on the type of IS being assessed .
IS success model
DeLone and McLean’s (1992) IS success model
DeLone and McLean  proposed a classification system consisting of six measures of IS success drawn from a review of 180 studies performed to evaluate the success of ISs. The proposed IS success taxonomy included System Quality, Information Quality, IS Use, User Satisfaction, Individual Impact, and Organizational Impact. These variables are interrelated, as shown in Figure 1. Furthermore, Figure 1 demonstrates how system quality and information quality collectively influence both IS use and user satisfaction which, in turn, are determinants of individual impact. Finally, individual impact affects organizational impact.
In the DeLone and McLean model, IS use was regarded as an IS success variable. IS use was viewed as the consumption of IS output , which was considered to be an antecedent of individual impact. According to their model, IS use was indispensable for the realization of the system’s benefits. However, DeLone and McLean do not empirically validate their model but they recommend further development and corroboration of their proposed model .
Seddon’s (1997) model
Seddon  asserted that the merger of causal and process concepts in the IS success model proposed by DeLone and McLean  could become a source of confusion. Therefore, Seddon  proposed three classes of variables in his revised model (Figure 1). These included measures of information and system quality, general measures of net benefits from IS use, and behavior with respect to IS use. An important discrepancy between Seddon’s  model and that of DeLone and McLean’s  was the definition and placement of IS use. Seddon viewed IS use as a behavioral outcome that manifested as an anticipation of net benefits from utilizing an IS. This latter definition of IS use implied that IS use resulted from IS success, rather than being an innate feature of IS success . Seddon’s  model included a direct path leading from system quality and information quality to perceived usefulness and user satisfaction. In addition, perceived usefulness was felt to influence user satisfaction.
DeLone and McLean’s (2003) updated IS success model
In 2003, DeLone and McLean  proposed an updated IS success model (Figure 1) and assessed its usefulness by considering vigorous changes in IS practice. They agreed with Seddon’s assertion that merging causal and process perspectives of IS success into a single model could be confounding. On the other hand, they argued that Seddon’s  restructuring of the DeLone and McLean  model into two partial causal models could confound the success model and deviate from the original intention of their model. Based on previous studies, DeLone and McLean  proposed an updated model by including service quality as a new dimension for measuring IS success, further assembling all the ‘impact’ measures into a single construct named net benefits.
Picture archiving and communication systems (PACS)
With progress in image processing technology and network communications, the Picture Archiving and Communication Systems (PACS) has being increasingly utilized to process digitized medical images in healthcare institutions [11, 17]. As a result, the PACS adoption rate among Taiwanese hospitals has risen from 24.9% in 2005 to 36% in 2013 [18, 19]. PACS is an electronic IS used to acquire, store, transmit, and display medical images . Proponents of PACS emphasize its numerous benefits such as the elimination of expensive silver-based films, reduction in physical storage requirements, improvement in access to images, and reduced personnel costs . However, the implementation of PACS is a complex process which demands vast resources .
The adoption of PACS is dissimilar to other types of ISs  due to its innate complexity. For instance, PACS is required to integrate and process data from various hospital information systems (HIS) in order to provide sufficient patient information to the entire healthcare staff. Furthermore, the establishment of PACS is an expensive undertaking requiring justification for its cost . Therefore, assessment of the success of PACS varies from that of traditional ISs.
When implementing new technologies, the existing literature [12, 23, 24] has emphasized the importance of focusing on wider organizational and human factors instead of centering solely on operational considerations. Technology, the individual, and the organization should not be assessed separately from each other . Furthermore, Van Der Meijden et al.  concluded (after a thorough literature review) that more in-depth assessments of healthcare ISs be undertaken in order to explore a wider scope of factors that may determine the success or failure of these systems. The success of PACS requires integration/cooperation involving project management, financial and human resources, social/behavioral influences (e.g., change and resistance management), and technology (e.g., integration with various hospital information systems) .
In spite of its significant value to healthcare institutions, its widespread use, and the sizable financial investments involved, there is a paucity of literature emphasizing the social and technical factors involved in the successful utilization of PACS . Most PACS-related studies have focused on evaluation of its effect on profit  or productivity [8, 9, 27]. Literature concerning the perceptions of PACS is still rare [11, 12, 17, 26]. As shown in Table 1, most studies involved a variety of healthcare professionals including radiologists, technologists, and clinicians [11, 12, 26]. Moreover, the majority of studies have investigated the acceptance/adoption perception of PACS [11, 12, 17, 28], and only one study  explored the success of PACS. Although such studies have advanced our knowledge regarding PACS, the use of a heterogeneous population may miss micro-level effects . In addition, most PACS acceptance studies [11, 12, 28] utilized Davis’s  TAM or TAM-related models (e.g., UTAUT), probably the most commonly adopted model in healthcare settings , as a theoretical basis. Thus, additional studies, that are more homogeneous and that focus on the success of PACS, are required to gain a better perspective on the topic.
The measurement of IS success, from the socio-technical perspective, should involve both social/human and technological factors . As with most ISs, the success of PACS depends upon the degree of its utilization , which, in turn, may be directly related to system quality, information quality, service quality, user satisfaction, and perceived usefulness. Hence, the technological factors (i.e., system quality, information quality, and service quality) and the social/human factors (i.e., usefulness, user satisfaction, and PACS dependence) are practical constructs for measuring the success of PACS.
The motive for investigating our specific target population, i.e., radiology technologists (RTs), can be illustrated using the concept of the value chain . A value chain can be regarded as a system of interdependent activities within an organization, including primary and supportive activities, which collectively generate value for the organization . These activities are connected by linkages which exist when the manner in which one activity is performed affects the cost or effectiveness of the other activities . Applying the concept of value chain to a radiology department, we can regard the radiologist’s interpretation of examinations as the primary activity and the RTs’ work as supportive activity. Without the RT’s ability to assess image quality via PACS, it is not possible for radiologists to interpret the images in a timely manner, which imposes a negative impact on patient care. Thus, the RT’s perception of PACS, as it relates to their jobs, is an important issue that hospitals must confront given the increasing rate of PACS adoption. Furthermore, literature that assesses the success of PACS from the perspective of non-physician healthcare providers [12, 15, 26, 35] is relatively scarce.
The primary objective of this study was to propose (and validate) a revised PACS success model from the socio-technical perspective by quantifying the subjective perception of RTs. The motive for choosing this specific target population was twofold. First, the use of a more homogenous population would make it easier to capture micro-level effects . Second, the concerns of RTs are important for gaining insight into how to promote wider PACS usage throughout the hospital.
Conceptual model formulation
DeLone and McLean  showed that abundant measures of IS success exist based on the different ways IS can be viewed. Consequently, the selection of success measures should rely upon the study context . Most researchers agree that service quality, when well measured, can be coupled with system quality and information quality to assess IS success . Furthermore, Seddon  asserted that ‘IS use’ was not an appropriate measure of IS success and felt that ‘perceived usefulness’ should be used, instead. However, other published literature  empirically affirmed that both DeLone and McLean’s  and Seddon’s  models exhibited reasonable fits. In other words, system quality, information quality, service quality, user satisfaction, and Seddon’s  perceived usefulness can all be regarded as valid measures of IS success.
According to DeLone and McLean , IS use as a measure of IS success makes sense for voluntary users only. Since PACS utilization is mandated in most healthcare institutions, PACS dependence (which measures the degree to which RTs are dependent on PACS for the execution of their tasks) is considered a more appropriate indicator of PACS success compared to DeLone and McLean’s  “use” or “intention to use”.
Paré et al.  adopted the IS success model to assess PACS success using radiologists, technologists, and clinicians as the study population. To the best of our knowledge, their study was the first of its kind to evaluate PACS success using the IS success model and it provided valuable insights into how to perform IS success studies. However, Paré et al.  did not test the full model with technologists because they felt that technologists interact with PACS in a more limited way. Thus, more in-depth investigations utilizing the perceptions of technologists are needed. Our study was based on the assertion that the full IS success model can be tested using RTs because its constructs, operational definitions, and measurements are properly adapted to the PACS context.
Adopting the viewpoint of DeLone and McLean , the proposed model was regarded as a causal model. The system quality, information quality, and service quality of PACS were pivotal in determining its perceived usefulness and RTs’ satisfaction with PACS. Furthermore, if the utilization of PACS enhanced the RTs’ job performance, they undoubtedly would experience a higher degree of satisfaction with the system and would depend upon it to a greater extent for their daily tasks. Finally, the RTs’ degree of satisfaction also influenced their dependence on PACS, as a greater extent of satisfaction would lead to an increasing dependence on PACS. The research framework justification, research variables, and their relationships in the proposed model are explained in detail in Figure 2.
This study adapted the variable definitions of prior IS success models [16, 37] and modified them for use in the PACS context. The research framework was composed of the following six variables: system quality (in RTs’ PACS usage context, it measures the extent to which PACS is easy to use); information quality (refers to the extent to which the information quality of the output via PACS is sufficient, accurate, and up-to-date); service quality (represents the extent to which PACS performs the service correctly the first time and is able to fulfill its agreements); perceived usefulness (gauges the extent to which RTs believe that utilizing PACS has improved their job performance); user satisfaction (measures the degree of RT’s satisfaction with PACS); and PACS dependence (defined as the degree to which the RT is dependent on PACS for the execution of their tasks). The variables used in this study, their generic IS definition, and PACS-specific definition are shown in Table 2.
The effect of system quality, information quality, and service quality on perceived usefulness
We aimed to study the effect of system quality, information quality, and service quality on the perceived usefulness of PACS. Since RTs are responsible for the correct entry of patient information into PACS and image storage , an easy-to-use PACS was anticipated. Furthermore, previous studies [38, 39] argued that the users’ perception, that useful information would be produced by an IS, was related to the success of the system. Therefore, if the quality of information provided by PACS was deemed superior by RTs, they were more likely to perceive that PACS made a tangible contribution to their job performance. Thus, it was felt that information quality leads to the perceived usefulness of PACS. If the service quality of PACS is poor, the likelihood that RTs will perceive the system as useful will be diminished. If PACS is unreliable (e.g., prolonged downtime), RTs will experience a much lower level of perceived usefulness of the system. Previous literature [7, 16, 30, 36, 40, 41] also supports positive relationships among system quality, information quality, and service quality, and perceived usefulness. Thus, the following three hypotheses were tested: H1: Higher levels of PACS system quality lead to higher levels of perceived usefulness of PACS; H2: Higher levels of PACS information quality lead to higher levels of perceived usefulness of PACS; H3: Higher levels of PACS service quality lead to higher levels of perceived usefulness of PACS.
The effect of system quality, information quality, and service quality on user satisfaction
DeLone and McLean  argued that system quality is an important antecedent to IT users’ satisfaction. A RT is unlikely to be satisfied if he/she has experienced problems in retrieving images or has had to wait a long time to acquire exam-related information via PACS. Furthermore, to carry out their tasks, RTs must acquire the most up-to-date and accurate information on the patients being examined. This implies that the search for patient examination information is one of the RTs’ most frequently cited reasons to use PACS. And it is common for RTs to use PACS to acquire the information they need. Regarding service quality, DeLone and McLean  argue that system quality and information quality may be the most important components for measuring a single system. In addition, most studies agree that service quality, when properly measured, can be lumped with system quality and information quality as constituents of IS success . Besides, user satisfaction may be considered a response to users’ aspiration of service quality for a system . That is, RTs subjectively assess the diverse consequences on a pleasant-unpleasant continuum after using PACS and formulate their own degree of satisfaction or dissatisfaction with PACS. Moreover, PACS can enhance RTs’ ability to obtain patient-related information, which leads to more accurate and comprehensive analyses of patient situations. Several prior studies [16, 26, 36, 40, 42–51] also suggest that system quality, information quality, service quality, and perceived usefulness are antecedents of user satisfaction. Thus, the following four additional hypotheses were tested: H4: Higher levels of PACS system quality lead to higher levels of user satisfaction with PACS; H5: Higher levels of PACS information quality lead to higher levels of user satisfaction with PACS; H6: Higher levels of PACS service quality lead to higher levels of user satisfaction with PACS; H7: Higher levels of perceived usefulness of PACS lead to higher levels of user satisfaction with PACS.
The effect of perceived usefulness and user satisfaction on PACS dependence
PACS may be the RT’s only viable choice for acquiring information pertaining to his or her task if no other means is available for accessing patient information. The more vital the information contained in an IS, the more the RTs will be forced to acquire the essential data via PACS. Furthermore, a key point of DeLone and McLean’s  model is that ‘IS use’ is regarded as an IS success measure. They defined ‘IS use’ as the utilization of IS output , which is viewed as an antecedent to individual impact. In other words, ‘IS use’ significantly influenced the realization of system benefits . Furthermore, DeLone and McLean  argued that user satisfaction influenced ‘IS use’, as higher satisfaction led to greater dependence on an IS [16, 49]. Since PACS utilization is mandatory in most healthcare institutions, it may be more appropriate to employ ‘dependence on PACS’ rather than ‘IS use’ to measure the success of PACS. Previous literature [16, 49] also supports that perceived usefulness and user satisfaction are antecedents of IS dependence. Hence, the following two additional hypotheses were tested: H8: Higher levels of perceived usefulness of PACS lead to higher levels of dependence on PACS; H9: Higher levels of user satisfaction with PACS lead to higher levels of dependence on PACS. The proposed relationships among key constructs and supportive literature are summarized in Table 3.
To assess the dependence of RTs on PACS, a cross-sectional survey was conducted at a large Taiwanese hospital. Information on the study hospital, study units, study users, tasks, processes, and policies regarding PACS are shown in Table 4. Questionnaires were used to collect the perceptions of RTs concerning PACS.
The questionnaire used in this study consisted of two parts. The first part collected respondents’ demographic data, and the second part dealt with their perceptions regarding system quality, information quality, service quality, perceived usefulness, user satisfaction, and PACS dependence. In accordance with Churchill’s  approach to questionnaire generation, our study combined scales from other relevant empirical studies to generate an initial pool of items for each construct. Then, an expert panel comprised of one senior RT, one radiologist, and one researcher (who was specialized in healthcare information management) reviewed these items. Due to language and cultural differences, some of the items from the initial pool were deemed unsuitable and were dropped and others were modified according to suggestions from this panel of experts. System quality, information quality, service quality, perceived usefulness, user satisfaction, and PACS dependence were measured using existing instruments. With the exception of demographic questions, all questions listed were based on a 5-point Likert scale (1 for ‘strongly disagree’ and 5 for ‘strongly agree’).
The instruments for system quality utilized four items adapted from Teo et al. . The information quality constructs were formed using four items from Rai et al. , Teo et al. , and Wang . Service quality was measured in terms of four items adapted from Teo et al.  and Wang . The instrument for perceived usefulness utilized four items based on Iivari . User satisfaction was measured with a four-item scale developed by Chen and Cheng , Teo et al.  and Wang and Liao . The instruments for PACS dependence utilized four items from Rai et al.  and Teo et al. .
A pilot test was conducted to develop these measures, followed by a broader survey. The pilot test was conducted using a convenient sample of 10 RTs, and items were subsequently modified resulting in a revised instrument that warranted further testing. Additional file 1 outlines the final measurement items.
Sample and data collection
Permission from the Institutional Review Board of Chi-Mei Foundation Hospital, Taiwan, was acquired before proceeding with the investigation.
The research model was empirically validated using data collected from a survey. Questionnaires were distributed to 128 RTs in a large medical center including three campuses in southern Taiwan. The study population included every RT in the department of medical imaging listed in the subject hospital’s database, except those selected for the pilot test. From February 1st to February 28th 2013, subjects were asked to voluntarily fill out the paper-and-pencil survey. From the 128 RTs who comprised the study population, a total of 109 usable questionnaires were collected, representing a response rate of 85.16%.
Of the 109 respondents, 49.54% were male and 50.46% were female. More than 87% of the respondents were younger than 40 years of age. In addition, the majority were frontline RTs (88.07%). Most respondents had less than 5 years of work experience (40.37%). More than 68% of the respondents’ had between 1 and 10 years of job experience. Most respondents (68.81%) have 4–9 years of PACS experience, as shown in Table 5.
Furthermore, the majority of RTs’ perceptions about PACS were higher than 3 (based on a 5-point Likert scale), indicating that most RTs held positive perceptions toward PACS, as shown in Table 6.
The proposed model and hypotheses were empirically validated using partial least square (PLS), a component-based structural equation modeling , supported by SmartPLS® 2.0 M3 software . PLS was selected because of the small size of the samples collected . A previous study  suggested a two-stage process for assessing the PLS model structure, including (1) the measurement model and (2) the structural model. The measurement model articulated the relationships between the latent variables and the measured (observed) variables, whereas the structural model articulated the relationships between the exogenous and endogenous latent variables .
The strength of the measurement model was demonstrated by calculating the reliability and validity (including convergent validity and discriminant validity) . The present study conducted three tests  to determine the reliability of constructs in a single instrument including the reliability of items, composite reliability (CR), and Cronbach’s alpha of constructs. Reliability of the items was first gauged by inspecting the loading of each item on a corresponding construct. The results demonstrated reliability scores for all the items over the criterion of 0.707 . Thus, the indicators measuring each construct in the present study all carried sufficient item reliability, as shown in Table 6. In the second step, PLS took into consideration the relationships among constructs when calculating the CR for each construct. The CR of all the constructs in the present study exceeded the criterion of 0.7 . Additional evidence concerning the reliability of the constructs used in this study was acquired by computing Cronbach’s alpha. A value of 0.7 revealed sufficient construct reliability . Based on these criteria, all constructs proposed were characterized by adequate reliability (Table 6). In the third step, PLS computed the average variance extracted (AVE) of each construct based on the degree to which corresponding indicators tapped into the same construct . A figure of 0.5 demonstrated an adequate level of convergent validity for each construct  (Table 6).
Discriminant validity refers to the extent a given construct is different from other constructs . As a rule of thumb, the square root of the AVE of each construct should be higher than the correlation of the specific construct with other constructs in the model  and should be at least 0.5 . The results indicated that none of the inter-correlations of the constructs employed in the study exceeded the square root of the AVE for the construct, as shown in Table 7. Furthermore, the AVE scores of all constructs were larger than 0.5. All criteria demonstrated satisfactory discriminant validity of the model.
With sufficient measurement models, the hypotheses were then tested by inspecting the structural models. After calculating path estimates in the structural model, PLS utilized a bootstrapping technique to acquire the corresponding t-values. Hypothesis testing (resulting from the structural equation modeling analyses) are summarized in Table 8, in which the standardized path coefficients and t-values are shown. The results indicated that all nine hypotheses proposed in this study were significant, with the exception of H5.
As in the interpretation of multiple regression, the R 2 indicates the amount of variance explained by the proposed model. The coefficient and R 2 resulting from the PLS model are demonstrated in Figure 3. In terms of goodness of fit indicators, systems quality, information quality, and service quality accounted for 29.6% of the variance in perceived usefulness. Systems quality, information quality, service quality, and perceived usefulness accounted for 72.7% of the variance in user satisfaction. Perceived usefulness and user satisfaction accounted for 56.4% of the variance in PACS dependence. Furthermore, the global fit measure, GoF, was used to validate the PLS model and was calculated as .
The average AVE = 0.74 and average R 2 = 0.53, thus GoF = 0.63. According to Wetzels et al. , a GoF value of 0.63 (which exceeds the 0.36 threshold for large effect sizes) indicated that our model was valid .
The direct and total effect of user satisfaction on PACS dependence was 0.53. However, the total effect of perceived usefulness on PACS dependence was 0.60. Perceived usefulness, despite showing a weaker direct effect than user satisfaction on PACS dependence, demonstrated a stronger total effect on PACS dependence than that of user satisfaction. Among the three quality-related constructs, information quality had the strongest total effect (0.31) on PACS dependence. The direct, indirect, and total effects of system quality, information quality, service quality, perceived usefulness, and user satisfaction on PACS dependence are detailed in Table 9.
This study proposed and verified a PACS success model from a socio-technical perspective based on prior IS success models. All its constructs, including system quality, information quality, service quality, perceived usefulness, user satisfaction and PACS dependence, were proven to possess adequate psychometric properties and thus can be used as effective measures of PACS success. Although the relationship between service quality and perceived usefulness was not significant, other hypothesized relationships amongst the six measurement parameters of success were all confirmed. Furthermore, similar to prior studies, a socio-technical view [26, 41] can be used to evaluate the success of an IS without placing sole emphasis on technology .
The effect of system quality, information quality, and service quality on perceived usefulness
The results of hypothesis H1 supported the proposition that a positive relationship exists between system quality and perceived usefulness of PACS. Previous studies [16, 30, 36, 40, 45] also reflect similar results. Therefore, the perceived ease of use is a vital indicator of the RTs’ confidence in the ability of PACS to improve their work performance. Paré et al.  argued that users benefit from the adoption of PACS because the system removes the workload of handling physical films. Indeed, prompt access to medical images anywhere/anytime in the healthcare setting is the greatest benefit of all, and the inconvenience of lost or misplaced films  is no longer an issue.
Regarding hypothesis H2, results showed a positive relationship between information quality and perceived usefulness and were consistent with prior studies [36, 41, 52]. Indeed, by integrating PACS images with the patient’s relevant clinical information (e.g., present illness and allergic history) from other HISs, RTs are able to easily obtain the patient’s information during the examination process. The goals of PACS are the management and dissemination of patient images and relevant clinical information. Thus, it is incumbent on the RTs to acquire and utilize images as well as patient-related information from PACS.
In testing hypothesis H3, the results failed to support the proposition. These results differed from prior literature [36, 52]. A plausible explanation might lie in the fact that although PACS provides the RT with obvious conveniences, the acquisition of patient images relies primarily upon the technical ability of the RTs, themselves . Without sufficient know-how regarding basic radiographic techniques, it is not possible to acquire quality images even with PACS. Furthermore, during the non-PACS era, RTs did not need to upload images to PACS after examining the patients, a process which some RTs might regard as an extra burden. Therefore, despite the excellent services being provided by PACS, it is still possible that RTs consider such conveniences a burden.
The effect of system quality, information quality, service quality, and perceived usefulness on user satisfaction
With regards to H4, the results support a positive correlation between system quality and user satisfaction with PACS and the results are consistent with prior studies [16, 26, 36, 44, 49, 50]. By using PACS, RTs are able to execute their tasks in a single action because the required protocols for different exams are predefined. This represents one of the greatest advantages made possible by PACS.
The results of hypothesis H5 support the proposition that a positive relationship exists between information quality and user satisfaction. These results are in line with previous studies [36, 44, 48, 52]. As DeLone and McLean  pointed out, user satisfaction might be regarded as the user’s anticipation of different properties of an IS: system quality, information quality, and service quality. The integration of HIS, Radiology Information Systems, and PACS enhances productivity as well as streamlines the acquisition of, and access to, patient information  such as subjective patient symptoms, objective clinical observations, diagnoses, and allergies. RTs are, therefore, likely to experience a greater degree of satisfaction with the comprehensive information provided by PACS.
In testing hypothesis H6, a positive relationship was found between service quality and user satisfaction. The results were in line with previous studies [36, 44, 48, 52]. In the subject hospital, the PACS was outsourced to a software provider who was responsible for the PACS servers. The software provider was obligated to keep PACS in a non-stop working state due to the critical demands of healthcare. Overall, PACS has performed well in the subject hospital. RTs do not usually experience interruptions in their work caused by PACS downtime in the process of carrying out their routine tasks. This may explain why service quality is such an important factor in their expression of satisfaction with the system.
In testing hypothesis H7, results demonstrated a positive association between perceived usefulness and user satisfaction. These results indicated that usefulness plays a vital role in increasing the RT’s satisfaction with PACS. The results were consistent with the previous literature [26, 36, 44, 45, 48, 52]. Indeed, the ability to easily produce and manage images is one of the greatest advantages when utilizing the system, which makes it possible to avoid the inconvenience of manually handling films. It is obvious that PACS reduces RTs’ workloads and increases in their degree of satisfaction.
The effect of perceived usefulness and user satisfaction on PACS dependence
In testing hypothesis H8, results showed a positive relationship between perceived usefulness and PACS dependence. The results were consistent with prior studies [12, 16, 45]. Despite the mixed results regarding the influences of perceived usefulness and perceived ease of use on outcome behavior [63, 64], our study was consistent with the findings related to the acceptance of technology in healthcare settings . The use of the technology depends more on its usefulness than on its ease of use. It does not really matter how difficult PACS is to use; RTs will assimilate PACS into their routine if it is shown to be useful and helpful on the job and beneficial to patients. When PACS is already a part of their work procedure, convincing them to continue using the system is easier.
Regarding hypothesis H9, results showed a positive relationship between user satisfaction and PACS dependence and the results were consistent with prior studies . The existing literature indicates that user satisfaction contributes to loyalty to a product/service [66, 67] or dependence on an IS . Thus, to further encourage the utilization of PACS, the hospital could enhance the RT’s satisfaction with PACS by strengthening its functions and versatility.
A comparison of the results with previous studies
Compared with prior studies, Paré et al.’s  study accounted for 40.4% of the variance in net benefits, 60.8% of the variance in user satisfaction, and 48.5% of the variance in system continuance intention. Our model explained 29.6%, 72.7%, and 56.4% of the variance in perceived usefulness, user satisfaction, and PACS dependence, respectively. Furthermore, the “acceptance” is one of the measures that can be used to assess IS success . Among these differing IS success measures, TAM and its related model (such as UTAUT) are the most commonly adopted theoretical bases in the literature . We also compared our study with other studies which assessed PACS acceptance. These studies explained about 41%-49% of variance of PACS acceptance, as shown in Table 10.
By modifying prior IS success models [3, 4], we constructed a model to evaluate the success of PACS from a socio-technical perspective by substituting ‘use’ with ‘perceived usefulness’  and adding PACS dependence to the equation. A survey of 109 RT questionnaires was used to test the hypotheses of the modified model. The model successfully predicted the dependence on PACS (R 2 = 56.4%). However, our results contradicted the notion, derived from other studies, that service quality does not influence an RT’s perceived usefulness of PACS.
Implications for research
The socio-technical model proposed in this study permitted simultaneous evaluation of the influences of system quality, information quality, and service quality, thus facilitating a more holistic approach to the evaluation of the success of PACS. In addition, most prior IS studies have focused on the associations among IS quality, satisfaction, and intention . Furthermore, IS dependence, as one kind of system usage measure [16, 68, 69], has been less explored in settings where IS utilization is mandatory, especially in healthcare institutions. Therefore, the inclusion of the PACS dependence construct into the research framework provides a more complete framework for relevant constructs and their causal relationships. Finally, our model predicts user satisfaction better than PACS dependence. Thus, future studies could explore other dependent variables in order to obtain more knowledge on the topic of IS success.
Implications for practice
Perceived usefulness and user satisfaction are vital to the success of PACS. Both were found to be significant antecedents to PACS dependence. Since perceived usefulness is the most important determinant of PACS dependence (total effect = 0.60), it is incumbent on managers to enhance the RTs’ dependence on PACS by strengthening their beliefs regarding how PACS can improve their performance and effectiveness.
In addition, perceived usefulness significantly mediates the effects imposed by system quality and information quality on PACS dependence. That is, the perceived usefulness of PACS reinforces user satisfaction, which, in turn, results in PACS dependence. The strategy for the target hospital is to increase perceived usefulness of PACS which can be developed by looking at its antecedents of system quality and information quality. This suggests that system quality and information quality must be provided to make RTs feel that PACS is helpful to them in their work routines.
Besides perceived usefulness, system quality is the second most important predictor of user satisfaction (total effect = 0.36). Consequently, enhancing the system quality increases the RTs’ satisfaction with PACS. Ease of use is the most important criterion of system quality. Thus, positive results regarding ease of use result in a positive impact on user satisfaction of PACS. Hospitals have an obligation to provide a system that is easy to use in order to enhance the RTs’ satisfaction. Furthermore it is also incumbent upon managers to improve the attributes of the system, such as ease of use, if it is to be more appealing to RTs.
Information quality is also a major factor affecting users’ satisfaction. Therefore, PACS should provide more relevant information to fulfill user requirements. This study suggests that when RTs felt that PACS was able to supply helpful information for carrying out their tasks, they are more likely to utilize it and experience an enhanced degree of satisfaction.
Most importantly, when RTs perceive that the information supplied by PACS is sufficient, accurate, and up-to-date for their tasks, they exhibit a higher degree of dependence on PACS. In addition, service quality also plays vital roles in the equation. Thus, managers should ensure that PACS provides reliable service with a superior user interface.
In conclusion, the findings of this study indicate that managers must improve the attributes of PACS because at the onset of deployment, an RT will establish his or her own personal perception regarding its quality (system quality, information quality, and service quality). When these perceptions are either reinforced or refuted, the RT will either be satisfied or dissatisfied with the system, and will also develop a perception of usefulness or lack of usefulness of PACS.
Contributions and limitations
The results of the present study contribute to the field in a number of ways. In addition to advancing our knowledge regarding a method to evaluate the success of PACS, the results also aid in the understanding of the RTs’ dependence on PACS. In addition, our findings strongly support the appropriateness of the DeLone and McLean [3, 4] model for understanding the determinants of success of PACS.
Our study had several limitations. The samples were collected in only one hospital within one country. Consequently, inferences to larger populations cannot be safely made. Future research should expand the present study’s findings by employing broader and more representative samples. Furthermore, a cross-cultural empirical study using a larger sample is essential for a more accurate generalization of the proposed model. In addition, although our model was based on the IS success model (a causal model asserted by DeLone and McLean), one must be careful when making such a causal declaration .
PACS is a new technology and generalization to other technologies is limited. Although our model predicted the success of PACS, other success measures such as benefits/return on investment, sustainability, global job satisfaction, and patient safety were not included in this study due to time and cost limitations. Due to the small sample size (n = 109), our results may suffer from insufficient statistical power. The definitions of constructs used in this study were adapted to RTs’ PACS usage context, which may differ from previous IS success studies. Further refinement of the operational definitions of constructs and measures is needed to better explain the constructs used. Our study used PACS dependence as a surrogate of system use which may not explain the variance completely. Although the relationship between user satisfaction and PACS dependence was confirmed in this study, there may be a bidirectional causal relationship between them. Further studies are needed to investigate the reciprocal relationship. Finally, our results may be limited by the depth of understanding permitted by closed-ended survey responses and the correspondence between self-reported perceptions and “reality.”
Goldschmidt PG: HIT and MIS: implications of health information technology and medical information systems. Commun ACM. 2005, 48 (10): 68-74. 10.1145/1089107.1089141.
Agarwal R, Gao G, DesRoches C, Jha AK: Research commentary–the digital transformation of healthcare: current status and the road ahead. Inform Syst Res. 2010, 21 (4): 796-809. 10.1287/isre.1100.0327.
DeLone WH, McLean ER: Information systems success: the quest for the dependent variable. Inform Syst Res. 1992, 3 (1): 60-95. 10.1287/isre.3.1.60.
DeLone WD, McLean ER: The DeLone and McLean model of information systems success: a ten-year update. J Manage Inform Syst. 2003, 19 (4): 9-30.
Kaplan B: Addressing organizational issues into the evaluation of medical systems. J Am Med Inform Assn. 1997, 4 (2): 94-101. 10.1136/jamia.1997.0040094.
Petter S, DeLone W, McLean ER: The past, present, and future of “IS Success”. J Assoc Inf Syst. 2012, 13: 5-
Seddon PB: A respecification and extension of the DeLone and McLean model of IS success. Inform Syst Res. 1997, 8 (3): 240-253. 10.1287/isre.8.3.240.
Ayal M, Seidmann A: An empirical investigation of the value of integrating enterprise information systems: the case of medical imaging informatics. J Manage Inform Syst. 2009, 26 (2): 43-68. 10.2753/MIS0742-1222260203.
Hurlen P, Østbye T, Borthne A, Gulbrandsen P: Introducing PACS to the late majority. A longitudinal study. J Digit Imaging. 2010, 23 (1): 87-94. 10.1007/s10278-008-9160-x.
Popovič A, Hackney R, Coelho PS, Jaklič J: Towards business intelligence systems success: effects of maturity and culture on analytical decision making. Decis Support Syst. 2012, 54 (1): 729-739. 10.1016/j.dss.2012.08.017.
Aldosari B: User acceptance of a picture archiving and communication system (PACS) in a Saudi Arabian hospital radiology department. BMC Med Inform Decis. 2012, 12 (1): 44-10.1186/1472-6947-12-44.
Duyck P, Pynoo B, Devolder P, Voet T, Adang L, Ovaere D, Vercruysse J: Monitoring the PACS Implementation Process in a Large University Hospital—Discrepancies Between Radiologists and Physicians. J Digit Imaging. 2010, 23 (1): 73-80. 10.1007/s10278-008-9163-7.
Sørebø O, Fuglseth AM: Proceedings of the NOKOBIT 2012: 19–21 November 2012. Information Systems Success and Tourism Employees’ Use of a Payment System: The Influence of User Motivation, Management Attitude and Ease of Use. 2012, Bodø, Norway: Universitetet i Nordland
Michel-Verkerke MB: Information quality of a nursing information system depends on the nurses: a combined quantitative and qualitative evaluation. Int J Med Inform. 2012, 81 (10): 662-673. 10.1016/j.ijmedinf.2012.07.006.
Chan L, Trambert M, Kywi A, Hartzman S: PACS in Private practice–effect on profits and productivity. J Digit Imaging. 2002, 15: 131-136. 10.1007/s10278-002-5019-8.
Rai A, Lang SS, Welker RB: Assessing the validity of IS success models: an empirical test and theoretical analysis. Inform Syst Res. 2002, 13 (1): 50-69. 10.1287/isre.18.104.22.168.
Chang IC, Hwang HG, Yen DC, Lian JW: Critical factors for adopting PACS in Taiwan: views of radiology department directors. Decis Support Syst. 2006, 42 (2): 1042-1053. 10.1016/j.dss.2005.08.007.
Department of Health: An Investigation of the Current Status of Computerized Medical Records Among Taiwanese Hospitals (in Chinese). 2005, Taipei: Department of Health, Executive Yuan, Taiwan (R.O.C.)
Department of Health: Bulletin of Electronic Medical Records. 2013, Taipei: Department of Health, Executive Yuan, Taiwan (R.O.C.)
Becker SH, Arenson RL: Costs and benefits of picture archiving and communication systems. J Am Med Inform Assn. 1994, 1 (5): 361-371. 10.1136/jamia.1994.95153424.
Allison SA, Sweet CF, Beall DP, Lewis TE, Monroe T: Department of defense picture archiving and communication system acceptance testing: results and identification of problem components. J Digit Imaging. 2005, 18 (3): 203-208. 10.1007/s10278-005-5164-y.
Hood MN, Scott H: Introduction to picture archive and communication systems. J Radiol Nurs. 2006, 25 (3): 69-74. 10.1016/j.jradnu.2006.06.003.
Maass M, Eriksson O: Proceedings of the 39th Hawaii International Conference on System Sciences. Challenges in the Adoption of Medical Information Systems. 2006, 95b-95b.
Paré G, Trudel MC: Knowledge barriers to PACS adoption and implementation in hospitals. Int J Med Inform. 2007, 76 (1): 22-33. 10.1016/j.ijmedinf.2006.01.004.
van der Meijden MJ, Tange HJ, Troost J, Hasman A: Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assn. 2003, 10 (3): 235-243. 10.1197/jamia.M1094.
Paré G, Lepanto L, Aubry D, Sicotte C: Toward a multidimensional assessment of picture archiving and communication system success. Int J Technol Assess. 2005, 21 (04): 471-479.
Lepanto L, Paré G, Aubry D, Robillard P, Lesage J: Impact of PACS on dictation turnaround time and productivity. J Digital Imaging. 2006, 19 (1): 92-97. 10.1007/s10278-005-9245-8.
Duyck P, Pynoo B, Devolder P, Voet T, Adang L, Vercruysse J: User acceptance of a picture archiving and communication system. Applying the unified theory of acceptance and use of technology in a radiological setting. Method Inform Med. 2008, 47 (2): 149-156.
Karahanna E, Straub DW, Chervany NL: Information technology adoption across time: A cross-sectional comparison of pre-adoption and post-adoption beliefs. Mis Q. 1999, 23 (2): 183-213. 10.2307/249751.
Davis FD: Perceived usefulness, perceived ease of use, and user acceptance of information technology. Mis Q. 1989, 13 (3): 319-340. 10.2307/249008.
Holden RJ, Karsh BT: The technology acceptance model: its past and its future in health care. J Biomed Inform. 2010, 43 (1): 159-172. 10.1016/j.jbi.2009.07.002.
Garrity EJ, Sanders GL: Introduction to information systems success measurement. Information systems success measurement. Edited by: Edward JG, Sanders GL. 1998, Hershey, USA: IGI Publishing, 1-12. ISBN 1
Porter ME: Competitive Advantage. 1985, New Yord, USA: Free Press
Porter ME, Millar VE: How information gives you competitive advantage. Harvard Bus Rev. 1985, 63 (4): 149-160.
Wilcox RB, Fallano J, Shannon KJ, Carrino JA, Sinclair J, Khorasani R: Picture archiving and communication system and its impact on image viewing in physical therapy practice. J Digit Imaging. 2006, 19 (4): 346-350. 10.1007/s10278-006-0586-8.
Wang YS: Assessing e-commerce systems success: a respecification and validation of the DeLone and McLean model of IS success. Inform Syst J. 2008, 18 (5): 529-557. 10.1111/j.1365-2575.2007.00268.x.
Lin HF, Lee GG: Determinants of success for online communities: an empirical study. Behav & Inform Technol. 2006, 25 (6): 479-488. 10.1080/01449290500330422.
Haga WJ, Zviran M: Information systems effectiveness: research designs for causal inference. Inform Syst J. 1994, 4 (2): 141-166. 10.1111/j.1365-2575.1994.tb00048.x.
Munro MC, Gordon BD: Determining management information needs: a comparison of methods. Mis Q. 1977, 1 (2): 55-67. 10.2307/249168.
Seddon P, Kiew MY: A partial test and development of DeLone and McLean’s model of IS success. Aust J Inform Syst. 1996, 4 (1): 90-109.
Wu JH, Wang YM: Measuring KMS success: a respecification of the DeLone and McLean’s model. Inform Manage-Amster. 2006, 43 (6): 728-739. 10.1016/j.im.2006.05.002.
Bharati P, Chaudhury A: An empirical investigation of decision-making satisfaction in web-based decision support systems. Decis Support Syst. 2004, 37 (2): 187-197. 10.1016/S0167-9236(03)00006-X.
Bhattacherjee A: Understanding information systems continuance: an expectation-confirmation model. Mis Q. 2001, 25 (3): 351-370. 10.2307/3250921.
Chen CWD, Cheng CYJ: Understanding consumer intention in online shopping: a respecification and validation of the DeLone and McLean model. Behav Inform Technol. 2009, 28 (4): 335-345. 10.1080/01449290701850111.
Hsieh J, Wang W: Explaining employees’ extended use of complex information systems. Eur J Inform Syst. 2007, 16 (3): 216-227. 10.1057/palgrave.ejis.3000663.
Kettinger WJ, Lee CC: Perceived service quality and user satisfaction with the information services function. Decision Sci. 1994, 25 (5–6): 737-766.
Kim G, Shin B, Lee HG: Understanding dynamics between initial trust and usage intentions of mobile banking. Inform Syst J. 2009, 19 (3): 283-311. 10.1111/j.1365-2575.2007.00269.x.
Kulkarni UR, Ravindran S, Freeze R: A knowledge management success model: theoretical development and empirical validation. J Manage Inform Syst. 2007, 23 (3): 309-347. 10.2753/MIS0742-1222230311.
McGill T, Hobbs V, Klobas J: User developed applications and information systems success: a test of DeLone and McLean’s model. Inform Res Manag J. 2003, 16 (1): 24-45.
Wang YS, Liao YW: Assessing eGovernment systems success: a validation of the DeLone and McLean model of information systems success. Gov Inform Q. 2008, 25 (4): 717-733. 10.1016/j.giq.2007.06.002.
Yang ZL, Cai SH, Zhou Z, Zhou N: Development and validation of an instrument to measure user perceived service quality of information presenting Web portals. Inform Manage-Amster. 2005, 42 (4): 575-589.
Landrum H, Prybutok VR: A service quality and success model for the information service industry. Eur J Oper Res. 2004, 156 (3): 628-642. 10.1016/S0377-2217(03)00125-5.
Churchill GA: A paradigm for developing better measures of marketing constructs. J Marketing Res. 1979, 16 (1): 64-73. 10.2307/3150876.
Teo TSH, Srivastava SC, Jiang LI: Trust and electronic government success: an empirical study. J Manage Inform Syst. 2008, 25 (3): 99-131. 10.2753/MIS0742-1222250303.
Iivari J: An empirical test of the DeLone-McLean model of information system success. ACM SIGMIS Database. 2005, 36 (2): 8-27. 10.1145/1066149.1066152.
Chin WW: The partial least squares approach to structural equation modeling. Modern methods for business research. Edited by: Marcoulides GA. 1998, NJ: Lawrence Erlbaum Associates, 295-336.
Ringle CM, Wende S, Will A: 2.0 Beta edn. SmartPLS (2M3). 2005, Hamburg, Germany
Henseler J, Ringle CM, Sinkovics RR: The use of partial least squares path modeling in international marketing. Adv Int Marketing. 2009, 20: 277-319.
Fornell C, Larcker DF: Evaluating structural equation models with unobservable variables and measurement error. J Marketing Res. 1981, 18 (1): 39-50. 10.2307/3151312.
Wetzels M, Odekerken-Schröder G, van Oppen C: Using PLS path modeling for assessing hierarchical construct models: guidelines and empirical illustration. Mis Q. 2009, 33 (1): 177-195.
Berg M: Patient care information systems and health care work: a sociotechnical approach. Int J Med Inform. 1999, 55 (2): 87-101. 10.1016/S1386-5056(99)00011-8.
Tzeng WS, Kuo KM, Liu CF, Yao HC, Chen CY, Lin HW: Managing repeat digital radiography images—a systematic approach and improvement. J Med Syst. 2011, 36 (4): 2697-2704.
Holden RJ, Brown RL, Scanlon MC, Karsh BT: Modeling nurses’ acceptance of bar coded medication administration technology at a pediatric hospital. J Am Med Inform Assn. 2012, 19 (6): 1050-1058. 10.1136/amiajnl-2011-000754.
Holden RJ, Brown RL, Scanlon MC, Karsh BT: Pharmacy workers’ perceptions and acceptance of bar-coded medication technology in a pediatric hospital. Res Soc Admin Ph. 2012, 8 (6): 509-522. 10.1016/j.sapharm.2012.01.004.
Aggelidis VP, Chatzoglou PD: Using a modified technology acceptance model in hospitals. Int J Med Inform. 2009, 78 (2): 115-126. 10.1016/j.ijmedinf.2008.06.006.
Caruana A, Fenech N: The effect of perceived value and overall satisfaction on loyalty: a study among dental patients. J Med Marketing. 2005, 5 (3): 245-255. 10.1057/palgrave.jmm.5040236.
Lin HH, Wang YS: An examination of the determinants of customer loyalty in mobile commerce contexts. Inform Manage-Amster. 2006, 43 (3): 271-282. 10.1016/j.im.2005.08.001.
Burton-Jones A: New Perspectives on the System Usage Construct. Doctoral dissertation. 2005, Atlanta, GA: Georgia State University
Goodhue DL, Thompson RL: Task-technology fit and individual-performance. Mis Q. 1995, 19 (2): 213-236. 10.2307/249689.
Mulaik SA: Linear Causal Modeling with Structural Equations. 2009, New York, USA: Chapman & Hall/CRC
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6947/13/109/prepub
This work was partially supported by National Science Council of Taiwan (R.O.C.) under grant No. NSC-101-2410-H-214-008 and I-Shou University under grant No. ISU 101-S-04.
The authors declare that they have no competing interests.
WS and KM conceived of this study and participated in its design, carried out the study, performed the statistical analysis, and drafted the manuscript. HW and TY helped to draft the manuscript and performed the statistical analysis. All authors read and approved the final manuscript.
Electronic supplementary material
About this article
Cite this article
Tzeng, WS., Kuo, KM., Lin, HW. et al. A Socio-technical assessment of the success of picture archiving and communication systems: the radiology technologist’s perspective. BMC Med Inform Decis Mak 13, 109 (2013). https://doi.org/10.1186/1472-6947-13-109