To our knowledge, this study is the first attempt to assess the factor validity and invariance of an anxiety measure used in decision aid trials. We used cognitive interviewing to refine the anxiety items, and we used factor analyses to evaluate factor validity and invariance. We also demonstrated the internal consistency reliability, the construct validity, and the discriminate validity by using various psychometric testing methods. Our findings from CFA support the use of a 1–factor, 6-item general anxiety measure for the decision to undergo PCS as a potential moderator in PCS decision aid studies that is appropriate to be used in private and public settings. The validity of the PSA and DRE subscales is uncertain.
Our results also provide partial support for the use of a 1–factor PSA anxiety measure. Although the measure was non-invariant across clinics, this result may not be clinically meaningful in this instance because the factor loading patterns were similar in direction and only slightly different in magnitude. Moreover, the small sample size for the public clinic (n = 149) may have affected the accuracy and precision of our measurement invariance testing . In addition, the factor loading estimates may be biased due to truncated distributions for the private clinic sample: categorical item frequencies indicated that men at the private clinic were less likely to report “no” (highest anxiety category) than men at the public clinic. This difference could be due to their history of PSA testing: men at the private clinic were more likely to report prior PSA testing than men at the public clinic were (70.4% versus 30.9%).
Our study results support the use of structural equation modeling techniques when psychometric properties of measures are being evaluated. Although our initial tests for internal consistency reliability and discriminant and construct validity indicated some evidence for the 3–factor model, factor analyses indicated a high degree of inter–correlation between items across factors. There was no evidence of discriminant validity to indicate that the items were three separate subscales. Many items loaded on factors other than those proposed, and there was a high degree of method variance. Some items seemed to be mostly related to method variance, some were mostly related to content, and some seemed to share variance with factors that were not conceptually identifiable.
The reported low levels of anxiety at both clinics may also have contributed to the lack of fit for the 3-factor PCS anxiety measure. Low variability can cause difficulty in distinguishing between factors. The low anxiety levels might be due to the prevalence of recent PSA testing. The low levels could also be due to the nature of the decision: men may not be anxious in general about PCS because messages about screening are that it is almost always beneficial .
On the basis of our results, we can suggest recommendations for future development of decision-specific anxiety measures. First, our findings that two of the three PCS subscales were non-invariant across the two samples that varied by clinic setting exemplify the importance to evaluate measurement invariance before comparing scale scores across samples. When measurement non-invariance is found, there may be indication that the samples differ in their underlying meaning of the anxiety construct. Therefore, comparisons of the scores across the samples may not be meaningful. Second, developing decision-specific measures may need more formative work (i.e., cognitive testing or focus groups) to explore the anxiety specifically related to the healthcare decision. For PCS, it may be necessary to understand what makes men anxious about the screening decision and the testing procedures: some men may be worried about the type of tests (blood drawn with a needle or rectal examination), while others may be anxious about the test results or accuracy. Third, we suggest avoiding using the same wording across subscales to minimize method variance. Finally, we recommend not to include negatively framed items in factor analyses. During cognitive interviewing, we discarded one negatively framed item (“calm”) that men did not interpret as the opposite of being anxious However, we included one negatively framed item (“relaxed”) in each subscale (3 items total) in our PCS anxiety measure. Two of these three negatively framed items loaded as a separate factor and not on the respective subscale factor.
Our findings have several limitations. First, we did not measure situational (state) anxiety or underlying (trait) anxiety. It would be helpful to know how PCS anxiety, a context–specific anxiety, is different than one’s general anxiety. Although the comparison groups were based on clinic site, one serving primarily privately insured patients and the other primarily publicly insured patients, there were statistically significant differences between the men at each clinic (i.e., race/ethnicity, recent history of PSA testing, education, and insurance status). Our results may not be generalizable to other private or publicly funded clinics with different patient compositions. Additionally, given the high percentage of white men at the private clinic and the high percentage of black men at the public clinic, the group comparison results may also be due to differences in race/ethnicity as well as factors such as insurance status. Future validity testing should explore differences in race/ethnicity with populations of the same insurance and socio-economic status. Finally, we used a well validated anxiety measure as a starting point to develop anxiety items specific for prostate cancer screening and used cognitive testing to verify the initial content validity of the items. To develop unique items related to aspects about the healthcare decision, other qualitative methods like using focus groups to find out what people find anxious about healthcare decisions may provide a better insight for content domain and to generate a pool of items.