Participants
Practicing physical therapists were surveyed at two time points as part of the ongoing evaluation of the developing website. The first survey was conducted in 2012 and included questions about ease of use, knowledge acquisition, and clinical reasoning. Additional questions were added to the 2013 user study, including questions about internet use and clinical decision making in practice. Between the first survey in 2012 and the second survey in 2013, PTNow.org was enhanced with:
More links to outcome measures and psychometric information
Search ability using G-code categories for functional limitation reporting
New clinical summaries
Full-text clinical practice guidelines
Participants were recruited via email in the summers of 2012 and 2013. PTs who attended national PT conferences in 2011 and 2012 who provided PTNow their contact information were initially emailed. Additional participants were recruited by purposive sampling including Catherine Worthingham Fellows of the APTA (highest honor accorded to leaders in the field, who are typically consulted on association initiatives), members of the boards of directors of the Connecticut and Massachusetts state chapters of APTA, PT professional education faculty in Connecticut, and PTs certified by the American Board of Physical Therapy Specialties (ABPTS). PTs who indicated interest were sent a consent letter via email, which indicated that initiation of the survey signified consent. The letter contained links to the survey. On entering the online survey, participants were given a unique identification number. The study was approved by the Rutgers University Institutional Review Board.
Patient scenarios
Once consented, participants were provided with links to four possible patient scenarios with the companion survey: benign paroxysmal positional vertigo (BPPV), chronic obstructive pulmonary disease (COPD), Parkinson disease (PD), and total knee arthroplasty (TKA). These scenarios were selected because they were the first clinical summaries posted on the PTNow.org site. The scenarios were written by one of the authors (JED) and reviewed for consistency by a second author (TJM). They contained comparable information and detail. Participants were asked to complete one or more patient scenarios and to use the clinical summaries on the PTNow.org site to answer corresponding questions. Each scenario was designed to take between 20 and 60 min to complete.
Patient scenarios consisted of a hypothetical case with a diagnosis of BPPV, PD, COPD, or TKA, and the participant was instructed to go to the PTNow.org site to answer questions related to the management of the case. These scenarios contained two parts. First, the patient was briefly described, and the study participant was asked to use PTNow.org to indicate how they would interview and examine the patient. After completing the knowledge acquisition questions, participants were then provided with information about examination results, and they were asked a set of clinical reasoning questions about direct patient care, patient/family education, and prognosis. To find out the format of the clinical summary that participants naturally selected (full clinical summary, quick takes and portable clinical summary), a forced choice was not imposed. An example of a patient scenario is below (see Additional file 1 for all of the patient scenarios and questions):
You are working as a home health care physical therapist, and you have just been assigned the case of a 65-year-old woman with PD (diagnosed 5 years ago and now in stage II of the Hoehn and Yahr Scale) who sustained a left hip fracture from a fall. She had her hip pinned and now is having some difficulties with bed mobility, transitional movements, ambulation in the home, and elevations. Her goals are for independence with bed mobility and with ambulation at home and in the community and for improved speed of movement. To formulate your examination strategy, you will use the information that you find in PTNow.org and the relevant resources in the clinical summary.
Validity
Validity was evaluated based on responses to the clinical scenarios. Content validity captures whether the universe of information is represented and is specific to the content universe as defined by the researcher [17]. Content validity was operationally defined as the participant’s ratings about whether the information in the clinical summary was complete and useful in answering clinical questions. The responses were obtained exclusively from expert physical therapists who were clinical specialists (a certification based on years of experience and completion of an exam, awarded by the American Board of Physical Therapy Specialties [ABPTS]) or who had more than 5 years of clinical experience in practice.
The clinical reasoning construct was based on cognitive flexibility theory, wherein knowledge from different concepts and perspectives is re-constructed into an ensemble used to solve the current problem [18]. The “problem” in this study was the addressing of the questions associated with the clinical scenario. Construct validity was operationally defined as the response to knowledge and clinical reasoning questions based on using PTNow.org. Knowledge acquisition included questions related to patient interview and examination techniques as part of patient care. This construct captured general information that was required to answer the patient case questions; this information could be obtained directly from the clinical summary as well as the clinician’s prior knowledge. Knowledge acquisition responses were coded as cut and paste, paraphrase, or prior knowledge. Cut-and-paste responses were identical to the content in the clinical summary, paraphrase responses were similar to the content and organization of the clinical summary, and prior knowledge responses provided information that was not contained in the clinical summary or occurred when participants indicated that they did not use the clinical summary to answer questions in the scenario.
Clinical reasoning questions were on the topics of direct patient care, patient/family education, and prognosis based on examination findings provided in the scenario. Clinical reasoning questions required a level of synthesis and interpretation and were also evaluated for accuracy [19]. Coding for clinical reasoning questions was based on: 1) Accuracy, determined by the investigator based on a priori responses, 2) Completeness, which required a response and rationale, and 3) Used the PTNow site material in the response. Responses were coded as cut and paste, paraphrase, and prior knowledge, in the same manner as described above. Coding was checked for agreement between investigators WR and JD. In the event of a discrepancy the investigators discussed the issue and arrived at a consensus.
Usability
Usability was assessed based on participants self-report. Participants rated their user experience on the site by responding to questions from the System Usability Scale (SUS) [20]. Participants were asked to rate on a five-point Likert scale (1 = completely disagree, 5 = completely agree) on the usability and content of each clinical summary format. Site usability was defined based on whether the information was (a) easy to find (navigation), (b) easy to interpret, (c) useful in answering questions, and (d) complete. In addition, open-ended questions were available for participants to discuss usability, completeness of the clinical summary, new information learned, and recommendations. Participants also responded to questions related to demographics, background, and their use of the internet.
Data analysis
The demographic and survey responses data were analyzed descriptively, including frequencies, means, and standard deviations. Content validity based on the ratings of the clinical summary being complete and useful in answering the patient case was supported if the average ratings exceeded 4 out of 5. Construct validity was considered supported if it surpassed a threshold of two thirds of the responses (66 %) used the PTNow.org site, to answer the questions in the patient case. An ANOVA using SPSS Version 20.0 was conducted with a Dunnett T3 post-hoc analysis to investigate the differences in usability of the three formats of the clinical summary (full, quick takes and portable). A repeated measures ANOVA was not conducted because not all participants responded to each question and the data set, therefore, had to be treated as independent. An alpha level of .05 was set for all analysis.