Skip to main content

Leveraging web-based prediction calculators to set patient expectations for elective spine surgery: a qualitative study to inform implementation



Prediction calculators can help set outcomes expectations following orthopaedic surgery, however effective implementation strategies for these tools are unknown. This study evaluated provider and patient perspectives on clinical implementation of web-based prediction calculators developed using national prospective spine surgery registry data from the Quality Outcomes Database.


We conducted semi-structured interviews in two health systems, Vanderbilt University Medical Center (VUMC) and Duke University Health System (DUHS) of orthopedic and neurosurgery health care providers (VUMC: n = 19; DUHS: n = 6), health care administrators (VUMC: n = 9; DUHS: n = 9), and patients undergoing elective spine surgery (VUMC: n = 16). Qualitative template analysis was used to analyze interview data, with a focus on end-user perspectives regarding clinical implementation of web-based prediction tools.


Health care providers, administrators and patients overwhelmingly supported the use of the calculators to help set realistic expectations for surgical outcomes. Some clinicians had questions about the validity and applicability of the calculators in their patient population. A consensus was that the calculators needed seamless integration into clinical workflows, but there was little agreement on best methods for selecting which patients to complete the calculators, timing, and mode of completion. Many interviewees expressed concerns that calculator results could influence payers, or expose risk of liability. Few patients expressed concerns over additional survey burden if they understood that the information would directly inform their care.


Interviewees had a largely positive opinion of the calculators, believing they could aid in discussions about expectations for pain and functional recovery after spine surgery. No single implementation strategy is likely to be successful, and strategies vary, even within the same healthcare system. Patients should be well-informed of how responses will be used to deliver better care, and concerns over how the calculators could impact payment and liability should be addressed prior to use. Future research is necessary to determine whether use of calculators improves management and outcomes for people seeking a surgical consult for spine pain.

Peer Review reports


Spine-related pain is among the largest drivers of disability and health care costs in the United States [1, 2]. It consistently ranks at or near the top in global reports of disease burden, and data suggest this burden is worsening. In 2016, US health care spending on spine pain was an estimated $134.5 billion (95% CI, $122.4-$146.9 billion), higher than spending attributed to any other health condition that year [1]. Surgery for spine pain is common, growing in incidence [3,4,5], and accounts for a significant proportion of spine-related health care costs.

Patient satisfaction has become an increasingly important metric by which patients and health care payers like Medicare and Medicaid judge the value of spine surgery [6,7,8]. Although most patients undergoing spine surgery experience significant benefits, recent work has found that up to 28% of patients are dissatisfied with their surgery despite achieving clinically relevant improvements in pain and function [9,10,11]. Some of the most important factors driving satisfaction after spine surgery include meeting expectations for return to work and return to previous physical activity [12,13,14,15]. Thus, one way to improve satisfaction with surgery is to ensure patients have realistic, evidence-informed post-surgical expectations during the decision-making process [16, 17].

Surgical prediction tools have been developed to guide pre-surgical counseling on patient-specific expectations for surgical outcomes [18]. These tools use patient-level characteristics to determine probable outcomes of surgery across domains such as adverse events (e.g. re-admission)[19, 20] and patient-centered outcomes (e.g. pain, disability) [21,22,23,24,25,26]. Recently, calculators were developed and internally validated using national data from the Quality Outcomes Database (QOD) [14, 22, 23, 27, 28]. These web-based tools provide individualized risk-adjusted postoperative projections for pain intensity, disability, quality of life, satisfaction, and return to work in patients undergoing elective lumbar and cervical spine surgeries.

Despite their potential value, health care providers have limited guidance on how to best implement prediction tools to facilitate pre-surgical counseling. Therefore, the objective of this study was to conduct interviews with health care providers, administrators, information technology (IT) professionals, and patients to assess barriers, opportunities, and optimal strategies for prediction tool implementation. In particular, we were interested in evaluating topics related to user characteristics, calculator content and interface, workflow integration, organizational culture, and external regulations that would inform implementation strategies for the QOD calculators as well as other prediction tools. We used the Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems [29] as a framework to inform interview content. This 8-dimensional model is specifically designed to address the socio-technical challenges involved in design, development, implementation, use, and evaluation of health IT within complex adaptive healthcare systems. We focused on QOD calculators predicting pain and disability for this project as these outcomes are often of highest importance to patients undergoing spine surgery [9, 30,31,32], and we expect similar implementation strategies to apply to use of other comparable calculators. We conducted interviews in two health care systems, Vanderbilt University Medical Center (VUMC) and Duke University Health System (DUHS), to compare implementation needs in systems that were familiar and unfamiliar, respectively, with the QOD calculators.


QOD predictive calculators

For this study, we focused on the QOD web-based calculators that predict disability and pain intensity (back/neck and leg/arm) following lumbar and cervical spine surgery. The calculators provide probabilities for (1) any improvement over current level of pain or disability, and (2) an improvement of 30%, which is a valid criterion for minimally clinical important difference (MCID) [22, 23]. Patients enter information about their demographics, symptoms, and pain and disability level into the calculator via a web-based interface. This information is then used to predict each outcome probability. Additional details on calculator content and development are provided elsewhere [22, 23, 33].

Participant selection

We used purposive sampling to recruit stakeholders located at VUMC and DUHS that met one of the following criteria: (1) surgeon involved in the care of individuals undergoing spine surgery; (2) non-surgeon health care provider involved in the care of individuals undergoing or potentially eligible for spine surgery; (3) health system administrator or health IT faculty or staff involved in the clinical spine care process; (4) patient with low back and/or neck pain consulting with a surgeon. We focused purposive sampling on ensuring a representative sample based on role (for both health care system interviewees) and experience with the calculators (for VUMC health care system interviewees). Purposive sampling of patients focused on gathering a representative sample of interviewees based upon the presence of neck versus back pain. We selected VUMC because familiarity of their health care providers with the calculators allowed us to explore their depth and breadth of experience, with key insights into potential implementation barriers of these specific tools. On the other hand, DUHS interviewees would provide important perspectives on potential barriers and facilitators for clinical sites implementing an entirely novel tool. Institutional Review Board approval was obtained prior to study activities at both sites. All interviewees provided informed consent before participation.

Data collection and management

Interviews (one-on-one) took place between April and December 2021. A PhD-trained investigator with 2 years of qualitative interview experience (TL, male) conducted all interviews by video call (i.e., Zoom). Provider and administrator/health IT interviews began with a short demographic survey (position/job title, stakeholder group, department, years in practice and at the institution, gender, race, ethnicity, and age). The patient demographic survey included age, race, ethnicity, employment status, level of education, primary diagnosis, acuity of symptoms, history of spine surgery, smoking status and self-reported health status compared to others. The interviewer then described the QOD calculators and showed screenshots of the data entry interface, as well as example outputs from mock data entered into the calculator (Additional File 1). The interviewer used the example outputs to describe how the outputs (i.e., the probabilities of achieving any improvement over current level of pain or disability, and an improvement of 30% over current level of pain or disability) would aid with setting expectations prior to surgery. After this demonstration, the interviewer provided opportunities for the interviewee to ask questions. The interviewer then followed a semi-structured interview guide, informed by the Socio-technical model and customized for each type of interviewee (Additional File 2 and 3). When discussing options for how the calculator should display probability outputs, we tried to minimize bias by showing the same output options to each interviewee, using a standard script, and asking standardized questions as outlined in the semi-structured interview guide. Interviews lasted 30–45 min and were audio recorded, transcribed verbatim, and assessed for data quality before analysis. Key topics covered by semi-structured interviews for each stakeholder type are provided in Table 1.

Table 1 Interview Topicsa

Data analysis. We used a qualitative template analysis approach [34], a form of thematic analysis emphasizing hierarchical coding. Dedoose software Version 9.0.18 was used to organize and analyze interview data. Two team members (TL, BR) read each interview for familiarity. The initial coding template consisted of interview guide prompts and expected categories of responses. TL and BR coded each transcript, and, in an iterative fashion, met to compare codes and reach consensus on coding application after each set of 4–5 interview transcripts were analyzed. Saturation of codes, meaning no new codes or code categories were found in subsequent interviews, occurred after 15 interviews (5 patients; 5 administrators; 5 health care providers) and coders agreed on the analytic template to use for coding subsequently scheduled interviews. We determined that data saturation was achieved with the number of interviews we completed (24 health care provider interviews, 18 administrator/health IT interviews, and 16 patient interviews). We did not plan to conduct a formal comparative analysis of results from VUMC and DUHS interviews, but rather identify where potential implementation challenges converged or diverged between sites that were familiar and unfamiliar with the calculators, respectively.


Health care provider stakeholders included n = 18 (10 surgeons, 8 non-surgeons) at VUMC and n = 6 (5 surgeons, 1 non-surgeon) at DUHS (Table 2). Administrator/health IT stakeholders included n = 9 at VUMC and n = 9 at DUHS. Patient stakeholders (n = 16) were from VUMC only, with 10 having neck pain and 6 with low back pain. Table 3 provides additional patient demographic and health related information. Tables 4 and 5 provide sample quotes from interviewees. Table 6 outlines actionable strategies to optimize implementation of the QOD calculators in clinical settings based on our holistic synthesis.

Table 2 Demographic information for health care provider participants
Table 3 Demographic and health related information for patient participants
Table 4 Sample health care provider, administrator, and health IT interviewee quotes by topic
Table 5 Sample patient interviewee quotes by topic

Characteristics of the user

Perceived utility of using an outcomes calculator by providers

Almost all administrators had a favorable impression of the calculators and felt implementation would vary by surgeon depending on whether they found it useful for patient education. Across surgeons, the most commonly stated potential benefits were as a resource to dissuade patients from wanting surgery if they were poor surgical candidates (i.e., no correctable pathology) and to set realistic outcomes expectations. Most surgeons felt the calculators would result in outcome probabilities that were generally similar to what they could surmise through more traditional methods (i.e., talking to the patient, considering imaging results). For this reason, a few surgeons felt the calculators would be of limited use. However, the majority of surgeons reported that having specific, evidence-based probability estimates would help reinforce their clinical impressions while setting appropriate surgical expectations for patients. Non-surgeon health care providers shared these perspectives on the potential benefits. DUHS stakeholders were largely unfamiliar with the QOD calculators and expressed more skepticism about the added benefit of these calculators and concerns over validity of the outcome probability estimates than those at VUMC.

What is important to patients

We solicited input from patients on what was most important to them when deciding whether to undergo surgery. The most common responses were the probability of returning to work or leisure activities, improvement in pain and function, and symptom severity. Each patient we interviewed had a generally favorable impression of the calculators. Most reported that the greatest value was having quantitative, evidence-informed estimates of outcomes probabilities that would reinforce expectations set by their surgeon. However, some patients stated the calculator would be more helpful if it compared outcomes probabilities for different surgical and non-surgical options.

Clinical content

Quality and applicability of the calculators

The QOD calculators was developed and tested in patients undergoing elective spine surgery for degenerative conditions. A common concern from surgeons was for how well the QOD sample represented their own patient population. These concerns were specific to surgeons that performed complex, or minimally-invasive, spine procedures that were not well-represented in the QOD. Some surgeons felt the calculators did not include all potentially-relevant predictors, like current opioid use. Others felt the utility of the calculators was limited because they do not provide probabilities of improvement with other treatments, and therefore would be unhelpful for setting expectations for surgical versus non-surgical outcomes.

Redundancy of information

A significant concern across health care providers and administrators was the increased response burden associated with adding questions to the existing battery of questionnaires patients already complete in conjunction with visits. Interestingly, additional response burden was not a major concern for most patients interviewed. While they stated that measures should not be redundant, they were largely in favor of completing calculator items. Patients reported that they would be more likely to complete additional questions if they understood how surgeons would use the information to inform their care.

Workflow and communication

The biggest concern for providers and administrators was interrupted clinical workflow due to use of the calculators. Interviewees had different perspectives on how to mitigate this concern by strategically selecting who should complete the calculator and when they should complete it.

Selecting who should complete the calculators

Across health care providers, there was strong consensus that only patients who are already determined to be a surgical candidate should complete the calculators. Providers felt it would otherwise lead to wasted effort, and potentially higher patient demand for unwarranted surgeries. However, many administrators and IT professionals felt that taking a standardized approach to administering the tools to all patients would optimize efficiencies and enhance implementation potential. These stakeholders suggested that patients would only see their results if surgery was deemed suitable.

Determining when to complete the calculator

We also observed a wide range of opinions on when and how calculators should be administered. Those advocating for universal administration supported completing the calculator at home prior to the initial visit. Links to the calculator would be sent through MyChart or MyHealth interfaces or through email. Clinical staff liked this option because frequent software and hardware challenges (e.g., connectivity issues, depleted batteries) make use of clinic tablets unreliable. Moreover, clinic staff reported that patients would commonly need assistance navigating the tablet interface, further slowing workflow. Most patients also preferred the at-home option because they could complete questionnaires “on their own time, on their own device” and “not when rushed.”

Those advocating for a more targeted approach to administration felt patients should complete the calculator after being identified as a surgical candidate, but before making a decision to undergo surgery. Often, this interval occurs while the patient is in clinic, which introduces the challenge of completing a calculator while minimizing disruption to clinical workflows. One suggestion was to have patients complete the calculator after being triaged as a potential surgical candidate by an advanced practice provider (APP). In most circumstances, the APP and surgeon follow-up visits occur on different days, allowing for an interval in which patients could complete the calculator at home. Another option was to have patients complete the calculators after the visit with their surgeon, but before scheduling surgery. This way, any concerns regarding outcomes could be discussed with the surgeon before scheduling.

A subset of surgeons and non-surgeons expressed strong preferences for completion of the calculator in clinic. They felt patients should only see calculator results while in the presence of a health care provider. The concern was that patients could misinterpret the findings or fail to comprehend how probabilities should inform care decisions. Moreover, some providers had concerns about the quality of data entered into the calculators, noting they would only trust calculator outcomes if they knew patients understood the questions, completed the questions themselves, and were not trying to “game the system” by entering inaccurate responses. Providers felt such concerns were mitigated by having patients complete the calculators in clinic. Regardless of the specific approach, administrators suggested a pilot of the calculators with one or more clinical champions. A pilot would enable health care systems to resolve workflow bottlenecks, identify optimal windows and formats for administration, gather feedback from end-users, and establish IT requirements.

Human/Computer interface

Communicating probabilities and key terms

Interviewees were presented with multiple options for visualizing results. A consensus among all patients and providers was that probabilities should be presented clearly, with explanations of key terms like “disability” and “improvement”. Patients preferred to see their current level of pain or disability and what a 30% improvement would be for each of those outcomes. Another consensus among patients was that results should not be presented using statistical graphs or figures, but in more simplistic forms with whole numbers and concise, layperson descriptions of the outcomes. While patients overwhelmingly supported the potential value of the calculators, some believed the benefit was limited, wanting to know probabilities for higher improvement thresholds like 50% improvement (“30% improvement would not be enough for me”), probability of no pain, or comparison of outcome probabilities across various treatments. Nearly all patients preferred to have a surgeon or other health care provider on hand to discuss results in detail.

Resource needs and constraints

Most surgeons stated they were comfortable discussing outcomes probabilities with patients. Nevertheless, some stated a script or talking points specific to capabilities and limitations of the calculators could be helpful. Regardless of which patients completed the calculators and when, nearly all providers and administrators agreed that at least some staff resources would be necessary for implementation (e.g., ensure patients have completed the calculator, be on hand to answer questions, remind surgeons to discuss calculator results).

Since the calculators represent new tools for surgeons, we were interested in processes that would help them remember to incorporate results into patient counseling discussions. There was no consensus on the best way to deliver reminders. Some surgeons and their staff preferred pop-up reminders in the electronic health record (EHR), while others were concerned for notification fatigue. A few suggested building reminders into the clinic notes, but qualified that these passive reminders alone would not be sufficient. Few saw benefit in placing flyers around the clinical workspace. The majority of providers felt support staff should be leveraged to remind providers, but that the optimal strategy would vary across clinics and surgeons. One consistent suggestion was that surgeons be reminded of the calculators during regular staff meetings.

Internal organizational policies, procedures, and culture

Culture of research and innovation

Interviewees at both VUMC and DUHS felt the strong research culture and emphasis on quality care in both systems created an environment that would support implementation. Many interviewees stated that practice variability and siloed care delivery across clinics would largely preclude a single institutional implementation strategy. Most administrators and non-surgeon health care providers felt that gaining buy-in from all surgeons would be nearly impossible because some would not see value in the calculators. This observation was supported by skepticism from some surgeons who, as previously mentioned, felt these tools might not apply to their patient population or include the appropriate risk factors. A common opinion across interviewees was that successful implementation would require strong buy-in from Department and Division chiefs.

External rules, regulations, and pressures

Many providers and administrators expressed concerns that payers could use calculator results to deny coverage. These concerns led some to recommend against documentation of results in the EHR and to suggest that health care systems are clear about which entities will have access to results. Some interviewees raised concerns over whether use of a calculator could expose providers and health care systems to liability if outcomes did not align with calculator results (e.g., continued pain and disability despite high probabilities of success). To address liability concerns, several providers suggested a “warning label” on the calculators and/or confidence intervals for the probability estimates. These additions would ensure patients understood the uncertainty around probability estimates and that results should be used solely as an adjunct to other information when making treatment decisions. Most patients understood the inherent uncertainty in predicting surgical outcomes, but agreed that surgeons should be transparent in discussing calculator strengths and limitations.

System Measurement and Monitoring

Implementing a new clinical tool consumes time and resources, therefore we were interested in learning how to measure the potential benefits of these efforts. We asked interviewees how they would determine if use of the tools to set outcomes expectations were a “success” in practice. Common metrics noted by health care providers and administrators were improvement in patient satisfaction scores and general improvements in patient management or decision-making from the provider perspective. Health care providers especially valued a reduction in decisional conflict among their patients who may be struggling with the decision to undergo surgery. Patients also reported the potential benefits of reducing decisional conflict, while emphasizing the most important measure of success was the degree to which the calculators accurately predicted their outcomes. Approximately half of the patients stated that improved quality of the patient-provider interaction should be another quality metric by which to evaluate the benefit of the calculators.


This work aims to inform implementation strategies for tools that support patient counseling on expectations and addresses the tension between additional automated tool application and the need for better ways to facilitate shared-decision-making. One strength of this qualitative analysis is the consideration of viewpoints across various stakeholder types at two institutions. We found common potential challenges across institutions, including the need for leadership buy-in, difficulties integrating a web-based tool into existing IT workflows, process variations across clinics that limit integration, and concerns over data quality, liability, and impact on payment. Prior implementation studies of comparable tools in different clinical settings have reported similar barriers [35,36,37,38]. While most interviewees agreed the calculators would help to set surgical expectations, they reported widely divergent viewpoints on the best way to implement these tools. These findings suggest that no single implementation strategy would be universally successful, even within the same healthcare system. Nevertheless, actionable strategies emerged throughout our interviews to guide implementation (Table 6). We believe, given the study design, that these determinations have broad capacity for translation in other clinical contexts that are similar to our own.

Table 6 Actionable strategies to optimize successful implementation of the QOD calculators

This study explored perspectives across stakeholders in two distinct health care systems that differed in their familiarity with the calculators. Notable similarities across systems include the need for department or division chief buy-in, challenges with integrating a web-based tool into existing EHR and documentation workflows, variation across clinics in workflows and processes that limit integration, and concerns over liability and impact on payment. As with VUMC interviewees, the majority of DUHS interviewees felt that a single implementation strategy was not feasible, but should be tailored to the unique needs, resources, and settings of each clinic and/or surgeon.

Although implementation challenges were similar across systems, the major differences were that DUHS interviewees (1) expressed more skepticism about the validity or accuracy of the calculators and (2) had more doubts about the operational feasibility of assimilating them into care. Most DUHS interviewees stated they would need more familiarity with the calculators before considering their use, and evidence of their benefit in setting expectations beyond current practice. Compared to health care providers, patients more readily identified the value of having access to quantifiable, evidence-based outcomes probabilities. The difference in value judgements between patients and providers on this point highlights an important opportunity to enhance patient-provider communication for surgical outcomes expectations. The majority of patients interviewed liked to see evidence-based estimates that supported their surgeons’ recommendations and felt this information would further reinforce their surgical decisions. Health care providers may not recognize or fully appreciate this patient perspective on the calculators’ value.

Our findings suggest a few critical implementation strategies. These include articulating the accuracy and validity of the calculators to providers, demonstrating their operational feasibility, and convincing providers of the potential value they bring in contributing measurable, evidence-based outcomes probabilities to the clinical encounter. Implementation strategies should also address concerns over who would have access to calculator results and how they could influence payment and liability, as interviewees at both sites raised these issues. Such efforts may include piloting the calculators with one or more clinical champions to provide “proof of concept” regarding benefits, a recommendation raised by a few administrators.

Many providers and administrators expressed doubt that patients would be willing to complete additional questions associated with the calculators. On the contrary, most patients were willing to complete the calculators as long as questions were not redundant and providers made it clear how the information would direct care. This feedback has two main implications. First, it suggests the need to limit redundancy of data entry by using information from existing data sources to populate calculator responses when possible (e.g., importing demographic or diagnostic data elements from the EHR). Second, this feedback reinforces the importance of clearly communicating the intent of clinical data collection to improve response rates and quality of entered data.

Perhaps the most important perceived driver of implementation success is the ease in which patients and providers can use the calculators during routine clinical care. Given the inherent variability in clinical workflows, successful implementation of the calculators at scale would require flexibility on who completes them, and when and how they are completed based on clinic resources, surgeon preferences, and patient needs. Strategies to enhance ease of use for patients includes providing flexible options for them to complete the calculator at-home or in-clinic, and having readily available explanations of key terms (e.g. disability, probability, and improvement). For providers, ease of use would be enhanced by integrating the calculators into the EHR, leveraging clinical support staff to remind surgeons to administer/discuss calculators with patients, receiving reminders about the calculators at staff meetings, and developing simple talking points or scripts to facilitate discussion with patients. The perceived optimal timing of when surgeons discuss probabilities with patients was highly variable and a very small proportion stated they were not interested in outcomes probabilities at all. Thus, health care providers should talk with patients about this resource and their preferences for its use in discussions about surgery.

This work has some limitations. First, we conducted interviews in two academic health systems. The characteristics of these systems and interviewees may not be generalizable to all healthcare settings. Additionally, this study evaluated the implementation potential of specific calculators. While most calculators share similar attributes, we do not know whether the challenges and implementation strategies we identified would apply to other calculators currently available, such as calculators that predict return to work [26].

A second limitation is that our sample of patients had limited diversity in race and ethnicity. We only recruited patient participants from one center. Therefore, the views of these patients may not generalize to more diverse populations in other clinical contexts. Our sample was similar in level of symptom acuity to studies on spine surgery populations using QOD data [14, 22, 23, 27, 28], however tended to be older (median age = 60 years, mean age = 62.2 years) compared to QOD studies (mean ranges 51–55 years). Our sample also had higher levels of education, a higher proportion of male participants, and was under-represented by non-White and Hispanic participants compared to published QOD studies. These comparisons are important because they suggest our study did not fully capture perspectives of younger, non-White, and Hispanic participants undergoing spine surgery, or those with lower educational attainment. These patients may have different communication, informational, or access needs. They may see different value in the calculators, have differing perspectives on the burden of extra questions, want information earlier or later in the care episode, or value information on other outcomes. Studying needs specific to these under-represented populations would further inform calculator implementation for an important care-seeking group. Finally, sample recruitment was driven by a desire to develop a representative cohort of interviewees, not necessarily by the desire for theme saturation. Although we did see similar and consistent responses across interviewees with similar roles, a larger sample may have uncovered additional perspectives.

Our findings have important implications for future work on the topic. Most notably, the calculators studied provide outcomes probabilities as the result of surgery, however are unable to provide outcomes probabilities for other non-surgical treatment options. Multiple patient interviewees noted the importance of comparing outcomes probabilities among different treatment options. To our knowledge, no single resource exists that directly compares personalized outcomes probabilities of different treatments for spine pain. Future work should focus on developing these resources, as well as calculators that predict longer-term outcomes.

Second, future research should focus on understanding how implementation strategies may differ across diverse healthcare settings and populations. This would include determining optimal strategies for communicating key terms (e.g., outcomes probabilities) to patients with differing levels of education and health literacy. It would also entail determining best ways to educate patients and providers on the limitations of the calculators, i.e., that they cannot be used to compare outcomes probabilities for different treatments. Our work also lays a foundation for future effectiveness studies. Specifically, it outlines strategies to ensure successful implementation of these tools and optimize fidelity to protocols that use these tools in pragmatic trials. Such trials are necessary to determine whether use of calculators improves management and outcomes for people seeking a surgical consult for spine pain.

A final direction for future work is to ensure that prediction tools, such as surgical outcomes calculators, do not lead to inequitable care by creating bias in clinical decision-making. This is a major concern as health care systems leverage new technologies and large, population-based datasets to create decision-support algorithms [39,40,41]. Some surgeons in this study raised concerns that the calculators may not apply to their specific patient population. This concern highlights an important principle; that outcomes probabilities will be most accurate when surgeons use calculators in a patient population similar to the one used to develop the calculator. Any future calculator refinement for more specific populations should pay careful attention to design and take a thoughtful approach to mitigate any potential biases.


Interviewees had a largely positive opinion of the QOD calculators, believing they would aid in setting appropriate expectations for spine surgery outcomes. Implementation strategies will need to vary according to clinic resources, surgeon preferences, and patient needs. The most successful strategies will provide flexible options for where and how patients complete the calculator, using existing EHR data elements when available to minimize response burden. To encourage calculator use by surgeons, implementation strategies should leverage clinical support staff to assist with administration, include reminders during regular staff meetings, provide scripts or talking points to aid in patient education on outcome probabilities, and address any concerns over data quality, liability, and impact on payment. Future research is necessary to determine whether use of calculators improves management and outcomes for people seeking a surgical consult for spine pain.

Data Availability

The datasets generated and/or analyzed during the current study are available from the corresponding author on reasonable request.



quality outcomes database


electronic health record


advanced practice provider


information technology


Vanderbilt University Medical Center


Duke University Health System


  1. Dieleman JL, Cao J, Chapin A, Chen C, Li Z, Liu A, et al. US Health Care spending by Payer and Health Condition, 1996–2016. JAMA. 2020;323:863–84.

    Article  PubMed  PubMed Central  Google Scholar 

  2. Institute of Medicine (US). Committee on Advancing Pain Research, Care, and Education. Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education, and Research [Internet]. Washington (DC): National Academies Press (US); 2011 [cited 2020 Apr 28]. Available from:

  3. Deyo RA, Mirza SK, Martin BI, Kreuter W, Goodman DC, Jarvik JG, Trends. Major Medical Complications, and charges Associated with surgery for lumbar spinal stenosis in older adults. JAMA. 2010;303:1259–65.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. Ragab A, Deshazo RD. Management of back pain in patients with previous back surgery. Am J Med. 2008;121:272–8.

    Article  PubMed  Google Scholar 

  5. Weiner DK, Kim Y-S, Bonino P, Wang T. Low back pain in older adults: are we utilizing healthcare resources wisely? Pain Med. 2006;7:143–50.

    Article  PubMed  Google Scholar 

  6. Centers for Medicare & Medicaid Services (CMS) H. Medicare program; hospital inpatient value-based purchasing program. Final rule. 2011.

  7. Sciubba DM, Pennington Z, Ehresman J. Guest Editorial: Predictive Analytics, calculators and cost modeling in spine surgery. Global spine Journal. Volume 11. SAGE Publications Inc; 2021. pp. 4S–6S.

  8. Damberg CL, Sorbero ME, Lovejoy SL, Martsolf GR, Raaen L, Mandel D. Measuring success in Health Care Value-Based Purchasing Programs: findings from an Environmental scan, Literature Review, and Expert Panel Discussions. Rand Health Q. 2014;4:9.

    PubMed  PubMed Central  Google Scholar 

  9. Sivaganesan A, Khan I, Pennings JS, Roth SG, Nolan ER, Oleisky ER, et al. Why are patients dissatisfied after spine surgery when improvements in disability and pain are clinically meaningful? Spine J. 2020;20:1535–43.

    Article  PubMed  Google Scholar 

  10. Asher AM, Oleisky ER, Pennings JS, Khan I, Sivaganesan A, Devin CJ, et al. Measuring clinically relevant improvement after lumbar spine surgery: is it time for something new? Spine J. 2020;20:847–56.

    Article  PubMed  Google Scholar 

  11. Khan I, Pennings JS, Devin CJ, Asher AM, Oleisky ER, Bydon M, et al. Clinically meaningful improvement following cervical spine surgery: 30% reduction Versus Absolute Point-change MCID values. Spine (Phila Pa 1976). 2021;46:717–25.

    Article  PubMed  Google Scholar 

  12. Hamilton DF, Lane JV, Gaston P, Patton JT, MacDonald D, Simpson AHRW, et al. What determines patient satisfaction with surgery? A prospective cohort study of 4709 patients following total joint replacement. BMJ Open British Medical Journal Publishing Group. 2013;3:e002525.

    Google Scholar 

  13. Mannion AF, Junge A, Elfering A, Dvorak J, Porchet F, Grob D. Great expectations: really the novel predictor of Outcome after spinal surgery? Spine. 2009;34:1590–9.

    Article  PubMed  Google Scholar 

  14. Asher AL, Devin CJ, Kerezoudis P, Nian H, Alvi MA, Khan I, et al. Predictors of patient satisfaction following 1- or 2-level anterior cervical discectomy and fusion: insights from the Quality Outcomes Database. J Neurosurgery: Spine Am Association Neurol Surg. 2019;31:835–43.

    Google Scholar 

  15. Cha EDK, Lynch CP, Jadczak CN, Mohan S, Geoghegan CE, Singh K. Meeting patient expectations or achieving a Minimum clinically important difference: predictors of satisfaction among lumbar Fusion Patients. Asian Spine J. 2022;16:478–85.

    Article  PubMed  Google Scholar 

  16. Rampersaud YR, Canizares M, Perruccio AV, Abraham E, Bailey CS, Christie SD, et al. Fulfillment of patient expectations after spine surgery is critical to patient satisfaction: a cohort study of spine surgery patients. Neurosurgery. 2022;91:173–81.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Eastwood D, Manson N, Bigney E, Darling M, Richardson E, Paixao R, et al. Improving postoperative patient reported benefits and satisfaction following spinal fusion with a single preoperative education session. Spine J. 2019;19:840–5.

    Article  PubMed  Google Scholar 

  18. Lubelski D, Hersh A, Azad TD, Ehresman J, Pennington Z, Lehner K, et al. Prediction models in degenerative spine surgery: a systematic review. Global Spine J. 2021;11:79S–88S.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Caplan IF, Winter E, Glauser G, Goodrich S, McClintock SD, Hume EL, et al. Composite score for prediction of 30-day orthopedic surgery outcomes. J Orthop Res. 2020;38:2189–96.

    Article  PubMed  Google Scholar 

  20. Dimentberg R, Caplan IF, Winter E, Glauser G, Goodrich S, McClintock SD, et al. Prediction of adverse outcomes within 90 days of surgery in a heterogeneous orthopedic surgery Population. J Healthc Qual. 2021;43:e53–63.

    Article  PubMed  Google Scholar 

  21. Anis HK, Strnad GJ, Klika AK, Zajichek A, Spindler KP, Barsoum WK, et al. Developing a personalized outcome prediction tool for knee arthroplasty. The Bone & Joint Journal. Volume 102–B. The British Editorial Society of Bone & Joint Surgery; 2020. pp. 1183–93.

  22. Archer KR, Bydon M, Khan I, Nian H, Pennings JS, Harrell FE, et al. Development and validation of cervical prediction models for patient-reported outcomes at 1 year after cervical spine surgery for Radiculopathy and Myelopathy. Spine (Phila Pa 1976). 2020;45:1541–52.

    Article  PubMed  Google Scholar 

  23. McGirt MJ, Bydon M, Archer KR, Devin CJ, Chotai S, Parker SL, et al. An analysis from the Quality Outcomes Database, Part 1. Disability, quality of life, and pain outcomes following lumbar spine surgery: predicting likely individual patient outcomes for shared decision-making. J Neurosurg Spine. 2017;27:357–69.

    Article  PubMed  Google Scholar 

  24. Curtin P, Conway A, Martin L, Lin E, Jayakumar P, Swart E. Compilation and analysis of web-based Orthopedic Personalized Predictive Tools: a scoping review. J Pers Med. 2020;10:223.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Tetreault LA, Côté P, Kopjar B, Arnold P, Fehlings MG, AOSpine North America and International Clinical Trial Research Network. A clinical prediction model to assess surgical outcome in patients with cervical spondylotic myelopathy: internal and external validations using the prospective multicenter AOSpine North American and international datasets of 743 patients. Spine J. 2015;15:388–97.

    Article  PubMed  Google Scholar 

  26. White HJ, Bradley J, Hadgis N, Wittke E, Piland B, Tuttle B, et al. Predicting patient-centered outcomes from spine surgery using Risk Assessment Tools: a systematic review. Curr Rev Musculoskelet Med. 2020;13:247–63.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Devin CJ, Bydon M, Alvi MA, Kerezoudis P, Khan I, Sivaganesan A, et al. A predictive model and nomogram for predicting return to work at 3 months after cervical spine surgery: an analysis from the Quality Outcomes Database. Neurosurg Focus. 2018;45:E9.

    Article  PubMed  Google Scholar 

  28. Asher AL, Devin CJ, Archer KR, Chotai S, Parker SL, Bydon M, et al. An analysis from the Quality Outcomes Database, Part 2. Predictive model for return to work after elective surgery for lumbar degenerative disease. J Neurosurg Spine. 2017;27:370–81.

    Article  PubMed  Google Scholar 

  29. Sittig DF, Singh H. A New Socio-technical model for studying Health Information Technology in Complex Adaptive Healthcare Systems. Qual Saf Health Care. 2010;19:i68–74.

    Article  PubMed  Google Scholar 

  30. Turk DC, Dworkin RH, Revicki D, Harding G, Burke LB, Cella D, et al. Identifying important outcome domains for chronic pain clinical trials: an IMMPACT survey of people with pain. Pain. 2008;137:276–85.

    Article  PubMed  Google Scholar 

  31. Ostelo RWJG, de Vet HCW. Clinically important outcomes in low back pain. Best Pract Res Clin Rheumatol. 2005;19:593–607.

    Article  PubMed  Google Scholar 

  32. Whitebird RR, Solberg LI, Norton CK, Ziegenfuss JY, Asche SE, Grossman ES. What outcomes matter to patients after joint or spine surgery? J Patient Cent Res Rev. 2020;7:157–64.

    Article  PubMed  PubMed Central  Google Scholar 

  33. Archer KR, Nian H, Khan I, Pennings JS. Streamlining the QOD Web-based Calculator for Clinical Integration: Development and Validation of a Reduced Prediction Model for Lumbar Spine Surgery. Spine. 2022;In Press.

  34. Brooks J, McCluskey S, Turley E, King N. The utility of Template Analysis in qualitative psychology research. Qualitative Res Psychol Routledge. 2015;12:202–22.

    Article  Google Scholar 

  35. Porter A, Kingston MR, Evans BA, Hutchings H, Whitman S, Snooks H. It could be a ‘Golden Goose’: a qualitative study of views in primary care on an emergency admission risk prediction tool prior to implementation. BMC Fam Pract. 2016;17:1.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Evans BA, Dale J, Davies J, Hutchings H, Kingston M, Porter A, et al. Implementing emergency admission risk prediction in general practice: a qualitative study. Br J Gen Pract. 2022;72:e138–47.

    Article  PubMed  Google Scholar 

  37. Trivedi MH, Daly EJ, Kern JK, Grannemann BD, Sunderajan P, Claassen CA. Barriers to implementation of a computerized decision support system for depression: an observational report on lessons learned in “real world” clinical settings. BMC Med Inf Decis Mak. 2009;9:6.

    Article  Google Scholar 

  38. Klarenbeek SE, Schuurbiers-Siebers OCJ, van den Heuvel MM, Prokop M, Tummers M. Barriers and facilitators for implementation of a computerized clinical decision support system in Lung Cancer Multidisciplinary Team Meetings-A qualitative Assessment. Biology (Basel). 2020;10:9.

    PubMed  Google Scholar 

  39. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366:447–53.

    Article  CAS  PubMed  Google Scholar 

  40. Benjamin R. Assessing risk, automating racism. Science. 2019;366:421–2.

    Article  CAS  PubMed  Google Scholar 

  41. Bærøe K, Gundersen T, Henden E, Rommetveit K. Can medical algorithms be fair? Three ethical quandaries and one dilemma. BMJ Health Care Inform. 2022;29:e100445.

    Article  PubMed  PubMed Central  Google Scholar 

Download references




This study was funded by the Center on Health Services Training and Research (CoHSTAR).

Author information

Authors and Affiliations



TL and KA contributed to the conception and design of the work; TL and SB contributed to data acquisition, TL and BR contributed to analysis; TL, BS, AA, JS, AS, BR, and KA contributed to interpretation of data; TL drafted the work; All authors substantively revised the draft and approved the submitted version.

Corresponding author

Correspondence to Trevor A. Lentz.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the Institutional Review Boards of Vanderbilt University Medical Center (Protocol #210128) and Duke University Health System (Pro00108956). All methods were performed in accordance with the Declaration of Helsinki. All participants provided informed consent electronically prior to their interview.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Supplementary Material 3

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lentz, T.A., Stephens, B.F., Abtahi, A.M. et al. Leveraging web-based prediction calculators to set patient expectations for elective spine surgery: a qualitative study to inform implementation. BMC Med Inform Decis Mak 23, 149 (2023).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: