Patient population and data collection

Informed consent was obtained from all patients and participation was neither mandatory, nor required to gain access to healthcare. According to Norwegian legislation, patients over the age of 15 can independently consent to participation in the registry. The registry protocol has been approved by the Data Inspectorate of Norway. This study was submitted to the regional ethical committee for medical research which categorized it as a clinical audit study, not in need of their formal approval [21].

At admission for surgery, the patients completed a baseline questionnaire on demographics, lifestyle issues, and PROMs. During the hospital stay, the surgeon recorded data concerning diagnosis, treatment, and comorbidity on a standard registration form. Twelve months after surgery, a questionnaire was distributed by regular post, completed at home by the patients, and returned to the central registry unit without involvement of the treating hospitals. One reminder with a new copy of the questionnaire was sent to those who did not respond.

Patient-reported outcome measures

The PROM questionnaires were identical at baseline and follow-up. The ODI version 2.0 was used to assess pain-related disability. It contains ten questions on limitations of activities of daily living. Each item is rated 0–5 and then transferred into a percentage score ranging from 0 (none) to 100 (maximum pain-related disability) [1].

Pain was reported on the numerical rating scale of 0–10 for both back pain (NRS back pain) and leg pain (NRS leg pain), where 0 = no pain and 10 = worst conceivable pain [2].

Generic health-related quality of life was assessed by the EQ-5D [22], which has been validated for a similar patient population [23]. It evaluates five dimensions: mobility, self-care, activities of daily living, pain, and anxiety and/or depression. For each dimension, the patient describes three possible levels of problems (none, mild-to-moderate, and severe). This descriptive system, therefore, contains 3⁵ = 243 combinations or index values for health status. The total score ranges from −0.59 to 1, where 1 corresponds to perfect health and 0 to death. Negative values are considered to be worse than death.

The patient-rated benefit of the operation was rated on a Global Perceived Effect scale (GPE) at follow-up [24]. The response alternatives were: 1 = “completely recovered”, 2 = “much better”, 3 = “somewhat better”, 4 = “no change”, 5 = “somewhat worse”, 6 = “much worse”, and 7 = “worse than ever”.

Definition of failure and worsening

Patients reporting to be unchanged or worse (categories 4–7) on the GPE scale at 12-month follow-up were classified as “failure”, and those considering themselves worse or worse than ever (GPE 6–7) were classified as “worsening”.

Statistics

All statistical analyses were performed with the Statistical Package for the Social Sciences (SPSS, IBM Version 23.0).

We excluded all patients who did not respond at 12 months, and compared baseline characteristics of both respondents and non-respondents. This strategy was based on a study on a comparable patient population from NORspine and a recent and similar Danish study [25, 26].

For all PROMs, the mean change, mean % change (except for EQ-5D), and mean final raw score were assessed against the GPE by one-way analyses of variance (ANOVA) with post hoc analysis (Tukey, α = 0.05) and by analyses of co-variance (ANCOVA, generalized linear model) with adjustment for baseline scores. Correlation analyses between PROMs and the GPE were done by Spearman rank correlation for all measures, except for the final raw scores in which Pearson was used.

Cutoffs for all scores were estimated by Receiver Operating Characteristic (ROC) curves. When analyzing criteria for “failure”, cases with failure were defined as those who reported to be unchanged or worse (categories 4–7) on the GPE scale at 12 months. All other categories on the GPE scale (1–3) were defined as “no failure”. When comparing patients, reporting being considerably worse (GPE 6–7), with those who reported an unchanged status (GPE 4–5), those reporting improvement (GPE 1–3) were excluded from these analyses. To determine the cutoff with the highest sensitivity and specificity for both failure and worsening, the closest point to the upper left corner of the ROC curve was calculated from the coordinates of the curve. Area under the curve (AUC) calculations were performed to determine how well the instruments differentiated between the outcome groups. An AUC value of >0.70 was considered acceptable [27]. The overall accuracy for each cutoff was calculated with a confusion matrix. In the presentation of the results, we included AUC and cutoff values only for variables with an AUC value above 0.70. Results for PROMS with poorer accuracy can be provided on request.

To investigate whether the optimal cutoffs differed between important subgroups in the registry sample, sensitivity analyses were performed between first time vs reoperation and between macroscopic (“open”) vs microscope or loupe-assisted discectomy. To evaluate the impact of different baseline scores on the cutoffs, cutoff calculations were also carried out on those with low- and high baseline disability.

Differences between elective and emergency cases at 12-month follow-up were calculated for all PROMs by Student’s t test and for the GPE by Mann–Whitney U test.

Floor and ceiling effects were assessed by calculating the frequency of the highest and lowest possible scores at baseline. If 15% of patients had a minimal or maximal score value at baseline, these were considered as floor or ceiling effects [27, 28].

Cutoff values

For differentiation between “failure” vs no failure in the whole cohort, all PROMs had an acceptable AUC of >0.70 (Table 4). The PROM with the highest accuracy was the mean ODI% change score with an AUC of 0.93 and a correct classification rate of 86% (Fig. 1).

Table 4

All cutoff values with corresponding sensitivity and specificity, area under the curve (95% confidence interval), and percentage of correctly classified

	Failure				Worsening
	Cutoff	Sens/spec	AUC (95% CI)	Corr. class %	Cutoff	Sens/spec	AUC (95% CI)	Corr. class %
ODI
Mean change	13	0.82, 0.82	0.89 (0.88–0.91)	82
Mean % change	33	0.86, 0.86	0.93 (0.92–0.94)	86
12 month raw	25	0.89, 0.81	0.92 (0.91–0.93)	86	48	0.70, 0.70	0.76 (0.72–0.80)	69
NRS leg pain
Mean change	1.5	0.81, 0.76	0.87 (0.86–0.88)	84
Mean % change	39	0.86, 0.81	0.89 (0.88–0.90)	84
12 month raw	4.5	0.91, 0.85	0.90 (0.88–0.91)	84	7.5	0.64, 0.68	0.70 (0.66–0.75)	67
NRS back pain
Mean change	1.5	0.74, 0.86	0.85 (0.84–0.86)	76
Mean % change	24	0.85, 0.81	0.87 (0.86–0.88)	86
12 month raw	5.5	0.81, 0.87	0.92 (0.91–0.93)	86	7.5	0.78, 0.64	0.77 (0.73–0.81)	68
EQ-5D
Mean change	0.1	0.76, 0.83	0.85 (0.84–0.87)	82
12 month raw	0.6	0.81, 0.85	0.91 (0.90–0.92)	85	0.1	0.76, 0.60	0.71 (0.67–0.75)	65

For worsening, only the 12-month raw scores were used, and all the other cutoffs had an AUC < 0.70

Open image in new window

For differentiation between “worsening” vs unchanged and slightly worse, the AUCs were poor (<0.70) for score changes of all outcome measures. The final raw scores of all four PROMs showed acceptable AUCs. The PROM with the highest accuracy was the ODI raw score with an AUC of 0.76 and a correct classification rate of 69% (Fig. 2). The ROCs for all of the PROMs are illustrated in the appendix (Figs. 5x–9x).

Open image in new window

Based on these cutoff values, the ODI change classified 26%, the ODI% change score 23%, and the ODI raw score at 12 months 27% of lumbar disc surgeries as failure. Failure rates assessed by cutoffs of the less accurate PROMs are shown in the appendix (Table 4x).

The percentages of patients classified as worsening by the cutoffs on the final PROM raw scores were 7% for ODI, 8% for EQ-5D, 7% for NRS leg pain, and 8% for NRS back pain.

Sensitivity analysis

When comparing patients operated for the first time with those who had been operated previously, values for cutoff, sensitivity, and specificity were similar (Tables 2x and 3x in appendix). When investigating the effect of low and high baseline disability (based on the 25th and 75th percentile of the baseline score for ODI), the cutoffs for “failure” and “worsening” in the PROMs varied considerably, both for change scores, % change scores, and the final raw score (Table 1x, appendix). For example, in the group with high disability at baseline, the failure cutoff for the mean % change in ODI was 30% higher than in the low disability group.

Compared to elective surgery, emergency cases had statistically significant worse baseline PROM scores and experienced a greater score improvement at 12 months. Accordingly, no statistically difference in any of the 12-month PROM raw scores was found between these two groups. Furthermore, they reported the same GPE after 12 months, with a median score of 2 (Table 5x, appendix).

Floor and ceiling effects

No floor or ceiling effects were detected. Only 9 (0.1%) patients scored 0 and 7 (0.1%) patients scored 100 on the baseline ODI. Furthermore, 107 (1.6%) scored 0 and 590 (8.8%) scored 10 in the NRS back-pain scale. For the NRS leg pain, scale numbers were 55 (0.8%) for 0 and 728 (10.9%) for 10. In the EQ-5D, only 12 (0.2%) patients scored the minimum and 20 (0.3%) the maximum at baseline.

Methodological challenges

The global perceived effect is a frequently used external anchor to define cutoffs on PROMs. Still, it has several weaknesses related to recall bias [17], lack of objectivity [36], and for not taking into account the measurement precision [6]. More objective criteria, such as return to work or use of pain killers, have been proposed [36]. However, they tend to be subgroup specific (e.g., only considering the working population) and may also be susceptible to confounding [37]. Some authors argue that the criteria should be defined prior to treatment by letting the patients quantify, e.g., on a pain scale, how great a satisfying improvement should be [38]. To the best of our knowledge, no such alternative and well-validated external anchors for self-reported questionnaires exist. Unlike the European Spine Tango registry, the NORspine does not collect data on the surgeon’s overall assessment of outcome [39]. Lack of “expert opinion” might represent a weakness. However, surgeons and patients agree only in 50% of cases when assessing outcomes, and surgeons tend to rate the end result over-optimistically [40]. Another weakness related to anchor-based methods is misclassification. In our population, the ODI% cutoff of 33% improvement at 12 months (AUC 0.93, sensitivity/specificity 86%) gave a false-positive rate of 14% and a false-negative rate of 15%.

Importantly, we found that the cutoffs also were highly depending on the baseline PROM score. For instance, severely disabled patients will require disproportionally greater improvements than the less disabled, not to consider the surgery as failed. This is in accordance with findings of other studies and illustrates the importance of taking into account the baseline score while interpreting PROM change scores, regardless of using absolute or percentage change scores [18, 41]. Consequently, one should adjust for the baseline score when using such outcome criteria in clinical trials and risk factor analyses. A possible cause might be higher expectations towards improvements among patients with high baseline pain and disability [42]. Fulfillment of expectations has also been identified as a major predictor for positive patient-rated positive outcome after surgery [35]. Similar to findings by Elkan et al., emergency cases presented with more severe symptoms and had a greater amount of change on the PROM scores, thus reported the same improvement on the GPE scale [43].

Limitations and strengths of this study

Loss to follow-up at 12 months was 31.1%. Two Scandinavian registry studies found that a loss to follow-up of 12–22% did not bias conclusions about treatment effects [25, 26]. Even if baseline PROMs were similar between respondents and non-respondents in our study, several baseline characteristics of non-respondents have been associated with poorer outcomes [44]. This could represent a selection bias, especially when measuring the exact failure and worsening rate, but less so when defining PROM cutoffs over a large range of outcomes. Follow-up was only 12 months, but previous studies have shown mean outcome values to be stable from 1 up to 8 years [26, 45].

An advantage of this study is the large sample size and high external validity due to patient recruitment from everyday practice. In a smaller single-center study from 2013, Gum et al. tried to define clinically important deterioration among patients operated with lumbar fusion for various diagnoses, but found it difficult to define cutoffs. They concluded that a larger patient population was needed to identify accurate cutoffs, since worsening is a relatively rare event [41]. We have used a much larger and more condition-specific cohort.

Future perspectives

Both clinicians and administrators have questioned whether quality registries can improve clinical practice and feedback comprehendible information to patients and clinicians [3]. An advantage of dichotomous outcomes is the possibility to provide risk estimates in terms of probability. In clinical decision-making, percentwise probability would be easier to understand than estimates based on continuous outcome data (e.g., linear regressing coefficients). More research is needed to identify risk factors for adverse outcomes and to learn how such new knowledge can be conveyed efficiently to patients and health care providers.

Implication

These criteria could facilitate shared decision-making among physicians and patients, quality assessment, and comparison of clinical outcomes across surgical units. In addition to clinically relevant improvements, we propose that rates of failure and worsening should be included in reporting from clinical trials.