Completeness and accuracy of data in spine registries: an independent audit-based study

Completeness and accuracy of data in spine registries: an independent audit-based study

Meyer, B., Shiban, E., Albers, L.E. et al. Completeness and accuracy of data in spine registries: an independent audit-based study. Eur Spine J 29, 1453–1461 (2020). https://doi.org/10.1007/s00586-020-06342-6

Abstract

Purpose

Clinical registries are used for quality management and clinical research. Due to the importance and implications of both aims, completeness and high quality of data are of paramount importance. However, this remains uncertain, as none of these registries have implemented independent monitoring. The aim of this study was to determine the accuracy and completeness of registry data o the example of the German Spine Society (DWG) registry.

Methods

In a prospective study, audits by a board-certified neurosurgeon were conducted at certified spine centers with mandatory registry input, a setting comparable to most existing registries worldwide. A 2-week period was analyzed, and any discrepancy between patients’ charts and the registry entry was evaluated. A median of 31 items per patient was evaluated including completeness and accuracy of data.

Results

Out of 17 centers willing to participate, 4 were still lacking any data entries. Even in the remaining 13 centers eligible for audits, 28.50% (95%-CI = [22.46–34.55]) of entries were finalized only after the audits were announced. Only 82.55% (95%-CI = [79.12–85.98]) of surgeries were documented, and on average 14.95% (95%-CI = [10.93–19.00]) of entries were not accurate with a wide variation (range; 6.21–27.44%) between centers. Aspects for improvement of the situation were identified.

Conclusion

Due to the high inaccuracy, the high number of centers lacking mandatory entries at all and the number of false entries, these data alert us to advocate unannounced audits and further measures to improve the situation. Data should not be used for the time being, since wrong conclusion will be drawn.

Graphic abstract

These slides can be retrieved under Electronic Supplementary Material.

Introduction

Registries were designed to acquire comprehensive data on treatment effects and natural course of diseases [1]. Particularly in spine surgery, registries have been adopted very early. The first structured and large-scale spine registry was the Swedish Spine Registry; Spine Tango was the first international [2, 3]. Thus, we look back on more than two decades of experience with data originating from registries, which are increasingly used for clinical science and quality management.

Randomized controlled trials (RCTs) enable us to gain level I evidence for treatment strategies but have the disadvantage of well-defined subgroups as their target of research and a standardized environment that does not reflect clinical reality.

Registries have the hypothetical advantage to acquire data from a complete population with a specific disease treated in a real-world setting. Thus, clinical studies with data from registries are considered to ideally complementing RCTs or vice versa.

Data originating from clinical registries become also increasingly important today for quality management purposes to all stakeholders of our healthcare systems, e.g., health insurances, healthcare providers, politicians, professionals and their respective scientific societies. For example, new legislation in Europe requires that medical products such as spinal implants can be individually followed and are correlated with clinical data. Only registries can fulfill these requirements. It is further to be expected or already in place that decisions regarding reimbursement of treatment or even sanctions against hospitals and/or surgeons are derived from comparative registry data.

Because these two purposes are highly important and may have far reaching consequences, data quality of registries itself with regard to completeness and accuracy becomes of utmost importance.

In strong contrast to this stands the fact that few spine registries have external monitoring, which would be a first prerequisite to assure data quality.

Another obstacle for almost all registries with exception of the Scandinavian ones is that most national data protection laws require strictly anonymized data entries, which prohibits any relevant and accurate follow-up for very important markers such as reoperation rates or 30-day readmissions.

Thus, the aim of this study was to investigate the accuracy and completeness of data from registries as they are today via the example of the German Spine Society Registry. Since we are convinced that the findings can be extrapolated to most other registries, we consider our message important beyond the spine community.

Methods

Ethics approval and consent to participate

The local ethics committee of our university confirmed the study (registration number 42/16). All patients gave written informed consent to the registry.

Literature search prior to the study

We performed a search for currently ongoing registries and their data quality via Medline, Google, clinicaltrials.gov, and inquiries to societies. We included registries with focus on scientific, socioeconomic and healthcare provider perspectives, which are used and acknowledged by national scientific societies or governmental organizations. For the purpose of this article, we did not consider registries of scientific organization and/or universities/hospitals, focusing on one specific spinal disease or one scientific question. Search criteria were “audit,” “registry,” “accuracy,” “data completeness,” and “monitoring.”

Study design

STROBE guidelines were used to execute this study in accordance with cohort studies.

The German Spine Society (Deutsche Wirbelsäulengesellschaft, DWG) was contacted and confirmed the institutional certification of 23 German centers. Institutional certification is linked to mandatory data input into the DWG registry.

All the centers were contacted, and the participating centers were chosen according to Fig. 1. The minimum mandatory requirement upon certification was to document in the registry all inpatient data of patients undergoing spine surgeries. For the audit visit, a 2-week period was defined which was 2 months before the initial contact to the centers, in which all patients, which had had spine surgery, were analyzed. In a prospective study, design audits by a board-certified neurosurgeon were then planned and conducted at all participating centers. The percentage of surgeries that were not documented in the registry at the time of the first contact concerning the study as well as the overall rate of undocumented cases were analyzed. Moreover, any discrepancy between the patients’ charts and the registry entry was detected and analyzed at the audit visit. A mean of 30.95 ± 3.05 items per patient (median 31, range 0–40) and a mean of 20.92 ± 7.96 (median 20, range 11–38) cases per center were analyzed.

Variables

The analyzed items per case included main diagnosis, type of fracture, type of degeneration, classification of spondylolisthesis and osteoporosis, affected levels, type of tumor, previous surgeries, type and duration of symptoms, date of surgery, surgeon, implants, type of surgery, postoperative course, complications, etc. Table 1 provides a comprehensive overview.Table 1 Analyzed itemsFull size table

Statistical analysis

The SPSS v.24 (IBM Inc., Armonk, New York) statistical software package and R statistical software (version 3.1.0; https://www.r-project.org/), Vienna, Austria were used for the analysis. The objective of our study was to deliver data on the general accuracy of data originating from a large national registry per se and on the other hand to identify potential factors leading to an increased rate of wrong entries depending on the item.

Results are presented as proportions or means with 95% confidence intervals (CI). Wilcoxon–Mann–Whitney statistics and Kruskal–Wallis statistics were used to test for differences in accuracy rates of the entries depending on a) monitoring of the data entry and b) the person performing the data entry were entered.

Results

Overall data completeness

Out of 23 certified spine centers contacted, 17 centers were willing to participate, but 4 were still lacking any data entries. Even in the remaining 13 centers eligible for audits, 28.50% (95%-CI = [22.46–34.55]) of entries were finalized only after the audits were announced. Thus, 71.50% (95%-CI = [65.45–77.54]; range between centers: 18.18–100%) of all operated patients were completely entered into the registry at the time of the first contact (Fig. 2).

At the time of the visit only, 82.55% (95%-CI = [79.12–85.98]; range between centers: 20.41–100%) of surgeries were documented in the registry although all centers are required to enter all operated spine cases without exception (Fig. 3).

Incorrect entries per center

On average, 14.95% (95%-CI = [10.93–19.00]) of entries were not accurate with a variation (range: 6.21–27.44%) between centers (Fig. 4). The setup also varied largely in the centers. While the registry entries were done by the operating surgeon in 10 centers, they were done by a study nurse in three centers. Looking at the influence of the person entering data, there is a significant difference with fewer errors if the surgeon (12.02% [10.47–13.58]) performs the data entry compared to a study nurse (15.65% [14.22–17.68]) (Fig. 5; p = .0004).

Table 1 shows the percentage of wrong registry entries per item. Wrong entries per item varied largely and were between .4 and 50% of the entries. It is worth mentioning that complications as one of the most useful data from large registries showed a failure rate of 50%.

Influence of internal monitoring

In four centers, no internal monitoring was performed. Two centers perform an internal check whether the patient was entered into the registry (quantity) but without any content control while there is a quality monitoring in seven centers, which is either done by an attending or by a study nurse. Yet, the percentage of wrong entries did not differ significantly for all three groups (no monitoring: 10.10% [8.75–11.46]; quantitative monitoring: 17.83% [11.96–23.67]; qualitative monitoring: 14.72% [12.29–17.15], p = .008; Fig. 6a). Comparing “no qualitative monitoring” (12.00% [10.16–13.84]) vs. “qualitative monitoring” (14.72% [12.29–17.15]), we also do not observe any significant difference (p = .084; Fig. 6b).

Discussion

Added value of this study

Our study will raise awareness on the general shortcomings of registries even if supervised by large societies and will help to improve future registry design and healthcare decisions. It is particularly valuable because it is actually the first study to determine the accuracy and completeness of a large nationwide registry.

In the light of today’s socioeconomic and health political influence of data gained from large national registries, this study provides profound evidence that the design and structure of most current registries does not provide reliable data sufficient for political, medical, or economic decision making.

General aspects

Our results indicate that the data of the DWG registry are neither suitable for clinical science purposes nor quality management for the time being. Compliance with mandatory data input was merely moderate. Completeness of data was not satisfactory on average, but especially inacceptable in a significant fraction of individual centers despite the fact that only the minimum data set was required, i.e., a reduced inpatient data set without follow-up. Accuracy of data was also not satisfactory, again with a wide variation between centers. Internal non-mandatory monitoring did neither influence completeness nor accuracy of data.

Data accuracy and audits

There are various aspects, which could improve data quality and should also be pursuit such as a strict definition of validated key metrics for each disease, enabling automated data extraction, and providing adequate financing for sufficient resources.

The DWG registry examined in this study is based on the Spine Tango platform. As mentioned above, the reliability and usefulness of the results are inherently dependent on the quality and completeness of data being entered into the registry. RCTs undergo regular monitoring visits to ensure accurate and complete data entry, but most of the current registries fail to have mandatory comprehensive-auditing procedures. The problems identified in this registry seem therefore inherent to the majority of spine registries and probably beyond.

Of course, in order to prevent selection bias and also to account for the actual variances between centers concerning outcomes, multivariate analyses are required adjusting for covariates [4]. But multivariate analyses can only handle known covariates or bias. If these are unknown or not recorded by the registry, we have no chance to clear the registry data from these influences potentially leading to wrong or unrepresentative results. Thus, the finer granularity is enabled by the required items per patients, the more accurate and the more unlikely that the data set suffers from unknown bias. However, fine granularity is more elaborate for the centers, thus leading to incomplete data sets. Regarding Table 1, there are items that are more prone to incorrect entries than others, and it is important to control these issues and change the items themselves accordingly.

It is important to say that the large variance between centers (Fig. 4) also points out that we need to differentiate between inherent problems of a registry itself causing low data accuracy and completeness and culture in the participating centers themselves.

Likewise, a large amount of missing data will also limit the value of the registry data. Thus, direct and real-time internal monitoring of data quality and integrity is crucial when designing and planning a registry. Moreover, without an active external monitoring or auditing process, the danger of selective reporting/entry is obvious. Particularly, the reported rate of complications seems extremely low in our study (Table 1). Bias can occur by factors such as omitting whole groups of high-risk patients or liberal interpretation of what constitutes as a complication. Both factors are known to occur if no rigorous guidelines for reporting are established and ultimately controlled. To our knowledge, this is not the case for any European spine registry at least.

Follow-up versus data protection

Another problem not examined in this study is the lack of relevant follow-up. Two aspects exist. First, follow-up rates tend to be low in general, but for clinical research, collecting follow-up data remains crucial. It is known that any loss of follow-up larger than 20% of the enrolled cases can lead to a significant bias in the statistical evaluation [5]. In a study by McGirt et al., the follow-up rate in spine registries was found to be between 22 and 79% [6]. Thus, even the registry with the best follow-up rate was worse than minimally recommended one. In our study, only 71.50% (95%-CI = [65.45–77.54]; range between centers: 18.18–100%) of all operated patients were completely entered into the registry at the time of the first contact (Fig. 2); and this even only reflects follow-up until discharge.

This issue relates to the current state of legislation regarding data protection in most countries and is problematic for quality management. To calculate some of the key indicators for quality of care such as 30-day readmission or reoperation rates, a patient would need to be traced, irrespective of the hospital taking care of the complication. Particularly in urban areas with a high density of specialists, a significant number of patients will seek a second opinion for problems occurring after the index surgery. Identification of a given patient for longitudinal follow-up is crucial. Outside Scandinavia, however, this is not possible in any other country we are aware of, which means that registries in those countries provide data that will clearly underestimate important quality indicators.

Further approaches to improve data quality

In order to find further solutions for the imminent problems raised by this study, awareness among peers teaching on the potentially crucial implications of registry data but also possible financial incentives could be used. Changing culture is another part.

Yet, when looking at the involved centers, those were the first certified spine centers in Germany and not only by this fact but also by personal knowledge highly motivated professionals, in part leaders in the field and well motivated. Thus, blaming lack of motivation, culture of education on the topic might not serve as complete solution. Other issues, such as shortness of staff, amount of required data and lacking usability of the registry software itself, should also be taken into account.

Conclusion

Due to the high inaccuracy, the high number of centers lacking mandatory entries at all and the number of false entries these data alert us to advocate unannounced audits and other measures to improve the situation such as financial incentives and education on the benefits and consequences of such data. The current data should not be used for the time being, since wrong conclusion will inevitably be drawn. Aspects for improvement of the situation were identified.

Abbreviations

DWG:

German Spine SocietyRCT:

Randomized controlled trialCI:

Confidence interval

References

1. 1.Melloh M, Staub L, Aghayev E, Zweig T, Barz T, Theis JC, Chavanne A, Grob D, Aebi M, Roeder C (2008) The international spine registry SPINE TANGO: status quo and first results. Eur Spine J 17(9):1201–1209. https://doi.org/10.1007/s00586-008-0665-2Article PubMed PubMed Central Google Scholar 
2. 2.Fritzell P, Stromqvist B, Hagg O (2006) A practical approach to spine registers in Europe: the Swedish experience. Eur Spine J 15(Suppl 1):S57–63. https://doi.org/10.1007/s00586-005-1051-yArticle PubMed Google Scholar 
3. 3.Stromqvist B, Fritzell P, Hagg O, Jonsson B, Sanden B (2013) Swespine: the Swedish spine register: the 2012 report. Eur Spine J 22(4):953–974. https://doi.org/10.1007/s00586-013-2758-9Article PubMed PubMed Central Google Scholar 
4. 4.Wood AM, White IR, Thompson SG (2004) Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials 1(4):368–376. https://doi.org/10.1191/1740774504cn032oaArticle PubMed Google Scholar 
5. 5.Schulz KF, Grimes DA (2002) Sample size slippages in randomised trials: exclusions and the lost and wayward. Lancet 359(9308):781–785. https://doi.org/10.1016/S0140-6736(02)07882-0Article PubMed Google Scholar 
6. 6.McGirt MJ, Parker SL, Asher AL, Norvell D, Sherry N, Devin CJ (2014) Role of prospective registries in defining the value and effectiveness of spine care. Spine (Phila Pa 1976) 39((22 Suppl 1)):S117–128Article Google Scholar 

Download references

Acknowledgements

Open Access funding provided by Projekt DEAL.

Funding

The study was completely financed by institutional grants from the Department of Neurosurgery.

Author information

Affiliations

1. Department of Neurosurgery, Klinikum rechts der Isar, Technical University Munich, Ismaninger Str. 22, 81675, Munich, Germany
   • Bernhard Meyer
   • , Ehab Shiban
   • , Lucia E. Albers
   •  & Sandro M. Krieg

Corresponding author

Correspondence to Sandro M. Krieg.

Ethics declarations

Conflict of interest

The study was financed by institutional grants. SK is consultant for Nexstim Plc (Helsinki, Finland) and Spineart Germany (Frankfurt, Germany) and received honoraria from Medtronic (Meerbusch, Germany) and Carl Zeiss Meditec (Oberkochen, Germany). SK and BM received research grants and are consultants for Brainlab AG (Munich, Germany). BM received honoraria, consulting fees, and research grants from Medtronic (Meerbusch, Germany), icotec ag (Altstätten, Switzerland), and Relievant Medsystems Inc., Sunnyvale, CA, USA), honoraria and research grants from Ulrich Medical (Ulm, Germany), honoraria and consulting fees from Spineart Deutschland GmbH (Frankfurt, Germany) and DePuy Synthes (West Chester, PA, USA), and royalties from Spineart Deutschland GmbH (Frankfurt, Germany). However, all authors declare that they have no conflict of interest regarding the materials used or the results presented in this study.

Ethical approval

Ethics committee registration number 42/16.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (PPTX 623 kb)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.