The learning curves of a validated virtual reality hip arthroscopy simulator

The learning curves of a validated virtual reality hip arthroscopy simulator

Bartlett, J.D., Lawrence, J.E., Yan, M. et al. The learning curves of a validated virtual reality hip arthroscopy simulator. Arch Orthop Trauma Surg 140, 761–767 (2020). https://doi.org/10.1007/s00402-020-03352-3

Abstract

Introduction

Decreases in trainees’ working hours, coupled with evidence of worse outcomes when hip arthroscopies are performed by inexperienced surgeons, mandate an additional means of training. Though virtual reality simulation has been adopted by other surgical specialities, its slow uptake in arthroscopic training is due to a lack of evidence as to its benefits. These benefits can be demonstrated through learning curves associated with simulator training—with practice reflecting increases in validated performance metrics.

Methods

Twenty-five medical students with no previous experience of hip arthroscopy completed seven weekly simulated arthroscopies of a healthy virtual hip joint using a 70° arthroscope in the supine position. Twelve targets were visualised within the central compartment, six via the anterior portal, three via the anterolateral portal and three via the posterolateral portal. Task duration, number of collisions (bone and soft-tissue), and distance travelled by arthroscope were measured by the simulator for every session of each student.

Results

Learning curves were demonstrated by the students, with improvements in time taken, number of collisions (bone and soft-tissue), collision length and efficiency of movement (all p < 0.01). Improvements in time taken, efficiency of movement and number of collisions with soft-tissue were first seen in session 3 and improvements in all other parameters were seen in session 4. No differences were found after session 5 for time taken and length of soft-tissue collision. No differences in number of collisions (bone and soft-tissue), length of collisions with bone, and efficiency of movement were found after session 6.

Conclusions

The results of this study demonstrate learning curves for a hip arthroscopy simulator, with significant improvements seen after three sessions. All performance metrics were found to improved, demonstrating sufficient visuo-haptic consistency within the virtual environment, enabling individuals to develop basic arthroscopic skills.

Introduction

The increasing utilisation of more technically demanding surgical procedures, coupled with decreases in trainees’ caseloads and working hours, has led to difficult learning curves in orthopaedic surgical training [1,2,3,4,5,6,7]. This is particularly true for the rapidly expanding field of hip arthroscopy. The geometry of the joint, combined with its thick soft-tissue envelope, makes hip arthroscopy particularly challenging and long learning curves have previously been described [3, 8,9,10]. Concerns regarding patient safety, fuelled by evidence of worse outcomes when arthroscopic procedures are performed by inexperienced surgeons, mandate the development of an additional means of training arthroscopic surgeons [3, 9].

Though virtual reality (VR) simulation training has been increasingly adopted by other surgical specialities, its use in arthroscopic surgical training remains limited [5, 11]. This slow uptake is, in part, the result of a lack of robust evidence as to the benefits of VR training to both the trainee and the patient [5, 11]. One means of demonstrating these benefits to the trainee is through the analysis of the learning curves associated with simulator training—with repeated practice reflecting measurable increases in validated performance metrics. Though virtual reality simulators have been suggested to shorten surgical learning curves, there are currently only reports in the literature on box simulators and low fidelity simulators [11,12,13]. Without demonstration of learning curves, it cannot be assumed that the visuo-haptic feedback provided by the VR simulator is sufficiently consistent to allow skill development.

The purpose of this study was to investigate how novice performance on a VR hip arthroscopy simulator’s visualisation module changed with repeated practice. We hypothesised that repeated use of the simulator would lead to significant decreases in time taken, number of collisions and distance travelled by the arthroscope before reaching a plateau, thus demonstrating learning curves.

Methods

Participants

Twenty-five medical students were voluntarily recruited for this study. Basic demographic information including gender, age, year of study and arthroscopic experience was collected.

Simulator

The previously validated Simbionix ArthroMentorVR simulator (3D Systems, Littleton, USA) was used in this study [14, 15]. This simulator consisted of a computer and monitor, a mannequin, and two haptic feedback devices capable of providing tactile feedback to a pair of instruments via connecting motors (Fig. 1a, b). The mannequin had four predefined 5 mm arthroscopy portals at the modified anterior, anterior, anterolateral, and posterolateral sites. The images of the virtual joint were displayed on the monitor in response to the movements of the operator.

Arthroscopy simulator protocol

All participants received an identical standardised introduction by the same individual. In this, participants were introduced to the basic principles of hip arthroscopy and VR simulation and shown a demonstration of the full diagnostic examination of the hip joint on the simulator. Following this, each participant was given a familiarisation period of exactly 3 min in which they could examine the hip joint from all three portals.

Each participant completed seven identical diagnostic hip examination modules in the supine position. Each participant completed one session every 7 days. Each module began with standardised written instructions on screen, with the procedure starting upon insertion of the arthroscope into the anterolateral portal. The module involved locating a series of 12 consecutive targets within the hip joint using an arthroscope. Six targets were visualized via the anterolateral portal, followed by three via the anterior, and finally three via the posterolateral (Table 1). The name of each target was displayed to the participant on-screen and the order in which the targets appeared was identical for each participant. Participants were required to place each target in the centre of the monitor for 3 s before the name of the next target in the examination sequence was displayed to them on-screen.Table 1 List of targets visualised during diagnostic hip moduleFull size table

During every attempt, the participants’ performance was evaluated by the simulator via a set of predefined metrics. These included the total time taken to complete the procedure, the number of collisions between the soft-tissue and the arthroscope, the number of collisions between bone and the arthroscope, the total time of soft-tissue collisions (seconds), the total length of femoral head scratches (mm), and the distance travelled by the arthroscope (cm).

Ethical approval

As per the National Health Service (NHS) Health Research Authority’s guidance, this study did not require approval from an NHS Research Ethics Committee [16]. This study was conducted in agreement with the ethical standards of the University of Cambridge, the NHS Research Ethics Committee and the 1964 Helsinki Declaration.

Statistical analysis

Statistical analysis was performed with version 3.2 of R (Foundation for Statistical Computing, Vienna, Austria). A power analysis was performed to estimate the sample size required to detect improvement across the sessions. On the basis of pilot attempts by the authors, the total task time was estimated to be 900 s for the first session. On the basis of previous studies, the standard deviation for a simulated arthroscopic task was estimated to be 40% of the task time, with a difference of 25% considered clinically relevant [13, 17]. Therefore, to achieve an 80% power (alpha = 0.05), the desired sample size was calculated to be 25.

Demographic data are presented as mean ± standard deviation. A Kilmogorov–Smirnov test revealed the data collected by the simulator to be non-parametric. As such, non-parametric tests were used, and all data presented as median ± interquartile range. A Friedman test with multiple comparison was used to analyse the differences between successive attempts at the module. Comparisons were made between session 1 and subsequent session for all metrics to define the first significant improvement. Comparisons were also made between session 7 and all session to elucidate when individuals’ improvements ceased to be significant.

Results

The mean age of the students was 21.7 ± 1.8 and the male to female ratio was 19:6. 12% of students were in their third year of study, 48% were in their fourth year of study and 40% were in their fifth year of study. The mean number of times using an arthroscope (simulated or real) amongst the students was 0.3 ± 0.7 and no student had ever used a hip arthroscope (simulated or real).

Learning curve were demonstrated by the students, with significant improvements in all parameters following training (Fig. 2). Average total time decreased by nearly 75% across the course—936 s (657–1688) to 261 s (207–379) (p < 0.0001) (Table 2). The first significant increase from the participants baseline was seen in session 3 (p = 0.024) and no significant difference from session 7 (peak performance) was first found at session 5.

Table 2 Median student performance in all parameters with Friedmann’s multiple comparisons testFull size table

Collisions between the arthroscope and soft-tissues decreased across the sessions by nearly 90%—91 (44–126) to 10 (5–27) (p < 0.0001). The first significant improvement from baseline was found at session 3 (p = 0.013) with no significant differences from session 7 found at session 6. Similarly, collisions between the femoral head and the arthroscope also decreased—34 (25–79) to 7 (4–9) (p < 0.0001). These improvements were first seen in session 4 (p = 0.0007) and no significant difference from session 7 was first found at session 6.

Total time of soft-tissue collision was found to decrease from 527 s (369–994) to 111 s (94–144) (p < 0.0001), with the first significant decrease seen in session 4 (p = 0.0003) and no significant difference from session 7 was first found in session 5. Length of femoral head scratches showed similar decreases—190 mm (120–285) to 34 mm (22–57) (p < 0.0001), with the first significant decrease in session 5 (p = 0.005) and no significant difference from session 7 first found in session 6.

Participant’s efficiency of movement was also found to improve, with total camera distance reaching 143 cm (67–212) from an initial 440 cm (355–1156) (p < 0.0001). The first significant increase was seen in session 3 and no significant difference from session 7 was first found in session 6.

Discussion

The most important finding of the present study was the demonstration a learning curve for a visualisation module on a previously validated hip arthroscopy simulator [15]. Significant increases were seen in six simulator measured parameters after a minimum of three sessions, before a plateau was reached after a maximum of six sessions. These results mimic those of similar studies into the learning curves associated with virtual reality orthopaedic simulators, and support its potential use in the development of basic hip arthroscopic skills such as visualisation [13, 18,19,20,21,22,23,24]. The assessment of learning curves, or ‘training effects’ is an essential aspect of VR simulator validation. It demonstrates that the VR environment generated by the simulator has sufficient visuo-haptic consistency, enabling individuals to develop psychomotor skills within it. Without demonstration of this training effect, it cannot be assumed that the visuo-haptic feedback provided by the VR simulator is effective in enabling dextrous development.

Previous studies have demonstrated learning curves and training effects for other arthroscopic simulators. Pollard et al. demonstrated a learning curve for lateral and supine patient positions in simulated hip arthroscopy using a box simulator, showing improvements in time taken and efficiency of movement [13]. Similar learning curves have been demonstrated using the Sheffield Knee Arthroscopy Training System and a passive haptic knee arthroscopy simulator with medical students [19]. A particularly difficult learning curve for knee arthroscopy was noted in a similar study by Rahm et al. using a passive haptic knee arthroscopy simulator [23]. Additionally, a learning curve has been demonstrated for the insightMIST (GMV, Madrid, Spain) shoulder VR simulator, further supporting shoulder VR simulators’ implementation in surgical training [20]. Furthermore, the retention of the skills acquired during simulation has been investigated by monitoring performance over a more protracted period, demonstrating limited degradation over time [18, 21].

A small number of studies have assessed the real-world benefits of VR simulators, showing improved performance in an operative setting following simulation training [5, 11]. Cannon et al. showed orthopaedic residents randomised to receive VR simulator training outperformed their control group counterparts in an operating theatre [25]. These benefits to knee arthroscopy were also assessed by Waterman et al. who measured the improvements to performance, and compared them to another group trained on cadaveric specimens [26]. Similarly, Banaszek et al.’s study assessed improvements in arthroscopic performance in medical students trained on a VR knee arthroscopy simulator, demonstrating significantly greater performance when compared to controls [27]. Contrary to these promising results, the benefits of knee arthroscopy simulation were not found to be significant in a study by Rebolledo et al. when testing orthopaedic residents’ performances on cadaveric models after simulation training [28]. However, this study did find significant improvements in shoulder arthroscopy performance compared to controls.

Limitations

Although the demonstration of a learning curve for this simulator is reassuring, it does not quantify the real-world benefits of training with this simulator—concurrent validity. Further studies are, therefore, required to confirm and quantify the concurrent validity of this simulator. This additional information will better inform orthopaedic training programmes as to the effectiveness of simulation training, and, as to when in a trainee’s career it should be utilised for the greatest benefit.

Additionally, this study is unable to provide insight into the potential cost benefits of simulator training, something particularly important given the high initial cost of purchasing a VR simulator. The ‘Transfer Effectiveness Ratio’ (TER), the only validated measure of cost effectiveness, is a tool commonly used in the aviation industry to quantify the difference in time required to achieve fully competent performance between virtual reality and real-life, with a ratio of 0.50 indicating that 1 h of simulator training saves approximately 30 min of operative training time [29, 30]. Implementation of such a measure in the future assessment of orthopaedic virtual reality simulators would allow both direct comparison between validated simulators and more accurate calculation of cost savings of simulator training.

This study has only demonstrated the learning curves for one of this simulator’s modules and one arthroscopic skill—‘scope manipulation’. This simulator contains numerous modules for hip arthroscopy, including probing modules, complex pathological cases, and shoulder and knee arthroscopy set-ups. Our results only show the learning curve of the module utilised and any conclusions drawn cannot be extended to the other modules of this simulator or other skills, for example, probing.

In addition to this, the rapid improvements displayed in this study are far quicker than the measurable improvements found in real-world hip arthroscopy learning curves. A recent systematic review suggested that after 30 cases significant reductions in operative time and complication rates are seen with surgeons training in hip arthroscopy [3]. Although the shorter learning curve found with this VR simulator suggests that VR simulation lacks the heterogeneous nature of arthroscopic cases seen in the real world, this is due to the role of the modules tested in training basic arthroscopic skills, such as camera manipulation and triangulation—they are not intended to mimic the complexity of real-world pathological cases. Furthermore, though a plateau was demonstrated after six sessions, this conclusion is limited as it was found toward the end of the testing period. As such further improvements may have been detected if additional sessions were tested.

Conclusions

The results of this study demonstrate learning curves for a hip arthroscopy simulator, with significant improvements seen after three sessions. All performance metrics were found to improved, demonstrating sufficient visuo-haptic consistency within the virtual environment, enabling individuals to develop basic arthroscopic skills.

References

1. 1.Connors RC, Doty JR, Bull DA et al (2009) Effect of work-hour restriction on operative experience in cardiothoracic surgical residency training. J Thorac Cardiovasc Surg 137:710–713. https://doi.org/10.1016/j.jtcvs.2008.11.038Article PubMed Google Scholar 
2. 2.Lee Y-K, Ha Y-C, Hwang D-S, Koo K-H (2013) Learning curve of basic hip arthroscopy technique: CUSUM analysis. Knee Surg Sports Traumatol Arthrosc 21:1940–1944. https://doi.org/10.1007/s00167-012-2241-xArticle PubMed Google Scholar 
3. 3.Hoppe DJ, de Sa D, Simunovic N et al (2014) The learning curve for hip arthroscopy: a systematic review. Arthrosc J Arthrosc Relat Surg 30:389–397. https://doi.org/10.1016/j.arthro.2013.11.012Article Google Scholar 
4. 4.Konan S, Rhee S-J, Haddad FS (2011) Hip arthroscopy: analysis of a single surgeon’s learning experience. J Bone Joint Surg Am 93(Suppl 2):52–56. https://doi.org/10.2106/JBJS.J.01587Article PubMed Google Scholar 
5. 5.Bartlett JD, Lawrence JE, Stewart ME et al (2018) Does virtual reality simulation have a role in training trauma and orthopaedic surgeons? Bone Jt J 100-B:559–565. https://doi.org/10.1302/0301-620X.100B5.BJJ-2017-1439CAS Article Google Scholar 
6. 6.Schüttler KF, Schramm R, El-Zayat BF et al (2018) The effect of surgeon’s learning curve: complications and outcome after hip arthroscopy. Arch Orthop Trauma Surg 138:1415–1421. https://doi.org/10.1007/s00402-018-2960-7Article PubMed Google Scholar 
7. 7.Olach M, Gerhard P, Giesinger K et al (2019) Clinical and radiological outcome at mean follow-up of 11 years after hip arthroscopy. Arch Orthop Trauma Surg 139:1–6. https://doi.org/10.1007/s00402-018-3024-8Article PubMed Google Scholar 
8. 8.Nakano N, Khanduja V (2016) Complications in hip arthroscopy. Muscles Ligaments Tendons J 6:402–409. https://doi.org/10.11138/mltj/2016.6.3.402Article PubMed PubMed Central Google Scholar 
9. 9.Mehta N, Chamberlin P, Marx RG et al (2018) Defining the learning curve for hip arthroscopy: a threshold analysis of the volume-outcomes relationship. Am J Sports Med 46:1284–1293. https://doi.org/10.1177/0363546517749219Article PubMed Google Scholar 
10. 10.Erturan G, Alvand A, Judge A et al (2018) Prior generic arthroscopic volume correlates with hip arthroscopic proficiency: a simulator study. J Bone Jt Surg Am 100:e3. https://doi.org/10.2106/JBJS.17.00352Article Google Scholar 
11. 11.Aïm F, Lonjon G, Hannouche D, Nizard R (2016) Effectiveness of virtual reality training in orthopaedic surgery. Arthrosc J Arthrosc Relat Surg 32:224–232. https://doi.org/10.1016/j.arthro.2015.07.023Article Google Scholar 
12. 12.Akhtar KSN, Chen A, Standfield NJ, Gupte CM (2014) The role of simulation in developing surgical skills. Curr Rev Musculoskelet Med 7:155–160. https://doi.org/10.1007/s12178-014-9209-zCAS Article PubMed PubMed Central Google Scholar 
13. 13.Pollard TCB, Khan T, Price AJ et al (2012) Simulated hip arthroscopy skills: learning curves with the lateral and supine patient positions. J Bone Jt Surg 94:e68. https://doi.org/10.2106/JBJS.K.00690Article Google Scholar 
14. 14.Bartlett JD, Lawrence JE, Khanduja V (2018) Virtual reality hip arthroscopy simulator demonstrates sufficient face validity. Knee Surg Sports Traumatol Arthrosc. https://doi.org/10.1007/s00167-018-5038-8Article PubMed PubMed Central Google Scholar 
15. 15.Khanduja V, Lawrence JE, Audenaert E (2017) Testing the construct validity of a virtual reality hip arthroscopy simulator. Arthrosc J Arthrosc Relat Surg 33:566–571. https://doi.org/10.1016/j.arthro.2016.09.028Article Google Scholar 
16. 16.NHS Health Research Authority (2019) Do I need NHS Ethics approval? In: Httpwwwhra-Decis. https://www.hra-decisiontools.org.uk/ethics/. Accessed 27 Jan 2019
17. 17.Howells NR, Auplish S, Hand GC et al (2009) Retention of arthroscopic shoulder skills learned with use of a simulator. Demonstration of a learning curve and loss of performance level after a time delay. J Bone Jt Surg Am 91:1207–1213. https://doi.org/10.2106/JBJS.H.00509CAS Article Google Scholar 
18. 18.Bliss JP, Hanner-Bailey HS, Scerbo MW (2005) Determining the efficacy of an immersive trainer for arthroscopy skills. Stud Health Technol Inform 111:54–56PubMed Google Scholar 
19. 19.McCarthy AD, Moody L, Waterworth AR, Bickerstaff DR (2006) Passive haptics in a knee arthroscopy simulator: is it valid for core skills training? Clin Orthop 442:13–20Article Google Scholar 
20. 20.Andersen C, Winding TN, Vesterby MS (2011) Development of simulated arthroscopic skills. Acta Orthop 82:90–95. https://doi.org/10.3109/17453674.2011.552776Article PubMed PubMed Central Google Scholar 
21. 21.Jackson WFM, Khan T, Alvand A et al (2012) Learning and retaining simulated arthroscopic meniscal repair s
    kills. J Bone Jt Surg Am 94:e132. https://doi.org/10.2106/JBJS.K.01438CAS Article Google Scholar 
22. 22.Sugand K, Mawkin M, Gupte C (2016) Training effect of using Touch Surgery^TM for intramedullary femoral nailing. Injury 47:448–452. https://doi.org/10.1016/j.injury.2015.09.036Article PubMed Google Scholar 
23. 23.Rahm S, Wieser K, Wicki I et al (2016) Performance of medical students on a virtual reality simulator for knee arthroscopy: an analysis of learning curves and predictors of performance. BMC Surg 16:14. https://doi.org/10.1186/s12893-016-0129-2Article PubMed PubMed Central Google Scholar 
24. 24.Sugand K, Akhtar K, Khatri C et al (2015) Training effect of a virtual reality haptics-enabled dynamic hip screw simulator. Acta Orthop 86:695–701. https://doi.org/10.3109/17453674.2015.1071111Article PubMed PubMed Central Google Scholar 
25. 25.Cannon WD, Garrett WE, Hunter RE et al (2014) Improving residency training in arthroscopic knee surgery with use of a virtual-reality simulator. A randomized blinded study. J Bone Jt Surg Am 96:1798–1806. https://doi.org/10.2106/JBJS.N.00058Article Google Scholar 
26. 26.Waterman BR, Martin KD, Cameron KL et al (2016) Simulation training improves surgical proficiency and safety during diagnostic shoulder arthroscopy performed by residents. Orthopedics 39:e479–485. https://doi.org/10.3928/01477447-20160427-02Article PubMed Google Scholar 
27. 27.Banaszek D, You D, Chang J et al (2017) Virtual reality compared with bench-top simulation in the acquisition of arthroscopic skill: a randomized controlled trial. J Bone Jt Surg Am 99:e34. https://doi.org/10.2106/JBJS.16.00324Article Google Scholar 
28. 28.Rebolledo BJ, Hammann-Scala J, Leali A, Ranawat AS (2015) Arthroscopy skills development with a surgical simulator: a comparative study in orthopaedic surgery residents. Am J Sports Med 43:1526–1529. https://doi.org/10.1177/0363546515574064Article PubMed Google Scholar 
29. 29.Moglia A, Ferrari V, Morelli L et al (2016) A systematic review of virtual reality simulators for robot-assisted surgery. Eur Urol 69:1065–1080. https://doi.org/10.1016/j.eururo.2015.09.021Article PubMed Google Scholar 
30. 30.Roscoe SN (1971) Incremental transfer effectiveness. Hum Factors 13:561–567. https://doi.org/10.1177/001872087101300607Article Google Scholar 

Download references

Funding

No funding was received for this study.

Author information

Affiliations

1. School of Clinical Medicine, Addenbrooke’s Hospital, Cambridge, UK
   • Jonathan D. Bartlett
   • , Matthew Yan
   •  & Max E. Stewart
2. Department of Trauma and Orthopaedics, Ghent University, Ghent, Belgium
   • Emmanuel Audenaert
3. Young Adult Hip Service, Department of Trauma and Orthopaedics, Addenbrooke’s, Cambridge University Hospitals NHS Foundation Trust, Addenbrooke’s Hospital, Box 37, Hills Road, Cambridge, CB2 0QQ, UK
   • John E. Lawrence
   • , Borna Guevel
   •  & Vikas Khanduja

Corresponding author

Correspondence to Vikas Khanduja.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.