Virtual Reality Environment for the Validation of Bone Fracture
Reduction Processes
J. J. Jim
´
enez-Delgado
a
, A. Calzado-Mart
´
ınez
b
, F. D. P
´
erez-Cano
c
and A. Luque-Luque
d
Graphics and Geomatics Group, University of Ja
´
en, Ja
´
en, Spain
Keywords:
Fracture Reduction, Virtual Reality, Reduction, Validation, Usability, Interaction.
Abstract:
This work presents a virtual environment for the validation by experts of computer-assisted bone fracture
reduction. This environment is composed of VR glasses and 3D controllers (HTC Vive) that allow interaction
and immersion in the scene in a realistic way. The virtual environment developed allows loading fractured
bone models (fragments) so that the specialist performs a virtual fracture reduction and its results can be used
for the validation of algorithms and assisted reduction techniques. Once the fragments are loaded, the user can
perform an interactive reduction of the fracture, visualizing the fragments in 3d from different views, moving
the fragments in 3d and placing the fragments in space to observe the reduction in detail. Once completed,
it allows the reduction to be exported so that it can be compared with other fracture reduction systems. The
system has been tested by specialists in traumatology and a usability study has been carried out. Finally,
the system has been empirically validated and used to compare the performance of other computer-assisted
reduction systems.
1 INTRODUCTION
Fracture reduction is a surgical procedure to repair a
fracture or dislocation in the correct alignment. This
sense of the term reduction does not imply any type
of elimination or quantitative decrease, but rather im-
plies a restoration. It must be taken under consider-
ation that multiple fragments can be generated in a
fracture, some of them with microscopic size. The re-
duction of a fracture requires the union of the larger
fragments, ignoring the smaller ones. Therefore, the
reduction of a fracture does not imply that the frag-
ments must be completely joined at all points.
When a bone is fractured, the fragments lose their
alignment in the form of displacement or angulation.
For the fractured bone to heal without any deformity,
the bone fragments must be realigned to their normal
anatomical position. Orthopedic surgery attempts to
recreate the normal anatomy of the fractured bone by
reducing displacement.
The goal of virtual reality is to create an immer-
sive environment that is as realistic as possible. The
a
https://orcid.org/0000-0003-3014-0496
b
https://orcid.org/0000-0001-8690-6409
c
https://orcid.org/0000-0002-8065-8173
d
https://orcid.org/0000-0001-5869-5683
use of virtual reality to work with bone fragments, im-
proves the visualization, the perception of the fracture
zone, to achieve a better reduction, and the interaction
of the user with the fragments is more realistic. These
features allow the use of the tool to perform precise
fracture reductions and their subsequent comparison
with reduced fractures using fracture reduction algo-
rithms allowing validation of the results. In addition,
it could be extended as a training tool for fracture re-
duction, although the focus of the tool should be the
inverse, with the help of experts, datasets of fractures
can be manually reduced to provide a reference to val-
idate the results of algorithm.
The summary of this article is as follows. In the
next section we analyze the previous work on reduc-
tion of bone fractures, analyzing the methods used for
the validation of the obtained models. It is comple-
mented with information on other fracture reduction
simulation environments. After the background the
design of the tool for the validation of fracture re-
duction processes is presented. The following section
presents and discusses the results obtained in the pro-
cess of using the simulator for validation. Finally, the
conclusions summarize the objectives of this tool and
propose the work to be developed in the future.
Jiménez-Delgado, J., Calzado-Martínez, A., Pérez-Cano, F. and Luque-Luque, A.
Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes.
DOI: 10.5220/0009175003990405
In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 1: GRAPP, pages
399-405
ISBN: 978-989-758-402-2; ISSN: 2184-4321
Copyright
c
2022 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
399
2 BACKGROUND
At the moment, data banks of bone fractures and
healthy bones are still inaccessible and a challenge
for the future. P
´
erez and Jim
´
enez (P
´
erez-Cano and
Jim
´
enez-Delgado, 2019a) have developed a tool that
allows the fracturing of bones and the obtaining of the
different fragments, figure 1. The problem with this
approach is that it is necessary the use of bone mod-
els and the results are based on the use of 2D frac-
ture patterns so that information about the fracture is
lost and is not entirely realistic. Therefore, the easi-
est, and most accurate method of obtaining 3D bone
models, is the segmentation of medical images of real
fracture cases. There are other approaches to obtain
geometric models of bones such as 3D scanning that
yield similar results, but the main problem they have
is time and loss of accuracy as identified by P
´
erez and
Jim
´
enez (P
´
erez-Cano and Jim
´
enez-Delgado, 2019b)
in evaluating the different approaches to obtaining
a geometric bone model. Paulano (Paulano et al.,
2014) concluded that traditional segmentation meth-
ods work correctly with healthy bones, but that in
fragmented bones, it was not possible to identify the
different fragments that make up a fracture. Paulano
also proposes a method based on 2D region growing
(Justice et al., 1997; Fan et al., 2005). This method is
based on the establishment of several seeds along the
bone so that the regions grow and allow the identifi-
cation of the different fragments.
Figure 1: Extraction of the fracture pattern through the frac-
ture of a femur after a process of mechanical experimenta-
tion. Figure a represents the femur at the time the fracture
occurs, figure b indicates the fracture zone with red colour
and figure c the fracture pattern obtained. Extracted from
(P
´
erez-Cano and Jim
´
enez-Delgado, 2019a).
Studies such as the directed by Citak (Citak et al.,
2008), emphasizes the importance of using new tech-
nologies to achieve a more realistic visualization. As
demonstrated in this study, improved visualization
and interaction allows for better planning of fracture
reduction and therefore obtaining more precise and
efficient reductions as a result. The advantages of vir-
tual reality in the field of bone fracturing have been
known for quite some time. Tsai (Tsai et al., 2001)
also conclude that simulations in a virtual environ-
ment allow better planning, mainly by improving the
visualization of the fracture, and obtaining better re-
sults. In terms of interaction in these environments,
Gusai (Gusai et al., 2017) analyzed the interaction of
a natural user in a realistic environment. The controls
of virtual reality systems have 6 degrees of freedom,
as in the case of HTC Vive. The use of these devices
are more intuitive in positioning tasks in immersive
environments compared to the use of other types of
options such as the use of gestural interfaces.
There are currently many works that attempt to
increase the degree of automation of bone fracture
reduction, but there is no standardized way to vali-
date the results obtained. The most direct and evident
method is a subjective visual assessment, but this does
not provide quantifiable results although it serves to
validate results from quantitative methods. One of
the forms used to objectively validate the reduction
of bone fractures is the comparison with the symmet-
rical bone (F
¨
urnstahl et al., 2012; Vlachopoulos et al.,
2018) of the same patient. This is not always possi-
ble since in real clinical cases only the study of the
fracture area is normally done, mainly because of the
cost in money and time. There are occasions in which
it is not possible to obtain the scan of the symmetri-
cal bone, bones that do not present symmetry, medical
anomalies or traumatisms with fracture in both bones.
When studies are performed on donated human bones
or on animal bones, which are mechanically frag-
mented to simulate different types of lesions, there
is the alternative of scanning the bone before and af-
ter the experiment, which permits comparison of the
reduced fragments with the intact model (Liu et al.,
2019). In the research by Paulano (Paulano-Godino
and Jim
´
enez-Delgado, 2017) a set of tools developed
by the same team (Paulano et al., 2012) is used for
the manual reduction carried out by experts, then the
measurements are extracted and compared with the
reduction obtained by its algorithm to extract the dif-
ference in translation and rotation between manual
and automatic reduction.
Lately a multitude of Virtual Reality, Augmented
or Mixed devices have appeared, which have been
tested in hospital environments and facilitate the re-
duction of a fracture. This work aims to address
the problem of validating the reduction of bone frac-
tures based on the work of Paulano (Paulano-Godino
and Jim
´
enez-Delgado, 2017), taking advantage of ad-
vances in computer person interaction and visualiza-
tion using Virtual Reality techniques.
GRAPP 2020 - 15th International Conference on Computer Graphics Theory and Applications
400
3 VIRTUAL ENVIRONMENT
The proposed virtual reality environment has been
implemented using the Unity graphics engine, as seen
in figure 2. This tool recreate a virtual world familiar
to future users of the system, an operating room, with
the aim of improving immersion and focusing users
in a surgical environment for fracture reduction. The
hardware incorporated in the HTC Vive bundle pro-
vides full immersion, the kit is formed by a VR helmet
with multiple sensors to determine its spatial position,
proximity sensors and gyroscope. For the control of
the environment, it incorporates controllers that allow
manipulating the fragments in 3D space and provide
feedback through vibration.
The system allows to select a fragment, place it in
the space and proceed to perform the reduction with
the remaining fragment, figure 3. The first time that
a fragment is selected, by clicking the right trigger
while the beam touch it, it jumps in front of the right
controller and stays connected to to motion of the con-
troller, once the user click the right trigger again the
fragment changes to a non selected status and starts
to follow the left controller. Non selected fragments
stay attached to the left controller, so the user can
move and rotate independently the inactive fragments
with the left hand and the selected fragment with the
right hand, this allows users to observe the reduction
process from different angles to improve results. The
fragments on the left hand can be selected as many
times as necessary, the second and subsequent times
when the fragments are selected they do not move to
the right controller, they only get attached to it so the
user can make small corrections.
Throughout the process the user receives contin-
uous feedback through vibration and visualization of
specific elements located on the fracture zone when
the fragments collide, to perceive the interaction be-
tween fragments like a real collision. These compo-
nents are called ”colliders” in the Unity system, in
figure 4 shows the active colliders as small yellow
spheres.
Once a satisfactory result is reached the reduction
can be finished by clicking the left trigger, figure 5
presents a reduced fracture with Windows Mixed Re-
ality controllers, the use of Unity allows the applica-
tion to be multiplatform and compatible with a wide
range of RV helmets. Finally, the relative positions
of both fragments, translation and rotation, are stored
and used as ground truth to compare the results from
other automatic or semi-automatic fracture reduction
systems.
4 RESULTS AND DISCUSSION
As has been seen in previous sections, virtual reality
provides better visualization and greater control over
daily actions. In the field of traumatology, an immer-
sive environment allows the users to have better con-
ditions when conducting studies. In this section, the
advantages of the tool for reducing bone fractures are
analysed, comparing the results achieved with the ob-
tained through studies of a similar scope. A research
has also been carried out about the experience of the
use of the tool by experts in fracture reduction pro-
cesses.
4.1 Automatic versus Manual
Reduction
A software has been developed that allows the valida-
tion of automatic algorithms for the reduction of bone
fractures using the bone reductions conducted by ex-
perts in a virtual environment as the ground truth.
Thus, the error is then calculated as the absolute value
of the difference between the result of the automatic
and manual reduction in translation and rotation, the
closer each value is to zero, the more accurate it is.
The first parameter is the distance error between
the two fragments measured from the center of mass
of each model. The second one is the rotation error
that was calculated in two ways, as the average differ-
ence around the fragments three 2nd moment vectors,
proposed by Paulano (Paulano-Godino and Jim
´
enez-
Delgado, 2017), and as α and β errors, used in the
work of F
¨
urnstahl (F
¨
urnstahl et al., 2012). α is the
difference around the two largest 2nd moment vec-
tors and β represents the rotational difference around
the smallest 2nd moment vectors.
In experiments, some cases has been tested and
compared with the results obtained by the algorithm
of Paulano (Paulano-Godino and Jim
´
enez-Delgado,
2017). In complex cases, when the fracture consist of
more than two fragments the reduction of the fracture
is applied in pairs, that means, first two fragments are
reduced, one to each other, and then the obtained frag-
ment is reduced with the remaining fragment. Figure
6 illustrates the automatic reduction of the fibula frac-
ture, the same fracture used in section 3 to demon-
strate the functioning of the virtual environment. The
fracture of the tibia is composed by three fragments,
the second reduction was performed using the previ-
ous reduction of fragment 1 and 3 and fragment 2.
At final stages of development, the system was
used to tune the parameters of a automatic frac-
ture reduction process based on a modification of
the ICP algorithm, which is being currently devel-
Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes
401
Figure 2: Scene designed in Unity of the virtual environment.
Figure 3: Bone fragment selection system, the beam emit-
ted from the controller improves accurate picking.
oped. The table 1 shows the results of the new al-
gorithm, which are very promising, improving those
obtained in the works of Paulano (Paulano-Godino
and Jim
´
enez-Delgado, 2017) and F
¨
urnstahl (F
¨
urnstahl
et al., 2012), table 2.The average translational error
has been reduced from 1.80 and 1.11, Paulano and
F
¨
urnstahl results respectively, to 0.58, which repre-
sents an average increase in accuracy of 100%. The
rotation error in the work of Paulano is 3.25, F
¨
urnstahl
Figure 4: The colliders are shown as small yellow spheres
that generate haptic feedback in the controllers.
obtains an error value in alpha and beta of 3.1 and
3.51, the algorithm in development has achieved val-
ues of 2.41, 3.30 and 0.92 in such parameters, only
the alpha error is slightly worse but the beta value has
been really improved.
The error of the fracture reduction algorithm could
be minimized thanks to this framework, once reached
a certain level of precision in the reduction of frac-
tures is difficult to differentiate visually if one result
is better than another. This is why it is important to
create an environment and a standardized process in
which the results in fractures of any bone can be com-
pared and improved.
One of the most important advantages of virtual
reality is the simplicity of use, replacing the interac-
tion with keyboard and mouse by a more immersive
GRAPP 2020 - 15th International Conference on Computer Graphics Theory and Applications
402
Table 1: Results of an improved experimental reduction algorithm achieved with the aid of the developed system.
Fracture Fragments Translation error (mm)
Rotation error (mm)
Average Alpha Beta
Humerus 1 2 1,3202 1,3902 1,8582 0,6494
Fibula 1 2 0,2945 3,2845 4,5854 1,1135
Tibia 1 3 0,3586 3,9431 5,3503 1,6489
Tibia 1 3 2 0,3283 1,0213 1,4163 0,2622
Figure 5: Successful reduction of a fibula fracture with a
Windows Mixed Reality device.
Figure 6: Improved automatic reduction of a fibula fracture
thanks to the use of the developed framework.
experience with elements that track in real time the
head and hands of the user, which achieves a com-
pletely natural interface to the 3D world. For this
reason, the learning time is practically non existent
compared to the use of a 2D environment. This soft-
ware has been tested by three different user profiles,
all users have been able to successfully reduce several
bone fractures without the need to spend a great deal
of time reducing a bone fracture.
The equipment needed to run the virtual reality
framework is noticeably more powerful, we have used
Table 2: Average results of different algorithms when eval-
uating errors.
Study Traslation Rotation Alpha Beta
Furnstahl 1,11 3,10 3,51
Paulano 1,80 3,25
Original 0,58 2,41 3,30 0,92
a PC equipped with a first generation i7 microproces-
sor with 8GB of RAM, an NVidia 1060 graphics and
the virtual reality glasses, HTC Vive.
4.2 Usability Tests
The system has passed through a validation process in
which a sample of specialists has been selected. This
user group formed by two experts with experience in
fracture reduction, two users with experience in the
use of fracture reduction applications and a radiolo-
gist expert in the analysis of traumatology images. In
addition to the difficulty of finding real cases of bone
fractures due to confidentiality reasons, the collabo-
ration with the medical community has proved more
costly than expected, mainly due to the lack of time
of the professionals and the rejection that such recent
technologies produce.
Table 3: Results of the user experience survey.
Feature Rating
Image quality 4.83±0.37
Immersive sensation 4.83±0.37
Realism 3.83±0.69
Graphics fluidity 4.50±0.50
Intuitive control 3.66±0.94
Learning curve 4.33±0.47
To conduct the usability study, the virtual reality
device was first shown to users, equipped with it, and
asked to observe and interact with the environment
without explanation of the application workflow. In
order to evaluate the first impression of the system, a
questionnaire had been designed with general ques-
tions about the quality of the images, immersive sen-
sation, realism, fluidity of the graphics, intuitive con-
trol and learning curve. The questionnaire has been
Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes
403
Table 4: Results of the questionnaire about the reduction process.
Fracture
Ease of selection
and movement
of fragments
Accuracy and feedback
of the collisions
Ease of the reduction
process
Accuracy
Fibula 4.66±0.62 3.83±0.99 4.16±0.69 4.16±0.69
Femur 1 4.75±0.43 3.83±0.80 3.66±0.75 4.00±0.58
Femur 2 4.66±0.47 3.75±0.43 3.33±1.11 3.50±0.76
Femur 3 4.83±0.37 3.58±0.49 3.16±0.69 3.33±0.47
Table 5: Score of the reductions granted by the experts.
Fracture Accuracy
Fibula 4.50±0.60
Femur 1 4.66±0.82
Femur 2 4.83±0.47
Femur 3 3.83±0.80
designed using five level Likert responses.
Table 3, shows that the experience of use is re-
ally satisfactory, only two values, ”realism” and ”in-
tuitive control” obtain a score slightly lower than 4
points. According to users, the low level of realism
is mainly due to the resolution of the models, which
have been extracted from CT images without apply-
ing any smoothing filter that could reduce accuracy,
and to the scale applied to easily handle the smaller
fragments. The score in this aspect could be improved
in the future by applying an algorithm to smooth the
model obtained after segmentation, but preserve as
much detail as possible.
The lowest value was ”intuitive control”, which is
to be expected considering that the use of the software
has not been previously indicated. Subsequently, brief
instructions for the reduction procedure have been
provided to each user, after which they have been
asked to perform four reductions of bone fractures
with two fragments each.
After completing the reductions, they were asked
to answer a questionnaire about the ease of selection
and movement of the fragments, the accuracy and
feedback of the collisions, the ease of the reduction
process, and finally each user was asked to rate the
accuracy of the result obtained (table 4). The values
obtained in relation to ease of movement and selec-
tion validate the user interface implemented using the
VR controllers, while the score associated with feed-
back indicates a need to adjust the vibration response
curve.
The ease of reduction and accuracy are markers
that are very interrelated, if users have a perception
that they do not get an optimal result they will nega-
tively evaluate both parameters. Considering the con-
clusions obtained in section 4.3, it could be assumed
that both aspects of the process have been subjectively
evaluated by the sensation of low performance of the
users.
4.3 Validation by Experts
In section 4.2, users have been asked about the ex-
perience using the system. As a result, it has been
obtained low values in the perception of accuracy in
the reduction. To confirm whether the value obtained
is objective or users rated their own result negatively,
users repeated each reduction three times and then the
result of their peers was presented to each expert to be
scored, to increase reliability the whole process was
conducted hiding the author of each reduction.
In table 5, it can be observed that the evaluation of
the reduction of fractures by the rest of the experts is
greater. The fragments of the fracture femur 3 show
loss of osseous material and some degree of defor-
mation, which explains the lower level of precision
appreciated.
5 CONCLUSIONS
This article proposes a system for reducing fractures
using a virtual environment that provides fast and re-
alistic results for the validation of bone fractures, as
an alternative to rapid prototyping reduction of bone
fragments and subsequent capture of fragment posi-
tions.
The main objective that has been sought and
achieved is the validation of automatic algorithms for
the reduction of fractures. For this purpose, the re-
sults generated by expert users have been compared
with the results obtained from fracture reduction al-
gorithms using different metrics defined in the results
section.These results have demonstrated that the pro-
posed process allows to compare and evaluate pre-
cisely the reductions of fractures improving those
present in the literature in different aspects.
Moreover, during the creation of the system sev-
eral tests have been carried out with different types
of users in order to study in depth how the immer-
sion of users in a virtual environment facilitated the
GRAPP 2020 - 15th International Conference on Computer Graphics Theory and Applications
404
reduction of fractures.These studies have proved that
the improvement in visualization and interaction have
been essential to the improvement of the results ob-
tained in the reduction of fractures.
In view of the ease of use and the natural way in
which bone fragments are manipulated, a future mod-
ification has been studied for use as training software
for traumatologists. Although this work has been used
exclusively for medical use, it could be extended to
other fields such as forensic medicine or archaeology.
ACKNOWLEDGEMENTS
This work has been supported by the Ministerio
de Econom
´
ıa y Competitividad and the European
Union (via ERDF funds) through the research project
DPI2015-65123-R.
REFERENCES
Citak, M., Gardner, M. J., Kendoff, D., Tarte, S., Krettek,
C., Nolte, L.-P., and H
¨
ufner, T. (2008). Virtual 3d
planning of acetabular fracture reduction. Journal of
Orthopaedic Research, 26(4):547–552.
Fan, J., Zeng, G., Body, M., and Hacid, M.-S. (2005).
Seeded region growing: an extensive and comparative
study. Pattern Recognition Letters, 26(8):1139–1156.
F
¨
urnstahl, P., Sz
´
ekely, G., Gerber, C., Hodler, J., Snedeker,
J. G., and Harders, M. (2012). Computer assisted re-
construction of complex proximal humerus fractures
for preoperative planning. Medical Image Analysis,
16(3):704–720.
Gusai, E., Bassano, C., Solari, F., and Chessa, M. (2017).
Interaction in an immersive collaborative virtual real-
ity environment: A comparison between leap motion
and HTC controllers. In New Trends in Image Anal-
ysis and Processing ICIAP 2017, pages 290–300.
Springer International Publishing.
Justice, R. K., Stokely, E. M., Strobel, J. S., Ideker, R. E.,
and Smith, W. M. (1997). Medical image segmen-
tation using 3d seeded region growing. In Hanson,
K. M., editor, Medical Imaging 1997: Image Process-
ing. SPIE.
Liu, B., Zhang, S., Zhang, J., Xu, Z., Chen, Y., Liu, S.,
Qi, W., and Yang, L. (2019). A personalized pre-
operative modeling system for internal fixation plates
in long bone fracture surgery—a straightforward way
from CT images to plate model. The International
Journal of Medical Robotics and Computer Assisted
Surgery, 15(5).
Paulano, F., Jim
´
enez, J. J., and Pulido, R. (2014). 3d seg-
mentation and labeling of fractured bone from CT im-
ages. The Visual Computer, 30(6-8):939–948.
Paulano, F., Pulido, R., and Jim
´
enez, J. J. (2012). An Appli-
cation to Interact with 3D Models Reconstructed from
Medical Images. In Navazo, I. and Patow, G., editors,
Spanish Computer Graphics Conference. The Euro-
graphics Association.
Paulano-Godino, F. and Jim
´
enez-Delgado, J. J. (2017).
Identification of fracture zones and its application in
automatic bone fracture reduction. Computer Meth-
ods and Programs in Biomedicine, 141:93–104.
P
´
erez-Cano, F. D. and Jim
´
enez-Delgado, J. J. (2019a).
Macroescale fracturation of osseous models. In Vip-
IMAGE 2019, pages 492–500. Springer International
Publishing.
P
´
erez-Cano, F. D. and Jim
´
enez-Delgado, J. J. (2019b). To-
wards the modelling of osseous tissue. In Proceedings
of the 14th International Joint Conference on Com-
puter Vision, Imaging and Computer Graphics Theory
and Applications. SCITEPRESS - Science and Tech-
nology Publications.
Tsai, M.-D., Hsieh, M.-S., and Jou, S.-B. (2001). Virtual
reality orthopedic surgery simulator. Computers in Bi-
ology and Medicine, 31(5):333–351.
Vlachopoulos, L., Sz
´
ekely, G., Gerber, C., and F
¨
urnstahl, P.
(2018). A scale-space curvature matching algorithm
for the reconstruction of complex proximal humeral
fractures. Medical Image Analysis, 43:142–156.
Virtual Reality Environment for the Validation of Bone Fracture Reduction Processes
405