Measuring the Latency of an Augmented Reality System for Robot-assisted Minimally Invasive Surgery

Martin Kibsgaard, Martin Kraus

Abstract

Minimal latency is important for augmented reality systems and teleoperation interfaces as even small increases in latency can affect user performance. Previously, we have developed an augmented reality system that can overlay stereoscopic video streams with computer graphics in order to improve visual communication in training for robot-assisted minimally invasive surgery with da Vinci surgical systems. To make sure that our augmented reality system provides the best possible user experience, we investigated the video latency of the da Vinci surgical system and how the components of our system affect the overall latency. To measure the photon-to-photon latency, we used a microcontroller to determine the time between the activation of a light-emitting diode in front of the endoscopic camera and the corresponding increase in intensity of the surgeon’s display as measured by a phototransistor. The latency of the da Vinci S surgical system was on average 62 ms. None of the components of our overlay system (separately or combined) significantly affected the latency. However, the latency of the assistant’s monitor increased by 14 ms. Passing the video streams through CPU or GPU memory increased the latency to 147 ms and 256 ms, respectively.

References

  1. Ali, M. R., Loggins, J. P., Fuller, W. D., Miller, B. E., Hasser, C. J., Yellowlees, P., Vidovszky, T. J., Rasmussen, J. J., and Pierce, J. (2008). 3-d telestration: a teaching tool for robotic surgery. Journal of Laparoendoscopic & Advanced Surgical Techniques. Vol. 18, Issue 1, pages 107-112.
  2. Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., and MacIntyre, B. (2001). Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6):34-47.
  3. Bachhuber, C. and Steinbach, E. (2015). A System for Precise End-to-End Delay Measurements in Video Communication. IEEE International Conference on Image Processing (ICIP 2016).
  4. Blissing, B. and Bruzelius, F. (2015). A technical platform using augmented reality for active safety testing. Road Safety & Simulation International Conference Proceedings, (October 2015):793-803.
  5. Ellis, S., Breant, F., Manges, B., Jacoby, R., and Adelstein, B. (1997). Factors influencing operator interaction with virtual objects viewed via head-mounted see-through displays: viewing conditions and rendering latency. Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality, pages 138- 145.
  6. Figl, M., Rueckert, D., Hawkes, D., Casula, R., Hu, M., Pedro, O., Zhang, D. P., Penney, G., Bello, F., and Edwards, P. (2010). Image guidance for robotic minimally invasive coronary artery bypass. Computerized Medical Imaging and Graphics: The Official Journal of the Computerized Medical Imaging Society, 34(1):61-68.
  7. Hattori, A., Suzuki, N., Hashizume, M., Akahoshi, T., Konishi, K., Yamaguchi, S., Shimada, M., and Hayashibe, M. (2003). A robotic surgery system (da vinci) with image guided function-system architecture and cholecystectomy application. Studies in Health Technology and Informatics, 94:110-116.
  8. Jacobs, M. C., Livingston, M. A., and State, A. (1997). Managing latency in complex augmented reality systems. In Proceedings of the 1997 symposium on Interactive 3D graphics - SI3D 7897, pages 49-ff., New York, New York, USA. ACM Press.
  9. Jarc, A. M., Shah, S. H., Adebar, T., Hwang, E., Aron, M., Gill, I. S., and Hung, A. J. (2016). Beyond 2D telestration: an evaluation of novel proctoring tools for robot-assisted minimally invasive surgery. Journal of Robotic Surgery, 10(2):103-109.
  10. Jefferson, M. (2015). Blackmagic Forum View topic - Internal keyer and channels on Decklink Quad 2.
  11. Kibsgaard, M. and Kraus, M. (1999). Real-time augmented reality for robotic-assisted surgery. In The 3rd AAU Workshop on Robotics: Proceedings, pages 19-23. Aalborg Universitetsforlag.
  12. Kibsgaard, M. and Kraus, M. (2016). Pointing with a oneeyed cursor for supervised training in minimally invasive robotic surgery. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9515, pages 12-21. Springer, Cham.
  13. Matu, F. O., Thøgersen, M., Galsgaard, B., Jensen, M. M., and Kraus, M. (2014). Stereoscopic augmented reality system for supervised training on minimal invasive surgery robots. In Proceedings of the 2014 Virtual Reality International Conference on - VRIC 7814, pages 1-4, New York, New York, USA. ACM Press.
  14. Sielhorst, T., Sa, W., Khamene, A., Sauer, F., and Navab, N. (2007). Measurement of absolute latency for video see through augmented reality. In 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR, pages 1-4. IEEE.
  15. Su, L. M., Vagvolgyi, B. P., Agarwal, R., Reiley, C. E., Taylor, R. H., and Hager, G. D. (2009). Augmented Reality During Robot-assisted Laparoscopic Partial Nephrectomy: Toward Real-Time 3D-CT to Stereoscopic Video Registration. Urology, 73(4):896-900.
  16. Ware, C. and Balakrishnan, R. (1994). Reaching for objects in VR displays: lag and frame rate. ACM Transactions on Computer-Human Interaction, 1(4):331-356.
Download


Paper Citation


in Harvard Style

Kibsgaard M. and Kraus M. (2017). Measuring the Latency of an Augmented Reality System for Robot-assisted Minimally Invasive Surgery . In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: GRAPP, (VISIGRAPP 2017) ISBN 978-989-758-224-0, pages 321-326. DOI: 10.5220/0006274203210326


in Bibtex Style

@conference{grapp17,
author={Martin Kibsgaard and Martin Kraus},
title={Measuring the Latency of an Augmented Reality System for Robot-assisted Minimally Invasive Surgery},
booktitle={Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: GRAPP, (VISIGRAPP 2017)},
year={2017},
pages={321-326},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006274203210326},
isbn={978-989-758-224-0},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 1: GRAPP, (VISIGRAPP 2017)
TI - Measuring the Latency of an Augmented Reality System for Robot-assisted Minimally Invasive Surgery
SN - 978-989-758-224-0
AU - Kibsgaard M.
AU - Kraus M.
PY - 2017
SP - 321
EP - 326
DO - 10.5220/0006274203210326