Towards a Diminished Reality System that Preserves Structures and Works in Real-time

Hugo Álvarez, Jon Arrieta, David Oyarzun

Abstract

This paper presents a Diminished Reality system that is able to propagate textures as well as structures with a low computational cost, almost in real-time. An existing inpainting algorithm is optimized in order to reduce the high computational cost by implementing some Computer Vision techniques. Although some of the presented optimizations can be applied to a single static image directly, the global system is mainly oriented to video sequences, where temporal coherence ideas can be applied. Given that, a novel pipeline is proposed to maintain the visual quality of the reconstructed image area without the need of calculating everything again despite slow camera motions. To the best of our knowledge, the prototype presented in this paper is the only Diminished Reality system focused on structure propagation that works near real-time. Apart from the technical description, this paper presents an extensive experimental study of the system, which evaluates the optimizations in terms of time and quality.

References

  1. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. (2009). Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics-TOG, 28(3):24.
  2. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C. (2000). Image inpainting. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417-424. ACM Press/AddisonWesley Publishing Co.
  3. Bouguet, J.-Y. (2001). Pyramidal implementation of the affine lucas kanade feature tracker description of the algorithm. Intel Corporation, 5(1-10):4.
  4. Bradski, G. R. (1998). Real time face and object tracking as a component of a perceptual user interface. In Applications of Computer Vision, 1998. WACV'98. Proceedings., Fourth IEEE Workshop on, pages 214-219. IEEE.
  5. Criminisi, A., Pérez, P., and Toyama, K. (2004). Region filling and object removal by exemplar-based image inpainting. Image Processing, IEEE Transactions on, 13(9):1200-1212.
  6. Fischler, M. A. and Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM, 24(6):381-395.
  7. Google Inc. (2013). Google AR Glasses. http:// www.google.es/glass/start/.
  8. Goyal, P., Diwakar, S., et al. (2010). Fast and enhanced algorithm for exemplar based image inpainting. In Image and Video Technology (PSIVT), 2010 Fourth Pacific-Rim Symposium on , pages 325-330. IEEE.
  9. Herling, J. and Broll, W. (2010). Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments. In Mixed and Augmented Reality (ISMAR), 2010 9th IEEE International Symposium on, pages 207-212. IEEE.
  10. Herling, J. and Broll, W. (2012). Pixmix: A real-time approach to high-quality diminished reality. In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on, pages 141-150. IEEE.
  11. Itseez (2016). Open source computer vision library. https://github.com/itseez/opencv.
  12. Kass, M., Witkin, A., and Terzopoulos, D. (1988). Snakes: Active Contour Models. International Journal of Computer Vision, 1(4):321-331.
  13. Kawai, N., Sato, T., and Yokoya, N. (2013a). Diminished reality considering background structures. In Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, pages 259-260. IEEE.
  14. Kawai, N., Yamasaki, M., Sato, T., and Yokoya, N. (2013b). [paper] diminished reality for ar marker hiding based on image inpainting with reflection of luminance changes. ITE Transactions on Media Technology and Applications, 1(4):343-353.
  15. Komodakis, N. and Tziritas, G. (2007). Image completion using efficient belief propagation via priority scheduling and dynamic pruning. Image Processing, IEEE Transactions on, 16(11):2649-2661.
  16. Lepetit, V., Berger, M.-O., and Lorraine, L.-I. (2001). An intuitive tool for outlining objects in video sequences: Applications to augmented and diminished reality. tC, 2:t3.
  17. Liu, Y. and Caselles, V. (2013). Exemplar-based image inpainting using multiscale graph cuts. Image Processing, IEEE Transactions on, 22(5):1699-1711.
  18. Lucas, B. D., Kanade, T., et al. (1981). An iterative image registration technique with an application to stereo vision. In IJCAI, volume 81, pages 674-679.
  19. Rahul Verma (2013). Inpainting algorithm source. https:// github.com/surahul/Inpaint.
  20. Rosten, E. and Drummond, T. (2005). Fusing points and lines for high performance tracking. In In Tenth IEEE International Conference on Computer Vision, pages 1508-1515.
  21. Simakov, D., Caspi, Y., Shechtman, E., and Irani, M. (2008). Summarizing visual data using bidirectional similarity. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pages 1-8. IEEE.
  22. Wexler, Y., Shechtman, E., and Irani, M. (2007). Space-time completion of video. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 29(3):463-476.
  23. Zokai, S., Esteve, J., Genc, Y., and Navab, N. (2003). Multiview paraperspective projection model for diminished reality. In Mixed and Augmented Reality, 2003. Proceedings. The Second IEEE and ACM International Symposium on, pages 217-226. IEEE.
Download


Paper Citation


in Harvard Style

Álvarez H., Arrieta J. and Oyarzun D. (2017). Towards a Diminished Reality System that Preserves Structures and Works in Real-time . In Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, (VISIGRAPP 2017) ISBN 978-989-758-225-7, pages 334-343. DOI: 10.5220/0006097803340343


in Bibtex Style

@conference{visapp17,
author={Hugo Álvarez and Jon Arrieta and David Oyarzun},
title={Towards a Diminished Reality System that Preserves Structures and Works in Real-time},
booktitle={Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, (VISIGRAPP 2017)},
year={2017},
pages={334-343},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006097803340343},
isbn={978-989-758-225-7},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP, (VISIGRAPP 2017)
TI - Towards a Diminished Reality System that Preserves Structures and Works in Real-time
SN - 978-989-758-225-7
AU - Álvarez H.
AU - Arrieta J.
AU - Oyarzun D.
PY - 2017
SP - 334
EP - 343
DO - 10.5220/0006097803340343