Authors:
Alessandro Farasin
1
;
2
;
Francesco Peciarolo
2
;
Marco Grangetto
3
;
Elena Gianaria
3
and
Paolo Garza
1
Affiliations:
1
Department of Control and Computer Engineering, Politecnico di Torino, Corso Duca degli Abruzzi 24, 10129, Turin, Italy
;
2
LINKS Foundation, Via Pier Carlo Boggio 61, 10138, Turin, Italy
;
3
Computer Science Department, University of Torino, Via Pessinetto 12, 10149, Turin, Italy
Keyword(s):
Computer Vision, Mixed Reality, Microsoft Hololens, Object Detection, Object Tracking, Deep Learning, Spatial Understanding.
Abstract:
This paper presents a mixed reality system that, using the sensors mounted on the Microsoft Hololens headset and a cloud service, acquires and processes in real-time data to detect and track different kinds of objects and finally superimposes geographically coherent holographic texts on the detected objects. Such a goal has been achieved dealing with the intrinsic headset hardware limitations, by performing part of the overall computation in a edge/cloud environment. In particular, the heavier object detection algorithms, based on Deep Neural Networks (DNNs), are executed in the cloud. At the same time we compensate for cloud transmission and computation latencies by running light scene detection and object tracking on board the headset. The proposed pipeline allows meeting the real-time constraint by exploiting at the same time the power of state of art DNNs and the potential of Microsoft Hololens. This paper presents the design choices and describes the original algorithmic steps w
e devised to achieve real time tracking in mixed reality. Finally, the proposed system is experimentally validated.
(More)