Real-time Integral Photography Holographic Pyramid
using a Game Engine
Shohei Anraku, Toshiaki Yamanouchi and Kazuhisa Yanaka
Kanagawa Institute of Technology, 1030 Shimo-ogino, Atsugi-shi, Kanagawa-ken, Japan
Keywords: 3D Display, Autostereoscopic, Integral Photography, Holographic Pyramid, Game Engine, Unity.
Abstract: A new holographic pyramid system that can display an animation of integral photography images that appear
to be floating is developed using a game engine and by writing its shader. An animation of the object, as
viewed from the front, rear, left, and right, are displayed on the four surfaces of the pyramid. All animations
are autostereoscopic and are provided in horizontal and vertical parallaxes. The user can rotate the object left
or right by operating the keyboard. This system can be regarded as an autostereoscopic mixed-reality system
because real and virtual objects can coexist in one pyramid.
1 INTRODUCTION
A holographic pyramid is a display that reflects the
image displayed on a flat panel display, such as a
liquid crystal display (LCD) on the surface of a
quadrangular pyramid made of a half mirror, so that
the image appears as an object floating in the
pyramid. Normally, the flat panel display is 2D, thus
the image displayed is a 2D image without depth, as
shown in Fig. 1. However, by displaying 3D images
in the flat panel, objects with depth can be manifested.
Among various 3D display systems, integral
photography (IP), which was proposed by Lippmann
(Lippmann 1908), is highly suitable for this purpose
because horizontal and vertical parallaxes can be
obtained without using stereo glasses. IP can be
constructed simply by placing a fly’s eye lens on top
of the LCD, as shown in Fig. 2. Yamanouchi et al.
(Yamanouchi et al. 2016) developed a holographic
pyramid system using IP in which a fly’s eye lens is
integrated between the flat panel display and the
pyramid, as shown in Fig. 3. Moreover, 3D CG
animation of IP images can be played with the
system, but images had to be pre-rendered because of
the heavy processing. Another issue of the system is
that only the same image can be displayed on the four
surfaces of the pyramid. It would be better if the
image of the object viewed from each direction is
displayed on the side of the pyramid.
Figure 1: Conventional holographic pyramid system.
Figure 2: Simple integral photography system.
2 METHOD
2.1 Use of a Game Engine and Its
Shader
We introduced a game engine (Unity 5) to solve these
problems.
Anraku, S., Yamanouchi, T. and Yanaka, K.
Real-time Integral Photography Holographic Pyramid using a Game Engine.
DOI: 10.5220/0006647106030607
In Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2018) - Volume 4: VISAPP, pages
603-607
ISBN: 978-989-758-290-5
Copyright © 2018 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
603
Figure 3: Holographic pyramid system using IP.
Game engines can reduce the time and labor of
creating games, as well as most multimedia content.
Moreover, IP multi-viewpoint rendering and IP image
synthesis can considerably be accelerated by using a
shader in general (Yanaka and Kimura 2013); and
Anraku et al. reported that the shader of Unity is no
exception. (Anraku et al. 2017).
Fig. 4 shows the IP image synthesis method using
Unity. In the lower right of the figure, four camera
arrays surround the subject. Each camera array
consists of 8 ×8 = 64 virtual cameras placed in the
front, back, left, and right of a subject that is to be
displayed in the pyramid. A memory area called a
render texture of 4096 × 4096 pixels is allocated to
each camera array. Each render texture is divided into
8 × 8 = 64 viewports (512 × 512 pixels each) and
images seen from different cameras are drawn.
As shown in the middle left of Fig. 1, four planes
(front, left, right, and back) are present, and each
render texture is texture-mapped to the corresponding
plane.
Figure 4: IP image synthesis method using Unity.
©UTJ/UCL.
A surface shader, which synthesizes the IP image
from the multi-view image stored in the render
texture, is assigned to each plane; thus, the IP image
is produced on the plane surface. The synthesis
algorithm is basically the same as written in previous
studies (Yanaka 2008).
Four IP images obtained in this method are taken
at once by Unity’s main camera, shown in the upper
left of Fig. 4. Fig. 5 shows an example of the image
taken by the main camera and displayed on the LCD
of the PC.
Figure 5: Image taken by the main camera and displayed on
the LCD of the PC. ©UTJ/UCL.
2.2 Rotation and Flipping of the Image
Please note that the character's feet must point to the
center of the pyramid so that the orientation of the
character is correct when the image is reflected on one
of the surfaces of the pyramid. For that purpose, the
images on the left and right planes must be rotated by
90 degrees in opposite directions, and the image of
the back plane must be rotated by 180 degrees, as
shown in Fig. 5. IP images cannot be rotated,
therefore, this rotation must be done before
synthesizing the IP images.
Rotating the subject towards one direction and
rotating the entire camera array in the opposite
direction are equivalent. Thus, we rotated all the
camera arrays, as shown in Figure 6.
Moreover, considering that the image displayed
on the LCD is reversed left and right when it is
reflected by the surface of the pyramid, the image is
horizontally reversed in advance when drawing on the
view port.
VISAPP 2018 - International Conference on Computer Vision Theory and Applications
604
Figure 6: Rotation of each camera arrays. ©UTJ/UCL.
2.3 Character Rotation by Keyboard
Operations
In this holographic pyramid, autostereoscopic images
of a character viewed from four different directions
are displayed on four surfaces of the pyramid.
Therefore, the image of the character viewed from
that direction can be seen if the user moves to another
surface. However, if the observer can rotate the
character in the pyramid, the user does not have to
walk around the pyramid to see the character from
various angles. Therefore, we added an interactive
function to rotate the character by the following
keyboard operations.
As shown in Fig. 7, the character rotates counter-
clockwise when the user presses “4” on the numeric
keyboard and rotates clockwise when “6” is pressed.
3 EXPERIMENTS
Fig. 8 shows an outlook of the system. Here we used
LowPolyUnityChan (Unity Technologies Japan
2016), which is distributed by Unity Technologies
Japan. The personal computer used in the experiment
is a Microsoft Surface Book equipped with 13.5-inch
high definition LCD of 3000×2000 pixels. The fly’s
eye lens used is by Fresnel Technology and is a 360,
a hexagonal convex lens array whose lens pitch is 1
mm. Despite the number of renderings being
increased by 256 times compared with the normal
CG, the frame rate was about 4 fps. It will be
improved by using a high performance GPU.
Figure 7: Character rotation by keyboard operations.
©UTJ/UCL.
Figure 8: Appearance of new holographic pyramid.
©UTJ/UCL.
Figs. 9, 10, 11, and 12 show the front, back, left-side,
and right-side views of the character displayed on the
surface of the pyramid, as seen from a user by rotating
the character by the user's own keyboard operation.
Figure 9: Character seen from the front. ©UTJ/UCL.
Real-time Integral Photography Holographic Pyramid using a Game Engine
605
Figure 10: Character seen from the back. ©UTJ/UCL.
Figure 11: Characters seen from the left side. ©UTJ/UCL.
Figure 12: Character seen from the right side. ©UTJ/UCL.
Naturally, the images of the remaining three faces
also change in conjunction with it. In either case, an
autostereoscopic effect of the IP method is obtained.
IP provides parallax in the horizontal and vertical
directions. Therefore, when the head is moved
vertically, the image changes accordingly.
For comparison, Fig. 13 shows an image of a
conventional holographic pyramid system in which
IP is not used. Even in this case, the character looks
like being inside the pyramid. However, the depth of
the object is not perceived because the surface of the
pyramid reflects the 2D image on the LCD. The
image seen around the center of the pyramid is not
3D, but 2D, as if in a billboard.
On the other hand, the holographic pyramid using
IP reflects the light from 3D image produced by the
IP system consisting of a LCD and a fly's eye lens.
Therefore, the viewer feels that the 3D character
really exists inside the pyramid.
Figure 13: Normal H\holographic pyramid without IP.
©UTJ/UCL.
4 CONCLUSIONS
In summary, we developed a new holographic
pyramid system that can display autostereoscopic
objects that appear to be floating and are animated
around the center of the pyramid using a game engine
and developing its shader to synthesize integral
photography images at high speed. On each of the
four surfaces of the pyramid, the scene which should
be seen from the direction is displayed.
In this system, real and virtual objects can coexist in
the pyramid. In this sense, this is a kind of
autostereoscopic mixed-reality system, which can be
applied in various situations, such as sales promotions
inside stores.
VISAPP 2018 - International Conference on Computer Vision Theory and Applications
606
ACKNOWLEDGEMENTS
This work was partially supported by KAKENHI
Grant Number 16K00284.
We appreciate Unity Technologies Japan because
Unity-chan was used in this research.
REFERENCES
Lippmann, M. G. 1908. Epreuves, reversibles donnant la
sensation du relief, J. Phys. Vol.4. 821-825.
Yanaka, K. 2008. Integral photography using hexagonal
fly's eye lens and fractional view. In Proc. SPIE 6803,
Stereoscopic Displays and Applications XIX, 68031K.
1-8.
Yanaka, K. and Kimura, S. 2013. GPU accelerated
interactive integral photography system using extended
fractional view method. In SIGGRAPH 2013 Posters,
one page.
Yamanouchi, T., Maki, N. and Yanaka, K. 2016.
Holographic Pyramid Using Integral Photography. In
Proceedings of EECSS'16, Paper No. MHCI 109, 14.
Anraku, S., Ishiwata, F., Maki, N., Yamanouchi, T. and
Yanaka, K. 2017. Real-Time Integral Photography
Using a Game Engine. In SIGGRAPH 2017 Posters,
two pages.
Unity Technologies. 2017. Unity. https://unity3d.com/jp.
Unity Technologies Japan. 2016. LowPolyUnityChan.
http://unity-chan.com/download/releaseNote.php?id=
LowPolyUnityChan&lang=en.
Real-time Integral Photography Holographic Pyramid using a Game Engine
607