Real-time Integral Photography Holographic Pyramid

using a Game Engine

Shohei Anraku, Toshiaki Yamanouchi and Kazuhisa Yanaka

Kanagawa Institute of Technology, 1030 Shimo-ogino, Atsugi-shi, Kanagawa-ken, Japan

Keywords: 3D Display, Autostereoscopic, Integral Photography, Holographic Pyramid, Game Engine, Unity.

Abstract: A new holographic pyramid system that can display an animation of integral photography images that appear

to be floating is developed using a game engine and by writing its shader. An animation of the object, as

viewed from the front, rear, left, and right, are displayed on the four surfaces of the pyramid. All animations

are autostereoscopic and are provided in horizontal and vertical parallaxes. The user can rotate the object left

or right by operating the keyboard. This system can be regarded as an autostereoscopic mixed-reality system

because real and virtual objects can coexist in one pyramid.

1 INTRODUCTION

A holographic pyramid is a display that reflects the

image displayed on a flat panel display, such as a

liquid crystal display (LCD) on the surface of a

quadrangular pyramid made of a half mirror, so that

the image appears as an object floating in the

pyramid. Normally, the flat panel display is 2D, thus

the image displayed is a 2D image without depth, as

shown in Fig. 1. However, by displaying 3D images

in the flat panel, objects with depth can be manifested.

Among various 3D display systems, integral

photography (IP), which was proposed by Lippmann

(Lippmann 1908), is highly suitable for this purpose

because horizontal and vertical parallaxes can be

obtained without using stereo glasses. IP can be

constructed simply by placing a fly’s eye lens on top

of the LCD, as shown in Fig. 2. Yamanouchi et al.

(Yamanouchi et al. 2016) developed a holographic

pyramid system using IP in which a fly’s eye lens is

integrated between the flat panel display and the

pyramid, as shown in Fig. 3. Moreover, 3D CG

animation of IP images can be played with the

system, but images had to be pre-rendered because of

the heavy processing. Another issue of the system is

that only the same image can be displayed on the four

surfaces of the pyramid. It would be better if the

image of the object viewed from each direction is

displayed on the side of the pyramid.

Figure 1: Conventional holographic pyramid system.

Figure 2: Simple integral photography system.

2 METHOD

2.1 Use of a Game Engine and Its

Shader

We introduced a game engine (Unity 5) to solve these

problems.

Anraku, S., Yamanouchi, T. and Yanaka, K.

Real-time Integral Photography Holographic Pyramid using a Game Engine.

DOI: 10.5220/0006647106030607

In Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2018) - Volume 4: VISAPP, pages

603-607

ISBN: 978-989-758-290-5

603

Figure 3: Holographic pyramid system using IP.

Game engines can reduce the time and labor of

creating games, as well as most multimedia content.

Moreover, IP multi-viewpoint rendering and IP image

synthesis can considerably be accelerated by using a

shader in general (Yanaka and Kimura 2013); and

Anraku et al. reported that the shader of Unity is no

exception. (Anraku et al. 2017).

Fig. 4 shows the IP image synthesis method using

Unity. In the lower right of the figure, four camera

arrays surround the subject. Each camera array

consists of 8 ×8 = 64 virtual cameras placed in the

front, back, left, and right of a subject that is to be

displayed in the pyramid. A memory area called a

render texture of 4096 × 4096 pixels is allocated to

each camera array. Each render texture is divided into

8 × 8 = 64 viewports (512 × 512 pixels each) and

images seen from different cameras are drawn.

As shown in the middle left of Fig. 1, four planes

(front, left, right, and back) are present, and each

render texture is texture-mapped to the corresponding

plane.

Figure 4: IP image synthesis method using Unity.

©UTJ/UCL.

A surface shader, which synthesizes the IP image

from the multi-view image stored in the render

texture, is assigned to each plane; thus, the IP image

is produced on the plane surface. The synthesis

algorithm is basically the same as written in previous

studies (Yanaka 2008).

Four IP images obtained in this method are taken

at once by Unity’s main camera, shown in the upper

left of Fig. 4. Fig. 5 shows an example of the image

taken by the main camera and displayed on the LCD

of the PC.

Figure 5: Image taken by the main camera and displayed on

the LCD of the PC. ©UTJ/UCL.

2.2 Rotation and Flipping of the Image

Please note that the character's feet must point to the

center of the pyramid so that the orientation of the

character is correct when the image is reflected on one

of the surfaces of the pyramid. For that purpose, the

images on the left and right planes must be rotated by

90 degrees in opposite directions, and the image of

the back plane must be rotated by 180 degrees, as

shown in Fig. 5. IP images cannot be rotated,

therefore, this rotation must be done before

synthesizing the IP images.

Rotating the subject towards one direction and

rotating the entire camera array in the opposite

direction are equivalent. Thus, we rotated all the

camera arrays, as shown in Figure 6.

Moreover, considering that the image displayed

on the LCD is reversed left and right when it is

reflected by the surface of the pyramid, the image is

horizontally reversed in advance when drawing on the

view port.

VISAPP 2018 - International Conference on Computer Vision Theory and Applications

604

Figure 6: Rotation of each camera arrays. ©UTJ/UCL.

2.3 Character Rotation by Keyboard

Operations

In this holographic pyramid, autostereoscopic images

of a character viewed from four different directions

are displayed on four surfaces of the pyramid.

Therefore, the image of the character viewed from

that direction can be seen if the user moves to another

surface. However, if the observer can rotate the

character in the pyramid, the user does not have to

walk around the pyramid to see the character from

various angles. Therefore, we added an interactive

function to rotate the character by the following

keyboard operations.

As shown in Fig. 7, the character rotates counter-

clockwise when the user presses “4” on the numeric

keyboard and rotates clockwise when “6” is pressed.

3 EXPERIMENTS

Fig. 8 shows an outlook of the system. Here we used

LowPolyUnityChan (Unity Technologies Japan

2016), which is distributed by Unity Technologies

Japan. The personal computer used in the experiment

is a Microsoft Surface Book equipped with 13.5-inch

high definition LCD of 3000×2000 pixels. The fly’s

eye lens used is by Fresnel Technology and is a 360,

a hexagonal convex lens array whose lens pitch is 1

mm. Despite the number of renderings being

increased by 256 times compared with the normal

CG, the frame rate was about 4 fps. It will be

improved by using a high performance GPU.

Figure 7: Character rotation by keyboard operations.

©UTJ/UCL.

Figure 8: Appearance of new holographic pyramid.

©UTJ/UCL.

Figs. 9, 10, 11, and 12 show the front, back, left-side,

and right-side views of the character displayed on the

surface of the pyramid, as seen from a user by rotating

the character by the user's own keyboard operation.

Figure 9: Character seen from the front. ©UTJ/UCL.

Real-time Integral Photography Holographic Pyramid using a Game Engine

605

Figure 10: Character seen from the back. ©UTJ/UCL.

Naturally, the images of the remaining three faces

also change in conjunction with it. In either case, an

autostereoscopic effect of the IP method is obtained.

IP provides parallax in the horizontal and vertical

directions. Therefore, when the head is moved

vertically, the image changes accordingly.

For comparison, Fig. 13 shows an image of a

conventional holographic pyramid system in which

IP is not used. Even in this case, the character looks

like being inside the pyramid. However, the depth of

the object is not perceived because the surface of the

pyramid reflects the 2D image on the LCD. The

image seen around the center of the pyramid is not

3D, but 2D, as if in a billboard.

On the other hand, the holographic pyramid using

IP reflects the light from 3D image produced by the

IP system consisting of a LCD and a fly's eye lens.

Therefore, the viewer feels that the 3D character

really exists inside the pyramid.

Figure 13: Normal H\holographic pyramid without IP.

©UTJ/UCL.

4 CONCLUSIONS

In summary, we developed a new holographic

pyramid system that can display autostereoscopic

objects that appear to be floating and are animated

around the center of the pyramid using a game engine

and developing its shader to synthesize integral

photography images at high speed. On each of the

four surfaces of the pyramid, the scene which should

be seen from the direction is displayed.

In this system, real and virtual objects can coexist in

the pyramid. In this sense, this is a kind of

autostereoscopic mixed-reality system, which can be

applied in various situations, such as sales promotions

inside stores.

VISAPP 2018 - International Conference on Computer Vision Theory and Applications

606

ACKNOWLEDGEMENTS

This work was partially supported by KAKENHI

Grant Number 16K00284.

We appreciate Unity Technologies Japan because

Unity-chan was used in this research.

REFERENCES

Lippmann, M. G. 1908. Epreuves, reversibles donnant la

sensation du relief, J. Phys. Vol.4. 821-825.

Yanaka, K. 2008. Integral photography using hexagonal

fly's eye lens and fractional view. In Proc. SPIE 6803,

Stereoscopic Displays and Applications XIX, 68031K.

1-8.

Yanaka, K. and Kimura, S. 2013. GPU accelerated

interactive integral photography system using extended

fractional view method. In SIGGRAPH 2013 Posters,

one page.

Yamanouchi, T., Maki, N. and Yanaka, K. 2016.

Holographic Pyramid Using Integral Photography. In

Proceedings of EECSS'16, Paper No. MHCI 109, 1–4.

Anraku, S., Ishiwata, F., Maki, N., Yamanouchi, T. and

Yanaka, K. 2017. Real-Time Integral Photography

Using a Game Engine. In SIGGRAPH 2017 Posters,

two pages.

Unity Technologies. 2017. Unity. https://unity3d.com/jp.

Unity Technologies Japan. 2016. LowPolyUnityChan.

http://unity-chan.com/download/releaseNote.php?id=

LowPolyUnityChan&lang=en.

Real-time Integral Photography Holographic Pyramid using a Game Engine

607