Milán Horváth
, Csaba Dániel
, Jacqueline Stark
and Cecília Sik Lanyi
1, 2, 4
Department of Image Processing and Neurocomputing, University of Pannonia, Veszprém, Hungary
Austrian Academy of Sciences, Department of Linguistics and Communication Research, Vienna, Austria
Keywords: Virtual reality, aphasia, language therapy.
Abstract: In this paper, the rationale for the development and the process of creating the Virtual ELA
(Everyday Life
Activities)-House are described. The Virtual ELA
-House is an innovative therapy program designed for
use with clients with language and speech disorders and/or with other cognitive neuropsychological
disorders, which result from brain damage, e.g. aphasia, apraxia of speech, neglect, etc. The Virtual Reality
setting is chosen as a modern and relevant therapy setting which imitates real everyday life scenarios.
Computer supported cognitive and language therapy allows for repetitive application in the clinical and
home setting which is necessary for learning to take place. The advantages of employing a software
program based on a Virtual Environment, in particular the Virtual ELA
-House, are discussed.
Virtual Reality offers a wide range of applications in
the field of cognitive neuropsychology, both in
diagnosing cognitive deficits and in treating them.
Virtual Reality is also a useful tool for skill-building
and training by creating a virtual setting, which
imitates the real environment including the attributes
to be trained. The virtual world is a computer-based,
simulated environment intended for its users to
inhabit and interact via avatars. This habitation is
usually represented in the form of two-or three-
dimensional graphical representations of humanoids
(or other graphical or text-based avatars) (Stark,
A Virtual ELA®-House was created to provide
an alternative means of language and cognitive
rehabilitation for use with clients suffering from
various disorders of higher cortical functioning
resulting from brain damage (The National Aphasia
Association). The decision to create a Virtual
ELA®-House was based on the impressive language
therapy results from single case studies of aphasia
based on the analogue version of the ELA- Photo
Series (Stark, 1992-1998), (Stark, 2003).
One possible reason for the significant
improvement in language skills following provision
of ELA®-based therapy protocols is that the picture
stimuli are realistic and relevant for everyday life.
Thus, an important goal in this project was to attain
a high degree of correspondence with the analogue
ELA®-picture stimuli, which is also required for an
authentic virtual world. In a virtual world the
designer of a task attempts to capture all aspects
which a particular task requires, but it is very crucial
for it to be true to nature. If the virtual world
established reaches this level, the user will feel that
he/she is part of the Virtual Environment. The term
Virtual Environment can refer to a room, a place, a
house, etc.
During the design process of a Virtual World, the
designer should be aware of the available
possibilities. To design or model a virtual world, a
personal computer is needed with 3D modeling
software and a programming environment.
2.1 Design and Modeling
Before modeling the Virtual World, the entire
project must be designed. In our case the layout for
the ELA®-Virtual House was designed by
Horváth M., Dániel C., Stark J. and Sik Lanyi C. (2008).
In Proceedings of the Third International Conference on Computer Graphics Theory and Applications, pages 519-523
DOI: 10.5220/0001099205190523
Jacqueline Stark. Figure 1 depicts the layout of the
house, which consists of various rooms located on
two floors and a basement:
Figure 1: The floor layout for the ELA®-Virtual House.
After completion of the room layout for ELA®-
Virtual House, modeling the components for the
ELA®-Virtual House was initiated. For this
operation, of the several available programs, we
chose the Autodesk Maya(Autodesk Maya) program
for the ELA®-Virtual House.
2.1.1 Criteria for Selecting the Autodesk
3D modeling is the process of creating three-
dimensional surfaces using a computer for the
purpose of rendering them into a picture or a
sequence of pictures (Harovas et. al, 2000). Maya is
a manifold 3D modeling program, which
encompasses textures, thus allowing for animating
software. This program is very versatile due to its
wide range of applications. The Maya project has a
scene where the house is modeled and built by
means of a wide range of tools. To model a building
- such as the ELA®-Virtual House – the first step
was to create a polygon cube. This cube can be
transformed, rotated or scaled to achieve the desired
position and size. The cube must then be divided for
the next modeling steps. Every polygon object has
faces, vertices and edges. In the case of a cube there
are 8 vertices, 6 faces and 12 edges. It is possible to
transform, rotate or scale these vertices, faces and
edges. The subdividing process generates more
vertices, faces and edges, which allows the building
of more exact and realistic objects.
Figure 2 shows the basic form of the cube after its
creation. In Figure 3 the result of the subdividing
process is shown, namely, that there are more
vertices, faces and edges which are necessary to
model the task.
Figure 2: A simple cube.
Figure 3: A cube with rotation, translation and scale.
In Maya there are eight fundamental polygon
objects: sphere, cube, cylinder, cone, plane, torus,
pyramid and pipe. With Maya it is possible to build
any objects from these polygons with the
aforementioned transformations, rotation and scales.
An object can consist of one polygon object, or
several polygon objects. In the second case, objects
which belong together can be combined by using the
Combine polygons tool.
2.2 Providing Realistic Textures and
In the modeling process, after the object has been
completed, the next step is to give it texture. For this
operation texture mapping was used. Texture
mapping is a method for adding detail, surface
texture, or color to a computer-generated graphic or
3D model (Texture mapping). A virtual house has
many textures, mostly in JPEG format. Maya can
work easily with this format as well as with almost
all of the other picture formats. The ELA
requires the use of precise and realistic images,
therefore the analogue photos from the original
GRAPP 2008 - International Conference on Computer Graphics Theory and Applications
-Photo Series were scanned and additional
digital photos were made of components of the real
rooms. These images provide the basis for the
modeling of the entire house and the individual
rooms. The important parts of the pictures were
copied and used to texture the objects in Maya.
There are several methods for texturing an object,
but the simplest way is to assign a material to an
object. First of all, the type of material is chosen. In
this project, lambert material was used for the wall
material. This is a material (shader) that represents
lusterless or dull, mat surfaces (such as chalk,
sheenless paint, unpolished surfaces) without
mirroring qualities. For metal-like materials, blinn
material is used. It is a material (shader), that is
particularly effective at simulating metallic surfaces
(for example, brass or aluminum), which typically
give soft specular, i.e., mirroring , highlighting. It
must be stressed that Maya supports many types of
materials. With Maya it is even possible to design
one’s own material by adding special colors,
transparency, ambient color or incandescence. Maya
saves the configurations and these can be used for
other models.
In the process of modeling and texturing,
numerous objects were created, including walls,
windows, floors, doors, etc. It is very useful to use
layers in Maya, especially if the project consists of
many individual objects. The project is more
comprehensible, if the individual parts are
categorized. Working with layers makes it easier to
maintain an overview of the whole project. The
procedure is simply to add a new layer to the project,
and select the objects which belong to the same
category, and then add selected objects to the layer.
First, the layer must be named. When the modeling
and texturing part of the layer’s objects has been
completed, it can simply be hidden, if necessary.
Proper layers represent categories. In each
category there are objects which have their own
texture map or color, and which belong together.
The result of modeling and texturing is shown in
Figure 4, where the layers present the individual
parts of the house, such as the roof, the inside walls,
the furniture, etc.
2.3 Converting: An Advantage of
Software Independence
This project was developed in the programming
environment Eclipse (Eclipse), where the
programming language is C++. Use of the engine
Irrlicht (Irrlicht Engine) enables the user to walk
around in the Virtual House, click on several objects
Figure 4: The ELA®-House in Maya.
around in the Virtual House, click on several objects
and respond interactively. Irrlicht is a free open
source 3D engine. However, one problem involved
in using Irrlicht is that it does not support the Maya
binary files. It was a challenge to find a way to
export the model to a file type supported by Irrlicht.
Maya can export its models to object format, but this
format contains only geometry, and textures cannot
be saved in this format. The solution to this problem
was the program Right Hemisphere’s Deep
Exploration (Deep Exploration).
2.3.1 The Advantages of using Deep
Deep Exploration supports almost all 3D or 2D file
types, including Maya’s scenes. When the program
runs, a scene, a file browser, and a little viewer
appears. It is possible to open a saved Maya file in
the scene field of Deep Exploration. Deep
Exploration is not merely a file converter. When the
file is loaded, the user is able to turn or fly around
the object, zoom in or zoom out, but it is not
possible to modify the object. Deep Exploration can
assign textures or colors to the objects, but this
operation and the possibilities are very far from
those which one is able to accomplish with Maya.
Once the scene is loaded, the software is ready to
export the result to all file types that Deep
Exploration supports. A very popular format is the
DirectX file format. The DirectX file format is an
architecture- and context-free file format. It is
template-driven and is free of any user-knowledge or
experience. The file format may be used by any
client application and currently is used by Direct3D
Retained Mode to describe geometry data, frame
hierarchies, and animations (DirectX). If the
exporting process is done correctly, the resulting file
is an .x file consisting of the whole geometry and
inside the file the paths of the texture files are also
included. As of this point the resulting .x file is
ready to be imported into the Irrlicht.
The Irrlicht Engine is an open source, high
performance, real time 3D engine written and usable
in C++ and also available for .NET languages. It is
completely cross-platform, using D3D, OpenGL and
its own software renderer, and it integrates all the
state-of-the-art features for visual representation
such as dynamic shadows, particle systems,
character animation, indoor and outdoor technology,
and collision detection, which can be found in
commercial 3D engines. It is a powerful,
high level
API for creating complete 3D and 2D applications
required in games or scientific visualizations. It
comes with an excellent documentation, which is
very useful in the development of software.
This engine is being used in many projects.
There are enhancements for Irrlicht all over the web,
including alternative terrain renderers, portal
renderers, exporters, world layers, tutorials, editors,
language bindings for java, perl, ruby, basic, python,
lua, and so on.
Irrlicht has its own graphical file manager,
named irrEdit (Ambierra: irrEdit), to load, rebuild
and test our C++ project every time. In irrEdit it is
possible to load many .x files, position, rotate or
scale the object, but it cannot be modified. After the
house model was loaded, some furniture models
were added, so it became possible to furnish the
whole house according to the layout. The irrEdit
loader can save the objects, transformations,
rotations and scales in its own file format, which at
the moment is best for Irrlicht. These saved scenes,
which can be modified for the specific exercises, are
loaded into the program.
The Virtual ELA
-House program can be executed
on Windows following installation. The project
supports two languages for the clients: German and
English. Before beginning the program, the user can
choose between a male and a female voice. After
that the user selects an exercise from the various
types of exercises. The picture stimuli used for all of
the language therapy tasks are taken from the ELA
Photo Series. A constant blue-red figure shows the
user where he/she is looking at the moment, when
he/she is walking in the house.
4.1 Types of Exercises for the Virtual
4.1.1 Discovery Task
In this task, the avatar wanders through the house on
his/her own to learn the vocabulary of the objects
and activities in each room of the house. Two
different symbols are used for showing the active
object. A blue cube is for an object, as shown in
Figure 5 and a yellow sphere is for an activity. When
the client clicks on an object by means of the right
mouse button, a window appears. In the window an
image of an object or an activity and four buttons are
shown. With three of these buttons the user can read
or hear separately, or see and hear the description of
the image simultaneously. With the fourth button
he/she can close the window, and continue
discovering the components of the house.
Figure 5: A virtual table with chairs.
4.1.2 Structured Discovery Task
A variation of the first task is that the avatar is
standing outside the house. The avatar receives
either an auditory command or one in written
language to which it must respond. The following
questions are included in this task:
‘Where does activity ‘x’ usually take
place?’ or, ‘Where is activity ‘x’ usually
performed in the house?’ A sentence is heard or
is written on the screen requesting the avatar to
GRAPP 2008 - International Conference on Computer Graphics Theory and Applications
move to that location in the house, where
he/she thinks the activity could be performed.
For one level of difficulty, when he/she clicks
on the correct room, a certain symbol blinks.
At a more difficult level, the avatar has to be
more specific and click on the exact location,
where the activity takes place. For example, if
it is a picture of a boy brushing his teeth, for the
first level the avatar would move into the
bathroom, To be more specific, the avatar must
click on or near the sink in the bathroom. If the
avatar makes a mistake in the selection of the
room, some response is heard or seen, signaling
to him/her to continue the search. If the avatar
is in the right room and makes a mistake
pertaining to the exact position, then he/she
also hears/sees a response: ‘You are getting
closer, keep trying/looking’.
‘Where does ‘x’ object belong in the
house?’ The avatar clicks on an icon on the
screen and an object appears. He/She should
then look for the location where that object
should be placed in the house. The same types
of responses are used as in the previous
4.1.3 Memory Task
Several items (i.e. objects) are shown on the screen
and the program then puts these pictures in different
places in the house, one at a time. (Range of
difficulty: the number can be set to include 3 to 5
objects). The avatar’s task is to find the hidden items
in the house in the same order as they were shown.
The items can be hidden in different places.
These are a few of the exercises included in the
present version of the Virtual ELA
The software development and the designing and
modeling of a Virtual ELA
-House is a very
complex task. The project has two significant parts,
which can be completely separated from each other.
The first part is the software development, and the
second is the modeling and texturing. During the
modeling, the most difficult task was to produce a
house which gives the users a feeling that the house
exists, and that they are actually walking in the
house. For this task digital photographs were used
from a real house. Difficulties arose during the
texturing part of the project. New images were
created based on the actual photos, with different
resolutions and compilation. In this project, the
programming task was completed after being tested
only once in a simple building. This was necessary
because the software development and the modeling
tasks were performed at the same time. After the
second task was completed, the two tasks were
integrated into the software environment, where the
finished project was built and tested.
In the future, other exercise types will be added
to the program which will be based on feedback
from the clients and an analysis of the data collected
while clients are using the ELA
-virtual house
program. An author system will also be developed
for the therapists working with individual clients to
adapt the program according to the individual needs
of each client.
The authors would like thank the Austrian Science
and Research Liaison Office (project number:
2007.ASO-N/4/5) for their support in the
development of the ELA®-Virtual House.
Stark, J. A., Analysing the language therapy process: The
implicit role of learning and memory. Aphasiology,
19, 10/11 (2005), 1074-1089.
The National Aphasia Association, which is available at:
Stark, J. A., Everyday Life Activities Photo Series,
Manuals to Set 1 to Set 3. Bösmüller/ Jentzsch, Vienna
Stark, J. A., Everyday Life Activities Object Photo Series
Set 1, Jentzsch, Vienna, (2003)
Autodesk Maya, available at:
Perry Harovas, John Kundert-Gibbs and Peter Lee,
Mastering MAYA Complete 2, ISBN: 0-7821-2521-2,
2000 SYBEX, Inc.
Texture mapping, which is available at: http://
Eclipse – an open development platform, which is
available at:
Irrlicht Engine, Available at:
Deep Exploration, which is available at:
DirectX file format, which is available at: http://
Ambierra: irrEdit, which is available at: