Usability and Sense of Presence in Virtual Worlds for Distance

Education: A Case Study with Virtual Reality Experts

Aliane Loureiro Krassmann

, Alex Eder da Rocha Mazzuco

, Miguel Melo

Maximino Bessa

2,3

and Magda Bercht

Universidade Federal do Rio Grande do Sul, Porto Alegre, Brazil

INESC TEC, Porto, Portugal

Universidade de Trás-os-Montes e Alto Douro, Vila Real, Portugal

Keywords: Usability, Sense of Presence, Virtual Reality, Virtual Worlds, Distance Education.

Abstract: This case study presents a virtual reality experts’ evaluation of a desktop-based virtual world developed

towards distance education, under the perspectives of usability and sense of presence, which are considered

factors that can potentially influence learning outcomes. Among the results, data from usability and sense of

presence were positively correlated. The sense of presence was achieved, with participants losing track of

time while performing the activity. Experts agreed that the virtual world is easy to use and can prepare students

for the real-world task. The findings outline positive and negative points that must be addressed in order to

optimize the experience of distance education students.

1 INTRODUCTION

Virtual Reality (VR) technology allows the creation

of interactive Virtual Environments (VE) in 3D

graphics. Applied to education, it allows students to

experience subject matter that would be difficult if

not impossible through conventional methods, in

addition to providing a more stimulating and

motivating environment, where learning can be both

challenging and fun. These aspects are special factors

for those who work alone at home, as is the case of

distance education students (Whitelock et al., 2000).

There are different levels of immersion in VR.

With the use of Head-Mounted Displays (HMD),

such as Oculus Rift or HTC Vive, VR systems can be

fully immersive, when the users’ field of view is

obstructed by the display (Christou, 2010). That is

when the stimuli from the physical reality are

suppressed, and from the VE are increased. However,

besides the need for costly devices, these settings

involve high-end computer systems and specific

technical knowledge, somehow restricting the

widespread use in formal education. Schott &

Marshall (2018) add the disadvantage of

cybersickness, saying that we are still unable to

display fully immersive and believable VE without

making a significant number of users feel nauseous.

Virtual world platforms, on the other hand, are

desktop-based semi-immersive VR systems, which

only require a conventional workstation for use, with

interaction provided by a default monitor interface

and controlling devices like mouse and keyboard

(Christou, 2010). Two known examples are the

Second Life and the OpenSimulator (OpenSim),

which are user-friendly platforms, with a sizable

developing community since the 2000s. Even though

unable to provide the same kind of immersive and

realistic experience, users can still interact in first or

third-person perspective, navigating freely through a

VE (Leung, Zulkernine & Isah, 2018). Therefore, it

becomes an affordable alternative, especially feasible

to be widely used in distance education.

When users are projected in a VE, usually through

their avatars, they can have the feeling of “being

there”, which is called the sense of presence. It refers

to the psychological state of experiencing more the

VE rather than the actual physical location, or the

perceptual illusion that the experience is non-

mediated (Lombard & Ditton, 1997). By allowing

students to feel authentic, this construct is considered

a key feature to ensure the transfer of knowledge from

virtual to real world (Dengel & Mäzdefrau, 2018).

Although the sense of presence is usually

associated with higher levels of immersion, studies

have argued that equivalent ratings can be achieved

Krassmann, A., Mazzuco, A., Melo, M., Bessa, M. and Bercht, M.

Usability and Sense of Presence in Virtual Worlds for Distance Education: A Case Study with Virtual Reality Experts.

DOI: 10.5220/0009350401550162

In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 1, pages 155-162

ISBN: 978-989-758-417-6

155

in semi-immersive conditions (e.g. Schroeder et al.,

2017). Additionally, Lee, Wong & Fung (2010) found

that the sense of presence was significantly and

positively correlated to learning outcomes in such

settings. In the study of Natsis et al. (2012), it was in

the monoscopic viewing condition (semi-immersive)

that users achieved better learning outcomes.

Usability, by its turn, is a factor that seems to

influence both performance (Schroeder et al., 2017)

and the sense of presence in a VE (Chow, 2016).

Szczurowski & Smith (2017), for example, explain

that failed interactions, which do not match expected

results, are likely to have a negative impact on the

sense of presence. Consequently, if users lack the

intention to engage in interaction with a VE, they will

likely feel present in the immediate physical

environment rather than in a mediated one. On the

other hand, if they find a VE to be useful and feel

confident in using it, they will be more willing to

suspend the disbelief and allocate attentional

resources to the mediated environment (Chow, 2016).

However, although the desktop-based virtual

world platforms have been explored for over 20 years,

few interface advancements concerning usability

(e.g. realism and natural ways of interaction) were

observed, aggravating a scenario in which the student

is, usually, unfamiliar with the VR technology.

Generally, studies focus on investigating the users’

perceptions, creating a gap of experts’ opinion.

In this sense, the objective of this study is to

perform an evaluation of an educational virtual world

with VR experts, before applying it with the target

audience of distance education students. We believe

they can have a more critical look, by knowing the

best of what the VR technology can offer, and on the

other hand, being aware of its overall limitations and

drawbacks. The evaluation is made under the

perspective of important aspects for learning in VE

(usability and sense of presence), by means of the

following research questions (RQ).

RQ1. What is the VR experts’ usability evaluation of

an educational virtual world?

RQ2. What is the VR experts’ sense of presence in an

educational virtual world?

Giving that the sense of presence has been

associated with a distorted experience of time relative

to the outside world; that is, a loss of the track of time

(Wallis & Tichon, 2013), a third RQ is proposed:

RQ3. Did the VR experts’ lost track of time in an

educational virtual world?

Few studies have included those specific variables

in the investigation of educational virtual worlds.

Ntokas, Maratou & Xenos (2015), for instance,

present a usability and presence evaluation of an

OpenSim virtual world which simulated a Computer

Science scenario. Overall, the results for usability

were considered acceptable, and for the sense of

presence were high. The authors conclude that the VE

contributed with an average improvement of 19.4%

of the students’ learning, suggesting as future work

the investigation of correlations between usability,

presence and educational value.

Naya & Ibáñez (2015), by their turn, analysed the

experience of young students in a virtual ﬁeld trip,

also based on OpenSim. The objective was testing the

viability of the virtual world as a platform for

educational activities. Three main aspects were

analysed: usability, presence, and learning. The

authors conclude that the use of virtual worlds may be

limited yet still powerful and thus not negligible.

The difference of this study consists of evaluating

these variables (except learning) with a sample of VR

experts, adding the temporal dissociation in the

investigation of the sense of presence. We consider as

VR experts professionals and academics with, at

least, one year of experience in developing and

investigating the use of VR technology, employing

high-tech systems, devices and tools.

2 MATERIALS AND METHOD

This research has an explorative case study design. In

the following sections, we present the subjects, the

educational activity developed in the virtual world,

the instruments and the procedure.

2.1 Subjects

We used a non-probability convenience sampling,

composed by 11 VR experts, being 10 (90.90%) male

and 1 (9.09%) female, aged 21-32 years (M=25.18,

SD=3.34), who are investigators and developers in

the MASSIVE laboratory (acronym for Multimodal

Acknowledgeable multiSenSory Immersive Virtual

Environments) from the INESC TEC, Portugal. Their

time of work with VR technology ranges from 1 to 10

years (M=3.00, SD=2.65). They were asked if they

have previously used a virtual world platform like

Second Life or OpenSim. Five answered yes

(45.45%), and only two have used it for development

purposes. That is, the majority have just worked with

more updated high-tech platforms, such as the Unity

engine and 3D Max.

Finally, participants were asked to rate their level

of interest in the subject addressed in the virtual

world, in a 5-point Likert scale. Five demonstrated a

level of interest (45,45%), and four manifested a level

CSEDU 2020 - 12th International Conference on Computer Supported Education

156

of disinterest (36,36%). The remaining two (18,18%)

rated themselves in the neutral point of the scale.

Thus, the sample is divided regarding the interest in

the subject presented in the VE.

2.2 The Virtual World Activity

The OpenSim virtual world is part of the AVATAR

Project (Portuguese acronym for Virtual Learning

Environment and Remote Academic Work), from the

Federal University of Rio Grande do Sul, Brazil. It

was implemented the version 0.8.1.1 in client-server

mode. With a bandwidth of 100Mbps, it supports a

load of approximately 20 users simultaneously. The

Singularity viewer, also free, is the software installed

on the client-side to render the 3D graphical part,

selected due to the compatibility with the native

language of the participants (Portuguese).

Focusing on the discipline of Financial

Mathematics, the virtual world activity covers the

curricular topics of percentage, simple and compound

interest.

It occurs in a building simulating an

accounting firm, populated by automated Non-Player

Characters (NPC). They express themselves bodily,

tapping on the keyboard at their workstations, and

verbally, participating in a textual narrative about the

company processes. As one way to evoke a greater

level of presence is the use of sound (Whitelock et al.,

2000), office-related background sounds are triggered

by the avatar’s presence throughout the environment.

The 3D objects that compose the scenarios were

created in part manually, and in part by importing

files from free online repositories. Didactic videos

from the Youtube repository were added. The scripts

were programmed in Linden Scripting Language

(LSL), including one routine to register the time each

user spent in the VE.

Students navigate the VE in a third-person

perspective, and are given the role of a trainee on the

first day of work, having to pass through five offices

in order to be admitted. Fifteen objective questions

are proposed along the way (a quiz with three

questions in each office).

At each office, the student is received by the

“chief”, who briefly explains the responsibilities of

the department, and then requests to answer the quiz,

which was developed using the Heads-Up Display

(HUD) device (Figure 1 - top). Among the choices of

answer, there is the button “Help”, which rotates the

user’s chair to a screen with a short didactic video

related to the subject matter. There is also a

“Calculator” button to help solving the questions.

After going through the five offices, the student

receives the news that he/she has been accepted into

the company and can start the internship. Then, is

instructed to sit in a workstation, placed in a room

similar to a corporate workspace (Figure 1 – bottom),

in which is congratulated and receives the total score

in the activity. The estimated time to perform the

activity is around 40 minutes.

Figure 1: Screen capture of the quiz (top) and the workspace

(bottom).

In this sense, the virtual world activity has two

main educational objectives for distance education:

1. To provide students with a differentiated fixing

exercise, in which they will practice the knowledge

acquired in the discipline;

2. To provide students with the opportunity to

engage and reflect on the socio-cultural practices of

the accountant profession, experiencing the real-

world task of being admitted in a company.

The virtual world can be applied to a number of

courses with the Financial Mathematics discipline,

and with other topics/areas, following the same or a

different type of narrative, as for example Human

Resource Management or Business Administration.

2.3 Instruments

The following three main instruments were used to

collect the data.

1. General Perceptions questionnaire: three 5-point

Likert scale questions and three open questions that

were elaborated focusing on the overall feedback

regarding the pros and cons of the virtual world.

Usability and Sense of Presence in Virtual Worlds for Distance Education: A Case Study with Virtual Reality Experts

157

2. System Usability Scale (SUS): ten 5-point Likert

scale questions (Brooke, 1996), interweaving positive

and negative assertions. We used the validated

Portuguese version from Martins et al. (2016).

3. ITC Sense of Presence Inventory (ITC-SOPI):

forty-four 7-point Likert scale questions (Lessiter et

al., 2001). The items are divided into four

dimensions: Ecological Validity/Naturalness (EVN),

Engagement (E), Spatial Presence (SP) and Negative

Effects (NE). We used a reduced 35 item Portuguese

version from Vasconcelos-Raposo et al. (2018).

Given that the sense of presence is related to the

sense of losing track of time (Wallis & Tichon, 2013),

we also analyse the user’s estimated time in the VE,

to compare it with the actual time of experience.

2.4 Procedure

Participants were informed about the overall purpose

of the study and signed a consent and demographics

form. Then, they were individually seated on a

workstation with headphones and were explained the

actions necessary to perform the activity. They were

requested to observe the operation and features of the

virtual world, thinking on the practical use by the

target audience. Time limits were not established.

After the experience, they were asked to answer the

instruments in printed format, being requested to

consider their expertise in VR technology. Finally,

they were thanked and dismissed.

3 RESULTS

The presentation of results is divided by instruments

of research. To allow a fast identification of pros and

cons (or strengths and weaknesses) the scores are

presented in decreasing order.

3.1 General Perceptions

Table 2 shows the results of the three closed questions

of this instrument, indicating that the idea of using

such virtual worlds platform in the distance education

(GP1) was the most positively evaluated item

(M=4.73, SD=0.47). Similarly, participants almost

totally agreed that they would like to have this

resource at their disposal if they were distance

education students (GP2, M=4.45, SD=0.82).

When asked about how well the virtual world can

prepare students for the real-world task portraited in

it (GP3), although in less proportion, participants

agreed with this possibility (M=4.27, SD=0.65).

Table 2: General perceptions evaluation.

Question M SD

GP1. Is using the virtual world a good

idea for the distance education mode of

instruction?

4.73 0.47

GP2. If you were a student of a distance

education, would you like to have a

similar virtual world at

our dis

osition?

4.45 0.82

GP3. How well can virtual world

training prepare for the real world task

ortraite

in it?

4.27 0.65

In the sequence, the first open question asked the

experts’ opinion about the strengths of the virtual

world (GP4). In general terms, they mentioned that

the OpenSim platform makes it easy to simulate

environments, any situation, anywhere. They

emphasized as positive the various spaces of the VE,

that it is aesthetically well developed, and the

interaction with characters, mentioning that “avatars

help in realism”. Also, the opportunity to explore the

work world without leaving home was highlighted,

saying that “distance is no longer an obstacle”.

Besides that, they added the possibility of a having a

preview of how a company can actually be, the ability

to get their full attention, and that “there is a

connection between two worlds (real and virtual)”.

Similarly, participants were asked to talk about

the weaknesses of the virtual world (GP5). They

mentioned the lack of: a) interactivity from the

avatars; b) uninterrupted background sounds; c)

immersion and multi-sensorial levels of immersion

(e.g. haptic). Furthermore, they highlighted the

technological limitations of the OpenSim platform,

with the failures of connection, the (too much) lag and

delayed response in actions, and that a long exposure

may eventually create eye fatigue or headache.

Finally, participants were asked to provide overall

feedback, with suggestions of improvements for the

virtual world (GP6). One said that it’s difficult to

improve it because it seems to be already using the

full capabilities of the software, which was

corroborated by other two who mentioned changing

the game engine or improving graphics and

interactivity. The remaining suggested a stronger

interaction with the avatars and to enrich the stimuli

at the auditory sense level, with the use of more

background sounds throughout the VE.

3.2 Usability Evaluation

By performing the SUS analysis procedure, the

overall result obtained was 68.40, which represents an

‘OK’ (acceptable) usability evaluation, on the C

percentile (Brooke, 1996).

CSEDU 2020 - 12th International Conference on Computer Supported Education

158

Table 3 presents the usability results, maintaining

the original mean values of reverse items. It shows

that the five positive assertions received scores

ranging from 3.27 to 4.00, allowing to infer that none

of the usability items was totally approved.

Table 3: The SUS evaluation.

System Usability Scale items M SD

U7. I would imagine that most people

would learn to use this system very

uickl

4.00 1.01

U5. I found the various functions in this

system were well integrate

3.82 1.17

U3. I thought the system was easy to use 3.73 0.90

U9. I felt very confident using the

system

3.55 1.33

U1. I think that I would like to use this

system frequently

3.27 0.60

U10. I needed to learn a lot of things

before I could get going with this

system

2.45 1.14

U2. I found the system unnecessarily

com

lex

2.18 1.00

U4. I think that I would need the support

of a technical person to be able to use

this system

2.18 1.14

U6. I thought there was too much

inconsistenc

in this s

stem

2.09 1.04

U8. I found the system very

cumbersome to use

2.09 1.10

The questions with the highest score address that

most people would learn to use the system very

quickly (U7, M=4.00, SD=1.01), and that its various

functions were well integrated (U5, M=3.82,

SD=1.17). Subsequently, the system was considered

easy to use (U3, M=3.73, SD=0.90), with participants

feeling confident in using it (U9, M=3.55, SD=1.33).

The positive assertion with the lowest score

questioned if they would like to use the system

frequently (U1), receiving an evaluation close to the

neutral point of the scale (M=3.27, SD=0.60).

However, it is not an objective of the virtual world to

be revisited, as it was configured as an activity to be

performed once.

In the bottom half of Table 3 are the negative

assertions. That is, lower scores correspond to better

evaluation. As a result, the scores ranged from 2.45 to

2.09, which represents a partial disagreement with the

aspects addressed, but not a total disagreement as it

would be ideally expected. The less disagreement

(more negative evaluation) occurred regarding the

need for the user to learn many things before get

going with the system (U10, M=2.45, SD=1.14).

In the sequence, participants partially disagreed

that the system was unnecessarily complex (U2,

M=2.18, SD=1.00) and that they would need the

support of a technical person to be able to use it (U4,

M=2.18, SD=1.14). Finally, the items with the most

disagreement (more positive evaluation) refer to the

system having too much inconsistency (U6, M=2.09,

SD=1.04) and being very cumbersome to use (U8,

M=2.09, SD=1.10).

3.3 Sense of Presence Evaluation

Figure 2 presents a graphic with the ITC-SOPI

results, identifying each item’s dimension by colour.

It allows observing that just one item (P17) received

a mean score in the “partially agree” point of the scale

(M=4.09, SD=0.83), and the remaining received

lower scores. This item refers to sensing that the

scenes depicted in the VE could really occur in the

real world. Thus, it can be inferred that participants

agreed that the virtual world was successful in its

attempt to simulating an accounting company.

Figure 2: The ITC-SOPI results.

Usability and Sense of Presence in Virtual Worlds for Distance Education: A Case Study with Virtual Reality Experts

159

Other six questions received mean scores with

values higher than 3.50, but lower than 4.00, which

indicates neutral to a partial agreement. They refer to

subjects feeling that they could interact with the VE

(P3, M=3.91, SD=0.70), that they were not just

watching something (P10, M=3.73, SD=0.90), that

they had the sensation of moving in response to the

VE (P11, M=3.73, SD=0.79), that they felt

participating (P33, M=3.64, SD=0.92) and involved

with the VE (P1, M=3.55, SD=0.69), and that they

lost track of time (P2, M=3.55, SD=1.04). Thus, these

aspects, which refer mostly to the SP dimension

(~66.66%), although not fully contemplated, emerged

as with the more potential to stimulate the

participants’ sense of presence in the virtual world.

The last ten items, in the lower extreme of Figure

2, are mostly from the NE dimension (60%), which,

as it would be expected, received the lowest mean

scores. The items with the most disagreement

(partially to totally), thus most positively evaluated,

refer to participants feelings nauseous (P22, M=1.27,

SD=0.65) and disoriented (P1, M=1.27, SD=0.47).

The disagreement was less for the NE items referring

to the feelings of eye strain (P18, M=1.64, SD=1.12),

headache (P32, M=1.55, SD=1.21) and dizziness

(P12, M=1.45, SD=0.82). The negative effect with

the highest score (M=2.00, SD=1.18, partially

disagree), refer to participants feeling tired (P8).

However, the other 40% of items in the lower

extreme of Figure 2 are from the SP dimension,

indicating that participants partially disagreed that all

their senses were stimulated at the same time (P27,

M=2.09, SD=1.14) and that the characters and/or

objects could almost touch them (P6, M=1.82,

SD=0.60). Participants more strongly disagreed that

the temperature changed to match the scenes (P25,

M=1.45, SD=0.82) and that they could almost smell

different features of the VE (P19, M=1.36, SD=0.67).

Although these aspects can be related to immersion

and imagination of users while interacting with a VE,

they can be more easily stimulated physically by

multi-sensory devices, which are usually not

contemplated by virtual world platforms.

Shapiro Wilks’ test indicated that data from the

ITC-SOPI dimensions (except the NE) and the SUS

score do not have a normal distribution. Therefore,

the Spearman’s correlation test was run, showing a

positive significant correlation between the EVN

dimension and the SUS score (r=0.815, p=0.002).

In a complementary analysis of the sense of

presence, it was investigated if subjects lost track of

time while performing the activity. To this end, their

estimated time was compared with the actual time of

experience, detected by the system logs. Signs of

losing track of time were inferred when this

difference was higher than five minutes. Table 4

presents the results of each participant, showing the

majority of cases (n=6, 54.54%) in this situation.

From these, four (~67%) spent more time than

estimated (More Than Estimated=MTE), and just two

(~33%) spent less time than estimated (Less Than

Estimated=LTE). The differences were also more

discrepant in the first case, with individuals spending

almost double the time estimated (ID 5 and ID 7).

Table 4: Time of the experiences in the virtual world.

ID Estimated

time (min.)

Actual time

(hh:mm:ss)

Difference (more

than 5 min.)

1 10 00:13:30 No

2 20 00:21:45 No

3 15 00:18:22 No

4 60 00:50:52 LTE

5 30 00:56:51 MTE

6 30 00:46:32 MTE

7 18 00:32:10 MTE

8 30 00:38:35 MTE

9 25 00:28:04 No

10 15 00:18:12 No

11 40 00:34:45 LTE

However, although the dissociation of time can be an

indication of the sense of presence (Wallis & Tichon,

2013), it was not observed any pattern of association

between these values and the ITC-SOPI scores.

4 DISCUSSION

The results show that, although knowing more cutting

edge technologies, the VR experts were positive to

the idea of using the OpenSim platform in the

distance education mode of instruction, mentioning

that it has the potential to prepare students for the real-

world task of being admitted in an accounting

company. The opportunity to explore the work world

without leaving home was highlighted.

Concerning suggestions of improvements, one

refers to increasing the auditory stimuli, making it

uninterrupted, as there were moments with no

background sounds, causing discomfort while using

headphones. Even though participants did not clearly

make this connection, it allows agreeing with

Whitelock et al. (2000), when arguing that this an

important component for the sense of presence.

As negative points, the VR experts emphasized

the technological limitations of the OpenSim

platform, as slowness and failures in interaction. In

CSEDU 2020 - 12th International Conference on Computer Supported Education

160

this sense, they see as difficult the possibility of

improvements, due to software limitations. Although

the OpenSim community is still active, with the last

version (0.9.1.1) launched in January 2020, few real

advancements from previous versions were observed.

We speculate that, besides the platform code

foundation limitations, there’s an effort to maintain

the easiness of access and development, limiting the

evolution of this open-source project.

In what concerns usability, none of the SUS items

was totally approved; a result that can also be linked

with the platform limitations. The aspect most

positively evaluated corresponds to the fast speed in

which people would learn to use the system. Despite

that, the importance of instruction was highlighted,

pointing out the need for users to learn many things

before using the system properly. This result suggests

an agreement with the study of Naya & Ibáñez

(2015), in which the operation of the interface

obtained very high marks, even though their

participants were children in their ﬁrst experience.

In the matter of the sense of presence, in

accordance with the general perceptions feedback, the

most positively evaluated aspect refers to participants

agreeing that the scenes depicted in the virtual world

could really occur in the real world. Also, they

highlighted as positive the feeling of interacting with

a VE instead of just being watching something, as it

would be, for example, in a video about the subject.

Another aspect that contributes with the inferring of

presence, lies on the fact that most VR experts were

not correct on guessing the time spent in the VE,

spending mostly more time than estimated. This

result corroborates with the self-report on the ITC-

SOPI instrument, in which participants demonstrated

a neutral to a partial agreement regarding losing the

track of time. Thus, it can be suggested that they had

a feeling of dissociation of time while doing the

activity in the OpenSim platform.

Besides that, the Spatial Presence dimension

emerged as with the more potential to stimulate the

user’s sense of presence. However, some items from

this dimension, related to level of multi-sensorial

stimuli, were also raised as weakest points, a result

that would be expected by the use of a desktop-based

VR interface with no addition of multi-sensorial

devices. This contradictory finding can also be related

to the divergent opinions regarding participants’ level

of interest on the subject matter of the VE. In the

study of Natsis et al. (2012), for example, students

who were more interested in the subject experienced

higher levels of spatial presence.

The negative effect that emerged as with the more

potential to prejudice (or reduce) participants’ sense

of presence refers to the feeling tiredness. This

finding can also be linked with the problems in the

OpenSim platform, evidenced in the previous

analysis, which could have made them feel bored.

Given that there’s not much to do about the

limitations of the platform itself, and that more

interaction with the VE could also mean more

problems, one action that could alleviate this negative

effect is to reduce the size of the virtual world,

consequently reducing the time of experience.

The data from the usability evaluation and the

Ecological Validity/Naturalness dimension of the

sense of presence were positively correlated. This

dimension is related to believability, realism,

naturalness and solidity of the environment. In this

sense, we corroborate with research that suggest that

these aspects are intrinsically associated, and

therefore must be planned together (Chow 2016).

5 CONCLUSION

In this study, a semi-immersive VR alternative, a

virtual world activity developed on a desktop-based

educational OpenSim platform, was evaluated by VR

experts. Usability and sense of presence issues were

diagnosed through specialized feedback, collecting

suggestions to optimize the experience of a target

audience of distance education students.

Answering the research questions, the usability

evaluation can be classified as intermediate. That is,

the VR experts see the potential of allowing students

to easily participate in a simulation of a company, and

the positive aspects of such an approach as a learning

tool, especially to distance education students, which

usually work alone at home, and have few or no

opportunity of professional practices. However, the

weaknesses of the platform were not denied, as

slowness and failures in the interaction. The

component analysis gave us indications of

improvement points that should be prioritized.

The sense of presence evaluation can be classified

as positive. Participants demonstrated that a level of

realism was achieved, even with graphics far from

detailed. Tiresome was highlighted as a negative

aspect that must be treated in order to improve the

user’s sense of presence. This finding indicates the

need for balance between a clean interface and levels

of interactivity between avatar and objects. It

contraries the study of Naya & Ibáñez (2015), using

the same platform, in which the activity lasted about

two hours. The authors claim that this is a similar time

to the real-world activity they simulated (a museum

visit), and obtained high rates for sense of presence.

Usability and Sense of Presence in Virtual Worlds for Distance Education: A Case Study with Virtual Reality Experts

161

Usability and the dimension referring to realism

in the sense of presence were positively correlated,

indicating that improvements in usability can reflect

on the believability of the VE. Alternatively, it means

that a better usability of the system can collaborate to

higher sense of presence and its associated benefits.

To answer the last research question, it was

identified that the VR experts did lose the tracking of

time while performing the activity. That is, it retained

their attention to a point in which they did not see time

passing by. This result demonstrates that even not so

realistic semi-immersive desktop-based VR

platforms have the potential to promote the

concentration of users and retain their attention,

which are useful aspects for education. Also, it shows

indications of presence through dissociation of time.

Besides the drawbacks inherent of a case study,

with the use of a small convenience sample, the

divergent level of participants’ interest on the subject

addressed in the virtual world is a limitation of the

study, which might have contributed to the

impossibility of correlating all dimensions of

presence to usability, or the dissociation of time to the

ratings of presence. Further research should focus on

investigating a larger sample, allowing to compare

the outcomes with different groups of interest on the

subject. In addition, it would be useful to compare the

results with a sample of non-VR experts, and with the

same virtual world in a full immersive setting.

ACKNOWLEDGEMENTS

This work is financed by National Funds through the

Portuguese funding agency, FCT - Fundação para a

Ciência e a Tecnologia within project UIDB/50014/

2020.

REFERENCES

Brooke, J. (1996). SUS-A quick and dirty usability scale.

Usability evaluation in industry, 189(194), 4-7.

Chow, M. (2016). Determinants of presence in 3d virtual

worlds: A structural equation modelling analysis.

Australasian Journal of Educational Technology, 32(1).

Christou, C. (2010). Virtual reality in education. In

Affective, interactive and cognitive methods for e-

learning design: creating an optimal education

experience (pp. 228-243). IGI Global.

Dengel, A., & Mägdefrau, J. (2019, June). Presence Is the

Key to Understanding Immersive Learning. In

International Conference on Immersive Learning (pp.

185-198). Springer, Cham.

Lee, E. A. L., Wong, K. W., & Fung, C. C. (2010). How

does desktop virtual reality enhance learning

outcomes? A structural equation modeling approach.

Computers & Education, 55(4), 1424-1442.

Lessiter, J., Freeman, J., Keogh, E., & Davidoff, J. (2001).

A cross-media presence questionnaire: The ITC-Sense

of Presence Inventory. Presence: Teleoperators &

Virtual Environments, 10(3), 282-297.

Leung, T., Zulkernine, F., & Isah, H. (2018). The use of

Virtual Reality in Enhancing Interdisciplinary Research

and Education. arXiv preprint arXiv:1809.08585.

Lombard, M., & Ditton, T. (1997). At the heart of it all: The

concept of presence. Journal of computer-mediated

communication, 3(2), JCMC321.

Martins, A. I., Rosa, A. F., Queirós, A., Silva, A., & Rocha,

N. P. (2015). European portuguese validation of the

system usability scale (SUS). Procedia Computer

Science, 67, 293-300.

Natsis, A., Vrellis, I., Papachristos, N. M., & Mikropoulos,

T. A. (2012, July). Technological factors, user

characteristics and didactic strategies in educational

virtual environments. In 2012 IEEE 12th International

Conference on Advanced Learning Technologies (pp.

531-535). IEEE.

Naya, V. B., & Ibáñez, L. A. H. (2015). Evaluating user

experience in joint activities between schools and

museums in virtual worlds. Universal Access in the

Information Society, 14(3), 389-398.

Ntokas, I., Maratou, V., & Xenos, M. (2015, September).

Usability and presence evaluation of a 3D virtual world

learning environment simulating information security

threats. In 2015 7th Computer Science and Electronic

Engineering Conference (CEEC) (pp. 71-76). IEEE.

Schroeder, B. L., Bailey, S. K., Johnson, C. I., & Gonzalez-

Holland, E. (2017, July). Presence and usability do not

directly predict procedural recall in virtual reality

training. In International Conference on Human-

Computer Interaction (pp. 54-61). Springer, Cham.

Schott, C., & Marshall, S. (2018). Virtual reality and

situated experiential education: A conceptualization

and exploratory trial. Journal of Computer Assisted

Learning, 34(6), 843-852.

Szczurowski, K., & Smith, M. (2017). Measuring presence:

Hypothetical quantitative framework. In 2017 23rd

International Conference on Virtual System &

Multimedia (VSMM) (pp. 1-8). IEEE.

Vasconcelos-Raposo, J., Melo, M., Teixeira, C., Cabral, L.,

& Bessa, M. (2019). Adaptation and validation of the

ITC-Sense of Presence Inventory for the Portuguese

language. International Journal of Human-Computer

Studies, 125, 1-6.

Wallis, G., & Tichon, J. (2013). Predicting the efficacy of

simulator-based training using a perceptual judgment

task versus questionnaire-based measures of presence.

Presence, 22(1), 67-85.

Whitelock, D., Romano, D., Jelfs, A., & Brna, P. (2000).

Perfect presence: What does this mean for the design of

virtual learning environments?. Education and

information technologies, 5(4), 277-289.

CSEDU 2020 - 12th International Conference on Computer Supported Education

162