Immersive Serious Game-style Virtual Environment for Training in

Electrical Live Line Maintenance Activities

Klaus de Geus

, Rafael T. Beê

, Vinícius M. Corrêa

, Ricardo C. R. dos Santos

Alexandre P. de Faria

, Elton M. Sato

, Vitoldo Swinka-Filho

, Awdry F. Miquelin

Sergio Scheer

, Paulo H. Siqueira

, Walmor C. Godoi

, Matheus Rosendo

and Yuri Gruber

Copel Geração e Transmissão S. A., Rua José Izidoro Biazetto, 158, 81.200-240 Curitiba, Brazil

Universidade Federal do Paraná, PPGMNE, CESEC, Centro Politécnico, 81.530-900 Curitiba, Brazil

Lactec, Rod BR 116, 8.813, Jardim das Américas, 81531-980 Curitiba, Brazil

Keywords: Virtual Reality, Gamification, Learning Theories, Gagné’s Learning Model, Electrical Energy Maintenance,

Critical Activities.

Abstract: This paper describes a virtual environment solution for the training of electricians in critical activities,

namely, live-line maintenance, in electrical energy substations. The main concept of the virtual environment

is the mapping between virtual reality technology and gamification methods with learning theories, in

particular, Gagné’s cognitive model. In order to explore the benefits of gamification, the system uses

concepts established by the Flow Theory, the Magic Circle concept as well as the Player Experience of Need

Satisfaction (PENS) theory. User Experience (UX) is used to assess how the system is perceived by the user.

Non-player characters are modelled to assist the trainee in the learning process. However, they may use

misleading information in order to induce the trainee to make mistakes and thus provide a means of

exercising decision making in adverse conditions, which is an important stage in the learning process,

especially in the context of critical activities. Additionally, an automatic feedback system based on the

visualization of error patterns highlights not only the mistakes made in the virtual experience, but also the

strategy for solving the proposed problem. Tests were carried out aiming at measuring several aspects,

ranging from usability, perception of benefits and learning effectiveness. A trainee classification process is

proposed based on the analysis of human error patterns during the execution of a task. The modelling of

knowledge is based on the literature on human reliability and results from the application of tools such as

task analysis and knowledge extraction from expert users when interacting with the system (expert

elicitation). Clustering techniques applied to error patterns allows for the identification of prototypes of

performance classes and their visualization in the form of distinct groups. Results of different assessment

processes, based on the view of potential users, are presented, analysed and discussed. Future work includes

the conclusion of the automatic evaluation process, based on the analysis and visualization of human error.

1 INTRODUCTION

Virtual reality, along with gamification, has played an

important role in alternative learning in recent years,

especially in the context of critical activities. This

paper describes a virtual environment aimed at

training electricians in power substation maintenance

activities.

https://orcid.org/0000-0003-0303-3548

https://orcid.org/

0000-0002-7459-3780

https://orcid.org/

0000-0003-3995-9780

https://orcid.org/

0000-0002-7498-0721

The general benefits of a complementary virtual

process include a) safety, since there are no risks and,

more precisely, there is no health and life related

threat in a virtual environment, as opposed to the

traditional maintenance activity; b) psychological

effects, since trainees know there are no risks

involved and they can concentrate solely in the

learning process; c) logistics, because the actual

de Geus, K., Beê, R., Corrêa, V., Santos, R., Faria, A., Sato, E., Swinka-Filho, V., Miquelin, A., Scheer, S., Siqueira, P., Godoi, W., Rosendo, M. and Gruber, Y.

Immersive Serious Game-style Virtual Environment for Training in Electrical Live Line Maintenance Activities.

DOI: 10.5220/0009343200420053

In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 2, pages 42-53

ISBN: 978-989-758-417-6

training process requires special arrangements,

favourable weather conditions, the involvement of a

whole team of professionals and displacement of

people and equipment, whereas the virtual process

can be used at any time and can be transported easily;

and d) the innovation of an effective learning through

the mediation of the virtual reality technology

involved.

The most important benefits, however, are related

to the improvement of the training process,

addressing the following issues:

a) the fact that the traditional training can only be

performed sporadically, at specific occasions,

due to the constraints implied in the process,

which makes it very difficult for the trainees

to recall learning, and thus reducing the

effectiveness of the training process;

b) the use of methods, concepts, and cognitive

aspects with great potential to improve the

learning process; and

c) risk activities require a lot of attention, since

they are by nature very dangerous. However,

experienced professionals tend to rely on their

muscle memory, when they have already

mastered the activity to be performed. This

may lead to accidents because of minor

mistakes. In a virtual environment, error-

inducing mechanisms are useful to bring the

issue from the muscle memory back to the

cognitive system. This way, professionals

activate their attention in the activities,

preventing mistakes which can lead to serious

accidents.

The immersive virtual environment relies on a

few concepts which address learning factors not

normally and suitably addressed in traditional

training processes. Thus, it provides an alternative

learning experience capable of a greater degree of

knowledge retention, exploring motivation as a key

factor for learning. This scheme can ultimately result

not only in better quality of services but also in human

security.

It is also very important that a virtual learning

system provide an adequate means of assessing the

effectiveness of learning. However, learning

evaluation is not a simple task, and it normally

considers the progress the learner has made since the

beginning of the training. Automatic evaluation

should be presented to the user as part of the

gamification process, since players admire scores and

feel motivated to carry on until they reach the desired

level.

Due to the professional nature of the critical

activities training, knowledge modelling was carried

out by means of human reliability tools such as task

analysis and expert elicitation during training

sessions. From the interaction data of expert users

with the virtual system during the execution of a task

sequence, error patterns of each user were mapped,

allowing for the proposal of a trainee model. Trainee

error patterns were grouped into performance classes

using simple data mining techniques, such as K-

means, implemented in the R language. The

identification of performance classes provides a basis

for developing a system of the evaluation of new

trainees. This can also be fed into a more advanced

evaluation model, capable of measuring how much

the player has actually learned.

2 RELATED WORK

This section describes the state-of-the-art in the areas

which play a major role in the work, namely, virtual

reality technology and applications, gamification,

virtual modelling and learning evaluation.

2.1 Virtual Reality Applied to Training

in Critical Activities

The attractiveness of virtual reality to training in

critical activities is high due to several aspects.

Trainees are able to train whenever they wish, for

how long they wish. Real world problems do not

represent any danger in a virtual environment, and

this corresponds to a significant advantage in terms of

psychological aspects. Gamification methods can

also enhance motivation, a key aspect of learning.

They provide means to deal with a problem which

affects to a great extent the performance of critical

activities, namely, overconfidence.

Works dealing with the application of virtual

reality in the area of power systems tend to focus on

providing the user with familiarity with an

environment to each there is limited access, so that

the user does not feel uncomfortable or even

threatened when present in the real scene. Such is the

case of the work by Cardoso et al (2017). The work

by Velosa et al. (2018), with application to power

substations, proposed the use of a virtual model to

provide a way to introduce a beginner to the scene, so

that safety distances can be assessed and dealt with

prior to an experience in a real scene.

Other works deal with the feasibility of

implementing a well modelled and fast enough

environment capable of adequately simulating a

power substation. This is the case of the work by

Immersive Serious Game-style Virtual Environment for Training in Electrical Live Line Maintenance Activities

Fanqi and Yunqi (2010) and the work by Nasyrov and

Excell (2018).

2.2 Gamification

Gamification is the application of game mechanics

outside the context of games, stimulating the reward

centre of the brain and creating a psychological

positive reaction to keep the subject interested in the

activity and, thus, to generate engagement and

increase the fidelity of the subject in the activity

implemented by the system.

The flow theory, central to the object of keeping

challenge and engagement, was developed by the

Hungarian-American psychologist Mihaly

Csikszentmihalyi in 1990 (Csikszentmihalyi, 1990)

and further explained in 1997 (Csikszentmihalyi,

1997), claims that the maximum motivation of the

player occurs when the challenge is large enough, but

not impossible to the point of generating either

anxiety or frustration. In this situation, the player can

spend hours playing without realizing the passage of

time.

There must be a goal that can actually be achieved

by the player when performing the activity, otherwise

it will become pointless and the player will lose

interest. Too easy a task implies boredom by the

player. Analogously, too difficult a task implies

frustration or anxiety on the part of the player. There

must be a well-defined relationship between skill and

challenge (Csikszentmihalyi, 1990).

In order to achieve a successful experience, the

player should receive feedback. The brain uses the

feedback as a means to predict performance, to assess

the chances of not reaching the desired goal and,

specifically in the case of a game, the chances of

losing (Tom et al., 2007).

The magic circle concept, within the theories of

Huizinga (1949), considered the pioneer of the study

of the play activity, explores the intrinsic limitations

of an activity, be it spatial, temporal, or even social,

and creates a parallel universe to everyday life, a self-

contained microcosm in which some rules are added

or removed from the player's context, so that errors

may have little or no consequence outside the magic

circle. Finally, the use of the PENS theory (Player

Experience of Need Satisfaction) can help create

virtual engaging experiences, mapping three aspects

that the player feels the need to have: competence,

autonomy and relationship (Scott Rigby, 2011).

The discipline of neuroscience has also shown

that the human mind is constantly subject to

interpretations that may not correspond to reality. It is

not difficult to deceive the human mind by inserting

it into a fictitious scenario and at the same time

convincing it that the scenario is real and that

everything within it is real.

The characteristics reported above led to the

conception of the learning virtual environment

described in this paper as a serious game.

2.3 Virtual Modelling

The modelling of virtual objects is commonly

performed using ontology, since they must interact

with the user and also with other objects in the scene,

that is, they must have a well-defined behaviour.

Zagal et al. (2005) developed a framework aiming

at an ontological language for game analysis. Their

model, called Game Ontology Project (GOP), is

based on the following taxonomy: a) Interface –

interaction of the player with the game; b) Rules –

what and how events happen within the game; c)

Goals – what must be fulfilled by the player; d)

Entities – all interactable objects; and e) Entity

Manipulation – actions by entities and the player.

Yusoff et al. (2009) present a framework for the

development of serious games combined with

learning theories. The model allows for the

specification of technical aspects of the game as well

as learning dynamics based on the ability to fulfil

certain tasks and promote reflection on the player

about the actions taken in the learning process.

A similar approach is used in the work of Tang

and Hanneghan (2010), which presents a model for

the automatic generation of serious games, based on

the following models: a) Game Content Model, used

for defining objects, attributes and relationships; b)

Game Technology Model, used for presenting the

game within a programmatic order; and c) Game

Software Model, used for implementing the model on

a specific platform.

The works listed above describe, in general,

development aspects in the production of serious

games. However, in the proposed virtual

environment, it is necessary to describe the elements

within the dynamics and story of the game, as well as

their behaviour.

2.4 Non-player Characters

Non-player characters (NPC) have played an

important role in virtual environments focused on

social skill training. Moon (2018), in his review paper

on social embodiment for design of NPCs in virtual

reality-based social skill training for autistic children,

stresses that NPCs are “believed to enhance the social

CSEDU 2020 - 12th International Conference on Computer Supported Education

awareness of users in simulated social worlds”. They

have three special roles in virtual training:

i. Serve as an aid to the trainee, giving hints as

to how to solve a problem;

ii. Serve as a sort of companion, providing the

learners with a desirable social engagement;

iii. Initiate problematic situations in order to

expose the learner to situations likely to be

experienced in real life.

Gamage and Ennis (2018) examined some effects

of NPCs in serious games both in terms of learning

and engagement. The results they obtained in their

experiments show that NPCs can lead to significant

positive effects on engagement and also that learners

prefer to use virtual environments which contain

virtual characters, which in turn leads to a positive

effect on learning.

The works by Buede et al. (2016) and Balint et al.

(2018) both address the issue of designing intelligent

NPCs in order to enhance interaction with users of the

virtual system and thus promote engagement.

2.5 Learning Evaluation

Learning virtual systems reported in the literature

normally use experimental tests in order to validate

results, but none of them comprises a module to

actually perform the evaluation of how much the

trainee has actually learned.

Shen et al. (2018) applied questionnaires to

students to assess the effectiveness of the use of

virtual reality in learning, by means of an immersive

VR experience. Analogously, Sankaranarayanan et

al. (2018) applied a questionnaire to a simulation

group and to a control group to assess the difference

in performance when training in a simulated

operating room fire scenario, resulting in a better

performance to the simulation group. Ekstrand et al.

(2018) compared an immersive virtual reality-based

training system with the traditional paper-based

method in the area of neuroanatomy, where 3D

techniques are essential for providing a better

understanding of the brain. The aim of the work by

Mavromihales et al. (2018) was to evaluate the

benefits of games-based learning for the performance

of mechanical engineering students, dividing them

into two groups and then comparing their

performance. Sreelakshmi et al. (2015) integrated a

serious game with the nine events of instructions by

Gagné, and evaluated its benefits. All these works,

regardless of the discipline to which virtual reality

and gamification technology have been applied,

highlight the fact that measuring learning is still a

scientific challenge.

Artificial intelligence techniques have also been

used for two relevant purposes regarding learning

evaluation: a) to map or to infer the knowledge state,

current and future, of the trainee from data generated

within the interaction with the virtual environment;

and b) to be an auxiliary tool for the analysis of

human error risks.

There are vast amounts of studies related to

human error in the scientific literature. This work is

based on some parameters that have been

consolidated in the area of knowledge: First, as stated

by Rasmussen (1982), back in 1982, every

instructional proposal must offer the opportunity for

trial and error experiments. According to Bateson

(1973), as interpreted by Pereira (1983), error is a

necessary consequence of any learning process and,

thus, error is an indicator of change in knowledge.

The recurrence of error, regardless of the context,

allows for identification of error patterns, which

provide indications of critical issues anchored in the

human-system interaction (Reason and Hobbs, 2003).

The virtual learning system described in this paper

goes beyond demonstrating that technology can

improve the learning process, providing a model for

the evaluation of trainees as well as of the learning

they achieve when subject to the training process. The

concept of the model is based on the analysis and

visualization of human error, and is the focus of

ongoing work.

3 METHODOLOGY

The work involves multiple areas of knowledge

aiming at providing an effective solution for the

virtual learning process. The most important areas

involved are geometric modelling, virtual reality,

which can be categorized as technological, and

gamification and learning models, which can be

categorized as cognitive sciences.

In addition, this work addresses two special

aspects. The first consists of the application of virtual

reality technology and gamification aiming at

enhancing the learning process, making special use of

the role of human errors in learning. The second

consists of conceiving an automatic evaluation

model, which is a very challenging task, since

measuring learning is rather difficult and methods to

accomplish that are still incipient.

3.1 Using Errors for Learning

Beyond the work towards the development of a

realistic virtual environment for learning purposes,

Immersive Serious Game-style Virtual Environment for Training in Electrical Live Line Maintenance Activities

the first research topic addressed in the project

attempts to answer the question on the effectiveness

of gamification on learning, especially in the context

of critical activities. The system should make use of

all gamification scheme which bring benefits to the

learning process.

The first thing to do is to map the methods

mentioned in the previous section onto a mechanism

of learning, which in turn is represented by a suitable

learning model. Due to the nature of the activities

addressed in the virtual environment, namely, “live

line maintenance of electrical substations”, which

englobe very rigorous protocols, the learning theory

adopted was Gagné’s learning model, with its nine

events of instruction. For further details on the

Gagné’s learning model, the reader must refer to the

seminal paper by Gagné, Briggs and Wager (1992).

After carefully identifying how each of the nine

events of instruction proposed by Gagné et al. relate

to the physical (standard) training process, it was

possible to analyse which of the gamification

schemes previously mentioned had potential of

application aiming at enhancing the learning process.

Apart from the geometric modelling challenges

involved in the development of the learning

environment, some important decisions had to be

made to ensure a satisfactory learning experience.

One of them was that the experience in the learning

environment should be of the type “single player”,

which implies that only one electrician is trained at

any particular time.

The second modelling challenge is to make every

agent (also known as actor) in the scenes behave

appropriately, according to the type of relationship

they have among themselves. For this purpose, an

ontology scheme was adopted.

For the modelling of behaviour and relationship

patterns of agents (whether they be simple electric

components or maintenance tools), seven

requirements were defined: 1) to take into

consideration the social relationship dynamics

present in real world practice, namely, subordination

and cooperation; 2) to convey to the player a sense of

the need for reaching a specific goal so that a larger

goal is achievable; 3) to consider that a specific

problem in this domain can have multiple possible

solutions; 4) to verify whether the player is able to

make a decision considered correct in activities such

as choice and use of maintenance tool; 5) to consider

possible consequences of not executing a specific

procedure; 6) to consider the nature of fortuitous

violations of good practices; and 7) to consider

possible adverse conditions which could jeopardize

or even prevent activities from being performed.

Logging user interactions with the system while

performing a task enables detection of different types

of errors. According to the taxonomy presented by

Reason and Hobbs (2003), based on Rasmussen's

Skill-Rules-Knowledge hierarchical scheme

(Rasmussen, 1982), these errors assume the following

forms of observable behaviours:

▪ Task not performed due to action omission;

▪ Inaccurate performance;

▪ Action performed in the wrong order;

▪ Performance outside the expected time;

▪ Incorrect action;

▪ Wrong action performed on the correct

component;

▪ Correct action performed on a wrong

component;

▪ Action performed at the wrong time.

Given the critical nature of training activities, the

role of error in assessing learning goes beyond its

pedagogical character. Often used in instructional

planning and learner classification (Bloom et al.

1983), human error is the central element in human

reliability studies and, therefore, was defined as the

fundamental parameter for mapping the trainee's

learning level.

These mechanisms help in the determination of

errors performed by the trainee. They can also

dynamically provide suitable feedback so that they

can estimate their performance and attempt to prevent

failure.

It is assumed that there is a projection of the state

of knowledge of the trainee in the form of behavioural

patterns. A pattern is a reflection of shared beliefs that

influence the performance of an individual. Thus, the

recognition and analysis of a critical situation, or the

execution and validation of a proposed solution, must

correspond to a certain level of performance in

relation to the knowledge of rules and procedures as

a strategy for problem solving and decision making in

critical situations. Data knowledge discovery

techniques have been successfully applied in

instructional systems to either statically or

dynamically infer the trainee's state of knowledge or

to map patterns of user interactions. The purpose of

analysing and visualizing such patterns is to reveal a

dynamic image that allows for the identification of

changes in individual performance and its

classification in relation to the proposed learning

objectives.

3.2 Evaluation Process

The evaluation of learning has always been a

challenge, since there is no definite method which is

CSEDU 2020 - 12th International Conference on Computer Supported Education

able to clearly state how much somebody has learned

from a particular experience or activity. One of the

clearest reasons is the subjective nature of scoring,

since every evaluator uses an internally conceived

scale and therefore different from that of any other

possible evaluator.

Evaluation processes in virtual learning systems

are performed on a learner model consisting of state

variables based on user interaction with the

environment. The trainee modelling was carried out

using tools and models to analyse human reliability

and, in particular, the study of human error in critical

activities. In this perspective, the state of knowledge

of a trainee corresponds to the likelihood of certain

types of errors occurring during the execution of a

task.

The trainee model is based on the probability of

violation or negligence towards implicit semantic

rules. Learning measure, as well as the measure of the

efficiency of the virtual environment, is based on a)

the ability by the trainee to respond to the requests

made by the system; and b) the evolution of this

ability.

Table 1: Error categories and types.

General category Error type

Choice of procedure (P) P1- incomplete

P2 - incorrect

P3 - superfluous

P4 - absent

P5 - unnecessary

Execution (E) E1 - omission

E2 - replication

E3 - inclusion

E4 - sequence

E5 - intervention at some

inappropriate time

E6 - incorrect operator

position

E7 - incomplete action

E8 - unrelated or

inappropriate action

E9 - right action on wrong

object

E 10- unintended action

Recovery from error (R) R1 - too late

R2 - late

R3 - immediate

Performance metrics of a trainee in a virtual

environment allow for the categorization according

to:

a) Trainee’s current level of knowledge;

b) Aims of the training;

c) Interactive processes with the system;

d) Risk associated with human error.

The categorization of human error, in turn, allows

for a) the identification of critical situations; and b)

the definition of preventive and corrective

interventions.

Error categories analysed in this paper were

selected from the work by Sherer et al. (2010) and are

presented in Table 1.

4 DEVELOPMENT

The immersive virtual learning environment was

developed using a game engine, Unreal® Engine 4. A

real electrical substation was digitalized, first using

laser scan and then a geometric modelling tool.

Details of objects, such as day and night cycle, dust

in the air and on insulators and 3D modelled gravel,

enhance realism and thus the sensation of immersion.

Figure 1 illustrates the modelling of a power

substation.

Figure 1: Image showing the modelling of a power

substation and tools used in maintenance activities.

Figure 2: Conceptual diagram showing an abstraction of

components of the live learning virtual environment in four

domains (fundamentals, methods, validation, and

application) along with their relationships. Continuous

arrows indicate direct relationship, whereas dotted arrows

indicate abstract relationship.

The concept of the system can be thought of as the

interaction between several different components.

Figure 2 attempts to represent, in terms of

methodology and technology, how the virtual

learning environment was conceived and developed.

Immersive Serious Game-style Virtual Environment for Training in Electrical Live Line Maintenance Activities

4.1 Virtual Learning Modelling

The system provides five modalities of virtual

training:

a) demonstration modality: the system shows

the procedures to be performed using a

movie-like animation, causing effects on the

components which must be moved or acted

upon.

b) instruction modality: the system takes one

step further, showing to the trainee how the

procedures must be carried out. In this

modality, the trainee is already inside the

virtual scene, but does not perform anything.

c) interaction modality: The trainee is now

responsible for carrying out the procedures

in their logical sequence. In this modality,

errors begin to have a role.

d) game modality: The system now provides

functionalities that make it resemble

entertainment games, including scoring.

This modality attempts to explore the

motivation to play and, thus, to learn.

e) unexpected situations modality: The last

layer of the system is built on top of the

game modality, inserting into the scene

agents with bad behaviour, which will try to

induce the trainee to make mistakes. In this

modality, errors are used to enhance the

learning experience.

4.2 Game Modelling

As far as the effectiveness of using interaction

equipment is concerned, with the intrinsic movement

restrictions, the magic circle concept was explored

and tested with potential users, showing that the

concept indeed works. The users easily transferred

their minds to the parallel world, with its new rules,

namely, the standard controllers and interface devices

of off-the-shelf game consoles, which were capable

of performing six degrees of freedom hand presence,

which contributed to the feeling of immersion by the

trainees.

The concept of Flow has also been explored and

tested with potential users, showing that the sensation

of passage of time had been altered. This means that

the users can train more without getting tired,

enhancing the level of learning. The tests also showed

that immersion significantly enhances the sensation

of a captivating experience and thus the learning

process.

As stated by Tom et al. (2007), adequate feedback

must be given to users, so that they can dynamically

predict their performance, their chances of reaching

the desired goal and, especially in the case of critical

activities, their chances of failing. Since the

environment resembles a game, users with prior game

experience tend to grasp the dynamic of the

environment almost instantaneously.

4.3 Multi-agents and NPCs

Since the environment was modelled as “single-

player”, it must count on NPCs representing the other

participants in the maintenance activity being trained.

The use of NPCs here is also based on what the

scientific literature claims, that it leads to a higher

level of engagement and learning.

In the current version, NPCs are used in order to

provide a means for the use of errors in the learning

process. In this scenario, one of the NPCs, whose

normal role is to assist the learner in the activity and

to perform the actions of other electricians involved

in the activity, can attempt to lead the player to make

a mistake. The mechanism is part of the “unexpected

situations modality” of the virtual environment, and

makes up its most advanced learning module.

The behaviour of NPCs was modelled based on a

solver best known as STRIPS, short for “the Stanford

Research Institute Problem Solver”. As stated by the

authors in their seminal paper (Fikes and Nilsson,

1971), the solver “attempts to find a sequence of

operators in a space of world models to transform a

given initial world model into a model in which a

given goal formula can be proven to be true”.

However, STRIPS lack two characteristics necessary

in the virtual environment: a) it does not account for

human (player) interference; and b) it does not

account for dynamic priorities.

An extension of STRIPS was then developed in

order to allow NPCs to update their tree of plans on

each iteration, and then to alter their priorities

according to their updated needs. This way, an NPC

is able to prioritize harmful actions and thus attempt

to lead the player to an error. The model also accounts

for an “internal distraction” degree, that is, an NPC

has less or more susceptibility of getting distracted in

activities such as trying to talk with the player.

In order to control the behaviour of NPCs, the so-

called BDI (short for “Belief, Desire, Intention”)

model was adopted, which provides NPCs with the

ability to make decisions on their next actions based

on all available information.

CSEDU 2020 - 12th International Conference on Computer Supported Education

4.4 Evaluation Model

The evaluation model of the virtual learning

environment is based on the acquisition, processing,

analysis and visualization of data collected during and

after training. The first issue to address in the

evaluation process is to provide the learner with

feedback on preliminary performance results, since

this is a fundamental principle in the learning process.

This is one of the nine events of Gagné’s model. The

performance assessment (learning results) comes

next, and then the enhancement of retention and

transfer (internalization of knowledge).

It is important to note that, in the work by Gagné

et al. (1992), the concept of instructional planning

was based on a cognitive model grounded on

information processing. Learning depends on

directing efforts and attention to learning outcomes.

A similar cognitive model is fundamental in

studies of human error, as in the model of human

failure proposed by Rasmussen (1982). This model

relates internal mechanisms of human errors with

their manifestation by means of observable

behaviour. This model is the basis for a learning

evaluation method which encompasses elements of

human reliability analysis within instructional design.

Analysing the interactions of learners in the

virtual environment allows for the mapping of their

performance during the execution of a task, which

then reveals patterns of behaviour. The key challenge

now is to identify and visualize these patterns by

means of data mining techniques which should allow

for the classification of patterns and the inference on

the knowledge states of the learners.

5 RESULTS

The evaluation of knowledge acquisition is a

challenge, encompassing many subjective aspects.

Four different types of tests were carried out

independently to validate the virtual environment.

Aspects in one test may be overlapped to some degree

with aspects in the other tests.

5.1 Usability and User Perceptions

The first test had a somewhat subjective nature,

addressing two aspects: usability and perception of

benefits to the learning process by potential users.

The second test was carried out with three different

groups and addressed four aspects: enjoyment,

usefulness, ease of use and immersion. It is worth

noting that it is not so easy to carry out tests due to

the low number of electricians qualified to perform

this kind of activity.

In the first test, each of the two aspects was

evaluated with a single question, as described below:

 Usability: Can you compare this virtual

environment, in terms of effectiveness, with

other computer programs used for learning

purposes?

 Learning benefits: Do you think this virtual

environment will be useful to complement

the training process, bringing benefits in

terms of interest and motivation?

Table 2 shows results of the tests carried out with

19 electricians, in three different virtual environment

experiments, attempting to assess “usability” and

“learning benefits”.

Table 2: Results of usability tests.

Criteria

Likert Scale

Much

worse /

strongly

disagree

Worse /

disagree

Similar

neutral

Better

agree

Much

better /

strongly

agree

Usability

0 0 7 6 6

Learning

benefits

1 2 0 8 8

After careful analysis of the answers, it was clear

that the negative observations on learning benefits

were, in fact, related to usability. This is confirmed

by other arguments they presented in their speeches.

This kind of confusion is fairly normal in experiments

like the ones made here.

The difficulties reported are presented below

along with how the system attempts to address them:

a) Lack of safety distances: the version he

tested did not include safety distances;

b) Lack of force feedback: the system should

address aspects of the learning process

which do not rely on force feedback;

c) Difficulties with the interface: users with no

experience in games or virtual applications

tend to perform significantly better on the

second experiment, improving their ability

in such a way as to match the skills of

experienced users.

The second test had a more general approach in

terms of public, and attempted to assess the virtual

environment as a game. Three different groups were

interviewed: a) game design professionals; b) non-

critical maintenance professionals; and c) critical

Immersive Serious Game-style Virtual Environment for Training in Electrical Live Line Maintenance Activities

activity maintenance professionals. This means that

only the third group corresponded to the public to

which the virtual environment was designed.

The method employed in these tests were by free

speech interview, where interviewees are allowed to

express themselves as they wish. The interviews were

later analysed to produce the results shown here.

Results presented in Figure 3 show satisfactory

perceptions on enjoyment and usefulness, relatively

satisfactory perceptions on immersion and reasonable

perceptions on ease of use. This will be discussed in

more details in the next section.

Figure 3: Responses of interviews with different groups on

four aspects of the virtual learning environment.

5.2 Analysis and Visualization of

Performance Patterns

The modelling of human error and the tracking of the

behaviour of the trainee, by means of the interaction

process in virtual systems, comprises a first stage of

the learning evaluation, which generates data about

their performance. Determining patterns from these

data requires the concomitant use of classification and

visualization techniques.

An important gap in the development of

interactive data visualization techniques, applied to

user interaction data from virtual training and

educational systems, according to Vieira et al. (2018),

is the lack of connection between data visualization

techniques and learning theories. This lack of

integration has produced a situation where the

sophistication and interactivity of visualization

methods are unrelated to a learning theory and thus

fail to communicate information regarding user

performance.

Based on the analysis of the activity called

“pedestal isolator exchange”, analysis of the system

database, expert interactions and literature review on

human error in the electrical sector, a knowledge

model and an elementary data structure are proposed

aimed at composing a “model of the trainee”. The

available data were extracted from a database of 23

training sessions of a simulated activity composed of

13 tasks represented in the unweighted and directed

graph in Figure 4. The complete execution of the

activity corresponds to a minimum spanning tree of

the graph subject to constraints imposed by edge

directions. The identification of violations which

occurred in the training sessions was based on

violations of the constraints derived from the

mapping of tasks relationships in the system database.

The classification of the mistakes made in training

sessions, as well as information about the time and

number of executions of each task, support the

modular structure for the trainee model. Thus, the

probability of error in a given task is subject to the

knowledge of the chance of occurrence of three error

categories (Table 1), illustrated in Figure 5a, namely,

(P) procedure choice, (E) execution and (R) recovery

from error, as well as the expectation of (T) the time

spent and (No) the number of times the task will be

performed. This model is applied to each task

performed, as illustrated in Figure 5b. A state of the

knowledge of the trainee throughout the 13 proposed

tasks is therefore given by a vector of 65 components.

Figure 4: Graph of possible paths for performing the tasks

within the simulated activity.

The error patterns created from the analysis and

treatment of the interaction data of the 23 training

sessions are represented in the heatmap of Figure 6.

Each line corresponds to the monitoring of trainee

interactions with the system. The columns, taken

from 13 to 13, correspond, respectively, to the values

of the variables T, No., P, R and E.

The mapping of interactions through heatmaps

has been explored in the study of social network

users. Cao et al. (2015) propose a system for user

CSEDU 2020 - 12th International Conference on Computer Supported Education

model visualization, called TargetVue, which allows

for the comparison of patterns of user behaviour over

time and supports the detection of anomalies from

different data sources.

Figure 5: Graphs of the model: a) Subgraph that models

probability of error occurrence in Task i; b) Graph of

simulated activity with concatenated tasks.

Figure 6: Heatmap of interactions of 23 training sessions

sorted according to the system entry.

Figure 7: Heatmap ordered and stratified according to five

performance classes.

In the heatmap shown in Figure 7, the matrix lines

were ordered according to similarity between the

patterns. The dendrogram to the left of the figure

allows for the visualization of possible hierarchical

clusters (highlighted line), according to the depth

levels of the groups.

The clusters found in this procedure demonstrate

the ability of the model to capture the state of the

trainee's knowledge by means of patterns which

identify distinct performance classes. The strong

correlation between similar patterns provides

important parameters for the implementation of tools

for automatic performance classification.

5.3 Error-inducing Scheme

The tests carried out to measure perceptions on the

error-inducing scheme involved five live line

electricians. It is worth noting that there are not so

many professionals available for tests, since it is a

very restricted public.

The methodology used in these tests were as

follows: First, electricians were asked to perform a

maintenance activity in the virtual environment,

being told that a virtual assistant (NPC) would help

them. Then, they were asked to do it for a second

time, but they were not advised about the change of

behaviour of the virtual assistant, who was supposed

to, at some point, mislead them. After the second

experience, they were told that the virtual assistant

tried to mislead them, and they were asked to perform

the activity for the third time, but this time the virtual

assistant would behave normally.

After the three performances, they were asked to

answer four questions (free speech) and an additional

question using the Likert scale, as follows:

1. Did you notice that the virtual assistant tried

to mislead you? Did you notice it

immediately? If not, how long did you take

to notice? Only two electricians noticed the

error immediately;

2. What did you think when you were sure that

there was a mistake? Three electricians

focused on what to do to correct the actions

when they realized there was an error. One

thought there was an error in the system;

3. Did this mistake lead you to change your

attitude during the experience? Four

participants said they changed their attitude.

One decided not to rely on the virtual

assistant. The other three said they decided

to pay more attention;

4. Did you expect another attempt by the

virtual assistant to mislead you in the third

session? All participants were expecting an

error in the third session;

5. Do you think the error-inducing mechanism

can bring benefits for the training? Does it

induce the trainee to think more about how

to perform the activity? (Likert scale: 1 -

Immersive Serious Game-style Virtual Environment for Training in Electrical Live Line Maintenance Activities

strongly disagree; 5 - strongly agree) Four

participants strongly agreed about the

benefits the error-inducing scheme can bring

(score 5) and one agreed to some degree

(score 4).

6 DISCUSSION

In terms of usability, most users approved the use of

the environment as an important tool to complement

traditional training, as the first test showed (Table 2).

The minority that was sceptical towards the system

based their opinion in arguments which are either out

of scope or dealt with either in future versions, with

the inclusion of other functionalities, or in further

virtual sessions, since other tests have already shown

that the cognitive load of the system is very low.

Furthermore, in one particular case the trainee even

misunderstood the purpose of the system, claiming

that it would not replace the traditional training

process. In fact, this has never been the purpose of the

system.

The second usability test, carried out with three

different groups using free speech interviews, showed

that all groups perceive the system as useful for its

purposes. Furthermore, despite not unanimous, they

find the system enjoyable, which is an important

aspect of the system since learning is affected by the

way learners feel about the process.

However, none of the groups found that the

system was easy to use. Live line maintenance

professionals had a slightly more positive view on

this aspect. This may be related to the unfamiliarity

other users had on the activity itself, which is

understandable. If one does not know what to do in

real practice, one tends to have more difficulties in

finding out what to do in the virtual environment.

As Figure 3 shows, ease of use received a

significantly lower score in comparison to the other

criteria. In order to analyse this, it is important to

distinguish the two types of population, as their

differences in profile cause significant impact as far

as this criterion is concerned. Technical audiences are

generally focused on solving problems. Indeed, the

literature is right when it says that games are a type of

problem-solving method that refrains from

unnecessary complications. Although the chart does

not segregate the groups, results show that active

maintenance personnel have scored significantly

better than the other groups, mostly due to the

knowledge of what tools should be used at every step

of the activity.

Interviews after the experiments also showed that

there was a general misunderstanding about the

meaning of the term "ease of use". From the thirteen

times the term appeared in the interview (that is, the

thirteen interviewees referred to the term after being

asked about it), twelve of them (all but one) were used

in the sense of "lack of familiarity" with the

equipment.

This means that the criterion "ease of use" has

been compromised, and should be addressed in a

different way. Analyses such as this, based on free

speech and using specific terms, proved to be more

complex than previously imagined. Nevertheless, the

whole experiment was presented here in order to

ensure its reliability and authenticity.

The proposed evaluation model, based on the

analysis of errors occurred during training sessions,

aims at providing parameters for the classification of

trainee performance. In this sense, a cluster analysis

was carried out in order to identify performance

classes. Data were normalized and clustering

techniques were implemented through the libraries

and functions available in the R-CRAN language and

the RStudio development environment (Williams,

2011).

As for the error-inducing tests, results show that

the use of the virtual environment brings benefits for

the learning process.

ACKNOWLEDGEMENTS

This work was developed by the OneReal Research

Group, R&D project PD-06491-0299/2013 proposed

by Copel Geração e Transmissão S.A., under the

auspices of the R&D Programme of Agência

Nacional de Energia Elétrica (ANEEL), Brazil.

REFERENCES

Balint, J.T.; Allbeck, J.M.; Bidarra, J. 2018. Understanding

Everything NPCs Can Do. In Proceedings of FDG,

Malmö, Sweden, August 7-10, 2018 (FDG’18), 10

pages. DOI: 10.1145/3235765.3235776.

Bateson, G. (1973). Steps to an ecology of mind. pp. 133-

149; 250-279, Granada, London.

Bloom, B. S.; Hastings, J. T.; Madaus, G. F. Evaluation

manual formative and summative of school learning.

Pioneer: São Paulo, 1983.

Buede, D.M.; Sticha, P.J.; Axelrad, E.T. (2016).

Conversational Non-Player Characters for Virtual

Training. In: Xu K., Reitter D., Lee D., Osgood N. (eds)

Social, Cultural, and Behavioral Modeling. SBP-

CSEDU 2020 - 12th International Conference on Computer Supported Education

BRiMS 2016. Lecture Notes in Computer Science, vol

9708. Springer, Cham.

Cardoso, A.; Prado, P.R.; Lima G.F.M.; Lamounier E.

(2017). A Virtual Reality Based Approach to Improve

Human Performance and to Minimize Safety Risks

When Operating Power Electric Systems. In: Cetiner

S., Fechtelkotter P., Legatt M. (eds) Advances in

Human Factors in Energy: Oil, Gas, Nuclear and

Electric Power Industries. Advances in Intelligent

Systems and Computing, vol 495. Springer, 2017.

Cao, N.; Shi, C.; Lin, S.; Yu-Ru, L.; Lin, C. Y. (2015)

TargetVue : Visual Analysis of Anomalous User

Behaviors in Online Communication Systems. IEEE

Transactions on Visualization and Computer Graphics,

v. 2626, n. c, 2015.

Csikszentmihalyi, M. Flow: The Psychology of Optimal

Experiences. Harper & Row Publishers, 1990.

Csikszentmihalyi, M. Finding Flow: The Psychology of

Engagement with Everyday Life, B. Books, Ed.

HarperCollins, 1997.

Ekstrand, C.; Jamal, A.; Nguyen, R.; Kudryk, A.; Mann, J.;

Mendez, I. Immersive and interactive virtual reality to

improve learning and retention of neuroanatomy in

medical students: a randomized controlled study.

CMAJ Open, vol. 6, no. 1. DOI

10.9778/cmajo.20170110, 2018.

Fanqi, M.; Yunqi, K. An improved virtual reality engine for

substation simulation. In: 2010 2nd International

Conference on Future Computer and Communication.

ISBN 978-1-4244-5824-0, DOI

10.1109/ICFCC.2010.5497308, 2010.

Fikes, R.E.; Nilsson, N.J. (Winter 1971). "STRIPS: A New

Approach to the Application of Theorem Proving to

Problem Solving" (PDF). Artificial Intelligence. 2 (3–

4): 189–208. doi:10.1016/0004-3702(71)90010-5.

Gagné, R. M.; Briggs, L. J.; Wager, W. W. (1992).

Principles of instructional design (4th ed.). Forth

Worth, TX: Harcourt Brace Jovanovich College

Publishers.

Gamage, V.; Ennis, C. 2018. Examining the effects of a

virtual character on learning and engagement in serious

games. In MIG ’18: Motion, Interaction and Games

(MIG ’18), November 8–10, 2018, Limassol, Cyprus.

ACM, New York, NY, USA, 9 pages.

https://doi.org/10.1145/3274247.3274499

Huizinga, J. Homo Ludens: A Study of the Play-Element in

Culture. Routledge & Kegan Paul, 1949.

Mavromihales, M.; Holmes, V.; Racasan, R. (2018).

‘Game-based learning in mechanical engineering

education: Case study of games-based learning

application in computer aided design assembly’,

International Journal of Mechanical Engineering

Education. doi: 10.1177/0306419018762571, 2018.

Moon, J. Reviews of Social Embodiment for Design of

Non-Player Characters in Virtual Reality-Based Social

Skill Training for Autistic Children. Multimodal

Technologies Interact. 2018, 2, 53.

Nasyrov R.R.; Excell P.S. (2018). New Approaches to

Training of Power Substation Operators Based on

Interactive Virtual Reality. In: Pammer-Schindler V.,

Pérez-Sanagustín M., Drachsler H., Elferink R.,

Scheffel M. (eds) Lifelong Technology-Enhanced

Learning. EC-TEL 2018. Lecture Notes in Computer

Science, vol 11082. Springer, 2018.

Pereira, O. G. Erro humano: uma conferência internacional.

Análise Psicológica, v. 3, 1983.

Rasmussen, J. Human Errors. A Taxonomy for describing

Human Malfunction in Industrial Installations. Journal

of Occupational Accidents, v. 4, p. 311–333, 1982.

Reason, J.; Hobbs, A. Managing maintenance error: a

practical guide. Florida: CRC Press, 2003.

Sankaranarayanan, G.; Wooley, L.; Hogg, D.; Dorozhkin,

D.; Olasky, J.; Chauhan, S.; Fleshman,m J.W.; De, S.;

Scott, D.; Jones, D.B. Immersive virtual reality-based

training improves response in a simulated operating

room fire scenario. Surgical Endoscopy (2018) 32:

3439-3449. https://doi.org/10.1007/s00464-018-6063-

Scott Rigby, R.R. Glued to Games: How Video Games

Draw Us in and Hold Us Spellbound. PRAEGER

FREDERICK A, 2011.

Shen, C.; Ho, J.; Ly, P.T.M.; Kuo, T. Behavioural

intentions of using virtual reality in learning:

perspectives of acceptance of information technology

and learning style. Virtual Reality (2018).

https://doi.org/10.1007/s10055-018-0348-1.

Sreelakshmi, R.; McLain, M.; Rajeshwaran, A.; Rao, B.;

Jayakrishnan, R.; Bijlani, K. "Gamification to enhance

Learning using Gagné's learning model," 2015 6th

International Conference on Computing,

Communication and Networking Technologies

(ICCCNT), Dallas-Fortworth, TX, USA, 2015, pp. 1-6.

DOI10.1109/ICCCNT.2015.7395197, 2015.

Tang, S.; Hanneghan, M. (2010). A model-driven

framework to support development of serious games for

game-based learning. In 2010 Developments in E-

systems Engineering, pages 95–100.

Tom, S.M.; Fox, C.R.; Trepel, C.; Poldrack, R.A. The

neural basis of loss aversion in decision-making under

risk, Science, vol. 315, no. 5811, pp. 515–518, 2007.

Velosa J.D.; Cobo L.; Castillo F.; Castillo C. (2018).

Methodological Proposal for Use of Virtual Reality VR

and Augmented Reality AR in the Formation of

Professional Skills in Industrial Maintenance and

Industrial Safety. In: Auer M., Zutin D. (eds) Online

Engineering & Internet of Things. Lecture Notes in

Networks and Systems, vol 22. Springer, 2018.

Williams, G, (2011). Data Mining with Rattle and R: The

Art of Excavating Data for Knowledge Discovery.

Springer, 2011.

Yusoff, A.; Crowder, R.; Gilbert, L.; Wills, G. (2009). A

conceptual framework for serious games. In

Proceedings of the 2009 Ninth IEEE International

Conference on Advanced Learning Technologies,

ICALT ’09, pages 21–23, Washington, DC, USA. IEEE

Computer Society.

Zagal, J. P.; Mateas, M.; Fernandez-Vara, C.; Hochhalter,

B.; Lichti, N. (2005). Towards an ontological language

for game analysis. In DIGRA Conf.

Immersive Serious Game-style Virtual Environment for Training in Electrical Live Line Maintenance Activities