Effectiveness of Gamification in Undergraduate Education
M. E. Sousa-Vieira, O. Ferreira-Pires, J. C. L
´
opez-Ardao and M. Fern
´
andez-Veiga
atlanTTic, Universidade de Vigo. 36310 Vigo, Spain
Keywords:
Gamification, Social Learning, Learning Analytics.
Abstract:
What is the quantitative effect of gamified learning/teaching on the performance of students? This basic
question has not received as much attention as it deserves in the related literature, despite the many works that
analyze the benefits and design principles of gamified activities across many domains. We present in this paper
an analysis of this question focused on identifying the network structure variables that quantitatively explain
the improvements reported when using gamification integrated in the course design. We compare the results
with those from a group that does not follow gamification at all, hence we are able to ascertain the strength
and impact that gamification has on numerical grades or scores.
1 INTRODUCTION
This paper is an attempt to clarify the following re-
search question: how effective is the use of gam-
ified activities and learning strategies for students?
As an educational approach, gamified learning (GL)
exhibits a number of clear advantages related to the
dynamics of the learning process (Aldemir et al.,
2018; Cakirogli et al., 2017), for instance in cre-
ating stronger and longer engagement from the stu-
dents, and in improving their perceived learning ex-
perience. However, when in comes to a quantita-
tive analysis of the real benefits of using technology-
mediated gamified activities in a course, the picture
extracted from the related research literature is less
clear or systematic. Whereas some studies (Buckley
and Doyle, 2017; Davis et al., 2018; Kyewski and
Kramer, 2018) adopt the point of view of students,
their attitudes and personal traits as one of the factors
that may impact on the effectiveness of GL, another
line of work is focused on less subjective experiments,
and attempts to determine how much improvement
in academic achievement is possible through a com-
prehensive embedding of GL into traditional teaching
styles (Yildirim, 2017).
Our previous work in this area has been focused
on identifying the network properties of the dynam-
ics induced by the GL activities designed for under-
graduates in engineering (Sousa et al., 2017). The
main finding has been not only that the networked
relations formed amongst the students in the online
social realm can be exploited as robust predictors of
final learning success (Sousa et al., 2018), but also
that there exist significant statistical correlation be-
tween basic structural properties in the network and
the graded performance of students (Ferreira et al.,
2020a). In this paper, we take one step further and
make a statistical comparison between groups of stu-
dents exposed to GL and students taught without GL
with respect to different types of GL activities. Our
aim is to estimate the quantitative measure that a
given GL teaching tool may have on the academic
performance, so as to guide in the selection of the GL
tools suitable for a given course.
The rest of the paper is organized as follows. Sec-
tion 2 summarizes some recent related work. The
methodology employed in the course under study is
reported in Section 3. Section 4 contains the main re-
sults of the analysis applied to the datasets. Finally,
some concluding remarks are included in Section 5.
2 RELATED WORK
Gamification is defined as the use of game design el-
ements in non-game contexts (Kapp, 2012). It can be
applied in several situations to influence the behav-
iors of individuals, mainly to increase engagement, to
motivate action or to promote learning. Due to the
fact that all these are major issues faced by teachers
of all educational levels, in recent years multiple im-
plementations of gamification in educational contexts
have emerged. In this section, we present a literature
review of those focused on higher education. Again, a
more extensive compilation can be found in (Subhash
and Cudney, 2018; Zainuddin et al., 2020). Table 1
summarizes, of the game design elements used in our
case study, those mentioned in each paper.
376
Sousa-Vieira, M., Ferreira-Pires, O., López-Ardao, J. and Fernández-Veiga, M.
Effectiveness of Gamification in Undergraduate Education.
DOI: 10.5220/0010495603760385
In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 1, pages 376-385
ISBN: 978-989-758-502-9
Copyright
c
2021 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
The work (Landers and Landers, 2014) studies
the effect of a gamified version of an online wiki-
based project in an industrial/organizational psychol-
ogy course, showing an increase of the interactions.
Moreover, results indicate that time-on-tasks predicts
learning outcomes. The results explained in (Ib
´
a
˜
nez
et al., 2014) show positive effects on the engagement
of C-programming students toward gamified learn-
ing activities and a moderate improvement in learn-
ing outcomes. In (Strmecki et al., 2015), authors
conduct an experimental study to investigate the ef-
fectiveness of gamification of an online informatics
course on computer graphics. Results show that stu-
dents enrolled in the gamified version of the course
achieved greater learning success. The article (Kuo
and Chuang, 2016), reports the application of gami-
fication to an online context for academic promotion
and dissemination. Both quantitative and qualitative
data were collected and analyzed, revealing that gam-
ification has the potential to attract, motivate, engage
and retain users. The research addressed in (Buck-
ley and Doyle, 2017) examines the impact of different
learning styles and personality traits on students’ per-
ceptions of engagement and overall performance in a
gamified business course. Findings suggest that stu-
dents who are oriented towards active or global learn-
ing as well as extroverted students have a positive im-
pression of gamification. The effect of gamified in-
structional process to ICT students engagement and
academic performance is studied in (Cakirogli et al.,
2017). Conclusions show that using the proposed
combination of elements provided a positive motiva-
tional impact on engagement and indirectly affected
the academic results. Similar results are observed
in (de Marcos et al., 2017), where authors found
evidence that gamification can be used to improve
the overall academic performance in practical assign-
ments and to promote social interaction in a qualifi-
cation for ICT users course. The work (Dias, 2017)
describes the application of gamification in an oper-
ations research/management science course, where it
was possible to observe an increase of participation
in class, better results and a good assessment of the
course made by the students. In the study (Sailer
et al., 2017) authors vary different configurations of
game design elements and analyze them in regard to
their effect on the fulfillment of basic psychological
needs. The article (S
´
anchez et al., 2017) presents a
gamification experience within prospective primary
teachers in a general science course. In an effort for
promoting collaborative dynamics rather than com-
petitive ones, a new variable called game index, that
takes into account the scoring of the whole class, was
introduced. A positive correlation among scoring and
academic marks was confirmed. The experiment de-
scribed in (Yildirim, 2017) was conducted to deter-
mine the effect of gamification-based teaching prac-
tices on achievements and attitudes towards learning,
in a course about teaching principles and methods.
The results show positive attitudes towards the lessons
and a moderate effect on achievements. Although
there was not difference between the final grades of
the gamified and the control groups, students regard
wiki and gamified activities positively.
Recently, the article (Aldemir et al., 2018)
presents the gamification process, iterations made into
the game elements and their features and students’
perceptions in a gamified teacher education course.
The study (Davis et al., 2018) investigates college stu-
dents’ experiences of a gamified informatics course,
showing positive trends with respect to students’ per-
ceptions of gamification’s impact on their learning,
achievement, and engagement in the course material.
The authors of (Ding et al., 2018) explore the ef-
fects of gamification on students’ engagement in on-
line discussions. Conclusions and interviews with
students and teachers suggest a positive effect of the
game-related features of the platform. The focus of
the study reported in (Huang and Hew, 2018) is to
explore whether gamification could be a good strat-
egy to motivate students to participate in more out-
of-class activities without decreasing the quality of
work. Results from two experiments conducted in
two master level courses on statistics reveal that the
gamified classes completed significantly more activi-
ties and produced higher quality work. In (Jo et al.,
2018), authors test the effectiveness of adding edu-
cational gaming elements into the online lecture sys-
tem of a flipped classroom, as a method to increase
interest in online preparation before class, obtain-
ing good results and better academic achievements of
mid-upper level students. The work (Jurgelaitis et al.,
2018) analyses the effect of using gamification ele-
ments in a course related to software development.
The study confirms that students’ grades and moti-
vation can increase as a result of applying gamifica-
tion to their learning process. In the experiment de-
scribed in (Kyewski and Kramer, 2018) students were
randomly assigned to three different conditions: no
badges, badges visible to peers and badges only vis-
ible to students themselves. Contrarily to expecta-
tions, the last one was evaluated more positively than
the second one. The effects of using gamification el-
ements in courses that make use of a wiki environ-
ment on the participation rates as well as on student
academic success is addressed in (Ozdener, 2018).
In (Roy and Zaman, 2018), authors measure the pos-
sible evolution of motivational levels in response to
Effectiveness of Gamification in Undergraduate Education
377
the interaction with the game elements used in a uni-
versity course. The findings illustrate the significance
of the individual nature of motivational processes,
the importance of sensitive longitudinal motivation
measurements and the relevance of the implemented
game elements’ design characteristics. The work ad-
dressed in (Tsay et al., 2018) indicates, from a cohort
of undergraduate business students, that course per-
formance was significantly higher among those stu-
dents who participated in the proposed gamified sys-
tem than in those who engaged with the traditional
delivery. The article (Zatarain et al., 2018) describes
an advanced learning environment that detects and re-
sponds to computer programming students’ emotions
by using ML techniques, and incorporates motivation
strategies by using gamification methods. In (Bay-
das and Cicek, 2019), authors develop a scale to mea-
sure the factors that may affect the gamification pro-
cess via kahoot in a pre-service teachers undergradu-
ate course. The study presented in (Ortiz et al., 2019)
describes the positive effect of gamification, based on
leaderboards, on learning performance in an introduc-
tory computer programming course. Finally, in (Toda
et al., 2019) authors propose a solution to help instruc-
tors to plan and deploy gamification concepts with
social network features in learning environments. A
case study over a programming course reveals that the
implemented gamified strategies achieved positive ac-
ceptance among teachers and students.
Related to our prior work in this area, (Sousa et al.,
2017) focused on the quantitative characterization of
non-formal learning methodologies. To this end, we
used one custom software platform for discovering
what factors or variables have statistically significant
correlation with the students’ academic achievements.
The dataset was collected along several consecutive
editions of an undergraduate course. Next, in (Sousa
et al., 2018) we compare and combine the power of
different classifiers for success/failure learning pre-
diction, using as inputs some of the features discov-
ered in previous works that have measurable correla-
tion with the students’ performance. Finally, in (Fer-
reira et al., 2020b) we focused on the analysis of fo-
rums engagement, modeling forums’ interactions as
social graphs. It was the first time that we encourage
and reward quality participation in this activity in the
undergraduate and master level courses under study.
In (Ferreira et al., 2020a) we extended this analysis
and we showed the power of some of the graphs prop-
erties for success/failure learning prediction. In this
work, as the last step of this longitudinal study, we
allow the following-up of the gamification method-
ology to be optional and we compare the results ob-
tained with each one of the two learning paths.
3 EDUCATIONAL CONTEXT &
DATASET
We have taken as our educational environment the
2019/2020 edition of a course on Computer Networks
directed to undergraduates of the second year of the
Telecommunications Technologies Engineering de-
gree. This course has a weekly schedule that spans
14 weeks (between January and May). The classroom
activities are organized as follows:
Lectures, that blend the presentation of concepts,
techniques and algorithms with the practice of
problem-solving skills and discussion of theoreti-
cal questions.
Laboratory sessions, where the students design
and analyze different network scenarios with dif-
ferent protocols, using real or simulated network-
ing equipment. Moreover, in some of these ses-
sions students make a programming assignment.
In both editions the activities are supported by a tai-
lored Moodle site to which the students and teachers
belong, and wherein general communication about
the topics covered takes place. To encourage net-
worked learning and collaborative work, different ac-
tivities are planned and carried out in the platform.
The students may gain different kinds of recognition
by completing or participating in these activities. In
the editions analyzed in this work, these online activ-
ities were proposed:
1. Homework tasks, to be worked out before the in-
class or the laboratory sessions. With this activity
teachers encourage the students to prepare some
of the material in advance.
2. Quizzes, proposed before the midterm exams for
self-training.
3. Collaborative participation in forums. Several fo-
rums were created in Moodle to allow the students
to post questions, doubts or puzzles related to the
organization of the course, the content of the in-
class lectures or the laboratory sessions and the
programming assignments.
4. Optional activities, such as games, peer assess-
ment of tasks, etc.
The score of tasks (and their peer assessment) and
quizzes is measured in so-called merit points, and rep-
resents the total score gained for accomplishment of
these activities in the modality B of the continuous
assessment (a 10% of the final grade). It is possible
to obtain extra merit points by doing the optional ac-
tivities in order to compensate for low scores or late
CSEDU 2021 - 13th International Conference on Computer Supported Education
378
Table 1: Game design elements used in the selected papers.
Badges Boards Levels Points Rewards
(Landers and Landers, 2014) X
(Ib
´
a
˜
nez et al., 2014) X X X X
(Strmecki et al., 2015) X X X X
(Kuo and Chuang, 2016) X X X X X
(Buckley and Doyle, 2017) X X X X X
(Cakirogli et al., 2017) X X X
(de Marcos et al., 2017) X X X
(Dias, 2017) X X X
(Sailer et al., 2017) X X X
(S
´
anchez et al., 2017) X
(Yildirim, 2017) X X X X
(Aldemir et al., 2018) X X X X X
(Davis et al., 2018) X X X X
(Ding et al., 2018) X X X X X
(Huang and Hew, 2018) X X X X X
(Jo et al., 2018) X X
(Jurgelaitis et al., 2018) X X X X X
(Kyewski and Kramer, 2018) X X
(Ozdener, 2018) X X X X X
(Roy and Zaman, 2018) X X
(Tsay et al., 2018) X X
(Zatarain et al., 2018) X X X
(Baydas and Cicek, 2019) X X X X X
(Ortiz et al., 2019) X
(Toda et al., 2019) X X X
submissions of some of the tasks or quizzes. Well-
done peer assessments and the best scores in tasks and
quizzess are rewarded with coins and badges.
Participation in forums, solving doubts or sharing
resources, is also valued with points or votes granted
by the teachers or the classmates. As new points or
votes are obtained, the so-called karma level of each
student increases, depending on different factors that
take into account the quality of the student’s actions
and the comparison with that of his/her classmates.
As the karma level increases, students can get coins.
The use of the virtual classroom is also rewarded
by the automatic scoring of different actions carried
out in the platform related to the normal activity un-
folded along the term, like posting or viewing re-
sources, posting new threads, replying to posts, com-
pleting tasks, etc. The so-called experience points
are organized into levels and are awarded in a con-
trolled environment with maximum values and their
frequency set by the teachers. When students level
up, they get coins.
At any time, a student can check his/her accumu-
lated merit points, karma level and accumulated expe-
rience points and level. Moreover, students can check
their positions in the global rankings and the averages
values of the course. And occasionally, the best stu-
dents of a ranking can be made public to the group.
The coins accumulated at the end of the course
can be converted into the following benefits helpful to
pass the subject (the final exam consist of 6 exercises;
each exercise scores 2 points and the maximun score
is 10 points).
- 32 coins can be changed by the extra exercise
wildcard: the 6 exercises of the exam are cor-
rected and their mark is added up to a maximum
of 10 points.
- 22 coins can be changed by the remove worse ex-
ercise wildcard: the 6 exercises of the exam are
corrected and the worse is not taken into account.
- 16 coins can be changed by the remove exer-
cise wildcard: students choose the exercise whose
score is not taken into account.
- 4 or 6 coins can be converted into one or two
pages of notes for the final exam.
- 3 coins can be changed by 5 bonus merit points up
to a maximum of 25.
For students who do not have a wildcard, the 6 exer-
cises are corrected and the score of each one is scaled
by 10/6. It is clear that the students that follow the
modality B of the continuous assessment can get more
benefit from the gamification strategy.
Students may pass the course after a single fi-
nal examination covering all the material (provided
the programming assignment meets the minimum re-
quirements), but they are encouraged to follow the
continuous assessment. In continuous assessment we
Effectiveness of Gamification in Undergraduate Education
379
allow two modalities, A and B. We weigh 40% the
final exam, but the rest is split as follows: 36% in
modality A and 24% in modality B from the midterm
exams, 24% from the programming assignment and
12% (only in modality B) coming out from the merit
points obtained by accomplishing the online activi-
ties (task, quizzes and optional tasks) described pre-
viously, devised as a tool to increase the level of par-
ticipation. Students have two opportunities to pass the
exam (non-exclusive), May and July.
To finish our description, among all students en-
rolled in this course, 121 students did not drop out
(i.e. they attendend the final exam). Among these
121 students, of the 114 students who followed the
continuous assessment (18 in modality A and 96 in
modality B), 84 finally passed the course (13 (72%) in
modality A, almost all second-taking students, and 71
(74%) in modality B). And none of the 7 students not
engaged in continuous assessment was able to pass.
Related to the rate of success in the final exam, it was
0% for the single final examination modality, 22% for
the continuous assessment modality A and 32% for
the continuous assessment modality B.
At this point, it is important to highlight that the
adaptation of this subject to the lockdown caused by
the pandemic, that affected the second part of the
term, was fast and without incidents for the vast ma-
jority of students, because they were already used to
the platform and the blended learning methodology
since January. In fact, among the students who fol-
lowed the continuous assessment, the dropout rate of
the students in modality A was 1.5 times higher than
that of the students modality B.
4 ANALYSIS OF THE DATASETS
4.1 MERIT POINTS
Figure 1 displays the points obtained by some rep-
resentative students that followed modality B of the
continuous assessment. These points include the
merit points obtained in the tasks and tests proposed,
the extra merit points and the bonus points. In these
figures we represent the accumulated number of merit
points earned by each student along the term. Stu-
dents are identified by his/her position in the ordered
list of final grades (we represent the first 8 students
with the best final grades and the 8 students with the
worst final grades among the 96 students that fol-
lowed the modality B of the continuous assessment
and did not drop out the course). As we can see, the
pattern of accomplishment of all the students repre-
sented in the figure on the top is similar, reaching all
0
20
40
60
80
100
120
0 20 40 60 80 100 120 140
1
2
3
4
5
6
7
8
0
20
40
60
80
100
120
0 20 40 60 80 100 120 140
120
119
118
117
116
115
114
113
Figure 1: Accumulated merit points. First positions in the
ordered list of final grades (top) and last positions (bottom).
or almost all the possible points. And although the ac-
tivity of the students represented in the figure on the
bottom is more irregular, almost all of them reached
more than 80% of all the possible points.
Table 2: Individual merit points.
ˆµ
ˆ
σ
tqmp 58.5253 18.1286
emp 13.8074 6.1131
tmp 75.8786 21.5459
tmps 0.7023 0.2013
Table 2 shows the estimated mean value and stan-
dard deviation of the individual merit points (merit
points form tasks and quizzes(tqmp), extra merit
points (emp) and total merit points (tmp)) and the
slope of total merit points (tmps), that is, the coef-
CSEDU 2021 - 13th International Conference on Computer Supported Education
380
ficient of a linear regression model of the total merit
points per time graph, of all the students that followed
the modality B of the continuous assessment.
Table 3: Correlation between individual merit points and
student’s performance in the final exam.
ˆ
ρ (
ˆ
β,t, P(> |t|))
tqmp 0.4783 (0.0689,5.3112, 7.21 · 10
7
)
emp 0.1677 (0.0559,1.6591, 1.01 · 10
1
)
tmp 0.3654 (0.0483,3.8261, 2.33 · 10
4
)
tmps 0.3944 (5.2706,4.1831, 6.41 · 10
5
)
In order to check the relationship among the patterns
of engagement along the term and knowledge acquisi-
tion, we have measured the statistical correlations be-
tween the individual merit points and the performance
in the final exam (taking into account the sum of the
scores of the 6 exercises before applying wildcards) of
the students that followed the continuous assessment
and did not drop out the course. For this purpose, the
sample correlations
ˆ
ρ were computed and the linear
regression statistical test was used to quantify such
correlations. The estimated linear coefficient is de-
noted by
ˆ
β. Under the null hypothesis (meaning that
there is not such linear dependence) the test statistic
follows a t-distribution and high values are very un-
likely to be observed empirically (James et al., 2013).
In Table 3 we can see statistically significant positive
dependences.
4.2 Forums Activity
We have applied standard SNA techniques (Newman,
2018) to mine the data collected in forums. For such
purpose, we have recorded the events that took place
in each forum, users who posted new threads, users
who replied and the average valuations they received.
This information is represented as a graph where two
nodes, the users, are connected by an edge if one has
given a reply to an entry posted by the other. More-
over, self-edges represent new threads. The weight of
each edge is related to the points or votes obtained by
the reply or the new thread post. Orange edges iden-
tify replies marked as useful by the owner of the ques-
tion and green edges identify the best replies based
on the opinion of the owner of the question and/or the
teachers. An illustration of the graphs related to each
forum is given in Figure 2, where every node is a stu-
dent identified by his/her position in the ordered list
of final grades. The node with label 0 corresponds to
the instructors.
Figure 2: Forums activity graphs. Lessons graph (top), pro-
gramming graph (middle) and organization graph (bottom).
4.3 Measures
Next, we report some of the typical measures of a
graph that can be obtained globally or individually for
each node, and their values in our datasets. Notice
that for some measures we consider simplified ver-
sions of the graphs, where the weight of each edge
is the sum of the weights of all the edges between
the underlying pair of nodes. Moreover, including
self-edges means including the opening of new forum
threads in the analysis.
For the case of degree centrality, we considered
separately the in-degree and out-degree centralities.
In this application, considering the simplified version
of the graphs, the in-degree centrality is the number
of neighbors whose replies a student receives, and the
out-degree centrality is the number of neighbors that
receive the replies given by a student. The results in
Tables 4-6 reveal that the in-degree centrality values
Effectiveness of Gamification in Undergraduate Education
381
Table 4: Summary of parameters of the lessons graph.
degree
in 0.2136
out 0.6244
closeness 0.6026
betweenness 0.6716
eigenvector 0.8454
# cliques
Size
2 110
3 68
4 15
number new threads (µ-σ) 0.4608 0.9485
number new threads (mod. B) (µ-σ) 0.5 0.9765
number replies (µ-σ) 0.9043 2.0475
number replies (mod. B) (µ-σ) 1.0408 2.1821
points new threads (µ-σ) 9.0591 27.5874
points new threads (mod. B) (µ-σ) 14.2755 27.9704
points replies (µ-σ) 20.5652 46.1802
points replies (mod. B) (µ-σ) 24 49.2351
Table 5: Summary of parameters of the programming
graph.
degree
in 0.3298
out 0.7161
closeness 0.6827
betweenness 0.7489
eigenvector 0.8425
# cliques
Size
2 84
3 43
4 4
number new threads (µ-σ) 0.3217 0.7199
number new threads (mod. B) (µ-σ) 0.3673 0.7651
number replies (µ-σ) 0.7304 1.8746
number replies (mod. B) (µ-σ) 0.8163 2.0017
points new threads (µ-σ) 8.5478 19.1532
points new threads (mod. B) (µ-σ) 9.7857 20.3734
points replies (µ-σ) 14.2434 33.9929
points replies (mod. B) (µ-σ) 15.6531 35.9713
are moderate, but the out-degree centrality is notice-
able, indicating a non-homogeneous distribution of
the number of neighbors that receive the replies sub-
mitted by the participants. A subset of few nodes act
as very active participants in forums (among them the
teachers). Nevertheless, more nodes act as generators
of new threads and recipients of information.
For the closeness centrality, that measures how
easily a node can reach other nodes computing the in-
verse of the average length of the shortest paths to all
the other nodes in the graph, the high values shown in
Tables 4-6 are again indicative of the existence of few
very active contributors.
In the case of the betweenness centrality, that tries
to capture the importance of a node in terms of its
role in connecting other nodes, computing the ratio
between the number of shortest paths that a node lies
on and the total number of possible shortest paths
between two nodes, the high values observed in Ta-
bles 4-6 suggest that few nodes act as bridges between
different parts of the graph.
Eigenvector centrality is a measure based on the
premise that a node’s importance is determined by
how important or influential its neighbors are. The
scores arise from a reciprocal process in which the
centrality of each node is proportional to the sum of
the centralities of the nodes it is connected to. Con-
sidering the version of the graph with self-edges, Ta-
Table 6: Summary of parameters of the organization graph.
degree
in 0.2185
out 0.5832
closeness 0.6071
betweenness 0.6417
eigenvector 0.8721
# cliques
Size
2 141
3 99
4 42
5 12
6 1
number new threads (µ-σ) 0.6086 1.2333
number new threads (mod. B) (µ-σ) 0.6428 1.2701
number replies (µ-σ) 1.2696 2.7859
number replies (mod. B) (µ-σ) 1.3061 2.6879
points new threads (µ-σ) 16.1391 32.3066
points new threads (mod. B) (µ-σ) 16.7959 33.0713
points replies (µ-σ) 27.9304 58.5248
points replies (mod. B) (µ-σ) 28.1428 54.4638
Table 7: Correlation between individual features and stu-
dent’s performance in the final exam (lessons graph).
ˆ
ρ (
ˆ
β,t, P(> |t|))
in degree 0.2218 (0.2441,2.4192, 1.72 · 10
2
)
out degree 0.1181 (0.1298,1.2643, 2.09 · 10
1
)
betweenness 0.1151 (0.0082,1.2332, 2.22 · 10
1
)
closeness 0.1593 (1.4924,1.7153, 8.91 · 10
2
)
eigenvector 0.1918 (2.9671, 2.0719,3.99 · 10
2
)
crossclique number 0.1473 (0.0401, 1.5831,1.16 · 10
1
)
number new threads 0.1728 (0.3498, 1.8652,6.47 · 10
2
)
number replies 0.0974 (0.0913,1.0404, 3.01 · 10
1
)
points new threads 0.1427 (0.0099,1.5332, 1.28 · 10
1
)
points replies 0.1156 (0.0048, 1.2376,1.77 · 10
1
)
bles 4-6 show that the measured eigenvector centrality
values are noticeable. Again, this clearly means that
there are substantial differences among the nodes in
their patterns of participation in this activity.
A clique is a completely connected subgraph of a
given graph. So, cliques represent strongly tied sub-
communities where each member interacts with any
other member. And the crossclique number accounts
for the number of cliques a node belongs to. Tables 4-
6 list the number of cliques in the graphs by their size.
Finally, if we consider the non-simplified version
of the graphs, the in-degree centrality is the number of
replies a student receives, and the out-degree central-
ity is the number of replies given by a student. More-
over, the number of self-edges accounts for the num-
ber of new threads opened by each student. In addi-
tion to the intensity of interactions, another important
factor is their quality that can be measured taking into
account the weights of the edges. The results in Ta-
bles 4-6 show the mean value and the standard devi-
ation of this measures for all the students that did not
drop out the course and for the students that followed
the modality B of the continuous assessment. We can
observe higher values for these, mainly in the lessons
forum.
CSEDU 2021 - 13th International Conference on Computer Supported Education
382
Table 8: Correlation between individual features and stu-
dent’s performance in the course (lessons graph).
ˆ
ρ (
ˆ
β,t, P(> |t|))
in degree 0.1654 (0.1951,1.7832, 7.72 · 10
2
)
out degree 0.1244 (0.1466,1.3331, 1.85 · 10
1
)
betweenness 0.1526 (0.0116,1.6424, 1.03 · 10
1
)
closeness 0.1072 (1.0054,1.3943, 1.66 · 10
1
)
eigenvector 0.1201 (1.9893, 1.2851,2.01 · 10
1
)
crossclique number 0.1249 (0.0363, 1.3381,1.83 · 10
1
)
number new threads 0.1286 (0.2791, 1.3791,6.47 · 10
2
)
number replies 0.1064 (0.1071,1.1385, 2.57 · 10
1
)
points new threads 0.1279 (0.0095,1.3711, 1.73 · 10
1
)
points replies 0.1268 (0.0056, 1.3591,1.77 · 10
1
)
Table 9: Correlation between individual features and stu-
dent’s performance in the final exam (programming graph).
ˆ
ρ (
ˆ
β,t, P(> |t|))
in degree 0.0461 (0.0575,0.4908, 6.25 · 10
1
)
out degree 0.0233 (0.1101,0.2481, 8.05 · 10
1
)
betweenness 0.0005 (0.0004,0.0001, 9.99 · 10
1
closeness 0.1162 (1.0895, 1.2442,2.16 · 10
1
)
eigenvector 0.0567 (0.9791,0.6041, 0.54 · 10
1
)
crossclique number 0.0598 (0.0242,0.6381, 5.25 · 10
1
)
number new threads 0.1042 (0.2781,1.1142, 2.68 · 10
1
)
number replies 0.0408 (0.0417,0.4342,6.65 · 10
1
)
points new threads 0.0644 (0.0064, 0.6861,4.94 · 10
1
)
points replies 0.0055 (0.0003,0.0591, 9.53 · 10
1
)
Table 10: Correlation between individual features and stu-
dent’s performance in the course (programming graph).
ˆ
ρ (
ˆ
β,t, P(> |t|))
in degree 0.0921 (0.1232,0.9832, 3.27 · 10
1
)
out degree 0.0785 (0.1101,0.8381, 4.04 · 10
1
)
betweenness 0.0361 (0.0037,0.3857, 7.01 · 10
1
)
closeness 0.1012 (1.5546,1.6656, 9.88 · 10
2
)
eigenvector 0.1098 (2.0321, 1.1742,0.24 · 10
1
)
crossclique number 0.1835 (0.0435, 1.0751,2.85 · 10
1
)
number new threads 0.1555 (0.4446, 1.6743,9.69 · 10
2
)
number replies 0.0948 (0.1041,1.0132, 3.13 · 10
1
)
points new threads 0.1211 (0.0231,1.2976, 1.97 · 10
1
)
points replies 0.0794 (0.0048, 0.8472,3.99 · 10
1
)
4.4 Correlations with Final Results
In order to check the relationship among the patterns
of participation in the forums and the achievements of
the course, we have measured the statistical correla-
tions between the features under study in this section
and the final grades in the final exam (taking into ac-
count the sum of the scores of the 6 exercises before
applying wildcards) and in the course of the students
that followed the continuous assessment and did not
drop out the course.
The results in Tables 7-12 show a statistically sig-
nificant positive dependence between many of the
considered factors and the students’ performance in
the lessons graph.
Table 11: Correlation between individual features and stu-
dent’s performance in the final exam (organization graph).
ˆ
ρ (
ˆ
β,t, P(> |t|))
in degree 0.0203 (0.0156,0.2176, 8.29 · 10
1
)
out degree 0.0279 (0.0252,0.2971,7.67 · 10
1
)
betweenness 0.0044 (0.0002,0.0487, 9.62 · 10
1
)
closeness 0.0362 (0.3258,0.3861, 7.01 · 10
1
)
eigenvector 0.0139 (0.2144, 1.4801,8.83 · 10
1
)
crossclique number 0.0308 (0.0039, 0.3281,7.43 · 10
1
)
number new threads 0.0347 (0.0541, 0.3691,7.13 · 10
1
)
number replies 0.0795 (0.0547,0.8487, 3.98 · 10
1
)
points new threads 0.0017 (0.0001,0.0183,9.85 · 10
1
)
points replies 0.0686 (0.0022, 0.7324,7.95 · 10
1
)
Table 12: Correlation between individual features and stu-
dent’s performance in the course (organization graph).
ˆ
ρ (
ˆ
β,t, P(> |t|))
in degree 0.0065 (0.0053,0.0692, 9.45 · 10
1
)
out degree 0.0095 (0.0092, 0.1021,9.19 · 10
1
)
betweenness 0.0301 (0.0014, 0.3221,7.56 · 10
1
)
closeness 0.0432 (0.8162,0.9044, 3.68 · 10
1
)
eigenvector 0.0068 (0.1128, 0.0732,9.42 · 10
1
)
crossclique number 0.0225 (0.0031, 0.3281,8.11 · 10
1
)
number new threads 0.0129 (0.0216, 0.1381,7.13 · 10
1
)
number replies 0.0519 (0.0383,0.5522, 5.82 · 10
1
)
points new threads 0.0175 (0.0011,0.1871,8.52 · 10
1
)
points replies 0.0245 (0.0008, 0.2611,7.95 · 10
1
)
5 CONCLUSIONS
We presented in this paper a broad correlation study
between individual features pertaining to the structu-
tal properties of the network relations graph formed in
an online social environment and final performance of
students, both for individuals following the GL-based
style and those who refuse this option. Consistently,
our results show that positive correlations are present
always between individuals in the GL group and bet-
ter academic achievement, almost irrespective of the
particular network feature studied. Thus, these stu-
dents attain better grades and better success probabil-
ity in the final exams, are much less prone to quit the
course, and provide more frequent contributions and
information to the forums, usually of high quality and
better cr itical thinking.
A possible limitation of our study is that the split
between the GL group and the non-GL group is not
totally random nor blind, which might introduce some
bias in our results. Nevertheless, we conjecture that,
since for most students this is their first exposure to
GL, their prior attitude toward gamification is incon-
sequential and little relevant as to the final outcomes.
Therefore, our study contributes to the identification
of the measurable variables that correlate positively
and significantly with the students performance.
Effectiveness of Gamification in Undergraduate Education
383
REFERENCES
Aldemir, T., Celik, B., and Kaplan, G. (2018). A qualitative
investigation of student perceptions of game elements
in a gamified course. Computers in Human Behavior,
78:235–254.
Baydas, O. and Cicek, M. (2019). The examination of the
gamification process in undergraduate education: A
scale development study. Technology, Pedagogy and
Education, 18(3):269–285.
Buckley, P. and Doyle, E. (2017). Individualising gamifica-
tion: An investigation of the impact of learning styles
and personality traits on the efficacy of gamification
using a prediction market. Computers & Education,
69:43–55.
Cakirogli, U., Basibuyuk, B., Guter, M., and Atabay, M.
(2017). Gamifying an ICT course: Influences on en-
gagement and academic performance. Computers in
Human Behavior, 69:98–107.
Davis, K., Sridharan, H., Koepke, L., Singh, S., and Boiko,
R. (2018). Learning and engagement in a gamified
course: Investigating the effects of student charac-
teristics. Journal of Computer Assisted Learning,
34(5):492–503.
de Marcos, L., Garc
´
ıa, A., and Garc
´
ıa, E. (2017). Towards
the social gamification of e-learning: A practical ex-
periment. International Journal of Engineering Edu-
cation, 33(1):66–73.
Dias, J. (2017). Teaching operations research to undergrad-
uate management students: The role of gamification.
The International Journal of Management Education,
15:98–111.
Ding, L., Er, E., and Orey, M. (2018). An exploratory study
of student engagement in gamified online discussions.
Computers & Education, 120:213–226.
Ferreira, O., Sousa, M., L
´
opez, J., and Fern
´
andez, M.
(2020a). An assessment of statistically classifica-
tion for socially oriented learning methodologies. In
CSEDU’20, 12
th
International Conference on Com-
puter Supported Education.
Ferreira, O., Sousa, M., L
´
opez, J., and Fern
´
andez, M.
(2020b). Studying relationships between network
structure in educational forums and students’ perfor-
mance. In Communications in Computer and In-
formation Science. Computer Supported Education.
Springer.
Huang, B. and Hew, K. (2018). Implementing a theory-
driven gamification model in higher education flipped
courses: Effects on out-of-class activity completion
and quality of artifacts. Computers & Education,
125:254–272.
Ib
´
a
˜
nez, M., Di-Serio, A., and Delgado, C. (2014). Gam-
ification for engaging computer science students in
learning activities: A case study. IEEE Transactions
on Learning Technologies, 7(3):291–301.
James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013).
An introduction to statistical learning with applica-
tions in R. Springer.
Jo, J., Jun, H., and Lim, H. (2018). A comparative
study on gamification of the flipped classroom in en-
gineering education to enhance the effects of learn-
ing. Computer Application on Engineering Education,
26(5):1626–1640.
Jurgelaitis, M., Ceponien
`
e, L., Ceponis, J., and Drungilas,
V. (2018). Implementing gamification in a university-
level uml modeling course: A case study. Computer
Applications in Engineering Education, 27(2):332–
343.
Kapp, F. (2012). The gamification of learning and instruc-
tion: Game-based methods and strategies for training
and education. Pfeiffer.
Kuo, M. and Chuang, T. (2016). How gamification moti-
vates visits and engagement for online academic dis-
semination: An empirical study. Computers in Human
Behavior, 55:16–27.
Kyewski, E. and Kramer, N. (2018). To gamify or not to
gamify? an experimental field study of the influence
of badges on motivation, activity and performance in
an online learning course. Computers & Education,
118:25–37.
Landers, R. and Landers, A. (2014). An empirical test of
the theory of gamified learning: The effect of leader-
boards on time-on-task and academic performance.
Simulation & Gaming, 45(6):769–785.
Newman, M. (2018). Networks. Oxford University Press.
Ortiz, M., Chiluiza, K., and Valcke, M. (2019). Gamifi-
cation through leaderboards: An empirical study in
engineering education. Computer Applications in En-
gineering Education, 27(4):777–788.
Ozdener, N. (2018). Gamification for enhancing web 2.0
based educational activities: The case of pre-service
grade school teachers using educational wiki pages.
Telematics and Informatics, 35(3):564–578.
Roy, R. and Zaman, B. (2018). Need-supporting gamifi-
cation in education: An assessment of motivational
effects over time. Computers & Education, 127:283–
297.
Sailer, M., Ulrich, J., Mayr, S., and Mandl, H. (2017). How
gamification motivates: An experimental study of the
effects of specific game design elements on psycho-
logical need satisfaction. Computers in Human Be-
havior, 69:371–380.
S
´
anchez, J., Ca
˜
nada, F., and D
´
avila, M. (2017). Just a
game? gamifying a general science class at univer-
sity. collaborative and competitive work implications.
Thinking Skills and Creativity, 26:51–59.
Sousa, M., L
´
opez, J., Fern
´
andez, M., Ferreira, O.,
Rodr
´
ıguez, M., and Rodr
´
ıguez, R. (2018). Predic-
tion of learning success/failure via pace of events in
a social learning network platform. Computer Appli-
cations in Engineering Education, 26(6):2047–2057.
Sousa, M., L
´
opez, J., Fern
´
andez, M., Rodr
´
ıguez, M., and
L
´
opez, C. (2017). Mining relations in learning-
oriented social networks. Computer Applications in
Engineering Education, 25(5):769–784.
Strmecki, D., Bernik, A., and Radosevic, D. (2015). Gami-
fication in e-learning: Introducing gamified design el-
ements into e-learning systems. Journal of Computer
Science, 11(12):1108–1117.
Subhash, S. and Cudney, E. (2018). Gamified learning in
higher education: A systematic review of the litera-
ture. Computers in Human Behavior, 87:192–206.
CSEDU 2021 - 13th International Conference on Computer Supported Education
384
Toda, A., Carmo, R., Silva, A., Bittencourt, I., and Isotani,
S. (2019). An approach for planning and deploying
gamification concepts with social networks within ed-
ucational contexts. International Journal of Informa-
tion Management, 46:294–303.
Tsay, C., Kofinas, A., and Luo, J. (2018). Enhancing student
learning experience with technology-mediated gamifi-
cation: An empirical study. Computers & Education,
121:1–17.
Yildirim, I. (2017). The effects of gamification-based teach-
ing practices on student achievement and students’ at-
titudes toward lessons. Internet and Higher Educa-
tion, 32:86–92.
Zainuddin, Z., Wah-Chu, S., Shujahat, M., and Perera, C.
(2020). The impact of gamification on learning and in-
struction: A systematic review of empirical evidence.
Educational Research Review, 30.
Zatarain, R., Barr
´
on, M., R
´
ıos, J., and Alor, G. (2018). A
virtual environment for learning computer coding us-
ing gamification and emotion recognition. Interactive
Learning Environments.
Effectiveness of Gamification in Undergraduate Education
385