First Insights into Hybrid AI-Fuzzy Tutoring System
for Boredom Identification
Viktors Zagorskis
1 a
, Ingrida Lavrinovica
2 b
and Atis Kapenieks
1
1
Distance Education Study Centre, Riga Technical University, Kronvalda 1, Riga, Latvia
2
Telecommunications Institute, Riga Technical University, Azenes 12, Riga, Latvia
Keywords:
e-Leaning Quality, Computer Agent, Artificial Intelligence, Emotions Recognition, Fuzzy Intelligent Control.
Abstract:
In this paper, we introduce the Hybrid AI-Fuzzy Intelligent Control (HAFiC) system prototype. The proposed
model is the add-on to online learning-tutoring environments to proactively detect learners’ emotional states
by measuring performance gaining or degradation in a game-like form. We introduce a system model and
experimental results implementing recently proposed Simple Algorithm for Boredom Identification (SABI)
(Zagorskis et al.,2019) along with the Fuzzy Intelligent control approach to evaluate whether proposed in-
direct data acquisition method allows retrieving performance data variability in correspondence to real user
emotional states. In the proposed system, the AI part cares about Image Processing and Text Recognition gath-
ered from mobile-handwriting devices. In contrast, Fuzzy Expert System part organises users’ performance
data utilisation and decision making based on adaptive fuzzy inference approach. First experiment results
described.
1 INTRODUCTION
This paper introduces with experience of a building
of hybrid intelligent learning-tutoring system mock-
up powered by Artificial Intelligence (AI), Machine
Learning (ML) (Russell and Norvig, 2016), and
Fuzzy Logic (Siddique, 2013) techniques.
Learning environments centered on the student
can help to address some weaknesses of traditional
education models such as complexity in learner’s per-
formance management and generation of personal-
ized learning environments (Lugo et al., 2015). Re-
spectively, there is a need in solutions that would meet
the learners’ needs, interests, rhythms and styles. Ac-
cording to the latest trends in e-learning education
market, personalized content is presented as an in-
telligent machine response to learner’s requirements.
Therefore, design of a personalized learning environ-
ment makes it necessary to cover the following set of
characteristics: understanding the situation of the stu-
dent in terms of emotional and cognitive states, previ-
ous knowledge background, skills, interests, response
to situations related to the teaching learning process
and learning style (Lugo et al., 2015). Individual
a
https://orcid.org/0000-0002-6155-0570
b
https://orcid.org/0000-0002-0343-167X
learning style directly characterizes student’s behav-
ior and plays a key-role in building predictive model
according to cognitive state analysis. For example,
sensing students prefer facts, data and experimenta-
tion whereas intuitive students prefer principles and
theories. Sensing students are patient with detail but
do not like complications, whereas intuitive students
are bored with detail and welcome complications (Li
and Rahman, 2018).
In this paper, we give insights into the implemen-
tation of Simple Algorithm for Boredom Identifica-
tion (SABI) (Zagorskis et al., 2019) in the experimen-
tal mockup. The essence of the SABI algorithm is
boredom identification analyzing periodic handwrit-
ing on mobile surfaces. Afterwards, we validate SABI
algorithm operation in action.
Aim of the research is to evaluate whether indi-
rect data acquisition method realized through SABI
algorithm, AI-based and Fuzzy Intelligent control
methods allows getting data variability suitable for
a successful boredom detection and optimization of
e-learning ecosystem architecture and technology re-
spectively. Hypothesis is formulated based on the re-
sults from previous research (Zagorskis et al., 2019).
The paper has been organized as follows: Sec-
tion 2 contains the reflection of related theories, meth-
ods, and approaches; in Section 3, which is the most
320
Zagorskis, V., Lavrinovica, I. and Kapenieks, A.
First Insights into Hybrid AI-Fuzzy Tutoring System for Boredom Identification.
DOI: 10.5220/0009413203200326
In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 2, pages 320-326
ISBN: 978-989-758-417-6
Copyright
c
2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
important part of the research, we propose design of
the hybrid Fuzzy-logic based tutoring system; in Sec-
tion 4, we describe the experimental results; Section
5 concludes the paper.
2 PRELIMINARIES
2.1 Identifying Learners’ Activity
Each learner’s individual characteristics like motiva-
tion, attitude, learning style, background knowledge,
ability to apply preliminary skills and the number of
asked questions - affect the learning process (Graf
et al., 2010). In the e-learning environment, know-
ing specific characteristics of user activity, it is pos-
sible to reveal hidden personality features like a ten-
dency to be bored or being engaged in the learning
process. Compared with other emotions, such as anx-
iety, boredom can be considered a relatively silent’
emotion (Tze et al., 2016). Summing up the facts
above, our research covers an evaluation of the re-
lationship between student’s academic boredom and
potential learning outcomes.
Analysis of user’s activity data will allow us to ap-
ply AI or ML methods to identify the user’s learning
style. Moreover, by conducting user self-assessment
and emotional states surveys, we will get one more
data tensor for analysis and supervising of machine
methods. Comparing both data, we can make con-
clusions regarding user behaviour patterns, emotional
conditions and learning styles.
Nowadays, instructional designers follow state-of-
the-art practices to engage learners by new instruc-
tional design and content. It has been noted that even
the same content enveloped in different frames can
have a disparate impact on learners attention time.
Learners’ activity and engagement defined by inter-
action environment settings gains better user experi-
ence (UX) and instructional design in a game-like ap-
proach.
2.2 Recent Boredom Detection Methods
There are at least two methods for boredom detec-
tion: direct (or explicit) and implicit method. Ex-
plicit implementation of surveys (or interviewing) is a
kind of intelligent methods involving post-processing
of the textual, audio, and video data. It is applicable
for the usage in small-scale environments while for
large-scaled online learning environments automated
and implicit methods are more relevant.
Environments. In recent years boredom detection
based on automatically inferring user behaviour activ-
ities from mobile phone usage has been actively stud-
ied. M.Pielot et al. introduce the user-independent
machine-learning model of boredom-leveraging fea-
tures related to (1) recency of communication, (2) us-
age intensity, (3) time of a day, and demographics
(Pielot et al., 2015). As results show, M.Pielot et al.
s experiments infer boredom with an accuracy of up
to 82.9%. Before that, Bixler and D’Mello show that
the most popular methods for boredom detection are:
(1) facial expressions, (2) speech and voice features,
text analysis, and physiological signals from mobile
devices (Bixler and D’Mello, 2013).
Self-Assessment Manikin (SAM) method
(Bradley and Lang, 1994) evaluates the pleasure,
arousal and dominance in one scale. In SAM method,
the perceived users’ emotions are obtained by ques-
tionnaires and recorded in three-dimensional spaces
(valence, arousal, and dominance). Each space has
five ranking levels giving the capability to identify
15 different emotions. Although the boredom is not
given in the emotions list, the new methods can be
elaborated modifying SAM method. For example,
fuzzy Hidden Markov Chains (FHMC) we find as one
of possible research directions in the future (Wang
et al., 2014), (Cannarile et al., 2018).
Stochastic Multi-player Games (SMGs). Another
way of modelling stochastic processes is the usage
of games-based methods. For example, for stochas-
tic two-player game modelling two competing entities
can be used: a) a user working on it’s mobile device,
and b) machine or host operating according to its own
or general strategy.
To specify the system evolution by the decisions
of multiple players taking into account the presence
of their probabilistic behaviour, we constrain our at-
tention to turn-based to stochastic games, in which a
single player controls
Definition One. (Stochastic Multi-player Game)
A stochastic multi-player game (SMG) is a tuple G :=
(Π, S, (S
i
)
iΠ
, s, A, δ, L), where:
Π is a finite set of players,
S is a finite state of states,
(S
i
)
iΠ
is a partition of S,
¯s S is an initial state,
A is a finite state of actions,
δ : S × A Dist(S) is a probabilistic transition
function,
L : S 2
AP
is a labeling function mapping states
to sets of atomic propositions.
First Insights into Hybrid AI-Fuzzy Tutoring System for Boredom Identification
321
Probabilistic behaviour typically coexists with
non-determinism. Both non-determinism and prob-
ability (in discrete state space) are present in
the classical model of Markov decision processes
(MDPs), which typically expressed in temporal log-
ics (Kwiatkowska, 2013).
Markov decision processes (MDPs) represent the
case when SMG contains only one player - Π := {1}.
Most common and widely studied class of SMG mod-
els involve two-players (Π := {1, 2}).
Definition Two. (Strategy) A strategy for player i Π
in SMG G is a function σ
i
: (SA) · S
i
Dist(A).
Summing up the above facts, we formulate the fol-
lowing research question: RQ - does indirect data ac-
quisition method allow getting data variability related
to the SABI algorithm?
About Intelligent Control Methods. Notably that
intelligent control systems are not defined in terms
of specific algorithms (Siddique, 2013), and there-
fore are suitably aimed at processes that are com-
plex, nonlinear, time-varying, and stochastic. The
area of intelligent control is inter-disciplinary and
combines methods from other disciplines, including
modern adaptive and optimal control, learning the-
ory, fuzzy logic, and artificial intelligence (Zaknich,
2006). Neural networks, fuzzy logic and genetic algo-
rithms are the constituent techniques of the methods
in a non-classical control engineering.
Another applicable methods belong to the dy-
namic programming (DP) class. In dynamic pro-
gramming exist subclass of reinforcement learning-
based techniques such as Q-learning, R-learning, and
action-dependent heuristic dynamic programming be-
longing to model-free methods. Much of the recent
research on continuous-time reinforcement learning
has focused on model-free methods promising for the
future studies (Kamalapurkar et al., 2018).
Methods for Stated Problem. To continue exper-
iment with SABI algorithm (Zagorskis et al., 2019)
in a hybrid environment, we select appropriate Fuzzy
Intelligent control methods. One of those methods
is an ANFIS - adaptive neuro-fuzzy inference system
(Jang, 1993). Motivated by the need to accommo-
date uncertainties in system model, much of the our
research has focused on Fuzzy methods.
For our experimental setup, we started reveal-
ing the level of boredom according to Bixler and
D’Mello, arguing that there are methods how to
detect boredom during writing tasks through log-
ging the writers’ keystrokes. As authors mentioned,
keystrokes had comparably low predictive power
- roughly 11% above chance - for discriminating
engagement-neutral and boredom-neutral states. By
adding stable traits of the learners to the model, as
Bixler and D’Mello argue, prediction performance
could be notably improved.
3 HYBRID SYSTEM DESIGN AND
EXPERIMENTAL SETUP
In this section, we describe the proof of the concept
of a hybrid boredom detection system based on fuzzy
logic methods.
The system setup involves an approach that during
the learning process organized in a game-like man-
ner, learners are given small regular assignments with
a content that matches learning objectives. Learner’s
interaction with learning content is required. After-
wards, learners apply feedback to the learning envi-
ronment. The response is partially processed, ana-
lyzed, and sent to an application running machine-
learning algorithm to identify specific behavior pat-
terns. The approach is: learners use the mobile sur-
face screens with several simultaneous data channels
providing Mobile device - Host communication. In
our experimental test-bed, the host sends instructional
data (JSON format) to the application deployed and
started on a mobile client device. Similarly, data pro-
duced on a mobile device is preliminarily processed
and sent to the host for analysis and control.
In our approach (Figure 1), we organize acqui-
sition and data processing based on mobile surface
touching events logs. We measure user’s activity
time from users’ typewriting, drawing, and screen
scrolling. The proposed scheme is applicable to each
user-machine interacting experiment (a game round).
To clarify interaction model, we follow notation; we
use subscript index: i - for user-machine interaction
rounds counting.
Figure 1: Simplified user-machine interaction protocol.
One interaction round.
CSEDU 2020 - 12th International Conference on Computer Supported Education
322
In our model, a time spent by a user for the inter-
action with different learning objects (LOs) may vary
in an arbitrary time slot. During interaction we define
the user behaviour as a signal S(t) with independent
variable t - time. The signal for user k is defined as
follows:
S
k
(t) = S
k
C
(t) + N
k
E
(t) + N
k
B
(t) (1)
Now, we clarify notation. The k
th
learner’s con-
scious response to assigned task S
C
is a process S
k
C
(t)
mixed with simultaneously upcoming noise signals
like emotional noise N
k
E
(t) and behavioural noise
N
k
B
(t). Equation 1 defines learner’s answer (a signal)
model responding on a learning task.
Emotional Modality detection from learner’s data
involves both measuring of attention time, spent on
LO, and reply classified as right or wrong answer in-
teracting with LOs. Since learners’ behaviour data is
encoded with a noise signal
N(t) = N
E
(t) + N
B
(t) (2)
, decoding process, in general, lead to signal S
C
(t)
detection errors due to noise stochastic characteris-
tics. Hence, user boredom detection problem relates
to the correct decoding of a signal transferred over a
noisy environment. Here, optimal filtering techniques
can be applied.
Figure 2: Hybrid AI-Fuzzy Intelligent Control (HAFiC)
system. The model of learner’s cognitive signal S
C
(t)
degradation due to impact of emotional N
E
(t) and be-
havioural N
B
(t) noise. Since behavioural data contains var-
ied hidden noise sources, decoding can lead to cognitive
response data detection errors.
To continue with experimental setup description,
we clarify algorithm named as Simple Algorithm for
Boredom Identification (SABI) recently proposed in
(Zagorskis et al., 2019). Also, we reveal crucial ex-
perimental settings structure, components, and usage
aspects before model validation experiment.
To rebuild the SABI algorithm ready for mod-
elling, authors follow the best findings and prac-
tice in the adaptive mobile learning (Cinquin et al.,
2019),(FOX, 2016). Also, into consideration are
taken some relevant findings starting from initial ideas
(Tappert et al., 1990), following through the pitfalls,
issues and challenges (Abuzaraida et al., 2013) to
recent achievements (Wang et al., 2016), that re-
searchers figured out in handwriting recognition.
In Figure 2, we illustrate components interaction
protocol for the Hybrid AI-Fuzzy Intelligent Con-
trol (HAFiC) system - the addon to online learning-
tutoring environment. During the learning process,
users interact with LOs producing a batch of vector
or raster–graphics data. AI and Fuzzy extensions to
SABI algorithm operate, allowing build sustainable
hybrid system with feedback and, what is essential,
forecasting options.
During the experiment calibration process, user
behaviour information is gathered and stored on the
host. Obtained data is also sent to the Fuzzy Expert
system. The goal of calibration is to determine spe-
cific, time-related user behaviour traits. In particular,
to identify an average (or problem-specific) response
time window, we provide experiments in game-like
approach (Figure 3). Gaming element is to control,
restrict, or just identify user response time window in
a sequence of organized experiments. Once Learner’s
Response Time increases in comparison to in calibra-
tion stored ”normal” response time - the learner is
classified as bored if his/her interaction quality addi-
tionally classified as diligent. Diligence detection is
also one of the objectives of the problem landscape.
Here, the fuzzy approach is helpful for experiment-
ing.
4 RESULTS AND DISCUSSION
In this section, we give insights into experimental data
on decision making in a fuzzy-logic equipped envi-
ronment. We represent student’s activity, which is
grouped into five classes based on recommendations
from 1) AI vector or raster–graphics data recognition
system, 2) AI learners’ diligence detection system,
and 3) explicit learners polling regarding their feel-
ings being bored (see Figure 4). We perform fuzzi-
fication as a mapping from an observed input space
to fuzzy sets in a time universe of discourse. In our
model, a time spent on a task is mapped to the enu-
meration of linguistically expressed emotional states
(bored, slow, active, fast, sharp).
Firstly, to represent the experiment data, we ap-
ply Fuzzy C-Means Clustering Algorithm to classify
First Insights into Hybrid AI-Fuzzy Tutoring System for Boredom Identification
323
Figure 3: SABI algorithm in Hybrid AI-Fuzzy Intelligent
Control (HAFiC) system. Example work-path: educator in-
vokes N-rounds Learning Game. Calibration Round starts:
1) learner responds within user-specific time interval. 2) re-
sponse time is recorded as a ”normal” for the user. Round
One: a similar or equal task is assigned. 1) SABI algorithm
identifies a gain of user-machine interaction performance -
user finishes task before calibrated time window ends. 2)
Fuzzy-Expert system algorithms compute learner’s perfor-
mance indicators and can respond with a feedback. Round
N
th
: a similar problem. 1) the user being bored - interaction
is slow or ignored. 2) SABI algorithm identifies signif-
icant time increase comparing to calibrated value. As a
result, 1) Fuzzy-Expert system responds to learners and ed-
ucators by giving a feedback, and 2) learner performance
model updates in HAFiC settings.
outcomes among the five subsets of the A set. The
”sharp” value corresponds to the fastest response time
and correct answer on assigned task, whereas ”bored”
state indicates the slow - non-responsive or discursive
or disjointed actions. Next, we use Gaussian Fuzzy
approximation method applied to each subset.
Finally, we depict one experiment results ( Fig-
ure 3 ). Here, on horizontal axis yellow curve indi-
cates maximum value at 1400 ms, which is the mean
value of data from all the games, and all the game
rounds classified as ”Active” since the first - cali-
bration round. Also, it should be mentioned that in
each individual game-case ”normal” response time is
different and depends on two factors: problem com-
plexity and learner’s personality traits. Also, in pre-
sented experiment data, ”bored” class is characterized
by task completion time greater than 2 sec.
Next, on Figure 5, after data classification and
re-scaling, we identify an unusual value of mem-
bership function associated with the emotional state
”Slow”. Such a result can be explained by an insuf-
ficient amount of data - as it was already mentioned,
in our experiment (n=54). The reason for such a lim-
itation lays within the boundaries of experiment - a
limited number of involved students (54) and a one
task in an assignment.
Figure 4: Students’ activity results. Five diverse judge-
ments about user emotional states on the universe of task
completion time. Dashed line identifies educator expecta-
tions regarding average task completion time. The dotted
line is the mean value of a data class ”normal activity”. In
this experiment, learners are more productive comparing to
educators forecast.
Figure 5: Students’ activity results. Five diverse judge-
ments about user emotional performance on the universe of
task completion time. Degree of membership normalized
and rescaled. The dashed line identifies educator expecta-
tions regarding average task completion time.
Before the experiment, assignment outcomes also
estimated by experts’ council suggesting outcome
equal (on the average) to 1500 ms.
On another hand, during the experiment, we iden-
tify average learners’ performance improvement -
”active” rescaled curve ( on Figure 5 ) is wider and
shifted towards better performance (approx. 1200
ms).
Ultimately, comparing students’ emotional states
after each experiment round (using an explicit ques-
tionnaire method) with data classified from the output
of the experimental Fuzzy Intelligent system mockup,
we acquired the level of confidence close to 70%.
CSEDU 2020 - 12th International Conference on Computer Supported Education
324
5 CONCLUSIONS
Experimental validation of SABI algorithm imple-
mentation in Hybrid AI-Fuzzy mockup system has
shown that such a hybrid system operates and is capa-
ble of solving aimed problems. Although, for build-
ing an accurate model with higher accuracy level,
there should be a sufficient amount of input data.
Summing up the results, we can conclude that AI-
Fuzzy models with simple components are compar-
atively easy to implement and use but they do not
always provide good accuracy. This finding can be
explained by algorithm’s dependence on the number
of students involved in the experiment, the number of
assigned tasks on the host side, as well as the honest
answers, difficulty, and fuzziness, to classify boredom
by students themselves.
The accuracy of experimental model can also be
improved by the usage of complex components and
advanced system architecture but those models are
difficult to implement.
An optimal choice of components for the fuzzy
model element design is especially important when
implementing the model in low-end hardware. Fur-
ther research will involve experimenting with more
data sources, wider range of tasks’ completion time
variety, ensuring feedback automatizing process and
work on the increase of data intelligence efficiency.
ACKNOWLEDGMENT
This research has been supported by a grant
from the European Regional Development Fund
(ERDF/ERAF) project ”Technology Enhanced
Learning E-ecosystem with Stochastic Interdepen-
dences - TELECI”, Project No.1.1.1.1/16/A/154.
REFERENCES
Abuzaraida, M. A., Zeki, A. M., and Zeki, A. M. (2013).
Problems of writing on digital surfaces in online hand-
writing recognition systems. 2013 5th International
Conference on Information and Communication Tech-
nology for the Muslim World, ICT4M 2013, pages 1–5.
Bixler, R. and D’Mello, S. (2013). Detecting boredom and
engagement during writing with keystroke analysis,
task appraisals, and stable traits. In Proceedings of
the 2013 international conference on Intelligent user
interfaces - IUI ’13, page 225, New York, New York,
USA. ACM Press.
Bradley, M. M. and Lang, P. J. (1994). Measuring emotion:
the self-assessment manikin and the semantic differ-
ential. Journal of behavior therapy and experimental
psychiatry, 25 1:49–59.
Cannarile, F., Compare, M., Baraldi, P., Di Maio, F.,
and Zio, E. (2018). Homogeneous Continuous-Time,
Finite-State Hidden Semi-Markov Modeling for En-
hancing Empirical Classification System Diagnostics
of Industrial Components. Machines, 6(3):34.
Cinquin, P.-A., Guitton, P., and Sauz
´
eon, H. (2019). Online
e-learning and cognitive disabilities: A systematic re-
view. Computers & Education, 130:152–167.
FOX, A. M. Y. (2016). Microlearning for Effective Perfor-
mance Management. (cover story). TD: Talent Devel-
opment.
Graf, S., Liu, T.-C., and Kinshuk (2010). Analysis of learn-
ers’ navigational behaviour and their learning styles in
an online course. J. Comp. Assisted Learning, 26:116–
131.
Jang, J. . R. (1993). Anfis: adaptive-network-based fuzzy
inference system. IEEE Transactions on Systems,
Man, and Cybernetics, 23(3):665–685.
Kamalapurkar, R., Walters, P., Rosenfeld, J., and Dixon, W.
(2018). Reinforcement Learning for Optimal Feed-
back Control: A Lyapunov-Based Approach. Com-
munications and Control Engineering. Springer Inter-
national Publishing.
Kwiatkowska, M. Z. (2013). Advances in quantitative veri-
fication for ubiquitous computing. In ICTAC.
Li, L. X. and Rahman, S. S. A. (2018). Students’ learning
style detection using tree augmented naive bayes. In
Royal Society open science.
Lugo, G. S., Rodr
´
ıguez, L.-F., L
´
opez, R. I. G., Mac
´
ıas-
Estrada, A., and Echeverr
´
ıa, M. R. (2015). Behavioral
patterns for automatic detection of learning styles in
learning management systems: a case study. Research
in Computing Science, 106:69–77.
Pielot, M., Dingler, T., Pedro, J. S., and Oliver, N. (2015).
When attention is not scarce - detecting boredom from
mobile phone usage. In Proceedings of the 2015
ACM International Joint Conference on Pervasive and
Ubiquitous Computing, pages 825–836, Osaka. ACM.
Russell, S. and Norvig, P. (2016). Artificial Intelligence: A
Modern Approach. Always learning. Pearson.
Siddique, N. (2013). Intelligent Control: A Hybrid Ap-
proach Based on Fuzzy Logic, Neural Networks and
Genetic Algorithms. Studies in Computational Intelli-
gence. Springer International Publishing.
Tappert, C. C., Suen, C. Y., and Wakahara, T. (1990). The
State of the Art in On-Line Handwriting Recognition.
IEEE Transactions on Pattern Analysis and Machine
Intelligence.
Tze, V. M. C., Daniels, L. M., and Klassen, R. M. (2016).
Evaluating the relationship between boredom and aca-
demic outcomes: A meta-analysis. Educational Psy-
chology Review, 28:119–144.
Wang, G., Zhang, X., Tang, S., Zheng, H., and Zhao, B. Y.
(2016). Unsupervised Clickstream Clustering for User
Behavior Analysis. In Proceedings of the 2016 CHI
Conference on Human Factors in Computing Systems
- CHI ’16.
Wang, N., Sun, S. D., Cai, Z. Q., Zhang, S., and Saygin, C.
(2014). A hidden semi-markov model with duration-
First Insights into Hybrid AI-Fuzzy Tutoring System for Boredom Identification
325
dependent state transition probabilities for prognos-
tics. Mathematical Problems in Engineering, 2014(1).
Zagorskis, V., Kapenieks, A., and Gorbunovs, A. (2019).
Emotions identification utilizing periodic handwriting
on mobile surfaces. Periodicals of Engineering and
Natural Sciences (PEN), 7(1):IC18.
Zaknich, A. (2006). Principles of Adaptive Filters and Self-
learning Systems. Advanced Textbooks in Control and
Signal Processing. Springer London.
CSEDU 2020 - 12th International Conference on Computer Supported Education
326