Good and Similar Learners’ Recommendation in Adaptive Learning
Systems
Dade Nurjanah
School of Computing, Telkom University, Bandung, Indonesia
Keywords: Recommender Systems, Collaborative Filtering, Content-based Filtering, Good Learners, Similar Learners,
Adaptation Engine, Adaptive Learning Systems.
Abstract: Classic challenges in adaptive learning systems are about performing adaptive navigation that recommends a
topic or concept to be learned next and learning materials relevant to the topic. Both recommendations have
to meet active learners’ needs. As adaptive navigation problems have been solved using artificial intelligence
techniques, learning material recommendation problems can be solved using recommender techniques that
have been successfully applied to other problems. Until recently there have been a number of techniques that
come with certain advantages and disadvantages. This paper proposes a new technique for recommending
learning materials that combine content-based filtering and collaborative filtering based on the similarity
between learners and learners’ competence. It aims to diminish the drawback of classic collaborative filtering,
which is based on the similarities between learners and does not consider learners’ competence. It also
diminishes problems arising from collaborative filtering based on good learners’ competence, which
potentially produces recommended objects that do not meet the learners’ condition. The results of a recent
experiment show that the proposed technique performs well, as indicated by the MAE score of 0.96 for a
rating scale of 1 to 10.
1 INTRODUCTION
Adaptive learning technologies aim to individualise
learning by taking into account the learner’s needs,
learning styles, preferences, competence, and
learning goals in tailoring content and teaching
strategies. The adaptation is delivered in various
types of adaptive techniques, including adaptive
navigation that selects and recommends concepts to
be learned next by the active learner. The
recommended concepts are the most appropriate
concepts for the learner regarding their
characteristics. A challenge emerges when adaptive
learning systems use existing online materials. The
challenge deals with how to find appropriate learning
materials for the recommended concept. Challenges
occur in matching multi-dimensional learner models
and a large number of learning materials with various
formats (Knutov et al., 2009). Previous studies have
implied that an individual user model should find
appropriate learning objects (Sicilia et al., 2010;
Wang et al., 2007).
As learning tends to be a social process, other
learners’ experiences become important and act as
references. According to social learning theories, a
learner learns better when accompanied by
experienced learners (McLeod, 2007; Vygotsky,
1978). Furthermore, learners learn by observing the
behaviour of others and the outcomes, and they most
likely copy the behaviour if the outcome is positive.
This is supported by another study which reports that
learners can build their knowledge from the help and
support they receive from peers who perform well;
these are called peer helpers (Topping, 2005).
The principles of social learning have been
implemented in recommender systems with
collaborative filtering approaches. A
recommendation for a learner is performed based on
recommendations from peers who share certain
similarities with the learner (Indrayadi and Nurjanah,
2015; Lops et al., 2011; Wang et al., 2007). This
approach is considered to produce appropriate
recommendations since similar learners will like
similar objects. However, in the context of learning,
the recommended objects are probably not the objects
needed to boost the learner’s performance. Hence,
other approaches that consider recommendations
from good learners will be more useful. Learning
material recommendation systems based on good
learners’ recommendations have been studied before
434
Nurjanah, D.
Good and Similar Learners’ Recommendation in Adaptive Learning Systems.
In Proceedings of the 8th International Conference on Computer Supported Education (CSEDU 2016) - Volume 1, pages 434-440
ISBN: 978-989-758-179-3
Copyright
c
2016 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
(Ghauth and Abdullah, 2011). According to precision
and recall scores, it has been proved that
recommendation results are better than
recommendations based on similarities between
learners.
The inclusion of good learners in learning the
process is not a new concept. Topping’s theory on
peer learning spots the importance of peer helpers in
learning (Topping, 2005). Peer helpers are chosen out
of the best learners and they have superior mastery in
a small part of the curriculum. In terms of the
recommender systems of learning materials,
Topping’s theory is implemented by including good
learners’ experiences in the recommendation. It is
different from conventional collaborative filtering,
which considers the similarity between learners in
rating learning materials without considering how
good the peer learners are.
The inclusion of good learners in learning process
is not a new concept. Topping’s theory on peer
learning spots the importance of peer helpers in
learning (Topping, 2005). Peer helpers are chosen
amongst the best learners and they have superior
mastery in a small part of curriculum. In term of
recommender systems of learning materials, the
Topping’s theory is implemented by including good
learners’ experience in recommendation. It is
different from the conventional collaborative filtering
that considers the similarity among learners in rating
learning materials without considering how good the
peer learners.
In this study, we combine collaborative tagging
and filtering and introduce a new approach to
collaborative filtering by applying good learners’
recommendations combined with similar learners’
recommendations. By considering good learners’
recommendations, the recommender will produce
learning materials that meet the learners’ needs. On
the other hand, the conventional method produces
recommendations appropriate to the learners’
characteristics in that learners like the recommended
objects. Hence, we argue that a combination of the
two methods will improve the quality and the
suitability of the recommendations for learners.
The remaining part of this paper is organised as
follows. Section two, on related work, discusses
current studies on recommender systems. Section
three, on recommenders of learning materials,
discusses our proposed framework for recommending
learning materials based on recommendations from
good learners who have been rated as similar to the
active learner. Section four discusses the experiments
and the results. It is followed by section five, which
includes the conclusion and discusses future work.
2 RELATED WORK
The process of recommending learning objects can refer
to previous studies on recommendation systems for
various objects, such as movies, learning materials,
books, goods, et cetera. A recommender system is a tool
that identifies items that are similar to the active user’s
interests. Recommendation systems help users to choose
objects they might find in their interests or that are
useful. The main purpose of recommendation systems is
to choose certain objects that meet the users’
requirements. The quality of the recommendation
depends on the experience of the active user in rating the
objects and the rating patterns the objects have received.
There are two main approaches in recommender
systems: content-based and user-based approaches.
The first approach recommends objects that share
similarities with other objects liked by the active user
in the past (Lops et al., 2011). The key to this approach
is that the objects that might interest the user must be
similar to the objects he or she has liked previously.
Content-based approaches identify new, interesting
items based on the similarities between the features of
the items. Hence, new items share similarities with the
items that the user has previously viewed. It treats the
recommendation problem as a search for related
objects. When a user rates an item, the algorithm
constructs a search query to find other items with
similar keywords or subjects that have been given
similar ratings. Information about objects is stored and
considered in the recommendation process. In previous
studies, content-based approaches have been combined
with the user’s preferences. In learning material
recommender systems, for example, the preferences
could be media, language, or the topic being learned by
the user (Wang et al., 2007).
On the other hand, user-based approaches or
collaborative filtering make recommendations based on
similarities between the active user and other users
(Sicilia et al., 2010). The principle is that users with
similar profiles will like similar objects. The similarities
could be measured according to users’ competence
(Cazella et al., 2010), preferences and rating pattern
(Indrayadi and Nurjanah, 2015; Wang et al., 2007), or
other parameters (Verbert et al., 2012). In terms of the
use of rating in collaborative filtering, users are required
to express their preferences by rating items.
There have been many previous studies on
recommender types of learning systems (Chen et al.,
2005; Lu, 2004; Verbert et al., 2012) proposed a
framework for a recommender system that helped
learners to find learning materials that meet their
needs based on the learners’ abilities. Another study
implementing collaborative filtering for
Good and Similar Learners’ Recommendation in Adaptive Learning Systems
435
recommending learning materials was carried out by
(Soonthornphisaj et al., 2006). This research also
proposed a mash-up technique to aggregate
recommended materials from several websites.
A combination of content-based filtering and
collaborative filtering can be found in (Liang et al.,
2012) who applied knowledge discovery techniques
to perform personalised recommendations for a
courseware selection module. On the other hand, to
compute relevant links for active users (Khribi et al.,
2008) used web mining to process the recent
navigation histories of learners combined with the
similarities and dissimilarities between user
preferences and the learning resources.
A further study exploiting collaborative filtering
and learners’ preferences was proposed by (Wang et
al., 2007). They suggested a personalised
recommendation mechanism based on content and
user similarity to choose learning materials out of a
large number of materials available on the web. They
combined two algorithms, a preference-based
algorithm and a correlation-based algorithm, to rank
the recommended results to advise a learner about the
most suitable learning objects. This model uses a
specific ontology of a certain course to infer objects
required for a learner. The inference is based on his
or her past studying history, which is recorded as the
learner’s personal preference pattern. Another
consideration in selecting learning objects is to refer
to the experiences of similar learners. The similarities
between learners can be inferred from similar values
for certain parameters.
Until recently, improvements for recommender
systems for learning materials have been made by
taking into account learners’ competence. In a
previous study (Tai et al., 2008), learners’
competence was used to retrieve relevant learning
materials from the web. A combination of
collaborative filtering and learners’ competence was
proposed by (Cazella et al., 2010). A learner’s
competence is relative to other learners’ competence
as it is assessed by comparing it to the average of all
the learners’ competence. On the other hand, Ghaut
in (Ghauth and Abdullah, 2011) proposed a
collaborative filtering method based on good
learners’ recommendations, rather than similar
learners’ recommendations.
3 THE FRAMEWORK
The method we propose is for learning material
recommendations. The recommender is part of the
adaptation engine in our proposed adaptive learning
architecture, as described in Figure 1. The difference
between such architecture and the conventional
adaptive learning systems lies in the existence of an
adaptation engine that produces recommendations for
topics or concepts to be learned next and relevant
learning materials.
There are two modules in the adaptation engine:
1. An adaptive navigation engine that decides which
topic or concept the learner will learn next. This
module applies the Bayesian network, but this is
not discussed in this paper.
2. A recommender system for learning materials.
Once the adaptive navigation engine recommends
a concept, the recommender starts working to
choose learning materials relevant to the concept
and the learner. The proposed collaborative
filtering technique is applied in this module.
Figure 1: Architecture of an adaptive learning system.
A domain model is important for the
recommender system as it contains learning concepts,
learning content (materials), tags, and ratings. The
concepts and learning content are designed by
teachers, while the tags and ratings are given by
teachers and learners. In Figure 2, tags are described
as hubs that link learning materials to concepts.
Before the recommendation begins, the teacher
has developed learning concepts and uploaded
learning materials in the domain model. Afterwards,
learners tag and rate the learning materials. All the
tags and rates are recorded in the domain model and
will be used for the recommendation process.
Once the active learner receives the next concept
to be learned, the recommender will run content-
based filtering to find all the learning materials tagged
with the concept. Furthermore, the recommender will
calculate its ratings by good learners and the
similarity between good learners and the active
learner. When a good learner has not rated a learning
material, the recommender will predict the rating
based on the similarity between the material and other
materials that have been rated by the good learner.
CSEDU 2016 - 8th International Conference on Computer Supported Education
436
Figure 2: Domain model.
We define good learners as those who show good
track records in all topics they have learned in the
course and they have mastered a given concept, that
is the concept being learned by the active learning.
The processes to measure and maintain learners’
achievements are parts of student modelling in the
adaptive learning system in which the recommender
resides.
The framework of the proposed recommendation
strategy is described in Figure 3. To conclude, the
better a good learner in mastering a concept and the
more similar she/he is with the active learner in rating
learning materials, the more contribution she/he has
in the recommendation process.
Figure 3: Recommender framework for learning materials
based on similar-good learners’ recommendations.
We define good learners as those who show good
track records in all topics they have learned on the
course and have mastered a given concept, that is, the
concept being learned during active learning. The
processes to measure and maintain learners’
achievements are part of student modelling in the
adaptive learning system in which the recommender
resides.
The framework for the proposed recommendation
strategy is described in Figure 3. To conclude, the
better a good learner is at mastering a concept and the
more similar he or she is to the active learner in rating
learning materials, the more contribution he or she
makes in the recommendation process.
4 THE RECOMMENDATION
MODEL
The proposed technique consists of three steps:
content-based filtering, collaborative filtering, and
recommendation score calculation. The proposed
technique.
4.1 Content-based Filtering
Content-based filtering is aimed at selecting learning
materials that meet the concept being learned by the
active learner. One concept can relate to a number of
learning materials and vice versa, as shown in Figure
2. The hub is created through collaborative tagging by
learners. In the first step, the weight of each tag in the
learning materials is calculated using the following
formula:
w
i,lm
=
|C|Max
|C|
x
|C|Max
|C|
i
lm
lm
lm
,i,i
(1)
where |C
i,lm
| is the frequency of tag C
i
on material lm,
which is similar to the number of learners tagging lm
with C, Max|C
lm
| is the maximum frequency among the
tags given to material lm, while Max|C
i
| is the maximum
frequency of concept C
i
put as a tag on all the materials.
The first step produces weights for all the tags in all the
learning material and it would be normalised by
comparing w
i,lm
to the maximum of the weights.
In the second step, the relevance scores (RS) of
learning materials are calculated. The higher the RS is,
the more relevant the learning material to the concept
being learned. The relevance of learning material lm is
calculated according to the following formula:
RS
lm
=
|w||w|
w.w
lmu
lmu
(2)
Good and Similar Learners’ Recommendation in Adaptive Learning Systems
437
Variable w
u
is a weight vector of learner u’s
competence. It is a dynamic vector of the learner
model, which is dynamically updated in adaptive
learning systems. Since this paper focuses on the
recommender, we do not discuss when and how the
vector is updated. Another vector counted in the
relevance score is w
lm
, a weight vector of tags given
to learning material lm. The second step produces L,
a set of materials that have the highest RS. At the end
of this process, a set of learning materials relevant to
the concept being learned by the active learner has
been defined. The kinds and number of tags are the
only parameters to determine the relevance or
irrelevance.
4.2 Collaborative Filtering based on
Recommendation from Similar
Good Learners
This part is the heart of the proposed collaborative
filtering technique. It aims to select good learners and
calculate the similarity scores between good learners
and the active learner based on ratings they have
given to learning materials. Good learners are those
who have mastered the concept being learned by the
active learner and consistently achieve well in all the
concepts they have learned so far. From the previous
parts, we have M, a set of learning materials with the
current learning context. We define G, a set of good
learners selected based on their overall competence
and their competence on the current topic being
learned by the active learner, al.
Once G has been defined, the similarity score
between the active learner and good learners will be
calculated. The similarity between learners is
identified from the ratings they have given to learning
materials. The similarity between the active learner,
al, and a good learner, gl, is calculated using the
following formula:
=
=
=
|M|
Mm
2
1gm,1g
2
alm,al
|M|
Mm
1gm,1galm,al
1i
ii
1i
ii
)rr()rr(
)rr)(rr(
)1g,al(sim
(3)
where
r
al,mi
is the rating of material mi given by active
learner, al. Furthermore, r
gl,mi
is the rating of material
mi given by good learner, g1. The formula also
applies the average of the ratings given by active
learner, al,
al
r
, and the average of the ratings given
by active learner, g1,
1g
r
. At the end of this module,
the similarity scores between the active learner and
each good learner will have been defined.
4.3 Recommendation Score Calculation
The final stage of the recommendation process is the
calculation of the recommendation scores. The
recommendation score of learning material lm is
calculated by considering the similarity between good
learners and active learners, and the ratings given by
good learners for lm. The formula is described as
follows:
=
=
=
|G|
1i
i
|G|
1i
lm.gi
lm
)g,al(sim
rating*)g,al(sim
R
i
(4)
where sim(al,g
i
) is the similarity score between the
active learner, al, and good learner, g
i
. On the other
hand, rating
gi
,
lm
is the rating for the learning material
lm given by the good learner, g
i
(equation 5). The
value of sim(al,g
i
) is set at 1 when the active learner
has not rated any learning material. In case a good
learner g has not rated learning material lm, then a
rating prediction will be calculated (equation 6).
|G|
rating
R
|G|
1i
lm.g
lm
i
=
=
(5)
=
=
=
|N|
1i
i
|M|
1i
mi,gi
lm,g
)m,lm(sim
rating*)m,lm(sim
P
(6)
where rating
gi
,
mi
is the rating of learning material mi
given by good learner g
i
and sim(lm,m
i
) is the
similarity score of the learning material lm and m
i
,
which is given by:
sim(lm,m
i
) =
|w||w|
w.w
milm
milm
(7)
At the end of this stage, a set of learning materials
with their recommendation scores has been defined.
5 IMPLEMENTATION AND
TESTING
To test the proposed method, an experiment is
designed. For this experiment, we design 100
documents for 10 topics in a programming course,
including variables and data types, expressions and
assignments, case analysis, functions, recursion,
CSEDU 2016 - 8th International Conference on Computer Supported Education
438
loops, array, searching, sorting, and a matrix in the
forms of slides or short articles. Teachers have given
some tags to the learning materials and afterwards the
participants have been invited to tag and give ratings.
We invite 171 undergraduate students to
participate in the experiment. They are students who
are registering on the programming course or have
completed the course. The experiment consists of
various steps. Firstly, we need to recognise which are
the good learners among them. As we have discussed
previously, good or other types of learners are
recognised in adaptive learning systems by
processing learner models. However, as this research
is focused on the recommender system itself, we
conducted a pre-test covering the aforementioned 10
programming topics, two problems for each topic. At
the end of this activity, each participant has a
competence model that records his or her competence
with respect to the 10 topics examined. The status for
each topic consists of good mastering or 1 if they
could correctly solve both given problems, half
mastering or 0.5 if they correctly solve one of the
given problems, and need to learn, or 0 if none of the
given problems can be solved. Good learners are
those who pass a threshold, for example 0.75, for the
mean score of mastered and half-mastered topics.
Secondly, 51 of the participants are invited to
participate in testing the recommender. They are
requested to rate and tag learning materials for topics
they have mastered or half-mastered. Then, the
recommender testing is carried out for a topic they
have not yet mastered. As we have previously
discussed, the recommender of learning materials, as
part of adaptive learning systems, should receive
input from the adaptive navigation engine in the form
of a topic where the active learner can go next. As we
focus on the recommender, we simulate the input
process with a query asking for a topic input from the
active learner. Then the recommender suggests
learning materials discussing the topic and they have
not been rated by the learner, along with the
predictive scores. The participants are requested to
input ratings for the recommended materials on rating
scale up to 10.
We use the mean absolute error (MAE). MAE
represents the deviation between the predicted ratings
and the user-given ratings.
MAE =
=
N
1i
ii
N
|rp|
(8)
The recent experiment produces an MAE score of
0.96 for the rating scale up to 10, which means that
the predicted and user-given ratings are not
significantly different. Compared to the standard for
the MAE score concluded from previous studies,
which is 0.73 for a rating scale up to 5, the proposed
algorithm shows better performance. The MAE score
will possibly change since more students will
participate in the experiment.
6 CONCLUSIONS
In this paper we have discussed our proposed
recommendation technique, which combines content-
based filtering and collaborative filtering, which
considers learners’ competence and similarities in
rating learning materials. The consideration of good
learners’ recommendations is inspired by the
existence of helpers in peer learning. The technique
aims to improve the suitability of the
recommendations with respect to learners’ needs. The
recent experiment to test the proposed technique
results in a low MAE score, 0.96 on rating scale up to
10. In comparison with the standard MAE score from
previous studies, which is 0.73 on a rating scale up to
5, the MAE score of the proposed method is relatively
low. The experiment is now still running so that more
participants can be invited to participate.
Following the results presented in this paper, there
is some work to carry out in the near future. Firstly,
since this recommender is part of adaptive learning
systems, this recommendation technique can be
improved by enhancing the method for identifying
good learners. This could be achieved by enhancing the
number of problems that learners have to solve in each
topic or using different types of problems, whereby
good learners can be more precisely identified.
Secondly, the collaborative filtering can be
extended by considering the similarities between
learners’ characteristics, for instance learners’
competence, in addition to similarities between rating
learning materials. By considering the learning
method, and various dimensions of learners’
characteristics, the recommendations are expected to
more precisely meet learners’ needs.
REFERENCES
Cazella, S. C., Reategui, E. B. & Behar, P., 2010.
Recommendation of Learning Objects Applying
Collaborative Filtering and Competencies. In the
proceeding of IFIP Advances in Information and
Communication Technology, pp. 35-43. Springer.
Chen, C. M., Lee, H. M. & Chen, Y. H., 2005. Personalized
E-Learning System Using Item Repository Theory.
Computers and Education, vol. 44, pp. 237-255.
Good and Similar Learners’ Recommendation in Adaptive Learning Systems
439
Ghauth, K. I. & Abdullah, N. A., 2011. The Effect of
Incorporating Good Learners’ Ratings in e-Learning
Recommender System. Educational Society and
Technology, vol. 14, pp. 248-257.
Indrayadi, F. & Nurjanah, D., 2015. Combining Learner's
Preference and Similar Peers' Experience in Adaptive
Learning. In the proceeding of The 7th International
Conference on Computer-Supported Education, pp.
486-493. ScitePress.
Khribi, M. K., Jemni, M. & Nasraoui, O., 2008. Automatic
Recommendations for E-Learning Personalization
Based on Web Usage Mining Techniques and
Information Retrieval. In the proceeding of Advanced
Learning Technologies, 2008. ICALT '08. Eighth IEEE
International Conference on, pp. 241-245.
Knutov, E., Bra, P. D. & Pechenizkiy, M., 2009. AH 12
Years Later: A Comprehensive Survey of Adaptive
Hypermedia Methods and Techniques. New Review of
Hypermedia and Multimedia vol. 15, pp. 5-38.
Liang, M., Guerra, J. & Brusilovsky, P., 2012. Building
multi-layer social knowledge maps with Google Maps
API. In the proceeding of Workshop on Semantic and
Adaptive Social Web (SASWeb 2012).
Lops, P., Gemmis, M. D. & Semerano, G., 2011. Content-
Based Recommender Systems: State of the Art and
Trends. In Recommender Systems Handbook, pp. 73-
105. Springer.
Lu, J., 2004. A Personalized E-Learning Material
Recommender System. In the proceeding of The 2nd
International Conference on Information Technology
for Application (ICITA 2004), pp. 374-379.
Mcleod, S., 2007. Vygotsky. From http://www.simplypsyc
hology.org/vygotsky.html. Accessed on 16 June, 2012.
Sicilia, M. A., Garcia-Barriocanal, E., Salvador Sanchez
Alonso & Cechinel, C., 2010. Exploring User-Based
Recommender Results in Large Learning Object
Repositories: The Case of MERLOT. In the proceeding
of 1st Workshop on Recommender Systems for
Technology Enhanced Learning (RecSysTEL 2010), pp.
2859–2864.
Soonthornphisaj, N., Rojsattarat, E. & Yim-Ngam, S.,
2006. Smart e-learning using recommender system. In
the proceeding of The 2006 international conference on
Intelligent computing: Part II, pp. 518-523. Springer-
Verlag.
Tai, D. W. S., Wu, H. J. & Li, P. H., 2008. Effective e
learning recommendation system based on self
organizing maps and association mining. The
Electronic Library, vol. 26, pp. 329-344.
Topping, K. J., 2005. Trends in Peer Learning. Educational
Psychology, vol. 25, pp. 631-645.
Verbert, K., Manouselis, N., Ochoa, X., Wolpers, M.,
Drachsler, H., Bosnic, I. & Duval, E., 2012. Context-
Aware Recommender Systems for Learning: A Survey
and Future Challenges. IEEE Transaction on Learning
Technology, vol. 5, pp. 318-335.
Vygotsky, L., 1978. Mind in Society. 1st edition. USA:
Harvard University Press. As cited by Mcleod, S.
(2007). Vygotsky. http://www.simplypsychology.org/v
ygotsky.html. Accessed on June 16, 2012.
Wang, T. I., Tsai, K. H., Lee, M. C. & Chiu, T. K., 2007.
Personalized Learning Objects Recommendation Based
on the Semantic-Aware Discovery and the Learner
Preference Pattern. Educational Technology and
Society, vol. 10, pp. 84-105.
CSEDU 2016 - 8th International Conference on Computer Supported Education
440