ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES
AND STUDENT KNOWLEDGE LEVEL
Boyan Bontchev and Dessislava Vassileva
Department of Software Engineering, Sofia University, 5, J. Baurchier blv., Sofia 1164, Bulgaria
Keywords: Adaptive assessment, Adaptive hypermedia systems, Learning styles, Learning object, Metadata,
Storyboard.
Abstract: Traditionally, adaptive assessment methods and tools have been addressed only by application of
computerized adaptive testing and item response theory as a key instrument for practical construction of
adaptive test assessments. The present paper tries to give a broader view of adaptive assessment, where tests
are not the only instrument for evaluation of learners outcomes gained during a course. It shows how
adaptive task-based assessment may be combined with traditional adaptive test assessment for achieving
better results and higher student satisfaction. Various types of tasks have been found suitable for adaptive
assessment based on learning styles and student knowledge level. By means of constructing course
storyboard with several branches for different learning styles, it has been proven that games, essay,
observation, comparative analysis tasks, projects and auto-generated tests may be used successfully for a
complex adaptive assessment. There have been explored approaches such as self-, peer- and teacher- (i.e.,
host) assessment by using appropriate types of estimable learning objects.
1 INTRODUCTION
Adaptive Web based learning plays a key role in
modern technology-enhanced learning research and
practical system implementation. First Brusilovsky
has pointed out (Brusilovsky, 1996) that learners
have different knowledge levels, needs and
preferences which should be considered for realising
personalized and adaptive hypermedia systems.
Moreover, learning styles have been used as a base
for creation of efficient adaptive hypermedia
systems (AHS) in many approaches (Milošević et al,
2007; Velsen, 2008). Learning style adaptivity may
be realised either by automatic generation of work
paths for each student character or, more precisely,
by construction of learning storyboards according
chosen instructional strategies (Vassileva et al,
2009a). In general, it has been proven in practice
that adaptation focussed on different aspects of
learning character is able to provide a higher
appealing and level of usefulness for the learner and,
thus, to lead to a better learning learning process
(Paramythis and Reisinger, 2003; Grimón et al,
2009).
Assessment is of key importance for both the
teacher and the learner because it tracks the learning
process, identifies strengths and weaknesses of the
learner and, thus, helps teacher in planning learning
steps. Introducing adaptivity to traditional
assessment may provide benefits not less then these
typical for AHS. However, until present, adaptive
assessment did not necessarily make a part of
adaptive hypermedia systems. Most of the present
works on adaptive assessment are focussed on
intelligent selection of questions based on prior
knowledge of the learner shown during a test run.
Thus, adaptive assessment tools use knowledge level
and proficiency of learners to select next question
within the test (Gouli et al, 2001). Testing is adapted
interactively to order to match the ability level of
any individual learner (Mansoor, 2006) by means of
using item response theory (IRT). En efficient
computer adaptive testing (CAT) system uses a
calibrated bank of questions developed according an
IRT model (Kovatcheva and Nikolov, 2009), initial
and intermediate questions' selection methods, a
scoring method, and terminating rules to stop (Gouli,
2001). For an intelligent selection of item (i.e.,
question) within an adaptive assessment run, proper
metadata of LOs of type questions has to be used. In
the approach of (Ibraheem, 2003), item metadata is
divided into two parts - descriptive and
psychometric.
449
Bontchev B. and Vassileva D..
ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES AND STUDENT KNOWLEDGE LEVEL.
DOI: 10.5220/0003484104490454
In Proceedings of the 3rd International Conference on Computer Supported Education (ATTeL-2011), pages 449-454
ISBN: 978-989-8425-50-8
Copyright
c
2011 SCITEPRESS (Science and Technology Publications, Lda.)
The present paper describes a practical approach
of realization of assessment adaptive to both
learning styles and learner’s knowledge level and
achieved within the scope of a platform for adaptive
content delivery based on learner both character and
knowledge. An adaptation control engine chooses
within a course storyboard appropriate work paths
best suited to given learner style. In the course of the
work path, the engine shows to the learner only
learning objects (LO) having a difficulty appropriate
to the individual learner, and tracks these objects. At
a control page, the engine selects questions related to
the learning objects shown to the learner in an
adaptive mode. Besides adaptive tests, the approach
includes a more general adaptive assessment based
on learning styles - it includes assessment of
estimable learning objects such as essays, projects,
single- and multi-user games, comparative analysis
and others – all appropriate to a given learning style.
Experimental results are obtained for applying
adaptivity for Honey and Mumford (Honey and
Mumford, 2000) styles comprising of theorist,
pragmatist, reflector and activist. Authors argue the
appropriateness of such tasks to given learning style
and assessment method, such as self-, peer- and
teacher assessment.
2 ADOPTA MODEL
AND PLATFORM
The assessment adaptive described in this paper is
implemented with the ADOPTA (ADaptive
technOlogy-enhanced Platform for eduTAinment)
platform for building edutainment (education plus
entertainment) content for both universities and
industry implementation. The triangular model of
ADOPTA is based principally on AHAM reference
model (De Bra et al, 1999) and its main idea is
explicitly separating narrative storyboard from the
content and adaptation engine. It adds some features
of contemporary adaptive e-learning systems as
support of different learning styles, learning content
and pedagogical strategy metadata and support of
several standards and more specifically LOM (Krull,
2004), SCORM (Rey-López et al, 2006) and OMV
(Vassileva et al, 2009).
The triangular model has hierarchical structure
with two levels (fig. 1). At each one level, it consists
of three sub-models. Thus its first level is divided
into the following three main models:
Learner model – it structures data for learners
and it is divided into three sub-models - Goals and
preferences, Learning styles and Knowledge and
Performance. They provide data for learners, which
are used from adaptive engine for adaptive content
delivery.
Adaptation model - it consists of following three
sub-models: Narrative metadata, Narrative
storyboard, and Storyboard rules. The sub-model
Narrative storyboard described course storyboard
graphs through directed graph. Each node of this
graph is narrative page or control page (CP).
Respectively narrative pages contain listed learning
object defined in different ontology graphs of the
Domain model, but CPs consist of randomly chosen
test questions which are based on visited learning
object of students. Each path from one CP to another
CP is called working path (WP). Each WP is
associated with a weight for each one learning
styles. These WP weights present how much they
are suitable for a particular learning style. The
metadata of narrative storyboard graphs are
described in the sub-model Narrative metadata. It
includes for example thresholds for each WP. These
thresholds gives the minimal results of test in CP
where the learner may continue to the next CP.
Rules for passing through narrative storyboard
graphs are stored in the sub-model Storyboard rules.
These rules define formulas which adaptive engine
used to calculate the most suitable WP for a learner
and conditions under which a learning object is
visible for a particular student.
Domain model – it is responsible for creating of
learning objects (through the sub-model Learning
objects), for structuring it in ontology graphs
(through the sub-model Ontology graph) and
description of ontologies and learning objects
metadata (through the sub-models Content
Metadata). The sub-model Learning objects
determines several types of learning objects such as
narrative content, projects, tasks, essays, games,
assessment questions. Each of them has level of
difficulty (easy, medium, difficult, more difficult,
and most difficult). Using this, the instructor of a
course defines for what test results in CP are visible
various in difficulty learning objects. The other sub-
model Ontology graph organized learning object in
ontology graphs. In these graphs learning objects are
connected between themselves by two types of links:
o is-a – it is used for connection between
learning objects of type narrative content
o has-a – it defined reference relations and
learning object of type projects, tasks, essays,
games, assessment questions can be associated with
one and more learning objects of type narrative
content.
CSEDU 2011 - 3rd International Conference on Computer Supported Education
450
Figure 1: Principal model and system workflow.
The architecture of ADOPTA system consists of
three main modules (fig. 1) – Authoring Tool,
Instructor Tool and Adaptation Control Engine.
The Authoring Tool is responsible for
implementation of functionalities of the Domain
model. It is used by content author for creating of
learning, for designing of ontology graphs and
describing of theirs metadata. For the last, this
module applies an inheritance mechanism.
After the learning content is created by the
content author, the instructor uses the Instructor
Tool for construction and for tuning of different
parameters of narrative storyboard and pages.
Finally, the Adaptation Control Engine uses all
available data for learning objects and storyboard
graphs for adaptive content delivery and adaptive
assessment. It can be used by instructor for
monitoring of the learning process, for controlling it
– adaptation behavior can be start or stop, for
correction of WP weights, for analysis of learner
performance, etc.
Each module includes three layers- persistence,
business and web (or client) layer. Each of the layers
is responsible for different specific problems. The
persistence layer stores and edits objects and it is
implemented with the Java Persistence API. The
business layer is build by EJB technology. The
business logic of application is presented by stateless
EJBs.
Finally the last web layer communicates with the
beans where the business logic resides with web
services and it is build with FLEX technology.
3 ADAPTIVE ASSESSMENTTO
LEARNING STYLES AND
KLOWLEDGE LEVEL
Realization of adaptive assessment within the
ADOPTA platform is a joint effort of content
authors, instructors and course supervisors. The
workflow phases are authoring of content for
adaptive courses, construction and tuning of
storyboard adaptive to learning styles and student
knowledge level and, finally, adaptive content
Figure 2: Partial view of ontology of the ADOPTA ontology authoring tool with LOs of a XML technology course.
ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES AND STUDENT KNOWLEDGE LEVEL
451
delivery and assessment by means of estimable LOs
such as essays, projects, test, etc.
3.1 Content Authoring
Content authoring is possible by means of special
authoring tool providing LO metadata support based
on inheritance (Vassileva et al, 2009). Here, the
author should provide many LOs of various types
such as narrative LO (lesson), exercise, project,
essay, problem solving and others, in order to be
used next by instructors when creating course
storyboard graphs by means of the instructor tool.
The author organizes LOs within a domain
ontology having two main types of relationships:
IS_A and HAS_A. Fig. 2 shows a partial view of
such an ontology for a course of XML technologies
given to bachelor students. Questions are special
types of learning objects in the ontology and may be
related to the narrative LO they concern. While
IS_A relations are used for typefication of most LOs
of the course, HAS_A relations link question LOs to
the objects they refer to. Like the approach of (Wen,
2007), we use for types of relations “Enable”,
“Disable”, “Plus” and “Contradictory”. As well,
there may be used another types, as follows:
Complementary – relation of such a type links
one question to other questions which are
complementary to the first one; means that one of
complementary questions may be used after the first
one;
Opposite – links one question to another being
opposite of the first; means that these two questions
are adversative.
As well, for each of the LOs of type question the
author is supposed to provide item metadata divided
into two parts - descriptive and psychometric as
specified in (Ibraheem, 2003). For an easy
manipulation of LO metadata, non-monotonous
inheritance support is provided meaning metadata is
inherited within the ontology while some of the
fields may be overridden or added for given LO.
3.2 Course Storyboarding
While designing storyboard graphs by means of the
ADOPTA instructor tool, the instructor is supposed
to be concerned about two issues:
development and tuning of a storyboard graph
containing paths with pages containing LOs
appropriate for different learning styles
selection and distribution among the graph pages
of LOs with different level of difficulty in order to
be delivered next to learners with appropriate
knowledge level.
Figure 3: Partial view of XML course ontology.
The first task is not so trivial, as far as there is no
existing stable mapping between LOs types and
learning styles. Fig. 3 presents a possible mapping of
types of LOs to the four basic learning styles of
Honey and Mumford resulted from practical
experience of the paper authors in teaching students
for many years. It has been found that games,
essays, projects, problem-solving, comparative
analysis and observation tasks may be used for
assessment not less efficiently than traditional tests.
Thanks to their appropriateness to different learning
styles, LOs of these types are selected as candidates
for assessment adaptive to learning styles and have
names shown in black letters while narrative LOs
have names in white labels. For example, games and
problem-solving tasks are suitable for assessment of
both pragmatists and activists, while comparative
analysis tasks are mostly suitable to theorists and
reflectors. Note, that these types are distinguished
according their suitability for one or more of the
assessment approaches mentioned in section 2:
for self assessment – there are nominated single
user games and test (auto-generated based on the
ontology relationships);
for peer assessment – there are used multi-user
games (the opponent of a learner appears as peer
assessor), essay, observation, and comparative
analysis tasks;
for teacher assessment - essay, observation, and
comparative analysis tasks plus projects and tests.
CSEDU 2011 - 3rd International Conference on Computer Supported Education
452
At the very beginning of the storyboard design
process, the instructor is guided only by possible
learning styles of students. For this purpose, the
storyboard graph has to have a few initial nodes or
to start from a node giving basic information and,
next, to fork the working (i.e., learning) paths (WPs)
within the graph designed by the instructor tool. For
construction of the adaptive course storyboard
graph, the instructor may follow a strongly
connected storyboard approach or graph design
using parallel branches. Educational content with the
branch for predominant activists contains more
examples, while materials for theorists are given
with more detailed explanations and formalization
within another branch. LOs intended for pragmatists
are presented by practical tasks and exercises, while
these for reflectors are reporting, analysis and
comparisons.
Finally, the instructor has to tune the weights for
the existing working paths. Each weight consists of
values for activist, theorist, reflector and pragmatist,
stating the level of suitability of the path for each
style. As well, the instructor has to select LOs of
various level of difficulty in order to meet different
student knowledge levels.
3.3 Adaptive Content Delivery
and Assessment
Before starting a course, individual learners pass
through a pre-test determining their learning
character. Usually, a learning character contains a
combination of the four learning styles expressed in
different degree by different weights. Next, the
adaptation control engine will select a path best
suited to individual learner character of given
learner. According distribution of LOs types to
learning styles, different students will receive
different narrative and estimable LOs. Estimable
LOs will be used for a total adaptive assessment
where students with different styles will receive
different tasks (essays, projects, problems to be
solved, etc.) as shown in fig. 3. There different tasks
will be used for a complex final grade of the course.
Of course, adaptive assessment tests do participate
into the final grade – in control pages the control
engine generates automatically adaptive tests by
selecting questions related to the LOs delivered to a
learner via the work path. The selection is based on
the relation types explained in section 4.1.
After solving the test at a control point, the
learner is passed back to the previous control point if
his/her result is less than the threshold set for
passing this control point. In this case, the adaptation
control engine will select a different path for this
learner. Otherwise, to the learner there is proposed a
new path leading to the next control point and most
appropriate to his/her learning style. The result of
the assessment (equal or greater the threshold set for
passing that control point) will be used by the engine
for selection of LOs with appropriate level of
difficulty. Thus, individual learners undergo
assessment process adaptive to their learning styles
and, as well, knowledge level.
4 PRACTICAL RESULTS
The ontology, LOs, and storyboard shown above in
this paper have been used for preparing a part of an
adaptive course in the domain of XML technologies.
This part of the course has been given to more than
70 bachelor students (fourth year) by means of
ADOPTA as both adaptive courseware delivery and
adaptive assessment, while the rest of the course has
been presented as a traditional non-adaptive course.
Students have passed separate assessments for both
the adaptive and non-adaptive part of the course.
Next, they have filled an inquiry with many
questions aiming at revealing their attitude to the
adaptive and non-adaptive e-learning approaches.
Having summarized the inquiries and assessment
results, we are able to conclude the following:
Students have shown better results when dealing
with adaptive content delivery and testing than with
non-adaptive ones;
Students do prefer adaptive content delivery and
assessment based on learning styles and knowledge
level than traditional methods;
All the learners like much more complex
assessment based on various types of tasks given
adaptively according learning styles, than traditional
test-based assessment;
Students do prefer having self- and peer
assessment before official assessments;
Students appreciate very much inclusion of
games - both single-user such as hangman, quizzes,
and word puzzles and multi-user games within the
ADOPTA platform
Students require mobile games, tasks and
assessments to be included into the adaptive e-
learning process.
ADAPTIVE ASSESSMENT BASED ON LEARNING STYLES AND STUDENT KNOWLEDGE LEVEL
453
5 CONCLUSIONS
Until present, researchers and practitioners regard
adaptive assessment mainly as application of
computerized adaptive testing. Such systems make
use of item response theory as a key instrument for
practical construction of adaptive assessments based
on adaptive tests. The present paper presented a
broader view of adaptive assessment, where tests are
not the only (and, eventually, the best) instrument
for evaluation of learners outcomes gained during a
course. It has shown that adaptive task-based
assessment may be combined with traditional
adaptive test assessment for achieving better results
and, not last, student satisfaction and approval.
Authors have explored suitability of various
types of tasks for assessment adaptive to learning
styles and student knowledge level. By means of
constructing course storyboard with several branches
for different learning styles, it has been proven that
games, essay, observation, and comparative analysis
tasks plus projects and auto-generated tests may be
used successfully for a complex adaptive
assessment. There have been explored the
approaches of self-, peer- and teacher- (i.e., host)
assessment by using appropriate types of estimable
learning objects.
The future works on adaptive assessment by
using the ADOPTA platform will explore the
abilities of more types of relationships among
questions for realization of adaptive test assessment.
As well, authors plan to develop and experiment
more types of games for both self- and peer
assessment.
ACKNOWLEDGEMENTS
This work is partially supported by both the SISTER
project funded by the EC in FP7-SP4 Capacities via
agreement no. 205030 and the ADOPTA funded by
the Bulgarian NSF under agreement no. D002/155.
REFERENCES
Brusilovsky, P., 1996. Methods and Techniques of
Adaptive Hypermedia. in User Modeling and User-
Adapted Interaction, Vol. 6, No. 2-3, pp. 87-129.
De Bra P. et al., 1999. AHAM: A Dexter-based Reference
Model for adaptive Hypermedia. Proc. of the ACM
Conference on Hypertext and Hypermedia, ISBN:1-
58113-064-3, pp. 147-156.
Gouli, E., Kornilakis, H., Papanikolaou, K., Grigoriadou,
M., 2001. Adaptive Assessment Improving Interaction
in an Educational Hypermedia System, Proc. of the
PanHellenic Conf. in Human-Computer Interaction,
pp.217-222.
Grimón, F., Monguet, J. M., Ojeda, J., 2009. Knowledge
Based Information Retrieval with an Adaptive
Hypermedia System, in LNCS, ISBN 978-3-642-
02263-0, pp. 457-463.
Honey, P., Mumford, A., 2000. The Learning Styles
Helper's Guide. Maidenhead, Peter Honey
Publications.
Ibraheem, A., M., Shaalan, K, Riad, M. B., Darwish, M.
G., 2003, A Model and Supporting Mechanism for
Item Evaluation in Distance Learning-Based
Environment, in Egyptian Information Journal, Vol.4,
No. 2, December 2003, pp 169-186.
Kovatcheva, E., Nikolov, R., 2009. An adaptive feedback
approach for e-learning systems, in IEEE
Multidisciplinary Eng. Education Magazine, Vol. 4,
No. 1/2, JUNE 2009, pp.55-57.
Krull, G (2004). An investigation of the development and
adoption of educational metadata standards for the
widespread use of learning objects, Master Thesis,
Rhodes University, November 2004.
Mansoor A., 2006. IRT-item response theory assessment
for an adaptive teaching assessment system, Proc. of
MATH'06, Stevens Point, Wisconsin, USA, ISBN:999-
6666-22-1, pp.518-522.
Milošević, D. et al., 2007. Adaptive Learning by Using
SCOs Metadata, in Interdisciplinary Journal of
Knowledge and Learning Objects, Vol. 3, pp. 163-
174.
Paramythis, A. Reisinger, L., 2003. S Adaptive Learning
Environments and e-Learning Standards, Proc. of
Conference on e-Learning (ECEL2003), Glasgow,
Scotland, pp. 369-379.
Petrov M., Bontchev, B., 2004. Question and test
interoperability – organization, reuse and management
of assessments’ contents, Proc. of IADAT Int. Conf. on
Education – IADATe-2004, ISBN: 84-933971-0-5,
Bilbao, Spain, pp.435-439.
Rey-López, M., Fernández-Vilas A., Díaz-Redondo R.,
Pazos-Arias J., 2006. Providing SCORM with
adaptivity. Proc. of the 15
th
Int. Conf. on World Wide
Web, ISBN:1-59593-323-9, pp.981-982.
Vassileva D., B. Bontchev, Sl. Grigorov, 2009. Mastering
Adaptive Hypermedia Courseware, in Acta
Electrotechnica et Informatica, Vol. 9, No. 1, pp. 57–
62.
Velsen., L., 2008. User-centered evaluation of adaptive
and adaptable systems: a literature review, in The
Knowledge Engineering Review Journal, Cambridge
University Press, Vol. 23:3, pp. 261–281.
Wen, D., Graf, S., Lan, C. H., Anderson, T., Kinshuk, K.
D., 2007. Adaptive Assessment in Web-based
learning, Proc. of the IEEE Int. Conf. on Multimedia
and Expo (ICME'2007), Beijing, China, July 2007, pp.
1846-1849.
CSEDU 2011 - 3rd International Conference on Computer Supported Education
454