General issues for a decision support system
Erla Morales
Departament of Theory and History of Education, Salamanca University, Canalejas Street 169, Salamanca, Spain
Francisco García
Department of Computer Science, Salamanca University, Merced Square, s/n, Salamanca,Spain
Keywords: Knowledge management, decisio
n support systems, learning objects
Abstract: In today’s world, reusable learning object concepts and standards for their treatment represent an advantage
for knowledge management systems to whatever kind of business that supports an on-line system. Users are
able to manage and reuse content according to their needs without interoperability problems. The possibility
of importing learning objects for e-learning aim to increase their information repository but the learning
object quality is not guaranteed. This work proposes a system to manage quality learning objects to support
teachers to select the best content to structure their course. To achieve this we suggest two subsystems:
First, an importation, normalization and evaluation subsystem; and second, a selection, delivery and post
evaluation subsystem.
Through an e-learning repository we can find a
myriad of content from academic research and
contributions, but how about the content’s quality?
Without doubt, an important contribution from
puter science to knowledge management and e-
learning systems is the learning object (LO) concept.
This element has characteristics of independent
units, which are able to be reused for other
educational situations. According to this, knowledge
management for e-learning based on reusable LOs
means the possibility to access specific content
according to the learners’ needs.
The stage mentioned above is possible due to
andards, which were established as an attempt to
avoid interoperability platform problems. Thanks to
reusable LOs and standards for their treatment,
knowledge management becomes more easy and
efficient but it doesn’t guarantee the content quality.
A great quantity of criteria exists about digital
earning sources evaluation. Nevertheless, for LO
content evaluation there are just a few proposals
which consider their characteristics. So it is
necessary for a knowledge management system to
frequently re-feed the content for an e-learning
repository together with the teacher’s expert
knowledge and the student’s learning experience. In
this case we are focused on teacher’s expert
On this basis, section 2, presents general issues
r LOs management. Section 3 presents the main
elements for systems to support decisions.
Subsection 3.1, presents a subsystem to import LOs
from external sources, followed by our
recommendation to normalize them according to a
knowledge model and finally to evaluate them
through an instrument and collaborative strategy.
Subsection 3.2 explains another subsystem that
supports decisions and content re-feed. Finally,
section 4 summarizes conclusions and further work.
The knowledge management we suggest is based on
reusable LOs. The most widespread definition is
from (IEEE LOM, 2002) that states the “digital or
non-digital entity that may be used, reused or
referenced while the learning receives technical
Morales E. and García F. (2005).
QUALITY CONTENT MANAGEMENT FOR E-LEARNING - General issues for a decision support system.
In Proceedings of the Seventh International Conference on Enterprise Information Systems, pages 343-346
DOI: 10.5220/0002524003430346
However, this concept is too broad to guarantee
an efficient resources management. We believe LOs
should represent at least a single instructional
objective and all of the related materials required to
support that objective
For this reason we propose a definition adapted
by (Polsani, 2003) “A Learning Object is an
independent and self-standing unit of learning
content that is predisposed to reuse in multiple
instructional contexts”, on this basis we refer to
learning objects which contain a unit of learning
with educational sense, for example a lesson.
According to LOs and standards capabilities, it is
necessary to consider how to manage quality LOs,
taking into account their characteristics to help
teachers to structure their courses. For this reason we
suggest a system to support decisions about how to
select the best content from a LOs repository. Figure
1 presents a general view about the system we
propose which will be explained in the next sections.
3.1 Importation, normalization and
evaluation subsystem
A first step we must consider is to import LOs
content to the e-learning repository. According to
this, for knowledge management it is necessary to
take into account some questions about content such
as what, how and why should it be managed
(Kuang-Tsae, 2000). Taking these issues into
consideration, imported LOs may be selected with
regard to context issues (Marquès, 2001). On this
basis, some keywords may be used for searching a
suitable LO.
After that, the imported LOs can be saved into a
non–normalized content repository, because each
one of them could has a different granularity level
than others ones. Therefore, to import LOs from
external sources the second step we suggest is to
normalize imported LO’s according to a knowledge
model. To achieve this, we suggest the next steps to
normalize LO’s.
1.- Clasiffy LO’s objectives according to their
complexity level, because this way it is easier
knowing if the LO is suitable for new educational
situations. Then we suggest Bloom’s cognitive
domain taxonomy (Bloom, 1956) because it has
been widely used in e-learning to define cognitive
objectives and also it divides the objectives into high
and low complexity levels.
2.- Define the difficulty level to each one of LO, for
this ussue we propose three kinds of complexity
levels: basic, medium and advanced because this
kind of clasiffication would help teachers to select
the LO content according to their teaching
3.- Classify the imported LO into three kind of
content areas: data and concept, procedure or
processes, and reflection or attitude. This
classification aims to define the kind of content
according to the learning objectives.
4.- Classify the imported LO into three kind of
activities: Initiation, Re-structuring and Application.
Initiation activities classification may be for all LOs,
which are designed to teach basic content for a
specific subject. Restructuring activities
classification may be directed to promote new
knowledge acquisition. Finally, applying
classification activities may be directed to promote
students’ experience in order to achieve their new
e mana
Teacher(s) Experts Teacher(s) Students
1.Import UL
2. Normalization
(knowledge model)
Non normalized
3. Evaluation
4. Selection
(quality LO’s)
5.Delivery courses
(e-learning system)
6.Student’s evaluation
and evaluated
normalization, and
evaluation subsystem
delivery and post
evaluation subsystem
ure 1: A decision su
ort s
concepts acquisition.
Nevertheless, the classification of the LO
according to a knowledge model like this is not
enough to guarantee the LO quality.
There exists is a plethora of quality criteria to
value digital sources but there are only a few
proposals about how to evaluate LOs
In order to achieve an optimal evaluation of the
LOs, it is necessary on one hand considering quality
criteria from different kinds of categories to each
one of LO, and on the other hand, considering the
LOs evaluation models (Merlot, 2003; Vargo et al.,
2003, Williams, 2000). In this way, it is possible to
consider different points of view with regard to the
same object. According to this we suggest a third
step for knowledge management to support
decisions, it is an instrument which considers
different evaluation criteria in four categories.
Psychopedagogical category (30%): This kind
of criteria aims to determine if the LO is suitable to
promote learning, for example, learner’s motivation.
Didactic-curricular category (30%): This kind
of criteria aims to evaluate if an object is related to
curricular objectives according to the context in
which it will be applied.
Technical- aesthetic category (20%):
Technical-aesthetic criteria aim to evaluate issues
like legibility, color-contrast, etc.
Functional category (20%): It aim to evaluate
if an object work correctly and doesn’t obstruct the
learning process.
From the stages mentioned above, the
psychopedagogical and didactic-curricular
categories are more important than technical-
aesthetics and functional categories within the
educational context, then, we do not propose
evaluating them with the same score weighting. We
suggest evaluating each object with the same rating
scale but applying a different percent.
For getting the final result, we propose
calculating the average score gained for each object
according to the percent weighted for each category
with the following rating scale: 0 = Criteria is not
present; 1 = Very low; 2 = Low; 3 = Medium, 4 =
High, 5 = Very high.
Due to the fact that an optimal LO evaluation
considers criteria from different kinds of categories,
we suggest the participation of different kinds of
experts during the evaluation, for example:
instructional designers, subject experts, and so on.
The participation of at least one participant from
each area encourages not only different points of
view over the subject under evaluation, but also a
critical objectivity and a reliable LO evaluation.
We propose two modes of applying the
instrument suggested above in order to value the
LO: individual and collaborative method.
According to this concept, individual evaluation
provides us with an initial appreciation of the quality
of the LO based on the judgment of each participant.
For making easier this evaluation firstly we
propose the possibility to view the LO Metadata
(IMS LOM, 2003) through the e-learning platform.
It allows to the evaluators knowing quickly LOs
characteristics. After, we propose that the evaluators
may view all the evaluation indicators classified into
each category. It allows that the evaluators may
know the meaning of the criteria that they are
For the evaluation of LOs characteristics we
suggest two criteria. The first one is LO reusability,
which means assessing whether the LO can be
reused for other educational situations (into didactic-
curricular category). The second one is ensuring
standard compliance (into technical-aesthetic
The possibility of completing an evaluation
through collaborative method enables one to contrast
the individual’s initial evaluation with the others
experts’ evaluations. It aims to share different points
of view to achieve an advanced and reliable
evaluation (Vargo et al., 2003). However, the
emergence of consensus is not always a fact, so we
suggest publishing evaluators’ disagreements
through the platform, and as a result it will be
possible to consider this information before the LO
is reused.
3.2 Selection, delivery and post-
evaluation subsystem
Once LOs evaluations are completed they will be
saved on a normalized repository, as shown between
three and four steps in Figure 1. This repository will
be required for teachers to search the content they
need to structure their courses, and from this
repository teachers can find quality and uniform
Numerical ratings provided through the
evaluations mentioned above allow quick
comparisons for searching LOs.
LOs classifications provided for the knowledge
model and their evaluation allow teachers to find
content according to the subject area, type of
content, type of activity, and level of difficulty
(retrieving content associated with Bloom’s
cognitive domain categories) and their numerical
evaluation, which reflect their quality.
To achieve an optimal LO selection for reuse, we
suggest a knowledge management system with the
possibility to view a list of all the final LOs
evaluations and the possibility to access evaluation
criteria by links. As a result, it becomes easier to
QUALITY CONTENT MANAGEMENT FOR E-LEARNING: General issues for a decision support system
recognize which elements of the LO are weak and
find a way to correct, improve or change them.
LOs needs to be enabled with other ones to build
the largest units (didactic units, courses, etc.)
possible to deliver selected LOs for students, such as
those shown in step number five in Figure 1.
To achieve this objective, an educational
modeling language is needed. We also suggest IMS
Learning Design (IMS LD, 2003) because it has a
flexible structure that supports pedagogical
diversity. The classification provided by the
knowledge model could help for this work.
However, the LOs evaluation we suggested is not
definitive. Once the LO evaluation has ended, it is
necessary to make a LO re-evaluation, which
considers a learners’ experience about the efficacy
of the LO to improve its quality as shows six steps in
Figure 1. Therefore a re-feeding process begins
taking into account students’ and teachers
contributions to the LOs quality. As Figure 1 shows,
the re-feeding process is a cycle in which content is
constantly evaluated for all the e-learning users.
We think the general issues discussed here have
important advantages. Nowadays, the LO concept is
widely discussed, however we are suggesting and
specific definition to their evaluation and
management for e-learning systems. In this way it is
possible to define what criteria and what quality
indicators we could use to evaluate them.
On other side, due to the different kind of LOs
definitions, a lot of LOs with different levels of
granularity exists. A knowledge model, like ours,
aims to normalize the imported LOs. In this way, it
is possible to manage uniform LOs for their
evaluation and classified them according to an
educational context.
The type of evaluation we are suggesting has
several advantages in comparison with other
proposals. There are few ways about how to evaluate
learning object, for example, MERLOT (2003)
proposes an evaluation with stars from 1 to 5,
considering just few evaluation criteria. However,
we propose an evaluation that involves different
kind of evaluation categories to get an integral
evaluation adding the possibility for evaluators to
view evaluation indicators to guide them.
Additionally, the proposed knowledge
management system could be an important
contribution for e-leaning systems. Educators could
make use of the information already existing and use
the information that most interests them to structure
their courses. Also, this proposal would help to
promote a more in-depth reflection and evaluation of
the syllabus by taking into account points of view
related to searching and utilizing quality educational
As a result, this proposal also could help students
make use of quality content and activities by taking
into account a variety of educational, curricular,
technical and functional points of view.
These would in turn guarantee the establishment
of an up-to-date knowledge-base that would be both
suitable and reliable in accordance with the needs
and requirements of learners.
In addition, feedback would assist in answering
the questions about how to manage a growing e-
learning information repository to meet the users’
needs. Our future work is to implement this model in
order to make possible adjustments and
Bloom, B., 1956. Taxonomy of Educational Objectives:
Handbook I, Cognitive Domain. David McKay.
IEEE LOM., 2002. IEEE Learning Object Metadata.
IMS LOM., 2003. Learning Resource Metadata
IMS LD., 2003. IMS Learning Design Specification.
Kuang-Tsae, H., Lee, Yang, W., Wang, Richard., 2000.
Calidad de la información y gestión del conocimiento.
Editorial AENOR, Madrid.
Marquès, P.,
2001. Orientaciones para la selección de
materiales didácticos. http://dewey.uab.es/pmarques.
MERLOT., 2003. Multimedia Educational Resources for
Learning and Online Teaching. http://merlot.org.
Polsani, P., 2003. Use and abuse of reusable learning
objects. Journal of digital information.
Vargo, J., Nesbit, J.C., Belfer, K., Archambault, A., 2003.
Williams, D., 2000. Evaluation of LOs and instruction
using LOs. In D. A. Wiley, The instructional use of
LOs http://reusability.org/read/chapters/williams.doc