KEY ISSUES FOR LEARNING OBJECTS EVALUATION
Erla M. Morales Morgado, Ángela Barrón Ruiz
Departament of Theory and History of Education, Salamanca University, Canalejas Street 169, Salamanca, Spain
Francisco J. García Peñalvo
Department of Computer Science, Salamanca University, Plaza de los Caídos Street, s/n, Salamanca, Spain
Keywords: Learning Objects, Evaluation, Knowledge Management.
Abstract: Web development is promoting important advantages for educational area specially e-learning systems. By
one side Learning Objects (LOs) aim the possibility to reuse specific information and by the other side they
can be interchanged though different context and platforms according to the user’s needs. However an
urgent necessity exists to guarantee the LOs quality content. There exists a plethora of quality criteria to
value digital sources but there are only a few suggestions about how to evaluate LOs to structure quality
courses. This work is a proposal to evaluate LOs as a continued process taking into account different kind of
LOs evaluation according to their characteristics and quality criteria related to metadata information,
pedagogical and usability issues, together with a strategy to ensure a continued LOs quality contents.
1 INTRODUCTION
As consequence of Semantic Web, an important
contribution from computer science to knowledge
management and e-learning systems is the learning
object (LO) concept. This element has
characteristics of independent units, which are able
to be reused for other educational situations and
platforms.
Each one of LOs has metadata (data about data)
for their description and administration. In this way
it is possible to know what kind of LO we are trying.
According to this, knowledge management for e-
learning based on reusable units of learning means
the possibility to access specific content according to
the learners’ needs. This stage is possible due to
standards, which were established as an attempt to
avoid interoperability platform problems, but they
don’t guarantee the LOs content quality.
A great quantity of criteria exists about digital
learning sources evaluation. Nevertheless, for LO
content evaluation; there are just a few proposals
that are interesting in order to consider their
characteristics about how to evaluate LOs to
structure quality courses together with the teacher’s
expert knowledge and the student’s learning
experience.
On this basis our proposal emphasizes the key
issues that need to be considered in order to achieve
a suitable LOs evaluation. To achieve this section 2
explains general issues for LOs evaluation
considering the LOs context and what kind of LO to
manage. It emphasizes the things that are needed to
consider for a possible LOs reuse. Section 2.1
presents an input evaluation where it is necessary to
value LOs characteristics taking into account
pedagogical, usability and metadata issues. Section
2.2 explains our LOs instrument to value LOs
according to quality criteria. To ensure LOs quality
evaluation and reliability we suggest combining
instrument application together with a collaborative
strategy which is explained in section 2.3
Finally, section 2.4 suggests LOs evaluation as a
Product. It means the possibility of users to make a
LOs evaluation after their use. To achieve this,
students have to answer questions about their
content quality and their-self satisfaction. All
information obtained may be given to experts and
teachers to advance contents design and to guarantee
a continuous quality contents re-feed. Finally section
3 points out our conclusions.
149
M. Morales Morgado E., Barrón Ruiz Á. and J. García Peñalvo F. (2007).
KEY ISSUES FOR LEARNING OBJECTS EVALUATION.
In Proceedings of the Ninth International Conference on Enterprise Information Systems - SAIC, pages 149-154
DOI: 10.5220/0002384901490154
Copyright
c
 SciTePress

2 GENERAL ISSUES FOR LOS
EVALUATION
The first thing to take into account for knowledge
management is to identify what kind of information
to manage. Knowledge is the principal factor which
supports innovation and change, and has a strategic
value for organizations. For this reason it is
fundamental to manage it accurately (Kuang-Tsae et
al., 2000).
According to this we define a LO “as a unit with
a learning objective, together with digital and
independent capabilities, accessible through
metadata to be reused in different contexts and
platforms” (Morales at al., 2006b).
In order to promote quality LOs management we
suggest evaluate LOs according to their
characteristics together with the suggestions about
who, evaluate, when and what instruments and
strategies to use.
In order to promote quality LOs it is necessary to
consider the possible context of use. Due to their
reusable capability LOs can be interchanged for
different educational situations.
According to Stufflebeam (1971) context
evaluation focuses on evaluating needs, priorities,
shared vision of participants, expectations of people
and organizations, and how their efforts fit into
broader time and location contexts. According to
this we think LOs context evaluation need to
consider the following issues:
Curricula: LOs must be suitable for the new
educational context curricula plans.
Student characteristics: LOs need to be suitable
for students’ previous knowledge.
Learning objectives: LOs need to have all the
necessary elements in order to achieve learning
objectives.
Technical requirement: The new context in
which LOs can be reused need to have suitable LOs
technical requirement, e.g. suitable computers and
Internet connection, etc.
According to reusable LOs capabilities, we
consider to evaluate external LOs (imported, buy,
etc.) or create them. The possibility to import or
create LOs enables to enrich a Knowledge
Management System. However the first thing to
consider is what kind of LOs we are trying. On this
basis we think it is necessary to normalize them
because in this way it is possible to guarantee a
suitable degree of granularity. To achieve this, we
suggest the next steps (Morales et al., 2006b).
• Classify LOs components: LOs may be
classified for different purpose by the
metadata “9.Classification”. According to
this users can define some characteristic
for them adding a vocabulary to the
metadata schema. To achieve a better
LOs management we suggest the
following LOs classification.
• Clasiffy LOs objectives according to their
cognitive domain: In this way it is easier
knowing about their compatibility for
suitable new educational situations. Then,
we suggest Bloom’s cognitive domain
taxonomy (Bloom, 1956) because it
defines what and how to learn according
to complexity levels: low level
(knowledge, comprehension and
application) and high level (analysis,
synthesis and evaluation).
• Classify LOs into three kind of content:
data and concept, procedure or processes,
and reflection or attitude. This
classification aims to define the kind of
content according to the learning
objectives. This is an important issue for
teachers because it aims them to search a
specific type of LOs and easily structure
their courses.
Evaluation which compare alternative inputs or
means for meeting the needs identified in context
evaluations, including but not limited to LOs
(Stufflebeam, 1971). It focuses on evaluating
alternative inputs that could be considered for
addressing concerns such as vision, purposes,
alternative curricula, instructional strategies,
participants, technologies, etc. According to this
we propose an input evaluation after LOs
normalization. In this way it is possible to evaluate
them taking into account a uniform structure.
2.1 LOs Characteristics Evaluation
There are a lot of KMS possibilities to support the
teaching and learning process through e-learning
systems, such as delivering and evaluating courses,
etc. (Rosenberg, 2001; Avgeriou, 2003). However,
according to LOs and standards capabilities, it is
necessary to consider how to manage quality LOs,
taking into account their characteristics.
In order to promote a whole LOs evaluation we
suggest evaluating them taking into account
different points of view: Pedagogical, Metadata and
usability.
ICEIS 2007 - International Conference on Enterprise Information Systems
150

a) Pedagogical: LOs are units of learning, for this
reason we think it is necessary to consider
pedagogical issues, according to this we propose to
evaluate psychopedagogical and didactic-curricular
characteristics.
• Psychopedagogical: Contains pedagogical
criteria related to the psychology of
learning. This kind of criteria aims to
determine if the LO is suitable to promote
learning.
• Didactic-curricular: This kind of criteria
aims to evaluate if a LO is related to
curricular objectives according to the
context in which it will be applied.
b) Usability: This concept is related with the ability
to achieve goals efficiently into a specific context.
Web design area has adopted this concept in order to
obtain a suitable and efficient interface design.
According to Nielsen (2001) we suggest to define
quality criteria for content design and site map
design.
• Content Design: It is directed to technical
issues about images, text, video,
animations, etc.
• Site Map Design: It encompasses main
page and site map navigation
c) Metadata: Taking into account this situation and
the low flexibility of LOM the only one metadata
specification approved some metadata schema
initiatives exist in order to attend specific context
and users that is called application profile.
LOs are characterized by the separation of
their content and presentation, for this reason an
important issue to consider evaluating them is their
metadata information. Metadata, provide LOs
information to their description and managing, in
this way it is possible to know if their characteristics
are suitable for other educational situations. Our
proposal is based on IMS specifications, for this
reason we refer to metadata according to IMS LOM
(IMS, 2005). However, in a way to made a context
evaluation we suggest the following application
profile.
• 1.General: The elements mentioned into
general category are very important in order
to manage LOs. It is because they contain
information which it is necessary for their
searching.
• 5.Educational: Educational information
aims to know pedagogical LOs
characteristics which it is necessary to take
into account before to reuse them in other
educational situations.
• 7.Relation: Relation metadata information
aims to know the type of LOs we are trying
and their relations with other ones. This
information is very useful in order to
establish LOs sequence.
• 8.Anotation: Description element into
annotation category aims to describe
experiences about LOs application. This
information is very useful in order to know
if the LO is suitable for a specific learning
situation.
• 9.Classification: This category aims to
describe LOs into a specific classification
system. IEEE LOM (2002) defines some
values for it (idea, learning objective,
accessibility, etc). However we think it is
necessary to consider LOs classification
according to users need. In this way we
suggest to use this category to classify LOs
according to kind of contents, cognitive
level and quality value.
Metadata Categories Metadata Elements
1.General
1.2 Title
1.4 Description,
1.5 Keywords,
1.6 Coverage,
5.Educational
5.1 Interactivity Type,
5.2Learning Resource
Type,
5.3 Interactivity Level,
5.4 Semantic Density,
5.6 Context,
5.7 Typical Age Range,
5.8 Difficulty,
5.9 Typical Learning Time,
5.10 Description,
5.11 Language
7.Relation
7.1 Type,
7.2 Resource,
8.Annotation 8.3 Description,
9.Classification
9.1 Purpose,
9.2 Taxon Path
9.2.1 Source
9.2.2 Taxon
9.2.2.1 Id
9.2.2.2 Entry
9.3 Description
9.4 Keyword
Figure 1: Metadata categories and elements suggested for
LOs management.
KEY ISSUES FOR LEARNING OBJECTS EVALUATION
151

The classifications of LOs provided for the
knowledge model allow teachers to find content
according to the cognitive domain and type of
content. By other side it provides to learners
different kind of content to achieve their educational
objectives. Nevertheless, LOs normalization is not
enough to guarantee their quality. To ensure a
quality LOs from a pedagogical point of view we
suggest to value LOs quality through and evaluation
instrument.
2.2 Evaluation Instrument
In order to evaluate LOs we designed an instrument
as shows figure 2. As we mentioned above the
instrument take into account quality criteria for
pedagogical issues (Psychopedagogical and
Didactic-Curricular) and usability issues (Content
Design and Site Map Design). The figure 2 shows an
example of psychopedagogical quality criteria.
For evaluating LOs there is a range scale, if
evaluators don’t know how to evaluate it or if they
have a doubt it is possible to select DN= Don’t
Know, otherwise they can to select the following
rate scale 1=very disagree, 2=disagree, 3=agree,
4=very agree.
For example, there are quality criteria related with
LOs objectives, contents, and activities, each one of
them has a final score that aim to know their
individual scoring and reinforce them if it be
necessary. According to this, the final scoring of
each category is average out at the field “final
score”. In case of any doubt, critic or suggestions
evaluators have a comments section. To evaluate LO
reusability the instrument contain a section called
“Reusability” where it is possible to comment
possible context of use.
2.3 Evaluation Strategy
Patton (1997) argues that the key to evaluation
utility is to identify people who are disposed to
learning from evaluation. He outlines several
procedures for identifying these users an then
working with them to clarify what they want to
know and what they are likely to do with
information gathered by an evaluation.
Input evaluation is directed to experts related
with educational area who have to evaluate LOs in
an individual and collaborative mode.
Individual evaluation provides us an initial
appreciation of the quality of the LO based on the
judgment of each participant. According to (Vargo
et al, 2003) collaborative evaluation aim to
encourages not only different points of view over the
subject under evaluation, but also a critical
objectivity and a reliable LOs evaluation.
The possibility of completing an evaluation
through collaborative method enables to contrast the
individual’s initial evaluation with the others
experts’ evaluations. It aims to share different points
of view to achieve an advanced and reliable
evaluation (Vargo et al, 2003).
In order to help teachers in this work by one side
our tool aim to analyze graphics which show
statistics that reflect individual an collaborative
evaluation and by the other side it provide a forum
for discussions to achieve an agreement for a final
evaluation.
PEDAGOGICAL ISSUES CRITERI
A
D/N= Don't know,
1=Very Desagree,
2=Desagree,
3=Agree,
4=Very Agree
PSICOPEDAGOGICAL Motivation and Atention
Presentation
: captute learners attention mantaning their motivation
Add important information
: Information need to be relevant according to the LO subject
Learners participation
: Learners need to find out what to do
Professional competence
learning objectives need to help users to achieve their professional competences
Difficult
y
level
contents difficulty level: It need to be suitable for user cognitive domain
Language
: It needs to be suitable for previous users knowledge
Interactivity
Interactivit
y
Level: It
p
romotes o
pp
ortunities to interact with LO in different wa
y
s
Interactivity type
: LO interaction aim to achieve learning objectives
Creativity
It promotes self-learning
It promotes cognitive domain development
COMMENTS
Figure 2: Pedagogical issues for LOs evaluation through evaluation instrument.
ICEIS 2007 - International Conference on Enterprise Information Systems
152

2.4 Product Evaluation
According to Williams (2000), students or learners
are some of the most important users of LOs. And of
course, they vary in their needs and values even
more than instructors do because there are more of
them. But eventually students have to evaluate any
given learning opportunity and choose to learn from
it or not. According to this learners are the key for
LOs evaluation.
Once students have finished their lesson they
have to respond questions about their satisfaction
with the LO. Each one of these questions is related
with LOs evaluation instrument; in this way it is
possible to contrast them with previous experts’
evaluation.
Product evaluation aims to re-feed LOs quality,
because it considers a learners’ experience about the
efficacy of the LO to improve their knowledge
(Morales et al., 2006a); (Morales et al, 2006c).
Taking users responses, evaluators may have to
re-feed LOs to guarantee their continued quality.
After LOs evaluation they must to be saved into
a repository which contains normalized and quality
contents. From this repository teachers could search
LOs to structure their courses offering quality
contents for their students. These contents will be
part of biggest units of learning like lesson, modules,
courses, etc. and they will be published by e-
learning system for their usability and be continually
evaluated to guarantee their quality.
Stufflebeam model (1971) define process
evaluations stage as an evaluation that formatively
assess the planning, design, development, and
implementation of learning objects and associated
efforts to use them, including attempts to adapt
instruction based on individual differences as
expressed in learner profiles, etc. Taking into
account Process evaluation focuses on evaluating the
processes being used to address needs clarified in
the context evaluation and the use of various inputs
to carry out a program or project. Examples of
processes include organizational structure,
instructional strategies, cooperation among
organizations, use of technologies, involvement of
faculty, curriculum development, course
development, organizational change, etc.
According to our proposal, we consider process
evaluation for course development. Therefore a re-
feeding process is needed which taking into account
students’ and teachers’ contributions to the LOs
quality. To achieve this, the process evaluation
consist on users comments during their interaction
with LOs. Users can contribute with valuable and
unexpected comments to consider improving LOs
quality and teaching and learning process. Then we
suggest creating a forum for users’ comments about
the process.
3 CONCLUSIONS
LOs evaluation is a complex area in development. It
is because there is not a consensus about things like
LOs definition, size, reusability, etc. Most of LOs
proposals are directed to achieve a suitable LOs
management from a technical point of view in order
to guarantee their characteristics reusability,
accessibility and interoperability for automatized
process.
Nowadays it is possible to find a lot of tools for
help this task like metadata editors, e-learning
platforms, etc. However in educational area LOs
needs a special attention. According to LOs
Figure 3: Pedagogical issues for LOs evaluation through evaluation instrument.
KEY ISSUES FOR LEARNING OBJECTS EVALUATION
153

definitions they must be directed to teach a little unit
of content. However to achieve this objective LOs
must have a suitable instructional design that aim to
achieve their educational objective.
We think our LOs definition may be suitable for
LOs management because it promotes simple LOs
contents that could help to reuse them in easily.
Our normalization proposal helps to promote a
uniform LOs level of granularity and the possibility
to increment LOs reusability to another specific
context. It is because relating a LO to knowledge
domain aim to attend different educational situations
for different requirements.
Each one of pedagogical evaluation criteria aim
to evaluate LOs characteristics into a concrete set,
providing specific criteria for LOs evaluation for
experts into an individual and a collaborative
strategy. This issue has a special value because
criteria are situated into psychopedagogical and
didactic-curricular areas. However an expert
evaluation must be reinforced with users’
evaluations, which might contribute their experience
and express their satisfaction.
IEEE LOM metadata elements have a complex
structure, thus it is not very clear what kind of
information to add. Our proposal considers specific
metadata elements for a suitable LOs management
and issues to consider in order to adding information
in a suitable way. By other side “classification”
metadata element is part of an official metadata
proposal, and it can be used for personalized
applications profiles in order to classify the LOs
according to their particular educational needs.
We want to emphasize that our proposal is an
attempt to solve questions about LOs evaluation.
However it doesn’t guarantee the quality LOs
management for e-learning systems because it
depends of many issues like platform capabilities,
usability, accessibility, etc. which are out of this
specific proposal. However this work proposes some
ideas to improve LOs quality from an instructional
design point of view that must be applied both,
instructional design and metadata information.
ACKNOWLEDGEMENTS
This work was partly financed by Ministry of
Education and Science as well as FEDER Keops
project (TSI2005-00960).
REFERENCES
Avgeriou, P., Retails, S., & Skordalakis, M. (2003). An
architecture for open learning management systems,
En Y. Manolopoulos, S. Evripidou, A.C. Kakas (Eds.)
Advances in Informatics. 8th Panhellenis Conference
on Informatics. Lectures Notes in Computer Science.
LNCS 2563. Berlin: Springer Verlag, pp.183-200.
Bloom, B. (1956). Taxonomy of educational objectives:
Handbook I, Cognitive Domain, Davis McKAy”.
IEEE Standard for Learning Object Metadata. (2002).
ANSI/IEEE. http://ltsc.ieee.org/wg12/
IMS LOM. (2005). Learning Resource Metadata
Specification.
http://www.imsglobal.org/metadata/mdinfov1p1.html
Kuang-Tsae, H., Lee, Yang, W., & Wang, Richard.
(2000). Calidad de la información y gestión del
conocimiento. Editorial AENOR, Madrid.
Morales, E., García, F., & Barrón, Á. (2006a). “Quality
Learning Objects Management: A proposal for e-
learning Systems” 8th International Conference on
Enterprise Information Systems (ICEIS’06). Paphos,
Cyprus, 23 al 27 de Mayo del 2006.
http://www.iceis.org.
Morales, E. M., García, F. J., & Barrón, Á. (2006b). "/LOs
Instructional Design based on an Ontological Model to
Improve their Quality/". In /Proceedings of the/ 8^th
International Symposium on Computers in Education,
SIIE'06 (León, Spain, October 24^th - 26^th , 2006).
L. Panizo Alonso, L. Sánchez González, B. Fernández
Majón, M. Llamas Nistal (Eds.). Vol. 1. Pages 441-
448. ISBN Obra completa 84-9773-303-7. ISBN Vol.
1 84-9773-301-0. 2006.
http://siie06.unileon.es/welcome.php
Morales, E., García, F. Barrón, A. & Gil, A. (2006c).
“Sistema de gestión de Objetos de Aprendizaje de
calidad” en III Simposio Pluridisciplinar sobre Objetos
y Diseños de Aprendizaje Apoyados en la Tecnología
25, 26 y 27 de Septiembre del 2006, Oviedo, España
http://www.spi.uniovi.es/od@06/inicio.htm
Nielsen J. (2001). Usabilidad Diseño de Sitios Web
Prentice Hall PTR; 1ª Edición 9 Noviembre.
Patton, M.Q. (1997). Utilization-focused evaluation. (3rd
ed.) Thousand Oaks, CA: Sage.
Rosenberg, M.J. (2001). E-learning. Strategies for
delivering knowledge in the digital age, Mc Graw Hill.
Stufflebeam, D.L. (1971). The relevance of the CIPP
evaluation model for educational accountability.
Journal of research and development in education.
5(1), 19-25.
Vargo, J., Nesbit, J., Belfer, K., & Archambault, A.
(2003). Learning object evaluation: computer-
mediated collaboration and inter-rater reliability,
International Journal of Computers and Applications,
25, 3.
Williams, D.D. (2000). “Evaluation of learning objects
and instruction using learning objects”. In D. A. Wiley
(Ed.), The instructional use of LOs,
http://reusability.org/read/chapters/williams.doc
ICEIS 2007 - International Conference on Enterprise Information Systems
154