Erla Morales
, Francisco García
and Ángela Barrón Ruiz
Department of Theory and History of Education
University of Salamanca, Salamanca, Campus Canalejas, Pº de Canalejas, 169, 37008, Salamanca, Spain
Department of Computer Science, University of Salamanca, Plaza de los Caídos s/n, 37008 Salamanca, Spain
Keywords: Metadata, learning objects, knowledge management.
Abstract: Although LO management is an interesting subject to study due to the current interoperability potential, it is
not promoted very much because a number of issues remain to be resolved. LOs need to be designed to
achieve educational goals, and the metadata schema must have the kind of information to make them
reusable in other contexts. This paper presents a pilot project in the design, implementation and evaluation
of learning objects in the field of university education, with a specific focus on the development of a
metadata typology and quality evaluation tool, concluding with a summary and analysis of the end results.
Many studies have been done on the concept of
learning objects (LOs) but no consensus has been
reached on a standard definition or on the technical
and pedagogical requirements. Specifications are
being developed but have yet to be normalized, and
the use of metadata schemas is still under discussion.
This has prevented LO creation and management
from becoming common practice.
This paper presents our research on the design,
implementation and evaluation of a prototype LO
management tool for e-learning systems, containing
quality criteria designed to enable LOs to be
standardized and attuned to educational needs. The
prototype was built on the basis of our own
knowledge model, and comprises specific metadata
value spaces for classifying LOs into the LOM “5.
Educational” metadata category (IEEE LOM, 2002).
The paper begins by outlining the development
of an initial prototype learning object (LO1) and
determines what type of metadata should be applied
(section 2). It goes on to describe how we
implemented and evaluated LO1 using our LO
evaluation tool (section 3); then describes how the
results of those trials were used to produce a second
prototype (LO2), which was also implemented and
evaluated (section 4). Finally it presents our
conclusions and plans for the next stages of our
work (section 5).
The first task to create our initial prototype learning
object (LO1) was to chose a context in which to
conduct our trials: the Object-Oriented Programming
(OOP) option of the Computer Science course at
Salamanca University (Morales, García, Barrón and
Gil, 2007c). We then defined a set of specific
learning objectives with which we built a knowledge
model (figure 1) that served to produce a basic unit
of learning for designing LO1, entitled “Object-
Oriented Programming: General Issues” (Morales,
García and Barrón, 2007b).
One of the key goals here was to enable a
knowledge model to be used to standardize LOs,
which is crucial for them to be tailored to
educational needs, taking into account key elements
for learning (Morales, García and Barrón, 2007a).
Sound LO management requires the
incorporation of reliable metadata, but the viability
of the only metadata schema currently regarded as a
standard (IEEE LOM, 2002) has been called into
question because it uses vast quantities of ill-defined
types of data, and some of its metadata categories do
not make it clear what kind of information has to be
added, thus further complicating the task of LO
management (Morales, García and Barrón, 2006).
Morales E., García F. and Barrón Ruiz Á. (2008).
In Proceedings of the Tenth International Conference on Enterpr ise Information Systems - AIDSS, pages 559-562
DOI: 10.5220/0001710705590562
Figure 1: Knowledge Model of LO1.
Although the lack of clarity in the IEEE LOM
standard makes its value spaces hard to interpret.
We set out to address this issue – and, hence, to
enable suitable LO management data to be
introduced into learning environments – by devising
a set of definitions to clarify the content of each
value space in the LOM “5. Educational” category:
5.1 Interactivity Type: expositive
LOs featuring a very low interactivity level,
with students receiving information yet
remaining unable to interact with the content
5.2 Learning Resource Type: web pages
5.3 Interactivity Level: low
LOs with an expositive interactivity level – minimal
student participation (web pages with few links)
5.4 Semantic Density: medium
LO content designed to promote smooth
learning and application of knowledge
5.5 Intended End User Role: learners
5.6 Context: university level
5.7 Typical Age Range: Unspecified
5.8 Difficulty: easy
Information is easily associated with previous
We then incorporated these definitions into our
prototype LO1.
Having designed LO1 based on our knowledge
model and incorporating our proposed metadata
typology, we set about implementing it with Moodle
together with the following additional elements:
a pdf file: so that our sample students could
print out the LO content
a self-assessment section: so that they could
see how much they knew about the content,
and to repeat the test whenever necessary
a forum: so that learners and teachers could
discuss the content
an evaluation tool: for the students to rate the
quality of LO1.
Current proposals for learning resource evaluation
tools include web sites (Marqués, 2003; Torres,
2005) and multimedia tools, (Marqués, 2000), and
other proposals have been made for assessing the
quality of LOs taking into account their instructional
use-oriented design (Williams, 2000) and
sequencing (Zapata, 2006). We drew on these to
design an instrument that would enable learners to
assess the value/quality of their LOs (see figure 2).
Our sample students were able to access the LO and
the evaluation tool via Moodle and to rate them on a
scale of 1 to 4: 1= very poor; 2=poor; 3=satisfactory;
4=high; 5=very high.
As seen in figure 2 (above), the evaluation tool
was designed to gather qualitative and quantitative
data about LO1.
The qualitative results show a general agreement
on its quality. The highest scoring value was the
difficulty level (3.87), followed by the objectives
and content (3.82). These results reflect our sample
students’ approval of the content in terms of its
quantity, consistency, reliability and so on.
Navigation was considered well-designed and user-
friendly (3.79).
The students were slightly less happy with the
overall design of LO1 (3.74), and suggested a
number of possible improvements. They also made a
number of positive comments on the feedback
(3.66). ‘Activities’ and ‘interactivity’ were rated
satisfactory (3.51), as was the lowest scoring
criterion: ‘motivation’ (3.41).
The feedback gained from the space provided in
LO evaluation tool for students to make comments
provided very useful pointers for us to see what
needed to be improved when developing our second
prototype (LO2).
ICEIS 2008 - International Conference on Enterprise Information Systems
Table 1: LO1 quality rating incorporated into LOM.
9. Classification
9.1 Purpose Quality
9.2 Taxon Path
9.2.1 Source Table 1. LO Eval. Rating
9.2.2 Taxon CA*: 3.64 (high) Id CA: 3.64 (high) Entry High
9.3 Description LO considered high quality by
sample students. Lowest
scoring quantitative items
were ‘motivation’, ‘activities’
and ‘interactivity’. Qualitative
feedback suggested adding a
glossary and examples…
9.4 Keyword quality, value, high, CA_3.64.
*CA: CALIDAD (quality)
To input the quantitative and qualitative data on
the quality of LO1 into our metadata typology, we
used the LOM “9. Classification” metatada category
in combination with our own LO quality rating
classification scheme. We believe that quality
measurement using a scale should be introduced into
the “9. Classification” metadata category. Table 1
shows our prototype adaptation using the final
quality score taken from the LO1 evaluation results.
Adding a quality value to the LO metadata
category would help locate and retrieve an LO
through a search based on keywords (e.g. quality,
value, high, etc.) An alphanumeric value (e.g.
CA_3.64). makes it possible to define a specific
vocabulary for running an LO search.
The sample students’ comments provided useful
pointers for producing an enhanced and more user-
friendly design for our second prototype (LO2), with
a different font, larger characters and links to further
reading.. The actual content of LO2 followed on
from LO1, taking the learning objectives to a more
advanced level.
LO2 was implemented in the same learning
environment as LO1, and was evaluated with an
enhanced version of our quality evaluation tool
(figure 2).
The final score reflects a similarly high average
quality rating on the part of our sample students
(3.66). The highest scoring item was ‘navigation’
(4.00), followed by ‘description’ and ‘activities’
(self-assessment) (3.91), both of which figure in the
Didactic Curricular Issues category.
Content design was considered high quality
(3.74), as were three other didactic-curricular issues:
– achievement of objectives (3.69), learning time,
and LO content (3.63) – and one psycho-
pedagogical issue: ‘difficulty’ (3.63) .
Student comments were even more positive for
LO2 than LO1, expressing their approval of the new
section with references, links to further reading, a
glossary and a list of acronyms.
Some, however, considered that the screen
resolution was better but needed further
improvement: there were still too many scroll bars
and accessing table cells remained an impediment to
sightless users.
Having completed our evaluation, we
incorporated the overall LO2 quality rating into the
corresponding LOM “9. Classification” metadata
Figure 2: LO2 Evaluation Results.
category, using the LO classification scheme based
on our proposed metadata typology (Morales, García
and Barrón, 2007b).
Our proposed adaptation of the LOM “9.
Classification” metadata category comprises the key
quantitative and qualitative data collected with our
LO quality evaluation tool. In presenting a summary
of learners’ comments on LO quality, item “9.3.
Description” provides a useful means of further
improving that quality.
Our prototype knowledge model sought to
demonstrate how LOs can be established as a basic
unit of learning, taking into account key educational
needs. It can be used to adapt an LO to a specific
type of course at university level.
The LO quality evaluation tool enabled us to
collect a wide range of information useful for
improving both LO1 and LO2. In attributing a
numerical value to LO quality, the rating scale
helped specify exactly which data to incorporate into
the metadata schema.
It is important to remember that metadata
editors today only classify LOs according to specific
established purposes. We used the LOM “9.
Classification” metadata category because we
believe it useful for defining and adapting new LO
classification schemes that would allow users to
acquire and manage LOs suited to their own
individual needs.
Finally, the results obtained with the LO quality
evaluation tool helped highlight exactly what
improvements needed to be made. Sorting
evaluation criteria into different categories made it
possible to evaluate the LOs from both pedagogical
and technical points of view.
Our future work will focus on developing an
LO creation tool based on our knowledge model. We
will also seek to improve the quality of LOs by
taking into account the accessibility issues that are
crucial to LO management.
This work was co-financed by the Spanish Ministry
of Education and Science, the FEDER-KEOPS
project (TSI2005-00960) and the Junta de Castilla y
León local government project (SA056A07).
IEEE LOM. 2002. IEEE 1484.12.1-2002 Standard for
Learning Object Metadata. Retrieved June, 2007, from
Marquèz, P. 2000. Elaboración de materiales formativos
multimedia. Criterios de calidad. Disponible en
Marquèz, P. 2003. Criterios de calidad para los espacios
Web de interés educativo. Disponible en
Morales, E. M., García, F. J., Barrón, Á. 2007a. Key
Issues for Learning Objects Evaluation. In ICEIS'07,
Ninth International Conference on Enterprise
Information Systems. Vol 4. pp. 149-154. INSTICC
Morales, E. M., García, F. J. & Barrón, Á. 2006. Quality
Learning Objects Management: A proposal for e-
learning Systems. In ICEIS’06. 8th International
Conference on Enterprise Information Systems
Artificial Intelligence and Decision Support Systems
Volume Pages 312-315. INSTICC Press. ISBN 972-
8865-42-2. 2006. (B3).
Morales, E. M., García, F. J., Barrón, A. 2007b. Improving
LO Quality through Instructional Design Based on an
Ontological Model and Metadata J.UCS. Journal of
Universal Computer Science, vol 13. nº 7. pp. 970-
979. ISSN 0948-695X
Morales, E.M., García, A.B., Barrón, A., Gil, A.B. 2007c.
Gestión de Objetos de Aprendizaje de calidad: Caso de
estudio. En SPDECE’07. IV Simposio Pluridisciplinar
sobre Objetos y Diseños de Aprendizaje Apoyados en
la Tecnología. ISBN 978-84-8373-992-1.
Torres, L. 2005. Elementos que deben contener las
paginas web educativas. Pixel-Bit: Revista de medios
y educación, Nº. 25, 2005, pags. 75-83.
Williams D. D. 2000 Evaluation of learning objects and
instruction using learning objects. In D. A. Wiley (Ed.),
The instructional use of LOs,
Zapata R. M. 2006. Calidad en entornos virtuales de
aprendizaje y secuenciación de Learning objects (LO).
Actas del Virtual Campus 2006. V Encuentro de
Universidades & eLearning, 111-119 ISBN 84-689-
ICEIS 2008 - International Conference on Enterprise Information Systems