ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED
QUALITY OF LMS SYSTEM AND E-COURSE DELIVERY
Matjaž Debevc, Martina Breg
Faculty of Electrical Engineering and Computer Science, University of Maribor, Smetanova ulica 17, Maribor, Slovenia
Julija Lapuh Bele
B2 d.o.o., Ljubljana, Slovenia
Keywords: Distance Learning, e-Learning, Learning Management Systems, e-Course, Learning Evaluation.
Abstract: In this research, a quality of e-Learning has been measured on the basis of students’ satisfaction with e-
learning environment (i.e. LMS system) and e-Course delivery. A questionnaire has been developed to
measure perceived quality of e-Learning technology and didactics. Results of this research also show
statistically significant correlation between the quality of e-Learning environment and the e-Course quality
in case of blended learning mode of delivery where only short introductory meeting and final examination
are carried out face-to-face. Students who were satisfied with e-Learning environment were also satisfied
with e-Course and vice versa. The research has been conducted by taking into consideration two different
Learning Management Systems and eight e-Courses facilitated with different teachers.
1 INTRODUCTION
e-Learning can be described with three basic criteria:
learning on demand, transfer of information to
students, and implementation of virtual classroom
(Rosenberg, 2001).
The success of any learning program is largely
dependent on the motivation and attitude of learners.
Therefore the key factors in e-learning delivery are
usability and didactic effectiveness (Ardito,
Costabile, Marsico, Lanzilotti, Levialdi, Roselli &
Rossano, 2006).
Since ICT is a crucial factor in each e-learning
setting, researchers emphasise various aspects to
evaluate e-learning platforms and other learning
applications (Ardito et al, 2006, Costabile, Marsico,
Lanzilotti, Plantamura & Roselli, 2005, Dringus &
Cohen, 2005, Squires & Preece, 1999, Zaharias &
Poylymenakou, 2006). Quality of computer software
often means usability (Nielsen, 1994).
Unfortunately, there is a lack of an evaluation
model that enables quick and easy evaluation of
e-Learning quality aspects. Existing models and
questionnaires are time consuming and
comprehensive (Bates & Obexer, 2005, Philips,
2005, Nielsen, 1994, Kirakowski, 1993). Since
students do not like to fill out long questionnaires,
such questionnaires are thus inappropriate for
regular implementation after e-Courses.
Therefore, this paper focuses on the newly
designed short-time e-learning quality evaluation
model, which we call DEMA model. The
questionnaire consists of only 28 items that cover
the following indicators of students’ satisfaction
with LMS system and e-Course delivery:
Usability,
Users Communication,
Functionality,
Safety,
Help and Support,
Learning Satisfaction,
e-Content Satisfaction,
Gained Knowledge and
Transferred Knowledge into Practise
The questionnaire is short enough to be
implemented after the completion of each e-Course.
In this study, the following research question is
also raised: is there a correlation between the
perceived e-learning portal quality and the e-Course
quality? Therefore, the following hypothesis has
311
Debevc M., Breg M. and Lapuh Bele J..
ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED QUALITY OF LMS SYSTEM AND E-COURSE DELIVERY.
DOI: 10.5220/0003304503110315
In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 311-315
ISBN: 978-989-8425-50-8
Copyright
c
2011 SCITEPRESS (Science and Technology Publications, Lda.)
been set: there is statistically significant relationship
between the quality of e-Learning environment and
the quality of e-Course. In the study, the term
e-Learning environment refers to e-Learning portal,
powered by Learning Management system (LMS).
There were total 197 students and 8 teachers
involved in the study. Each student has attended one
of eight blended learning courses (i.e. blend of
face to face and facilitated e-Learning) and has been
mentored by one of eight teachers. Face-to-face part
of each blended learning course consisted of a short
introductory meeting and final examination in
classic classroom. The major part of each course has
been conducted online. Participating students have
used learning environments eCampus (Debevc &
Bele, 2008) or Moodle (Martin-Blas & Serrano-
Fernandez, 2009).
2 DEMA
MODEL - QUESTIONNAIRE
In order to evaluate the quality of e-Courses and to
design our own questionnaire, we have chosen
Kirkpatrick’s model (Kirkpatrick & Kirkpatrick,
2006) and adjusted it to the population that
participated in the study.
Disadvantage of Kirkpatrick’s model is in its
complexity, in relation to a number of questions and
the time needed to conduct evaluation and data
analysis. Therefore, we have tried to search for a
shorter, faster model, which would confirm our
hypothesis that there is statistically an important
relationship between the quality of e-learning
environment (LMS system) and the quality of e-
Course.
To reach these requirements, the Kirkpatrick’s
model, based on his first three levels, needed to be
redesigned.
The goal of the anonymous DEMA
questionnaire is to measure learners’ satisfaction
with e-Learning fundamentals, e-Learning
environment and e-Course as well as to confirm or
reject the hypothesis that the quality of e-Learning
environment and e-Course quality are significantly
correlated.
2.1 First Part - Quality of e-Learning
Environment
The first part of the questionnaire evaluates the
quality of e-Learning environment. This part
consists of thirteen 1-to-5 rating Likert scales of
quality measures divided into five categories and is
based mainly on European Computer Driving
License Certified Training Professional trainers
evidence record (ECDL CTP, 2010), adapted for
DEMA model.
It measures simplicity of use, ability of
communication between users, functionality,
security and additional help and support.
2.2 Second Part - e-Course Quality
The second part of the questionnaire evaluates the e-
Course quality based on first three levels of
Kirkpatrick’s model. It consists of fifth teen 1-to-5
rating Likert scales of quality measures divided into
five categories. It measures learner’s satisfaction
with course organisation, given information,
teacher’s support and meeting individual
expectations. The second factor of e-Course quality
is the e-content quality. The questionnaire measures
intelligibility, conciseness, graphic design, quality of
used multimedia and hyperlinks and quality of tests
for knowledge evaluation.
The next factor measures the gained knowledge.
Even though in most cases, the gained knowledge is
evaluated with written tests, we have measured
participant’s opinion on gained knowledge.
The evaluation ends with questions about
usefulness of gained knowledge (i.e. transfer of
knowledge into practice).
3 RESEARCH
The purpose of this study is to confirm or reject the
following hypothesis: there is a positive relationship
between quality of e-Learning environment (i.e. e-
Learning portal powered by LMS system) and e-
Course quality in blended learning setting.
3.1 Methodology
3.1.1 Participants
Full-time students from Faculty of Electrical
Engineering and Computer Science (FERI) and part-
time students from B2 Vocational College have
participated in the study. Each student has attended
one of eight blended learning courses (i.e. blend of
face to face and facilitated e-Learning) and has been
mentored by one of eight teachers.
Students at FERI used LMS system Moodle
while those students at B2 learned via LMS system
eCampus.
CSEDU 2011 - 3rd International Conference on Computer Supported Education
312
3.1.2 Procedure
All participants attended blended-learning courses
(BL). Each blended learning course started in the
classroom, with an introductory face to face (F2F)
meeting. Teacher presented the course, announced
learning goals, learning tasks (e.g. project work,
assessments), code of behaviour within the course
and gave advice on e-Learning strategies. Then, the
e-Learning activities began. Most of them run
asynchronously. Students learned from e-content
(i.e. interactive learning materials that included
automated feedback and dynamic graphics such as
video, animations and simulations) and performed
learning tasks according to a weekly schedule that
determines learning activities (i.e. real-time online
meetings, readings, discussions, project work, on-
line assessments etc.). All activities had deadlines.
Students could carry them out according to their
own schedule. Teacher used the following activities
to facilitate learning:
Followed student work and monitored their
progress using LMS tools.
Facilitated, motivated and encouraged students
using communication tools.
Stimulated communication and collaboration
among students.
Actively participated, promoted and led
interactive discussions.
Provided answers to questions, feedback and
recommendations on course activities.
The exam was the last activity in the blended
learning course and the examination took place in
the classroom. After the exam, the students filled out
questionnaires.
The research was carried out during a period of
one semester.
3.1.3 Measurements
All participants filled out a questionnaire where they
specified gender, age group, school, course,
employment status and expressed their opinion on
the quality of e-Learning environment and e-Course
quality.
As we expected that individuals would attempt
to quantify constructs, which have not been directly
measurable, we have used multiple-item scales and
summated ratings to quantify the constructs of
interest (i.e. e-Learning environment quality, e-
Course quality). The quality of e-Learning
environment is measured with a scale of 13
questions and e-Course quality is measured with a
set of 15 questions.
For all questions, 1-to-5 rating Likert scale (1-
strongly disagree, 2-disagree, 3-neutral, 4-agree, 5-
strongly agree) is used. New variable portal quality
is calculated as the arithmetic mean of 13 values,
which are measured with questions about aspects of
e-Learning environment quality. The variable e-
Course quality is calculated as the arithmetic mean
of 15 values is measured with questions about
aspects of e-Course quality. Both variables are
numerical.
3.1.4 Statistical Treatment
Since a new questionnaire is developed, it has to be
found out whether the instrument is reliable.
Therefore, Cronbach's alpha is computed to measure
the reliability (internal consistency) of scales for e-
Learning environment quality and e-Course quality.
Descriptive statistics are used to analyse
demographics data.
Pearson's correlation is performed to determine
if there is a significant relationship between the e-
Learning portal quality and e-Course quality. A
significance level of p < 0.05 is adopted for the
study.
SPSS is used for data analysis.
3.2 Results and Interpretation
3.2.1 Reliability of the Questionnaire
For each scale (e-Learning environment quality and
e-Course quality), the Cronbach’s α is used to check
how closely a set of items is related as a group.
Values of Cronbachs’ coefficients α are 0.913
(for the scale e-Learning environment quality) and
0.937 (for the scale e-Course quality). Since each
value is greater than 0.8, it can be concluded that the
questionnaire is sufficiently reliable. Alpha
coefficients, for both scales, are above 0.9, which
suggests that items have relatively high internal
consistency.
3.2.2 Participants
197 students participated in the study and they have
ranged from 19 to 64 years of age. 48% of
participants were males and 52% were females.
Participants used two different LMS systems. 53%
of them used Moodle and 47% of them used
eCampus. Each participant attended one of eight
e courses. Each e-Course was led by one of eight
teachers.
ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED QUALITY OF LMS SYSTEM AND E-COURSE
DELIVERY
313
3.2.3 Participant Satisfaction
Participants’ opinion on e-Learning environment
(i.e. e-Learning portal powered by LMS system)
quality and e-Course quality has been measured by
descriptive statistics.
Descriptive statistics of portal quality and e-
Course quality are reported in Table 1.
Table 1: Descriptive Statistics – all Students.
N Min Max Mean
Std.
Dev.
Portal Qual. 197 1.5 5.0 3.94 .66
e-Course Qual. 197 1.4 5.0 3.93 .69
As it can be seen, participants assessed e-portal
quality and e-Course quality rather high. The
expected mean of both variables was 3 (neither
unsatisfied nor satisfied). The measured values were
almost 4 in the scale from 1 to 5.
Students who have used the LMS system Moodle
have assessed their learning experienced as shown in
Table 2.
Table 2: Descriptive Statistics - Students Using Moodle.
N Min Max Mean
Std.
Dev.
Portal Qual. 103 1.7 4.8 3.75 .57
e-Course Qual. 103 1.4 4.9 3.69 .61
As it can be seen in Table 3, those students who
have studied via LMS system eCampus have been
on average a bit more satisfied.
Table 3: Descriptive Statistics for eCampus.
N Min Max Mean
Std.
Dev.
Portal Qual. 94 1.5 5.0 4.13 .69
e-Course Qual. 94 1.5 5.0 4.19 .69
The descriptive statistics suggest a relationship
between the variables. The question is, if this
relationship is statistically significant.
3.2.4 Significant Correlation between Portal
Quality and e-Course Quality
Correlation analysis is used to verify the relationship
between numeric variables of portal quality and e-
Course quality.
The following null hypothesis is set: there is no
relationship between portal quality and e-Course
quality in blended learning setting.
Firstly, it is checked as to which test is the most
appropriate for our data?
Pearson’s correlation is appropriate if two
conditions are met: both variables are normally
distributed and that the correlation between variables
is linear.
To determine whether the variables are normally
distributed, the Kolmogorov-Smirnov test is used to
confirm normal distribution of both variables.
Scattered plot (Figure 1) shows a positive linear
relationship between both quality measures.
The results from obtaining Pearson’s correlation
are shown in Table 4.
According to the obtained results, the null
hypothesis is rejected and it can be concluded that
there is a significant relationship between the portal
quality and e-Course quality, r=.804, p (one tailed)
<0.01.
Students who are satisfied with e-Course are also
satisfied with e-Learning environment and vice
versa.
Figure 1: Scatter plot.
Table 4: Correlation.
e-Course
quality
portal
quality
e-Course Pearson Correlation 1 .804
quality Sig. (1-tailed) .000
N 197 197
portal Pearson Correlation .804 1
quality Sig. (1-tailed) .000
N 197 197
4 CONCLUSIONS
The quality of e-Learning environment and e-Course
quality is measured, in this research, with the use of,
DEMA model, which has been developed in this
CSEDU 2011 - 3rd International Conference on Computer Supported Education
314
study. Evaluation results with this model prove that
the quality of e-Learning environment and the
quality of e-Course is significantly correlated.
On an average, it can be concluded that observed
e-Learning portals show a positive attitude in
relation to their use. e-Courses have also been
assessed above expected average. From the
evaluation, it was evident that there is a pattern in
which case:
Students who are satisfied with e-Courses are
also satisfied with e-Learning environment.
Students who are unsatisfied with e-Courses are
also unsatisfied with e-Learning environment.
However, it is not evident from this research, which
of these two factors (e-Course, e-Learning
environment) has a greater impact on students’
satisfaction. However, the role of the teacher, who
tutored the course in e-Learning environment, is
very important. He/she is the one who facilitates
students’ learning through the use of technology.
He/she should use it in a way that technology helps
students to learn efficiently and effectively. If LMS
system is of good quality and does not cause any
troubles to students, students probably assess
technology useful, if they like the teacher’s activities
and e-Learning materials.
The experiment was carried out in case of
blended learning courses with minor face to face
part. Further research will show if there is also a
significant relationship between e-Learning
environment quality and e-Course quality in the case
of complete e-Learning settings (i.e. courses without
any face to face meetings).
In this study, DEMA model includes short
questionnaire with 28 items. In future studies we
will apply more extensive and standardised
questionnaires for evaluating students’ satisfaction
with LMS system and e-Course as well as to verify
the significance of correlation between e-Learning
portal quality and e-Learning course quality.
REFERENCES
Ardito, C., Costabile, M., Marsico, M., Lanzilotti, R.,
Levialdi, S., Roselli, T., & Rossano, V. (2006). An
approach to usability evaluation of e-learning
applications. Universal Access in the Information
Society, 4(3), 270-283. doi: 10.1007/s10209-005-
0008-6.
Bates, P., & Obexer, R. (2005). Evaluating student-
centred teaching and learning strategies for Aviation
students using a quality framework for online learning
environments. In ASCILITE 2005 conference
proceedings: Balance, Fidelity, Mobility: Maintaining
the Momentum.
Costabile, M. F., De Marsico, M., Lanzilotti, R.,
Plantamura, V. L., & Roselli, T. (2005). On the
Usability Evaluation of E-Learning Applications.
Paper presented at the System Sciences, 2005. HICSS
'05. Proceedings of the 38th Annual Hawaii
International Conference on.
Debevc, M., & Lapuh Bele, J. (2008). Usability testing of
e-learning content as used in two learning
management systems. EURODL, (1). Retrieved from:
http://www.eurodl.org/?p=archives&year=2008&halfy
ear=1&article=296
Dringus, L. P., & Cohen M. S. (2005). An adaptable
Usability Heuristic Checklist for Online Courses. In
Proceedings of the ASEE/IEEE Frontiers in Education
Conference, Indianapolis, USA.
ECDL CTP (2010). European Computer Driving License
Certified Training Professional Programme.
Eisner, E. W. (1985) The Art of Educational Evaluation.
Lewes, Falmer Press.
Kirakowski, J. & Corbett, M. (1993). SUMI: The Software
Usability Measurement Inventory. British Journal of
Educational Technology, 24, 210-212.
Kirkpatrick, D., & Kirkpatrick, P. (2006). Evaluating
Training Programs (3rd ed.). San Francisco, CA,
Berrett-Koehler Publishers, Inc.
Martin-Blas, T., & Serrano-Fernandez, A. (2009). The role
of new technologies in the learning process: Moodle as
a teaching tool in Physics. Computers & Education,
52(1), 35-44.
Nielsen, J. (1994). Usability Engineering. San Francisco,
USA Morgan Kaufmann.
Phillips, R. (2005). We can’t evaluate e-learning if we
don’t know what we mean by evaluating e-learning!
Interact, 30, 3-6. Learning Technology Support
Service, University of Bristol.
Rosenberg, M. J. (2001). E-learning: Strategies for
delivering knowledge in the digital age. New York,
McGraw-Hill.
Squires, D., & Preece, J. (1999). Predicting quality in
educational software: Evaluating for learning, usability
and the synergy between them, Interacting with
Computers, 11(5), 467-483.
Zaharias, P., & Poylymenakou, A. (2009). Developing a
Usability Evaluation Method for e-Learning
Applications: Beyond Functional Usability.
International Journal of Human-Computer
Interaction, 25(1), 75-98.
ESTABLISHING RELATIONSHIP BETWEEN PERCEIVED QUALITY OF LMS SYSTEM AND E-COURSE
DELIVERY
315