Usability and User Experience Evaluation of Learning Management Systems - A Systematic Mapping Study

Walter Takashi Nakamura, Elaine Harada Teixeira de Oliveira, Tayana Conte

2017

Abstract

Background: Advances in technology made possible the development of powerful platforms called Learning Management Systems (LMSs), designed to help the teaching and learning process. Studies show that usability and User Experience (UX) of such platforms may influence in this process. Although several studies had been conducted in this area, most of them are at initial stages and need improvements or deeper empirical studies. Aim: This work aims to analyze scientific publications in order to characterize the usability and UX evaluation techniques in the context of LMSs. Method: We performed a systematic mapping study regarding the usability and UX evaluation techniques in the context of LMSs. Results: A total of 62 publications were accepted in this mapping, which helped identifying the techniques used to evaluate the usability and UX of LMSs and their characteristics such as its origin, type, performing method, learning factors, restriction and availability. Conclusion: Several studies were conducted regarding the evaluation of LMSs. However, there are still some gaps such as the lack of techniques with some features, e.g., feedback with suggestions to correct the identified problems. Besides, there is no sufficient evidence of which of them is best suited for this context.

References

  1. Alkhattabi, M., 2015. Usability evaluation of Tadarus: Student perceptions. Procedia Computer Science Journal, 62(1), pp. 631-636.
  2. Ardito, C., Costabile, M. F., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., Rossano, V., 2006. An approach to usability evaluation of e-learning applications. Universal Access in the Information Society Journal, 4(3), pp. 270-283.
  3. Blecken, A., Brüggemann, D., Marx, W., 2010. Usability evaluation of a Learning Management System. In Proceedings of the 43rd Hawaii International Conference on System Sciences. Washington DC, USA: IEEE, pp. 1-9.
  4. Brusilovsky, P., 2004. KnowledgeTree: A distributed architecture for adaptive e-learning. In Proceedings of the 13th International World Wide Web conference on Alternate track papers & posters (WWW Alt. 7804). New York, USA: ACM Press.
  5. Cohen, J., 1960. A coefficient of agreement of nominal scales. Educational and Psychological Measurement, 20(1), pp.37-46.
  6. Cota, C. X. N., Díaz, A. I. M., Duque, M. Á. R., 2014. Evaluation framework for m-learning systems: Current situation and proposal. In Proceedings of the XV International Conference on Human Computer Interaction (INTERACCION 2014). New York, USA: ACM Press.
  7. De Carvalho, A. F. P., Anacleto, J. C., 2008. The importance of usability criteria on Learning Management Systems: Lessons learned. In Proceedings of the 10th International Conference on Enterprise Information Systems (ICEIS 2008). Barcelona, Spain: SciTePress, pp. 154-159.
  8. De Kock, E., Van Biljon, J., Pretorius, M., 2009. Usability evaluation methods: Mind the gaps. In Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists. Vaal River, South Africa: ACM Press, pp. 122-131.
  9. Dubost, A., Klein, M. R., Dang, J., 2004. Building interoperability between LMS and Brokerage Platforms. In Proceedings of the E-ACTIVITIES conference (WSEAS 2004). Crete, Greece.
  10. Fetaji, M., Dika, Z., Fetaji, B., 2008. Usability testing and evaluation of a mobile software solution: A case study. In Proceedings of the ITI 2008 30th International Conference on Information Technology Interfaces. Cavtat, Croatia: IEEE, pp. 501-506.
  11. Freire, L. L., Arezes, P. M., Campos, J. C., 2012. A literature review about usability evaluation methods for e-learning platforms. In 18th World congress on Ergonomics - Designing a sustainable future (IEA 2012). Recife, Brazil: IOS Press, pp. 1038-1044.
  12. Gordillo, A., Barra, E., Aguirre, S., Quemada, J., 2014. The usefulness of Usability and User Experience evaluation methods on an e-Learning platform development from a developer's perspective: A case study. In Frontiers in Education Conference (FIE) 2014. Madrid, Spain: IEEE.
  13. Granic, A., Cukušic, M., 2011. Usability testing and expert inspections complemented by educational evaluation: A case study of an e-learning platform. Journal of Educational Technology & Society, 14(2), pp. 107-123.
  14. Hamdi, N., Elbadrawy, R., Aziz, R. A., 2011. An experimental study to measure mobile learning system usability at the AASTMT. In International Conference Mobile Learning (IADIS 2011). Avila, Spain: IADIS Press, pp. 167-174.
  15. Hassenzahl, M., Platz, A., Burmester, M., Lehner, K., 2000. Hedonic and ergonomic quality aspects determine a software's appeal. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (CHI 7800). The Hague, Holland: ACM Press, pp. 201-208.
  16. Hassenzahl, M., Law, E. L-C., Hvannberg, E. T., 2006. User Experience - Towards a unified view. In: User Experience - Towards a Unified View: Second International COST294-MAUSE Open Workshop - NordICHI'06, pp. 1-3.
  17. Hijon-Neira, R., Velázquez-Iturbide, Á., Pizarro-Romero, C., Carriço, L., 2014. Merlin-know, an interactive virtual teacher for improving learning in Moodle. In Proceedings of Frontiers in Education Conference (FIE 2015). Madrid, Spain: IEEE.
  18. Ivory, M. Y., Hearst, M. A., 2001. The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4), pp. 470-516.
  19. Junus, I. S., Santoso, H. B., Isal, R. Y. K., Utomo, A. Y., 2015. Usability evaluation of the student centered elearning environment. International Review of Research in Open and Distributed Learning, 16(4), pp. 62-82.
  20. Kakasevski, G., Mihajlov, M., Arsenovski, S., Chungurski. S., 2008. Evaluating usability in Learning Management System Moodle. In 30th International Conference on Information Technology Interfaces (ITI 2008). Cavtat, Croatia: IEEE, pp. 613-618.
  21. Kitchenham, B., Chartes, S., 2007. Guidelines for performing systematic literature reviews in software engineering. Technical Report (EBSE 2007). Durham, UK: Keele.
  22. Landis, J.R. & Koch, G.G., 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), pp. 159-174.
  23. Lanzilotti, R., Ardito, C., Costabile, M. F., 2006. eLSE methodology: A systematic approach to the eLearning systems evaluation. Journal of Educational Technology & Society, 9(4), pp. 42-53.
  24. Lanzilotti, R., Ardito, C., Costabile, M. F., De Angeli, A., 2011. Do patterns help novice evaluators? A comparative study. International Journal of HumanComputer Studies, 69(1), pp. 52-69.
  25. Medina-Flores, R., Morales-Gamboa, R., 2015. Usability evaluation by experts of a Learning Management System. IEEE Revista Iberoamericana de Tecnologias del Aprendizaje, 10(4), pp. 197-2013.
  26. Mehlenbacher, B., Bennet, L., Bird, T., Ivey, M., Lucas, J., Morton, J., Whitman, L., 2005. Usable e-learning: A conceptual model for evaluation and design. In Proceedings of HCI International 2005: 11th International Conference on Human-Computer Interaction, 4(1), pp. 1-10.
  27. Mtebe, J. S., Kissaka, M. M., 2015. Heuristics for evaluating usability of Learning Management Systems in Africa. In IST-Africa 2015 Conference Proceedings. Lilongwe, Malawi: IEEE.
  28. Nielsen, J., (1994). Heuristic evaluation. In Usability Inspection Methods. New York, USA: John Wiley & Sons.
  29. Orfanou, K., Tselios, N., Katsanos, C., 2015. Perceived usability evaluation of Learning Management Systems: Empirical evaluation of the System Usability Scale. International Review of Research in Open and Distributed Learning, 16(2), pp. 227-246.
  30. Oztekin, A., Kong, Z. J., Uysal, O., 2010. UseLearn: A novel checklist and usability evaluation method for eLearning systems by criticality metric analysis. International Journal of Industrial Ergonomics, 40(4), pp. 455-469.
  31. Oztekin, A., Delen, D., Turkyilmaz, A., Zaim, S., 2013. A machine learning-based usability evaluation method for eLearning systems. Decision Support Systems Journal, 56(1), pp. 63-73.
  32. Pireva, K., Imran, A. S., Dalipi, F., 2015. User behaviour analysis on LMS and MOOC. IEEE Conference on eLearning, e-Management and e-Services (IC3e 2015). Malacca, Malaysia: IEEE, pp. 21-26.
  33. Reeves, T. C., Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., Loh, C. S., 2002. Usability and instructional design heuristics for e-learning evaluation. In ED-MEDIA 2002 World Conference on Educational Multimedia, Hypermedia & Telecommunications. Denver, USA: AACE, pp. 1615-1621.
  34. Rodrigues, J., Diniz, M., Ferreira, S. B. L., Silveira, D. S., Capra, E., 2011. Evaluation of usability in a remote learning system utilizing markov models. In Proceedings of the IADIS International Conference on WWW/Internet. Rio de Janeiro, Brazil: IADIS Press.
  35. Sánchez-Chamochin, E. L., Cambranes-Martínez, E., Menéndez-Domínguez, V., 2008. Adapting mobile access scheme for a legacy e-learning platform. In Mexican International Conference on Computer Science. Mexicali, Mexico: IEEE, pp. 96-103.
  36. Santoso, H. B., Isal, R. Y. K., Basaruddin, T., Sadita, L., 2014. Research-in-progress: User experience evaluation of student centered e-learning environment for computer science program. In 3rd International Conference on User Science and Engineering (iUSEr). Sha Alam, Malaysia, pp. 52-55.
  37. Shneiderman, B., Plaisant, C., 2010. Designing the user interface: Strategies for effective Human-Computer Interaction. Boston, MA, USA: Pearson.
  38. Squires, D., Preece, J., 1999. Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers Journal, 11(1), pp. 467-483.
  39. Ssekakubo, G., Suleman, H., Marsden, G., 2014. A streamlined mobile user-interface for improved access to LMS Services. In eLmL 2014: Sixth International Conference on Mobile, Hybrid, and On-line Learning. Barcelona, pp. 92-101.
  40. Ssemugabi, S., De Villiers, R., 2007. A comparative study of two usability evaluation methods using a web-based e-learning application. In Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists. Port Elizabeth, South Africa: ACM Press, pp. 132-142.
  41. Stickel, C., Scerbakov, A., Kaufmann, T., Ebner, M., 2008. Usability metrics of time and stress - Biological enhanced performance test of a university wide Learning Management System. HCI and Usability for Education and Work. Graz, Austria: Springer, pp. 173- 184.
  42. Tee, S. S., Wook, T. S. M. T., Zainudin, S., 2013. User testing for Moodle application. International Journal of Software Engineering and Its Applications, 7(5), pp. 243-252.
  43. Theng, Y-L., Sin, J., 2012. Evaluating usability and efficaciousness of an e-learning system: A quantitative, model-driven approach. In 12th IEEE International Conference on Advanced Learning Technologies. Rome, Italy: IEEE, pp. 303-307.
  44. Vogten, H., Koper, R., 2014. Towards a new generation of Learning Management Systems. Proceedings of the 6th International Conference on Computer Supported Education (CSEDU 2014). Barcelona, Spain: SciTePress, pp. 514-519.
  45. Wohlin, C., Höst, M., Regnell, B., 2012. Experimentation in Software Engineering. Verlag: Springer.
  46. Yusoff, S. R. M., Zin, N. A. M., 2014. Design and evaluation of Collaborative Learning Management System (CLMS) framework for teaching technical subject. In Proceedings of Revised Selected Papers on New Horizons in Web Based Learning, 7.697(1), pp. 79-89.
  47. Zaharias, P., 2006. A usability evaluation method for elearning: Focus on motivation to learn. In Proceedings of CHI EA 7806 Extended Abstracts on Human Factors in Computing Systems. New York, USA: ACM Press, pp. 1571-1576.
  48. Zaharias, P., 2008. Cross-cultural differences in perceptions of e-learning usability: An empirical investigation. International Journal of Technology and Human Interaction, 4(3), pp. 1-26.
  49. Zaharias, P., Poylymenakou, A., 2009. Developing a usability evaluation method for e-learning applications: Beyond functional usability. International Journal of Human-Computer Interaction, 25(1), pp. 75-98.
  50. Zaharias, P., Koutsabasis, P., 2012. Heuristic evaluation of e-learning courses: A comparative analysis of two elearning heuristic sets. Campus-Wide Information Systems. 29(1), pp. 45-60.
Download


Paper Citation


in Harvard Style

Takashi Nakamura W., Harada Teixeira de Oliveira E. and Conte T. (2017). Usability and User Experience Evaluation of Learning Management Systems - A Systematic Mapping Study . In Proceedings of the 19th International Conference on Enterprise Information Systems - Volume 3: ICEIS, ISBN 978-989-758-249-3, pages 97-108. DOI: 10.5220/0006363100970108


in Bibtex Style

@conference{iceis17,
author={Walter Takashi Nakamura and Elaine Harada Teixeira de Oliveira and Tayana Conte},
title={Usability and User Experience Evaluation of Learning Management Systems - A Systematic Mapping Study},
booktitle={Proceedings of the 19th International Conference on Enterprise Information Systems - Volume 3: ICEIS,},
year={2017},
pages={97-108},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006363100970108},
isbn={978-989-758-249-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 19th International Conference on Enterprise Information Systems - Volume 3: ICEIS,
TI - Usability and User Experience Evaluation of Learning Management Systems - A Systematic Mapping Study
SN - 978-989-758-249-3
AU - Takashi Nakamura W.
AU - Harada Teixeira de Oliveira E.
AU - Conte T.
PY - 2017
SP - 97
EP - 108
DO - 10.5220/0006363100970108