Christian Saul, Heinz-Dietrich Wuttke


As our society moves from an information-based to an innovation-based environment, it is not just important what you know, but how you can use your own knowledge in order to solve problems and create new knowledge. Hence, assessment systems need to evaluate not just the students' factual knowledge, but also their problem-solving and reasoning strategies. This leads to a demand for the assessment of higher-order thinking skills (HOTS). This paper analyzes HOTS and possibilities for their measurement. As a result, adaptive assessment systems (AASs) are in response to the emerging need of personalization while assessing HOTS. AASs take student’s individual context, prior knowledge and preferences into account in order to personalize the assessment. This personalized support helps students develop HOTS. But, this paper also reveals several arising issues, which need to be addressed when measuring HOTS with AASs.


  1. Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., et al., 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Addison Wesley Longman, Inc.
  2. Bennett, R. E., 1998. Reinventing Assessment. Speculations on the Future of Large-Scale Educational Testing, Policy Information Center.
  3. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. and Krathwohl, D. R., 1956. Taxonomy of Educational Objectives, Handbook 1: Cognitive Domain, Longman.
  4. Burstein, J., and Marcu, D., 2003. A Machine Learning Approach for Identification Thesis and Conclusion Statements in Student Essays. In Computers and the Humanities, 37(4), 455-467.
  5. Cleave-Hogg, D., Morgan, P. and Guest, C., 2000. Evaluation of Medical Students' Performance in Anaesthesia Using a CAE Med-Link Simulator System. In Proceedings of the Fourth International Computer Assisted Assessment Conference, 119-126.
  6. Conejo , R., Guzman, E., Millan, E., Trella, M., Perez-dela-Cruz, J. and Rios, A., 2004. SIETTE: A Web-Based Tool for Adaptive Testing, In International Journal of Artificial Intelligence in Education, 14, 29-61.
  7. Conole, G. and Warburton, B., 2005. A review of computer-assisted assessment, In Alt-J, Research in Learning Technology, 13(1), 17-31.
  8. Fadel, C., Honey, M. and Pasnik, S., 2007. Assessment in the Age of Innovation. In Education Week, 26(38), 34- 40.
  9. Gouli, E., Papanikolaou, K. and Grigoriadou, M., 2002. Personalizing assessment in adaptive educational hypermedia systems. In Proceedings of the Second International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, 153-163.
  10. IMS LIP, 2005. Learner Information Package,
  11. IEEE PAPI, 2002. Public and Private Information,
  12. IMS QTI, 2006. Question and Test Interoperability,
  13. King, F. J., Goodson, L. and Rohani, F., 1998. Higher Order Thinking Skills, Retrieved January 31, 2011, from thinking _skills.pdf.
  14. Kozloff, M. A. and Wilmington N. C., 2002. Three requirements of effective instruction: Providing sufficient scaffolding, helping students organize and activate knowledge, and sustaining high engaged time.
  15. Lalos, P., Retalis, S. and Psaromiligkos, Y., 2005. Creating personalised quizzes both to the learner and to the access device characteristics: the Case of CosyQTI. In Proceedings of Workshop on Authoring of Adaptive and Adaptable Educational Hypermedia, 1-7.
  16. Lankes, A. M. D., 1995. Electronic portfolios: A new idea in assessment, Retrieved February 10, 2011, from
  17. Lazarinis, F., Green, S. and Pearson, E., 2009. Focusing on content reusability and interoperability in a personalized hypermedia assessment tool. In Multimedia Tools and Applications, 47(2), 257-278.
  18. Lewis, A. and Smith, D., 1993. Defining Higher Order Thinking. In Theory into Practice, 32(3), 131-137.
  19. Mitchell, T., Russell, T., Broomhead, P. and Aldridge, N., 2002. Towards robust computerised marking of freetext responses. In Proceedings of the 6th CAA Conference, 233-249.
  20. Norris, S., 1989. Can we test validly for critical thinking? In Educational Researcher, 18(9), 21-26.
  21. Pitkow, J. and Recker, M., 1995. Using the Web as a survey tool: results from the second WWW user survey, In Computer Networks and ISDN Systems, 27, 809-822.
  22. Rice, J. W., 2007. Assessing higher order thinking in video games. In Journal of Technology and Teacher Education, 15(1), 87-100.
  23. Sangwin, C. J., 2003. Assessing higher mathematical skills using computer algebra marking through AIM. In Proceedings of the Engineering Mathematics and Applications Conference (EMAC03), 229-234.
  24. Saul, C., Runardotter, M. and Wuttke, H.-D., 2010. Towards Feedback Personalisation in Adaptive Assessment, In Proceedings of the Sixth EDEN Research Workshop.
  25. Shepherd, E. and Godwin, J., 2004. Assessments through the Learning Process, Retrieved December 3, 2010, from
  26. Sugrue, B., 1995. A Theory-Based Framework for Assessing Domain-Specific Problem-Solving Ability. In Educational Measurement: Issues and Practice, 14(3), 29-35.
  27. Thomas, R., Ashton, H., Austin, B., Beevers, C., Edwards, D. and Milligan, C., 2004. Assessing Higher Order Skills Using Simulations. In Proceedings of the 8th CAA Conference.
  28. Thomas, R., Ashton, H.S., Austin, B., et al., 2005. Cost effective use of simulations in Online Assessment. In Proceedings of the 9th CAA Conference.
  29. Van der Linden, W. and Glas, C., 2000. Computerized adaptive testing: Theory and practice, Springer Netherlands.
  30. Wainer, H., Dorans, N., Eignor, D., Flaugher, R., Green, B. and Mislevy, R., 2000. Computerized Adaptive Testing: A Primer, Lawrence Erlbaum Associates.
  31. Wuttke, H.-D., Ubar, R., Henke, K., Jutman, A., 2008. The synthesis level in Blooms Taxonomy a nightmare for an LMS. In Proceedings of the 19th EAEEIE Annual Conference, 199-204.

Paper Citation

in Harvard Style

Saul C. and Wuttke H. (2011). PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS . In Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011) ISBN 978-989-8425-50-8, pages 425-430. DOI: 10.5220/0003480204250430

in Bibtex Style

author={Christian Saul and Heinz-Dietrich Wuttke},
booktitle={Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011)},

in EndNote Style

JO - Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011)
SN - 978-989-8425-50-8
AU - Saul C.
AU - Wuttke H.
PY - 2011
SP - 425
EP - 430
DO - 10.5220/0003480204250430