PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS

Christian Saul, Heinz-Dietrich Wuttke

2011

Abstract

As our society moves from an information-based to an innovation-based environment, it is not just important what you know, but how you can use your own knowledge in order to solve problems and create new knowledge. Hence, assessment systems need to evaluate not just the students' factual knowledge, but also their problem-solving and reasoning strategies. This leads to a demand for the assessment of higher-order thinking skills (HOTS). This paper analyzes HOTS and possibilities for their measurement. As a result, adaptive assessment systems (AASs) are in response to the emerging need of personalization while assessing HOTS. AASs take student’s individual context, prior knowledge and preferences into account in order to personalize the assessment. This personalized support helps students develop HOTS. But, this paper also reveals several arising issues, which need to be addressed when measuring HOTS with AASs.

References

  1. Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., et al., 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Addison Wesley Longman, Inc.
  2. Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., et al., 2001. A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives, Addison Wesley Longman, Inc.
  3. Bennett, R. E., 1998. Reinventing Assessment. Speculations on the Future of Large-Scale Educational Testing, Policy Information Center.
  4. Bennett, R. E., 1998. Reinventing Assessment. Speculations on the Future of Large-Scale Educational Testing, Policy Information Center.
  5. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. and Krathwohl, D. R., 1956. Taxonomy of Educational Objectives, Handbook 1: Cognitive Domain, Longman.
  6. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H. and Krathwohl, D. R., 1956. Taxonomy of Educational Objectives, Handbook 1: Cognitive Domain, Longman.
  7. Burstein, J., and Marcu, D., 2003. A Machine Learning Approach for Identification Thesis and Conclusion Statements in Student Essays. In Computers and the Humanities, 37(4), 455-467.
  8. Burstein, J., and Marcu, D., 2003. A Machine Learning Approach for Identification Thesis and Conclusion Statements in Student Essays. In Computers and the Humanities, 37(4), 455-467.
  9. Cleave-Hogg, D., Morgan, P. and Guest, C., 2000. Evaluation of Medical Students' Performance in Anaesthesia Using a CAE Med-Link Simulator System. In Proceedings of the Fourth International Computer Assisted Assessment Conference, 119-126.
  10. Cleave-Hogg, D., Morgan, P. and Guest, C., 2000. Evaluation of Medical Students' Performance in Anaesthesia Using a CAE Med-Link Simulator System. In Proceedings of the Fourth International Computer Assisted Assessment Conference, 119-126.
  11. Conejo , R., Guzman, E., Millan, E., Trella, M., Perez-dela-Cruz, J. and Rios, A., 2004. SIETTE: A Web-Based Tool for Adaptive Testing, In International Journal of Artificial Intelligence in Education, 14, 29-61.
  12. Conejo , R., Guzman, E., Millan, E., Trella, M., Perez-dela-Cruz, J. and Rios, A., 2004. SIETTE: A Web-Based Tool for Adaptive Testing, In International Journal of Artificial Intelligence in Education, 14, 29-61.
  13. Conole, G. and Warburton, B., 2005. A review of computer-assisted assessment, In Alt-J, Research in Learning Technology, 13(1), 17-31.
  14. Conole, G. and Warburton, B., 2005. A review of computer-assisted assessment, In Alt-J, Research in Learning Technology, 13(1), 17-31.
  15. Fadel, C., Honey, M. and Pasnik, S., 2007. Assessment in the Age of Innovation. In Education Week, 26(38), 34- 40.
  16. Fadel, C., Honey, M. and Pasnik, S., 2007. Assessment in the Age of Innovation. In Education Week, 26(38), 34- 40.
  17. Gouli, E., Papanikolaou, K. and Grigoriadou, M., 2002. Personalizing assessment in adaptive educational hypermedia systems. In Proceedings of the Second International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, 153-163.
  18. Gouli, E., Papanikolaou, K. and Grigoriadou, M., 2002. Personalizing assessment in adaptive educational hypermedia systems. In Proceedings of the Second International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, 153-163.
  19. IMS LIP, 2005. Learner Information Package, http://www.imsglobal.org/profiles
  20. IMS LIP, 2005. Learner Information Package, http://www.imsglobal.org/profiles
  21. IEEE PAPI, 2002. Public and Private Information, http://www.cen-ltso.net/Main.aspx?put=230
  22. IEEE PAPI, 2002. Public and Private Information, http://www.cen-ltso.net/Main.aspx?put=230
  23. IMS QTI, 2006. Question and Test Interoperability, http://www.imsglobal.org/question
  24. IMS QTI, 2006. Question and Test Interoperability, http://www.imsglobal.org/question
  25. King, F. J., Goodson, L. and Rohani, F., 1998. Higher Order Thinking Skills, Retrieved January 31, 2011, from http://www.cala.fsu.edu/files/higher_order_ thinking _skills.pdf.
  26. King, F. J., Goodson, L. and Rohani, F., 1998. Higher Order Thinking Skills, Retrieved January 31, 2011, from http://www.cala.fsu.edu/files/higher_order_ thinking _skills.pdf.
  27. Kozloff, M. A. and Wilmington N. C., 2002. Three requirements of effective instruction: Providing sufficient scaffolding, helping students organize and activate knowledge, and sustaining high engaged time.
  28. Kozloff, M. A. and Wilmington N. C., 2002. Three requirements of effective instruction: Providing sufficient scaffolding, helping students organize and activate knowledge, and sustaining high engaged time.
  29. Lalos, P., Retalis, S. and Psaromiligkos, Y., 2005. Creating personalised quizzes both to the learner and to the access device characteristics: the Case of CosyQTI. In Proceedings of Workshop on Authoring of Adaptive and Adaptable Educational Hypermedia, 1-7.
  30. Lalos, P., Retalis, S. and Psaromiligkos, Y., 2005. Creating personalised quizzes both to the learner and to the access device characteristics: the Case of CosyQTI. In Proceedings of Workshop on Authoring of Adaptive and Adaptable Educational Hypermedia, 1-7.
  31. Lankes, A. M. D., 1995. Electronic portfolios: A new idea in assessment, Retrieved February 10, 2011, from www.eric.ed.gov.
  32. Lankes, A. M. D., 1995. Electronic portfolios: A new idea in assessment, Retrieved February 10, 2011, from www.eric.ed.gov.
  33. Lazarinis, F., Green, S. and Pearson, E., 2009. Focusing on content reusability and interoperability in a personalized hypermedia assessment tool. In Multimedia Tools and Applications, 47(2), 257-278.
  34. Lazarinis, F., Green, S. and Pearson, E., 2009. Focusing on content reusability and interoperability in a personalized hypermedia assessment tool. In Multimedia Tools and Applications, 47(2), 257-278.
  35. Lewis, A. and Smith, D., 1993. Defining Higher Order Thinking. In Theory into Practice, 32(3), 131-137.
  36. Lewis, A. and Smith, D., 1993. Defining Higher Order Thinking. In Theory into Practice, 32(3), 131-137.
  37. Mitchell, T., Russell, T., Broomhead, P. and Aldridge, N., 2002. Towards robust computerised marking of freetext responses. In Proceedings of the 6th CAA Conference, 233-249.
  38. Mitchell, T., Russell, T., Broomhead, P. and Aldridge, N., 2002. Towards robust computerised marking of freetext responses. In Proceedings of the 6th CAA Conference, 233-249.
  39. Norris, S., 1989. Can we test validly for critical thinking? In Educational Researcher, 18(9), 21-26.
  40. Norris, S., 1989. Can we test validly for critical thinking? In Educational Researcher, 18(9), 21-26.
  41. Pitkow, J. and Recker, M., 1995. Using the Web as a survey tool: results from the second WWW user survey, In Computer Networks and ISDN Systems, 27, 809-822.
  42. Pitkow, J. and Recker, M., 1995. Using the Web as a survey tool: results from the second WWW user survey, In Computer Networks and ISDN Systems, 27, 809-822.
  43. Rice, J. W., 2007. Assessing higher order thinking in video games. In Journal of Technology and Teacher Education, 15(1), 87-100.
  44. Rice, J. W., 2007. Assessing higher order thinking in video games. In Journal of Technology and Teacher Education, 15(1), 87-100.
  45. Sangwin, C. J., 2003. Assessing higher mathematical skills using computer algebra marking through AIM. In Proceedings of the Engineering Mathematics and Applications Conference (EMAC03), 229-234.
  46. Sangwin, C. J., 2003. Assessing higher mathematical skills using computer algebra marking through AIM. In Proceedings of the Engineering Mathematics and Applications Conference (EMAC03), 229-234.
  47. Saul, C., Runardotter, M. and Wuttke, H.-D., 2010. Towards Feedback Personalisation in Adaptive Assessment, In Proceedings of the Sixth EDEN Research Workshop.
  48. Saul, C., Runardotter, M. and Wuttke, H.-D., 2010. Towards Feedback Personalisation in Adaptive Assessment, In Proceedings of the Sixth EDEN Research Workshop.
  49. Shepherd, E. and Godwin, J., 2004. Assessments through the Learning Process, Retrieved December 3, 2010, from http://www.questionmark.com/us/whitepapers.
  50. Shepherd, E. and Godwin, J., 2004. Assessments through the Learning Process, Retrieved December 3, 2010, from http://www.questionmark.com/us/whitepapers.
  51. Sugrue, B., 1995. A Theory-Based Framework for Assessing Domain-Specific Problem-Solving Ability. In Educational Measurement: Issues and Practice, 14(3), 29-35.
  52. Sugrue, B., 1995. A Theory-Based Framework for Assessing Domain-Specific Problem-Solving Ability. In Educational Measurement: Issues and Practice, 14(3), 29-35.
  53. Thomas, R., Ashton, H., Austin, B., Beevers, C., Edwards, D. and Milligan, C., 2004. Assessing Higher Order Skills Using Simulations. In Proceedings of the 8th CAA Conference.
  54. Thomas, R., Ashton, H., Austin, B., Beevers, C., Edwards, D. and Milligan, C., 2004. Assessing Higher Order Skills Using Simulations. In Proceedings of the 8th CAA Conference.
  55. Thomas, R., Ashton, H.S., Austin, B., et al., 2005. Cost effective use of simulations in Online Assessment. In Proceedings of the 9th CAA Conference.
  56. Thomas, R., Ashton, H.S., Austin, B., et al., 2005. Cost effective use of simulations in Online Assessment. In Proceedings of the 9th CAA Conference.
  57. Van der Linden, W. and Glas, C., 2000. Computerized adaptive testing: Theory and practice, Springer Netherlands.
  58. Van der Linden, W. and Glas, C., 2000. Computerized adaptive testing: Theory and practice, Springer Netherlands.
  59. Wainer, H., Dorans, N., Eignor, D., Flaugher, R., Green, B. and Mislevy, R., 2000. Computerized Adaptive Testing: A Primer, Lawrence Erlbaum Associates.
  60. Wainer, H., Dorans, N., Eignor, D., Flaugher, R., Green, B. and Mislevy, R., 2000. Computerized Adaptive Testing: A Primer, Lawrence Erlbaum Associates.
  61. Wuttke, H.-D., Ubar, R., Henke, K., Jutman, A., 2008. The synthesis level in Blooms Taxonomy a nightmare for an LMS. In Proceedings of the 19th EAEEIE Annual Conference, 199-204.
  62. Wuttke, H.-D., Ubar, R., Henke, K., Jutman, A., 2008. The synthesis level in Blooms Taxonomy a nightmare for an LMS. In Proceedings of the 19th EAEEIE Annual Conference, 199-204.
Download


Paper Citation


in Harvard Style

Saul C. and Wuttke H. (2011). PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS . In Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011) ISBN 978-989-8425-50-8, pages 425-430. DOI: 10.5220/0003480204250430


in Harvard Style

Saul C. and Wuttke H. (2011). PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS . In Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011) ISBN 978-989-8425-50-8, pages 425-430. DOI: 10.5220/0003480204250430


in Bibtex Style

@conference{attel11,
author={Christian Saul and Heinz-Dietrich Wuttke},
title={PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS},
booktitle={Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011)},
year={2011},
pages={425-430},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003480204250430},
isbn={978-989-8425-50-8},
}


in Bibtex Style

@conference{attel11,
author={Christian Saul and Heinz-Dietrich Wuttke},
title={PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS},
booktitle={Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011)},
year={2011},
pages={425-430},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003480204250430},
isbn={978-989-8425-50-8},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011)
TI - PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS
SN - 978-989-8425-50-8
AU - Saul C.
AU - Wuttke H.
PY - 2011
SP - 425
EP - 430
DO - 10.5220/0003480204250430


in EndNote Style

TY - CONF
JO - Proceedings of the 3rd International Conference on Computer Supported Education - Volume 2: ATTeL, (CSEDU 2011)
TI - PERSONALIZED ASSESSMENT OF HIGHER-ORDER THINKING SKILLS
SN - 978-989-8425-50-8
AU - Saul C.
AU - Wuttke H.
PY - 2011
SP - 425
EP - 430
DO - 10.5220/0003480204250430