Towards Computer-based Exams in CS1

Vesa Lappalainen, Antti-Jussi Lakanen, Harri Högmander

2017

Abstract

Even though IDEs are often a central tool when learning to program in CS1, many teachers still lean on paper-based exams. In this study, we examine the “test mode effect” in CS1 exams using the Rainfall problem. The test mode was two-phased. Half of the participants started working on the problem with pen and paper, while the other half had access to an IDE. After submitting their solution, all students could rework their solution on an IDE. The experiment was repeated twice during subsequent course instances. The results were mixed. From the marking perspective, there was no statistically significant difference resulting from the mode. However, the students starting with the paper-based part tended to make more errors in their code, but after the computer-based reworking phase, they matched or exceeded the level of the students who started with the computer-based phase. We also discuss the reliability of automatic assessment that is based on a unit test suite that was developed for the purposes of this study.

References

  1. Bodmann, S. M. and Robinson, D. H. (2004). Speed and performance differences among Computer-Based and Paper-Pencil tests. Journal of Educational Computing Research, 31(1):51-60.
  2. Carter, J., Mutka, K. A., Fuller, U., Dick, M., English, J., Fone, W., and Sheard, J. (2003). How shall we assess this? SIGCSE Bull., 35(4):107-123.
  3. de Raadt, M. (2009). Teaching Programming Strategies Explicitly to Novice Programmers: Can the Way We Teach Strategies Improve Novice Outcomes? VDM Verlag, Saarbrücken, Germany, Germany.
  4. Ebrahimi, A. (1994). Novice programmer errors: language constructs and plan composition. International Journal of Human-Computer Studies, 41(4):457-480.
  5. Fisler, K. (2014). The recurring rainfall problem. In Proceedings of the Tenth Annual Conference on International Computing Education Research, ICER 7814, pages 35-42, New York, NY, USA. ACM.
  6. Grissom, S., Murphy, L., McCauley, R., and Fitzgerald, S. (2016). Paper vs. computer-based exams: A study of errors in recursive binary tree algorithms. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education, SIGCSE 7816, pages 6-11, New York, NY, USA. ACM.
  7. Isomöttönen, V. and Lappalainen, V. (2012). CSI with games and an emphasis on TDD and unit testing: piling a trend upon a trend. ACM Inroads, 3(3):62-68.
  8. Lakanen, A. J., Lappalainen, V., and Isomöttönen, V. (2015). Revisiting rainfall to explore exam questions and performance on CS1. In Proceedings of the 15th Koli Calling Conference on Computing Education Research, Koli Calling 7815, pages 40-49, New York, NY, USA. ACM.
  9. Leeson, H. V. (2006). The mode effect: A literature review of human and technological issues in computerized testing. International Journal of Testing, 6(1):1- 24.
  10. McDonald, A. S. (2002). The impact of individual differences on the equivalence of computer-based and paper-and-pencil educational assessments. Computers & Education, 39(3):299-312.
  11. Ozalp-Yaman, S. and C¸agiltay, N. E. (2010). Paper-based versus computer-based testing in engineering education. In IEEE EDUCON Education Engineering. IEEE.
  12. Russell, M., Goldberg, A., and O'connor, K. (2003). Computer-based testing and validity: a look back into the future. Assessment in Education: Principles, Policy & Practice, 10(3):279-293.
  13. Seppälä, O., Ihantola, P., Isohanni, E., Sorva, J., and Vihavainen, A. (2015). Do we know how difficult the rainfall problem is? In Proceedings of the 15th Koli Calling Conference on Computing Education Research, Koli Calling 7815, pages 87-96, New York, NY, USA. ACM.
  14. Simon (2013). Soloway's rainfall problem has become harder. In Learning and Teaching in Computing and Engineering (LaTiCE), 2013, pages 130-135. IEEE.
  15. Soloway, E. (1986). Learning to program= learning to construct mechanisms and explanations. Communications of the ACM, 29(9):850-858.
  16. Venables, A., Tan, G., and Lister, R. (2009). A closer look at tracing, explaining and code writing skills in the novice programmer. In Proceedings of the Fifth International Workshop on Computing Education Research Workshop, ICER 7809, pages 117-128, New York, NY, USA. ACM.
Download


Paper Citation


in Harvard Style

Lappalainen V., Lakanen A. and Högmander H. (2017). Towards Computer-based Exams in CS1 . In Proceedings of the 9th International Conference on Computer Supported Education - Volume 2: CSEDU, ISBN 978-989-758-240-0, pages 125-136. DOI: 10.5220/0006323501250136


in Bibtex Style

@conference{csedu17,
author={Vesa Lappalainen and Antti-Jussi Lakanen and Harri Högmander},
title={Towards Computer-based Exams in CS1},
booktitle={Proceedings of the 9th International Conference on Computer Supported Education - Volume 2: CSEDU,},
year={2017},
pages={125-136},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006323501250136},
isbn={978-989-758-240-0},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 9th International Conference on Computer Supported Education - Volume 2: CSEDU,
TI - Towards Computer-based Exams in CS1
SN - 978-989-758-240-0
AU - Lappalainen V.
AU - Lakanen A.
AU - Högmander H.
PY - 2017
SP - 125
EP - 136
DO - 10.5220/0006323501250136