Using Affordances and Constraints to Evaluate the Use of a
Formative e-Assessment System in Mathematics Education
Said Hadjerrouit
Institute of Mathematical Sciences, University of Agder, Kristiansand, Norway
Keywords: Affordance, Constraint, e-Assessment System, Formative Feedback, Mathematical Learning, Numbas.
Abstract: e-Assessment systems provide affordances for learning mathematics by means of formative feedback.
However, there is a lack of research on affordances of e-assessment systems, and work remains to be done
before evaluating their effect on mathematical learning. This paper uses the e-assessment system Numbas and
proposes a framework to capture the affordances and constraints of the system at the technological, student,
classroom, mathematics subject, and assessment level. The aim of the paper is to explore affordances and
constraints that emerge at these levels, and the effect of formative feedback on mathematical learning. Based
on the results, some concluding remarks and recommendations for future work are proposed.
A core component of e-assessment systems involves
offering formative feedback to students about the
quality and level of their mathematical performance.
Formative feedback occurs in the course of
mathematical problem-solving and offers information
that allows students to change their behaviour and
way of thinking (Clark, 2012; Shute, 2008).
The potentialities of formative feedback, which is
an essential part of e-assessment systems, to make
contributions to mathematical learning are important.
However, although there has been great enthusiasm
about the potential of e-assessment systems to support
learning, there is a lack of research studies on their
affordances that might lead to enhanced student
mathematical understanding. Although several
research studies provide good examples of
mathematical learning by means of e-assessment
systems (Bokhove, & Drijvers, 2012; Fujita, Jones, &
Miyazaki, 2018; Gresalfi, & Barnes, 2016; Hoogland,
& Tout, 2018; Olsson, 2018), there has yet to be
systematic explorations into how affordances and
constraints of the systems might support or hinder
student mathematical learning.
This work aims at exploring the impacts of the e-
assessment system Numbas on students’
mathematical learning drawing on the theoretical
background consisting of two central issues:
formative feedback, on the one hand and affordances
and constraints, on the other hand. It addresses two
research questions: a) What are the affordances and
constraints that emerge at the technological, student,
classroom, mathematical, and assessment level when
students interact with Numbas? and b) How do
students experience Numbas formative feedback?
The contribution of this work is twofold. Firstly,
it applies affordances and constraints to Numbas.
Secondly, it assesses the formative feedback of
Numbas in terms of affordances and constraints.
This theoretical background of the work consists of
two key elements: formative feedback, and
affordances and constraints.
2.1 Formative Feedback
Feedback is considered as “information with which a
learner can confirm, add to, and overwrite, tune, or
restructure information in memory, whether that
information is domain knowledge, meta-cognitive
knowledge, beliefs about self and tasks, or cognitive
tactics and strategies” (Winne, & Butler, 1994, p.
5740). The purpose of feedback is thus to restructure
and achieve change in student thinking. Feedback that
occurs in problem solving is called formative
feedback, in contrast to summative feedback, which
Hadjerrouit, S.
Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education.
DOI: 10.5220/0009352503660373
In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 1, pages 366-373
ISBN: 978-989-758-417-6
2020 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
occurs at the end of an activity and does not normally
allow the students to change their thinking. Formative
feedback is normally given by a teacher or a peer, but
it could be viewed as being the result of the student’s
interaction with an e-assessment system.
Feedback in e-assessment systems is primarily
formative. Shute (2008) identified two main functions
of formative feedback: verification, that is simple
judgement of whether an answer is correct; and
elaboration that provides relevant cues to guide the
student towards the correct answer. Clark (2012)
states that the “objective of formative feedback is the
deep involvement of students in meta-cognitive
strategies such as personal goal-planning,
monitoring, and reflection” (p. 210), and, as such, it
is related to self-regulated learning. Likewise, Hattie
and Timperley (2007) identified two types of
feedback: task-based and process feedback. Task-
based feedback is about a task or product, such as
whether a response to a test is correct or incorrect.
Process based feedback is “information about the
processes underlying a task that can act as a cueing
mechanism and lead to more effective information
search and use of task strategies” (p. 93). Rakoczy et
al. (2013, p. 64) found that written process-oriented
feedback, that is “suggesting how and when a
particular strategy is appropriate” might foster
students’ mathematical learning. They argue that
while task-based feedback may be the least effective
form, it can help when the task information is
subsequently used for “improving strategy processing
or enhancing self-regulation” (pp. 90–91).
2.2 Affordances and Constraints
Gibson (1977) developed the concept of affordance to
describe the interactions between a goal-oriented
actor and an object in the environment in terms of
what it “affords” the actor, or in other words in terms
of action possibilities for meeting the actor’s goal.
According to Gibson, affordances are not intrinsic
properties of the object. Rather, affordances emerge
from the relationship between the object and the actor
with which it is interacting. Moreover, affordances
are not inherent characteristics of the object and
independent of the actor. Rather, affordances are
neither objective nor subjective properties: They
simply cut across the subjective-objective dichotomy.
Finally, affordances are not without constraints.
When one thing is afforded, something else is
simultaneously constrained. Affordances and
constraints are simply not separable, because
constraints are complementary and not the opposite
of affordances (Brown, Stillman, & Herbert, 2004).
The concept of affordance was introduced to the
Human-Computer Interaction community by Norman
(1988) to describe the perceived and actual properties
of the tool’s user interface to determine just how it
could possibly be used. Several research studies draw
on Gibson’s and Norman’s work to investigate the
concept of affordance in various educational settings.
For example, Kirchner et al. (2004) described a three-
layer definition of affordance: Technological
affordances that cover usability issues, educational
affordances that facilitate teaching and learning, and
social affordances to foster social interactions.
Likewise, Chiappini (2013) applied the notions of
perceived, ergonomic, and cultural affordances to
Alnuset, a digital tool for high school algebra. Finally,
Hadjerrouit (2017) proposed two types of affordances
at five different levels in teacher education:
Technological affordances at the ergonomic and
functional level, and pedagogical affordances at the
student, classroom, and subject level. Based on the
research literature, the specificities of mathematics
education and e-assessment systems, this work
proposes a model of affordances and constraints that
can emerge at five different levels: Technological
level, student level or mathematical task level,
classroom level or student-teacher interaction level,
mathematics subject level, and assessment level.
Given this background, the following affordances
may emerge at the technological level as students
interact with Numbas. These are ease-of-use, ease-of-
navigation, accessibility of Numbas at any time and
place, and accuracy and quick completion of
mathematical operations. Moreover, Numbas may
help to perform calculations, draw graphs and
functions, solve equations, and construct diagrams.
Secondly, several affordances may emerge at the
student level or mathematical task level:
Numbas presents the mathematical content in
several ways using text, graphs, symbolic,
interactive diagrams, videos
Numbas helps to transform expressions that
support conceptual understanding
Numbas facilitates mathematical activities such
as exercises, multiple choice, quizzes, etc.
Numbas is congruent with textbook and paper-
pencil mathematics
Numbas offers a flexible way to handle a wide
range of assessment questions
Thirdly, several affordances may emerge at the
classroom or student-teacher interaction level:
Numbas enables a high degree of autonomy
and help students to work on their own
Numbas offers multiple levels of difficulty, and
can be adapted to different knowledge levels
Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education
Numbas provides opportunities for the teacher
to make individual adjustments for students
Numbas allows to choose the level of difficulty
Numbas stimulates students to cooperate and
share their knowledge
Moreover, several affordances may emerge at the
mathematics subject level:
Numbas offers a high quality of mathematical
Numbas questions are useful to foster
reflections and higher-level mathematical
Numbas provides opportunities to exploit the
constraints and limitations of the tool to provoke
students’ mathematical thinking
Numbas provides opportunities to foster
conceptual rather than procedural understanding
Numbas displays formulas, functions, graphs,
numbers, algebraic expressions, and
geometrical figures correctly
Numbas simplifies mathematical expressions so
they look as if there are written on paper
Finally, several types of affordances may emerge at
the assessment level:
Numbas provides several assessment tests, e.g.
questions, practical exercises, quizzes, etc.
The order and wording of the assessment
questions in Numbas are appropriate
The questions are relevant to test mathematical
Numbas gives immediate feedback
Numbas provides several types of feedback such
as expected answers and advices to the solution,
and give hints to problem-solving step by step
Numbas takes the profile and knowledge level
of the student into account and serves up
appropriate questions
Numbas provides an answer to a question, and
whether it is correct or not
Numbas provides a summary of the test,
students’ answers to questions, what they have
done wrong or right, and statistics on students’
answers to questions and their performances.
3 Numbas
Numbas is an e-assessment system with an emphasis
on formative feedback (Perfect, 2015). It is used to
create mathematical tasks that help teachers build
tests with videos, visualizations, and interactive
diagrams that they can use to challenge their students
individually. The primary design goal of Numbas is
to enable a student to submit a mathematical answer
in the form of an algebraic expression. The student
selects an option from a list of mathematical tasks
designed by the teacher. Numbas provides feedback
to the student, and generates information by, for
example, drawing graphs according to the student’s
submitted formulas and expressions. In Numbas,
feedback is often provided to students based on their
correct or incorrect answer to a mathematical task,
either immediately or with a small delay. Numbas can
also reveal the solution to the problem.
According to Perfect (2015), the great advantage
of Numbas is the large range of marking algorithms
and input types, which make it easy to assess a range
of answers to mathematical questions that are entered
by the students as symbolic expressions or as
numbers. Another advantage is that students can
access Numbas and produce a test through web
browsers without any set-up. Also, the randomisation
system, through the definition of question variables
and substitution into the question text is particularly
powerful compared to other e-assessment systems.
Figure 1 shows an example of test in Numbas:
Figure 1: Numbas test: Differentiation of a function.
The main constraint of Numbas is the limited range
of mathematical expressions the student can submit,
since each input must be automatically marked. It is
very difficult to set up a question that gives credit to
a student who does a lot of mathematical reasoning
while solving a problem, but fails to produce a final
result, because it is hard to capture the student’s
thinking process (Perfect, 2015).
4.1 Context and Participants
This study was conducted in the context of a master
course on the use of digital tools for mathematical
learning in teacher education. The participants
(N=15) were students from one class enrolled in the
CSEDU 2020 - 12th International Conference on Computer Supported Education
course in 2018. The students had varied knowledge
background in mathematics ranging from arithmetics
and algebra to differentiation and derivation. They
had also varied experience with digital tools such as
Excel, GeoGebra, Khan Academy, etc. In terms of
mathematical knowledge, the basic requirement of
the course is the completion of a bachelor-degree in
teacher or mathematics education. In terms of digital
tools, the recommended prerequisites were basic
knowledge in digital technologies such as
spreadsheets, calculators and Internet. None of the
students had any prior experience with Numbas.
4.2 Methods
Teaching activities over a period of two weeks were
designed. These covered mathematical tasks at the
primary, middle and secondary level, which include
numbers, fractions, algebra, linear equations, and
Both quantitative and qualitative methods were
used to answer the research questions described in the
introductory section. Firstly, a survey questionnaire
with a five-point Likert scale from 1 to 5, and
quantitative analysis of the results, where 1 was coded
as the highest and 5 as the lowest. Secondly, Students’
comments in their own words on each of the items of
the survey, and open-ended questions to collect and
analyse supplementary information on the use of
Numbas. The data collection and analysis methods
were guided by the theoretical background in terms
of formative feedback, affordances and constraints,
and identification of central themes in students’
comments to bring to the fore information that was
not sufficiently covered by the survey.
Students’ perceptions of affordances and constraints,
and their views of formative feedback are presented
in the following sections. The results are presented in
qualitative rather than quantitative terms due to space
5.1 Affordances and Constraints at the
Technological Level
The survey results show that the vast majority of the
students pointed out that Numbas has a user-friendly
interface and that it is easy to use, to start and to exit.
Numbas has also a ready-made mathematical content
that can be extended to include more study material
using video lessons, simulations, animations, etc.
Technological affordances are reflected in students’
comments: “easy and fine design”; “easy to find and
navigate through the information”; “very positive that
we get immediate feedback”. No constraints have
been reported. These results show the importance of
a user-friendly interface for teachers and students.
5.2 Affordances and Constraints at the
Student or Mathematical Task Level
The survey results show that Numbas present the
mathematical content in several ways by means of
text, graphs, symbolic expressions, interactive
diagrams, videos, GeoGebra worksheets, etc.
Numbas also facilitates various mathematical
activities in terms of problem-solving, exercises,
multiple choice, quizzes, etc. It can be used to
reinforce textbooks mathematics. Likewise, Numbas
supports the delivery of mathematical tests outside
classroom, and it is flexible to handle a wide range of
assessment questions. No constraints have been
reported at the student level.
A qualitative analysis of the students’ comments
indicates three main themes: multiple representations
of tasks and variation, feedback with the teacher, and
rigidity and constraints of the tool.
Concerning the first theme, it seems that different
and multiple representations of the mathematical
tasks were highly valued by the students. These are
reflected in their comments:
Many similar technical tasks. Had been
interesting if we could enter GeoGebra tasks to
increase variation.
Good that one can use different representations
on the tasks, so that one can test different types of
understanding among the students. Good that the
students can be tested at home so that they can test
themselves how much they can.
There is a multi-representation of the
mathematical content, which is really important.
Likewise, students considered Numbas as an
alternative tool to traditional testing:
Can be used as a supplement, but students must
also have training in mathematical reasoning.
Seems this is a good alternative to traditional
testing. Can also be a good tool for testing students
who for various reasons cannot be tested at school.
The third theme is the rigidity and constraints of
Numbas in terms of quality of assessment in
comparison to human beings. The constraints make
teacher assistance necessary:
Seems to lack assessment skills. A program system
is rigid and has trouble seeing if the student is
thinking properly.
Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education
Numbas is designed to be an assessment tool, but
since it is a computer program and not a human, it
has some obvious limitations, e.g., customized
feedback beyond correct/wrong and general hints.
Programming errors (….) force students to ask
the teacher for assistance.
5.3 Affordances and Constraints at the
Student-teacher Interaction Level
Most participants think that Numbas enables a high
degree of autonomy for the students to work at their
own pace. Numbas also contains multiple levels of
difficulty, but it is up to the teacher to adjust the level
and make individual adjustments. Students also partly
agreed that they can ask the teacher for help, but most
of them did not need to use the textbook. Numbas
does normally not enable collaborative work.
Moreover, many students think that Numbas is not
designed to allow them to choose the level of
A qualitative analysis of participants’ comments
reveals four main themes: teacher help,
individualization, collaboration, and use of other
external resources such as textbooks and internet.
Regarding the first theme, students indicated that
they appreciate well the role of the teacher to provide
help, design and adapt tasks to their knowledge
levels. The constraints of the tool also make teacher
help necessary as already mentioned above. Some
representative comments are:
The teacher should adapt the tasks to how the
students respond and give easier tasks when the
students fail. Good that students can use hints and
help themselves.
Want to believe that the students will ask for help
despite hints and feedback from Numbas.
It is the teacher who makes the tasks for the
students, (…), and it is positive that they can design
more tasks of the same type and several times.
The students can work at their own pace, but it is
the teacher who decides how far they can proceed.
Good that it is not predetermined in Numbas.
Teacher help will always be needed when using
Numbas, as it has no feedback in terms of syntax
errors, especially when a student insists on the
correctness of his/her answer. Using a textbook is a
choice depending on the students’ judgment.
Regarding the second theme, most students
agreed that Numbas has an individual focus and that
individualization and adaptation of tasks at different
levels are important in the learning process:
Very individual focus.
Very good that Numbas provided the opportunity
for hints and feedback underway.
Numbas preserves the individuality of the process
of practicing.
Can provide various tests at different levels so
that the students themselves can choose the levels
they want to work with, possibly begin at an easy level
and move on to more difficult ones. In this way, they
can challenge themselves.
In contrast to the individual focus of Numbas, it
seems that students do not think that the tool provides
opportunities for collaborative problem-solving and
discussion with peers, even though collaboration is
considered important for many reasons. The teacher
may also play an important role in designing
collaborative tasks.
The assignment can be a good starting point for
collaboration where the students can explain how
they think.
The students can collaborate on certain tasks, but
basically Numbas stimulates individual work.
Numbas does not facilitate cooperation, but it
could provide problem solving tasks to promote
Initially, Numbas seems to be designed for the
individual student, (…) but it does obviously not open
up for cooperation. That said, the teacher has of
course designed the tests/questions so that the
students can work together on them.
As a teacher, I can decide whether the students
will work together or alone.
The last theme is Numbas as supplementary
digital resource in addition to textbooks and other
resources available online. Some comments:
May be wise for the students to have the textbook
open. Numbas can be used without teacher help.
I use textbooks and the internet as well because
there was some topics of the mathematics I can’t quite
5.4 Affordances and Constraints at the
Mathematics Subject Level
Most students agreed that Numbas provides a high
quality of mathematical content, and that the
questions and tasks are well-designed and formulated.
Likewise, most students found that Numbas displays
mathematical notations and expressions correctly,
which means Numbas has a high degree of
mathematical fidelity. Moreover, Numbas is
mathematically correct and it simplifies mathematical
expressions. Likewise, many students think that
Numbas provides opportunities to foster
mathematical thinking through various entry points to
Numbas, such as “submit answer”, “submit part”, “try
another question like this one” or “reveal answers”,
CSEDU 2020 - 12th International Conference on Computer Supported Education
which help the students to decide on their own
whether they want to submit an answer or part of it,
let Numbas reveal the answers, or just choose another
similar question. In contrast to these positive
comments, more than the majority answered
negatively to issues related to conceptual
understanding, even though there is a relatively big
variation in their responses. Hence, some work
remains to be done to provide tasks that foster
conceptual understanding, metacognition, and high-
level thinking in mathematics (proving, reasoning), as
well as exploit the anomalies and constraints of the
tool (machine mathematics). This confirms the
designers view (Perfect, 2015) that it is difficult for
Numbas to capture student’s thinking and reseaoning
A qualitative analysis of the students’ comments
reveals three main themes: machine mathematics,
congruence of Numbas with paper-pencil techniques,
and conceptual understanding.
Machine mathematics is about the way Numbas
represents mathematics, e.g., numbers, arithmetic
operations or algebraic expressions. The following
comments highlight the constraints of machine
mathematics versus “ideal” mathematics:
Can be problematic if you don’t write “,” but must
use a dot for a decimal number. A fraction is also not
always mathematically correct if the numbers are
very large.
Writing fractions such 1/3 can be difficult when
the task is written in form of decimal numbers, and it
will be wrong if you use it, as the dot is the preferred
one. Both parts should be approved. The system is
rigid and unable to respond to the wrong answer.
In contrast to machine mathematics, some
students think there is a congruence and
complementarity between Numbas and paper-pencil
techniques in some situations:
It was very good that one could enter the formulas
in the fields and calculate the answer here.
Very good that Numbas writes my answer as you
see it on paper even though I write it differently.
I use paper to figure out the answer that one can
have in different steps and enter, not just the answer.
Numbas measures right / wrong and has little
focus on process (conceptual understanding) skills,
even though one can object that if a student gave the
correct answer, he/she might have understood the
mathematical concept.
In terms of conceptual understanding, students
think that this issue is dependent on the teacher and
his/her knowledge, and the way he/she designs the
questions and feedback. Errors in Numbas may also
foster reflection.
Depends on how the questions are asked.
Again, it really depends on whether the teacher
has designed and programmed the questions
correctly. On the other hand, errors in the program
can also help to stimulate reflection if they try to
understand what has gone wrong.
Something I think what is very positive is the given
response to answers, the possibility of hints and the
possibility of showing calculations. This information
can help the students to reinforce their
understanding. The teachers have a lot of power to
control how this program will affect the student.
The degree to which Numbas responds to the
criteria in the questions above, is entirely dependent
on the teacher who creates the questions, since the
program is very flexible in terms of how to create
these, as well as has more advanced features as
mentioned earlier.
5.5 Affordances and Constraints at the
Assessment Level
Most students think that Numbas provides several
assessment tests in terms of single questions,
exercises, quizzes, and multiple-choice questions as
well. Likewise, students think that the order of the
questions given to the students is appropriate. The
wording of the questions is understandable as well.
However, the students pointed out that these issues
depend entirely on the teacher who creates the
questions, but they also added that Numbas offers the
possibility of letting the order of questions be random.
In terms of formative feedback, most students
agreed that Numbas gives immediate feedback to a
question. It also provides several types of feedback
such as expected answers and advices to the solution.
For most students, Numbas provides a summary of
the test, students’ answers to questions, and what they
have done wrong or right, and in a lesser degree
whether it is correct or not. In contrast, some students
did not find that Numbas feedback contains useful
information that may help them understand the
exercises and answer the questions. Moreover, hints
in form of videos to problem-solving step by step
were not always useful. Most students also think that
Numbas provides statistics on students’ answers to
questions and their performance and grading. Finally,
according to the students, the most important
constraint of Numbas is that it does not consider the
profile and knowledge level of the student. This
confirms the designer’s view that Numbas cannot
capture students’ characteristics.
The most important themes that emerged from the
qualitative data analysis and emphasized by the
students are the affordances and constraints of
Numbas feedback and the role of the teacher in
Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education
designing the feedback rather than Numbas alone.
This confirms somehow the survey results.
(…) The forms of feedback both to the students
and teachers, as far as I can see, are not very good,
and therefore should not be based on such tests alone.
The fact that the students receive feedback right
away is positive, which means that they can make
self-assessments to a greater extent. In the event of a
difficult test, the result will not come from the teacher.
To some extent, it might have been better that
Numbas gives more concrete feedback if I had made
an obvious mistake as for example, a wrong sign.
I feel that the students get a little more control
over their own test results as they can choose how
much help and support that they want themselves.
Again, I think it depends much on the design of
individual tests that determine the degree to which
feedback satisfies the needs of individual students.
Feedback quality depends on pre-programmed
(…) It depends on the teacher who creates the
questions, but it can be added that Numbas offers the
possibility of letting the order of questions be random.
(…) The summaries I have seen are not particularly
rich in terms of information and give the teacher little
hint about what the student can do.
The research questions addressed in this work are: a)
What are the affordances and constraints that emerge
at the technological, student, classroom,
mathematical, and assessment level when students
interact with Numbas? and b) How do students
experience Numbas formative feedback?
Regarding the first question, the study shows that
the affordance model was useful to capture and make
visible many of the potential affordances described in
section 2. Indeed, several affordances and constraints
emerged at the technological, student, classroom,
mathematics subject, and assessment level when
students interact with Numbas.
The technological affordances are ease-of-use,
ready-made mathematical content, and extensions to
include more study material such as video lessons,
simulations, and animations. This is possible, because
Numbas has an advanced extension system, which
enables the inclusion of a wide range of material and
A user-friendly interface with an
understandable language, and usability issues in
general are extremely important for both teacher
educators and students. At the student level, several
affordances emerged. The most important ones are
the presentation of the mathematical content in a wide
variety of ways and the facilitation of various
mathematical activities. The affordances also provide
opportunities to reinforce textbook-mathematics and
deliver a wide range of tests to the students based on
material from textbooks, but this is entirely dependent
on the teacher.
Both affordances and constraints emerged at the
classroom level. Firstly, Numbas enabled a high
degree of autonomy and individualization, and
allowed students to work at their own pace, test
mathematical tasks, and practice their skills.
Moreover, Numbas contains varied mathematical
tasks, but it is up to the teacher to design material with
multiple level of difficulty to challenge the students
and make individual adjustments. Finally, Numbas
does not stimulate students to cooperate and share
their knowledge, but it is possible for the teacher to
design collaborative tasks using Numbas.
At the mathematics subject level, Numbas
provided a high level of mathematical content that is
correct, sound, and congruent with textbooks and
paper-pencil mathematics. Numbas helps to test
problem-solving skills, and in a lesser degree
conceptual understanding and reasoning such as
proofs. Nevertheless, the teacher has the possibility to
assess some of these skills indirectly using the
available functionalities.
At the assessment level, Numbas provided several
assessment tasks to test students’ mathematical
knowledge, and in particular, the immediate
feedback, which was useful in terms of correctness of
answers, but it does not take into account the
student’s profile and knowledge level. This constraint
may be considered in future work, even though it is
hard to implement.
Regarding the second question, the participants
valued the feedback provided by Numbas as this was
helpful for mathematical problem-solving, even if it
does not automatically promote conceptual
understanding. In terms of feedback in comparison to
traditional testing, the study shows that the immediate
feedback of Numbas is important to many students,
but some felt it is limited as it provides mostly
wrong/right answers, which do not automatically
promote conceptual understanding and higher order-
thinking in mathematics as already mentioned above.
This is an important constraint that might be
considered in future designs and tests. Nevertheless,
the feedback function provided help and hint to test a
great spectrum of mathematical questions ranging
from primary to upper secondary school levels.
Clearly, Numbas revealed to be a good formative
assessment system for tasks that involve using an
algorithm for verifying whether a result is correct or
CSEDU 2020 - 12th International Conference on Computer Supported Education
not. Clearly, Numbas feedback made it easy to assess
a range of answers to mathematical questions that
students submit as algebraic expressions or as
numbers. The teacher can also benefit from the ease-
of-use of Numbas to create challenging mathematical
tasks with different and varied levels of difficulty.
The results cannot be generalized due to the limited
number of participants (N=15). However, some
preliminary conclusions can be drawn for the use of
Numbas in teacher education.
Firstly, the study confirms that affordances and
constraints emerge at the technological, student,
classroom, mathematics subject, and assessment level
in the context of teacher education, where Numbas
was used to test students’ mathematical problem-
solving skills in a master course on the use of digital
tools for mathematical learning. The affordances and
constraints reported in this study are specific to the
particular context of teacher education.
Secondly, considering the affordances of Numbas
that emerged at the assessment level, it appears that a
combination of various types of feedback may be the
most effective form to support mathematical
understanding. The way Numbas shows where a
student has gone wrong, giving a full working
solution, and not only a right or wrong answer, giving
a detailed solution to a task with additional comments
on mistakes, and other mathematical misconceptions
provide useful information that can make students
more confident in their mathematical learning. Thus,
Numbas fulfils some of the functions described by
Shute (2008) and Hattie and Timperley (2007).
Nevertheless, teacher assistance is still important
because of the constraints and limitations of Numbas.
Future research will focus on both students’ and
teachers’ perspectives, and a triangulation of their
views. It will also include more varied tasks that
visualize mathematical concepts, resources such as
Geogebra dynamic figures and videos, and the ability
to let students make graphs that contribute to more
variety, and the opportunity for the teacher to design
intrinsically motivating tasks. Students will thus be
able to receive information and feedback tailored to
their activities, and teachers will receive better
feedback on both students’ successful and failed
solutions and their thinking processes. Finally,
collaborative tasks should be addressed in future
work as collaboration becomes increasingly
important in mathematics education.
Bokhove, C., & Drijvers, P. (2012). Effects of feedback in
an online algebra intervention. Tech Know Learn 17,
pp. 43–59.
Brown, J.P., Stillman, G., Herbert, S. (2004). Can the
notion of affordances be of use in the design of a
technology enriched mathematics curriculum? 27th
Annual Conference of MERGA, pp. 119-126.
Gibson, J. J. (1977). The ecological approach to visual
perception. Houghton Mifflin, Boston.
Clark, I. (2012). Formative assessment: Assessment for
self-regulated learning. Educational Psychology
Review, 24(2), pp. 205–249.
Chiappini, G. (2013). Cultural affordances of digital
artifacts in the teaching and learning of mathematics.
Proceedings of ICTMT11.
Fujita, T., Jones, K., & Miyazaki, M. (2018). Learners’ use
of domain-specific computer-based feedback to
overcome logical circularity in deductive proving in
geometry. ZDM 50, pp. 699–713.
Gresalfi, M. S., & Barnes, J. (2016). Designing feedback in
an immersive videogame: supporting student
mathematical engagement. Education Tech Research
Dev 64, pp. 65–86.
Hadjerrouit, S. (2017). Assessing the affordances of
SimReal+ and their applicability to support the learning
of mathematics in teacher education. Issues in
Informing Science and Information Technology 14, pp.
Hattie, J., & Timperley, H. (2007). The power of feedback.
Review of Educational Research, 2(1), pp. 81-112.
Hoogland, K., & Tout, D. (2018). Computer-based
assessment of mathematics into the twenty-first
century: pressures and tensions. ZDM 50, pp. 675–686
Kirchner, P., Strijbos, J-W., Kreijns, K., Beers, B. J. (2004):
Designing electronic collaborative learning
environments. Educational Technology Research and
Development, 52(3), pp. 47–66.
Norman, D.A. (1988). The psychology of everyday things.
Basic Books, New York.
Numbas. Retrieved from
Olsson, J. (2018). The contribution of reasoning to the
utilization of feedback from software when solving
mathematical problems. Int Journal of Science and
Mathematics Education 16, pp.715–735.
Perfect, C. (2015). A demonstration of Numbas, an
assessment system for mathematical disciplines. CAA
Conference, pp. 1-8.
Rakoczy, K., Harks, B., Klieme, E., Blum, W., &
Hochweber, J. (2013). Written feedback in
mathematics: Mediated by students’ perception,
moderated by goal orientation. Learning and
Instruction, 27, pp. 63–73.
Shute, V. J. (2008). Focus on formative feedback. Review
of Educational Research, 78(1), pp. 153 -189.
Winne, P. H., & Butler, D. L. (1994). Student cognitive
processing and learning. In: T. Husen, & T. N.
Postlethwaite (Eds.). The International Encyclopedia of
Education (pp. 5739-5745). Oxford, UK: Pergamon.
Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education