Using Affordances and Constraints to Evaluate the Use of a

Formative e-Assessment System in Mathematics Education

Said Hadjerrouit

Institute of Mathematical Sciences, University of Agder, Kristiansand, Norway

Keywords: Affordance, Constraint, e-Assessment System, Formative Feedback, Mathematical Learning, Numbas.

Abstract: e-Assessment systems provide affordances for learning mathematics by means of formative feedback.

However, there is a lack of research on affordances of e-assessment systems, and work remains to be done

before evaluating their effect on mathematical learning. This paper uses the e-assessment system Numbas and

proposes a framework to capture the affordances and constraints of the system at the technological, student,

classroom, mathematics subject, and assessment level. The aim of the paper is to explore affordances and

constraints that emerge at these levels, and the effect of formative feedback on mathematical learning. Based

on the results, some concluding remarks and recommendations for future work are proposed.

1 INTRODUCTION

A core component of e-assessment systems involves

offering formative feedback to students about the

quality and level of their mathematical performance.

Formative feedback occurs in the course of

mathematical problem-solving and offers information

that allows students to change their behaviour and

way of thinking (Clark, 2012; Shute, 2008).

The potentialities of formative feedback, which is

an essential part of e-assessment systems, to make

contributions to mathematical learning are important.

However, although there has been great enthusiasm

about the potential of e-assessment systems to support

learning, there is a lack of research studies on their

affordances that might lead to enhanced student

mathematical understanding. Although several

research studies provide good examples of

mathematical learning by means of e-assessment

systems (Bokhove, & Drijvers, 2012; Fujita, Jones, &

Miyazaki, 2018; Gresalfi, & Barnes, 2016; Hoogland,

& Tout, 2018; Olsson, 2018), there has yet to be

systematic explorations into how affordances and

constraints of the systems might support or hinder

student mathematical learning.

This work aims at exploring the impacts of the e-

assessment system Numbas on students’

mathematical learning drawing on the theoretical

background consisting of two central issues:

formative feedback, on the one hand and affordances

and constraints, on the other hand. It addresses two

research questions: a) What are the affordances and

constraints that emerge at the technological, student,

classroom, mathematical, and assessment level when

students interact with Numbas? and b) How do

students experience Numbas formative feedback?

The contribution of this work is twofold. Firstly,

it applies affordances and constraints to Numbas.

Secondly, it assesses the formative feedback of

Numbas in terms of affordances and constraints.

2 THEORETICAL BACKGOUND

This theoretical background of the work consists of

two key elements: formative feedback, and

affordances and constraints.

2.1 Formative Feedback

Feedback is considered as “information with which a

learner can confirm, add to, and overwrite, tune, or

restructure information in memory, whether that

information is domain knowledge, meta-cognitive

knowledge, beliefs about self and tasks, or cognitive

tactics and strategies” (Winne, & Butler, 1994, p.

5740). The purpose of feedback is thus to restructure

and achieve change in student thinking. Feedback that

occurs in problem solving is called formative

feedback, in contrast to summative feedback, which

366

Hadjerrouit, S.

Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education.

DOI: 10.5220/0009352503660373

In Proceedings of the 12th International Conference on Computer Supported Education (CSEDU 2020) - Volume 1, pages 366-373

ISBN: 978-989-758-417-6

occurs at the end of an activity and does not normally

allow the students to change their thinking. Formative

feedback is normally given by a teacher or a peer, but

it could be viewed as being the result of the student’s

interaction with an e-assessment system.

Feedback in e-assessment systems is primarily

formative. Shute (2008) identified two main functions

of formative feedback: verification, that is simple

judgement of whether an answer is correct; and

elaboration that provides relevant cues to guide the

student towards the correct answer. Clark (2012)

states that the “objective of formative feedback is the

deep involvement of students in meta-cognitive

strategies such as personal goal-planning,

monitoring, and reflection” (p. 210), and, as such, it

is related to self-regulated learning. Likewise, Hattie

and Timperley (2007) identified two types of

feedback: task-based and process feedback. Task-

based feedback is about a task or product, such as

whether a response to a test is correct or incorrect.

Process based feedback is “information about the

processes underlying a task that can act as a cueing

mechanism and lead to more effective information

search and use of task strategies” (p. 93). Rakoczy et

al. (2013, p. 64) found that written process-oriented

feedback, that is “suggesting how and when a

particular strategy is appropriate” might foster

students’ mathematical learning. They argue that

while task-based feedback may be the least effective

form, it can help when the task information is

subsequently used for “improving strategy processing

or enhancing self-regulation” (pp. 90–91).

2.2 Affordances and Constraints

Gibson (1977) developed the concept of affordance to

describe the interactions between a goal-oriented

actor and an object in the environment in terms of

what it “affords” the actor, or in other words in terms

of action possibilities for meeting the actor’s goal.

According to Gibson, affordances are not intrinsic

properties of the object. Rather, affordances emerge

from the relationship between the object and the actor

with which it is interacting. Moreover, affordances

are not inherent characteristics of the object and

independent of the actor. Rather, affordances are

neither objective nor subjective properties: They

simply cut across the subjective-objective dichotomy.

Finally, affordances are not without constraints.

When one thing is afforded, something else is

simultaneously constrained. Affordances and

constraints are simply not separable, because

constraints are complementary and not the opposite

of affordances (Brown, Stillman, & Herbert, 2004).

The concept of affordance was introduced to the

Human-Computer Interaction community by Norman

(1988) to describe the perceived and actual properties

of the tool’s user interface to determine just how it

could possibly be used. Several research studies draw

on Gibson’s and Norman’s work to investigate the

concept of affordance in various educational settings.

For example, Kirchner et al. (2004) described a three-

layer definition of affordance: Technological

affordances that cover usability issues, educational

affordances that facilitate teaching and learning, and

social affordances to foster social interactions.

Likewise, Chiappini (2013) applied the notions of

perceived, ergonomic, and cultural affordances to

Alnuset, a digital tool for high school algebra. Finally,

Hadjerrouit (2017) proposed two types of affordances

at five different levels in teacher education:

Technological affordances at the ergonomic and

functional level, and pedagogical affordances at the

student, classroom, and subject level. Based on the

research literature, the specificities of mathematics

education and e-assessment systems, this work

proposes a model of affordances and constraints that

can emerge at five different levels: Technological

level, student level or mathematical task level,

classroom level or student-teacher interaction level,

mathematics subject level, and assessment level.

Given this background, the following affordances

may emerge at the technological level as students

interact with Numbas. These are ease-of-use, ease-of-

navigation, accessibility of Numbas at any time and

place, and accuracy and quick completion of

mathematical operations. Moreover, Numbas may

help to perform calculations, draw graphs and

functions, solve equations, and construct diagrams.

Secondly, several affordances may emerge at the

student level or mathematical task level:

 Numbas presents the mathematical content in

several ways using text, graphs, symbolic,

interactive diagrams, videos

 Numbas helps to transform expressions that

support conceptual understanding

 Numbas facilitates mathematical activities such

as exercises, multiple choice, quizzes, etc.

 Numbas is congruent with textbook and paper-

pencil mathematics

 Numbas offers a flexible way to handle a wide

range of assessment questions

Thirdly, several affordances may emerge at the

classroom or student-teacher interaction level:

 Numbas enables a high degree of autonomy

and help students to work on their own

 Numbas offers multiple levels of difficulty, and

can be adapted to different knowledge levels

Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education

367

 Numbas provides opportunities for the teacher

to make individual adjustments for students

 Numbas allows to choose the level of difficulty

 Numbas stimulates students to cooperate and

share their knowledge

Moreover, several affordances may emerge at the

mathematics subject level:

 Numbas offers a high quality of mathematical

content

 Numbas questions are useful to foster

reflections and higher-level mathematical

thinking

 Numbas provides opportunities to exploit the

constraints and limitations of the tool to provoke

students’ mathematical thinking

 Numbas provides opportunities to foster

conceptual rather than procedural understanding

 Numbas displays formulas, functions, graphs,

numbers, algebraic expressions, and

geometrical figures correctly

 Numbas simplifies mathematical expressions so

they look as if there are written on paper

Finally, several types of affordances may emerge at

the assessment level:

 Numbas provides several assessment tests, e.g.

questions, practical exercises, quizzes, etc.

 The order and wording of the assessment

questions in Numbas are appropriate

 The questions are relevant to test mathematical

knowledge

 Numbas gives immediate feedback

 Numbas provides several types of feedback such

as expected answers and advices to the solution,

and give hints to problem-solving step by step

 Numbas takes the profile and knowledge level

of the student into account and serves up

appropriate questions

 Numbas provides an answer to a question, and

whether it is correct or not

 Numbas provides a summary of the test,

students’ answers to questions, what they have

done wrong or right, and statistics on students’

answers to questions and their performances.

3 Numbas

Numbas is an e-assessment system with an emphasis

on formative feedback (Perfect, 2015). It is used to

create mathematical tasks that help teachers build

tests with videos, visualizations, and interactive

diagrams that they can use to challenge their students

individually. The primary design goal of Numbas is

to enable a student to submit a mathematical answer

in the form of an algebraic expression. The student

selects an option from a list of mathematical tasks

to the student, and generates information by, for

example, drawing graphs according to the student’s

submitted formulas and expressions. In Numbas,

feedback is often provided to students based on their

correct or incorrect answer to a mathematical task,

either immediately or with a small delay. Numbas can

also reveal the solution to the problem.

According to Perfect (2015), the great advantage

of Numbas is the large range of marking algorithms

and input types, which make it easy to assess a range

of answers to mathematical questions that are entered

by the students as symbolic expressions or as

numbers. Another advantage is that students can

access Numbas and produce a test through web

browsers without any set-up. Also, the randomisation

system, through the definition of question variables

and substitution into the question text is particularly

powerful compared to other e-assessment systems.

Figure 1 shows an example of test in Numbas:

Figure 1: Numbas test: Differentiation of a function.

The main constraint of Numbas is the limited range

of mathematical expressions the student can submit,

since each input must be automatically marked. It is

very difficult to set up a question that gives credit to

a student who does a lot of mathematical reasoning

while solving a problem, but fails to produce a final

result, because it is hard to capture the student’s

thinking process (Perfect, 2015).

4 THE STUDY

4.1 Context and Participants

This study was conducted in the context of a master

course on the use of digital tools for mathematical

learning in teacher education. The participants

(N=15) were students from one class enrolled in the

CSEDU 2020 - 12th International Conference on Computer Supported Education

368

course in 2018. The students had varied knowledge

background in mathematics ranging from arithmetics

and algebra to differentiation and derivation. They

had also varied experience with digital tools such as

Excel, GeoGebra, Khan Academy, etc. In terms of

mathematical knowledge, the basic requirement of

the course is the completion of a bachelor-degree in

teacher or mathematics education. In terms of digital

tools, the recommended prerequisites were basic

knowledge in digital technologies such as

spreadsheets, calculators and Internet. None of the

students had any prior experience with Numbas.

4.2 Methods

Teaching activities over a period of two weeks were

designed. These covered mathematical tasks at the

primary, middle and secondary level, which include

numbers, fractions, algebra, linear equations, and

differentiation.

Both quantitative and qualitative methods were

used to answer the research questions described in the

introductory section. Firstly, a survey questionnaire

with a five-point Likert scale from 1 to 5, and

quantitative analysis of the results, where 1 was coded

as the highest and 5 as the lowest. Secondly, Students’

comments in their own words on each of the items of

the survey, and open-ended questions to collect and

analyse supplementary information on the use of

Numbas. The data collection and analysis methods

were guided by the theoretical background in terms

of formative feedback, affordances and constraints,

and identification of central themes in students’

comments to bring to the fore information that was

not sufficiently covered by the survey.

5 RESULTS

Students’ perceptions of affordances and constraints,

and their views of formative feedback are presented

in the following sections. The results are presented in

qualitative rather than quantitative terms due to space

restrictions.

5.1 Affordances and Constraints at the

Technological Level

The survey results show that the vast majority of the

students pointed out that Numbas has a user-friendly

interface and that it is easy to use, to start and to exit.

Numbas has also a ready-made mathematical content

that can be extended to include more study material

using video lessons, simulations, animations, etc.

Technological affordances are reflected in students’

comments: “easy and fine design”; “easy to find and

navigate through the information”; “very positive that

we get immediate feedback”. No constraints have

been reported. These results show the importance of

a user-friendly interface for teachers and students.

5.2 Affordances and Constraints at the

Student or Mathematical Task Level

The survey results show that Numbas present the

mathematical content in several ways by means of

text, graphs, symbolic expressions, interactive

diagrams, videos, GeoGebra worksheets, etc.

Numbas also facilitates various mathematical

activities in terms of problem-solving, exercises,

multiple choice, quizzes, etc. It can be used to

reinforce textbooks mathematics. Likewise, Numbas

supports the delivery of mathematical tests outside

classroom, and it is flexible to handle a wide range of

assessment questions. No constraints have been

reported at the student level.

A qualitative analysis of the students’ comments

indicates three main themes: multiple representations

of tasks and variation, feedback with the teacher, and

rigidity and constraints of the tool.

Concerning the first theme, it seems that different

and multiple representations of the mathematical

tasks were highly valued by the students. These are

reflected in their comments:

Many similar technical tasks. Had been

interesting if we could enter GeoGebra tasks to

increase variation.

Good that one can use different representations

on the tasks, so that one can test different types of

understanding among the students. Good that the

students can be tested at home so that they can test

themselves how much they can.

There is a multi-representation of the

mathematical content, which is really important.

Likewise, students considered Numbas as an

alternative tool to traditional testing:

Can be used as a supplement, but students must

also have training in mathematical reasoning.

Seems this is a good alternative to traditional

testing. Can also be a good tool for testing students

who for various reasons cannot be tested at school.

The third theme is the rigidity and constraints of

Numbas in terms of quality of assessment in

comparison to human beings. The constraints make

teacher assistance necessary:

Seems to lack assessment skills. A program system

is rigid and has trouble seeing if the student is

thinking properly.

Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education

369

Numbas is designed to be an assessment tool, but

since it is a computer program and not a human, it

has some obvious limitations, e.g., customized

feedback beyond correct/wrong and general hints.

Programming errors (….) force students to ask

the teacher for assistance.

5.3 Affordances and Constraints at the

Student-teacher Interaction Level

Most participants think that Numbas enables a high

degree of autonomy for the students to work at their

own pace. Numbas also contains multiple levels of

difficulty, but it is up to the teacher to adjust the level

and make individual adjustments. Students also partly

agreed that they can ask the teacher for help, but most

of them did not need to use the textbook. Numbas

does normally not enable collaborative work.

Moreover, many students think that Numbas is not

designed to allow them to choose the level of

difficulty.

A qualitative analysis of participants’ comments

reveals four main themes: teacher help,

individualization, collaboration, and use of other

external resources such as textbooks and internet.

Regarding the first theme, students indicated that

they appreciate well the role of the teacher to provide

help, design and adapt tasks to their knowledge

levels. The constraints of the tool also make teacher

help necessary as already mentioned above. Some

representative comments are:

The teacher should adapt the tasks to how the

students respond and give easier tasks when the

students fail. Good that students can use hints and

help themselves.

Want to believe that the students will ask for help

despite hints and feedback from Numbas.

It is the teacher who makes the tasks for the

students, (…), and it is positive that they can design

more tasks of the same type and several times.

The students can work at their own pace, but it is

the teacher who decides how far they can proceed.

Good that it is not predetermined in Numbas.

Teacher help will always be needed when using

Numbas, as it has no feedback in terms of syntax

errors, especially when a student insists on the

correctness of his/her answer. Using a textbook is a

choice depending on the students’ judgment.

Regarding the second theme, most students

agreed that Numbas has an individual focus and that

individualization and adaptation of tasks at different

levels are important in the learning process:

Very individual focus.

Very good that Numbas provided the opportunity

for hints and feedback underway.

Numbas preserves the individuality of the process

of practicing.

Can provide various tests at different levels so

that the students themselves can choose the levels

they want to work with, possibly begin at an easy level

and move on to more difficult ones. In this way, they

can challenge themselves.

In contrast to the individual focus of Numbas, it

seems that students do not think that the tool provides

opportunities for collaborative problem-solving and

discussion with peers, even though collaboration is

considered important for many reasons. The teacher

may also play an important role in designing

collaborative tasks.

The assignment can be a good starting point for

collaboration where the students can explain how

they think.

The students can collaborate on certain tasks, but

basically Numbas stimulates individual work.

Numbas does not facilitate cooperation, but it

could provide problem solving tasks to promote

discussion.

Initially, Numbas seems to be designed for the

individual student, (…) but it does obviously not open

up for cooperation. That said, the teacher has of

course designed the tests/questions so that the

students can work together on them.

As a teacher, I can decide whether the students

will work together or alone.

The last theme is Numbas as supplementary

digital resource in addition to textbooks and other

resources available online. Some comments:

May be wise for the students to have the textbook

open. Numbas can be used without teacher help.

I use textbooks and the internet as well because

there was some topics of the mathematics I can’t quite

remember.

5.4 Affordances and Constraints at the

Mathematics Subject Level

Most students agreed that Numbas provides a high

quality of mathematical content, and that the

questions and tasks are well-designed and formulated.

Likewise, most students found that Numbas displays

mathematical notations and expressions correctly,

which means Numbas has a high degree of

mathematical fidelity. Moreover, Numbas is

mathematically correct and it simplifies mathematical

expressions. Likewise, many students think that

Numbas provides opportunities to foster

mathematical thinking through various entry points to

Numbas, such as “submit answer”, “submit part”, “try

another question like this one” or “reveal answers”,

CSEDU 2020 - 12th International Conference on Computer Supported Education

370

which help the students to decide on their own

whether they want to submit an answer or part of it,

let Numbas reveal the answers, or just choose another

similar question. In contrast to these positive

comments, more than the majority answered

negatively to issues related to conceptual

understanding, even though there is a relatively big

variation in their responses. Hence, some work

remains to be done to provide tasks that foster

conceptual understanding, metacognition, and high-

level thinking in mathematics (proving, reasoning), as

well as exploit the anomalies and constraints of the

tool (machine mathematics). This confirms the

designer’s view (Perfect, 2015) that it is difficult for

Numbas to capture student’s thinking and reseaoning

processes.

A qualitative analysis of the students’ comments

reveals three main themes: machine mathematics,

congruence of Numbas with paper-pencil techniques,

and conceptual understanding.

Machine mathematics is about the way Numbas

represents mathematics, e.g., numbers, arithmetic

operations or algebraic expressions. The following

comments highlight the constraints of machine

mathematics versus “ideal” mathematics:

Can be problematic if you don’t write “,” but must

use a dot for a decimal number. A fraction is also not

always mathematically correct if the numbers are

very large.

Writing fractions such 1/3 can be difficult when

the task is written in form of decimal numbers, and it

will be wrong if you use it, as the dot is the preferred

one. Both parts should be approved. The system is

rigid and unable to respond to the wrong answer.

In contrast to machine mathematics, some

students think there is a congruence and

complementarity between Numbas and paper-pencil

techniques in some situations:

It was very good that one could enter the formulas

in the fields and calculate the answer here.

Very good that Numbas writes my answer as you

see it on paper even though I write it differently.

I use paper to figure out the answer that one can

have in different steps and enter, not just the answer.

Numbas measures right / wrong and has little

focus on process (conceptual understanding) skills,

even though one can object that if a student gave the

correct answer, he/she might have understood the

mathematical concept.

In terms of conceptual understanding, students

think that this issue is dependent on the teacher and

his/her knowledge, and the way he/she designs the

questions and feedback. Errors in Numbas may also

foster reflection.

Depends on how the questions are asked.

Again, it really depends on whether the teacher

has designed and programmed the questions

correctly. On the other hand, errors in the program

can also help to stimulate reflection if they try to

understand what has gone wrong.

Something I think what is very positive is the given

response to answers, the possibility of hints and the

possibility of showing calculations. This information

can help the students to reinforce their

understanding. The teachers have a lot of power to

control how this program will affect the student.

The degree to which Numbas responds to the

criteria in the questions above, is entirely dependent

on the teacher who creates the questions, since the

program is very flexible in terms of how to create

these, as well as has more advanced features as

mentioned earlier.

5.5 Affordances and Constraints at the

Assessment Level

Most students think that Numbas provides several

assessment tests in terms of single questions,

exercises, quizzes, and multiple-choice questions as

well. Likewise, students think that the order of the

questions given to the students is appropriate. The

wording of the questions is understandable as well.

However, the students pointed out that these issues

depend entirely on the teacher who creates the

questions, but they also added that Numbas offers the

possibility of letting the order of questions be random.

In terms of formative feedback, most students

agreed that Numbas gives immediate feedback to a

question. It also provides several types of feedback

such as expected answers and advices to the solution.

For most students, Numbas provides a summary of

the test, students’ answers to questions, and what they

have done wrong or right, and in a lesser degree

whether it is correct or not. In contrast, some students

did not find that Numbas feedback contains useful

information that may help them understand the

exercises and answer the questions. Moreover, hints

in form of videos to problem-solving step by step

were not always useful. Most students also think that

Numbas provides statistics on students’ answers to

questions and their performance and grading. Finally,

according to the students, the most important

constraint of Numbas is that it does not consider the

profile and knowledge level of the student. This

confirms the designer’s view that Numbas cannot

capture students’ characteristics.

The most important themes that emerged from the

qualitative data analysis and emphasized by the

students are the affordances and constraints of

Numbas feedback and the role of the teacher in

Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education

371

designing the feedback rather than Numbas alone.

This confirms somehow the survey results.

(…) The forms of feedback both to the students

and teachers, as far as I can see, are not very good,

and therefore should not be based on such tests alone.

The fact that the students receive feedback right

away is positive, which means that they can make

self-assessments to a greater extent. In the event of a

difficult test, the result will not come from the teacher.

To some extent, it might have been better that

Numbas gives more concrete feedback if I had made

an obvious mistake as for example, a wrong sign.

I feel that the students get a little more control

over their own test results as they can choose how

much help and support that they want themselves.

Again, I think it depends much on the design of

individual tests that determine the degree to which

feedback satisfies the needs of individual students.

Feedback quality depends on pre-programmed

solutions.

(…) It depends on the teacher who creates the

questions, but it can be added that Numbas offers the

possibility of letting the order of questions be random.

(…) The summaries I have seen are not particularly

rich in terms of information and give the teacher little

hint about what the student can do.

6 DISCUSSION

The research questions addressed in this work are: a)

What are the affordances and constraints that emerge

at the technological, student, classroom,

mathematical, and assessment level when students

interact with Numbas? and b) How do students

experience Numbas formative feedback?

Regarding the first question, the study shows that

the affordance model was useful to capture and make

visible many of the potential affordances described in

section 2. Indeed, several affordances and constraints

emerged at the technological, student, classroom,

mathematics subject, and assessment level when

students interact with Numbas.

The technological affordances are ease-of-use,

ready-made mathematical content, and extensions to

include more study material such as video lessons,

simulations, and animations. This is possible, because

Numbas has an advanced extension system, which

enables the inclusion of a wide range of material and

subjects.

 A user-friendly interface with an

understandable language, and usability issues in

general are extremely important for both teacher

educators and students. At the student level, several

affordances emerged. The most important ones are

the presentation of the mathematical content in a wide

variety of ways and the facilitation of various

mathematical activities. The affordances also provide

opportunities to reinforce textbook-mathematics and

deliver a wide range of tests to the students based on

material from textbooks, but this is entirely dependent

on the teacher.

Both affordances and constraints emerged at the

classroom level. Firstly, Numbas enabled a high

degree of autonomy and individualization, and

allowed students to work at their own pace, test

mathematical tasks, and practice their skills.

Moreover, Numbas contains varied mathematical

tasks, but it is up to the teacher to design material with

multiple level of difficulty to challenge the students

and make individual adjustments. Finally, Numbas

does not stimulate students to cooperate and share

their knowledge, but it is possible for the teacher to

design collaborative tasks using Numbas.

At the mathematics subject level, Numbas

provided a high level of mathematical content that is

correct, sound, and congruent with textbooks and

paper-pencil mathematics. Numbas helps to test

problem-solving skills, and in a lesser degree

conceptual understanding and reasoning such as

proofs. Nevertheless, the teacher has the possibility to

assess some of these skills indirectly using the

available functionalities.

At the assessment level, Numbas provided several

assessment tasks to test students’ mathematical

knowledge, and in particular, the immediate

feedback, which was useful in terms of correctness of

answers, but it does not take into account the

student’s profile and knowledge level. This constraint

may be considered in future work, even though it is

hard to implement.

Regarding the second question, the participants

valued the feedback provided by Numbas as this was

helpful for mathematical problem-solving, even if it

does not automatically promote conceptual

understanding. In terms of feedback in comparison to

traditional testing, the study shows that the immediate

feedback of Numbas is important to many students,

but some felt it is limited as it provides mostly

wrong/right answers, which do not automatically

promote conceptual understanding and higher order-

thinking in mathematics as already mentioned above.

This is an important constraint that might be

considered in future designs and tests. Nevertheless,

the feedback function provided help and hint to test a

great spectrum of mathematical questions ranging

from primary to upper secondary school levels.

Clearly, Numbas revealed to be a good formative

assessment system for tasks that involve using an

algorithm for verifying whether a result is correct or

CSEDU 2020 - 12th International Conference on Computer Supported Education

372

not. Clearly, Numbas feedback made it easy to assess

a range of answers to mathematical questions that

students submit as algebraic expressions or as

numbers. The teacher can also benefit from the ease-

of-use of Numbas to create challenging mathematical

tasks with different and varied levels of difficulty.

7 CONCLUDING REMARKS

The results cannot be generalized due to the limited

number of participants (N=15). However, some

preliminary conclusions can be drawn for the use of

Numbas in teacher education.

Firstly, the study confirms that affordances and

constraints emerge at the technological, student,

classroom, mathematics subject, and assessment level

in the context of teacher education, where Numbas

was used to test students’ mathematical problem-

solving skills in a master course on the use of digital

tools for mathematical learning. The affordances and

constraints reported in this study are specific to the

particular context of teacher education.

Secondly, considering the affordances of Numbas

that emerged at the assessment level, it appears that a

combination of various types of feedback may be the

most effective form to support mathematical

understanding. The way Numbas shows where a

student has gone wrong, giving a full working

solution, and not only a right or wrong answer, giving

a detailed solution to a task with additional comments

on mistakes, and other mathematical misconceptions

provide useful information that can make students

more confident in their mathematical learning. Thus,

Numbas fulfils some of the functions described by

Shute (2008) and Hattie and Timperley (2007).

Nevertheless, teacher assistance is still important

because of the constraints and limitations of Numbas.

Future research will focus on both students’ and

teachers’ perspectives, and a triangulation of their

views. It will also include more varied tasks that

visualize mathematical concepts, resources such as

Geogebra dynamic figures and videos, and the ability

to let students make graphs that contribute to more

variety, and the opportunity for the teacher to design

intrinsically motivating tasks. Students will thus be

able to receive information and feedback tailored to

their activities, and teachers will receive better

feedback on both students’ successful and failed

solutions and their thinking processes. Finally,

collaborative tasks should be addressed in future

work as collaboration becomes increasingly

important in mathematics education.

REFERENCES

Bokhove, C., & Drijvers, P. (2012). Effects of feedback in

an online algebra intervention. Tech Know Learn 17,

pp. 43–59.

Brown, J.P., Stillman, G., Herbert, S. (2004). Can the

notion of affordances be of use in the design of a

technology enriched mathematics curriculum? 27th

Annual Conference of MERGA, pp. 119-126.

Gibson, J. J. (1977). The ecological approach to visual

perception. Houghton Mifflin, Boston.

Clark, I. (2012). Formative assessment: Assessment for

self-regulated learning. Educational Psychology

Review, 24(2), pp. 205–249.

Chiappini, G. (2013). Cultural affordances of digital

artifacts in the teaching and learning of mathematics.

Proceedings of ICTMT11.

Fujita, T., Jones, K., & Miyazaki, M. (2018). Learners’ use

of domain-specific computer-based feedback to

overcome logical circularity in deductive proving in

geometry. ZDM 50, pp. 699–713.

Gresalfi, M. S., & Barnes, J. (2016). Designing feedback in

an immersive videogame: supporting student

mathematical engagement. Education Tech Research

Dev 64, pp. 65–86.

Hadjerrouit, S. (2017). Assessing the affordances of

SimReal+ and their applicability to support the learning

of mathematics in teacher education. Issues in

Informing Science and Information Technology 14, pp.

121-138.

Hattie, J., & Timperley, H. (2007). The power of feedback.

Review of Educational Research, 2(1), pp. 81-112.

Hoogland, K., & Tout, D. (2018). Computer-based

assessment of mathematics into the twenty-first

century: pressures and tensions. ZDM 50, pp. 675–686

Kirchner, P., Strijbos, J-W., Kreijns, K., Beers, B. J. (2004):

Designing electronic collaborative learning

environments. Educational Technology Research and

Development, 52(3), pp. 47–66.

Norman, D.A. (1988). The psychology of everyday things.

Basic Books, New York.

Numbas. Retrieved from https://www.numbas.org.uk/

Olsson, J. (2018). The contribution of reasoning to the

utilization of feedback from software when solving

mathematical problems. Int Journal of Science and

Mathematics Education 16, pp.715–735.

Perfect, C. (2015). A demonstration of Numbas, an

assessment system for mathematical disciplines. CAA

Conference, pp. 1-8.

Rakoczy, K., Harks, B., Klieme, E., Blum, W., &

Hochweber, J. (2013). Written feedback in

mathematics: Mediated by students’ perception,

moderated by goal orientation. Learning and

Instruction, 27, pp. 63–73.

Shute, V. J. (2008). Focus on formative feedback. Review

of Educational Research, 78(1), pp. 153 -189.

Winne, P. H., & Butler, D. L. (1994). Student cognitive

processing and learning. In: T. Husen, & T. N.

Postlethwaite (Eds.). The International Encyclopedia of

Education (pp. 5739-5745). Oxford, UK: Pergamon.

Using Affordances and Constraints to Evaluate the Use of a Formative e-Assessment System in Mathematics Education

373