How Do Mathematics Teachers in Higher Education Look at

E-assessment with Multiple-Choice Questions

Jose Manuel Azevedo

1,*

, Ema Patrícia Oliveira

and Patrícia Damas Beites

Instituto Politécnico do Porto (ISCAP/IPP), Rua Jaime Lopes de Amorim, Porto, Portugal

Departamento de Psicologia e Educação and LabCom.IFP, Universidade da Beira Interior, Covilhã, Portugal

Departamento de Matemática and CMA-UBI, Universidade da Beira Interior, Covilhã, Portugal

Keywords: E–assessment, Multiple-Choice Questions, Mathematics, Higher Education, Qualitative Data Analysis.

Abstract: This paper presents a part of a global proposal aimed to create and put forward an e–assessment strategy using

tests with multiple–choice questions (MCQ) implemented in Moodle. This strategy was planned in order to

allow the use of continuous summative assessment in mathematics’ courses in a higher education institution,

in classes with a large number of students. The main goal of this work was to analyse how this procedure can

affect the teaching and learning processes. Changes in educational practices were ascertained using interviews

with teachers. It was found an improvement in the way teachers create questions as well as in teamwork

promotion. Furthermore, teachers reported that they pay more attention now on how they teach. Thus, the

implementation of this e–assessment approach can be considered successful, namely because it allowed an

adequate response to the main needs initially identified.

1 INTRODUCTION AND

BACKGROUND

New challenges have emerged from Information and

Communication Technologies (ICT) and, at the same

time, the opportunity to create differentiated learning

opportunities for students, through multiple tools

available to teachers. The use of ICT is recommended

by several European organizations such as the

European Parliament (Redecker, 2013; Redecker and

Johannessen, 2013). In the assessment process, it

becomes an useful resource, through electronic

format or e-assessment.

The use of ICT in the student assessment process,

from where the concept of e-assessment arose, is

largely associated with the need to adapt the

traditional forms of assessment in e-learning courses.

ICT is used throughout the assessment process from

the design of the tests to the storage of the results

(Stödberg, 2012).

Closed-response questions remain the most

commonly used format in e-assessment (Stödberg,

2012). Among those, multiple-choice questions

Doutorando no Departamento de Psicologia e Educação da

Universidade da Beira Interior

(MCQ) are of particular relevance and have some

specificities. Based on a review of the literature, some

advantages and some limitations of MCQ are

presented in tables 1 and 2, respectively. In both

tables, the ones presented in bold are related to

mathematics, and the ones with the grey background

concern the point of view of students.

Structurally, MCQ are composed of 3 elements:

(1) a stem that presents the problem and which can

take the form of an incomplete sentence or a question;

(2) the correct option or answer key; and (3) several

distractors, which are incorrect alternatives, but

equally plausible for students who do not fully master

the subjects to be tested (Clegg and Cashin, 1986;

Burton et al., 1991; Bush, 2015).

From our point of view, one of the limitations of

MCQ in paper format tests, which we have not found

explicitly mentioned in the literature, has to do with

the possibility of students to cheat more easily than

they would do when the questions are open-ended. To

obviate this limitation, some teachers construct

several versions of the same test, introducing slight

changes to it, but trying to maintain the feasibility and

the reliability of the assessment. In the case of e-

assessment, a database of questions duly conceived

Azevedo, J., Oliveira, E. and Beites, P.

How Do Mathematics Teachers in Higher Education Look at E-assessment with Multiple-Choice Questions.

DOI: 10.5220/0006324801370145

In Proceedings of the 9th International Conference on Computer Supported Education (CSEDU 2017) - Volume 2, pages 137-145

ISBN: 978-989-758-240-0

137

and implemented represents a fundamental role here,

and may even reach the limit of obtaining a different

version of the test for each student, generated

randomly by the computer system (Azevedo, 2015).

Table 1: Advantages of Multiple Choice Questions.

 Saving time (e.g. in obtaining ratings) and resources

 Ease of assessment of large numbers of students in

large-scale tests

 Ease of calculating statistical analyses and test

results

 Obtaining a greater and faster scope with respect to

the contents of the course, which allows to assess a

wider range of topics and knowledge

 Compatibility between web-based courses

 Greater objectivity and reliability in classifications

 Existence of question banks for future use

 Ease of implementation through computers

 Easier to manage

 It is a standard method

 Great format variety

 Existence of balance between validity and reliability

with the logistical facility

 It has the potential to measure comprehension,

analysis, problem-solving ability and calculus

skills

 It avoids the introduction of symbolic notation by

students, in the specific case of mathematics

 Greater confidence in getting the correct answer

from processes of eliminating the wrong answers

 It allows to evaluate the students' knowledge per se

and not their writing abilities

 Perception that MCQ tests are more objective and

reliable

 Useful for self-evaluation and revision

 Feedback is fast and impersonal (no concept of

judgment)

 Increased student motivation and engagement

When it is intended to use e-assessment in general and

the MCQ, in particular, it is useful to construct a

question bank (Yorke, 2001). Question banks can

contribute to ensure the validity and reliability of the

assessment process, saving resources, time and

money (Bull and Danson, 2001; McAlpine, 2002). An

important aspect in the development of MCQ to be

included in a question bank is the assurance that the

items are of high quality. Several guidelines for the

quality of MCQ writing can be found in the literature

(Clegg and Cashin, 1986; Burton et al., 1991;

Haladyna, Downing and Rodriguez, 2002; Haladyna,

2004; Camilo and Silva, 2008).

Assessment can profoundly influence the

motivation of those who learn, as well as their

perspectives on learning. So, the introduction of

different assessment systems may have a major

impact on the whole educational process (Smith et al.,

1996; Wild et al., 1997; Scouller, 1998; Brown, 2001;

Bull and Danson, 2001; Jarvis et al., 2003; Jacob et

al., 2006; Frankland, 2007; JISC, 2007; Garfield and

Ben-Zvi, 2008; Stödberg, 2012; Redecker, 2013;

Holmes, 2015).

Table 2: Limitations of Multiple Choice Questions.

 They may not assess the same levels of

understanding that are assessed by open-end

questions

 Possible ambiguity in the questions themselves

 Inability to adequately measure certain skills at

higher cognitive levels

 The development of properly structured questions is

quite time consuming and requires a lot of training

 They may favour the superficial memorization of

concepts

 Students may try to hit the answer randomly

 Students can reverse the resolution and one is not

evaluating what is supposed to

 In questions with calculations the student can

arrive at a solution that does not exist in the

options concluding that his answer is incorrect

 It does not allow students to explain their answers, so

they are limiting

 They may penalize students who do not tend to take

risks

 Feedback personalization is limited

This paper presents the last part of a seven years’

implementation process of an e-assessment strategy

using MCQ, implemented in Moodle, which started in

fall 2008. This strategy was implemented in order to

allow the use of continuous summative assessment in

Mathematics Courses in a Higher Education

Institution (ISCAP/IPP), in classes with a high

number of students. The goal of this work is to reflect

on the teachers' practices for the development of

MCQ in the area of Mathematics and how e-

assessment can influence the teaching-learning

process in this institution.

The research method is described in section 2. A

discussion and the conclusions are presented,

respectively, in sections 3 and 4.

2 METHOD

Six mathematics teacher volunteers were interviewed

about the e-assessment process with MCQ, since they

used this test format for continuous summative

assessment. Four of them were female and the

average age was 55 years, ranging from 44 to 71

years. The average length of service at ISCAP was 22

years, so the teachers interviewed were quite

CSEDU 2017 - 9th International Conference on Computer Supported Education

138

experienced. In order to safeguard the participants'

anonymity, we identified each of the interviews with

"E" followed by a serial number (E1, E2,...).

A semi-structured interview was carried out in

order to allow a better organization of the topics to be

addressed and, moreover, to enable teachers to freely

express their views. The main objective was to

analyse and discuss the implementation process of e-

assessment, namely possible changes in teaching or

learning experiences and resulting advantages and

disadvantages for the teacher.

The interview script consisted of descriptive

information of the participants (gender, age,

disciplinary area and length of service at ISCAP) and

seven questions, focused on the following

dimensions: i) opinion on the e-assessment procedure

implemented; ii) major difficulties found during

implementation; iii) changes in teachers' pedagogical

practices; iv) advantages of this procedure for the

teacher; v) disadvantages of this procedure for the

teacher; vi) changes in students' practices in their

learning process. Question 7 aimed to assess whether

teachers identify any important information beyond

those previously considered for the topic being

analysed.

The interviews were recorded with the consent of

the teachers and later were entirely transcribed.

Content analysis techniques were used following two

steps: first, a careful reading of all the interviews, in

order to capture the general meaning of the speeches

and, then, the answers were grouped by each of the

defined dimensions already mentioned above.

3 RESULTS

We present below the analysis of the content of the

interviews regarding each of the defined dimensions.

We recall that each question, presented in the

interview script, corresponds to a dimension to be

studied.

Opinion on the e-assessment procedure used

All teachers reported a very positive opinion about

the type of assessment implemented and unanimously

agreed that this was a good assessment system:

"It was very appealing to the students and I think it

increased their interest and it had good results."

(E3)

"I think it's an assessment that was worth

developing. I like the assessment. I think it's a very

good assessment when you want to do several tests

during the semester to the students and the number

of students is quite high." (E6)

In any case, two teachers said they were reluctant

at the beginning of the process:

"At first I was very sceptical when we started using

this system, especially with regard to multiple

choice." (E1)

"At first I was not very receptive because I thought

it was not as functional as it actually was." (E2)

Two of the teachers pointed out that there was a

positive evolution over time, which resulted from a

learning process. The statement of one of them well

summarizes this opinion:

"Over the years, I think we have been refining,

learning to build multiple choice questions and I

think it has improved a lot." (E1)

It was also mentioned by two teachers that there

was a need for greater learning regarding how

questions should be developed and that this

learning was done:

"We read and reflect and we studied a little bit

about how to improve the formulation, especially

the multiple choice where we really could get better

results." (E1)

Major difficulties found during implementation

The initial lack of knowledge about how to prepare

good questions was pointed out by three teachers as

one of the difficulties encountered in the

implementation of this e-assessment process:

"We had to study the best way to ask the questions

because the way we work is different from a

traditional test. And at the outset there was a

difficulty in formulating the questions so that they

would be objective and would not evaluate more than

one objective in each question. "(E5)

Also pointed out by three teachers as a great

difficulty were the problems related to technology,

namely with Moodle functionality, the computers for

the students to carry out the tests or with the servers

contained the Moodle:

"First, the use of laptops. At the beginning some

didn’t have. Then I think it turned better. Also at the

beginning, sometimes the system crashed and blocked

a lot. Sometimes also the adaptation of teachers to

electronic/ computer methods." (E3)

Two teachers referred the introduction of complex

mathematical formulas in Moodle tests as a restraint.

Although it is a difficulty related to Moodle usage, it

is important to mention it, since it is directly related

to mathematics. One teacher said:

"When we had not yet mastered TeX, and then it

was all filled with question marks and just with an

extra space a point appeared... Correcting an error

was complicated and time-consuming." (E1)

Two interviewees pointed out how difficult it was

How Do Mathematics Teachers in Higher Education Look at E-assessment with Multiple-Choice Questions

139

to put a team in action due to the initial resistance of

some teachers:

"At first there was some resistance because it was

a new method." (E3)

Finally, it was pointed out by one teacher that

initially there was a lack of support from the school

management bodies:

"This was the great difficulty that put or could have

unable the start of this assessment. They did not

want, the Management, at the time, in 2006, that the

assessment proceeded." (E6)

Changes in teachers' pedagogical practices

Two teachers reported that there were no changes in

their pedagogical practices with e-assessment. One of

them reported that there had to be a greater process of

students’ adaptation. These teachers stated the

following:

"Basically no. I mean, the classes continued to

be given in the same ..." (E3)

"There had to be more adaptation on the part of

the students than ours." (E4)

However, three teachers acknowledged that there

were changes, but that they were not a consequence

of the type of assessment. One of them even affirmed

that the change in the assessment procedure was itself

a result of the global change process that was

implemented in the course:

"I do not think my practices have changed due

to the choice of this method of assessment. We are

changing according to other things: the level of

knowledge that students bring." (E1)

"Little has changed. It was not because of the

assessment system that there were changes." (E5)

"The type of assessment was more a

consequence of all the changes that were made than

the other way around." (E6)

Only one teacher recognized some changes, but

said that it only changed the way he approached the

resolution of the exercises in the practical classes:

"I think it's different from saying ‘solve this

exercise’ and that's it ... because they have to know

how to analyse the answers. Therefore, I think

practices are guided in another way. But in

theoretical terms I think nothing has changed ...”

(E2)

Advantages for the teacher

The main advantage for the teachers that was pointed

out by all the interviewees has to do with the

automatic process of obtaining the student grades,

which represents a great saving of time:

"The advantage in correction is evident, it is a

gain of many hours." (E4)

Two teachers considered this type of assessment

more objective:

"Much more objective assessment and much

easier correction." (E3)

"The criteria are very objective and therefore

there is no disparity of correction even elaborating

detailed grids of correction in the normal tests that

has now gone... There is always disparity of

correction between teachers. Here, therefore, the

questions are objective, either right or wrong." (E5)

Another of the advantages, pointed out by three

teachers, was that the question bank allowed the

creation of more practical and faster tests, for

example for the special examinations, which can be

requested by some of the students during the

semester:

"When we build our tests it also becomes very

simple, because we just go to the database and

choose this category or that subcategory and

therefore quickly build the test. That's why it's all

faster. If you were to build a test from scratch now,

you had to waste more time." (E1)

"At any moment one can resort to a test" (E4)

"If there is a need to take a test for the next day

or the next hour it is easier to have the question

bank." (E5)

One of the teachers said that this process made it

easier to integrate new teachers into the course. This

same professor pointed out as a great advantage the

fact of being able to carry out a more creative work:

"In the matter of the time that one has to spend

on course is more about creative aspects and less

about 'minor' aspects like those of correcting tests."

(E6)

Disadvantages for the teacher

The main disadvantage, which was reported by four

teachers, was that it takes a lot of initial effort and a

lot of time to develop the question bank, but that had

improved over the years. Here are the statements of

two teachers, who mirror very well the opinion of all:

"Put the questions, then the options and create the

database itself, divide it into categories and

subcategories... All this, we have been building over

the years, took a lot of time. There were many hours

there." (E1)

"Of course they give a lot of work to build the

question, it’s not it... It's more the work of building

the questions. But also with some practice ... initially

you invest enough time in this and we write and do

not like and then it is not how it should be ... With

practice this disadvantage is diluted." (E5)

Another disadvantage pointed out by two

teachers, when comparing the formulation of open-

ended and multiple choice questions, was that MCQ

were more difficult:

CSEDU 2017 - 9th International Conference on Computer Supported Education

140

"If you have to ask an open question it's very fast.

And when you are formulating the answers, you must

have a lot of attention ... it's a lot more attention ..."

(E2)

"In the beginning there was a difficulty in

elaborating the questions so that they were objective

and did not assessment more than one objective in

each question." (E5)

Another disadvantage, in the view of two

interviewees, had to do with the impossibility to

evaluate students’ creativity and reasoning:

"the tests he did not allow to see the students'

reasoning. Therefore, it was enough to have an error

in the way and sometimes even could arrive at the

right result with wrong reasoning or the other way

around." (E3)

Changes in students’ practices

Teachers perceived some changes in students'

practices. Most of the teachers (five of them),

reported that there was an increase to the classes’

attendance. The following statements reflect the

opinion of all teachers:

"What I think in what they have changed is that

they were much more ‘seduced’ by continuous

assessment in these terms... it gives me the impression

that if it were in a much more open regime, we would

have more absences than there were. If we had the

traditional continuous assessment they would missing

more often." (E2)

"I think with our assessment system we got the

students attending the classes. And when they come

to school more, they end up learning more." (E6)

Three teachers emphasized that there had been an

evolution in the way students behaved in responding

to the assessment questions. It was found that initially

the students attempted to respond at random and then

became aware of the penalties in case of wrong

answers, beginning to be more careful about the

selection of answers. One teacher said that:

"It is noted that, for example, they have come

accustomed from the secondary to having multiple

choice. In the case of mathematics, they have multiple

choice in the national exam only that does not

discount. And that, I think at the beginning, when the

students reach the first year they tried to fill the whole

grid not having well the notion, this in the first test, of

what it would discount. But then they will learn and

select." (E1)

Another behavioural change identified by three

teachers was that students study more regularly:

"They study more assiduously, I do not say every

day because I think they should work even harder, but

in the next test they always work harder." (E5)

"If there are several moments of assessment, the

students will also have to study something else and

that is good too." (E6)

One teacher pointed out as a negative aspect the

fact that the students were limited to mechanize the

processes without worrying about the reasoning:

"It mechanized the students more. The students

became more mechanical. I noticed a disinterest, one

might say, in reasoning. (...) The main goal was to

achieve the result. "(E3)

Other aspects emphasised during the interviews

In addition to the aspects related to the dimensions

under study, during the interviews there were some

aspects mentioned by the teachers that we consider

relevant. One aspect pointed out by four teachers was

the fact that the high number of students in the classes

made it difficult to carry out continuous assessment

and that this form of assessment actually made it

possible to implement continuous summative

assessment. The following statement sums up this

aspect:

"(This type of assessment) was a good choice,

given the large number of students we have. As we

want to make continuous assessment, the only way to

be able to implement the continuous assessment

system was to opt for such a system." (E5)

Three teachers reported that students’ grades

improved with continuous assessment:

"I think that, given the results, it has gone better

now. From what I can remember from our statistics,

the results have been a lot better. Well, we also have

the advantage of doing some tests, more than three at

least ... three or four... we've had four already. And

the distributed subjects may be a little easier." (E2)

"The assessment was so much better that the

results with continuous assessment in terms of

approvals have improved a lot." (E5)

Two teachers mentioned the advantages of

performing the "recovery" test, saying that it is good

because it allows students not to give up right after

the first test if the grade is low:

"It was really good because, in fact, when they did

the first test and it went bad, they became

unmotivated... And now they know that there is a

chance of catching the first one or any and it keeps

them in class. And I think that's pretty important."

(E2)

"Another aspect that does not have to do directly

with this assessment but the fact that there are

“recovery” tests... I think that was a very important

aspect, they give up less in the middle of the semester.

And so... because they have yet another chance in

case something goes wrong, to recover. And this

aspect was fundamental." (E6)

One teacher mentioned his concern about

How Do Mathematics Teachers in Higher Education Look at E-assessment with Multiple-Choice Questions

141

students’ fraud, having even been confronted with

this problem by some students:

"I think my biggest concerns are in that sense. Do

not cheat. And the student do not tell us: ‘Ah! I know

someone approved that did not know the contents but

I knew...’“ (E2)

Another teacher stated that there was a need to

make several changes:

"…The first step was to realize that things were

not going well, because there were too many failures,

school failures and, taking that into consideration, we

tried to diagnose problems (…) without pretending to

analyse everything at one time, but because there

were some problems that were being worked out over

the years, there were things that were clearly

necessary to do to make sure that we all follow the

syllabus more or less the same way, without limiting

the freedom of each one. But there was a well-defined

orientation on syllabus and what was important for

us to do. And for this we were all involved in some

way in the preparation of class notes of the classes.

So, in my point of view, when people are involved in

this process, then changes occur. And people also had

to try to act always by consensus, generate consensus

in the team... there was a syllabus that we had to

accomplish and from the syllabus we started to create

tools and we discussed many times. (…) And more.

How the class notes themselves have been worked

out. The work was distributed, then passed me to...

let's say I was perhaps the person who later gave

some unity to things and made some revisions... but

all people were involved in the process. And in that

way it was possible to make a consensual change of

things in which people would join and participate.

Then, the assessment system… at the beginning not

everyone was in agreement. And it ended up with

everyone, I think, adhering to the process and

realizing, or at least starting to believe, that that was

the way. According to the objectives we had and

according to the circumstances it was the best way

forward." (E6)

This professor emphasized the importance of

teamwork with motivation:

"I think it was funny to get a relatively large team

working, pulling everyone to the same side and

everyone working together. This demonstrates once

again that, above all, people are capable of doing

things, that there is a minimum of motivation and that

they believe in what they are doing." (E6)

4 DISCUSSION

An e-assessment process was developed and

implemented to allow the use of continuous

summative assessment in mathematics’ courses in a

higher education institution, in classes with a large

number of students. In this study, we analyse the

perceptions of teachers involved in this process

concerning its application, the difficulties they found,

potential changes in their pedagogical practices,

advantages and disadvantages of this procedure for

the teacher, and potential changes in students'

practices in their learning process.

Although some teachers were sceptical at the

beginning of the process, all teachers interviewed in

this study reported a very positive opinion about the

type of assessment implemented and agreed that it

was a good assessment system. Some of them

highlighted the progress made over time, namely the

need for greater learning on how questions should be

formulated and the importance of effective teamwork.

In fact, the initial lack of knowledge about how to

prepare good questions was pointed out as one of the

difficulties encountered in the implementation of e-

assessment.

Another difficulty experienced by the teachers

during the creation of the question bank was about the

use of TeX to write the mathematical expressions to

put in Moodle. For teachers who start writing

mathematical characters in e-assessment this is an

added problem, since the software used for the

construction of the MCQ does not always allow

quickly writing (Brito et al., 2009, p. 167). Most of

the teachers did not master TeX and therefore TeXaide

software was used to help teachers with more

difficulties. Even so, teachers had constraints in

writing the mathematical expressions. In recent years

the experience of teachers in the use of TeX already

allows to overcome this problem. Furthermore, the

latest versions of Moodle also have a good built-in

TeX editor, which allows to insert mathematical

formulas directly into Moodle for those who do not

master TeX. Even with the evolution and

improvement in the introduction and interpretation of

TeX by Moodle, some students still mentioned a

certain difficulty regarding the reading of some of the

formulas, especially when they were very similar.

However, in recent years, these problems have

specially affected to those using Tablets with small

screen. We think that with the implementation of the

latest version of Moodle this problem should

disappear. At the moment, all the teachers are very

familiar with its usage, being able to solve the few

problems that arise, especially during the tests. In

addition, at this time all the teachers can, without any

difficulty, introduce the MCQ in Moodle.

Most teachers reported that there were no changes

CSEDU 2017 - 9th International Conference on Computer Supported Education

142

in their pedagogical practices resulting from the e-

assessment. In spite of this, teachers recognized that

they have improved the preparation of the MCQ and

that now they pay more attention to the formulation

of the questions and to the way they teach. Another

aspect of change has to do with the introduction of

ICT in the teaching and learning process, which was

incorporated in a natural way after the entire

developmental process that took place throughout the

implementation of e-assessment.

There were teachers who acknowledged that some

changes were not triggered by the implementation of

e-assessment. For these teachers, the change in

assessment was, rather, the result of an institutional

global change. They were referring to the required

adaptations to implement the Bologna Process,

namely the equipment installed and all the investment

made in the assessment process. Contrasting with this

view, other colleagues considered that those broader

adaptations were only possible due to the use of the

MCQ in assessment, which served as a catalyst for a

global change in the institution.

A great advantage of this tool for the teachers

seems to be the automatic process of obtaining

ratings, and then saving a lot of time. This is also one

of the main potentialities stated in the literature, as

well as the easier assessment of large numbers of

students (Clegg and Cashin, 1986; Nicol, 2007; Bible

et al., 2008; Camilo and Silva, 2008; Green and

Mitchell, 2009; Liu et al., 2011; Douglas et al., 2012;

Jordan, 2013; Heron and Lerpiniere, 2013).

Another two benefits pointed out by teachers, and

emphasized by many authors, were an increased

objectivity in classifications (Burton et al., 1991;

Wild et al., 1997; Haladyna, 2004; Bible et al., 2008;

Jordan, 2013) and the existence of question banks for

future use (Ferrão, 2010; Guo et al., 2014).

On the other side, the amount of the initial effort

and time needed to build the question bank seemed to

be the major constraint of this tool for the teachers.

This was somehow expectable, since the development

of properly structured questions is quite time

consuming and requires a lot of training (Clegg and

Cashin, 1986; Burton et al., 1991; Ferrão, 2010; Liu

et al., 2011; Jordan, 2013; Guo et al., 2014). Other

issue that deserves attention is teachers’ perception of

the impossibility to evaluate students’ creativity and

reasoning through MCQ. In fact, some authors

acknowledge for the limitation of this kind of tests or

exams in properly measuring certain skills at higher

cognitive levels (Nicol, 2007; Bible et al., 2008;

Green and Mitchell, 2009; Ferrão, 2010; Rod et al.,

2010; Lee et al., 2011). Related to this, some of them

note that MCQ may favour the superficial

memorization of concepts (Nicol, 2007; Liu et al.,

2011; Heron and Lerpiniere, 2013).

With the implementation of this tool in continuous

summative assessment, teachers perceived some

changes in students' practices. They point out, for

example, the increase to the classes’ attendance,

greater regularity in academic work and the evolution

in the way students behaved in responding to the

assessment questions (less randomly). Other authors

have found an increased student motivation and

involvement (Green and Mitchell, 2009; Jordan,

2013).

We can thus conclude that the change in the type

of assessment had impact in both teacher and student

practices, as documented in the literature, in which it

is stated that the introduction of different assessment

systems can have an important impact throughout the

educational process (Smith et al., 1996; Scouller,

1998; Bull and Danson, 2001; Jarvis et al., 2003;

Jacob, Issac and Sebastian, 2006; Frankland, 2007;

JISC, 2007; Boticki and Milasinovic, 2008; Garfield

and Ben-Zvi, 2008; Stödberg, 2012; Redecker and

Johannessen, 2013; Holmes, 2015).

5 CONCLUSIONS

In this study, teachers’ perceptions about an e-

assessment tool with multiple-choice questions were

analysed. This strategy was planned in order to allow

the use of continuous summative assessment in

mathematics courses in a higher education institution,

in classes with a large number of students.

Despite all the initial resistance and difficulties,

both in terms of technology proficiency and

formulation of the MCQ, it can be concluded that

teachers developed a favourable opinion about the

accomplished evaluation process. To put forward this

strategy, it was fundamental teachers’ effort and

commitment to adapt an entire process that they were

not used to do. We can say that the initial resistance

was triggered by a certain "fear" by the unknown.

There was a need to rebuild the way they do

assessment, and to embrace a new methodology.

Once experienced, this tool would become very

useful given its objectivity and time savings, due to

the immediate test ratings and to a whole process of

automations that made certain bureaucratic tasks

much easier. However, there are some disadvantages,

such as a major initial effort required to construct a

database of questions. Teachers also noticed great

improvements in students, especially in the way they

study and an increase in class attendance.

Future research directions should include a deeper

How Do Mathematics Teachers in Higher Education Look at E-assessment with Multiple-Choice Questions

143

analysis of the practice changes and a confrontation

with the ones pointed out by the teachers in these

interviews. It would also be relevant to analyse

students’ views on the implementation of this

approach, as well as on its value, namely the use of

MCQ tests and exams in mathematics assessment.

REFERENCES

Azevedo, J. (2015) ‘e-Assessment in Mathematics Courses

with Multiple-choice Questions Tests’, in Proceedings

of the 7th International Conference on Computer

Supported Education (CSEDU 2015). Lisboa, pp. 260–

266. doi: 10.5220/0005452702600266.

Bible, L., Simkin, M. G. and Kuechler, W. L. (2008) ‘Using

multiple-choice tests to evaluate students’

understanding of accounting’, Accounting Education,

17(sup1), pp. S55–S68. doi:

10.1080/09639280802009249.

Botički, I. and Milašinović, B. (2008) ‘Knowledge

assessment at the faculty of electrical engineering and

computing’, in Proceedings of the International

Conference on Information Technology Interfaces, ITI.

Cavtat, pp. 111–116. doi: 10.1109/ITI.2008.4588392.

Brito, I., Figueiredo, J., Flores, M., Jesus, A., Machado, G.,

Malheiro, T., Pereira, P., Pereira, R. M. S. and Vaz, E.

(2009) ‘Using e-learning to self regulate the learning

process of mathematics for engineering students.’, in

Bulucea, CA and Mladenov, V and Pop, E and Leba, M

and Mastorakis, N. (ed.) Recent Advances in Applied

Mathematics. ATHENS: WORLD SCIENTIFIC AND

ENGINEERING ACAD AND SOC (Mathematics and

Computers in Science and Engineering), pp. 165–169.

Brown, G. (2001) Assessment series n.o 3 - assessment: a

guide for lectures. York: Learning and Teaching

Support Network (LTNS).

Bull, J. and Danson, M. (2001) Assessment series N.o 14 -

computer-assisted assessment (CAA). York: Learning

and Teaching Support Network (LTNS).

Burton, S., Sudweeks, R., Merrill, P. and Wood, B. (1991)

How to prepare better multiple-choice test items:

guidelines for university faculty, Brigham Young

University Testing Services and The Department of

Instructional Science. Available at: http://

testing.byu.edu/info/handbooks/betteritems.pdf.

Bush, M. (2015) ‘Reducing the need for guesswork in

multiple-choice tests’, Assessment & Evaluation in

Higher Education, 40(2), pp. 218–231. doi:

10.1080/02602938.2014.902192.

Camilo, H. and Silva, J. A. P. da (2008) Os testes de escolha

múltipla (TEM), Essências EDUcare. Departamento de

Educação Médica da Faculdade de Medicina -

Universidade de Coimbra.

Clegg, V. L. and Cashin, W. E. (1986) Improving multiple-

choice tests. Kansas State University: Center for

Faculty Evaluation & Development.

Douglas, M., Wilson, J. and Ennis, S. (2012) ‘Multiple-

choice question tests: a convenient, flexible and

effective learning tool? A case study’, Innovations in

Education and Teaching International, 49(2), pp. 111–

121. doi: 10.1080/14703297.2012.677596.

Ferrão, M. (2010) ‘E-assessment within the bologna

paradigm: evidence from Portugal’, Assessment &

Evaluation in Higher Education, 35(7), pp. 819–830.

doi: 10.1080/02602930903060990.

Frankland, S. (2007) Enhancing teaching and learning

through assessment. Dordrecth: Springer.

Garfield, J. B. and Ben-Zvi, D. (2008) Developing students’

statistical reasoning: connecting research and teaching

practice. Dordrecht: Springer. doi: 10.1007/978-1-

4020-8383-9.

Green, A. and Mitchell, C. (2009) ‘E-assessment:

opportunities and challenges for the sports marketing

and educator’, in Proceedings of the 2nd International

Conference of Teaching and Learning (ICTL 2009).

Kuching, pp. 1–9.

Guo, R., Palmer-Brown, D., Lee, S. W. and Cai, F. F.

(2014) ‘Intelligent diagnostic feedback for online

multiple-choice questions’, Artificial Intelligence

Review, 42(3), pp. 369–383. doi: 10.1007/s10462-013-

9419-6.

Haladyna, T. M. (2004) Developing and validating

multiple-choice test items - third edition. 3rd edn.

Mahwah, New Jersey: Lawrence Erlbaum Associates.

doi: 10.1177/0146621605280143.

Haladyna, T. M., Downing, S. M. and Rodriguez, M. C.

(2002) ‘A review of multiple-choice item-writing

guidelines for classroom assessment’, Applied

Measurement in Education, 15(3), pp. 309–333. doi:

10.1207/S15324818AME1503_5.

Heron, G. and Lerpiniere, J. (2013) ‘Re-engineering the

multiple choice question exam for social work’,

European Journal of Social Work, 16(4), pp. 521–535.

doi: 10.1080/13691457.2012.691873.

Holmes, N. (2015) ‘Student perceptions of their learning

engagement in response to the use of a continuous e-

assement in a undergratuate module’, Assessment &

Evaluation in Higher Education, 40(1), pp. 1–14. doi:

10.1080/02602938.2014.881978.

Jacob, S. M., Issac, B. and Sebastian, Y. (2006) ‘Impact on

student learning from traditional continuous assessment

and an e-assessment proposal’, in Proceedings of the

PACIS 2006 - The 10th Pacific Asia Conference on

Information Systems. Kuala Lumpur, pp. 1482–1496.

Jarvis, P., Holford, J. and Griffin, C. (2003) Theory &

practice of learning - 2nd edition. 2nd edn. New York:

Routledge Falmer.

JISC (2007) Effective practice with e-assessment: an

overview of technologies, policies and practice in

further and higher education, Joint Information Systems

Committe. Available at: http://www.jisc.ac.uk/media/

documents/ themes/ elearning/ effpraceassess. pdf

(Accessed: 15 September 2014).

Jordan, S. (2013) ‘E-assessment: past, present and future’,

New Directions, 9(1), pp. 87–106.

Lee, H.-S., Liu, L. and Linn, M. C. (2011) ‘Validating

measurement of knowledge integration in science using

multiple-choice and explanation items’, Applied

CSEDU 2017 - 9th International Conference on Computer Supported Education

144

Measurement in Education, 24(2), pp. 115–136. doi:

10.1080/08957347.2011.554604.

Liu, O. L., Lee, H.-S. and Linn, M. C. (2011) ‘An

investigation of explanation multiple-choice items in

science assessment’, Educational Assessment, 16(3),

pp. 164–184. doi: 10.1080/10627197.2011.611702.

McAlpine, M. (2002) Design requirements of a databank.

Leicestershire: The CAA Centre TLTP Project.

Nicol, D. (2007) ‘E-assessment by design: using multiple-

choice tests to good effect’, Journal of Further and

Higher Education, 31(1), pp. 53–64. doi:

10.1080/03098770601167922.

Redecker, C. (2013) The use of ICT for the assessment of

key competences. Luxembourg: European Union. doi:

10.2791/87007.

Redecker, C. and Johannessen, Ø. (2013) ‘Changing

assessment - towards a new assessment paradigm using

ICT’, European Journal of Education, 48(1), pp. 79–96.

doi: 10.1111/ejed.12018.

Rod, J. K., Eiksund, S. and Fjaer, O. (2010) ‘Assessment

based on exercise work and multiple-choice tests’,

Journal of Geography in Higher Education, 34(1), pp.

141–153. doi: 10.1080/03098260903062039.

Scouller, K. (1998) ‘The influence of assessment method

on students’ learning approaches: multiple choice

question examinations versus assignment essay’,

Higher Education, 35(4), pp. 453–472. doi:

10.1023/A:1003196224280.

Smith, G., Wood, L., Coupland, M., Stephenson, B.,

Crawford, K. and Ball, G. (1996) ‘Constructing

mathematical examinations to access a range of

knowledge and skills’, International Journal of

Mathematical Education in Science and Technology,

27(1), pp. 65–77. doi: 10.1080/0020739960270109.

Stödberg, U. (2012) ‘A research review of e-assessment’,

Assessment & Evaluation in Higher Education, 37(5),

pp. 591–604. doi: 10.1080/02602938.2011.557496.

Wild, C., Triggs, C. and Pfannkuch, M. (1997) ‘Assessment

on a budget: using traditional methods imaginatively’,

in Gal, I. and Garfield, J. B. (eds) The assessment

challenge in statistics education. Amsterdam: IOS

Press, pp. 205–220.

Yorke, M. (2001) Assessment series n.o 1 - assessment: a

guide for senior managers. York: Learning and

Teaching Support Network (LTNS).

How Do Mathematics Teachers in Higher Education Look at E-assessment with Multiple-Choice Questions

145