Assessment of the Quality of Distance Learning during the Period of

COVID-19

V. V. Grigoriev-Golubev

, N. V. Vasileva

and I. V. Evgrafova

Saint-Petersburg state marine technical University, Saint-Petersburg, Russia

Keywords: Test Control, Quality of Education, Computer Program, Mathematical Tests, the Format of the Test Session,

the Database of Tests, the Formation of Tests.

Abstract: This paper discusses the issues of building the educational process in the distance learning system, the

associated difficulties and ways to overcome them. The main topic of the study is the methodology for

assessing the quality of knowledge of students through test control during the entire distance educational

process. The authors set out both the necessary requirements for test items that ensure the quality of

assessment of students' knowledge, and various formats of testing in the educational process, their purpose

and ease of use in distance learning. Particular attention is paid to the effectiveness of the testing systems used,

the simplicity of the formation of the test task, the tools for replenishing the test base, automation of both the

testing session and the announcement and saving of its results. The authors describe the different formats of

the content and organization of the test control, as well as the learning elements introduced into the test control.

As an example, several testing systems developed by the authors are demonstrated and are successfully used

in the distance learning process.

1 INTRODUCTION

During the pandemic, especially dramatic changes

took place in the field of education. The need to

conduct classes in a distance format posed a difficult

task for teachers:

 provide the educational process with electronic

educational materials available via the Internet

in a short time;

 think over the format of the distance learning;

 to acquire the equipment necessary for distance

learning.

An equally difficult task faced the leadership of

the Russian universities – to quickly deploy

educational platforms for distance learning. Despite

the fact that many Russian universities were not ready

for this, most of them, in particular, the St. Petersburg

State Maritime Technical University (SPbGMTU)

quite quickly created portals in the form of distance

learning centers, providing a convenient form of

interactive communication between students and

https://orcid.org/0000-0002-4310-7452

https://orcid.org/0000-0002-4930-6938

https://orcid.org/0000-0002-0972-2275

teachers, and also purchased platforms for on-line

learning: Zoom, Webex, and etc.

The same is true for teachers. Never before the

period of the pandemic did university professors

improve their qualifications in the field of using

computer technologies so massively and intensively.

Moreover, during the student holidays, many

universities, in particular, the SPbGMTU, conducted

advanced training courses in order to expand the

competencies of teachers in the use of modern

information technologies and software in the distance

educational process.

It is clear that Russia has made a real leap forward

in digital educational technologies. “Noone in the

world has done more to digitalize education than the

coronavirus” (Demakov, 2020).

Grigoriev-Golubev, V., Vasileva, N. and Evgrafova, I.

Assessment of the Quality of Distance Learning during the Period of COVID-19.

DOI: 10.5220/0011117400003439

In Proceedings of the 2nd International Scientiﬁc and Practical Conference "COVID-19: Implementation of the Sustainable Development Goals" (RTCOV 2021), pages 189-196

ISBN: 978-989-758-617-0

189

2 NEW FORMAT OF THE

EDUCATIONAL PROCESS

2.1 Using Online Platforms

Surprisingly, university professors, even the most

active opponents of computer technologies, promptly

prepared the electronic educational materials

necessary for studying the discipline being taught,

developed a method of conducting distance learning

that was convenient for them and adapted to the

characteristics of the course being studied, and

bought the necessary equipment for this: graphic

tablets, web cameras, and etc. Despite the fact that in

the universities of the Russian Federation, classes are

held both remotely and in a mixed form, lectures are

delivered remotely. The distance form of lecturing

was especially difficult for teachers of physical and

mathematical disciplines, whose educational material

contains a large number of formulas and figures. One

of the most effective ways to conduct such exercises

is “working with two screens”. If a lecture is being

held, then the prepared material of this lecture is open

on one screen, and the other is filled in "in real time",

i.e. the necessary additions and explanations, answers

to questions arising from the listeners, etc. are written

on it.

If a practical lesson is held in a remote format

(Fig. 1), then the theoretical material that is used in

solving the problem is displayed on the left screen,

and on the right screen, again, the solution itself is

demonstrated in real time. Wherein, students can even

see the set and editing of formulas. Teachers who are

not proficient in fast typing and editing formulas use

graphic tablets and just writing on a piece of paper

that is displayed on the screen through a webcam.

2.2 Form of Dialogue with Students

Dialogue between the teacher and the students, i.e.

checking homework, tests, etc. with distance

learning, as a rule, it takes place via e-mail.

This significantly increases the load on the teacher,

since with this format of training, it is necessary not

only to point out the mistakes made in the work, but

also to give a detailed explanation, indicate the way

to correct them, give a link to the corresponding

theoretical material and to the demonstrated solution

of similar problems.

Surprisingly, it turned out that this method of

exchange gives the teacher a broader and more

accurate idea of the level of knowledge and

performance of both the entire study group and the

individual student.

Moreover, almost daily correspondence with the

teacher, in which the student receives a detailed

explanation of his/her mistakes and shortcomings,

contributes to their greater involvement in the

educational process and increases the motivation for

learning, in comparison with full-time training.

Figure 1: Conducting a lesson through the on-line Zoom platform.

RTCOV 2021 - II International Scientiﬁc and Practical Conference " COVID-19: Implementation of the Sustainable Development Goals

(RTCOV )

190

2.3 Assessment of the Education

Quality

One of the ways to assess the quality of education, in

particular mathematics education, is the monitoring

of the assimilation of the studied material at all stages

of the educational process, carried out in the form of

test control. If the educational process takes place in

a distance form or in a mixed (part-time) form, then

test control is its necessary attribute.

Test control in distance learning is carried out

both to assess the educational activities of students

and to arrange and manage the educational process

(Verbickij, 2013). Moreover, given the correct

content of test items, testing can also be an element of

learning.

The benefits of knowledge control in the form of

testing include:

 obtaining an objective assessment of

knowledge;

 participation in a testing session for the entire

study group;

 obtaining information about the level of

knowledge of students within a few minutes;

 formation of students' skills in solving

problems;

 increasing the motivation of students to

systematically prepare for classes.

2.3.1 Test Requirements

Considering testing as a tool for assessing students'

knowledge, it is possible to formulate the necessary

requirements for the test:

1. Differentiating ability of the test – the division

of the tested according to the level of competence.

2. Test Validity – Test items should cover all

aspects of the topic being tested and be consistent

with the competencies they are assessing.

3. Reliability of the test – the results shall not be

random, which is determined by both the quality of

the test tasks and the comfort of the conditions of its

conduct.

If at the same time to take care that the teacher

does not spend too much time on formation of test

tasks and on their assessment, then it is necessary to

ensure the automation of the formation of tests from

the available test base and the output of the test

session results both for each test taker and for the

entire study group.

Compliance of the test with all the mentioned

criteria is checked on the basis of repeated testing

sessions in different academic groups. The difficulty

level of the test is also experimentally verified. Too

simple (correctly performed by the overwhelming

majority or all subjects) or too complex test (the

overwhelming majority of subjects or all of them fail)

is useless, since it does not solve the problem of

assessing the level of assimilation of the studied

material. Since the tests, as a rule, are formed from

the base (pool) of test items, structured by discipline

topics and by the level of complexity, “manually” or

through the program, then too simple or too complex

tasks shall be excluded from it or significantly

reworked.

Sometimes the test turns out to be "difficult" due

to the small amount of time allotted for it. In this case,

you need to increase the time for passing the test. It is

even better not to limit the time of the testing session,

but to include the time spent by the subjects in the

rating, which determines the given grade. This format

of test control is also preferable because it creates a

more comfortable environment in the audience (at an

on-line conference or in a computer room), since it

gives the subjects an opportunity to show their

knowledge without fear that the session is about to

end.

If the tests are generated programmatically from

a database of test items, structured not only by the

topics of the studied material, but also by the level of

complexity, then the implementation of the so-called

adapted testing is possible. In this case, tests of

different levels of complexity are formed, depending

on the previous results of the test control.

As you know, there are two forms of tests: closed

and open tests. Closed types of tests (multiple choice

tests) involve choosing one or more answers from the

proposed options or establishing the truth/falsity of

the statements presented. Open forms of tests involve

students' answers to the essence of the task, the

wording of which ensures that there is only one

correct answer. This form of test items is more

difficult for the test subject, since in closed tests it is

sometimes possible to choose the correct answer

based not on knowledge, but on intuition, or on the

basis of an analysis of deliberately unrealistic results,

or simply guessing. Open test assignments are more

valuable for the teacher, however, the implementation

of testing systems in the form of a program focused

on open tests is associated with great difficulties: the

form of recording the answer is strictly regulated and

any deviation from it when checking the results of a

testing session programmatically can be perceived as

an error.

2.3.2 Teaching Function of Tests

Testing that accompanies the student throughout the

educational process, with a methodologically

competent organization, can also be of a teaching

nature. “Tests don't just measure learning – they

promote learning” (Butler, 2018). Test control

contributes to the formation of problem-solving skills

Assessment of the Quality of Distance Learning during the Period of COVID-19

191

if the subjects get the opportunity to find out the

questions in which mistakes are made, to realize them

and pass the test again. This is demonstrated by

examples in (Author, 2019), (Kunturova, 2019), and

in (Author, 2017), a test control system adapted for

small screens is described, i.e. allowing to conduct a

testing session in a regular classroom using

smartphones.

Numerous studies have shown that closed tests

(multiple choice tests) are educational in nature no

less than open ones, since they not only improve the

memorization of correct information, but also make it

easier to remember information related to incorrect

alternatives (Little, 2012). Multiple choice tests

(closed tests) are very effective if the number of

answer options contains a corresponding number of

plausible incorrect alternatives. Wherein, studies

have shown that test tasks in which none of the listed

answer options is correct are “distracting” in nature

and do not solve the goal set for using the testing

technology.

Similarly, tests in math disciplines are useless if

multiple answers are correct. Although, tests in the

humanities, it is sometimes advisable to form tests

with several correct answer options, differing in

completeness, generalization and level of

understanding, as described in (Little, 2012).

2.3.3 Regular Test Management Function

The main function of testing is not to control

knowledge, but to assess its quality, on the basis of

which the educational process is controlled by the

teacher and students' self-government. Regular

(ongoing) testing allows the teacher to manage the

learning process, i.e. correct it in accordance with the

obtained test results:

 it is possible to consider again in more detail

some sections of the passed topic;

 adaptively, depending on the results for the

whole group, present further material;

 refer some sections of the educational material

with a shortage of time and a sufficiently high

level of assimilation of the material of the

educational group for independent study, and etc.

In more detail, the issues of management of the

educational process are investigated in the paper

(Author, 2020).

2.3.4 Test Control Format

Modern computer technologies allow testing sessions

in three directions:

 - as a personal monitoring of students through

the testing system;

 - to carry out control testing in the mode off line

in the "paper version";

 - to conduct control testing online using

electronic means.

Personal monitoring of students, as a rule, is

carried out through a testing program built into an

electronic training course (ETC) designed for self-

study. The need to include an element of personal

monitoring in the ETC is discussed in the paper

(Petrovic-Dzerdz, 2019). "The typical student usually

does not close the textbook after studying, but

voluntarily answers the questions posed at the end of

each chapter." Studies show that he/she

conscientiously participates in self-tests in order to

know about the state of his/her knowledge level." If

testing is carried out solely to assess the knowledge

gained by the subjects, then there is a temptation to

look for alternative ways in order to earn a good

grade. An effective personal monitoring system built

into the ETC is demonstrated by examples in (Author,

2014, 2017, 2019).

Off line mode testing system does not require a

computer class, it can be carried out in any auditorium

or online through the conference. The advantage of

testing in the of line system is the ability to use open

tests when developing test tasks, which is not always

possible when testing through a computer program.

Wherein, computer technologies are used only to

form a test base (a pool of questions) and form tests

from it at a client request.

Testing in online format requires the development

of a testing program. It is possible to develop such a

program based on platforms: Drupal (Author, 2014),

Moodle (Medvedeva, 2002), Sakai (Author, 2015),

Blackboard, and etc. However, they all have limited

application, especially for mathematical tests

associated with a large number of formulas integrated

into the program. It is much more efficient to develop

your own shell designed for testing control.

The original shell will take into account all the

requirements that correspond to the set goal, include

all the necessary options. If at the same time the test

base is well structured and based on modern data

storage technologies, then it is easy to automate the

formation of test tasks and announcement of test

results. The widespread use of the testing program in

the educational process requires a convenient,

accessible to any user, method of replenishing and

updating the database of test questions – “a database

adds what we call delivery time flexibility”

(Brusilovsky, 1999). Only its own original shell will

make it possible to develop new tests based on it

and/or update and replenish the test base

RTCOV 2021 - II International Scientiﬁc and Practical Conference " COVID-19: Implementation of the Sustainable Development Goals

(RTCOV )

192

2.3.5 Online Testing

At SPbGMTU, for the current computer control of the

level of knowledge of the academic group, the

original testing program, made by the authors, is used.

When using this program, the teacher sets the rating

and grade in accordance with the number of correctly

solved problems in the corresponding options of the

program.

All results of the testing session are displayed to

the user on the screen. The testing program provides

for free navigation through the test items, i.e. the user

can answer test questions in a convenient order.

The tests contain tasks with closed answers. Some

of the tasks are of a theoretical nature, while the other

part of the test tasks requires the ability to apply the

knowledge gained to solve practical problems. When

an attempt is made to end a testing session, the user is

presented with a warning that prevents accidental

termination of the session.

After passing the test, its results are announced in

the window that opens (Fig. 2). The test results

include:

 the number of correctly solved problems;

 the number of correctly solved problems as a

percentage of the total number;

 numbers of incorrectly solved problems.

Testing with this program is intended for use in a

computer room. Since at this each entry into the

program, both the order of the test tasks and the

numbers of the answer options are mixed, then

“cheating from a neighbor” is excluded.

The program can also be used in the remote access

system, since for the teacher the results are saved in

the "User Diary", which can be viewed by him/her at

any convenient time.

Updating tests when using this program is carried

out directly through its open source, both by adding

new questions and by forming a new test from

existing test items. More details about the program's

capabilities are described in (Author 2016, 2017).

2.3.6 off Line Testing

The SPbGMTU developed a computer program

"Formation of Tests" (Author 2008), on the basis of

which test options are formed from the existing

database on the given topics, which, together with the

matrix of answers, are issued in printed form for

testing. This program is successfully used in the

educational process of SPbGMTU.

The test base is developed using ACCESS

technologies, and formation is based on client

requests and is carried out using SQL – Script

operators. The program is very easy to use, because

test problems are added to the database as they are

developed. In this case, it is possible to use the table

mode and the form mode (Fig. 3). The text of the

problems is written in the Microsoft Word editor with

the ability to use the Mat Type formula editor.

Formation of the test is carried out through the start

form at the client's request, which includes (Fig. 4):

 - the number of tasks in the test;

 - number of options;

 - names of testing topics;

 - the level of difficulty of the test items.

Figure 2: Visualization of test results.

Assessment of the Quality of Distance Learning during the Period of COVID-19

193

Figure 3: Database in Table Mode and Form Mode.

Figure 4: Forming a test and viewing it on the screen.

The generated tests can be viewed on the screen, they

can be edited, they can be saved using the provided

"Menu of ready tests", they can be printed by

accessing the "Print test" tab. The user is given two

reports: the ordered number of test options and a

matrix of answers to them (Fig. 5).

Use of the program allows for a testing session in

any classroom. Tests in printed form are offered to the

test subjects. At the end of the session, they present a

ticket with the numbers of the correct (according to

their solution) answers, and also present solutions to

some problems indicated by the teacher. This format

of testing sessions is very similar to the form of the

USE and, therefore, is well known to students.

3 DISTANCE LEARNING

EXPERIENCE

The experience of two years of distance learning

shows that there has been no decline in the quality of

education. The teaching load of teachers and the

volume of independent work of students have

increased. The form of personal contact between

teacher and student has changed, but modern

communication means make it possible to

successfully implement this contact in the remote

access system.

RTCOV 2021 - II International Scientiﬁc and Practical Conference " COVID-19: Implementation of the Sustainable Development Goals

(RTCOV )

194

Figure 5: Test format and response matrix.

Moreover, academic performance not only did not

decrease, but even increased, especially in the first

year, which from the first days of training was

involved in the educational process in a distance

format. The number of unsatisfactory ratings has not

decreased in comparison with the previous streams,

but the number of "good" ratings has increased

significantly. This rating is becoming the most

popular.

Despite the fact that online testing can be carried

out in a remote access system, it turned out that

repeated testing outside the computer room is not

convenient, since the generated user diaries that save

the results for the teacher need to be deleted regularly,

which takes a lot of time.

However, it is convenient to use the testing program

posted on the site for personal monitoring of the level

of knowledge. The student enters the program

through authorization (Fig. 6) and selects a test from

the list (Fig. 7).

Figure 6: Authorization of the test subject.

Figure 7: Test selection.

Since he/she is given a message about the

numbers of incorrectly solved problems, he/she can

view the test, understand his/her mistakes and pass it

again. In difficult cases, the student can contact the

teacher.

During the period of distance learning, the

program for generating tests from a database turned

out to be very useful and in demand (Fig. 4, 5). It

turned out that it can also be used when conducting a

lesson in the form of an online conference: the

formation of a test takes several minutes, the teacher

sends a set of formed test options to students via e-

mail. After completing all the assignments, students

fill out a ticket with answers. Since the teacher has a

matrix of answers, he/she can conduct a preliminary

check right in the classroom, asking to show the

completed ticket with answers to the camera and read

out the numbers of incorrectly solved tasks, and as

homework ask to complete their detailed solution.

Assessment of the Quality of Distance Learning during the Period of COVID-19

195

4 CONCLUSIONS

The pandemic has come, the pandemic will go away,

and the use of powerful information resources during

the period of the distance educational process will

form a different quality of competencies for both

teachers and students. The widespread use of

elements of distance learning and computer

technologies in the subsequent full-time form of the

educational process will inevitably increase its

quality.

REFERENCES

Brusilovsky, P., Miller, P., 1999. Web-based testing for

distance education. In World Conference of the WWW

and Internet. Honolulu, AACE, pp. 149 – 154.

Butler, A.C., 2018. Multiple-choice testing in education:

Are the best practices for assessment also good for

learning. Journal of Applied Research in Memory and

Cognition, 7: 323 – 331.

Demakov, I.S., 2020. Coronavirus has like no one else

contributed to the digitalization of education. In

Federalpress, M. http://fedpres.ru.

Grigoriev-Golubev, V.V., Vasileva, N.V. et al., 2008. A

software complex for generating test tasks from a

database. Journal Computer learning programs and

innovations, 12: 7 – 8.

Grigoriev-Golubev, V.V., Vasileva, N.V. et al., 2014.

Elements of distance learning mathematical discipline

in the higher education system for ship construction

majors. Journal of Marine Intellectual Technologies,

26(4): pp. 136 – 142.

Grigoriev-Golubev, V.V., Vasileva, N.V., et., 2015.

Complex electronic educational resources for the

organization of educational process in the form distance

learning. In Scientific notes of ISGS, 13(1): 154 – 160.

Grigoriev-Golubev, V.V., Vasileva, N.V., Kunturova,

N.B., Rakov, Yu.A., 2019. Aspects of distance learning

to mathematical disciplines, Proceedings of the 1st

International Scientific Conference. In Modern

Management Trends and the Digital Economy: from

Regional Development to Global Economic Growth

(MTDE 2019), pp. 530 – 536.

Kunturova, N.B., Prokofieva, A.L., Vasileva, N.V., et al.,

2019. The Basic Requirements for the Development of

Efficient Applications for E-Learning. In Proceedings

of the 2019 International Conference on Pedagogy,

Advances in Social Science, Education and Humanities

Research Communication and Sociology (ICPCS

2019), 315: 1–8.

Little, J.L, Bjork, E.L., Robert A., Bjork, R.A., 2012.

Multiple-Choice Tests Exonerated, at Least of Some

Charges: Fostering Test-Induced Learning and

Avoiding Test-Induced Forgetting. Journal of

Psychological Science, 23(11): 1337 – 1344.

Medvedeva, S.N., Tutubalin, P.I., 2002. Information

technologies of control and assessment of knowledge in

system of distance learning Moodle. Journal of

Educational Technology & Society, 15(1): 555 – 566.

Petrovic–Dzerdz, M., 2019. Gamifying Online Tests to

Promote Retrieval-Based Learning. International

Review of Research in Open and Distributed Learning,

20(2): 25 – 42.

Vasileva, N.V., Kunturova, N.B., Jakovleva, N.A., 2016.

Creation and pedagogical conditions for the use of

electronic means of teaching mathematical disciplines

in educational organizations of law enforcement

agencies. Journal Bulletin of the St. Petersburg

University of the Ministry of Internal Affairs of Russia,

4 (72): 164–168.

Vasileva, N.V., Grigoriev-Golubev, V.V. and Evgrafova,

I.V., 2017. On improving the effectiveness of software

for electronic (e-learning) and mobile (m-learning)

learning. Journal of Marine Intellectual Technologies,

36(2): 75 – 80.

Vasileva, N.V., Grigoriev-Golubev, V.V. and Evgrafova,

I.V., 2020. Problems of modern education, Journal of

Virtual educational process management environment,

2: 201–214.

Verbickij, A.A., Puchkova, Е.B., 2013. The possibilities of

the test as a means of diagnosing the quality of

education: myths and reality. Journal of Higher

Education in Russia, 6: 33 – 44.

RTCOV 2021 - II International Scientiﬁc and Practical Conference " COVID-19: Implementation of the Sustainable Development Goals

(RTCOV )

196