Analyzing Interactions in Automatic Formative Assessment Activities

for Mathematics in Digital Learning Environments

Alice Barana

and Marina Marchisio

Department of Molecular Biotechnology and Health Sciences, University of Turin, Via Nizza 52, 10126 Turin, Italy

Keywords: Automatic Formative Assessment, Digital Learning Environment, Interactions, Mathematics Education,

Online Learning.

Abstract: This paper discusses the theme of the analysis of the interactions in a Digital Learning Environment (DLE)

to study formative assessment processes. We propose a definition for a Digital Learning Environment based

on the concept of a learning ecosystem, and we provide a model to analyze the interactions occurring among

the components of a DLE during automatic formative assessment activities for Mathematics. Using the model,

we qualitatively analyze two different activities of symbolic computations, carried out by 396 students of

grade 8 in different contexts, to identify the interactions through which formative assessment strategies are

developed. In the conclusions, we suggest ways to adopt this model for learning analytics, to analyze the

interactions in large online courses.

1 INTRODUCTION

According to Wilson (1995), a learning environment

is “a place where learning is fostered and supported”.

It includes at least two elements: the learner, and a

“setting or space wherein the learner acts, using tools

and devices, collecting and interpreting information,

interacting perhaps with others, etc.” (Wilson, 1995).

The traditional learning environment that everyone

knows is the classroom, where the teacher teaches,

students learn, individually or with their peers, using

tools such as paper, pen, and a blackboard. The

diffusion of technology transformed this traditional

learning environment by adding digital tools, as

tablets or computers, and the IWB (Interactive White

Board). Besides equipping physical places with

technologies, the technological revolution brought to

the creation of a new learning environment, situated

in a non-physical dimension: that of the Internet,

accessible from everywhere via computers, tablets, or

even smartphones. This is the essence of the “Digital

Learning Environment” (DLE); besides the learner

and a setting, which can be virtual, a device is needed

to access the activities.

Today, due to the COVID-19 pandemic, online

platforms have known increasing popularity,

https://orcid.org/0000-0001-9947-5580

https://orcid.org/0000-0003-1007-5404

supporting smart-schooling, and class attendance

from home (Giovannella et al., 2020). They have been

invaluable to permit students from all social and

cultural backgrounds to carry on their education. The

interest in DLEs in the research has increased

accordingly, making different theories and models

come to life (Fissore et al., 2020).

This paper intends to contribute to the discussion

about the essence of DLEs providing a definition and

a model for analyzing learning interactions in a DLE.

The theoretical framework includes a review of

various studies on DLEs and a proposal of definition.

Particular characteristics of DLEs for Mathematics

are considered, based on theories on formative

assessment and Automatic Formative Assessment

(AFA). Then, a model for the interactions among the

members of a DLE is proposed, to highlight the

interactions during AFA activities. In the following

sections, an AFA activity for grade 8 Mathematics in

a classroom context is presented. Some episodes

involving students working on this activity are

analyzed using our models, to show what kinds of

interactions can support formative assessment

strategies. The conclusions suggest how these

findings could be used in learning analytics research.

Barana, A. and Marchisio, M.

Analyzing Interactions in Automatic Formative Assessment Activities for Mathematics in Digital Learning Environments.

DOI: 10.5220/0010474004970504

In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 1, pages 497-504

ISBN: 978-989-758-502-9

497

2 THEORETICAL FRAMEWORK

2.1 Definition of Digital Learning

Environment

The concept of “Digital Learning Environment” has

a long history, and it has known several developments

and many different names over the years: Virtual

Learning Environments (Wilson, 1996), Online

Learning Environments (Khan, 1997), Computerized

Learning Environments (Abdelraheem, 2003), and

Digital Learning Environments (Suhonen, 2005). The

common factor among all these definitions is the use

of the Internet and its tools to provide an environment

where learning is supported, generally represented by

a Learning Management System (LMS). An LMS,

according to Watson and Watson (2007), is “the

infrastructure that delivers and manages

instructional content, identifies and assesses

individual and organizational learning or training

goals, tracks the progress towards meeting those

goals, and collects and presents data for supervising

the learning process of an organization as a whole.”

While similar environments are mainly used to

support online educational processes, we are

convinced and have proof of the fact that web-based

platforms can also be successfully adopted in

classroom-based settings: in our conception, DLEs

should not only be confined to distance education

(Barana, Marchisio, & Miori, 2019; Barana &

Marchisio, 2020; Borba et al., 2018).

More recently, many authors have developed an

interest in conceptualizing digital learning

environments as ecosystems, borrowing the term

from ecology (García-Holgado & García-Peñalvo,

2018; Giovannella et al., 2020; Guetl & Chang, 2008;

Uden et al., 2007; Väljataga et al., 2020). According

to Encyclopaedia Britannica (www.britannica.com),

an ecosystem is “a complex of living organisms, their

physical environment, and all their interrelationships

in a particular unit of space.” The natural ecosystem,

constituted by a biological community in a physical

environment, is the fundamental example; however,

this definition can be applied to any domain, even

artificial environments, by specifying the living

community, environment, and space unit.

There are several models of learning or e-learning

ecosystems in the literature, which vary for the

components included based on the theoretical

assumptions considered. In general, they contemplate

individuals, computer-based agents, communities,

and organizations in a network of relations and

exchanges of data that supports the co-evolutions and

adaptations of the components themselves (Guetl &

Chang, 2008). Following this trend, in this thesis, we

chose to use the term “Digital Learning Environment”

to indicate a learning ecosystem in which teaching,

learning, and the development of competence are

fostered in classroom-based, online or blended

settings. It is composed of a human component, a

technological component, and the interrelations

between the two. The human component consists of

one or more learning communities whose members

can be: teachers or tutors, students or learners, and

their peers, the administrators of the online

environment. The technological component includes:

 A Learning Management System, together

with software, other tools, and integrations

which accomplish specific purposes of

learning (such as web-conference tools,

assessment tools, sector-specific software, and

many others);

 Activities and resources, static or interactive,

which can be used in synchronous or

asynchronous modality;

 Technological devices through which the

learning community has access to the online

environment (such as smartphones,

computers, tablets, IWB);

 Systems and tools for collecting and recording

data and tracking the community's activities

related to learning (such as sensors, eye-

trackers, video cameras.

The interrelations between the two components

can include the interactions and learning processes

activated within the community and through the use

of the technologies as well as pedagogies and

methodologies through which the learning

environment is designed.

Independently of the fact that the DLSs are based

on a web-based platform, teaching and learning can

occur in one of the following modalities:

 Face to face, in the classroom or a computer

lab, with students working autonomously or in

groups through digital devices, or solving

tasks displayed on the Interactive White Board

with paper and pen or other tools;

 Entirely online, using the DLE as the only

learning environment in online courses or

MOOCs;

 In a blended approach, using online activities

to integrate classroom work, such as asking

students to complete them as homework.

These three modalities can be adapted to different

situations, grades, aims, and needs. For example, the

face-to-face modality can be suitable with students of

the lowest grades and in scholastic situations where

CSEDU 2021 - 13th International Conference on Computer Supported Education

498

the classroom work is predominant. The blended

approach can offer useful support to the face-to-face

lessons at secondary school or university (Marchisio

et al., 2020). Online courses are generally used for

training and professional courses, university courses,

or learning in sparse communities, where face-to-face

meetings are difficult to organize (Abdelraheem,

2003; Marchisio et al., 2020).

In this conceptualization, the DLE is not limited

to technological artifacts, even if they play a crucial

role. The learning community takes a prominent

place: it can include, according to the kind of DLE,

students and peers, teachers and tutors (who are

facilitators of learning activities), designers of

educational materials, and administrators of the

digital environment. The use of these technologies,

such as web-based platforms, assessment tools, and

other systems such as sensors or eye-trackers, allows

for collecting, recording, and using learning data.

These data can be elaborated within the DLE to

provide information useful to make decisions and

take action. In the following paragraphs, we will

explain how these data can be used to improve

learning, teaching, and the development of

competences.

2.2 Formative Assessment

Formative assessment is one of the key principles

which, according to the majority of scholars, should

be included in the design of a learning environment,

being it physical or virtual (Barana & Marchisio,

2016, Barana, Fissore, & Marchisio, 2020; Gagatsis

et al., 2019). In this study, we refer to Black and

Wiliam’s definition and framework of formative

assessment (Black & Wiliam, 2009). According to

them, “a practice in a classroom is formative to the

extent that evidence about student achievement is

elicited, interpreted, and used by teachers, learners,

or their peers, to make decisions about the next steps

in instruction that are likely to be better, or better

founded, than the decisions they would have taken in

the absence of the evidence that was elicited”. They

identified three agents that are principally activated

during formative practices: the teacher, the student,

and peers. Moreover, they theorized five key

strategies enacted by the three agents during the three

different processes of instruction:

 KS1: clarifying and sharing learning

intentions and criteria for success;

 KS2: engineering effective classroom

discussions and other learning tasks that elicit

evidence of student understanding;

 KS3: providing feedback that moves learners

forward;

 KS4: activating students as instructional

resources; and

 KS5: activating students as the owners of their

own learning.

2.3 DLEs for Mathematics

In this paper, we consider particular DLEs for

working with Mathematics through suitable

technologies and methodologies. The LMS that we

use is based on a Moodle platform and it is integrated

with an Advanced Computing Environment (ACE),

which is a system for doing Mathematics through

symbolic computations, geometric visualization, and

embedding of interactive components (Barana,

Brancaccio, Conte, et al., 2019), and with an

Automatic Assessment System based on the ACE

engine. In particular, we chose Maple ACE and

Moebius AAS. Through this system, we create

interactive activities for Mathematics based on

problem solving and Automatic Formative

Assessment (AFA), which are the main

methodologies used in the DLE, and that we have

better defined and characterized in previous works

(Barana, Conte, et al., 2018; Fissore et al., 2020). In

detail, the characteristics of the Mathematics

activities that we propose are the following:

 Availability of the activities for a self-paced

use, allowing multiple attempts;

 Algorithm-based questions and answers, so

that at each attempt different numbers,

formulas, graphs, and texts are displayed,

computed on the base of random parameters;

 Open mathematical answers, accepted for its

Mathematical equivalence to the correct one;

 Immediate feedback, returned when the

student is still focused on the task;

 Interactive feedback, which provides a sample

of a correct solving process for the task, which

students can follow step-by-step;

 Contextualization in real-life or other relevant

contexts.

2.4 Modelling Interactions in a Digital

Learning Environment

The technological apparatus of a DLE, particularly

when the LMS is integrated with tools for automatic

assessment, has a mediating role in the learning

processes. We can identify the following functions

through which it can support the learning activities

(Barana, Conte, Fissore, et al., 2019):

Analyzing Interactions in Automatic Formative Assessment Activities for Mathematics in Digital Learning Environments

499

 Creating and Managing: supporting the

design, creation, editing, and managing of

resources (e.g., interactive files, theoretical

lessons, glossaries, videos), activities (e.g.,

tests, chats for synchronous discussions,

forums for asynchronous discussions,

questionnaires, submission of tasks) and more

generally of the learning environment by

teachers, but also by students or peers;

 Delivering and Displaying: making the

materials and activities available to the users;

 Collecting: collecting all the quantitative and

qualitative data concerning the actions of the

students (such as movements and dialogues),

the use of the materials (for example, if a

resource has been viewed or not, how many

times and how long), and the participation in

the activities (such as given answers, forum

interventions, number of tasks delivered,

number of times a test has been performed,

evaluations achieved);

 Analyzing and Elaborating: analyzing and

elaborating all the data collected through the

technologies related to teaching, learning, and

the development of competences;

 Providing Feedback: giving the students

feedback on the activity carried out and

providing teachers, as well as students, with

the elaboration of learning data.

To schematize these functions, we propose the

diagram shown in Figure 1. The external cycle

represents the five functions; the black dashed arrows

represent how data are exchanged within the DLE

through automatic processes. The technologies of a

DLE, to accomplish one function, uses the data or the

outputs resulting from the previous one: the learning

materials, created through the LMS or other sector-

specific software through the “creating and managing

function”, are displayed via devices through the

“delivering and displaying function”. Information

about the students’ activities is collected by the LMS,

other software, or tools through the “collecting

function” and it is analyzed by these systems, which

may use mathematical engines, learning analytics

techniques, algorithms of machine learning, or

artificial intelligence, through the “analyzing and

elaborating” function. The results of the analysis are

feedback in the sense of Hattie’s definition (i.e.,

information provided by an agent regarding aspects

of one’s performance or understanding) (Hattie &

Timperley, 2007). They can be returned to students

and teachers through the “providing feedback”

function, and they can be used to create new activities

or edit the existing ones. This circle represents a

perfect adaptive system from the technological

perspective (Di Caro et al., 2018).

Figure 1: Diagram of the interactions among the

components of a DLE through the functions of the

technology.

In a human-centered approach, at the center of the

DLE, there is the learning community, composed of

students, teachers, and peers (who are the agents in

the Black and Wiliam’s theory of formative

assessment): they can interact with the DLE through

its functions receiving and sending information. The

blue dotted arrows represent the interactions between

the community and the digital systems that occur

through human actions, such as reading, receiving,

inserting, providing, digiting. For example, the

teacher, or designer, or tutor can create the digital

activities through the “creating and managing”

functions of the DLE; tasks are displayed (“delivering

and displaying” function) and received, seen, or read

by the students through some device. The students,

individually or with their peers, can insert their

answers or work. The technology collects them

through the “collecting” function. The system

analyzes the students’ answers and provides feedback

(“providing feedback” function) returned to the

student. Simultaneously, the information about the

students’ activity is returned to the teacher through

the “providing feedback” function; the teacher can

use it to edit the existing task or create new ones. The

continuous double-ended orange arrows represent the

interactions among students, teachers, and peers,

which in classroom-based settings can be verbal

while in online settings can be mediated by the

technology. This model allows us to identify some

outcomes that the adoption of a similar DLE with

AFA, through the functions previously shown, makes

it possible to achieve:

CSEDU 2021 - 13th International Conference on Computer Supported Education

500

 To Create an Interactive Learning

Environment: all the materials for learning

and assessment can be collected in a single

environment and be accessible at any time.

They can activate the students who can be

engaged in the navigation of the learning path,

solve the tasks and receive feedback;

 To Support Collaborative Learning,

through specific activities, delivered to groups

of students, which enhance the

communication and sharing of materials,

ideas, understanding;

 To Promote Formative Assessment, by

offering immediate feedback to students about

their results, their knowledge and skills

acquired, and their learning level. Feedback

can also be returned to the teachers on the

students’ results and their activities,

supporting decision-making.

The identification and classification of a DLE's

functions can allow us to identify the interactions in a

DLE, to analyze their nature and the contribution of

technology that mediates them. In this sense, the

diagram in Figure 1 is a proposal of schematization of

the interactions among the components of a DLE. It

helps us understand how data are shared among the

components of a DLE, elaborated, and used. The

information gained is useful from a learning analytics

perspective since it allows us to identify the role of

data during the learning processes. Moreover, this

model helps us identify and separate the functions and

outcomes of technology in learning processes, which

is necessary to have a clear frame and find causal

connections, especially when analyzing large data

quantities.

3 METHODOLOGY

In this study, we aim at showing how the diagram of

the interactions among the components of a DLE can

be used to model learning processes, and in particular

to understand how formative assessment can be

enacted in a DLE for Mathematics. To this purpose,

we analyzed an AFA activity concerning symbolic

computations for students of grade 8, experimented in

a classroom-based context. The task (Figure 2) asks

students to formulate, represent, and compare

different formulas derived from a geometrical shape.

The shape is non-standard and students are asked to

find as many formulas as they can to express its area.

Figure 2: Part of the activity on symbolic computations.

In the first section, they have 3 attempts to write a

formula for the area. In the second one, they are asked

to fill other 4 response areas with different formulas

expressing the same area. The intent is to make them

explore the symbolic manipulation of an algebraic

formulas through the geometric context, to confer a

more concrete meaning to the technical operations. In

the last part, students have to substitute the variable

with a given value and compute the area. This activity

was tested in a classroom-based setting in an

experiment involving 97 students of 4 different

classes of grade 8. In the classrooms there were the

teacher and 2 researchers of our research group; the

students worked in pairs using a computer or a tablet.

The work and discussions of some pairs of students

were recorded through a video camera. Data from the

platform were analyzed through the diagram of the

interactions in a DLE presented in the previous

section; the video recordings were analyzed as well.

4 RESULTS

We analyzed the videos realized during the activity in

the classrooms, to understand how the interactions

among the components of the DLE changed and how

the formative assessment strategies took place during

Analyzing Interactions in Automatic Formative Assessment Activities for Mathematics in Digital Learning Environments

501

a group activity. We choose some episodes which we

considered most significant. Here, the learning

community includes a class of students and a teacher;

the digital activities are created in a LMS integrated

with an AAS, and the devices used to access them are

an IWB and computers.

The first episode involves the teacher who

illustrates the task to the class. The teacher was at the

IWB and was pointing at the figure shown.

TEACHER: Look at this figure. Write the formula

which expresses how the area of this figure varies when a

varies. That is, [pointing at one of the sizes of the yellow

triangles] how long is this side?

STUDENTS: a

TEACHER: Well, you have to calculate the area of this

figure using a. Those sides measure a. What does it mean?

What is a?

STUDENTS: A variable.

In this excerpt, the teacher introduced the activity

and explained to the students what their task was. The

explanation took the form of a dialogue, as he

engaged the students with questions to make sure that

they were following the discourse. The teacher

exploited the “delivering and displaying” function of

the technology to display the task and, in particular,

the figure; then, she interacted with the students. If we

consider the diagram, we are in the right part; the

parts of the model involved in this excerpt are shown

in yellow in Figure 3. While explaining the tasks, she

developed the KS1 “clarifying and sharing learning

intentions and criteria for success”. The KS2

“engineering effective classroom discussions and

other learning tasks that elicit evidence of student

understanding” was accomplished during the phase of

the creation of this activity by the researchers (that we

can include in the “Teacher” subject of our analysis)

through the “creating and managing” function of the

technologies; it is also activated when the teacher

asks questions to the class aimed at making students

reason in the correct direction.

The second episode involves Marco (M) and

Giulia (G), two students of medium level who were

trying to solve the first part of the activity, working

together. In the beginning, they observed the figure

displayed on the screen of their computer and tried to

understand the task.

M: We have to compute the area, but we don’t have any

data!

G: But we have a.

M: But a is not a number!

G: Ok, but we can compute the area using a.

M: Teacher, how can we compute the area without

numbers? Can we use a?

T: Yes, it is like a generic number.

G: We have to write a formula using a, isn’t it?

T: That’s right.

Figure 3: Diagram of the formative assessment strategies

enacted in the first episode of activity 2 through the

interactions in the DLE.

The two students started reasoning together on the

figure trying a way to compute the area. After about

15 minutes, they came up with a quite complex

formula, built subtracting the area of the inner white

square to that of the external square. They used the

Pythagorean theorem to compute the length of the

white square’s side. They inserted the formula in the

response area and the system returned a green tick

with positive feedback. They passed to the following

part, which asked them to find other 4 formulas for

the same area. For the first two formulas, they

reasoned algebraically, manipulating the original

formula. For the other two, they reasoned

geometrically, developing new ways to compute the

area. The peer discussion allowed them to correct

mistakes before entering the formulas in the response

areas, so their answers were marked as correct at their

first attempt.

In this episode, the students look at the task

displayed on the screen through the “delivering and

displaying” function, then interact among them

discussing the task. They also interact with the teacher

asking questions about their doubts. Then they insert

their answers in the system, which collects them

through the “collecting” function, analyzes them, and

returns feedback. They repeat the same cycle several

times. The students activate KS4 “activating students

as instructional resources” when discussing in pair.

KS5 “activating students as the owners of their own

learning” is enacted when they insert their answers in

the AAS, and KS3 is developed when they receive

feedback from the AAS, but also by the teacher. The

yellow parts in Figure 11 schematize the interactions

CSEDU 2021 - 13th International Conference on Computer Supported Education

502

that occurred in this episode and the formative

assessment strategies developed.

Figure 4: Diagram of the formative assessment strategies

enacted in the second episode of activity 2 through the

interactions in the DLE.

5 DISCUSSION AND

CONCLUSIONS

The episode presented in the previous section helps

clarify how the interactions among the members of a

DLE occur during AFA Mathematics activities in a

classroom-based setting. The main feedback is

provided by the social interactions within the learning

community and especially among peers; in fact,

Marco and Giulia reasoned more time on the tasks

and they tended to answer correctly at the first

attempt. In other cases, we could observe that the

computerized interactive feedback has a key role in

providing a feedback and in engaging the students.

The design of the activity enables KS3 and KS5,

which keep students engaged with the task until its

full comprehension, demonstrated by the repeated

attempts. Similar activities can lead to a deep

understanding of fundamental Mathematics concepts;

the technologies and methodologies used – in

particular, an AAS based on a mathematical engine

and AFA – supported the design and implementation

of interesting activities for the development of

mathematical competences.

The diagram used for the analyses helped clarify

what functions of the technologies and through which

kinds of interactions the formative assessment

strategies are elicited in different situations. In

particular, we can see that all the Black and Wiliam’s

strategies of formative assessment can be enacted

through AFA activities, and all of them are identified

and located along the arrows of our diagram, that is

during the interactions among the human components

of the DLE or between human and technological

components. Thus, we can include a fourth agent in

Black and Wiliam’s framework: in the DLEs that we

consider, the technology is also an agent of the

formative assessment strategies, especially for

providing feedback and engaging students (KS3 and

KS5).

Through the analysis of the interactions among

the members of these DLEs, we can also point out that

the three outcomes mentioned in our framework are

achieved. In particular: the analyzed learning

environments are interactive, since students are

actively engaged in the activities, they are stimulated

to reflect and have the opportunity to achieve

important understanding; the formative assessment is

promoted by the activities, as all the 5 key strategies

are enacted; collaboration among students is

supported, especially in the classroom-based settings

where students are asked to work together.

The diagram used in the analyses allows us to

conceptualize the DLE as an ecosystem: we can see

that the human and technological components are

strictly related, and the interrelations among them

cause the development of the learning community, in

terms of learning processes, knowledge, and

competences gained; but also an improvement of the

learning activities on the base of the results obtained.

The analyses conducted in this study have a

qualitative nature: they are aimed at showing how the

schema of the interactions among the components of

a DLE can be used to model formative assessment

practices, especially when the AFA is adopted.

However, they can be a starting point for the research

about learning analytics for formative assessment.

This model can be used to create a taxonomy of the

interactions occurring in a DLE, identifying which

support formative assessment or other learning

processes. Since interactions in a DLE can be

described using log data, this model can also be used

with extensive learning data to identify the formative

assessment strategies or other learning processes

occurring in large online courses. This would allow

us to identify the learning activities which are better

related to the development of knowledge, abilities,

and competences or the elicitation of interactions and

engagement. The results of similar analyses could

help adjust and improve the digital materials in online

courses. Using other technologies and different

learning methodologies to build suitable activities,

this model of analysis could also be adapted to other

disciplines.

Analyzing Interactions in Automatic Formative Assessment Activities for Mathematics in Digital Learning Environments

503

REFERENCES

Abdelraheem, A. Y., 2003. Computerized Learning

Environments: Problems, Design Challenges and

Future Promises. The Journal of Interactive Online

Learning, 2(2), 1–9.

Barana, A., Brancaccio, A., Conte, A., Fissore, C., Floris,

F., Marchisio, M., & Pardini, C., 2019. The Role of an

Advanced Computing Environment in Teaching and

Learning Mathematics through Problem Posing and

Solving. In Proceedings of the 15th International

Scientific Conference ELearning and Software for

Education, 2, 11-18.

Barana, A., Conte, A., Fioravera, M., Marchisio, M., &

Rabellino, S., 2018. A Model of Formative Automatic

Assessment and Interactive Feedback for STEM. In

Proceedings of 2018 IEEE 42nd Annual Computer

Software and Applications Conference (COMPSAC),

1016–1025.

Barana, A., Conte, A., Fissore, C., Marchisio, M., &

Rabellino, S., 2019. Learning Analytics to improve

Formative Assessment strategies. Journal of E-

Learning and Knowledge Society, 15(3), 75–88.

Barana, A., Fissore, C., & Marchisio, M., 2020. From

Standardized Assessment to Automatic Formative

Assessment for Adaptive Teaching. In Proceedings of

the 12th International Conference on Computer

Supported Education, 1, 285–296.

Barana, A., & Marchisio, M. (2016). From digital mate

training experience to alternating school work

activities. Mondo Digitale, 15(64), 63–82.

Barana, A., & Marchisio, M., 2020. An interactive learning

environment to empower engagement in Mathematics.

Interaction Design and Architecture(s) Journal -

IxD&A, 45, 302–321.

Barana, A., Marchisio, M., & Miori, R., 2019. MATE-

BOOSTER: Design of an e-Learning Course to Boost

Mathematical Competence. In Proceedings of the 11th

International Conference on Computer Supported

Education (CSEDU 2019), 1, 280–291.

Black, P., & Wiliam, D., 2009. Developing the theory of

formative assessment. Educational Assessment,

Evaluation and Accountability, 21(1), 5–31.

Borba, M. C., Chiari, A. S. de S., & de Almeida, H. R. F.

L., 2018. Interactions in virtual learning environments:

New roles for digital technology. Educational Studies

in Mathematics, 98(3), 269–286.

Di Caro, L., Rabellino, S., Fioravera, M., & Marchisio, M.,

2018. A model for enriching automatic assessment

resources with free-text annotations. In Proceedings of

the 15th International Conference on Cognition and

Exploratory Learning in the Digital Age, 186–194.

Fissore, C., Marchisio, M., & Rabellino, S., 2020.

Secondary school teacher support and training for

online teaching during the covid-19 pandemic. In

European Distance and E-Learning Network (EDEN)

Proceedings

, 311–320.

Gagatsis, A., Michael-Chrysanthou, P., Christodoulou, T.,

Iliada, E., Bolondi, G., Vanini, I., Ferretti, F., &

Sbaragli, S., 2019. Formative assessment in the

teaching and learning of mathematics: Teachers’ and

students’ beliefs about mathematical error. Scientia

Paedagogica Experimentalis, 56(2), 145–180.

García-Holgado, A., & García-Peñalvo, F. J., 2018. Human

Interaction in Learning Ecosystems Based on Open

Source Solutions. In P. Zaphiris & A. Ioannou (Eds.),

Learning and Collaboration Technologies. Design,

Development and Technological Innovation. LCT 2018

(Vol. 10924, pp. 218–232). Springer.

Giovannella, C., Passarelli, M., & Persico, D., 2020. The

Effects of the Covid-19 Pandemic on Italian Learning

Ecosystems: The School Teachers’ Perspective at the

steady state. Interaction Design and Architecture(s)

Journal - IxD&A, 45, 264–286.

Guetl, C., & Chang, V., 2008. Ecosystem-based Theoretical

Models for Learning in Environments of the 21st

Century. International Journal of Emerging

Technologies in Learning (IJET), 3(1), 50–60.

Hattie, J., & Timperley, H., 2007. The Power of Feedback.

Review of Educational Research, 77(1), 81–112.

Khan, B. H., 1997. Web-Based Instruction: What Is It and

Why Is It? In B. H. Khan, Web-Based Instruction,

Educational Technology Publications (pp. 5–18).

Englewood Cliffs.

Marchisio, M., Rabellino, S., & Sacchet, M. (2020).

Start@unito as Open Educational Practice in Higher

Education. Journal of E-Learning and Knowledge

Society, 16(4), 46–55.

Marchisio, M., Remogna, S., Roman, F., & Sacchet, M.,

2020. Teaching Mathematics in Scientific Bachelor

Degrees Using a Blended Approach. In Proceedings -

2020 IEEE 44th Annual Computers, Software, and

Applications Conference, COMPSAC 2020, 190–195.

Suhonen, J., 2005. A formative development method for

digital learning environments in sparse learning

communities.

Uden, L., Wangsa, I. T., & Damiani, E., 2007. The future

of E-learning: E-learning ecosystem. In Inaugural

IEEE International Conference on Digital Ecosystems

and Technologies, 113–117.

Väljataga, T., Poom-Valickis, K., Rumma, K., & Aus, K.,

2020. Transforming Higher Education Learning

Ecosystem: Teachers’ Perspective. Interaction Design

and Architecture(s) Journal - IxD&A, 46, 47–69.

Watson, W. R., & Watson, S. L., 2007. An Argument for

Clarity: What are Learning Management Systems,

What are They Not, and What Should They Become?

TechTrends, 51(2), 28–34.

Wilson, B. G., 1995. Metaphors for instruction: Why we

talk about learning environments. Educational

Technology, 35(5), 25–30.

Wilson, B. G., 1996. Constructivist learning environments:

Case studies in instructional design. Educational

Technology Pubns.

CSEDU 2021 - 13th International Conference on Computer Supported Education

504