TOM

A Multi-touch System for Learning Math

Daniela Blanke and Georg Schneider

Dept of Computer Science, University of Applied Sciences Trier, Schneidershof, Trier, Germany

Keywords: e-Learning hardware and software, Multi-touch table application, e-Learning application for mathematics.

Abstract: Learning of mathematical concepts is a crucial point for high school students as well as students in science

and engineering disciplines. In order to facilitate the learning of these abstract concepts we have developed

a system to interactively explore these concepts based on a multi-touch table. The table is used as I/O device

to additionally stimulate the haptic sense and to allow simple and intuitive gesture based interaction.

Consequently we intend to further improve the learning outcome when using the TOM (Touchable Math)

system. In this paper we will describe the concepts and the realisation of TOM, which is a modular

framework using Interactive Math Objects, which represent basic mathematical concepts together with

exercises related to these objects and a gesture based interaction concept.

1 INTRODUCTION

Learning mathematics is a demanding task,

especially when the concepts become more abstract

and complex. Whereas small children use toys

where they move pearls on a bar back and forth for

learning counting or simple mathematical operations

such as addition and subtraction to literally grasp

these ideas, no such support is given for more

complex mathematical tasks as middle school

students will encounter. Based on findings

concerning the process of learning, which we will

sketch in the following section our purpose is to

transfer these ideas to a learning application for the

more advanced mathematical topics.

In the following sections we will start with a

motivation of our approach based on results about

learning in general. Afterwards we will present in

the related work section systems that use a similar

approach for learning mathematical concepts. Then

we will introduce the concept of our system and

describe the realization on a multi-touch table. We

conclude the paper with a short summary and an

outlook for our future work.

2 MOTIVATION

The most natural way to understand our environment

is to touch and examine surrounding objects to con-

ceive their meaning. Small children naturally use

their hands to discover the world around them and

by doing so they make their first learning

experiences. Following these principles, in

elementary schools the teachers use tangible objects

to teach the children in counting and calculating.

The concept of "experiential learning" was

described in (Kolb, 1984). Kolb describes the

learning as a cycle of concrete experience, reflective

observation, abstract conceptualization and active

experimentation. This cycle is passed repeatedly and

leads the learner to a higher comprehension level.

Active experimentation is an important part of this

cycle. So the learner can deepen and reinforce the

learning contents.

These principles are also applied in applications

that use dynamical geometrical contexts to help

pupils and students to understand more complex

mathematical concepts as well. Many studies show

that learning with those applications as applets via

internet or as other types of programs provides

significant advantage for the learner. The user study

from Daher (Daher, 2009) analysing the use of

applets for solving mathematical problems shows

that applets are not really necessary but they can

help to "feel and see mathematical terms, objects

and concepts" (Daher, 2009, p. 389) because they

visualize the problems so that these problems can

easier be understood.

The study by (Martinez-Santaolalla, Bienvenida,

Tunez, 2005) explores the use of ICT in mathemati-

199

Blanke D. and Schneider G..

TOM - A Multi-touch System for Learning Math.

DOI: 10.5220/0003301601990206

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 199-206

ISBN: 978-989-8425-49-2

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

cal education for geometrical problems. They point

out that "dynamic geometry software" facilitates

specific types of learning activities, for example

exploration and visualisation, and can enhance

others, such as "proof and proving" (Martinez-

Santaolalla, Bienvenida, Tunez, 2005, p. 5).

The use of e-learning platforms in the internet

has been explored in the user study of (Albano,

Ferrari 2008). The interviewed students valued e-

learning as "more interesting" and the learning as

"faster and easier".

Regarding the findings of these studies, we can

conclude that dynamical maths and the use of

specialized software systems are beneficial for

pupils and students and help to visualize problems

and interact with mathematical terms as described in

the principles of Kolb's learning cycle.

In order to stress the fact of interactively

exploring mathematical concepts further we came up

with the idea to develop a mathematical application

using a multi-touch table. Those tables provide an

intuitive touch input with multiple fingers instead of

the traditional mouse interaction on a PC. So

dynamical geometry and the interaction on a multi-

touch table can be combined to design a new kind of

learning environment for pupils.

3 RELATED WORK

Following we will present the most relevant related

applications in the field of mathematical e-learning

as in regard to our application domain. A part of

them belongs to classical PC based programs. We

also discuss new applications already using multi-

touch tables.

3.1 Mathematical Learning Programs

for PC

The programs "GeoGebra" (Hohenwarter, Fuchs,

2004) and "Cinderella" (Kortenkamp, Richter-

Gebert, 2002) aim at a similar direction as our TOM

system. GeoGebra is a software for learning

dynamical geometry. Objects can be drawn and

altered after drawing. The coordinates of the points

and equations are also displayed but only in a

subordinate role.

To draw a special object, the kind of the object

has to be selected by a button before, i.e. recognition

of gestures is not part of the system. Additionally the

system can be used to solve mathematical problems,

e.g. from a textbook. The system itself does not

provide a functionality to integrate exercises for the

individual topics, which can be used to help a

learner to experience the different facets of a

problem and to guide and support the learning

process.

The program Cinderella is more comprehensive

than GeoGebra. A higher number of objects can be

drawn and the system provides more actions to be

performed with these objects. The price to pay for

this is that a longer training period is necessary to

use the entire functionality of the system. To draw

the objects and perform actions on these objects, the

pupils have to choose the type of the action or object

to draw using buttons comparable to GeoGebra.

An additional window must be opened to display

equations and the belonging coordinates.

As in GeoGebra the system can be used to solve

exercises from a textbook. However integrating

exercises into the system is not possible.

Both applications provide a range of

functionalities to interactively explore mathematical

objects. However the interaction is rather

cumbersome. Consequently, an application for

dynamical maths on a multi-touch table shall be

assessable by simple intuitive gestures. The

possibility to integrate exercises shall be envisaged

for the system as well.

3.2 Mathematical Learning Programs

on Tabletops

First mathematical applications for multi-touch

tables have been developed recently.

"Cinderella Multi-touch" is an interactive

geometry software on a multi-touch table. This

software is an experimental realization of the

"Cinderella" system on a tabletop, as described in

the previous section. It has been developed by

students of the University of Applied Sciences in

Schwäbisch Gmünd, Germany (Hochschule für

Gestaltung Schwäbisch Gmünd, 2010). The program

differs strongly from its PC version.

The target audience for this system are young

children. It has been designed to help them

understand shape and style of geometrical objects.

For example, the children can learn what a triangle

is and that it consists of three points and three

connecting lines.

"Smart Table" is a commercial teaching table.

(SMART, 2010). According to the informations on

the producers web site the table can be used for

different educational purposes. A mathematical

learning environment for simple mathematical tasks

like learning to count or simple geometrical games,

like for example Tangram puzzle, are provided. The

CSEDU 2011 - 3rd International Conference on Computer Supported Education

200

table is recommended for younger children and so

are the applications.

The "Teaching Table" (Khandelwal, Mazalek,

2007) is a table used for pre-kindergarten children.

This prototype uses an horizontal fixed LCD screen

covered with a sensing grid. This grid gathers the

interaction of the pupils with tangible objects on the

screen. Some exercises like recognition, assignment

and duplication of geometric shapes are

implemented as well.

Another approach in the use of mathematical

learning programs on tabletops originates from the

combination of CAS (Computer Algebra Systems)

with a multi-touch interface. The origin were

programs like “MathPad2” (LaViola, 2006) and

“MathBrush” (Labahn et. al, 2008) that have been

built for tablet PCs and were used to write

mathematical expressions through handwriting

recognition. In these systems the user can write the

expression with a pen on the screen instead of typing

it in on a keyboard. By writing special symbols the

user can trigger calculations or create the graphical

representation, which corresponds to the expression.

This technique has been recently adapted for

tabletops.

“NiCE” (Leitner et. al, 2010) is such a system. It

uses an Interpolated Force Sensitive Resistor (IFSR)

and can be used together with a desktop or a

withboard. Here, pen and multi-touch input are

combined. The written expressions can be solved

and rearranged by performing multi-touch gestures,

like combining two terms together.

Another project is “Hands-On Math” (Zeleznik

et. al, 2010). It has been developed for the Microsoft

Surface technology. It combines the technique used

in “NiCE” with the concept of virtual paper. The

expressions can be written on virtual pages and the

graphical representations are added to the pages.

Calculation gestures like dragging parts of the

expression or to split/join them are equivalent to

those presented in ”NiCE”.

The combination of CAS and digital pen and

multi-touch interfaces provides an easier use of

CAS. However the mathematical expressions can

only be altered but the graphical representation of

the expression cannot be manipulated. The graphics

can only be scaled but not changed.

Looking at the systems above we can sumarize,

that current existing mathematical applications on

multi-touch tables are either targeted to small

children without any relation of graph and

expression or they allow the modification of graphs

but do not provide sufficient feedback about the

corresponding changes of the equation or work.

Only the expressions are modified but not the

corresponding graph.

The system presented in this paper is based on

the idea of creating mathematical objects (like

GeoGebra and Cinderella), which can be

interactively modified. Different to these systems the

objects shall be closely tied to the representation.

Objects possess a representation, which is the

mathematical formula, that is updated in the same

way the objects are manipulated. The intuitive way

of interacting with the application shall be similar as

in the multi-touch implementations. As it concerns

the target audience, we develop the application for

maths education in middle schools. For this age, the

concepts can be more complex and the learners

possess already a certain level of abstraction.

4 CONCEPTS

OF THE TOM SYSTEM

The basic idea of the TOM system is to develop a

mathematical learning software for a multi-touch

table that uses the principles of dynamic geometry,

which shall help students to learn these concepts

easier and to gain a deeper insight in the interrelation

of these concepts.

The system should rather be used by pupils to

experiment and practise than by a teachers in a

classroom situation. So we decided to use a tabletop

instead of an interactive whiteboard.

4.1 Multi-touch Gestures

In order to incorporate a great amount of

interactivity in the system and to guarantee an

intuitive usage, all mathematical objects shall be

changeable by the user by performing multi-touch

gestures. Functions like straight lines, parabolas,

vectors etc. shall be drawn by using easy and

intuitive gestures. The recognition what to draw

shall be done only by the style of the gesture and not

by choosing a special draw option (e.g. click on a

"create an ellipse"-button) before.

To remove objects from the screen, also a

gesture shall be used. Therefore a simple wipe

gesture shall be performed on the object to remove

it.

4.2 Interactive Math Objects

In addition to the multi-touch gestures it shall be

possible, as far as it is mathematical useful, to drag

TOM - A Multi-touch System for Learning Math

201

and enlarge the elements or modify them in a

specific way. To emphasize the relation between the

function drawn and the equation, the equation shall

be drawn concise and in the same colour as the

graph. If the graph is changed, the equation shall

change accordingly.

Therefore we have decided to show both views

on the same screen by splitting the screen into a

drawing area and an area, in which the equations are

rendered. The drawing area is a coordinate system so

it is intuitively understandable where to draw the

objects because the students already know the

concept of a coordinate system from school.

The pupils shall have the option to experiment on

their own with the objects, to recognize how the

modifications refer to the equations and which part

of it is changed when the object is changed in a

specific way. For example, if the x radius of an

ellipse is changed the corresponding parameter in

the equation changes simultaneously. Thus the

different parts of such an equation do not have to be

learned by heart but can be experienced. In that way

the meaning of the equation will be retained much

better.

The user shall be able to draw and use objects

like straight lines, parabolas, ellipses, vectors and

angles. It shall be possible to modify the objects

shape and size as far as it is mathematical useful.

Besides matrix operations like rotation and

mirroring shall be feasible as well. Looking at this

topic from the opposite side, it shall also be possible

to provide the matrices and to check if the

corresponding object behaves as intended.

Furthermore the different objects shall be

brought in relation to each other so that things like

the slope of a parabola can be understood using the

concept of a tangent.

4.3 Built-in Exercises

To deepen the learning content, exercises to the

different topics shall be provided by the system as

well. These exercises shall be solved by drawing and

modifying the mathematical objects. The response

shall be communicated in an easy and intuitive way.

Therefore the concept of a traffic light is

incorporated. The traffic light is red as long as the

exercise is not solved correctly. If the answer

approaches the correct solution the light will turn to

yellow. When the answer is correct the light turns to

green and the next exercise can be selected.

In addition persons like tutors or teachers should

have the possibility to define own exercises.

Therefore a simple authoring tool shall be part of the

system where the question text and the test value can

be specified.

4.4 Tangible Objects

A connection to the physical world can be made by

using tangible objects. The use of objects equipped

with fiducial markers is conceivable. So those

objects can be used to identify different users or

objects, e.g. a sponge can be used to remove the

drawing.

5 SYSTEM DESIGN

In the following sections, we will sketch the system

design. We will present the hardware used to build

the TOM system. Afterwards we will illustrate the

realization of the concepts mentioned in the

preceding section.

5.1 Multi-touch Table

The multi-touch table we have used for the

development of our system has been developed in

our department. It uses the diffused surface

illumination (DSI) technology. The touch inputs are

gathered by CCV (community core vision) (CCV,

2010) and forwarded to the application by the

tangible user interface protocol (TUIO-Protocol)

(TUIO.org, 2010).

One of the reasons to build a table on our own is

to make use of the DSI technology (Nuigroup,

2010). Likewise it is possible to recognize physical

objects that are put on its surface.

5.2 Programming Language

and Framework used

The program has been developed in Java by using

the multi-touch framework MT4J (Multi-touch for

Java) (MT4j.org, 2010). This framework supports

the TUIO protocol and provides basic gesture

recognition. Additionally own gestures can be

defined.

Besides objects like rectangles, lines, ellipses and

polygons that can be used as basic objects for the

system graphical representations can be provided

implementing an individual view for these objects.

5.3 System Design

In general, our design aims at building a modular

system in a way that Interactive Math Objects,gestu-

CSEDU 2011 - 3rd International Conference on Computer Supported Education

202

Figure 1: Change x radius (1). Figure 2: Change x radius (2).

res and exercises can be added individually in order

to extend or adapt the system to the special needs of

a learning situation.

The user interacts with the system in the context

of so called scenes, which are presented to him. In

the main scene a coordinate system is displayed.

Different listeners are connected to the coordinate

system that process the drawing gestures. Objects

can be drawn into the coordinate system and in

parallel the corresponding equation is displayed in

the same colour as the graph. When the drawing is

modified the equation changes as well. All objects

can only be drawn, moved and modified inside the

coordinate system. In the next section the objects

and their modification operations are described in

more detail.

5.3.1 Realization

of the Interactive Math Objects

In the following we describe the objects realized yet

and the interaction with these objects.

5.3.1.1 Straight Line

A straight line is defined by two points. For this

reason we construct a line in our system by touching

the surface with two fingers. The straight line is

displayed by the two points and a line going through

these points.

Afterwards the Interactive Math Object can be

modified in different ways. If the user wants to drag

the line, he touches the line at an arbitrary position

(besides the two distinguished points) and the whole

straight line is moved within the coordinate system

following the gesture on the table.

Additionally each point can be moved separately

and also simultaneously.

Furthermore the straight line can be rotated if the

line is fixed at one point and rotated with a second

finger. If one point gets out of the coordinate system

during the rotation the straight line is shortened so

that the point will be kept inside the system.

5.3.1.2 Ellipse

Another Interactive Math Object is the ellipse. The

gesture to construct an ellipse is fairly intuitive as

well. Like drawing a circle with a compass a circle

can be drawn on the table by fixing the centre an

move another finger in a circular motion. In order to

facilitate the creation of this object the user must not

perform a whole circular movement. A segment of

circle is already sufficient, i.e. if the motion

describes an angle at about 60 degrees the gesture is

recognized and the ellipse with the same value for

the x and the y radius is drawn as a point for the

centre with a circle line.

The ellipse can be interactively modified as well.

If the centre is moved the whole ellipse is moved

along the dragging gesture as long as it stays inside

the coordinate system. Two fingers are necessary to

change the size of the ellipse. If the centre is fixed

the circle line can be moved with another finger.

Thereby the x and y radius are resized in equal ratio.

Additionally the x and y radius can be changed

separately. This is performed when the circle line is

pushed or pulled along the x or y axis. Figure 1 and

2 show the changing of the x radius. The hand

symbols represent the touching points. In figure 2

the changed equation parameter is marked.

If the x radius should be changed like in the

figures above the fingers have to be approximately

beside the centre. If the y radius should be changed,

the fingers have to be on the circle line

approximately above and below the centre.

5.3.1.3 Parabola

The third Interactive Math Object that we have

realized so far is a parabola. To draw a parabola a

three finger gesture is used. To fulfil the condition

TOM - A Multi-touch System for Learning Math

203

that three points are necessary to establish a parabola

equation. For the correct gesture the apex and two

point on the legs of the parabola are defined. These

points are than checked if they match the given

conditions and are slightly adjusted to match the

formula. Afterwards the values for the graph are

calculated by using the Gauss algorithm and the

graphic view and the corresponding equation are

displayed.

Afterwards the apex can be moved up and down

an the width an length of the legs can be changed

when the points are moved. The legs can only be

changed if both points are moved synchronously.

The parabola can also be moved through the

coordinate system if it is picked by the line. Similar

to the other Interactive Math Objects the

modification are assigned to the equation.

If the user intends to delete an object he may

perform a simple wipe gesture on the object to

remove that view and the corresponding equation.

To clear the coordinate system and remove all

graphs and equations the user can push the sponge-

button.

5.3.2 Calculation

of Robust Pointing Gestures

The precondition for the pointing gestures is that one

finger is close to an object. In order to simplify the

use of these gestures we use an adjustable threshold,

which allows that a gesture is recognized if the user

is "close enough" to the object.

The following code snippet shows the

calculation:

getFunctionValue(Vector3D p){

Vector3D pLoc = globalToLocal(p);

Vector3D cor = new Vector3D(

(pLoc.getX() - centerPoint.getX()),

(centerPoint.getY() - pLoc.getY()));

float v1 = (cor.getX() / radiusX) *

(cor.getX() / radiusX);

float v2 = (cor.getY() / radiusY) *

(cor.getY() / radiusY);

float value = v1 + v2;

return value;

}

boolean isOnTheLine(Vector3D p){

float val = getFunctionValue(p);

return (0.93 < val && val < 1.07);

}

First of all the function value for the drawing

object is calculated using the coordinate of one

touching point by the function "getFunctionValue".

For an ellipse, this value is lower than 1 if the finger

is inside the ellipse, greater than 1 if it is outside and

exactly 1 if the line got hit. The second function

checks if the value is approximately 1.

5.3.3 Menus

The systems comes up with three different types of

menus: a help menu, an exercise selection menu and

a menu that provides mathematical basics. All

menus are located at the edges of the screen and

look out a little bit like cue cards in a box. This

encourages the pupil to pull them out. The menus

can only be pulled out up to a specified value

likewise it is not possible to push the entire menu

under the edge. The basics menu and the exercise

selection menu are located at the left edge an can be

moved horizontally. The help menu is located at the

bottom and can be moved vertically.

The contents of both menus are created

dynamically depending on the current objects. So

they provide help and mathematical fundamentals to

the topics the user is currently interacting with. In

addition the help contains general information how

to draw and modify objects.

5.3.4 Exercises

The program provides exercises on different

mathematical topic. These exercises can be chosen

by the exercise selection menu. The queries, the

related button and the traffic light symbol are

displayed on the left area on the surface. The traffic

light shows a red light if the answer is not yet

correct, a yellow light if the answer is approximately

right and a green light it is correct. After that the

next question can be activated using the arrow

button. Some of the exercises require text input.

Therefore a soft keyboard can be displayed as input

device as well.

Figure 3 shows an example of an exercise. Here

the slope of the parabola at the position x = 1 shall

be shown with help of a tangent. The tangent is at

the correct position and the fact that the answer is

right is displayed showing the green traffic light.

Afterwards the next exercise can be chosen.

5.3.5 Fiducial Marker

Recently we have implemented a feature, which

allows to remove objects not only by performing the

already mentioned wipe gesture but also by using a

normal sponge. This functionality is realized by

sticking a printed out fiducial marker under that

sponge.

To remove an object the pupil has to put the

sponge on the table an wipe over the view in the sa-

CSEDU 2011 - 3rd International Conference on Computer Supported Education

204

Figure 3: Exercise example.

me way the finger wipe gesture is performed. The

use of a real sponge to delete functions is not

necessary but it is funny and could support the

pupils pleasure while working with the application.

Furthermore an identification of different users

by physical objects with fiducial markers will be

implemented so that for example already started

exercise could be saved and continued without a

ordinary user registration.

6 CONCLUSIONS

AND FUTURE WORK

Up to now we have implemented the basic

functionality of the TOM system. Consequently the

systems serves as a proof of concept that Multi-

touch tables could not only be used for mathematical

applications for kindergarten and elementary school

but provides advantages for middle school

education.

The next steps will be experiments with the users

of the targeted user group in order to receive

feedback concerning our concepts and

implementation.

We want to gather this information in order to

direct the future direction of our system, e.g. to

figure out if additional functionality is needed on the

existing Interactive Math Objects or if we can carry

on to explore the interaction with further objects.

REFERENCES

Albano, G., Ferrari, P. L. (2008). Integrating technology

and research in mathematical education: The case of

e-Learning. In Garcia Penalvo (ed.): Advances in e-

learning: Experiences and Methodologies, 132-148.

CCV, Community Core Vision, http://ccv.nuigroup.com,

(21.09.10).

Daher, W. (2009). Preservice Teachers' Perception of

Applets for Solving Mathematical Problems: Need,

Difficulties and Functions. Educational Technology &

Society, 12(4), 383-395

Hochschule für Gestaltung Schwäbisch Gmünd,

Cinderella Multi-touch Wiki, Techert T. et al.,

http://ig.hfg-gmuend.de/Members/thomas_techert/

meine-projekte/cinderella-dynamische-geometrie-am-

multi-touch, (23.08.2010).

Hohenwarter, M., Fuchs, K. (2004). Combination of

Dynamic Geometry, Algebra and Calculus in the

Software System GeoGebra. Computer Algebra

Systems and Dynamic Geometry Systems in

Mathematics Teaching Conference 2004. Pecs,

Hungary

Khandelwal, M. and Mazalek, A. (2007). Teaching table:

a tangible mentor for pre-k math education.

Processings of the 1st international conference on

Tangible and embedded interaction, 191–194.

ACMPress.

Kolb, D., (1984). Experiential learning: Experience as the

source of learning and development. Prentice-Hal.

Kortenkamp U., Richter-Gebert, J. (2002). Making the

move: The next version of Cinderella. Mathematical

Software, Proceedings of the First International

Congress of Mathematical Software, 208–216, World

Scientific.

Labahn, G., Lank, E., MacLean, S., Marzouk, M., Tausky,

D., (2008). MathBrush: A System for Doing Math on

Pen-Based Devices. The Eighth IAPR International

Workshop on Document Analysis Systems.

LaViola, J., (2006). An Initial Evaluation of a Pen-Based

Tool for Creating Dynamic Mathematical Illustrations.

Eurographics Workshop on Sketch-Based Interfaces

and Modeling. 157-164.

TOM - A Multi-touch System for Learning Math

205

Leitner, J., Rendl, C., Perteneder, F., Gokcezade, A.,

Seifried, T., Haller, M., Zeleznik, R., Bragdon, A.,

(2010). NiCE Formular Editor. SIGGRAPH 2010.

Martinez-Santaolalla, M. J., Bienvenida, B. F., Tunez, R.

S. (2005). ICT in Mathematics Education: geometry

problem solving with applets. Recent Research

Developments In Learning Technologies, La Canada

De San Urbano: Department of languages and

computation, University of Almeria.

MT4j.org, MT4J – Multi-touch For Java, http://mt4j.org,

(21.09.2010).

Nuigroup, Diffused Surface Illumination - NUI Group

Community Wiki, http://wiki.nuigroup.com/Diffused_

Surface_Illumination, (21.09.10).

SMART Technologies, SMART – SMART Table,

http://www.smarttech.de/index.php?page=smart-table-

tactile, (21.09.2010).

TUIO.org, TUIO, http://tuio.org/, _(21.09.10).

Zeleznik, R., Bragdon, A, Adeputra, F., Ko, H. (2010).

Hands-on math: a page-based multi-touch and pen

desktop for technical work and problem solving.

Proceedings of the 23nd annual ACM symposium on

User interface software and technology. ACMPress.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

206