AN EVALUATION FRAMEWORK FOR M-LEARNING

Gerald C. Gannod and Kristen M. Bachman

Department of Computer Science and Software Engineering, Miami University, Oxford OH, U.S.A.

Keywords: m-Learning, Evaluation, Survey.

Abstract: We have developed an evaluative framework that can be used to place m-learning projects and technologies

within a context that associates a project with a broad learning objective. We do this through the

identification of dimensions within the aspects of FRAME, a framework developed by Koole (2009) that

looks at three different aspects: device, learning, and social. We have modified this framework to form what

we call Augmented FRAME. Augmented FRAME refines each of the aspects of FRAME into finer-grained

elements in order to gain a better understanding about the degree to which different approaches meet m-

learning goals. To illustrate this evaluation framework, we have surveyed a small but representative set of

m-learning approaches and discuss initial trends observed from using the framework.

1 INTRODUCTION

In order to help educators address issues related to

adopting and using m-learning approaches, we have

developed an evaluative framework that can be used

to place m-learning projects and technologies within

a context that associates a project with a broad

learning objective. We do this through the

identification of dimensions within the aspects of the

Framework for the Rational Analysis of Mobile

Education (FRAME), a framework developed by

Koole (2009) that looks at three different aspects:

device, learning, and social. We have modified this

framework to form what we call Augmented

FRAME, which refines each of the aspects into

finer-grained elements in order to gain a better

understanding about the degree to which different

approaches meet m-learning goals. For instance, we

have unpacked the social aspect to account for

critical learning activities most often associated with

communication, such as reading, writing, speaking,

and teaming. To illustrate this evaluation framework

we have surveyed a representative set of m-learning

approaches and discuss initial trends observed from

using the framework.

The remainder of this paper is organized as

follows: Section 2 describes background material,

including FRAME (Koole, 2009) to provide context

for the discussion contained later in the paper.

Section 2 also discusses our evaluative framework

while Section 3 surveys a number of existing

approaches from literature and provides some

evaluation of the context of these techniques by

placing them into the framework. Finally, Section 4

draws conclusions and suggests future

investigations.

2 APPROACH

m-learning brings a promise of extending the

learning experience beyond the classroom. Traxler

identified three properties that characterize effective

m-learning. Specifically, that m-learning has the

potential to provide an experience that is

personalized, authentic and situated (Traxler, 2007).

As the smart phone becomes the device of

choice, more K-12 institutions have begun to

explore how the device impacts learning while

lowering costs. While the benefits of m-learning are

intriguing, the challenges that accompany m-

learning pose barriers for adoption (Corbell and

Valdes-Corbell, 2007).

2.1 FRAME

FRAME (Koole, 2009) serves as the basis for the

work described in this paper. FRAME is a model

that describes how social interactions (social aspect),

mobile technologies (device aspect), and human

learning capacities (learning aspect) all work

330

C. Gannod G. and M. Bachman K..

AN EVALUATION FRAMEWORK FOR M-LEARNING.

DOI: 10.5220/0003349203300333

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 330-333

ISBN: 978-989-8425-50-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

together in forming an ideal m-learning

environment.

The device aspect focuses on the physical,

technical, and functional characteristics of a mobile

device. The learner aspect describes how learners

use their knowledge and how they encode, store, and

transfer information. Finally, the social aspect of

FRAME takes into account the processes of social

interaction and cooperation. (Koole, 2009)

In addition to the three aspects above, the

FRAME model discusses intersections. The Device

Usability (Device + Learner or DL) intersection ties

characteristics of mobile devices to cognitive tasks

related to manipulation and storage of information.

The Social Technology (Device + Social or DS)

aspect describes how mobile devices enable

communication and collaboration amongst multiple

individuals and systems. The Interaction Learning

(Learning + Social or LS) intersection focuses on

how learning is collaborative with meaning

negotiated from multiple aspects. Finally, the

Mobile Learning intersection (Device + Learner +

Social or DLS) refers to the “sweet spot” in the

FRAME model where all of the different aspects

come together to form a confluence of all of the

benefits of each concern (Koole, 2009).

2.2 Augmented FRAME

We found FRAME to be one of the few models that

attempts to catalogue different m-learning

techniques, but as is proves difficult for analyzing

current m-learning trends due to the broadness of the

categories. We have augmented FRAME in order to

facilitate a more fine-grained analysis of approaches

so that identifying where they fall in the framework

is more systematic. In particular, we have taken the

device, learner, and social aspects of FRAME and

identified different dimensions within each as a

means for differentiating between different m-

learning approaches. In this section, we discuss each

of the aspects in detail by introducing the additional

properties that we have identified.

Device. In studying different approaches, we

identified three additional characteristics that fall

under the FRAME category of device; namely type,

infrastructure support, and mobility. Type refers to

the kind of device being used in a particular

approach (e.g. netbook, cell phone, etc.).

Infrastructure Support refers to the kind of network

support required to facilitate the devices. Mobility

refers to the degree to which an approach supports

an un-tethered experience.

Learner. For this aspect we identified six

different learner-oriented characteristics that

differentiate m-learning approaches. In particular,

we were interested in whether the given approaches

facilitated certain kinds of learning as identified by

Traxler (2009) (e.g., personal, authentic, and situated

experiences as a form of information transfer). In

addition, we identified whether the approaches were

meant to facilitate authoring, content delivery

(content-oriented), or distance learning.

Social. We identified four characteristics that are

commonly associated with communication: reading,

writing, speaking/listening, and teaming (or

collaboration). In particular, the purpose of these

dimensions is to determine whether one form of

communication is being used more than others.

Since the use of mobile devices is in many ways a

visual activity, the use of the devices in the m-

learning approaches most often will involve reading,

but the more interactive activities of

speaking/listening and teaming appear to be more

interesting as they provide an ability to connect

learners that are not necessarily situated in the same

location. In regards to speaking/listening, we have

combined these skills into one area to cover forms of

one-way verbal or aural communication.

In order to organize and adequately compare

different approaches, we use a table to represent all

of the different properties described above. Section 3

contains an evaluation of twelve different

approaches that we have catalogued, with the

purpose being to support not just analysis of these

particular approaches, but to provide exemplars of

how to use the evaluation framework. In this sense,

we believe that our approach has some benefit in

that it facilitates:

• Characterization – the evaluation framework

provides a high-level view with respect to basic

FRAME while also facilitating a deeper look into

more specific characteristics that are related to

each of the FRAME aspects.

• Adoption – the evaluation framework facilitates a

broad understanding of different dimensions that

may encourage adoption.

• Comparative – the evaluation framework shows

that not all approaches meet different m-learning

goals and thus places each approach into

discernable contexts.

One of the primary tasks when using our evaluation

technique is the identification of where a given

approach falls within FRAME. Specifically, we are

interested in identifying which of the

aforementioned intersections an approach falls (e.g.,

DS, DL, LS, or DLS).

AN EVALUATION FRAMEWORK FOR M-LEARNING

331

Our taxonomy is meant to assist in providing

criterion for identifying how a particular m-learning

approach addresses the different FRAME aspects.

To do so, we arrange different approaches in a table,

with the different aspects (and their respective fine-

grained dimensions) in the columns. As an approach

is examined, the table is marked with properties that

characterize that approach (e.g., serves the

characterization role).

The information that is used to characterize

approaches provides a potential adopter with data

about the context of the approach within FRAME

and can then facilitate matching the approach with

learning goals. For instance, an adopter may be

interested in finding an approach that is heavy on the

social side (e.g., falls in the DS intersection). By

examining a table similar to the one shown in Table

1, an adopter can then view how the approach

addresses a particular aspect.

Another potential use of the Augmented FRAME

approach is as a comparative tool. The obvious

comparisons are between different approaches based

on the characterizations within the FRAME

intersections. However, another comparative use of

the framework is in the analysis of which

approaches meet certain dimensions within the

aspects. That is, as more approaches are catalogued

with our technique, identification of interesting areas

of investigation can be facilitated. For instance, if

we find that a particular dimension within an aspect

is not being adequately covered by existing

approaches, we can analyze whether that dimension

is indeed of interest for m-learning, and if it is, focus

attention on developing new methodologies that

address that particular area.

3 ANALYSIS

In order to evaluate the effectiveness of the

Augmented FRAME evaluation framework, we

analyzed a set of twelve approaches that target

audiences from K-12 to higher education.

Table 1 contains characterizations based on the

dimensions of Augmented FRAME. The Mobility

column refers to a 5-point scale, where 5 is the most

mobile and 1 is the least. In general, mobile

applications are typically in the 3-5 range, with

kiosks at a 2, and desktop or other fixed devices at a

1. An “X” in the Learner and Social columns

indicate that a particular approach addresses that

dimension. The Frame Intersection column uses the

acronyms of DS, DL, and DLS referred to earlier.

In constructing our table, we have observed

some interesting trends. While our analysis is far

from being comprehensive, we believe that these

trends are interesting enough to determine whether

they lead to new research or perhaps the

modification of our evaluation framework. One such

trend that we have observed is that most of the

approaches we have looked at involve reading as a

Table 1: Properties of M-Learning Approaches.

CSEDU 2011 - 3rd International Conference on Computer Supported Education

332

primary communication (Social) aspect. On the

surface, this would lead us to say that inherently,

mobile devices are visual instruments. However, this

perhaps leads to wondering about whether other

learning modes can be facilitated. For instance, aural

learners might be more responsive to audio.

Another trend we observed was that the personal

dimension within the Learner aspect was not

addressed. The fact that mobile devices are

considered personal devices means that we have the

opportunity to provide experiences that are highly

configured to meet the needs of the individual. In

addition, writing and speaking/listening found

limited support. Considering that these dimensions

are related to interactive communication skills, this

perhaps means that m-learning approaches are ripe

for addressing these skills.

Our primary goal in developing the Augmented

FRAME evaluation approach was to assist in

understanding the current state of m-learning. We

have found other evaluation frameworks to be

difficult to use for quickly sizing up approaches as

well as for looking at a big picture view of the field.

In order to further validate our approach, we intend

to expand the number of approaches that we

catalogue. In doing so, we hope to identify whether

some of the initial trends we have observed are true

of the field.

4 CONCLUSIONS

Mobile learning has received an influx of energy

with the release of mobile technology that has

offered a significant bump in utility. Features such

as GPS, cameras, accelerometers, magnetometers,

and other capabilities believed to be only wishful

thinking during the first generation of PDA in the

early 2000’s are now commonplace. As more

educational institutions move towards using m-

learning, effective tools that assist educators in

evaluating and selecting appropriate m-learning

strategies are needed. In this paper, we described the

Augmented FRAME evaluation framework based on

the FRAME evaluation model by Koole (2009).

In order to further validate our approach, we will

be focusing on building a larger catalogue of m-

learning techniques, with the intention of studying

trends as well as determining whether the

dimensions we have identified are sufficient.

Ultimately, our work is focused on using the

framework to inform policy makers about methods

to use by providing information about potential

learning outcomes that are relevant for m-learning.

REFERENCES

Cabrera, J. S., Frutos, H. M., Stoica, A. G., Avouris, N.,

Dimitriadis, Y., Fiotakis, G., & Liveri, K.D. (2005).

Mystery in the museum: collaborative learning

activities using handheld devices. Proc. of the 2005

ACM Mobile HCI Conference.

Corbell, J. R. & Valdes-Corbell, M. E. (2007). Are you

ready for mobile learning? EDUCAUSE Quarterly

Magazine, Vol 30, No 2.

Hagen, P., Robertson, T., Kan, M., & Sadler, K. (2005).

Emerging research methods for understanding mobile

technology use. Proc. of OZCHI 2005.

Huizenga, J., Admiraal, W., Akkerman, S., & ten Dam, G.

(2009). Mobile game-based learning in secondary

education: engagement, motivation, and learning in a

mobile city game. Journal of Computer Assisted

Learning, 25, 332-344.

Koole, M. L. (2009). A model for framing mobile

learning. In M. Ally (Ed.), Empowering Learners and

Educators with Mobile Learning. Athabasca, AB:

Athabasca University Press, 25-47.

Norris, C. & Soloway, E. (2004). Envisioning the

handheld-centric classroom. Journal of Educational

Computing Research, 30(4), 281-294.

Reckles, D. (2007). Lessons in wireless for K-12 schools.

Aruba Networks Whitepaper.

Schwabe, G. & Goth, C. (2005). Mobile learning with a

mobile game: design and motivational effects. Journal

of Computer Assisted Learning, 21, 204-216.

Sharples, M., Corlett, D., & Westmancott, O. (2002). The

design and implementation of a mobile learning

resource. Personal and Ubiquitous Computing, 6, 220-

234, Springer-Verlag.

Shen, R., Wang M., & Pan, X. (2008). Increasing

interactivity in blended classrooms through a cutting-

edge mobile learning system. British Journal of

Educational Technology, 39(6), 1073-1086.

Sung, Y.-T., Chang, K.-E., Lee, Y.-H., & Yu, W.-C.

(2008). Effects of a Mobile Electronic Guidebook on

Visitors’ Attention and Visiting Behaviors.

Educational Technology & Society, 11 (2), 67-80.

Tatar, D., Roschelle, J., Vahey, P. & Penuel, W. R. (2003).

Handhelds go to school: Lessons Learned. IEEE

Computer, September, 30-37.

Traxler, J. (2007). Defining, Discussing and Evaluating

Mobile Learning: the moving finger writes and having

writ… The Int’l Review of Research in Open and

Distance Learning, Vol 8, No 2, ISSN: 1492-3831.

van t’ Hooft, M. & McNeal T. (2010). Mobile Phones for

Mobile Learning: The Geo-Historian Project. Proc. of

the American Educational Research Assoc. Conf.

Zurita, G. & Nussbaum, M. (2004). Computer supported

collaborative learning using wirelessly interconnected

handheld computers. Computers & Education, 42,

289-314, Elsevier Publishers.

AN EVALUATION FRAMEWORK FOR M-LEARNING

333