EXPERIENCE WITH USABILITY TESTING OF WEB PORTALS

Andrina Granić

, Ivica Mitrović

and Nikola Marangunić

Faculty of Natural Sciences, Mathematics and Education, University of Split, Nikole Tesle 12, Split, Croatia

Arts Academy, University of Split, Glagoljaska bb, Split, Croatia

Keywords: End user testing, guideline inspection, methodology for web portal usability evaluation,

broad-reach web portals.

Abstract: The paper reports on the experience regarding usability evaluation of broad-reach web portals. The designed

methodology advocates a number of usability test methods along with specialists' inspection and provides a

comprehensive quantitative and qualitative assessment feedback. Our experience accords with the claim that

we should not rely on isolated evaluations, but instead on combine assessment methods. In this paper we

present and discuss mainly the results obtained through user-based usability evaluation. The results indicate

that chosen research instruments, measures and methods for usability testing were consistent. Conversely,

the results of the designed guideline inspection did not conform to the ones obtained through the end user

testing. Although proved promising, some of the guideline-based evaluation aspects need improvement.

1 INTRODUCTION

Usability is the key issue in human-computer

interaction (HCI) and is related to ease of use and to

ease of learning. It becomes defined within a

relationship among task, user and system purpose.

For this reason there is no simple definition, or

meaningful single measure of usability.

When considering usability of a web portal, a

site that functions as a point of access to information

on the Web, it should be noted that current research

on usability evaluation is mostly concerned with

focused, domain specific portals. This is the result of

the global trend of portal specialization. In such

context it is difficult to find any studies related to

broad-reach web portal assessments.

This paper reports on the experience regarding

the design of a methodology for usability evaluation

of broad-reach web portals. The main motivation

for our research came from reports stating that the

most visited Croatian web sites are broad-reach web

portals, implying end users and designers' familiarity

with such kind of sites.

We conducted the experiment in order to

evaluate how easy and efficient those broad-reach

portals are. Portal usability is viewed as efficacy in

use, considered primarily as involving measures of

user performance and ease of use, envisaged as

involving subjective judgments. Additionally it is

augmented with a significant feedback acquired

through guideline inspection, a less formal heuristic

evaluation.

A strong point in such an approach is the chance

to supplement results from the guideline-based

assessment with the user-based one, enhanced with a

feedback on the users' pleasure while working with

the portal. The results go in hand with the assertion

that we should not rely on isolated evaluations.

Instead, usability assessment methods should be

combined, hence obtaining different kinds of

usability improvement suggestions.

In this paper we discuss and present interpretations

of the results achieved through employed user-based

usability methods. The complementing test methods

proved to be consistent. Conversely, results of the

designed guideline-based evaluation were not in

agreement with the ones obtained from the usability

testing. Consequently, our method for guideline

inspection raised a couple of concerns which will be

addressed in our future work.

The paper has the following organization.

Section 2 briefly introduces some background of the

research, addressing web portals and web usability,

providing the rationale for the methodology. An

approach to usability evaluation along with achieved

results is described in what follows. Section 4 brings

the discussion and interpretation of findings,

161

Grani

c A., Mitrovi

c I. and Maranguni

c N. (2008).

EXPERIENCE WITH USABILITY TESTING OF WEB PORTALS.

In Proceedings of the Fourth International Conference on Web Information Systems and Technologies, pages 161-168

DOI: 10.5220/0001520801610168

 SciTePress

additionally offering some directions for future

research. Section 5 closes the paper.

2 WEB PORTALS AND WEB

USABILITY

Web portal, generally considered as a single point of

access to information, resources and services on a

wide range of topics (Waloszek, 2001), is typically

based on the more advanced technologies that go

beyond simple interface of the just information

based standard web page. Broad-reach portals, also

called "general" or "generic" portals, bring together

services such as search engines, e-mail, news,

forums, event guides, maps, on-line shopping, travel

information and the like. Accordingly, information

presented in every page can be delivered to a

number of users having different needs, motivations

and goals which the portal design has to reflect.

Market research findings related to the Croatian

web sphere undertaken in the last few years report

that broad-reach portals are the most visited web

sites (GFK Croatia, 2006; GemiusAudience, 2007).

This is the basic distinction from the countries with

high level of Internet literacy, where there are much

more specialized web portals seen as gateways to

the variety of web information related to the specific

context (Tatnall, 2005). In terms of the choice of

content and layout, major Croatian broad-reach

portals have the semblance of the web sites of

broadsheet newspapers or public service

broadcasters (Tomić-Koludrović and Petrić, 2004).

On the other hand, an improvement of usability

as a quality of use in context (Bevan and Macleod,

1994) is nowadays perhaps the most important goal

of current research in the field of human-computer

interaction. Most used assessment methods are

grouped into two categories, e.g. (Nielsen, 1993;

Lewis, 2005):

• usability test methods; user-based which involve

end users, hence including user testing, focus

groups, interviews, questionnaires and surveys;

• usability inspection methods, without end users

consisting of heuristic evaluations and cognitive

walkthroughs as the most often used ones.

Research related to web usability recently has

had a tendency to bring together those two basic

approaches, cf. (Hornbæk, 2006). Concerning web

portal usability, it should be noted that the current

studies are mostly concerned with domain specific

portals. Namely, in the context of the global trend of

web portal specialization, recent research related to

usability evaluation is mostly concerned with the

focused portals (somewhere called "vertical" or

"domain specific" ones) such as enterprise or

corporate portals (Boye, 2006), travel portals (Shelat

and Stewart, 2004; Carstens and Patterson, 2005),

news portals (Tsui and Paynter, 2004), library web

portals (Brantley et al., 2006), tourist portals

(Klausegger, 2006), healthcare web portals (Theng

and Soh, 2005) and similar.

Apparently, while there is a number of studies

related to usability evaluation of specialized web

portals, some additionally taking into account the

particular cultural context e.g. (Theng and Soh,

2005; Tsui and Paynter, 2004; Fang and Rau, 2003),

there is hardly any research dealing with broad-

reach portal assessment.

Taking into consideration outcomes of the

undertaken research concerning web portals and

usually employed usability assessment methods, the

aim of our study is related to the design of

appropriate methodology for broad-reach portal

usability evaluation. The study is placed in Croatian

web sphere where broad-reach web portals are much

more popular and accepted than specialized ones.

Such context seems appropriate for this research,

since it implies end users and designers familiarity

with such kind of portals.

3 THE EXPERIMENT

In order to achieve the key objective, the design of

the most visited Croatian broad-reach portals was

evaluated. We conducted a controlled experiment

which advocates scenario-guided user evaluations

involving a number of usability testing methods used

to collect both quantitative data and qualitative

"remarks", cf. (Shackel, 1991). Furthermore, user-

based testing is supplemented with less strict

heuristic evaluation (Nielsen, 1994), i.e., guideline

inspection. This follows the literature that suggests

that usability inspection needs to be combined with

usability test methods, e.g. (Holzinger, 2005;

Hornbæk, 2006; Uldall-Espersen et al., 2007).

Such approach, understood to be a combination

of behaviour and opinion based measures obtained

under experimental control which is additionally

augmented with expert assessment, is described in

the following.

3.1 Methodology

In the following sections we describe the

experimental methodology adopted to assess the

effectiveness, efficiency and satisfaction in addition

WEBIST 2008 - International Conference on Web Information Systems and Technologies

162

to expert evaluation of selected Croatian broad-reach

portals and the results obtained.

3.1.1 Participants

The study involved thirty participants with basic

computer literacy. According to their practical

experience in web design, they were classified in

two different groups composed of fifteen

participants. The "practitioner" group was composed

of three independent sub-groups of randomly chosen

participants including computer science experts,

marketing experts knowledgeable in Internet issues

as well as students of web design. On the other hand,

the "non-practitioner" group was consisted of three

independent sub-groups of randomly chosen

participants consisting of young participants, middle

aged and elderly ones.

Furthermore, a particular group of ten "instant

experts" (Wright and Monk, 1991) for guideline-

based evaluation was formed. Those were web

design practitioners who are familiar with the

principles of the good user-centred designs in

addition to the evaluation approaches and who

provided usability expert assessment of the selected

web portals.

3.1.2 Instruments, Measures and Portals

User assessment was conducted individually, with

Internet access and a screen capturing software for

tracing the users' actions and navigation. We

measured task time and achievement. End user

testing was based on criteria expressed in terms of

few measures (ISO/IEC, 2006):

• objective performance measurement of

effectiveness (percent task completion) in using

the portal,

• objective performance measurement of efficiency

(time on task) in using the portal and

• users' subjective assessment of the web portal

usage.

System Usability Scale (SUS), as a simple standard,

ten-item attitude questionnaire with five-point

Likert scale (Brook, 1996), was used for the

subjective valuation. As an additional subjective

feedback, answers to the semi-structured interview

were collected.

In order to perform a less formal heuristic

evaluation, specifically guideline inspection, an

evaluation form consisting of a set of adapted

principles augmented with portal-related auxiliary

guidelines was prepared. Individual expert’s marks

and comments were collected. The score for every

portal was calculated as an average mark on a seven-

point Likert scale.

We included four broad-reach web portals in

our experiment: Index portal (www.index.hr), Net

portal (www.net.hr), Vip portal (www.vip.hr) and T-

Portal portal (www.tportal.hr). As already stated

before, those portals were selected as the most

visited whilst also the earliest broad-reach web

portals. For an insight to a broad-reach portal user

interface, a screenshot of the Vip portal is illustrated

in Figure 1.

Figure 1: Screenshot of Vip broad-reach web portal.

3.1.3 Experiment Design and Procedure

In order to understand the effect of web portal

design in a sample work situation, we elaborated a

work scenario, a sequence of typical tasks and user

actions. To test assigned tasks and time interval,

clarity and unambiguousness of measuring

instruments for subjective assessment and adequacy

of hardware and software support, pilot testing was

performed.

We chose three typical tasks which structure and

location on the portals was not changed over time.

The tasks covered different topics, offering to

diverse groups of involved participants' a similar

opportunity for finding task-related information. For

each selected portal, undertaken tasks were the same

and the probability of their completion was similar

as well. See Figure 2 to get insight into the

performed tasks.

The evaluation procedure was carried out

individually with each test user, using a personal

computer with Internet access in addition to a

software and hardware support for tracing and

recording results of task completion. Within each

evaluation session all the portals were assessed,

where the order of their evaluation was randomly

selected. The allocated session's average time per

every participant was 45 minutes.

EXPERIENCE WITH USABILITY TESTING OF WEB PORTALS

163

Figure 2: Undertaken tasks.

An evaluation procedure consisted of the following

steps:

• task-based end user testing,

• usability satisfaction questionnaire,

• semi-structured interview and

• guideline inspection.

Task-based end user testing involved a scenario-

guided user assessment with three tasks selected to

show the portal basic functionality. It enabled us to

determine user efficiency and effectiveness while

working with the web portal. A user's objective

accomplishment measure, labelled as fulfilment, was

calculated as an average time spent on all allocated

tasks weighted with successfulness of task

completion. For each user, the time limit for all

assigned tasks was 15 minutes per portal.

An usability satisfaction questionnaire supported the

assessment of the users' subjective satisfaction with

diverse interaction aspects. We used SUS

questionnaire, as it is argued that it yields the most

reliable results across sample sizes (Tullis and

Stetson, 2004). Its questions addressed different

aspects of the user's reaction to the portal as a whole,

providing an indication of a level of statement

agreement on a five-point Likert scale. Furthermore,

the feedback from the questionnaire was augmented

with the users' answers in a semi-structured

interview. In this interview we additionally asked the

participants to rate and comment on the portal’s

visual attractiveness as well.

End user testing of web portal usability

additionally brought together with guideline-based

evaluation provided a more precise broad-reach

portal usability assessment. In order to overcome the

problem of not having enough usability experts who

could be involved in the portal evaluation process,

we had the guideline inspection performed by

"instant experts", web design practitioners familiar

with the HCI principles. A detailed evaluation form

with Nielsen's usability heuristics (Nielsen, 1994), as

a set of ten key principles, was adapted to portal

context and augmented with a series of auxiliary

guidelines, as additional explanations of web portals

design cf. (Granić et al., 2004; Wood, 2004; MIT,

2004; Preece et al., 2002; Nielsen and Tahir, 2002).

In the provided evaluation form experts had (i) to

specify a level of their agreement with the

principle/guideline and related set of auxiliary

guidelines on a seven-point Likert scale as well as

(ii) to provide comments in order to justify the

assigned mark. Furthermore, remarks and

observations concerning the overall guideline-based

assessment procedure were more then welcome.

The guideline-based evaluation coupled with the

task-based end user testing, the attitude

questionnaire and the interview, provided a

sufficient quantitative and qualitative assessment

feedback. In the following we present experimental

results and findings, considering in details the ones

obtained by means of usability testing. Namely, due

to paper length limitations, the results of guideline

inspection will be addressed in our further report.

3.2 Results

Results acquired through the usability test methods

in addition to the main findings obtained in

guideline-based inspection are addressed in what

follows.

3.2.1 Results of Usability Test Methods

Descriptive statistics of the objective

accomplishment measure fulfilment, including

arithmetic means, standard deviations and

significance levels of Kolmogorov-Smirnov

coefficient for normality of distribution is presented

in Table 1. Result of the distribution for measure

fulfilment on the T-Portal web portal differs

significantly from normal distribution (K-S = 0.008).

Accordingly, Friedman's test as a non-parametrical

procedure was performed. Statistically significant

value of chi square (χ

= 49.4, df = 3, p < 0.01)

indicates the existence of differences in the objective

accomplishment measure among portals (see Table

1).

SCENARIO FOR TASK-BASED

TESTING

Imagine that you are interested in some information

related to diverse aspects of everyday living. In order to

find relevant data, you are kindly asked to refer to the

respective site of Index web portal (www.index.hr) and

complete three tasks.

The tasks are the following:

1. find information related to the traffic situation on

A1 highway in the direction from Zagreb to Split as well

as the daily temperature in Split

2. search for the today’s first show on RTL

television which is scheduled after 9 PM

3. find and print one recipe for the main meal

After your complete the allocated tasks, you are

kindly asked to proceed with the evaluation session

WEBIST 2008 - International Conference on Web Information Systems and Technologies

164

Table 1: Results of objective accomplishment measure fulfilment for the four selected portals (note that lower M score

means better result).

fulfilment M SD K-S M Rank df χ

Index portal 59.77 38.726 0.292 1.57 3 49.4 <0.01

Net portal 108.40 46.300 0.720 2.93

Vip portal 62.13 17.211 0.656 1.87

T-Portal portal 171.64 168.143 0.008 3.63

Table 2: Results of subjective satisfaction measure SUS for the four selected portals (note that higher M score means better

result).

SUS M SD K-S df F P

Index portal 75.33 18.820 0.819 29 746.94 <0.01

Net portal 56.00 25.194 0.902

Vip portal 77.83 15.821 0.319

T-Portal portal 51.75 23.836 0.961

Descriptive statistics of results acquired for

subjective satisfaction measure SUS for each and

every participant on every web portal is shown in

Table 2. No statistical difference in the distribution

of the results from the expected normal distribution

was found (K-S

1,2,3,4

> 0.05).

In order to test the difference among portals, the

analysis of variance as a parametric procedure was

applied. Significant F-ratio (F = 746.94, df = 29,

p < 0.01) indicates the existence of differences

among the portals in the results related to the

obtained subjective measure (see Table 2).

Additionally, we also considered all accomplished

experimental results related to the two groups of

participants – the practitioner group and the non-

practitioner one. The differences in the user's

objective accomplishment and subjective

satisfaction usability measures between the two

groups were tested with t-tests for small independent

samples.

Statistically significant difference between the

groups was found for results of the fulfilment

measure (t = 2.95, p < 0.01). The group of

practitioners showed better results on mean values

(mean = 308.4, SD = 57.217) than the non-

practitioners group (mean = 495.46, SD = 238.479).

The mean fulfilment results are shown in Table 3.

Table 3: Results of objective accomplishment measure

fulfilment for the two groups of participants (note that

lower score means better result).

fulfilment practitioners non-practitioners

N 15 15

M 308,40 495,46

SD 57,217 238,479

t 2,95

df 28

p <0,01

On the other hand, a statistically significant

difference was not found for the results related to the

subjective satisfaction measure SUS (t = 1.95,

p = 0.062) between the practitioner group of

participants (mean = 243.17, SD = 51.317) and the

non-practitioner one (mean = 278.67, SD = 48.531).

Pearson's correlation coefficients for the participants'

results in the achieved usability objective and

subjective measures are shown in Table 4.

Table 4: Pearson's correlation coefficients of overall

results (* significant correlations at level of p<0.05).

Correlations of overall results r

SUS –fulfilment 0.14

SUS –visual attractiveness 0.41*

A statistically significant correlation coefficient was

not found between overall SUS and overall

fulfilment. Conversely, significant correlation was

EXPERIENCE WITH USABILITY TESTING OF WEB PORTALS

165

found between overall SUS and score for overall

visual attractiveness.

3.2.2 Results of Usability Inspection Method

The arithmetic means of the marks from a seven-

point Likert scale of the guideline evaluation forms

for assessed portals show the following results. The

highest mark was given to Vip web portal

(mean = 5.38), followed by Net portal

(mean = 4.85), T-Portal portal (mean = 4.64) and

Index portal (mean = 4.01).

Furthermore, we considered the outcomes

achieved in the guideline-based evaluation in

addition to the ones obtained throughout the

usability testing. The ranking of the selected broad-

reach web portals was compared.

Considering the objective accomplishment

measure, Index portal and Vip portal scored the best

result, followed by Net portal and T-Portal web

portal. The subjective satisfaction measure provided

similar rankings too. On the other hand, in the

guideline-based evaluation Index portal achieved the

lowest rank, while Vip portal remained on a high-

level, followed by Net portal and T-Portal broad-

reach web portal.

4 DISCUSSION AND FUTURE

WORKS

4.1 Interpretation of the Results

The results of the task-based end user testing

showed statistically significant differences among

the assessed portals according to the measure of

user's objective accomplishment. This suggests that

web portals could be ranked by mean values. The

results of the subjective satisfaction measure also

showed differences among evaluated portals and

their ranking by mean values.

It is important to point out that the measures of

user's objective accomplishment and her/his

subjective satisfaction were not significantly

correlated. Our results are in accordance with the

ones of the meta-analytic research report on

correlations among usability measures calculated

from the raw data of 73 studies (Hornbæk and Law,

2007). This yet again implies the appropriateness of

the applied usability test methods for portal

evaluation, since it confirms independence of the

objective and subjective measurements.

The overall achieved results could be further

related to the most frequent statements from the

interviews. Namely, the participants felt especially

pleased and comfortable working with the portals

where their objective achievement was high. The

end users considered them as broad-reach web sites

with good quality of information structure,

respectable layout and straightforward navigation.

Correlation between overall SUS results and

overall visual attractiveness indicates that a pleasant

appearance influences the subjective perception of

portal usability. The interview statements also

support this finding. Namely, it happened that the

users usually emphasized the portals' visual

attractiveness, assigning high subjective ratings.

Such assumption is in line with related studies

which, along with the related HCI issues, also

address aesthetics aspects of design, cf. (Tractinsky

et al., 2000; Hassenzahl, 2004).

The results of the objective accomplishment

measure revealed expected differences between non-

practitioners and practitioners, the latter being faster

and more successful in the tasks' achievement. This

indicates that the selected tasks and the objective

accomplishment measure were consistent. On the

other hand, the measure of subjective satisfaction

did not show any statistically significant difference

between these two groups. Such finding indicates

that the questionnaire itself and its translation to

Croatian language could be considered as an

appropriate instrument for user subjective

assessment.

Moreover, our experience suggests that the

choice of the sample size in addition to the structure

of engaged end users ("practitioner" and "non-

practitioner" groups of participants) is also in line

with the outcomes of related studies. Specifically, in

the Hornbæk and Law's (2007) meta-analysis of

usability measures, the average number of

participants involved per study was 32 (SD = 29,

ranging from 6 to 181).

Conversely, the achieved result of guideline-

based assessment did not conform to the ones

obtained throughout applied usability testing. The

highest ranked web portal in the end user testing

scored as the lowest one in the "instant expert"

usability evaluation. There are two possible reasons

for such an outcome − the designed evaluation form

and/or the selection of usability specialists involved

in the web portal guideline-based assessment.

Concerning the first reason, some of adapted

Nielsen’s principles showed poor applicability in the

web portal context, not providing any useful

information for portal usability improvement.

Moreover, a number of guidelines should be more

comprehensible and auxiliary guidelines revised and

redundant ones excluded. Accordingly, a new set of

guidelines is needed, the one which is not so strictly

based on Nielsen’s heuristics.

Regarding the choice of usability "instant

experts", significant difference in acquired

WEBIST 2008 - International Conference on Web Information Systems and Technologies

166

information suggests non-homogeneity of the group

concerning their HCI expertise. Such problem was

hard to prevent due to inadequate number of resident

HCI specialists as well as high costs of possible

foreign experts' engagement.

Although our guideline inspection has proved

very promising, some of its aspects need

improvement. Conducted assessment obtained

comprehensive quantitative and qualitative data

which analysis defined the first steps of our future

work: (i) revision of the designed evaluation form

and (ii) subsequent assessment with adequate sample

of experts.

4.2 Future Steps

The results acquired throughout the applied

methodology show that the designed guideline

inspection, as a less formal heuristic evaluation,

although proved very promising needs to be

improved. This is in line with recent research on

heuristic evaluation which is focused on improving

its effectiveness and efficiency with respect to user

testing, cf. (Hvannberg et al., 2007). Consequently,

our future work will be focused on quantitative and

qualitative analysis of the results obtained through

guideline-based assessment and will first and

foremost address two important aspects: the

evaluation form issues and the instant experts

selection.

Furthermore, future work will subsequently go

into two directions also taking into account the

following issues:

• first, in order to upgrade the applied usability

inspection method and develop a methodology

for web portal usability evaluation (i) the instant

experts selection and the evaluation form issues

should be revised according to the quantitative

and qualitative analysis of the obtained results

and (ii) the redesigned methods should be

applied for the assessment of more specialized

Croatian web portals

• second, in order to improve the applicability of

the developed methodology to practice and to

achieve its broad generalization, an inclusion of a

cross-cultural sample should be considered.

5 CONCLUDING REMARK

Design of most visited Croatian broad-reach web

portals was assessed both through a number of

usability test methods with diverse groups of end

users on the one hand and the usability inspection

method on the other. The designed evaluation

methodology, as an approach which is based on (i)

scenario-guided user evaluations used to collect both

quantitative data and qualitative "remarks" and (ii)

specialists' assessment, provided a comprehensive

valuation feedback.

The experience reported on in this paper

indicates that the chosen research instruments,

measures and methods for user-based usability

evaluation were consistent. Conversely, the results

of the designed usability inspection method were not

in agreement with the ones obtained from the user

test methods. Consequently, the guideline-based

evaluation, even though showing respectable

potential, raised a couple of concerns which will be

addressed in our future work.

ACKNOWLEDGEMENTS

This paper describes the results of research being

carried out within the project 177-0361994-1998

Usability and Adaptivity of Interfaces for Intelligent

Authoring Shells funded by the Ministry of Science,

Education and Sports of the Republic of Croatia.

Experiment was conducted at Department of

Visual Communication Design, Arts Academy,

University of Split.

REFERENCES

Bevan, N., Macleod, M., 1994. Usability measurement in

context. Behaviour & Information Technology, 13, pp.

132-145.

Boye, J., 2006. Improving portal usability. CMS Watch.

http://www.cmswatch.com/Feature/151-Portal-

Usability.

Brantley, S., Armstrong, A., Lewis, K.M., 2006. Usability

Testing of a Customizable Library Web Portal.

College & Research Libraries 67, 2.

Brooke, J., 1996. SUS: a "quick and dirty" usability scale.

In Jordan, P.W., Thomas, B., Weerdmeester, B.A,

McClelland, A.L. (Eds.): Usability Evaluation in

Industry. Taylor and Francis, London.

Carstens, D.S., Patterson, P., 2005. Usability Study of

travel websites. Journal of Usability Studies, 1, 1.

Fang, X., Rau, P.L.P., 2003. Culture differences in design

of portal sites. Ergonomics 46, 1-3.GemiusAudience,

2007. http://www.cati.hr/

GemiusAudience-ozujak.pdf.

GFK Croatia, 2006. http://www.gfk.hr/press1/

internet.htm.

Granić, A., Glavinić, V., Stankov, S., 2004. Usability

Evaluation Methodology for Web-based Educational

Systems. In Stary, C., Stephanidis, C. (Eds.):

Proceedings of the 8

ERCIM Workshop "User

EXPERIENCE WITH USABILITY TESTING OF WEB PORTALS

167

Interfaces for All". ERCIM, Heraklion (Crete),

Greece, 28.1-28.15.

Hassenzahl, M., 2004. The interplay of beauty, goodness,

and usability in interactive products. Human-

Computer interaction 19, 4.

Holzinger, A., 2005. Usability Engineering methods for

Software developers. Communications of the ACM 48,

1, 71-74.

Hornbæk, K., 2006. Current practice in measuring

usability: Challenges to usability studies and research.

International Journal of Man-Machine Studies 64, 2.

Hornbæk, K., Law, E.L-C., 2007. Meta-Analysis of

Correlations Among Usability Measures. CHI 2007

Proceedings, April 28–May 3, San Jose, California,

USA.

Hvannberg, E., Law, E.L-C., Larusdottir, M., 2007.

Heuristic evaluation: Comparing ways of finding and

reporting usability problems. Interacting with

Computers 19, 225–240.

ISO/IEC 25062:2006, 2006. Software engineering –

Software product Quality Requirements and

Evaluation (SQuaRE) – Common Industry Format

(CIF) for usability test reports.

Klausegger, C., 2006. Evaluating Internet Portals - An

Empirical Study of Acceptance Measurement Based

on the Austrian National Tourist Office's Service

Portal. Journal of Quality Assurance in Hospitality &

Tourism 6, 3-4.

Lewis, J. R., 2005. Introduction to Usability Testing.

Tutorial given at HCI International 2005. July 22-27.

Las Vegas, Nevada.MIT Usability Guidelines, 2004.

http://web.mit.edu/is/usability/selected.guidelines.pdf.

Nielsen, J., 1993. Usability Engineering. Academic Press

Limited. Boston.

Nielsen, J., 1994. Heuristic Evaluation. In Nielsen, J.,

Mack, R.L. (Eds.): Usability Inspection Methods. John

Wiley and Sons Inc, New York.

Nielsen, J., Tahir, M., 2002. Homepage Usability: 50

Websites Deconstructed. New Riders Publishing,

Indianapolis.

Preece, J., Rogers, Y., Sharp, H., 2002. Interaction design:

beyond human-computer interaction. John Wiley &

Sons, New York.

Shackel, B., 1991. Usability – context, framework, design

and evaluation. In Shackel, B., Richardson, S. (eds.):

Human Factors for Informatics Usability, Cambridge

University Press, Cambridge, 21-38.

Shelat, B., Stewart, T., 2004. Transport Direct Portal.

Usability Testing Report. http://

www.transportdirect.gov.uk/research/pdf/mr61r1.pdf

Tatnall, A. (Ed.), 2005. Web portals: the new gateways to

Internet information and services. Idea Group

Publishing, Hershey.

Theng, Y.L., Soh, E.S., 2005. An Asian Study of

Healthcare Web Portals: Implications for Healthcare

Digital Libraries. Proceedings of the 8th international

conference on Asian digital libraries, ICADL.

Bangkok.

Tomić-Koludrović, I., Petrić, M., 2004. Identities on the

Net: Gender and National Stereotypes on Croatian

Broad-Reach Portals. Društvena istraživanja 13, 4-5.

Tractinsky, N., Katz, A., Ikar, D., 2000. What is beautiful

is usable. Interacting with Computers 13, 2.

Tsui, W.C., Paynter, J., 2004. Cultural Usability in the

Globalisation of News Portal. In Masoodian, M.,

Jones, S., and Rogers, B. (Eds.): Computer Human

Interaction, Proceedings of the 6

Asia Pacific

Conference, APCHI. Lecture Notes in Computer

Science 3101. Springer.

Tullis, T.S., Stetson, J.N., 2004. A Comparison of

Questionnaires for Assessing Website Usability.

Proceedings of UPA Conference. Minneapolis.

http://home.comcast.net/~tomtullis/publications/UPA2

004TullisStetson.pdf.

Uldall-Espersen, T., Frøkjær, E., Hornbæk, K., 2007.

Tracing Impact in a Usability Improvement.

Interacting with Computers. (in press)

Waloszek, G., 2001. Portal Usability – Is There Such A

Thing? SAP Design Guild, Edition 3. http://

www.sapdesignguild.org/editions/edition3/

overview_edition3.asp.

Wood, J., 2004. Usability Heuristics Explained, iQ

Content.http://www.iqcontent.com/publications/featur

es/article_32.

Wright, P., Monk, A., 1991. A cost-effective evaluation

method for use by designers. International Journal of

Man-Machine Studies 35, 6, 891-912.

WEBIST 2008 - International Conference on Web Information Systems and Technologies

168