Andrina Granić
, Ivica Mitrović
and Nikola Marangunić
Faculty of Natural Sciences, Mathematics and Education, University of Split, Nikole Tesle 12, Split, Croatia
Arts Academy, University of Split, Glagoljaska bb, Split, Croatia
Keywords: End user testing, guideline inspection, methodology for web portal usability evaluation,
broad-reach web portals.
Abstract: The paper reports on the experience regarding usability evaluation of broad-reach web portals. The designed
methodology advocates a number of usability test methods along with specialists' inspection and provides a
comprehensive quantitative and qualitative assessment feedback. Our experience accords with the claim that
we should not rely on isolated evaluations, but instead on combine assessment methods. In this paper we
present and discuss mainly the results obtained through user-based usability evaluation. The results indicate
that chosen research instruments, measures and methods for usability testing were consistent. Conversely,
the results of the designed guideline inspection did not conform to the ones obtained through the end user
testing. Although proved promising, some of the guideline-based evaluation aspects need improvement.
Usability is the key issue in human-computer
interaction (HCI) and is related to ease of use and to
ease of learning. It becomes defined within a
relationship among task, user and system purpose.
For this reason there is no simple definition, or
meaningful single measure of usability.
When considering usability of a web portal, a
site that functions as a point of access to information
on the Web, it should be noted that current research
on usability evaluation is mostly concerned with
focused, domain specific portals. This is the result of
the global trend of portal specialization. In such
context it is difficult to find any studies related to
broad-reach web portal assessments.
This paper reports on the experience regarding
the design of a methodology for usability evaluation
of broad-reach web portals. The main motivation
for our research came from reports stating that the
most visited Croatian web sites are broad-reach web
portals, implying end users and designers' familiarity
with such kind of sites.
We conducted the experiment in order to
evaluate how easy and efficient those broad-reach
portals are. Portal usability is viewed as efficacy in
use, considered primarily as involving measures of
user performance and ease of use, envisaged as
involving subjective judgments. Additionally it is
augmented with a significant feedback acquired
through guideline inspection, a less formal heuristic
A strong point in such an approach is the chance
to supplement results from the guideline-based
assessment with the user-based one, enhanced with a
feedback on the users' pleasure while working with
the portal. The results go in hand with the assertion
that we should not rely on isolated evaluations.
Instead, usability assessment methods should be
combined, hence obtaining different kinds of
usability improvement suggestions.
In this paper we discuss and present interpretations
of the results achieved through employed user-based
usability methods. The complementing test methods
proved to be consistent. Conversely, results of the
designed guideline-based evaluation were not in
agreement with the ones obtained from the usability
testing. Consequently, our method for guideline
inspection raised a couple of concerns which will be
addressed in our future work.
The paper has the following organization.
Section 2 briefly introduces some background of the
research, addressing web portals and web usability,
providing the rationale for the methodology. An
approach to usability evaluation along with achieved
results is described in what follows. Section 4 brings
the discussion and interpretation of findings,
c A., Mitrovi
c I. and Maranguni
c N. (2008).
In Proceedings of the Fourth International Conference on Web Information Systems and Technologies, pages 161-168
DOI: 10.5220/0001520801610168
additionally offering some directions for future
research. Section 5 closes the paper.
Web portal, generally considered as a single point of
access to information, resources and services on a
wide range of topics (Waloszek, 2001), is typically
based on the more advanced technologies that go
beyond simple interface of the just information
based standard web page. Broad-reach portals, also
called "general" or "generic" portals, bring together
services such as search engines, e-mail, news,
forums, event guides, maps, on-line shopping, travel
information and the like. Accordingly, information
presented in every page can be delivered to a
number of users having different needs, motivations
and goals which the portal design has to reflect.
Market research findings related to the Croatian
web sphere undertaken in the last few years report
that broad-reach portals are the most visited web
sites (GFK Croatia, 2006; GemiusAudience, 2007).
This is the basic distinction from the countries with
high level of Internet literacy, where there are much
more specialized web portals seen as gateways to
the variety of web information related to the specific
context (Tatnall, 2005). In terms of the choice of
content and layout, major Croatian broad-reach
portals have the semblance of the web sites of
broadsheet newspapers or public service
broadcasters (Tomić-Koludrović and Petrić, 2004).
On the other hand, an improvement of usability
as a quality of use in context (Bevan and Macleod,
1994) is nowadays perhaps the most important goal
of current research in the field of human-computer
interaction. Most used assessment methods are
grouped into two categories, e.g. (Nielsen, 1993;
Lewis, 2005):
usability test methods; user-based which involve
end users, hence including user testing, focus
groups, interviews, questionnaires and surveys;
usability inspection methods, without end users
consisting of heuristic evaluations and cognitive
walkthroughs as the most often used ones.
Research related to web usability recently has
had a tendency to bring together those two basic
approaches, cf. (Hornbæk, 2006). Concerning web
portal usability, it should be noted that the current
studies are mostly concerned with domain specific
portals. Namely, in the context of the global trend of
web portal specialization, recent research related to
usability evaluation is mostly concerned with the
focused portals (somewhere called "vertical" or
"domain specific" ones) such as enterprise or
corporate portals (Boye, 2006), travel portals (Shelat
and Stewart, 2004; Carstens and Patterson, 2005),
news portals (Tsui and Paynter, 2004), library web
portals (Brantley et al., 2006), tourist portals
(Klausegger, 2006), healthcare web portals (Theng
and Soh, 2005) and similar.
Apparently, while there is a number of studies
related to usability evaluation of specialized web
portals, some additionally taking into account the
particular cultural context e.g. (Theng and Soh,
2005; Tsui and Paynter, 2004; Fang and Rau, 2003),
there is hardly any research dealing with broad-
reach portal assessment.
Taking into consideration outcomes of the
undertaken research concerning web portals and
usually employed usability assessment methods, the
aim of our study is related to the design of
appropriate methodology for broad-reach portal
usability evaluation. The study is placed in Croatian
web sphere where broad-reach web portals are much
more popular and accepted than specialized ones.
Such context seems appropriate for this research,
since it implies end users and designers familiarity
with such kind of portals.
In order to achieve the key objective, the design of
the most visited Croatian broad-reach portals was
evaluated. We conducted a controlled experiment
which advocates scenario-guided user evaluations
involving a number of usability testing methods used
to collect both quantitative data and qualitative
"remarks", cf. (Shackel, 1991). Furthermore, user-
based testing is supplemented with less strict
heuristic evaluation (Nielsen, 1994), i.e., guideline
inspection. This follows the literature that suggests
that usability inspection needs to be combined with
usability test methods, e.g. (Holzinger, 2005;
Hornbæk, 2006; Uldall-Espersen et al., 2007).
Such approach, understood to be a combination
of behaviour and opinion based measures obtained
under experimental control which is additionally
augmented with expert assessment, is described in
the following.
3.1 Methodology
In the following sections we describe the
experimental methodology adopted to assess the
effectiveness, efficiency and satisfaction in addition
WEBIST 2008 - International Conference on Web Information Systems and Technologies
to expert evaluation of selected Croatian broad-reach
portals and the results obtained.
3.1.1 Participants
The study involved thirty participants with basic
computer literacy. According to their practical
experience in web design, they were classified in
two different groups composed of fifteen
participants. The "practitioner" group was composed
of three independent sub-groups of randomly chosen
participants including computer science experts,
marketing experts knowledgeable in Internet issues
as well as students of web design. On the other hand,
the "non-practitioner" group was consisted of three
independent sub-groups of randomly chosen
participants consisting of young participants, middle
aged and elderly ones.
Furthermore, a particular group of ten "instant
experts" (Wright and Monk, 1991) for guideline-
based evaluation was formed. Those were web
design practitioners who are familiar with the
principles of the good user-centred designs in
addition to the evaluation approaches and who
provided usability expert assessment of the selected
web portals.
3.1.2 Instruments, Measures and Portals
User assessment was conducted individually, with
Internet access and a screen capturing software for
tracing the users' actions and navigation. We
measured task time and achievement. End user
testing was based on criteria expressed in terms of
few measures (ISO/IEC, 2006):
objective performance measurement of
effectiveness (percent task completion) in using
the portal,
objective performance measurement of efficiency
(time on task) in using the portal and
users' subjective assessment of the web portal
System Usability Scale (SUS), as a simple standard,
ten-item attitude questionnaire with five-point
Likert scale (Brook, 1996), was used for the
subjective valuation. As an additional subjective
feedback, answers to the semi-structured interview
were collected.
In order to perform a less formal heuristic
evaluation, specifically guideline inspection, an
evaluation form consisting of a set of adapted
principles augmented with portal-related auxiliary
guidelines was prepared. Individual expert’s marks
and comments were collected. The score for every
portal was calculated as an average mark on a seven-
point Likert scale.
We included four broad-reach web portals in
our experiment: Index portal (www.index.hr), Net
portal (www.net.hr), Vip portal (www.vip.hr) and T-
Portal portal (www.tportal.hr). As already stated
before, those portals were selected as the most
visited whilst also the earliest broad-reach web
portals. For an insight to a broad-reach portal user
interface, a screenshot of the Vip portal is illustrated
in Figure 1.
Figure 1: Screenshot of Vip broad-reach web portal.
3.1.3 Experiment Design and Procedure
In order to understand the effect of web portal
design in a sample work situation, we elaborated a
work scenario, a sequence of typical tasks and user
actions. To test assigned tasks and time interval,
clarity and unambiguousness of measuring
instruments for subjective assessment and adequacy
of hardware and software support, pilot testing was
We chose three typical tasks which structure and
location on the portals was not changed over time.
The tasks covered different topics, offering to
diverse groups of involved participants' a similar
opportunity for finding task-related information. For
each selected portal, undertaken tasks were the same
and the probability of their completion was similar
as well. See Figure 2 to get insight into the
performed tasks.
The evaluation procedure was carried out
individually with each test user, using a personal
computer with Internet access in addition to a
software and hardware support for tracing and
recording results of task completion. Within each
evaluation session all the portals were assessed,
where the order of their evaluation was randomly
selected. The allocated session's average time per
every participant was 45 minutes.
Figure 2: Undertaken tasks.
An evaluation procedure consisted of the following
task-based end user testing,
usability satisfaction questionnaire,
semi-structured interview and
guideline inspection.
Task-based end user testing involved a scenario-
guided user assessment with three tasks selected to
show the portal basic functionality. It enabled us to
determine user efficiency and effectiveness while
working with the web portal. A user's objective
accomplishment measure, labelled as fulfilment, was
calculated as an average time spent on all allocated
tasks weighted with successfulness of task
completion. For each user, the time limit for all
assigned tasks was 15 minutes per portal.
An usability satisfaction questionnaire supported the
assessment of the users' subjective satisfaction with
diverse interaction aspects. We used SUS
questionnaire, as it is argued that it yields the most
reliable results across sample sizes (Tullis and
Stetson, 2004). Its questions addressed different
aspects of the user's reaction to the portal as a whole,
providing an indication of a level of statement
agreement on a five-point Likert scale. Furthermore,
the feedback from the questionnaire was augmented
with the users' answers in a semi-structured
interview. In this interview we additionally asked the
participants to rate and comment on the portal’s
visual attractiveness as well.
End user testing of web portal usability
additionally brought together with guideline-based
evaluation provided a more precise broad-reach
portal usability assessment. In order to overcome the
problem of not having enough usability experts who
could be involved in the portal evaluation process,
we had the guideline inspection performed by
"instant experts", web design practitioners familiar
with the HCI principles. A detailed evaluation form
with Nielsen's usability heuristics (Nielsen, 1994), as
a set of ten key principles, was adapted to portal
context and augmented with a series of auxiliary
guidelines, as additional explanations of web portals
design cf. (Granić et al., 2004; Wood, 2004; MIT,
2004; Preece et al., 2002; Nielsen and Tahir, 2002).
In the provided evaluation form experts had (i) to
specify a level of their agreement with the
principle/guideline and related set of auxiliary
guidelines on a seven-point Likert scale as well as
(ii) to provide comments in order to justify the
assigned mark. Furthermore, remarks and
observations concerning the overall guideline-based
assessment procedure were more then welcome.
The guideline-based evaluation coupled with the
task-based end user testing, the attitude
questionnaire and the interview, provided a
sufficient quantitative and qualitative assessment
feedback. In the following we present experimental
results and findings, considering in details the ones
obtained by means of usability testing. Namely, due
to paper length limitations, the results of guideline
inspection will be addressed in our further report.
3.2 Results
Results acquired through the usability test methods
in addition to the main findings obtained in
guideline-based inspection are addressed in what
3.2.1 Results of Usability Test Methods
Descriptive statistics of the objective
accomplishment measure fulfilment, including
arithmetic means, standard deviations and
significance levels of Kolmogorov-Smirnov
coefficient for normality of distribution is presented
in Table 1. Result of the distribution for measure
fulfilment on the T-Portal web portal differs
significantly from normal distribution (K-S = 0.008).
Accordingly, Friedman's test as a non-parametrical
procedure was performed. Statistically significant
value of chi square (χ
= 49.4, df = 3, p < 0.01)
indicates the existence of differences in the objective
accomplishment measure among portals (see Table
Imagine that you are interested in some information
related to diverse aspects of everyday living. In order to
find relevant data, you are kindly asked to refer to the
respective site of Index web portal (www.index.hr) and
complete three tasks.
The tasks are the following:
1. find information related to the traffic situation on
A1 highway in the direction from Zagreb to Split as well
as the daily temperature in Split
2. search for the today’s first show on RTL
television which is scheduled after 9 PM
3. find and print one recipe for the main meal
After your complete the allocated tasks, you are
kindly asked to proceed with the evaluation session
WEBIST 2008 - International Conference on Web Information Systems and Technologies
Table 1: Results of objective accomplishment measure fulfilment for the four selected portals (note that lower M score
means better result).
fulfilment M SD K-S M Rank df χ
Index portal 59.77 38.726 0.292 1.57 3 49.4 <0.01
Net portal 108.40 46.300 0.720 2.93
Vip portal 62.13 17.211 0.656 1.87
T-Portal portal 171.64 168.143 0.008 3.63
Table 2: Results of subjective satisfaction measure SUS for the four selected portals (note that higher M score means better
Index portal 75.33 18.820 0.819 29 746.94 <0.01
Net portal 56.00 25.194 0.902
Vip portal 77.83 15.821 0.319
T-Portal portal 51.75 23.836 0.961
Descriptive statistics of results acquired for
subjective satisfaction measure SUS for each and
every participant on every web portal is shown in
Table 2. No statistical difference in the distribution
of the results from the expected normal distribution
was found (K-S
> 0.05).
In order to test the difference among portals, the
analysis of variance as a parametric procedure was
applied. Significant F-ratio (F = 746.94, df = 29,
p < 0.01) indicates the existence of differences
among the portals in the results related to the
obtained subjective measure (see Table 2).
Additionally, we also considered all accomplished
experimental results related to the two groups of
participants – the practitioner group and the non-
practitioner one. The differences in the user's
objective accomplishment and subjective
satisfaction usability measures between the two
groups were tested with t-tests for small independent
Statistically significant difference between the
groups was found for results of the fulfilment
measure (t = 2.95, p < 0.01). The group of
practitioners showed better results on mean values
(mean = 308.4, SD = 57.217) than the non-
practitioners group (mean = 495.46, SD = 238.479).
The mean fulfilment results are shown in Table 3.
Table 3: Results of objective accomplishment measure
fulfilment for the two groups of participants (note that
lower score means better result).
fulfilment practitioners non-practitioners
N 15 15
M 308,40 495,46
SD 57,217 238,479
t 2,95
df 28
p <0,01
On the other hand, a statistically significant
difference was not found for the results related to the
subjective satisfaction measure SUS (t = 1.95,
p = 0.062) between the practitioner group of
participants (mean = 243.17, SD = 51.317) and the
non-practitioner one (mean = 278.67, SD = 48.531).
Pearson's correlation coefficients for the participants'
results in the achieved usability objective and
subjective measures are shown in Table 4.
Table 4: Pearson's correlation coefficients of overall
results (* significant correlations at level of p<0.05).
Correlations of overall results r
SUSfulfilment 0.14
SUSvisual attractiveness 0.41*
A statistically significant correlation coefficient was
not found between overall SUS and overall
fulfilment. Conversely, significant correlation was
found between overall SUS and score for overall
visual attractiveness.
3.2.2 Results of Usability Inspection Method
The arithmetic means of the marks from a seven-
point Likert scale of the guideline evaluation forms
for assessed portals show the following results. The
highest mark was given to Vip web portal
(mean = 5.38), followed by Net portal
(mean = 4.85), T-Portal portal (mean = 4.64) and
Index portal (mean = 4.01).
Furthermore, we considered the outcomes
achieved in the guideline-based evaluation in
addition to the ones obtained throughout the
usability testing. The ranking of the selected broad-
reach web portals was compared.
Considering the objective accomplishment
measure, Index portal and Vip portal scored the best
result, followed by Net portal and T-Portal web
portal. The subjective satisfaction measure provided
similar rankings too. On the other hand, in the
guideline-based evaluation Index portal achieved the
lowest rank, while Vip portal remained on a high-
level, followed by Net portal and T-Portal broad-
reach web portal.
4.1 Interpretation of the Results
The results of the task-based end user testing
showed statistically significant differences among
the assessed portals according to the measure of
user's objective accomplishment. This suggests that
web portals could be ranked by mean values. The
results of the subjective satisfaction measure also
showed differences among evaluated portals and
their ranking by mean values.
It is important to point out that the measures of
user's objective accomplishment and her/his
subjective satisfaction were not significantly
correlated. Our results are in accordance with the
ones of the meta-analytic research report on
correlations among usability measures calculated
from the raw data of 73 studies (Hornbæk and Law,
2007). This yet again implies the appropriateness of
the applied usability test methods for portal
evaluation, since it confirms independence of the
objective and subjective measurements.
The overall achieved results could be further
related to the most frequent statements from the
interviews. Namely, the participants felt especially
pleased and comfortable working with the portals
where their objective achievement was high. The
end users considered them as broad-reach web sites
with good quality of information structure,
respectable layout and straightforward navigation.
Correlation between overall SUS results and
overall visual attractiveness indicates that a pleasant
appearance influences the subjective perception of
portal usability. The interview statements also
support this finding. Namely, it happened that the
users usually emphasized the portals' visual
attractiveness, assigning high subjective ratings.
Such assumption is in line with related studies
which, along with the related HCI issues, also
address aesthetics aspects of design, cf. (Tractinsky
et al., 2000; Hassenzahl, 2004).
The results of the objective accomplishment
measure revealed expected differences between non-
practitioners and practitioners, the latter being faster
and more successful in the tasks' achievement. This
indicates that the selected tasks and the objective
accomplishment measure were consistent. On the
other hand, the measure of subjective satisfaction
did not show any statistically significant difference
between these two groups. Such finding indicates
that the questionnaire itself and its translation to
Croatian language could be considered as an
appropriate instrument for user subjective
Moreover, our experience suggests that the
choice of the sample size in addition to the structure
of engaged end users ("practitioner" and "non-
practitioner" groups of participants) is also in line
with the outcomes of related studies. Specifically, in
the Hornbæk and Law's (2007) meta-analysis of
usability measures, the average number of
participants involved per study was 32 (SD = 29,
ranging from 6 to 181).
Conversely, the achieved result of guideline-
based assessment did not conform to the ones
obtained throughout applied usability testing. The
highest ranked web portal in the end user testing
scored as the lowest one in the "instant expert"
usability evaluation. There are two possible reasons
for such an outcome the designed evaluation form
and/or the selection of usability specialists involved
in the web portal guideline-based assessment.
Concerning the first reason, some of adapted
Nielsen’s principles showed poor applicability in the
web portal context, not providing any useful
information for portal usability improvement.
Moreover, a number of guidelines should be more
comprehensible and auxiliary guidelines revised and
redundant ones excluded. Accordingly, a new set of
guidelines is needed, the one which is not so strictly
based on Nielsen’s heuristics.
Regarding the choice of usability "instant
experts", significant difference in acquired
WEBIST 2008 - International Conference on Web Information Systems and Technologies
information suggests non-homogeneity of the group
concerning their HCI expertise. Such problem was
hard to prevent due to inadequate number of resident
HCI specialists as well as high costs of possible
foreign experts' engagement.
Although our guideline inspection has proved
very promising, some of its aspects need
improvement. Conducted assessment obtained
comprehensive quantitative and qualitative data
which analysis defined the first steps of our future
work: (i) revision of the designed evaluation form
and (ii) subsequent assessment with adequate sample
of experts.
4.2 Future Steps
The results acquired throughout the applied
methodology show that the designed guideline
inspection, as a less formal heuristic evaluation,
although proved very promising needs to be
improved. This is in line with recent research on
heuristic evaluation which is focused on improving
its effectiveness and efficiency with respect to user
testing, cf. (Hvannberg et al., 2007). Consequently,
our future work will be focused on quantitative and
qualitative analysis of the results obtained through
guideline-based assessment and will first and
foremost address two important aspects: the
evaluation form issues and the instant experts
Furthermore, future work will subsequently go
into two directions also taking into account the
following issues:
first, in order to upgrade the applied usability
inspection method and develop a methodology
for web portal usability evaluation (i) the instant
experts selection and the evaluation form issues
should be revised according to the quantitative
and qualitative analysis of the obtained results
and (ii) the redesigned methods should be
applied for the assessment of more specialized
Croatian web portals
second, in order to improve the applicability of
the developed methodology to practice and to
achieve its broad generalization, an inclusion of a
cross-cultural sample should be considered.
Design of most visited Croatian broad-reach web
portals was assessed both through a number of
usability test methods with diverse groups of end
users on the one hand and the usability inspection
method on the other. The designed evaluation
methodology, as an approach which is based on (i)
scenario-guided user evaluations used to collect both
quantitative data and qualitative "remarks" and (ii)
specialists' assessment, provided a comprehensive
valuation feedback.
The experience reported on in this paper
indicates that the chosen research instruments,
measures and methods for user-based usability
evaluation were consistent. Conversely, the results
of the designed usability inspection method were not
in agreement with the ones obtained from the user
test methods. Consequently, the guideline-based
evaluation, even though showing respectable
potential, raised a couple of concerns which will be
addressed in our future work.
This paper describes the results of research being
carried out within the project 177-0361994-1998
Usability and Adaptivity of Interfaces for Intelligent
Authoring Shells funded by the Ministry of Science,
Education and Sports of the Republic of Croatia.
Experiment was conducted at Department of
Visual Communication Design, Arts Academy,
University of Split.
Bevan, N., Macleod, M., 1994. Usability measurement in
context. Behaviour & Information Technology, 13, pp.
Boye, J., 2006. Improving portal usability. CMS Watch.
Brantley, S., Armstrong, A., Lewis, K.M., 2006. Usability
Testing of a Customizable Library Web Portal.
College & Research Libraries 67, 2.
Brooke, J., 1996. SUS: a "quick and dirty" usability scale.
In Jordan, P.W., Thomas, B., Weerdmeester, B.A,
McClelland, A.L. (Eds.): Usability Evaluation in
Industry. Taylor and Francis, London.
Carstens, D.S., Patterson, P., 2005. Usability Study of
travel websites. Journal of Usability Studies, 1, 1.
Fang, X., Rau, P.L.P., 2003. Culture differences in design
of portal sites. Ergonomics 46, 1-3.GemiusAudience,
2007. http://www.cati.hr/
GFK Croatia, 2006. http://www.gfk.hr/press1/
Granić, A., Glavinić, V., Stankov, S., 2004. Usability
Evaluation Methodology for Web-based Educational
Systems. In Stary, C., Stephanidis, C. (Eds.):
Proceedings of the 8
ERCIM Workshop "User
Interfaces for All". ERCIM, Heraklion (Crete),
Greece, 28.1-28.15.
Hassenzahl, M., 2004. The interplay of beauty, goodness,
and usability in interactive products. Human-
Computer interaction 19, 4.
Holzinger, A., 2005. Usability Engineering methods for
Software developers. Communications of the ACM 48,
1, 71-74.
Hornbæk, K., 2006. Current practice in measuring
usability: Challenges to usability studies and research.
International Journal of Man-Machine Studies 64, 2.
Hornbæk, K., Law, E.L-C., 2007. Meta-Analysis of
Correlations Among Usability Measures. CHI 2007
Proceedings, April 28–May 3, San Jose, California,
Hvannberg, E., Law, E.L-C., Larusdottir, M., 2007.
Heuristic evaluation: Comparing ways of finding and
reporting usability problems. Interacting with
Computers 19, 225–240.
ISO/IEC 25062:2006, 2006. Software engineering –
Software product Quality Requirements and
Evaluation (SQuaRE) – Common Industry Format
(CIF) for usability test reports.
Klausegger, C., 2006. Evaluating Internet Portals - An
Empirical Study of Acceptance Measurement Based
on the Austrian National Tourist Office's Service
Portal. Journal of Quality Assurance in Hospitality &
Tourism 6, 3-4.
Lewis, J. R., 2005. Introduction to Usability Testing.
Tutorial given at HCI International 2005. July 22-27.
Las Vegas, Nevada.MIT Usability Guidelines, 2004.
Nielsen, J., 1993. Usability Engineering. Academic Press
Limited. Boston.
Nielsen, J., 1994. Heuristic Evaluation. In Nielsen, J.,
Mack, R.L. (Eds.): Usability Inspection Methods. John
Wiley and Sons Inc, New York.
Nielsen, J., Tahir, M., 2002. Homepage Usability: 50
Websites Deconstructed. New Riders Publishing,
Preece, J., Rogers, Y., Sharp, H., 2002. Interaction design:
beyond human-computer interaction. John Wiley &
Sons, New York.
Shackel, B., 1991. Usability – context, framework, design
and evaluation. In Shackel, B., Richardson, S. (eds.):
Human Factors for Informatics Usability, Cambridge
University Press, Cambridge, 21-38.
Shelat, B., Stewart, T., 2004. Transport Direct Portal.
Usability Testing Report. http://
Tatnall, A. (Ed.), 2005. Web portals: the new gateways to
Internet information and services. Idea Group
Publishing, Hershey.
Theng, Y.L., Soh, E.S., 2005. An Asian Study of
Healthcare Web Portals: Implications for Healthcare
Digital Libraries. Proceedings of the 8th international
conference on Asian digital libraries, ICADL.
Tomić-Koludrović, I., Petrić, M., 2004. Identities on the
Net: Gender and National Stereotypes on Croatian
Broad-Reach Portals. Društvena istraživanja 13, 4-5.
Tractinsky, N., Katz, A., Ikar, D., 2000. What is beautiful
is usable. Interacting with Computers 13, 2.
Tsui, W.C., Paynter, J., 2004. Cultural Usability in the
Globalisation of News Portal. In Masoodian, M.,
Jones, S., and Rogers, B. (Eds.): Computer Human
Interaction, Proceedings of the 6
Asia Pacific
Conference, APCHI. Lecture Notes in Computer
Science 3101. Springer.
Tullis, T.S., Stetson, J.N., 2004. A Comparison of
Questionnaires for Assessing Website Usability.
Proceedings of UPA Conference. Minneapolis.
Uldall-Espersen, T., Frøkjær, E., Hornbæk, K., 2007.
Tracing Impact in a Usability Improvement.
Interacting with Computers. (in press)
Waloszek, G., 2001. Portal Usability – Is There Such A
Thing? SAP Design Guild, Edition 3. http://
Wood, J., 2004. Usability Heuristics Explained, iQ
Wright, P., Monk, A., 1991. A cost-effective evaluation
method for use by designers. International Journal of
Man-Machine Studies 35, 6, 891-912.
WEBIST 2008 - International Conference on Web Information Systems and Technologies