Andrina Granić, Ivica Mitrović* and Nikola Marangunić
Faculty of Natural Sciences, Mathematics and Education, University of Split, Nikole Tesle 12, Split, Croatia
* Arts Academy, University of Split, Glagoljaska bb, Split, Croatia
Keywords: Usability evaluation, guideline inspection, evaluation form, broad-reach web portals.
Abstract: Overall research is aiming to design discount evaluation methodology for web portal assessment. A number
of problems was identified during testing user tasks in scenario-based usability testing while others were
detected through tasks mentally simulated by HCI experts using an inspection method. This paper reports on
the experience regarding the latter one. Designed inspection has proved very promising, although obtained
comprehensive quantitative and qualitative data revealed the need for some improvements. Revision of the
evaluation form along with subsequent assessment with adequate expert sample is needed.
Web portal, generally considered as a single point of
access to information, resources and services on a
wide range of topics (Waloszek, 2001), is typically
based on the more advanced technologies that go
beyond simple interface of the just information
based standard web page. Broad-reach portals, also
called "general" or "generic" portals, bring together
services such as search engines, e-mail, news,
forums, event guides, maps, on-line shopping, travel
information and the like. Accordingly, information
presented in every page can be delivered to a
number of users having different needs, motivations
and goals which portal design has to reflect.
Market research findings related to Croatian web
sphere undertaken in the last few years report that
broad-reach portals are the most visited web sites
(GFK Croatia, 2006; GemiusAudience, 2007). This
is the basic distinction from countries with high
level of Internet literacy, where there are much more
specialized web portals seen as gateways to the
variety of web information related to the specific
context (Tatnall, 2005).
When considering web portal usability, it should
be noted that current research on usability evaluation
is mostly concerned with focused, domain specific
portals, e.g. (Boye, 2006; Carstens and Patterson,
2005; Brantley et al., 2006). This is the result of
global trend of portal specialization when is quite
difficult to find any studies related to broad-reach
web portal assessments.
The objective of the overall study is the design
of "discount evaluation" approach to web portal
usability assessment. We conducted the experiment
in order to evaluate how easy and efficient the most
visited portals are. The study is placed in Croatian
web sphere where broad-reach web portals, as the
most visited sites, are much more popular than the
specialized ones. Such context seems appropriate for
this research, since it implies end users and
designers familiarity. We expected to find different
sets of problems because a wide variation in tasks
and applied diverse usability evaluation methods,
both empirical and analytic. A number of problems
was identified through testing user tasks in scenario-
based usability testing, which comprised a number
of test methods. Other problems were discovered
throughout tasks mentally simulated by experts from
the field of human-computer interaction (HCI) using
an inspection method.
Strong point in such approach is the chance to
supplement results from the guideline-based
inspection and the user-based one, enhanced with
feedback on users' pleasantness while working with
the portal. In this paper we report on and discuss the
results obtained through guideline-based inspection.
c I., Grani
c A. and Maranguni
c N. (2008).
In Proceedings of the Tenth International Conference on Enterprise Information Systems - HCI, pages 401-404
DOI: 10.5220/0001719904010404
We conducted a controlled experiment which
included less strict heuristic evaluation (Nielsen,
1994), i.e. guideline inspection, supplemented with
scenario-guided user evaluations involving task-
based end user testing, usability satisfaction
questionnaire and semi-structured interview, all used
to collect both quantitative data and qualitative
"remarks", cf. (Shackel, 1991). In the following we
describe the guideline inspection along with the
acquired results and discussion of findings.
Scenario-guided user evaluation along with the
achieved results is addressed in (Granić et al., 2008).
2.1 Experts, Instruments and Measures
Particular procedure was conducted with a group of
ten "instant" specialists from the HCI field. With the
intention of overcoming the problem of not having
enough usability experts who could be involved in
the web portal evaluation process, we had the
guideline inspection performed by "instant experts"
(Wright and Monk, 1991). Those were web design
practitioners who learnt principles of good user-
centered design and provided usability expert
assessment of the selected portals.
Evaluation form consisting of a set of
principles/guidelines augmented with auxiliary
guidelines, as additional explanations related to web
portals, was prepared. Individual expert’s marks and
comments concerning the assessed portals were
collected. The score for every portal was calculated
as an average mark on a seven-point Likert scale.
Additional observations concerning the inspection
procedure could be provided as well. Four broad-
reach web portals were included in the study: Index
portal (, Net portal (, Vip
portal ( and T-Portal (
Those portals were selected as the most visited
whilst also the earliest broad-reach web portals.
2.2 Assessment Procedure
Document containing detailed instructions and
prepared evaluation form was sent to chosen experts.
Aiming to discover possible problems in the
interface design, they had to mentally simulate the
tasks to be performed on portals, mark and comment
on in the evaluation form, following the instructions
and the provided guidelines along with the auxiliary
ones. Consequently, in order to supply all necessary
information, the evaluation form had to be very
detailed and self-explanatory. Nielsen’s usability
heuristics as a set of ten key principles (Nielsen,
1994) is explained and adjusted to portal usage.
Besides, as additional explanation to the guideline, a
series of auxiliary ones concerning portal design was
also provided, cf. (Wood, 2004; MIT, 2004).
Experts had (i) to specify a level of their
conformity with a principle/guideline and related set
of auxiliary guidelines on a seven-point Likert scale
and (ii) to provide a comment to justify assigned
mark since they were encouraged to offer additional
notes related to advantages and disadvantages of
assessed portals. Furthermore, observations and
remarks concerning the overall guideline assessment
procedure were more then welcome.
2.3 Results
Experimental results and findings acquired through
guideline inspection are addressed in what follows.
Arithmetic means of marks from a seven-point
Likert scale provided by ten "instant experts"
according to ten usability guidelines show that
highest mark was given to Vip portal (mean = 5.38),
followed by Net portal (mean = 4.85), T-Portal
portal (mean = 4.64) and Index portal (mean = 4.01).
Overall achieved results could be further related
to the key experts’ comments from the form. For
Vip portal experts emphasized well-adjusted,
consistent layout, simple navigation and feeling of
control. Specialists criticized portal’s slow respond,
errors while loading and awkward forum. Net portal
is characterized as coherent with good quality of
information structure, but with poor and old fashion
visual appearance. According to experts, its main
limitations are too long initial page, unclear names
of categories, shallow navigation structure and
absence of contact information. Identified problems
in T-Portal portal are related to employed diverse
types of navigation, too long initial page, absence of
contact information, help and FAQ. Lack of
consistency and aggressive "visual noise" were the
main reasons why Index portal got severely bad
marks. Recognized problems include ambiguous
home page, lack of consistency concerning names,
categories and navigational graphics, navigation
overload, absence of help and FAQ.
The evaluation form analysis included the
assessment of adapted guidelines themselves and
judgment of the quality of experts' evaluation.
Qualitative analysis criteria were expressed in terms
of the mark span along with the significance and
usefulness of comments (see columns Mark Span
and Info in Table 1). The guidelines were examined
through expert's comments and observations,
ICEIS 2008 - International Conference on Enterprise Information Systems
assigning low (L), medium (M) and high (H) values
according to quantity and level of details of provided
comments (Info column). The range of marks
expresses the lowest and highest ones (Mark span
column). Quality of experts' evaluation was assessed
too. It comprised inspection of specialist's answers
to the guideline compliance related to the four
assessed portals. Similar information quality criteria
were used while analyzing experts' work: number,
percentage and quality of provided comments as
well as number of additional observations.
In the following the comprehensive analysis of data
obtained through ten evaluation forms is presented.
3.1 Guidelines Issues
The quality of information achieved through
individual expert’s comments was categorized as
low, medium and high. E.g. remarks like "there is no
mistake" or "not good at all" represent comments of
low information quality. Conversely, detailed ones
which list specific observations related to page
layout, fonts, navigation/links and graphics, are
classified in medium and high information quality
Analyses of the marks and comments indicate
that some provide extremely poor information.
Those guidelines have wide mark span, which could
imply limitations in their understanding and/or their
vague formulation. Besides, those guidelines also
offer minimal amount of comments because they do
not stir/provoke experts in sufficient portal analysis.
Moreover, any additional information regarding
problems and possible solutions in portal design is
not offered. Examples of such guidelines are no. 1,
no. 5 and no. 9. Conversely, narrow mark span for
guidelines no. 2, no. 3 and no. 7 points toward their
clarity and unambiguity, also indicating high degree
of experts’ accordance in their valuation. Moreover,
acquired comments provided new and detailed
information about particular design problems. Our
results suggest that a good guideline is the one (i)
characterized with a narrow mark span and (ii)
"provoking" good quality comments and criticism
which detects interface design problems offering
solutions at the same time. We assume that obtained
quite low span of some guidelines implies their good
quality, despite the non-uniform group of specialists.
Table 1: Guideline analysis.
Info Mark
Info Mark
Info Mark
1. Portal is actively informing user about its'
processes (information about what is going on
is always present).
4 – 6 L 2 – 6 L 3 – 6 L 2 – 6 L
2. Concept of portal is well adjusted to the user
5 – 6 M 5 – 7 M 5 – 6 L 2 – 6 M
3. While working with portal users have feeling
of control, safety and navigation freedom.
3 – 6 H 4 – 7 M 2 – 6 H 2 – 6 H
4. Portal respects media standards and usual
1 – 4 H 5 – 7 M 4 – 6 M 2 – 7 M
5. Portal prevents possible user errors. 2 – 6 L 3 – 7 L 3 – 7 L 2 – 7 L
6. User is intuitively getting information on
portal, i.e. user doesn't have to remember
information path but recognize it.
4 – 5 M 4 – 7 L 3 – 6 M 2 – 6 L
7. Portal is adjusted for efficient use by novice
users as well as by experts.
3 – 5 H 4 – 6 M 3 – 6 H 2 – 6 M
8. Portals' design is clear, understandable and
transparent, i.e. most needed information are
at the same time most visible.
2 – 5 M 5 – 7 M 4 – 6 M 2 – 6 L
9. Portal enables user recognize and help
recover from errors.
2 – 6 L 3 – 7 L 2 – 7 L 3 – 7 L
10. Portal offers help while working on it. 2 – 5 M 3 – 6 M 3 – 6 M 2 – 7 M
3.2 Instant Expert Issues
Further analysis of evaluation forms provides insight
into instant experts sample selection. Apart from
marks, three specialists did not provide any
supplementary comment or observation concerning
the overall assessment procedure. Two experts
supported their marks with very poor explanations
and quality of acquired information didn’t help to
get insight into design strengths and weaknesses.
On the other hand, five experts provided
evaluation forms with marks and detailed comments
of every guideline for each portal. Additionally they
offered remarks and suggestions for an improvement
of the evaluation process in general and individual
guidelines in particular. Such considerable
distinction in acquired information quantity and
quality suggests quite heterogeneous group of
experts in their HCI knowledge and usability
expertise. This problem was very difficult to avoid
due to limited number of HCI experts in Croatia on
the one hand and high costs of foreign experts’
engagement on the other.
The objective of the research is the design of
"discount evaluation approach" to web portal
assessment. Aiming to achieve this goal, design of
most visited Croatian broad-reach portals was
assessed both through usability inspection method
and a number of usability test methods.
We found that designed evaluation form used in
guideline inspection needs significant revision.
Some of adapted Nielsen’s principles showed poor
applicability in web portal context. The derived
guidelines did not provide any information which is
useful for improving portals’ usability. For that
reason, a number of guidelines should be more
comprehensible, auxiliary guidelines revised and
redundant ones excluded. A new set of guidelines is
needed, the one which is not so strictly based on
Nielsen’s heuristics.
Significant difference in acquired experts’
information suggests the non-homogeneity of
"instant experts" group concerning their HCI
expertise. Such problem was hard to prevent due to
inadequate number of resident HCI experts as well
as high costs of possible foreign specialists’
The guideline-based evaluation, even though
showing respectable potential, raised a couple of
concerns. In order to upgrade the applied usability
inspection method (i) the "instant experts" selection
and the evaluation form issues should be revised
according to the quantitative and qualitative analysis
of the obtained results and (ii) the redesigned
method should be applied in an assessment of more
specialized Croatian web portals.
This paper describes the results of research being
carried out within the project 177-0361994-1998
Usability and Adaptivity of Interfaces for Intelligent
Authoring Shells funded by MZOŠ of the Republic
of Croatia. The experiment was conducted at Dept.
of Visual Communication Design, Arts Academy.
Boye, J., 2006. Improving portal usability. CMS Watch.
Brantley, S., Armstrong, A., Lewis, K.M., 2006. Usability
Testing of a Customizable Library Web Portal.
College & Research Libraries 67, 2.
Carstens, D.S., Patterson, P., 2005. Usability Study of
travel websites. Journal of Usability Studies, 1, 1.
GemiusAudience, 2007.
GFK Croatia, 2006. ttp://
Granić, A., Mitrović, I., Marangunić, N., 2008. Experience
with Usability Testing of Web Portals. 4
Int. Conf.
on Web Information Systems and Technologies. 4-7
May, 2008, Madeira, Portugal (accepted for
MIT Usability Guidelines, 2004.
Nielsen, J., 1994. Heuristic Evaluation. In Nielsen, J.,
Mack, R.L. (Eds.): Usability Inspection Methods. John
Wiley and Sons Inc, New York.
Shackel, B., 1991. Usability – context, framework, design
and evaluation. In Shackel, B., Richardson, S. (eds.):
Human Factors for Informatics Usability, Cambridge
University Press, Cambridge, 21-38.
Tatnall, A. (Ed.), 2005. Web portals: the new gateways to
Internet information and services. Idea Group
Publishing, Hershey.
Waloszek, G., 2001. Portal Usability – Is There Such A
Thing?. SAP Design Guild, Edition 3.
Wood, J., 2004. Usability Heuristics Explained, iQ
Wright, P., Monk, A., 1991. A cost-effective evaluation
method for use by designers. International Journal of
Man-Machine Studies 35, 6, 891-912.
ICEIS 2008 - International Conference on Enterprise Information Systems