Usability Evaluation of a Collaborative Health Information System
Lessons from a User-centred Design Process
Berglind Smaradottir
1
, Santiago Martinez
2
, Elisabeth Holen-Rabbersvik
3
,
Torunn Vatnøy
3
and Rune Fensli
1
1
Department of ICT , University of Agder, Jon Lilletuns vei 9, N-4879 Grimstad, Norway
2
Department of Psychosocial Health, University of Agder, Grimstad, Norway
3
Department of Health and Nursing Sciences, University of Agder, Grimstad, Norway
Keywords: Health Information System, User-centred Design, Usability, Inter-municipal Cooperation, Dementia
Assessment, Human-computer Interaction.
Abstract: In Norway, a recent health reform urged municipalities to implement new primary health care services for
their citizens. In order to optimise resources, municipalities have established inter-municipal coordination
(IMC) to collaborate across organisational borders. Information systems become a necessary tool to support
collaboration and shared access to information in an IMC. In this context, the research project eHealth-
extended Care Coordination identified a specific need for a collaborative information system for the process
of evaluation and assessment of dementia in IMC teams. This paper presents the usability evaluation of a
collaborative information system for dementia assessment built using a user-centred design approach.
Mixed methods such as observations, semi-structured interviews and a questionnaire were used for data
collection. The results showed that the new information system supported the collaborative work of the
inter-municipal dementia team with a sufficient level of satisfaction among the end-users. The prototyped
solution established the foundations for the system implemented in the Norwegian trials of the FP7 EU
project United4Health, dedicated to Point-of-Care Services.
1 INTRODUCTION
In Norway, the Coordination reform (Norwegian
Ministry of Health and Care Services, 2008-2009)
addressed the continuity of care in national health
and care services. Services that traditionally were
carried out by specialised health care were
transferred to primary health care provided by
municipalities. Small and medium size Norwegian
municipalities faced the challenge of providing
specialised services to their citizens, accomplishing
the need for structural, organisational and
technological changes. This brought to light the need
for an effective coordination and collaboration
across organisational borders.
In this context, the research project eHealth-
extended Care Coordination (Samhandling uten
grenser) 2011-2015, focused on information flow in
inter-municipal cooperation (IMC) health care
teams. In the first phase of the project, a field study
identified the need for a collaborative information
system (CIS) to improve the information flow in
IMC health care teams. In the second phase, an IMC
dementia team participated in a user-centred design
(UCD) process entailing user workshops, laboratory
evaluations and interviews for developing a
functional prototype for a CIS for dementia
assessment (Smaradottir et al., 2015b). In the third
phase, a usability evaluation of electronic dementia
assessment forms for home visits and a
videoconference solution for collaborative report
writing were performed with the participation of an
IMC dementia team (Smaradottir et al., 2014).
This paper reports from the fourth phase of the
project. The final version of the CIS was developed
and a usability evaluation was carried out together
with end-users in order to validate whether the
system accomplished acceptable levels of
effectiveness, efficiency and satisfaction. In
addition, reflections from the UCD process that
involved the IMC dementia team are presented.
The research questions (RQs) of this study were:
RQ1: How can an information system be evaluated
taking into account the needs and requirements of
306
Smaradottir, B., Martinez, S., Holen-Rabbersvik, E., Vatnøy, T. and Fensli, R.
Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process.
DOI: 10.5220/0005703603060313
In Proceedings of the 9th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2016) - Volume 5: HEALTHINF, pages 306-313
ISBN: 978-989-758-170-0
Copyright
c
2016 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
the end-users for collaborative access and
information sharing by an inter-municipal team of
dementia assessment?
RQ2: What lessons and methodological
procedures from this study are transferable and
applicable to development of technological solutions
for other clinical assessment workflows?
2 RESEARCH BACKGROUND
Dementia is a clinical syndrome with deterioration
of mental abilities and cognitive skills (Gustafson,
1996). For assessment of the cognitive aspects of
dementia, a widely used method is the cognitive
mental status examination, the Mini-Mental State
(MMS), (Folstein et al., 1975). The MMS is a set of
questions whose scored answers result in a category
of cognitive mental status.
A dementia plan was implemented in Norway in
2007 (Norwegian Ministry of Health and Care
Services, 2007), aiming at improving capacity,
competence and quality in dementia care and
enhancing the need for qualified competence in
primary care. However, due to the small or medium
size of many Norwegian municipalities, specialised
IMC dementia teams have been established
(Directorate of Health, 2011) to collaboratively carry
out the assessment of people with dementia in
neighbour municipalities.
Recently, a Delphi study with experts in
coordination and IMC in health services reached
consensus about the challenges concerning
electronic communication. Specifically, the lack of
tools impeded the collaboration of IMCs (Holen-
Rabbersvik et al., 2013). Therefore, IMC dementia
teams face challenges generated by their nature of
operation, such as limited information flow across
the municipalities and interoperability problems
between different information systems (IS).The aim
of developing a CIS for IMC dementia teams was to
provide a platform that supported the information
flow and collaborative work across municipal
borders.
An effective IS requires a detailed analysis of
end-users’ needs to inform system design. In
addition, the usability of such application is crucial
for the continuous, efficient and satisfactory use of
the system. In system development, the approach of
UCD involves end-users throughout the each stage
of the development cycle
(Lazar, 2006; Gulliksen et
al., 2003; Nielsen 1993). UCD considers the needs
of the end-users through field studies, evaluations
and task analysis, helping to understand context of
use and workflow, which are key elements for the
construction of an IS for a clinical workflow (Chan
et al., 2011; Goldberg et al., 2011). In addition,
usability evaluation is necessary to analyse user’s
interaction and user satisfaction with the system
(Jaspers, 2009; Kushniruk and Patel, 2004; Lazar et
al., 2010).
3 MATERIALS AND METHODS
The prototype from the earlier phases of the
eHealth-extended Care Coordination project was
further developed by an industry partner as a full
functioning version of the CIS which was
implemented within the secure Norwegian Health
Network (NHN, 2015). The evaluation of the CIS
was executed during two days in June 2015 and
entailed three steps: (1) test in usability laboratory
with end-users, (2) individual questionnaire and (3)
group interview. A mixed methods research
approach was used including observations,
interviews and a questionnaire.
3.1 Usability Evaluation
The usability evaluation was made with end-users in
a laboratory. The facilities had two rooms (test and
observation) connected through one-way mirror
(described in Gerdes et al., 2014). In the test room,
the system was accessed and used on a laptop
connected to an external screen and keyboard. In the
observation room, the evaluation was followed by
the research team in real-time through four monitors
connected to two stationary computers.
5 participants (4 female, 1 male; aged 41-57,
average 55.6 years) with the professions nurse, nurse
coordinator and social educator, took part in the
tests. They were all members of an IMC dementia
team from 4 municipalities. They reported an
average of 16.8 years of experience using clinical
systems and evaluated their computer skills as
medium.
Each test session started with a pre-test interview
with questions about background and experience
with clinical systems. A member of the research
team moderated each session. Participants were
asked about their first impression of the graphical
user interface (GUI). A concurrent Think Aloud
protocol (Jaspers, 2009; Kushniruk and Patel 2004;
Ericson and Simon, 1980) was employed. The task
list included 9 differentiated tasks to perform within
the system. After each task, the participants were
asked to score the task solving into five categories:
Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process
307
very easy, easy, medium, difficult and very difficult.
The tasks were based on the IMC dementia team
workflow description from the UCD workshops
(Smaradottir et al., 2015b). The test sessions had
duration of 39 to 62 minutes (average 47 minutes).
3.2 System Usability Scale
In order to evaluate the user satisfaction, the
participants individually answered the post-test
questionnaire System Usability Scale (SUS)
consisting of with 10 questions (Brooke, 1996).
3.3 Group Interviews
In order to complete the feedback, two post-test
semi-structured group interviews (n=2, n=3) were
conducted to qualitatively analyse the output of the
test (average duration 37 minutes). The CIS was
shown on a screen during the interviews, allowing
the participants to follow in detail the GUI and
comment on its functionality. The main findings
from the usability evaluations were also discussed.
3.4 Data Collection
Audio-visual recordings were made with two
cameras (1 fixed and 1 portable) and a screen
capture tool (in usability evaluation) merged into
one single video file using the software Wirecast
v.4.3.1. The recordings (.mov format) were imported
into QSR NVIVO 10 for transcription and a
qualitative content analysis (Lazar et al., 2010). This
study was approved by Norwegian Social Science
Data Services (NSD, 2015) with project number:
37920. All participants signed a consent form.
4 RESULTS
The results are presented following the three steps of
the evaluation.
4.1 Usability Evaluation
The test started with the screen patients’ overview
(see Figure 1) and questions about participants’ first
impression of the GUI. They generally stated that
the screen was useful to get a fair overview of
patients. Three participants positively commented on
the search function used to find a specific patient.
About the GUI, comments highlighted the
appropriate choice of colours, with the exception of
poor readability and contrast of black text over blue
Figure 1: GUI of patients’ overview.
background in patients’ overview screen. It was
pointed out that it could be difficult to read white
text sections, especially in rooms with bright light.
In addition, the insufficient font size both in text and
headings was stated recommending to adapt the GUI
to the full screen size. Suggestions included being
able to run a search writing only 3 letters and
increasing the speed of the search results.
On each individual patient’s view, comments of
the GUI (see Figure 2) confirmed the abundance of
colours, intended to visually inform about the
sections’ functionality. In this line, participants
commented: I liked the choice of colour and graphic
design. Very clear and easy to read. When you are
working on a patient, the colours can tell you where
you are. Patient’s key information was coloured as a
yellow section and placed at the top right.
4.1.1 Task Performance
All 5 participants successfully solved all the tasks,
with different degrees of help from the moderator.
Task 1: Add a new patient to CIS
Participants had to click the ‘+’ sign to access
administrative functionalities and be able to
register a new patient into the system (see
Figure 3). The task was unanimously scored
as easy. 2 participants had errors with the
Figure 2: Individual patient’s view.
HEALTHINF 2016 - 9th International Conference on Health Informatics
308
Figure 3: GUI of administrative functionalities.
input format while registering patient’s
birthdate, having to try few extra times.
Suggestions were made about having text
boxes with the exact format of the field to
avoid errors. Error messages would have to be
written in colour to improve readability. When
typing a post code, the city would have to
automatically appear. The labelling of the
button to register a new patient was suggested
as save instead of create as a more intuitive
description The list of patients was suggested
to be sorted either alphabetically or
chronologically; for the chronological order,
the newest patients would be placed at the top.
Task 2: Add General Practitioner to CIS
To solve the task, the administrative
functionality of the GUI had to be accessed
and health care professional be chosen as
action for data input. All participants needed
help to solve the task, one participant needing
up to nine attempts. Task was scored as
difficult and finding administrative
functionality was tagged as problematic. 2
system errors were identified relating to
repetition of information: 1) when clicking
twice on create. In this case, patient was
stored twice with the same name without
notifying the user. 2) when typing a long
email address the phone number field became
invisible due to of lack of space.
Comments on navigation issues in the GUI:
Information input was ok, but the navigation
was difficult. The task was difficult to solve,
because the problem was navigation.
Task 3: Add relative into CIS
To solve the task, the administrative
functionalities of the GUI had to be accessed.
Then, health care professional had to be
chosen and change the role to relative for data
input. 4 participants successfully solved the
task without help; one of them tried few times
before succeeding and another asked for help.
3 participants scored the task as easy, 1 as
medium and 1 as difficult. Participants
suggested being able to add different types of
relatives such as closest relative, friend,
guardian or other. They also suggested that it
would have been preferable to be able to make
a priority list of whom to contact in case of
multiple relatives registered. Comments
related to understanding how the roles were
interpreted in CIS: I found health care
professional but did not understand that it was
the right one, and the role had to be changed
to relative. It is difficult when I have not seen
the system before… Difficult to navigate, the
input was easy.
Task 4: Navigate to patient’s view in CIS
To find the new patient’s view, firstly the icon
home had to be selected and then selecting
patient’s name in order to enter patient’s view.
4 participants successfully solved the task and
one needed help after two incorrect actions.
Task 5: Add a task into the Patient’s View
It was necessary to click on ‘+’ symbol in the
section Tasks to solve the task. 3 participants
successfully solved the task, although 2
needed help: I did not see the heading Tasks…
I did not see tasks, did not understand to
watch on top. 2 participants scored the task as
easy
, 1 as medium and 1 as difficult. One word
regarding who to perform an action was
misunderstood and that led to confusion.
Task 6: Upload a referral into CIS
Participant had to click the ‘+’ symbol in
Documents section and upload a document to
solve the task. 2 participants successfully
solved the task and 3 needed help. 2
participants evaluated the task as easy, 1 as
medium and 1 as difficult.
Task 7: Upload a dementia assessment
report into CIS
The task was similarly solved as task 6,
adding a document and uploading it. All
participants successfully solved the task and
graded it as easy: Now I have tried this once
before.
Task 8: Upload the clock-test into CIS
The task was solved similarly as task 6 and 7,
adding a document and uploading it. All
participants successfully solved the task and
graded it as easy: Now I start to understand
how the program is organised.
Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process
309
Task 9: Write a journal note into the just-
registered patient’s view
Participants had to click on the ‘+’ symbol in
the Journal note section to solve the task. All
5 participants successfully solved the task and
graded it as easy.
4.2 System Usability Scale
The scores of the SUS questionnaire are presented in
Table 1. The colour visualisation scheme presented
is a modified version of MacLellan et al., 2012, and
Bangor et al., 2009. Overall, the mean of the
satisfaction ratings were on the range of Agree,
Strongly Agree or Neutral for the majority of
answers to the positively enunciated questions and in
the range of Disagree, Strongly Disagree or Neutral
for the majority of answers for the negatively
enunciated questions.
Table 1: Responses of System Usability Scale (SUS).
Question P1 P2 P3 P4 P5 M SD
Q1 3 3 4 4 3 3.4 0.5
Q2 4 2 1 3 3 2.6 1.1
Q3 3 4 4 3 3 3.4 0.5
Q4 1 2 1 3 1 1.6 0.9
Q5 4 4 5 4 3 4.0 0.7
Q6 2 2 1 3 2 2.0 0.7
Q7 5 4 4 4 4 4.2 0.4
Q8 2 5 1 2 2 2.4 1.5
Q9 3 4 3 3 3 3.2 0.4
Q10 2 4 1 3 2 2.4 1.1
Pi = participant i; M = mean; SD = Standard Deviation
Positive Response: Agree or Strongly Agree for positive
questions; Disagree or Strongly Disagree for negative
questions
Neutral: neither Agree nor Disagree
Negative Response: Agree or Strongly Agree for
negative questions; Disagree or Strongly Disagree for
positive questions
4.3 Post-test Group Interviews
The group interview results are presented in four
categories.
4.3.1 Test Scenario and Procedure
Participants defined the test experience as exciting
and similar to the earlier tests. They found that the
questions made after each task to grade the difficulty
of the task accomplishment were slightly more
difficult to answer. A participant commented on it: I
did not find everything, but still I don’t think this
was a complicated program. When you receive help
once, then you learn how to do it and it is easier next
time. If I had used longer time during each task I
would have probably found it by myself.
4.3.2 User Training
Participants suggested that having user training in
advance would have been useful and, in addition,
would help them to provide more feedback. They
commented that in their daily basis, they have user
training when new functions and updates are
implemented in the systems. One participant
commented: If I had been allowed to sit 10 minutes
by myself to explore the system, the test would have
been much easier. […] If we had been instructed in
advance about the three main elements I would have
understood the structure earlier.
4.3.3 Navigation
Participants found the data input for all tasks easy
with good visibility of the displayed information.
Participants found difficult how to access patient’s
journal from the administrative section (task 4). In
this line, one participant commented: It was not
obvious; I would not have found it without help. In
general, they suggested as few clicks as possible,
displaying some information only in request (e.g.,
email address, contact information to GP) using
icons.
4.3.4 Municipal EHR versus CIS
The IMC dementia team used a municipal electronic
health record (EHR) system in their daily work.
Even though participants were positive earlier in the
research project towards the new collaborative
information system, in this evaluation they
expressed some scepticism about the co-existence of
the new system with the ones previously used by the
team: I would find it a bit cumbersome to have two
different systems, one system for the inter-municipal
dementia team, and one for everything else. […] I
would not like to change the system we have now,
since we would have two systems to use. I don’t
think that is smart and would be more difficult to
work.
One of the reasons argued was that the initial
circumstances when the project started have changed
throughout the project period. One participant
commented: We get new tasks all the time and that
demands more from us. We need to ease the working
processes as much as possible. We should not have
too many programs to use. I am afraid that this
system will cause double work, instead of having one
HEALTHINF 2016 - 9th International Conference on Health Informatics
310
single system. Four years ago, I was much more
positive, because then, we did not have e-messages
or access to EHR systems in other municipalities.
Some of the problems we had at that time are now
solved. The implementation of e-message (1.5 years
ago) did revolutionise our daily work. The dementia
report is now sent as e-message. In addition, we are
now used to the tablet and to take a picture of the
clock test to upload into the municipality EHR,
instead of scanning as we did earlier. So there are
fewer papers involved now.
Another participant stated: Instead of implementing
a new system, I would suggest a collaborative space
in the [municipal EHR]. That would be helpful, with
collaborative access for the inter-municipal
dementia team to the patients undergoing dementia
assessment.
Another reason was that, when the project
started, the IMC dementia team was recently
established and they were inexperienced as a
collaborative team. Since then, they have had over a
100 dementia assessments. Routines have been
improved and less time is now used on each home
visit and in the report writing. In addition, the laws
regarding shared access to medical information
across health organisations have been changed
during the project period, and the nurse coordinator
now had acquired legal access to the EHR systems
in the involved municipalities (even though with
separate username and passwords for each system to
log in). Although the participants expressed a
sceptical attitude towards implementation of the
CIS, it was stated: I like this new system and would
find it helpful. In [municipal EHR] there are too
many clicks and the information input is much more
complicated. Another participant commented:
Anyhow, I think this system would be useful. In
[municipal EHR], I need to search a lot for
information. I liked the visibility of the key
information.
Overall, participants positively commented the
participation in the research project: The
participation in this project has been interesting.
They received the news that the outcome of the
earlier phases of this project informed the creation of
another IS for remote monitoring of COPD patients:
Nice to hear that what we have participated in has
been used in another system, living its own life. […]
So our contribution already has come to use.
5 DISCUSSION
In this paper, the usability evaluation of a
collaborative information system for an IMC
dementia team has been presented. The aim of the IS
was to provide a platform that supported the
information flow and collaborative work across
municipalities’ borders. An effective IS requires a
detailed analysis of end-users’ needs, preferences
and suggestions to inform system design. For this
reason, a UCD process was employed involving
end-users in design and evaluation throughout the
entire development cycle.
The two research questions presented at the
beginning of the paper were answered based on the
results of this study. About the RQ1, which asked
about how to take into account user needs and
requirements in the evaluation of a new IS, the study
showed that the mixed methods approach efficiently
considered user needs in the evaluation of the
system. The approach was divided in three stages.
The first stage was the evaluation in the usability
laboratory, were participants performed a series of
tasks based on the IMC dementia team workflow
description provided by the users in earlier UCD
workshops. This test enabled users to give useful
feedback and first impressions about the GUI,
functionality and interactions with the system. The
second stage included a questionnaire (SUS) with 10
questions related to user satisfaction after task
solving. It showed that, overall, participants were
generally satisfied with the use of the system. The
third stage included post-evaluation semi-structured
group interviews that allowed participants to discuss
the main findings with the research team and
spontaneously make any suggestions. This stage
gave the opportunity to participants to make
comments and exchange impressions in a group,
rather than individually, what presented the research
team with new situations to learn from and which
were not previously considered (e.g., slight
reluctance to final implementation due to potential
integration problems with coexisting systems and
user work overload).
Several lessons were learned during the UCD
process that can be transferable for the development
of solutions for other clinical assessment workflows
(RQ2). Firstly, the creation of clinical systems
requires active and continuous involvement of the
end-users in the design and evaluation of the
solution. Secondly, the circumstances for the context
of use may change over the study’s time span. The
nature of this research was linked to a Norwegian
research project with the time duration of four years.
Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process
311
The key requirements for the system that were
gathered in a field study and several user workshops
in an early project phase changed as the project
evolved. For instance, new functionalities provided
and included in the collaborative information system
were, during the project time, also implemented in
parallel in existing systems. At the end of the
project, this resulted in a reduction of end-user
interest in using the new system because they
reported that improvements were already in place in
existing systems. In addition, due to recent law
changes, shared access across municipal borders was
now allowed improving information flow and
electronic communication. Thirdly, new system
integration with existing systems is vital to, at least,
not increase user workload. This is a logical
consequence of the previous lesson.
There were some limitations associated to this
study, such as the use of a simulated test
environment and a reduced number of end-users.
Firstly, although the laboratory setting realistically
simulated the work environment and representative
end-users carried out the tests for validation of the
system, the study was performed in a simulated
instead of real environment. This should be seen as a
first step in the validation, complemented by a test
of the system in real clinical settings through a field
trial would be recommended before final
implementation. Secondly, the reduced number of
participants in the usability evaluation might be seen
as an impediment of the applicability of the findings
in a larger scale. However, the participants
meaningfully represented the end-users of the
system and in qualitative usability studies, a small
number of participants can be sufficient for having
valid results (Nielsen, 1994; Nielsen and Landauer,
1993).
6 CONCLUSIONS
This study was framed inside the project eHealth-
extended Care Coordination, which aimed to
develop a collaborative information system to be
used in dementia assessment to improve the
information flow between the members of an inter-
municipal team. The system would ideally be the
core for IMC health care teams, potentially
adaptable for other clinical workflows. A UCD
process was employed throughout the whole
duration of the project, in which all the versions of
the system were evaluated and tested. The usability
evaluation, together with graphical assessment and
group interviews of the system, identified
refinements in order to improve the functionality and
effectiveness of the solution before implementation.
The SUS questionnaire showed a high score of user
satisfaction.
The time span of the project, to which this study
belonged, was four years. This period represents a
substantial amount of time in clinical environments,
usually associated with an increased demand for
technological solutions that quickly and easily adapt
to continuously evolving workflows, requirements
and existing systems. Therefore, when implementing
a new system, functionality should not duplicate the
one from existing systems. In addition, there is a
need of rapid development of new ICT capable of
integration with other parallel activities and systems.
These systems are typically used within
organisations facing continuous changes as in the
health care services.
The initial GUI of the CIS for dementia
assessment established the foundation for the user-
centred design and development of an information
and management system for remote telemedicine
monitoring of COPD patients at home (Smaradottir
et al., 2015a), which has been implemented in the
FP7 EU project United4Health (United4Health,
2015), currently being successfully used in 3 inter-
municipal telemedicine centres in Norway.
Future research would include a full
implementation of the system, with its
corresponding evaluation in the field from a
usability and operational perspective. In addition, a
comparison of the new and the already existing
system would provide useful results.
ACKNOWLEDGEMENTS
The authors thank all participants for their
contribution in the research study and Devoteam AS
for the development and implementation of the
system. Financial support was provided by the
Regional Research Fund, Agder, Norway (Grant
number 204119-2011).
REFERENCES
Bangor, A., Kortum, P., Miller, J., 2009. Determining
What Individual SUS Scores Mean: Adding an
Adjective Rating Scale. Journal of Usability Studies,
Vol. 4, Issue 3, 114-123.
Brooke, J., 1996. SUS: a quick and dirty usability scale.
Usability Evaluation in Industry. London: Taylor &
Francis, p.189-194.
HEALTHINF 2016 - 9th International Conference on Health Informatics
312
Chan, J., Shojania, K. G., Easty, A. C., Etchells, E. E.,
2011. Does user-centred design affect the efficiency,
usability and safety of CPOE order sets? Journal of
the American Medical Informatics Association, 18,
276e281. doi:10.1136/amiajnl-2010-000026.
Directorate of Health and National Expertise Service for
Ageing and Health, 2011. Etablering og drift av.
Demensteam, Demensutredning i primærhelsetjenesten.
[cited 2015 Dec 10]. Available from: http://
www.aldringoghelse.no/ViewFile.aspx?ItemID=2824.
Ericsson, K. A., Simon, H. A., 1980. Verbal reports as
data. Psychological Review, 87(3), 215-251.
Folstein, M. F., Folstein, S. E., McHugh, P. R., 1975.
“Mini-mental state”: A practical method for grading
the cognitive state of patients for the clinician. Journal
of Psychiatric Research, Vol 12, Issue 3, 189-198.
doi:10.1016/0022-3956(75)90026-6.
Gerdes, M., Smaradottir, B., Fensli, R., 2014. End-to-end
infrastructure for usability evaluation of ehealth
applications and services. Proceedings of
Scandinavian Conference on Health Informatics,
Grimstad, Norway, August 21-22, 53-59. ISSN(print):
1650-3686, ISSN(online): 1650-3740, ISBN: 978-91-
7519-241-3.
Goldberg, L., Lide, B., Lowry, S., Massett, H. A.,
O'Connell, T., Preece, J., Quesenbery, W.,
Shneiderman, B., 2011. Usability and accessibility in
consumer health informatics current trends and future
challenges. American Journal of Preventive Medicine
Vol 40, Issue 5, Supplement 2, S187–S197.
doi:10.1016/j.amepre.2011.01.009.
Gulliksen, J., Göransson, B., Boivie, I., Blomkvist, S.,
Persson J., Cajander, Å., 2003. Key principles for
user-centred systems design. Behaviour & Information
Technology, 22:6, 397-409.
doi:10.1080/01449290310001624329.
Gustafson, L., 1996. What is dementia? Acta Neurologica
Scandinavica, Volume 94, Issue Supplement s168,
22–24. doi: 10.1111/j.1600-0404.1996.tb00367.x.
Holen-Rabbersvik, E., Eikebrokk, T. R., Fensli, R. W.,
Thygesen, E., Slettebø, Å., 2013. Important challenges
for coordination and inter-municipal cooperation in
health care services: a Delphi study. BMC health
services research, 13(1), 451. doi: 10.1186/1472-
6963-13-451.
Jaspers, M. W., 2009. A comparison of usability methods
for testing interactive health technologies:
methodological aspects and empirical evidence.
International Journal of Medical Informatics, 78(5),
340-353.
Kushniruk, A. W., Patel, V. L., 2004. Cognitive and
usability engineering methods for the evaluation of
clinical information systems. Journal of Biomedical
Informatics, 37(1), 56-76.
Lazar, J. 2006. Web Usability- a user-centered design
approach. Pearson Education. ISBN-10: 0321321359
ISBN-13: 9780321321350.
Lazar, J., Feng, J. H., Hochheiser, H., 2010. Research
methods in human-computer interaction, Chichester,
United Kingdom, John Wiley & Sons, ISBN 978-0-
470-72337-1.
MacLellan, S., Muddimer, A., Peres S., C. 2012. The
Effect of Experience on System Usability Scale
Ratings. Journal of Usability Studies, Vol. 7, Issue 2,
56-67.
NHN, 2015: Norwegian Health Network. [cited 2015 Dec
10]. Available from: https://www.nhn.no/english/
Pages/default.aspx.
Nielsen, J., 1994. Estimating the number of subjects
needed for a thinking aloud test. International Journal
of Human-Computer Studies, 41(3), 385-397.
doi:10.1006/ijhc.1994.1065.
Nielsen, J. 1993. Usability engineering, Elsevier. ISBN-
13: 978-0125184069, ISBN-10: 0125184069.
Nielsen, J., and Landauer, T. K., 1993. A mathematical
model of the finding of usability problems.
Proceedings of ACM INTERCHI'93 Conference,
Amsterdam, The Netherlands, April 24-29, pp. 206-
213.
Norwegian Ministry of Health and Care Services, 2007.
Dementiaplan 2015 (Demensplan 2015 “Den gode
dagen” Delplan til omsorgsplan 2015). [cited 2015
Dec 10]. Available from: http://www.regjeringen.no/
upload/HOD/Vedlegg/Omsorgsplan_2015/Demenspla
n2015.pdf.
Norwegian Ministry of Health and Care Services, 2008-
2009. Report No. 47. The Coordination Reform,
Proper treatment – at the right place and right time.
[cited 2015 Dec 10]. Available from:
https://www.regjeringen.no/contentassets/d4f0e16ad3
2e4bbd8d8ab5c21445a5dc/no/pdfs/stm200820090047
000dddpdfs.pdf.
NSD, 2015: Norwegian Social Science Data Services.
[cited 2015 Dec 10]. Available from:
http://www.nsd.uib.no/personvern/en/index.html.
Smaradottir, B., Gerdes, M., Martinez, S., Fensli, R.,
2015a. The EU-project United4Health: user-centred
design of an information system for a Norwegian
telemedicine service. Journal of Telemedicine and
Telecare Nov 2015. doi:10.1177/1357633X15615048.
Smaradottir B., Holen-Rabbersvik, E., Thygesen, E.,
Fensli, R., Martinez, S., 2015b. User-centred design
and usability evaluation of the user interface of a
collaborative information system for inter-municipal
dementia team. Proceedings of International
Conference on Health Informatics, Lisbon, Portugal
January 12-15, 446-453. doi:
10.5220/0005222704460453.
Smaradottir B., Holen-Rabbersvik, E., Thygesen, E.,
Fensli, R., Martinez, S., 2014. Usability evaluation of
electronic forms and collaborative assessment report in
an inter-municipality health care team for dementia
diagnose. Proceedings of Scandinavian Conference on
Health Informatics, Grimstad, Norway, August 21-22,
45-54. ISSN(print): 1650-3686, ISSN(online): 1650-
3740, ISBN: 978-91-7519-241-3.
United4Health 2015, European Commission
Competitiveness Innovation Programme, ICT
Programme to Support Policy. [cited 2015 Dec 10].
Available from: http://www.united4health.eu.
Usability Evaluation of a Collaborative Health Information System - Lessons from a User-centred Design Process
313