SClinico: Usability Study

João Pavão

, Rute Bastardo

, Marta Covêlo

, Luís Torres Pereira

, Paula Oliveira

, Catarina Pedrosa

Ana Silva

, Victor Costa

, Ana Isabel Martins

, Alexandra Queirós

and Nelson Pacheco Rocha

INESC-TEC, Science and Technology School, University of Trás-os-Montes and Alto Douro, Quinta de Prado,

5001-801 Vila Real, Portugal

Science and Technology School, University of Trás-os-Montes and Alto Douro, Quinta de Prado,

5001-801 Vila Real, Portugal

CITAB, Science and Technology School, University of Trás-os-Montes and Alto Douro, Quinta de Prado,

5001-801 Vila Real, Portugal

Centro Hospitalar of Trás-os-Montes e Alto Douro, Avenida da Noruega, 5000-508 Vila Real, Portugal

IEETA, Department of Medical Sciences, University of Aveiro, Campo Universitário de Santiago,

3810-193 Aveiro, Portugal

IEETA, Health Sciences School, University of Aveiro, Campo Universitário de Santiago, 3810-193 Aveiro, Portugal

Keywords: Usability, Electronic Health Record, Sclinico.

Abstract: The use of electronic health records (EHR) to support clinical practices is widespread worldwide, due to the

need to optimize health care delivery. Therefore, the usability assessment of EHR systems is crucial. The

objective of this study was to perform a qualitative and quantitative assessment of the usability of SClinico,

the most used EHR system within the Portuguese National Health Service. This observational study to assess

SClinico usability took place in several clinical services of the Centro Hospitalar de Trás-os-Montes e Alto

Douro. The results show that SClinico has some usability issues that influence the clinical practice and,

therefore, need to be improved.

1 INTRODUCTION

In 1998, the International Organization for

Standardization (ISO) defined usability as being the

“extent to which a product can be used by specified

users to achieve specified goals with effectiveness,

efficiency, and satisfaction in a specified context of

use” (International Organization for Standardization,

1998).

The present paper reports an observational study

to assess the usability of SClinico, the most used

electronic health records (EHR) system within the

Portuguese National Health Service. This study was

performed in several clinical services of the Centro

Hospitalar de Trás-os-Montes e Alto Douro (Trás-os-

Montes and Alto Douro Hospital Centre), known as

CHTMAD, composed by three hospitals of the

Portuguese Serviço Nacional de Saúde (National

Health Service).

In addition to this introductory section, the paper

comprises five more sections: Related Work,

Methods, Results, Discussion, and Conclusion.

2 RELATED WORK

An EHR, in its simplest form, consists of an

electronic file containing clinical information about

the individuals and can help to personalize care,

prevent medical errors, promote the consistency of

care, support the referral of the care receivers to the

correct services, control costs or support clinical

research. In terms of health care delivery, EHR

systems represent a benefit to both the patient and

organizations, since they enable secure, accessible

and efficient clinical information reporting and

retrieving and, therefore, contribute to the quality of

the health care delivery.

EHR systems allow the clinical report, as well as

the access, exchange and share of the required clinical

information (Rouleau et al., 2015). Therefore, EHR

systems support highly collaborative and demanding

Pavão, J., Bastardo, R., Covêlo, M., Pereira, L., Oliveira, P., Pedrosa, C., Silva, A., Costa, V., Martins, A., Queirós, A. and Rocha, N.

SClinico: Usability Study.

DOI: 10.5220/0006545300480056

In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 5: HEALTHINF, pages 48-56

ISBN: 978-989-758-281-3

processes, involving large number of clinicians with

distinct profiles (e.g. physicians, nurses or therapists),

social workers, managers and administrative

personnel (Marin, 2010; Mair et al., 2012).

Adequate and efficient mechanisms must be made

available to allow the visualization of the required

information to prevent and treat the disease,

whenever and wherever required (Zahabi et al.,

2015). In fact, the use of clinical information is often

carried out in different contexts, such as, for example,

an emergency service of any hospital, where the rapid

access to comprehensive patient’s clinical

information is essential. The adaptability of health

care applications to different situations might

influence the way the clinical information is reported

and retrieved, which, necessarily, has impact on the

health care delivery.

According to Zahabi and colleagues (Zahabi et al.,

2015), some of the problems with the implementation

of EHR systems may be related to the usability of the

respective user interfaces, such as poor readability,

poor alarm identification or insufficient feedback on

the actions being taken. Considering that EHR

systems are fundamental tools for the health care

delivery, it is therefore important to assess their

usability (Bhutkar et al., 2013).

Internationally there are many usability studies of

EHR systems, targeting different clinical domains

(American Medical Association, 2014; Feng, Chang,

2015; Villa, Cabezas, 2014; Choi et al., 2015; Czaja

et al., 2015; Clarke et al., 2014). There are also efforts

to standardize the usability assessment methods

attempting to find a common ground, which is

fundamental to perform comparative studies

(Johnson et al., 2011; Zhang and Walji, 2014).

According to Saitwal and colleagues (Saitwal et

al., 2010), some EHR systems do not have user-

friendly interfaces, as they often do not take in

consideration the user centric development. The

referred authors (Saitwal et al., 2010) applied a

cognitive analysis method to assess the usability of

EHR systems, which made possible the identification

of issues that could be improved, namely the

reduction of the number of steps to perform certain

tasks, together with the minimization of the cognitive

effort required to execute them (Marquié et al., 2015).

Edward and colleagues (Edwards et al., 2008)

argue that the usability of EHR systems is crucial to

ensure safety and to enable clinicians to focus more

on patients and less on technology. These authors

used a heuristic assessment method to identify points

of possible usability improvements of an EHR system

supporting a paediatrics service. The results of this

usability assessment allowed immediate

improvements in the EHR system configuration and

training materials.

Since there is a worldwide trend to consider

usability as one of the important factors of the

implementation of EHR systems, the American

Medical Informatics Association (AMIA) created a

working group to propose relevant recommendations

(Middleton et al., 2013). Moreover, the National

Institute of Standards and Technology (NIST)

describes a set of procedures for design evaluation

and user performance testing of EHR systems by

defining an EHR usability protocol (EUP) (Lowry et

al., 2012). The purpose of this protocol is to provide

methods to measure and validate user performance

prior to deployment and it comprises a three-step

process: application analysis, user interface expert

review, and user interface validation testing (Lowry

et al., 2012).

Thus, since there is no solid evidence on the

usability assessment of SClinico, the most used EHR

system within the Portuguese National Health

Service, and given the growing interest of the

international community in the usability of EHR

systems, the authors considered important to assess

the usability of SClinico (Pavão et al., 2016).

3 METHODS

3.1 SClinico

In the Portuguese public hospitals, clinical records

have been carried out in electronic format since 1988,

using a common information system, the Sistema

Integrado de Informação Hospitalar (Integrated

Hospital Information System) also known as

SONHO, and two specific health care applications for

physicians and nurses, respectively the Sistema de

Apoio ao Médico (Medical Support Service), known

as SAM, and the Sistema de Apoio à Prática de

Enfermagem (Nursing Practice Support System),

known as SAPE. These two applications were used

until 2013. In 2014, the application known as

SClinico replaced the previous two, by integrating in

a single application the medical and nurse profiles.

According to the organization responsible for the

development of SClinico, the Serviços Partilhados do

Ministério da Saúde (Shared Services of the Ministry

of Health), known as SPMS, SClinico is present in

more than 50 public hospitals, as well as in more than

300 primary care centres (Serviços Partilhados do

Ministério da Saúde, 2017a; 2017b).

The SClinico is part of the strategy defined by the

Portuguese Ministry of Health to introduce

SClinico: Usability Study

information technologies in the Portuguese National

Health Service. SClinico was born from the vast

experience with two previous applications used by

thousands of clinicians of the Portuguese National

Health Service: the SAM and SAPE applications,

which have evolved to be a single application for all

clinicians and focused on the patient (Serviços

Partilhados do Ministério da Saúde, 2017a; 2017b).

With SClinico, the Portuguese Ministry of Health

foresees the provision of a standardized tool for

reporting and retrieving clinical information to

promote the homogenization of procedures at

national level (Serviços Partilhados do Ministério da

Saúde, 2017a; 2017b). This might contribute to the

efficacy and efficiency of health care delivery,

allowing the clinicians to perform better their role in

multidisciplinary teams and to provide better support,

assistance and follow-up to their patients (Serviços

Partilhados do Ministério da Saúde, 2017a; 2017b).

The SClinico has two versions, SClinico Hospital

and SClinico Primary Health Care. The first, SClinico

Hospital, has more than sixty thousand registered

clinicians. In turn, more than thirteen thousand

clinicians are using the SClinico Primary Health Care

(Serviços Partilhados do Ministério da Saúde, 2017a;

2017b). Particularly, in this study the SClinico

version that was assessed was the SClinico Hospital

version.

3.2 Usability Assessment

The usability of an application, decisive when dealing

with health care application, is influenced by the

efficiency and efficacy of the respective user

interface, as well as the required cognitive effort and

overall user’s satisfaction (Nielsen, 1993). It is

expected that the focus of clinicians will be the

patients and, in this sense, the more intuitive user

interface the more it will contribute to adequate and

proficient care.

A user interface with adequate usability allows

the reduction of memorization efforts, tasks

fulfilment duration and errors, as well as the

promotion of its user’s satisfaction. As Nielsen

(Nielsen, 1993) points out, these characteristics

influence the acceptability of a system by their users.

Usability assessment is an important part of the

overall design of the user interfaces, which consists

of interactive prototyping, design and validation

cycles (Ivory and Hearst, 2001). Usability assessment

must be understood in a comprehensive way in the

context of a given project and cannot be done without

connection to the intended functions of the system.

There is a wide range of tools and methods to

identify and assess system’s usability and to

contribute, directly or indirectly, to its improvements

(Martins et al., 2013; Martins et al., 2014). In

particular, user-centred development methods ensure

that real systems can be used by real people to

perform their tasks in the real world (Bevan, n.d.).

3.3 Objectives of the Study

The primary objective of the observational study

reported in this paper was to assess the usability of the

SClinico, particularly the SClinico Hospital version,

as well as the level of satisfaction of its users. As a

secondary objective, it was intended to identify, in

terms of usability, improvements that should be

implemented.

3.4 Research Plan

Data collection was performed in several clinical

services of CHTMAD.

The protocol of the observational study

comprised two stages. In each of them, a specific

usability assessment method was used: in the first

stage, a qualitative assessment was carried out; while

in the second stage, a quantitative assessment was

performed using a validated usability assessment

scale.

The first stage of SClinico’s assessment took

place few weeks after it has being introduced in the

clinical services of CHTMAD (i.e. May and June

2014). In turn, the second step was performed

eighteen months after the first assessment, when all

potential users were already familiar with SClinico.

The qualitative assessment for the first stage was

performed by a specially prepared questionnaire

(Covêlo, 2015). In turn, the second stage consisted in

a quantitative self-reported assessment of the

SClinico usability (user opinion) using the Post-Study

System Usability Questionnaire (PSSUQ) (Lewis,

2002). Specifically, after using SClinico the

clinicians were invited to participate in the study and

to complete the PSSUQ.

3.5 Instruments

The development of the questionnaire used on the

qualitative assessment was based on a literature

research related to relevant questions that should be

considered when assessing the usability of EHR

systems (Covêlo, 2015; Pavão et al., 2016). It is

composed of two sections, the first one dealing with

demographic aspects and the second with the

HEALTHINF 2018 - 11th International Conference on Health Informatics

interaction with SClinico. This second section has 45

questions, and if a particular inquired clinician was

allocated to the emergency service, he/she would

have to answer four more questions. Most of the

questions are closed answer questions, being seven of

them open answer questions aimed to detail some of

the closed answer questions.

The questions that constitute the second section

were grouped in the following classes, accordingly to

the recommendations of other studies (Covêlo, 2015):

• System performance (10 questions).

• Clarity of information (12 questions).

• Quality of the graphical interface (12

questions).

• Adequacy of the system's functions to the

tasks performed (11 generic questions plus

four questions specifically related to the

emergency service).

In turn, the PSSUQ, used for the quantitative

usability assessment, was developed by International

Business Machines (IBM) and consists of 19 items

rated on a 7-point scale (from ‘strongly agree’ - 1, to

‘strongly disagree’ - 7) (Lewis, 2002). The PSSUQ

addresses five usability characteristics of a system:

rapid completion of the task; ease of learning; high

quality documentation; online information; and

functional adequacy (Lewis, 2002).

According to the authors of the original version,

the score of the PSSUQ can be specified by three sub-

scores, system utility (SysUse), information quality

(InfoQual), and interface quality (IntQual), that can

be obtained as follows (Lewis, 2002):

• The mean value related to the system utility

(SysUse) - items 1 to 8;

• The mean value related to the quality of

information (InfoQual) - items 9 to 15;

• The mean value related to the quality of the

interface (IntQual) - items 16 to 18.

The final score of PSSUQ is the average of the

scores of its items. If a participant fails to respond to

an item or classify it as N/A (not applicable), then that

item should be scored with the average score of the

remaining items. Finally, higher scores indicate lower

usability and vice-versa.

The PSSUQ was translated and adapted from the

cultural and linguistic point of view to the European

Portuguese (Rosa et al., 2015). This instrument was

completed by the participants considering their

opinions about the interaction with SClinico.

3.6 Sample Selection

All physicians and nurses of CHTMAD using

SClinico in their clinical practice were eligible to

participate if they previously gave a written informed

consent accepting to participate in the study, whereby

the available population covered by the questionnaire

was composed of 1253 elements, out of which 426

were physicians and 827 were nurses. As the SClinico

interface is different for physicians and nurses, the

data analysis was performed separately.

3.7 Regulatory, Ethical and Data

Protection Aspects

To address the ethical issues, a request of approval

was directed to the administration and ethical

committee of the CHTMAD.

Additionally, all the involved subjects received

all the information regarding the study and their

participation before completing the informed consent.

This study considered all ethical principles

underlying the Helsinki Declaration (World Medical

Association, 2013), the Good Epidemiological

Practice Guidelines (International Epidemiological

Association, 2007), and the applicable legislation and

regulations.

All necessary steps were taken to protect the

participant's privacy and all relevant guidelines on

data privacy were followed: information that may

allow the identification of participants was not stored

in the study database; participants were identified in

all study documents by a unique identification

number; and the only document linking the

identification number to personal information (e.g.

names, addresses or telephone numbers) was stored

in a safe that could only be accessed by the principal

investigator. Moreover, professional secrecy was

mandatory for all investigators involved in

conducting the study and doing reports. The

investigators were told not to disclose personal

information, obtained by having access to research

data, to someone outside the research team. In

accordance with Good Epidemiological Practice

(International Epidemiological Association, 2007),

all study investigators agreed to respect the legislation

and regulations applicable to data processing

resulting from research on humans. The principal

investigator was in charge to ensure that all team

members were aware of these rules.

SClinico: Usability Study

4 RESULTS

4.1 Results of the Qualitative

Assessment

The qualitative assessment involved a convenience

sample of 22 physicians and 47 nurses from the

following clinical services of CHTMAD: medicine,

surgery, intensive care, oncology, gastroenterology,

paediatrics and emergency (Pavão et al., 2016).

The age of the participants varied between 26 and

58 years. Considering the physicians group, the

participant’s age varied between 26 and 58 years,

while in the nurses group, the participant’s age varied

between 27 and 55 years. The mean age was 43 years

(SD = 11.8) for the physicians group and 38 years (SD

= 8.3) for the nurses group. Most participants were

female in both groups, being 75% female and 25%

male.

Regarding the number of years with clinical

experience, 15% of the clinicians had 5 or less years,

32% from 5 to 15 years, 34% from 16 to 25 years and

19% from 26 to 35 years of clinical experience.

Figure 1: Percentage of answers to the question: do you

consider the SClinico appropriate tool to report and retrieve

clinical information?

Concerning the opinion of the participants about

SClinico as a clinical information reporting and

retrieving tool (Figure 1), only 44% of physicians

think it is an appropriate system, while 17% did not

respond. Regarding the nurses group, 59% of the

participants answered positively when questioned

whether SClinico is an appropriate EHR system, with

only 13% who did not respond.

Regarding the assessment of whether SClinico

facilitates the clinical practice (Figure 2), 22% of the

physicians answered negatively and 35% did not

respond, while 23% of the nurses answered

negatively and 11% did not respond. The main

concern, in the case of physicians, is related to the

organization of information that seems to be

inadequate to the needs of these clinicians. In the case

of the nurses, the ambiguity of some information

fields together with the low contrast of the colours

selected for the user interface (e.g. in the initial menu,

the design of the icons is ambiguous with colours

difficult to differentiate) do not facilitated the user

interaction.

When questioned about whether the graphic

environment is favourable after consecutive working

hours (Figure 3), most physicians were satisfied

(52%), while 31% of the physicians were dissatisfied.

There is a great difference of opinion on this issue

among physicians and nurses, since 66% of the nurses

were dissatisfied, with only 34% satisfied.

Figure 2: Percentage of answers to the question: do you

consider that SClinico facilitates your clinical practice?

Figure 3: Percentage of answers to the question: do you

consider that the SClinico's graphical environment

facilitates your clinical practice, after hours of consecutive

work?

Regarding the clinicians allocated to the

emergency service, specifically when it is required to

rapidly access to a comprehensive perspective of the

clinical history of the patient (Figure 4), most of the

HEALTHINF 2018 - 11th International Conference on Health Informatics

participants believe SClinico is not adequate, since

57% of physicians and 53% of nurses answered

accordingly.

Most clinicians think that it is very important, or

even decisive, to have a direct access to relevant

patient information in cases of urgency, to speed up

and facilitate the evaluation of the clinical history of

the patients. Nevertheless, the participants have

expressed disappointment with SClinico in this

regard.

The main reason given is related to the slowness

of the system, and it is unclear whether this is due to

issues related to the communication network of the

CHTMAD or related to usability issues of SClinico

(e.g. the need for a large number of tasks using the

pointing device or a deficient organization of the

information).

One of the most desired features for all clinicians

is the ability to view patient’s information in a

comprehensive way to cover the entire clinical

situation in a glance. This form of visualization would

allow a rapid assessment and correlation of different

clinical aspects of a patient. This question was asked

in the questionnaire and 100% of the participants of

both groups considered important the existence of this

type of function.

Figure 4: Percentage of responses to the question: in urgent

cases, do you consider SClinico adequate for the clinical

evaluation of the patient?

4.2 Results of Quantitative Assessment

For the quantitative assessment using the PSSUQ, the

sample consisted of 33 physicians and 21 nurses of all

the clinical services of one of the three hospitals that

constitute the CHTMAD: the Hospital of Lamego.

Regarding the physicians group, the average age

was 39 years (SD = 11,5). The maximum age was 66

years and the minimum age was 25 years. In terms of

gender, 48% where males and 52% were females.

The average score of PSSUQ for all physicians

was 4,10 out of 7,00 (SD = 1,35), which indicates a

medium low degree of usability and satisfaction.

The results of the sub-scales associated with

PSSUQ were:

• System Utility (SysUse): average of the

responses from items 1 to 8 = 3,75 (SD =

1,48).

• Quality of information (InfoQual): average of

the responses from items 9 to 15 = 4,47 (SD =

1,47).

• Interface Quality (IntQual): average of the

responses from items 16 to 18 = 4,00 (SD =

1,60).

In a more detailed analysis, the questions that

obtained better results were ‘I felt comfortable using

this system’ (3,20 out of 7,00) and ‘it was easy to

learn to use this system’ (3,30 out of 7,00). Both

questions are related to the System Utility (SysUse)

sub-scale.

The questions that obtained worse results were

‘the system gave error messages that clearly told me

how to fix problems’ (5,20 out of 7,00) and

‘whenever I made a mistake using the system, I could

recover easily and quickly’ (5,03 out of 7,00). Both

questions are part of the quality of information

(InfoQual) sub-scale related to the ability to recover

from errors and the availability of support when

recovering from errors.

Regarding the nurses group, the average age was

46 years (SD = 9,3). The maximum age was 57 years

and the minimum age was 24 years. Considering the

gender, 33% were male and 67% were female.

The average score of PSSUQ for all nurses was

4,83 out of 7,00 (SD = 1,21), which indicates a low

degree of usability and satisfaction.

The results of the sub-scales associated with

PSSUQ were:

• System Utility (SysUse): average of the

responses from items 1 to 8 = 4,83 (SD =

1,38).

• Quality of information (InfoQual): average of

the responses from items 9 to 15 = 4,84 (SD =

1,40).

• Interface Quality (IntQual): average of the

responses from items 16 to 18 = 4,67 (SD =

1,32).

In a more detailed analysis, the questions that

obtained better results were ‘I felt comfortable using

this system’ (4,38 out of 7,00) and ‘the interface of

this system was pleasant’ (4,43 out of 7,00).

SClinico: Usability Study

The questions that obtained worse results were

‘overall, I am satisfied with this system’ (5,38 out of

7,00) and ‘I believe I could become productive

quickly using this system’ (5,33 out of 7,00).

5 DISCUSSION

The analysis of the qualitative assessment results

identifies several usability problems that need to be

considered in more detail. One of these problems has

to do with the cognitive load. According to Zahavi

and colleagues (Zahabi et al., 2015) information

overload appears when the ability to perceive and

understand is exceeded by the amount of information

presented by a user interface, to the point of

facilitating information processing errors.

Moreover, an issue that all participants consider

important, known and discussed in the area of

information visualization (Hansen et al., 2011), was

related to the user interface adaptability and

flexibility to allow a comprehensive visualization of

a patient's clinical information. Particularly, most

participants allocated to the emergency service

answered that it is difficult, in the context of

emergency services, the rapid access to the clinical

history of the patients, which they considered a

fundamental requirement.

The quantitative results corroborate the

discontent of the SClinico users patent in the

qualitative analysis.

The physicians group considered that SClinico

has low usability. The sub-scale that presented the

best values refers to the utility of the system (SysUse)

and the one that presented worse values is the one that

refers to the quality of the information (InfoQual).

This seems to indicate that there is a recognition of

the utility of SClinico, however the clinicians were

unsatisfied with the mode the information is

presented and the general usability of the system.

The nurses group classified the usability of

SClinico even lower than the physicians group. One

of the reasons that may explain this result is the fact

that nurses spend more time reporting and retrieving

clinical information using SClinico than physicians.

This fact may justify a greater frustration of the nurses

group in terms of the SClinico usability. The overall

functioning of the system and the ability to use it

productively were the most critical aspects for the

nurses group.

For both groups of participants, the best-

performing score of PSSUQ was the system utility

(SysUse), which suggests that clinicians consider that

a system such as SClinico benefits their clinical

practice. The second was the quality of interface

(IntQual) and lastly the score of the information

quality (InfoQual).

The values of InfoQual sub-scale (i.e. 4,47 for

physicians and 4,84 for nurses) seem to suggest that

the usability problems of SClinico are reflected in the

perception that clinicians have of the information

quality that SClinico provides. This must be deeply

studied since the quality of the health care delivery

relies on the quality of the information.

6 CONCLUSIONS

In terms of study limitation, the small sample size

stands out. However, this is not significant, since the

size of the sample does not preclude some

conclusions from being drawn from the results, in

particular as regards the need to improve the usability

of the SClinico.

The results of the usability assessment

questionnaire shows that the use of SClinico would

clearly benefit from alternative forms of information

visualization, as well as a better organization of the

interfaces.

Moreover, the results of the application of the

PSSUQ, which was applied 18 months after the

introduction of SClinico (i.e. at a time when clinicians

were already accustomed to its use) point clearly to

usability problems: a general score of 4,10 (SD =

1,35) for physicians and of 4,83 (SD = 1,21) for

nurses clearly indicates low degree of usability and

satisfaction.

Given the widespread use of SClinico in the

Portuguese National Health Service, the less positive

points pointed out by the clinicians should be

considered. Therefore, further developments are

required to improve SClinico, namely in terms of

visualization of the information and organization of

the interfaces. These developments are required to

facilitate a comprehensive visualization of the

relevant information, as well as to facilitate

correlations of possible past situations with the

current episode. Furthermore, an improved version of

the graphical design, too flat with poor colour

contrast, should also be considered.

Therefore, based on the study reported in this

paper, the authors are preparing a set of suggestions

that may improve the usability of SClinico and are

planning to evaluate these suggestions in another

observational study to be conducted in the

CHTMAD.

HEALTHINF 2018 - 11th International Conference on Health Informatics

ACKNOWLEDGMENTS

The authors thank the administration board of

CHTMAD for all the support provided, as well as all

the clinicians of this hospital centre who have

accepted to participate in the study reported in this

paper.

REFERENCES

American Medical Association, 2014. Improving Care:

Priorities to Improve Electronic Health Record

Usability. AMA.

International Epidemiological Association, 2007. Good

Epidemiological Practice (GEP). IEA Guidelines for

Proper Conduct in Epidemiologic Research. IEA.

Bevan, N., European Usability Support Centres: Support

for a More Usable Information Society.

Bhutkar, G. et al., 2013. A review: healthcare usability

evaluation methods. Biomed Instrum Technol, Suppl,

pp.45–53.

Choi, M., Lee, H.S., Park, J.H., 2015. Usability of

Academic Electronic Medical Record Application for

Nursing Students’ Clinical Practicum. Healthc Inform

Res, 21(3), pp.191–195.

Clarke, M.A., Belden, J.L., Kim, M.S., 2014. Determining

differences in user performance between expert and

novice primary care doctors when using an electronic

health record (EHR). J Eval Clin Pract, 20(6), pp.1153–

1161.

Covêlo, M., 2015. Interface para Aplicação Informática de

Suporte Clínico em Ambiente Hospitalar. Vila Real:

Universidade de Trás-os-Montes e Alto Douro.

Czaja, S. J., Zarcadoolas, C., Vaughon, W. L., Lee, C. C.,

Rockoff, M. L., Levy, J., 2015. The usability of

electronic personal health record systems for an

underserved adult population. Hum Factors, 57(3),

pp.491–506.

Edwards, P. J., Moloney, K. P., Jacko, J. A., Sainfort, F.,

2008. Evaluating usability of a commercial electronic

health record: A case study. International Journal of

Human Computer Studies, 66(10), pp.718–728.

Feng, R.-C., Chang, P., 2015. Usability of the Clinical Care

Classification System for Representing Nursing

Practice According to Specialty. Comput Inform Nurs,

33(10), pp.448–455.

Hansen, D.L., Schneiderman, B., Smith, M.A., 2011.

Analyzing social media networks with NodeXL :

insights from a connected world, Burlington, MA:

Morgan Kaufmann.

International Organization for Standardization, 1998.

ISO/IEC 9241-14: 1998 (E) Ergonomic requirements

for office work with visual display terminals (VDT)s –

Part 14 Menu dialogue. ISO.

Ivory, M.Y., Hearst, M.A., 2001. The state of the art in

automating usability evaluation of user interfaces. ACM

Computing Surveys, 33(4), pp.470–516.

Johnson, C. M., Johnston, D., Crowle, P. K., 2011. EHR

Usability Toolkit: A Background Report on Usability

and Electronic Health Records. Rockville, MD:

Agency for Healthcare Research and Quality.

Lewis, J.R., 2002. Psychometric evaluation of the PSSUQ

using data from five years of usability studies.

International Journal of Human-Computer Interaction,

14(3-4), pp.463–488.

Lowry, S.Z. et al., 2012. Technical Evaluation, Testing, and

Validation of the Usability of Electronic Health

Records. USA: National Institute of Standards and

Technology.

Mair, F.S. et al., 2012. Factors that promote or inhibit the

implementation of e-health systems: an explanatory

systematic review. Bull World Health Organ, 90(5),

pp.357–364.

Marin, H., 2010. Sistemas de informação em saúde:

considerações gerais. Journal of Health Informatics,

2(1), pp.20–24.

Marquié, J.-C.C. et al., 2015. Chronic effects of shift work

on cognition: findings from the VISAT longitudinal

study. Occup Environ Med, 72(4), pp.258–264.

Martins, A.I. et al., 2013. Usability evaluation: A

systematic literature review [Avaliação de usabilidade:

Uma revisão sistemática da literatura]. RISTI - Revista

Iberica de Sistemas e Tecnologias de Informacao, 11

(1), pp. 31-44.

Martins, A. I., Queirós, A., Silva, A. G., Rocha, N. P., 2014.

Usability evaluation methods: A systematic review. In

Human Factors in Software Development and Design.

IGI Global.

Middleton, B. et al., 2013. Enhancing patient safety and

quality of care by improving the usability of electronic

health record systems: recommendations from AMIA.

J Am Med Inform Assoc, 20(e1), pp.e2–8.

Nielsen, J., 1993. Usability Engineering. Elsevier.

Pavão, J. et al., 2016. Usability study of SClinico.

Information Systems and Technologies (CISTI), 2016

11th Iberian Conference on, pp.1–6.

Rosa, A. et al., 2015. European Portuguese validation of the

Post-Study System Usability Questionnaire (PSSUQ).

In 2015 10th Iberian Conference on Information

Systems and Technologies (CISTI). pp. 1–5.

Rouleau, G., Gagnon, M.-P.P., Côté, J., 2015. Impacts of

information and communication technologies on

nursing care: an overview of systematic reviews

(protocol). Syst Rev, 4(1), p.75.

Saitwal, H. et al., 2010. Assessing performance of an

Electronic Health Record (EHR) using Cognitive Task

Analysis. Int J Med Inform, 79(7), pp.501–506.

Serviços Partilhados do Ministério da Saúde, 2017.

SClinico Hospitalar. Available at: http://spms.min-

saude.pt/product/sclinicohospitalar/ [Accessed July 17,

2017].

Serviços Partilhados do Ministério da Saúde, 2017.

SClinico Cuidados Primários. Available at:

http://spms.min-saude.pt/product/sclinicocsp/

[Accessed July 17, 2017].

Villa, L.B., Cabezas, I., 2014. A review on usability

features for designing electronic health records. In e-

SClinico: Usability Study

Health Networking, Applications and Services

(Healthcom), 2014 IEEE 16th International

Conference on. pp. 49–54.

World Medical Association, 2013. Declaration of Helsinki

- Ethical Principles for Medical Research Involving

Human Subjects. WMA.

Zahabi, M., Kaber, D.B., Swangnetr, M., 2015. Usability

and Safety in Electronic Medical Records Interface

Design: A Review of Recent Literature and Guideline

Formulation. Hum Factors, 57(5), pp.805–834.

Zhang, J., Walji, M., 2014. Better EHR: usability, workflow

and cognitive support in electronic health records.

Strategic Health IT Advanced Research Projects

(SHARPC). The National Center for Cognitive

Informatics and Decision Making in Healthcare

(NCCD).

HEALTHINF 2018 - 11th International Conference on Health Informatics