Usability Evaluation of Medical Devices during Clinical Studies: First

Results of a Scoping Review

Laura Douze

, Jessica Schiro

and Sylvia Pelayo

1,2 c

Univ. Lille, Inserm, CHU Lille, ULR 2694 - METRICS (Évaluation des Technologies de Santé et des Pratiques Médicales),

F-59000 Lille, France

Inserm, Tech4Health, F-59000 Toulouse, France

Keywords: Usability, Medical Device, Clinical Study.

Abstract: This scoping review is interested in mapping the clinical studies protocols of medical devices on usability

evaluation. The research question is as follows: How is usability of medical devices evaluated in clinical

studies? The paper presents some first results from a sample of 47 protocols within a set of 188 potentially

eligible protocols. Results highlight that a non-negligible part of usability evaluations are carried out

combined with clinical studies. Very often, usability outcomes are part of the secondary outcomes of the

clinical study. The most claimed usability-related outcomes are ease of use, handling and satisfaction.

Usability is mainly addressed through questionnaires which provide actually perceived usability (not usability

per se). Some protocols appear to be quite comprehensive in terms of usability evaluation methods.

1 INTRODUCTION

Medical devices diagnose, prevent, monitor, treat,

alleviate, or compensate for disease or injury (World

Health Organization, 2018). Their importance is

rising due to several factors, including advances in

technology, increases in lifestyle-associated disease

(Menotti, Puddu, Maiani, & Catasta, 2015;

Weisburger, 2002), and an aging population. Medical

devices developed with usability principles and

methods not only make devices easier to learn, more

efficient to use, more satisfying, and better able to fit

into peoples’ lives, but they also reduce the likelihood

of injury to patients, caregivers, and health-care

providers (Wiklund & Weinger, 2011).

The EU’s Medical Device Regulations (The

European Parliament and the Council of the European

Union, 2007, 2017) regulate the market access of new

medical technology. Since 2010, these regulations

have included the obligation to adopt a usability

engineering process. The main objective is to

optimize medical device usability as it relates to

safety, but also to task accuracy, completeness and

efficiency, and user satisfaction.

https://orcid.org/0000-0003-1759-0273

https://orcid.org/0000-0002-6710-8310

https://orcid.org/0000-0003-2830-2548

The usability engineering process is supposed to

be applied as early as possible during the

development process of a medical device. It includes

iterative usability evaluations of medical devices (i.e.

formative evaluations) and a final validation (i.e.

sommative evaluation). This final validation must

prove that the residual risk as it relates to usability is

acceptable. The EU regulation also mentions the

importance of the usability post-deployment

monitoring to follow-up the usage of the medical

device.

One of the challenges of this usability process is

to anticipate as well as possible the risks of use errors

before the deployment in real life of the medical

device. This supposes to conduct usability

evaluations as close as possible to the reality of

clinical settings. There is a need to “bring context into

the design and evaluation of usable and safe health

information technologies” (Kushniruk et al, 2013).

With this in mind, clinical studies are a good

opportunity to test usability and gather information

about the usage of a device. To our knowledge, no

studies have focused on usability studies conducted in

combination with clinical studies. This is precisely

Douze, L., Schiro, J. and Pelayo, S.

Usability Evaluation of Medical Devices during Clinical Studies: First Results of a Scoping Review.

DOI: 10.5220/0010385602930299

In Proceedings of the 14th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2021) - Volume 1: BIODEVICES, pages 293-299

ISBN: 978-989-758-490-9

293

the aim of this scoping review. The objective is to

identify the outcomes of clinical studies related to

usability evaluation and the methods used to collect

corresponding data. This paper presents the method

and first results of the study.

2 MATERIALS AND METHODS

We used the scoping review as the method for this

study. Our aim is to map the clinical studies protocols

of medical devices on usability evaluation. The

research question is as follows: How is usability of

medical devices evaluated in clinical studies?

2.1 Information Sources

The US National Library of Medicine database,

ClinicalTrials.gov, was searched. It is a well-known

database of privately and publicly funded clinical

studies conducted around the world.

2.2 Search Strategy

The search strategy was developed by two authors (JS

and LD). The general search terms were usability,

human factor, usage, use errors, satisfaction,

acceptability, acceptance, utility. Searches were

conducted between September 2020 and October

2020.

The following search string was used: (usability

OR satisfaction OR usage OR use errors OR

acceptability OR utility OR acceptance OR human

factors OR adherence OR adoption).

Any protocol about medical devices using

empirical methods of usability evaluation published

between January 2015 and October 2020 with the .pdf

protocol attached was considered. This means that the

following additional criteria were used as filters: only

Study Protocols as Study documents in the

ClinicalTrial.gov database, 01/01/2015 as Study start

date, and Medical device as Intervention/treatment.

2.3 Inclusion and Exclusion Criteria

The eligibility criteria were developed by two authors

(JS and LD). The usability definition provided by the

ISO 9241-11 was considered: “Extent to which a

product can be used by specified users to achieve

specified goals with effectiveness, efficiency and

satisfaction in a specified context of use.” All

protocols that included the collection of device usage

data to link device effectiveness and efficiency to its

intrinsic qualities were included in the analysis.

Objective data (e.g. use of a device, handling, ease of

use, safety of the procedure, adverse events,

successes) as well as subjective data (e.g. satisfaction,

perceived usability, barriers to adherence) were

considered for the analysis. The terms usage,

compliance or adherence if motivations were

collected, in terms of barriers for example, were

included in the analysis.

All in all, a protocol was included for analysis if

the following criteria were met:

 The protocol included evaluation of a medical

device or a combination product.

 The protocol concerned usability evaluation as

described in the outcomes of the protocol (e.g.

satisfaction, perceived usability, ease of use,

difficulties to use, handling, safety of the

procedure, utility).

A protocol was excluded of the analysis if the

following criteria were met:

 The protocol evaluated a product that was not a

medical device or a combination product (e.g.

a drug, a behaviour, a procedure).

 The protocol didn’t evaluated usability (e.g.

evaluate rather comfort, time spent for

procedure, clinical performance).

 The protocol focused only on the satisfaction of

a patient and/or his/her family while they were

not the end users (e.g. medical device used by

healthcare professionals, while the patients and

their families were the beneficiaries).

 The protocol concerned adherence to the

medical device without gathering information

about the motivations/reasons for the

adherence (or non-adherence) or acceptability.

 The protocol was not a clinical study (e.g.

authors claimed in the protocol that it was not

a clinical study, but a “classical” evaluation).

2.4 Search Results and Selection

Two authors (JS and LD) searched the

ClinicalTrials.gov database which yielded 883

protocols for possible inclusion in the scoping review

(Figure 1). In a first step, two of the authors (JS and

LD) independently screened 50 protocols on titles and

outcome measures.

Then, they pooled the results and discussed non-

agreements until consensus was reached. This first

step allowed a refinement of the eligibility criteria. In

a second step, the same two authors (JS and LD)

independently screened 300 other protocols on titles

ClinMed 2021 - Special Session on Dealing with the Change in European Regulations for Medical Devices

294

Figure 1: Flowchart of protocol selection.

and outcome measures. Then, they pooled the results

to assess the inter-rater reliability; the Cohen’s kappa

was 0.65 which indicated a strong agreement

(Krippendorff, 2013). The remaining protocols (534

protocols) were therefore screened by one of two

authors (JS and LD).

The screening of the protocols based on the titles

and the outcome measures led to 188 protocols to be

reviewed on their full text. A random selection of 47

protocols among the 188 was made for analysis.

Among these 47 protocols, 21 were excluded since

the studies did not met the eligibility criteria. Twenty-

six protocols were finally included for the next step.

2.5 Data Extraction and Categorisation

As for the selection of the protocols, the different

categories of information extracted from the

protocols were the result of an iterative and

collaborative work. In a first step, 5 protocols were

independently reviewed by two of the authors (JS and

LD) and then pooled in order to validate the

categories of information and their definition and to

refine the eligibility criteria. Then, in a second step,

19 protocols were reviewed by one of the authors (JS

or LD with respectively 9 and 10 protocols) to extract

the information. All the 19 protocols were cross-

checked to verify the extraction and completion of the

information. The remaining 23 protocols were

reviewed by one of the authors (JS or LD or SP).

General and specific information about the 26

protocols was extracted from the .pdf document

protocol and some metadata provided by the

ClinicalTrials.gov database in the study design. Table

1 lists all the extracted information along with their

definition.

Data extracted from each protocol was recorded

on an Excel computer worksheet in order to

categorise and compare characteristics. This study

was a scoping review with a focus on mapping the

clinical studies protocols of medical devices on

usability evaluation. As the objective was not to

collect the best available evidence, critical appraisal

of the selected articles was not performed.

Usability Evaluation of Medical Devices during Clinical Studies: First Results of a Scoping Review

295

Table 1: Information categories extracted during the analysis and their definition.

NCT ClinicalTrials.gov Identifier.

Title Title of the protocol.

Medical device

specification

Condition or disease

Disease, disorder, syndrome, illness, or injury that is being

studied. On ClinicalTrials.gov, conditions may also include

other health-related issues, such as lifespan, quality of life, and

health risks.

Device Medical device that is the focus of the clinical study.

End user considered in the

clinical study

Person that will use the medical device during the clinical study,

i.e. Patient and/or Healthcare professional or Other

Study design

Study type

Nature of the clinical study, includes interventional studies (also

called clinical trials), observational studies (including patient

registries), and expanded access.

Intervention model

Intervention model of the study includes one group assignment,

parallel assignment, crossover assignment or cohort.

Number of participants Number of participants that is planned to be recruited.

Post Market Surveillance Is the clinical study a post market surveillance study?

Usability

evaluation*

Category of primary

outcome measure related to

usability

Category of the planned outcome measure that is the most

important for evaluating the effect of the medical device, if

related to usability evaluation.

Category of secondary

outcome measure related to

usability

Category of the planned outcome measure that is not as

important as the primary outcome measure for evaluating the

effect of the medical device but that is still of interest, if related

to usability evaluation.

Type of usability-related

methods

Methods specified in the protocols to collect usability-related

outcomes.

* The values of these categories were established by the authors based on the information provided in the protocols.

3 RESULTS

3.1 Medical Device Specifications

Several types of medical devices are concerned by the

clinical studies including usability evaluations (Table

2), e.g. digital health devices (e.g. app to monitor

glucose, virtual reality systems, image guidance

system), biomaterials, orthoses, contact lenses,

therapeutic boot or shoe. The medical devices are

intended for both patients and healthcare

professionals.

3.2 Study Design

From our sample, the clinical studies incorporating

usability assessment that are reported in

ClinicalTrials.gov are essentially intervention studies

(Table 3), only one observational study has been

identified, a cohort study. These clinical studies are

either follow-up studies of the use of a medical device

with one group of participants, crossover studies or

comparative studies (2 groups). Only one post market

surveillance study has been identified. Sample sizes

are highly variable.

3.3 Usability Evaluation

Among the 26 analysed protocols, 5 protocols have

primary outcomes related to usability evaluation.

Figure 2 presents the different categories of outcomes

considered in each of the 26 protocols. Seventeen

protocols include one outcome related to usability

while 9 protocols include at least two usability-related

outcomes. The three most claimed outcomes (at least

8 protocols from our sample) correspond to the ease

of use of the medical device, its handling and the

satisfaction it provides when using it. Some studies

are interested in use errors or barriers to medical

device adherence (at least 3 protocols from our

sample). Other outcomes are also sometimes used,

such as willingness to use, acceptability, user-

friendliness, clarity of information or usefulness as

related to usability.

Figure 3 shows the different methods on which the

protocols are based on. The classical methods of the

usability field are used, e.g. shadowing,

questionnaires, interviews, user testing. The

questionnaire is the most widely used technique,

followed by the interview. Most of the protocols

(16/26) rely on one technique to collect usability-

ClinMed 2021 - Special Session on Dealing with the Change in European Regulations for Medical Devices

296

Table 2: Specifications of the medical device concerned by the clinical studies included in the analysis.

Condition or disease Device (NCT)

End user considered in

the clinical study

Diabetes

Sealed therapeutic shoe (NCT04085926) Patient

Offloading boot (NCT02783066) Patient

Diabetes app to assess diabetes control

(NCT03252964)

Patient

Continuous glucose monitor combined with an

activity tracker (NCT03165110)

Patient

Myopia, Astigmatism, visual

acuity

Contact lens (NCT03086447; NCT03024970;

NCT03006458; NCT03139578;

NCT03098745; NCT03707821;

NCT03679741)

Patient

Accidental falls Ankle Foot Orthoses (NCT02819011) Patient

Hearing Loss Successor hearing aid (NCT03086018) Patient

Amblyopia

Virtual reality based digital therapeutic that

applies therapeutic modifications in real-time to

cinematic content to rebalance visual input and

treat amblyopia (NCT03608150; )

Patient

Fecal Incontinence Anal tape (NCT02989545) Patient

Stroke

Smart Glove (home based virtual reality

biofeedback system) (NCT03559829)

Patient

Asthma

Propeller Health device + asthma navigator

(NCT03065205)

Healthcare professional

& Patient

Medication Adherence

Device for Dispensing Pain Medications in

Hospice Patients (NCT03940534)

Healthcare professional

& Patient

Feeding tube dysphagia

Enhanced enteral feeding device

(NCT03007511)

Healthcare professional

Hypothermia Neonatal Non-Electric Infant Warmer (NCT03031431) Healthcare professional

Spinal Diseases New image-guidance software (NCT03015142) Healthcare professional

Wounds and Injuries,

Lacerations, Surgical Incision

Polyurethane-based skin adhesive

(NCT03688880)

Healthcare professional

Coronary Artery Bypass

Grafting

Pliable and absorbable bone hemostats

(NCT03085017)

Healthcare professional

Aortic Valve Stenosis

Portico TF and ALT Delivery System

(NCT03056573)

Healthcare professional

Table 3: Study designs of the clinical studies included in the analysis.

Interventional model

Single group

assignment

Parallel

assignment

Crossover

assignment

Cohort

Interventional

study

PMS 1 / 1 /

No PMS 7 10 6 /

Observational

study

PMS / / / /

No PMS / / / 1

Number of participants:

Mean (SD)

Min-Max

95, 33 (122,04)

15-400

116, 4 (91, 34)

10-267

49, 42 (36, 41)

20-120

240 screw

placements (15

to 25 patients)

related data, mostly on questionnaires. Almost 40%

of protocols (10/26) combine several techniques,

quite often interviews and questionnaires, but also

shadowing and interview. The category Other refers

to techniques used once in one of the protocols; the

log analysis was used once, as the technique of the

diary study (the end user document in a journal some

Usability Evaluation of Medical Devices during Clinical Studies: First Results of a Scoping Review

297

Figure 2: Categories of outcome concerned by each of the

protocols.

elements related to the use of the device), or the

analysis of adverse events to identify use errors.

When crossing outcomes with methods (Table 4),

not surprisingly questionnaires are used to address

ease of use or satisfaction of the participants with the

medical device, but more surprisingly also to

collectinformation on the handling of the device

which is supposed to be more objective data.

Interestingly the shadowing technique is exclusively

used with healthcare professionals (Table 5).

Figure 3: Methods concerned by each of the protocols.

Table 5: Methods depending on the participants in clinical

studies included in the analysis.

Healthcare

professional

Patient Other

Shadowing 6 0 0

User testing 0 1 0

Interview 3 8 1

Questionnaire 7 26 0

Other 2 2 0

Table 4: Categories of outcomes and methods of clinical studies included in the analysis.

Shadowing

User

testing

Interview Questionnaire Other

Ease of use 5 0 5 7 2

Handling 1 1 3 11 2

Satisfaction 0 0 0 8 0

Use error 3 0 3 2 1

Barrier to adherence 0 0 3 3 1

Willingness to use 0 0 0 1 0

Usefulness/Utility 0 0 0 2 0

Acceptability 0 0 0 1 0

User friendly/User Experience 0 0 0 2 0

Clarity of information 0 0 0 1 0

ABCDEFGHI J

AEase of use

BHandling

CSatisfaction

DUse error

E Barriers to adherence

F Willingness to use

G Acceptability

H Userfriendliness/UX

I Clarity of information

J Usefulness/Utility

ABCDE

A Shadowing

B User testing

C I nterview

D Questionnaire

EOther

ClinMed 2021 - Special Session on Dealing with the Change in European Regulations for Medical Devices

298

4 DISCUSSION

The first results of this scoping review highlight that

a non-negligible part of usability evaluations is

carried out combined with clinical studies (or planned

to be carried out as only protocols have been

analysed). Very often, usability outcomes are part of

the secondary outcomes of the clinical study. The

most claimed usability-related outcomes are ease of

use, handling and satisfaction. Usability is mainly

addressed through questionnaires which provide

actually perceived usability of the medical device

instead of usability per se. While several protocols

appear to be quite comprehensive in terms of usability

evaluation methods, the vast majority of protocols

refer to notions close to that of usability, but not to

usability.

But this paper presents only some first results of

the scoping review and are maybe not representative

of the results out of the total of 188 protocols.

Moreover, not all usability studies conducted with

medical devices are necessarily reported on a

database such as ClinicalTirals.gov. But these first

results show the validity of the methodology and

some interesting trends.

REFERENCES

Krippendorff K. Content analysis. An introduction to is

methodology. Third. USA: Sage; 2013.

Kushniruk, A., Nohr, C., Jensen, S., Borycki, E. M. From

usability testing to clinical simulations: Bringing

context into the design and evaluation of usable and

safe health information technologies, Yearb. Med.

Inform. 22 (01) (2013) 78–85, http://dx.doi.org/

10.1055/s-0038-1638836.

Menotti, A., Puddu, P. E., Maiani, G., & Catasta, G. (2015).

Lifestyle behaviour and lifetime incidence of heart

diseases. International Journal of Cardiology, 201,

293_299.

The European Parliament and the Council of the European

Union, Regulation (EU) 2017/745 of the European

Parliament and of the Council of 5 april 2017 on

medical devices, 2017, https://eur-lex.europa.eu/legal-

content/EN/TXT/PDF/?uri=CELEX:32017R0745&fro

m=ENThe European Parliament and the Council of the

European Union, Directive 2007/47/EC of the

European Parliament and the Council of the European

Union of 5 september 2007 on medical devices, 2007,

https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?

uri=OJ:L:2007:247:0021:0055:en:PDF

World Health Organization. (2018). Medical device -

Full definition. http://www.who.int/medical_devices/

full_deffinition/en/. Retrieved September 16, 2018.

Weisburger, J. H. (2002). Lifestyle, health and disease

prevention: the underlying mechanisms. European

Journal of Cancer Prevention, 11, 1_7.

Wiklund, M. E., & Weinger, M. B. (2011). General

principles. In M. B. Weinger, M. E. Wiklund, & D. J.

Gardner-Bonneau (Eds.), Handbook of human factors

in medical device design. New York: CRC Press.

Usability Evaluation of Medical Devices during Clinical Studies: First Results of a Scoping Review

299