Is Usability Engineering Anticipation Possible during the

Initial Research Actions? An Example with the R-Link in

Vitro Self-monitoring Device

K. Charrière

, C. L. Azzopardi

, S. Pelayo

3,4 c

, M. Nicolier

1,5 d

, T. Lihoreau

1,4 e

F. Bellivier

6,7,8,9 f

, E. Haffen

1,5,10 g

and B. Wacogne

1,2 h

Centre Hospitalier Universitaire de Besançon, Centre d’Investigation Clinique, INSERM CIC 1431,

25030, Besançon Cedex, France

FEMTO-ST Institute, Université de Bourgogne Franche-Comté, CNRS, 15B Avenue des Montboucons,

25030 Besancon, Cedex, France

Centre Hospitalier Universitaire de Lille, Centre d’Investigation Clinique, INSERM CIC-IT 1403, F-59000 Lille, France

Tech4Health Network - FCRIN, France

Department of Clinical Psychiatry, CHU de Besançon, UR LINC 481 Laboratoire de Recherches Intégratives en

Neurosciences & psychologie Cognitive, University Bourgogne Franche-Comté, 25030, Besançon Cedex, France

AP-HP, GH Saint-Louis, Lariboisière, F. Widal, Department of Psychiatry and Addiction Medicine,

75475 Paris Cedex 10, France

Inserm, U1144, Paris, F-75006, France

Université Paris Descartes, UMR-S 1144, Paris, F-75006, France

Université Paris Diderot, Sorbonne Paris Cité, UMR-S 1144, Paris, F-75013, France

FondaMental Foundation, Creteil, Hôpital Albert Chenevier, Pôle Psychiatrie, 40 rue de Mesly, 94000 Créteil, France

{emmanuel.haffen, bruno.wacogne}@univ-fcomte.fr, frank.bellivier@inserm.fr

Keywords: Self-monitoring in Vitro Medical Device, Lithium, Usability, European in Vitro Medical Devices Regulation.

Abstract: Bipolar disorders are severe and complex psychiatric disorders and lithium remains one of the most effective

drugs for relapse prevention. Despite its effectiveness, prescription of lithium therapy can be complicated

because of its narrow therapeutic range. Furthermore, adherence to treatment is generally low. One means of

improving adherence would be to make the patient an actor of his/her treatment. The possibility to control the

lithium level with a device that can be used at home would favor this involvement. Although the main part of

the work to produce a device is research and development, regulatory analysis, including usability, should not

be neglected. Indeed, some design choices should be made taking into account usability constraints. This

ensure the fabrication of a device which will be safe, effective and well accepted by the intended users. In this

conference, we present actions taken in this direction during the R-Link project.

1 INTRODUCTION

The R-Link project, "Response to Lithium Network",

is a collaborative project funded by the European

Commission (Grant agreement n° 754907). It

proposes a clinical study involving people with

https://orcid.org/0000-0003-4542-8003

https://orcid.org/0000-0003-2147-2042

https://orcid.org/0000-0003-2830-2548

https://orcid.org/0000-0001-5135-5109

https://orcid.org/0000-0001-8417-6609

https://orcid.org/0000-0002-3660-6640

https://orcid.org/0000-0002-4091-518X

https://orcid.org/0000-0003-1490-5831

bipolar disorder type I when lithium treatment is

initiated (NCT04209140). The consortium includes

22 European partners among which research

institutes, hospitals, clinical investigation centers and

companies. It is led by Prof. Franck Bellivier

(Department of Psychiatry and Addiction Medicine -

Charrière, K., Azzopardi, C., Pelayo, S., Nicolier, M., Lihoreau, T., Bellivier, F., Haffen, E. and Wacogne, B.

Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring Device.

DOI: 10.5220/0010971600003123

In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - Volume 1: BIODEVICES, pages 259-269

ISBN: 978-989-758-552-4; ISSN: 2184-4305

259

Expert Centers University of Paris Diderot - INSERM

UMR-S144).

The goal is to identify early biomarkers that will

allow stratification of patients with bipolar I disorder

according to their Lithium (Li) response. This response

is being assessed prospectively over a two-year period

based on a thorough clinical assessment coupled with

measurements of blood omics, anatomical/structural

magnetic resonance imaging (MRI) and

Li MRI

derived markers. These markers will be tested as

predictors of response status at the end of the study.

Each patient will be involved in the study for two years.

Translation will be assessed in terms of positive and

negative predictive values of the markers, usefulness

of the markers when used alone or in combination,

patient acceptability, and cost-effectiveness. As it is

essential to monitor adherence to treatment, interactive

software for self-assessment of mental status will be

introduced and electronic reminders will be offered

throughout the study. A device that will allow self-

monitoring of salivary lithium levels at home will be

developed to be provided to patients. This last point is

the focus of this paper.

Indeed, the design and development of this device

raise some interesting questions related to the

compatibility between (i) the design choices of the

device and its usability and (ii) the regulatory

framework to be compliant with. The regulatory

analysis guides some design choices. In a context

where the device is still at the conceptual stage and its

design is progressing at the pace of the complex

regulatory analysis, can we already plan and conduct

a usability engineering process?

In this paper, we will present the different aspects

of usability engineering process on a general basis

and we will specify what was performed in the frame

of the R-Link project. Regulatory aspects must be

treated but will not be described in this

communication. After an introductory part on bipolar

disorders and the technical progress of the R-Link

device, we will detail the usability studies plan before

concluding.

2 BIPOLAR DISORDERS

Bipolar disorders are severe and complex psychiatric

disorders that affect approximately 45 million people

worldwide (James et al., 2018). In France, it is

estimated that between 1 and 2.5% of the population

is affected by these disorders, but it seems that these

figures are underestimated. It is one of the most

serious psychiatric pathologies, frequently leading to

suicide attempts: 50% of patients with bipolar

disorder will make at least one suicide attempt, and

15% will die (Troubles bipolaires, n.d.)

[not dated]. In

addition, bipolar disorder often leads to functional

impairment and reduced quality of life (Oldis et al.,

2016) and is associated with a decrease in lifespan of

approximately 10 years. The World Health

Organization has ranked this condition among the 10

most worrying of the 21

century (WHO | The Global

Burden of Disease, n.d.).

According to the DSM-5 (Diagnostic and

Statistical Manual of Mental Disorders-5th edition),

bipolar disorders can be classified into bipolar I

disorder, bipolar II disorder, cyclothymia and residual

categories. This sub-classification depends on the

severity and duration of manic (or hypomanic) and

depressive episodes (Vieta et al., 2018).

Bipolar disorder is recurrent, even when

diagnosed and treated. Various molecules are

available to treat bipolar disorders, among them are

mood stabilizing agents. Clinically, the main actions

that qualify a molecule as a mood stabilizer are its

effects at both ends of the mood spectrum (depression

and mania) and its ability to maintain euthymia by

preventing future mood instability. According to

these factors, lithium is the best and therefore the gold

standard mood stabilizing agent (Malhi et al., 2021).

According to the network meta-analysis by Miura

et al., lithium remains one of the most effective drugs

for relapse prevention and should remain the first-line

treatment (Miura et al., 2014).

Current recommendations call for a serum lithium

concentration between 0.6 mM and 0.8 mM for the

most effective treatment. In the acute manic phase,

concentrations can be increased to 1 mM, depending

on the patient's tolerance (Malhi et al., 2020). Despite

its effectiveness, lithium therapy can be complicated to

administer. Indeed, lithium can cause safety problems

due to its narrow therapeutic range. Below 0.5 mM

lithium, treatment may be ineffective and may lead to

relapse. Above 1.5 mM, there is a risk of toxicity. The

Li intoxication symptoms are variable and depend on

the intoxication severity. Nevertheless, if lithium levels

are correctly controlled, it seems that its long-term

toxicity may be limited (Malhi et al., 2020). According

to the practical guide of Malhi et al., follow-up should

be performed during the initial maintenance phase as

well as whenever there is a significant change in

therapy or when adverse effects occur (Malhi et al.,

2011, 2016).

Despite existing guidelines, many clinicians

remain reliant on an empirical "trial and error"

approach to effective lithium prescribing. Indeed, 18

to 24 months is often required to ensure a clinically

meaningful effect of lithium, with shorter-term

ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices

260

outcomes not reliably predicting prophylactic

outcomes. In addition to concerns about potential side

effects, this trial-and-error strategy likely leads to

increased non-adherence to treatment potentially

increasing the likelihood of treatment failure. For

example, only 30% of patients treated with lithium

show an excellent long-term response, most show a

partial response, and up to one-third do not respond

(Scott et al., 2018).

Furthermore, adherence to prescribed treatment is

generally low in most chronic illnesses including

bipolar disorder, with nonadherence as high as 50%

of most patients (Goodwin et al., 2016). The

possibility to strongly involve patients through

regular and home self-monitoring would be a

valuable help, probably allowing for increased

adherence to treatment but also for finer monitoring

of lithium levels. This is why a part of the R-Link

H2020 project aims to develop such a device.

3 THE SALIVARY LITHIUM

SELF-MONITORING DEVICE

The R-Link device aims to improve adherence to

treatment for patients with bipolar disorder type I,

prevent lithium overdose, prevent relapse into a

manic or depressive phase.

To achieve these goals, the idea is to help patients

to become active in their treatment - and more

particularly in its monitoring - by regularly

monitoring their salivary lithium levels.

Although there are still many uncertainties to be

resolved before an usable product is available for the

first pilot studies, the final configuration of the device

is already broadly defined (Figure 1). It will consist

of three distinct parts. Two parts will be single-use: a

system for collecting the patient's saliva (A) and a

"cartridge" containing the reactive zone and the

solutions necessary for the reaction (B). The third part

will be the device itself, i.e. the reusable apparatus (C)

allowing: (i) the driving of the solutions on the

dedicated reaction zone, (ii) the reading of the

reaction, (iii) the display and recording of the results.

Figure 1: Diagram of the 3 parts of the final device. A.

saliva collection system, B. cartridge with reagent area and

C. reader-actuator for performing, reading and interpreting

the reaction.

4 USABILITY STUDIES FOR

MEDICAL DEVICES

As mentioned above, the prototype is not yet

available but some technical solutions have already

been defined and technical validation tests are

currently underway. It is therefore possible - and

necessary to meet the time constraints set by the

H2020 project - to move forward in parallel on certain

tasks, including the implementation of a usability

plan.

Usability is an integral part of the MDR/IVD, in

particular point 19, chapter II of Annex VIII

concerning “protection against risks arising from

devices intended for self-diagnosis or diagnosis near

the patient […]”. So, usability engineering process

aims to improve the safety of use of the device and

ultimately the safety of the patients as end-users by

reducing the risks associated with errors in use during

normal use of the medical device. Usability studies

have to be mobilized to anticipate the risks of

abnormal use, in order to avoid, as much as possible,

the associated errors. The process should be

documented in the usability studies file for obtaining

CE marking.

Usability is defined by the 62366-1 standard (NF

EN 62366-1/A1 - Août 2020, n.d.) as "the

characteristic of the user interface that facilitates use

and thus establishes the effectiveness, performance

and satisfaction of the user in the intended use

environment". The usability engineering process is a

risk management process focused on potential use

errors. This usability process is closely intertwined

with the standard 14971 for the application of risk

management to MD (Medical Device) (NF EN ISO

14971 - Décembre 2019, n.d.).

The usability engineering process is an iterative

process that applies to all stages of the MD life cycle

and for all users. It concerns, of course, the use of the

device itself with the user interface, but also the

accompanying documentation and the delivered

training. It must take into account the end users

(patients and non-medical caregivers) and the

secondary users such as the medical staff who will be

responsible for training in the use of the device or the

staff who will have to manufacture, package, store,

maintain, recycle or dispose of the device. We have

related here only the end users: patients and non-

medical caregivers.

Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring

Device

261

Figure 2: Schematic of the usability plan for the R-Link device.

4.1 Use Specifications

Establishment part of the use and functional

specifications was done during the functional analysis

(Charrière et al., 2021). However, the usage

specifications do not only include the required

functions of the final device but must also establish

the characteristics of the environment in which the

device will be used, as well as the characteristics of

the users, considering both the physical and cognitive

characteristics of the primary and secondary users.

The main steps of the proposed usability

engineering plan for the R-Link device are

summarized in Figure 2. They consist in establishing

first, the usability specifications and second, the

functional specifications (Figure 2, points 1 and 2).

The usability-related safety characteristics must be

then established accordingly and will complete the

technical risk analysis made by the manufacturer

(Figure 2, point 2). On this basis, the dangerous

situations and the different scenarios arising from

them can be identified to guide the MD design. Future

assessments can then be planned to test to what extent

the design of the device prevents that use errors occur

(Figure 2, points 3 and 4). The evaluation plan for the

user interface (Figure 2, point 5) should be

established integrating formative evaluations (Figure

2, point 6). It may be necessary to run iterative

evaluations with several models or demonstrators

(Figure 2, points 6 and 7), before reaching a system

satisfactory for conducting summative evaluation(s)

(Figure 2, point 9).

4.1.1 Intended Use Environment

The device is intended to be used at the patient's

home, by the patient himself or by non-professional

caregivers. Environmental characteristics are

therefore likely to vary according to location,

especially countries. For example, the first models of

the R-Link device will have to be connected into the

mains. In France, the voltage is 220 V, whereas it is

110 V in the United States.

The patient could be away from home at the time

of the test. Ideally, the device should be easily

transportable and usable in mobile conditions. It will

therefore be important to provide an appropriate

device size and weight.

The appropriate luminous flux to illuminate a

space varies according to the room. Recommended

levels can be found in NF EN 12464-1 standard

"indoor lighting for workplaces" (NF EN 12464-1 -

Juillet 2011, n.d.). It is desirable that the result can be

read from 20 cm to 50 cm under appropriate light

conditions.

Since the device is intended for home use, the

temperature can be varied in the range of 14°C to

35°C. However, previous summer heat waves should

be taken into account. If this is not the case, the

manufacturer will ensure that this risk is controlled by

clearly indicating it in the instructions or by adding an

internal control to the device.

Based on the reagent cost, the estimated

production costs after industrialization and, above all,

the recommendations of the project's partner

physicians, patients will be encouraged to perform a

test every 15 days. This frequency could be adapted

throughout the project duration.

1. Prepare use specification

Conduct meetings and brainstorming with medi cal staff

and manufacturer

6. Establish user interface specification

7. Design and implement the user interface - 8. Perform formative evaluations

Conceptualize Implement Evaluate

Develop user

interface design

concepts

Models with

instructions for use

Formative evaluation:

face-to-face interview

Ite rati on 1

Refine user

interface design

Hight fidelity

demonstrators +

instructions for use

Formative evaluation:

usability test

Itera ti on 2

9. Perform summative evaluation

Usability test Evaluate residual risks related to usability

3. Identify hazards, hazardous situations and hazard-

related scenarios

4. Select the hazard-related use scenarios for

evaluations

2. Identify characteristics for safety

Ta s k a n a l y s i s

Functional

specifications

Review publicly

available

databases

Conduct meetings, brainstormings with partners

(medical, research, manufacturer)

Conduct meetings and brainstorming with medical staff

and manufacturer

5. Establish user interface evaluation plan

Refine user interface

design

Refined models and

instructions for use

Formative evaluation:

face-to-face interview

Iteration 2+X (if needed)

Refine user interface

design

Refined models and

instructions for use

Formative evaluation:

face-to-face interview

Iteration 1+X (if needed)

ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices

262

The description of the technical environment of

the device cannot yet be finalized at this stage of the

project. However, some characteristics can now be

specified: hardware configuration such as processor

speed, memory size, network, storage, input and

output devices; screen type and size, resolution and

color depth; whether or not the visual interface

elements (such as text or symbols) can vary in size

(and size(s) available); configuration of the electronic

board; assistive technologies available if required.

4.1.2 Target Users

User characteristics (functional, physical, sensory

and cognitive capabilities, experience, knowledge

levels and behaviors) could impact the safe and

effective use of the device.

For example, elderly people may have reduced

visual acuity or polyarthritis problems. A small text

on a screen or a too complicated handling of the

device will most likely lead to user errors. Since the

ultimate goal is to eliminate sources of error related

to perception, cognition or handling as much as

possible, it is important to correctly identify the

primary users (i.e. the person who will use the device

in its actual medical use) and the secondary users (i.e.

all persons who may have the device in their hands

during its life cycle, from manufacture to disposal).

In the case of the R-Link device, the primary users

(see Table 1) of the device are patients with bipolar

disorder type I. Bipolar disorder affects both men and

women, regardless of social class or location. The

illness can occur throughout the lifespan, from the age

of 15 to over 60. If patients are unable to use the

device due to physical or cognitive impairments,

caregivers may do it for them and then become the

primary users. Bipolar disorder causes comorbidity

that can lead to impairments, and patients (or

caregivers) may have age-related physical and

cognitive impairments, such as loss of vision,

hearing, dexterity, etc. Patient and non-professional

caregiver categories for the device should include:

adults (18-49 years old), seniors (50-64 years old),

and the elderly (65 years old and older).

4.2 User-centered Safety Features

Risk analysis is often understood as an analysis of

technical risks like electrical, thermal or biological

risks. They are related to a failure of the device or of

a component, and therefore do not depend on the way

the device is used, i.e. on the interaction between user

and interface.

However, some risks are directly related to this

interface/user interaction and can be the result of user

interface design problems. For example, the result is

not clearly readable or difficult to interpret, resulting

in a more or less serious damage (Health, 2019).

Therefore, the risk analysis - and the entire risk

Table 1: Primary user characteristics.

Patients

Demographic

characteristics

Age range: 18 years and above. Education and literacy: all levels of education, including

illiterate. All types of socio-economic, ethnic, cultural status. Language: French.

Physical

characteristics,

potential

disabilities

Many women have long nails. Patients may have xerostomia and other co-morbidities

(alcohol and drug use, panic disorders, obsessive-compulsive disorders, eating disorders,

personality disorders, overweight and obesity, diabetes, cardiovascular disease). The

majority of patients have no physical disorders. Some patients - particularly those over 40

years of age - may have age-related vision problems (such as presbyopia). Some patients -

particularly those aged 50 and over - may have progressive hearing loss and/or dexterity

and strength limiting disorders such as tremors, arthritis...Some patients may have native

disorders such as visual impairment, hearing impairment or physical disability. Some

patients may have cognitive impairments in executive function, learning and verbal

memory. With advancing age, patients may have much greater impairments in information

processing.

Competence

Patients are generally not proficient in the use of medical devices. Some patients -

particularly older ones - may not be comfortable using the device, as they may be

latecomers to computer technology.

Type of

learning

Unknown. To be determined in formative and summative evaluations. There is a strong

preference among partners for learning in a consultation, delivered by the doctor and/or

trained medical staff.

Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring

Device

263

management plan - must also include the risks

associated with the use of the device throughout its

life cycle. It is therefore necessary to be able to

identify the hazards, estimate and quantify the

associated risks, control them and be able to monitor

the effectiveness of these measures (NF EN ISO

14971 - Décembre 2019, n.d.).

Here, the analysis is focused on the risks related

to the use of the device. The analysis of technical

risks, resulting from a failure of the device, will not

be dealt with. Use errors analysis is difficult to carried

out when the technical solutions are not yet known

and when the development of the device is not

advanced, which is the case for the R-Link device.

Some of main trends are already decided in terms of

design: a saliva sample is inserted in the system

manually or automatically, a chemical reaction takes

place, the result is read by an analyzer and delivered

to the patient who must interpret it and react

accordingly.

There are analytical approaches for identifying

hazard-related tasks or scenarios. Such an approach is

based on the task analysis method, which breaks

down the process of using the device into discrete

sequences of tasks. This analysis has been applied to

the R-Link device.

To perform the salivary lithium level self-test, all

parts of the R-Link device are required: the reader, a

cartridge and a saliva collector (Figure 1). The

cartridge and saliva sampler are independent of the

reader. Five major steps have been identified for

performing salivary lithium self-testing with the R-

Link device: (i) collect saliva using a saliva sampler,

(ii) insert the saliva sample in the designated area, (iii)

insert the cartridge into the R-Link reader, (iv) after a

few minutes, the result appears on the screen and (v)

the patient reads and interprets the result.

For each of these tasks, a questioning based on the

WWWWHW model (Who, What, Where, When,

How, Why) is performed. Based on this questioning,

we identified anticipated subtasks that will be

performed by the patient, with the exception of the

automated tasks.

Based on these identified subtasks, the user risk

analysis can start relying on a Failure Mode and

Effects Analysis (FMEA) method. It is used to

identify all the hazards and harms associated with the

use of the device according to its characteristics and

its intended use. In order to conduct this analysis in

the best way, all project partners (clinicians,

researchers and manufacturers) must be involved. For

each of the previously defined subtasks, it is

determined whether or not a hazard can be associated

with. This hazard may lead - either on its own or as a

result of a sequence of events - to a dangerous

situation that will result in damage for the user. The

risk level is then assessed according to the probability

and severity of the damage.

If the risk level is high, risk control measures must

be put in place to ensure that the residual risk is

acceptable. At the research and development step, a

certain number of control methods could be

suggested. The final choice of the control method will

be made considering the adequacy between use added

value and production costs.

For the R-Link device, several types of damage

have been identified. The most serious is an erroneous

chemical reaction leading to a false result, namely an

over- or under-estimation of the lithium level. In both

cases, the damage is severe.

In case of overestimation of the lithium level by

the device, the patient might actually be beyond the

zone for which no toxicity is to be feared.

Nevertheless, this risk is to be put in comparison with

the patient's feeling. Indeed, lithium overdoses are

often well estimated by the patient who then

immediately contacts his doctor.

In case of lithium level underestimation, the

patient would probably not be aware of it and would

risk a relapse - either into a manic state or into a

depressive state. It is precisely these cases that the R-

Link device targets in priority. Thus, in both cases,

the damage to the patient could be significant and

countermeasures must be taken to reduce it.

Several causes could be at the origin of this bad

estimation: too high temperature, expired

consumable, bad salivary sampling, bad reading and

bad interpretation of the result delivered by the

device. To reduce these risks, several control methods

are suggested: designing the device with a

thermostatic chamber, or at least incorporating a

temperature controller; designing the device with an

integrated expiration date controller; training end

users in saliva sampling and deliver clear instructions

for use; making sure that the result is clearly

displayed.

Other non-critical errors of use have been

identified. For example, if the patient does not

connect the device properly to the power source, the

test cannot be performed. Nevertheless, this problem

should be rare and will not cause any direct damage

since the test cannot be performed. It should also be

easily controlled by learning how to use the device

and a clear instruction manual.

ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices

264

4.2.1 Review of Public Databases

A review of available databases was also conducted

to identify known use errors with similar devices:

MAUDE (Manufacturer and User Facility Device

Experience), Web of Science, PUBMED. Only one

search carried out with the key words "self-test

lithium" on google gave interesting results (Self Test

Lithium - Google Search, n.d.). The first comes from

the Dutch company FISIC: the Medimate Multireader

(Fisic | Lithium Self Test, n.d.). The second comes

from ReliaLAB, an American company: the Instaread

lithium system (Finger-Stick Lithium Test, n.d.).

For the Instaread lithium system an adverse

reaction report exist. . This report mentions that the

results obtained with the Instaread lithium system can

differ of up to 0.5 mM compared with the results

obtained during a laboratory test. (INSTAREAD

LITHIUM SYSTEM * Adverse Event MAUDE, n.d.).

Finally, the 510k data sheet for this MD/IVD is

available, but it only enumerates device performance

data (510(k) Premarket Notification, n.d.). No data

regarding usability was found.

More documentation is available from the second

MD/IVD, the Medimate Multireader from the

company FISIC (Fisic | Documentation, n.d.). This

one is not FDA approved but is EC labelled according

to the European Directive for IVDs (98/79/EC). In a

study, authors aim to evaluate the usability of the

Medimate Multireader when used by the patient for

self-testing at home, or when used in a health care

facility for point-of-care testing. Healthcare workers

(for point-of-care testing) and patients (for home

testing) completed a System Usability Scale (SUS)

questionnaire. The SUS is a validated method to

quickly assess the perceived usability of a system and

consists of 10 items covering different aspects such

as complexity, ease of learning, frequency of use

(Affairs, 2013; Bangor et al., 2008). Based on this

scale, authors concluded that the usability of their

device is "good", even if the blood collection was

considered unpleasant and/or difficult in terms of

sampled volumes.

The analysis of the competing devices is a key

point, which allows to anticipate the requirements

expected for similar devices. Thus, the studies for the

design and then the validation of the R-Link device -

similar in its specification of use to the Medimate

Multireader and Instaread lithium system - could be

inspired by this already compliant competition for a

diffusion on the European market or for the American

market. For the Instaread lithium system, the 510k

data sheet of the system could be a source of

inspiration for the performance validations of our

MD/IVD as well as the instructions for use (complete

and abbreviated), the study designs used and the

various articles published in peer-reviewed journals

from the company FISIC (Floris et al., 2010; Muñoz

et al., 2011; Nieuwe Mogelijkheden Voor Een

Lithiummeting Op de Poli En in de Huiskamer, 2019;

Staal et al., 2015).

4.3 Formative and Summative

Evaluations

Although the R-Link device is at a very early

development stage, it is possible to anticipate future

evaluations. In addition to the 62366-1 and 2 standard

(IEC/TR 62366-2:2016 - Avril 2016, n.d.; NF EN

62366-1/A1 - Août 2020, n.d., pp. 62344–2), the FDA

guide for manufacturers and their staff is freely

available and is a good support to design the plan of

the different usability evaluations of a device (Health,

2019). Usability evaluations can be classified into

two categories depending on the objective: formative

and summative evaluations.

4.3.1 Formative Evaluations

Formative evaluations should help in the design of the

MD during its development and focus primarily on

points that could jeopardize the safety of use

identified during the risk analysis and on undefined

design options. They should complement the

preliminary analyses (task analyses, risk analyses)

and reveal previously unidentified errors in use. Thus,

formative evaluations should be performed

throughout the development process, depending on

the amount of information needed for the design, the

complexity of the device and its use, the variability of

the user population or the conditions of use. They can

be done with very simple mockups, even drawings, or

with very advanced prototypes (Health, 2019).

Standard 62366-2 recommends several types of

methodologies for conducting these formative

evaluations, including face-to-face interviews,

cognitive walkthroughs, and/or usability tests. For

face-to-face interviews to be productive, the

objectives must be established beforehand and an

interview guide defined. This guide should not

present closed questions but include short, open-

ended, organized questions around topics of

discussion. In the cognitive walk, a very preliminary

design - which may be in the form of drawings - is

presented to a small group of people. A session

involves a single participant who must imagine

his/her reactions to the MD and verbalize all his/her

thoughts and actions. Usability tests are conducted

Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring

Device

265

Table 2: Example of summary sheet; task "Insert the sample in the slot provided in the cartridge".

Hazardous

event

Description of the use scenario related to the hazardous

phenomenon

Associated

damage(s)

Hazardous

situation

Wrong

test result

The system for transferring saliva from the collection tube to the

cassette has not yet been determined. The user has difficulties in

transferring saliva from one container to another. The user does

not insert a sufficient volume into the cassette and/or causes

numerous bubbles in the reaction area. The chemical reaction does

not take place correctly, leading to an over- or underestimation of

the lithium level.

Anxiety, relapse

or risk of toxicity

Use of the saliva

collection device

is difficult for the

user.

Formative evaluation(s) - "Sample tube / cartridge / leaflet" interface

Face-to-face interviews

Objective: To assess the understanding of the instructions in relation to the use of the system and the clarity

of the training.

Method: Face-to-face interviews with an interview grid focused on the understanding of the instructions and

the instructions given by the trainer.

Presentation of a low definition model, then high definition, allowing the sample to be placed in the

cassette, with the associated instructions. Collect opinions on the clarity of instructions. Explanation of the

use of the device. Collect opinions on the clarity of the use of the device after explanation.

Data collection: audio recording and note taking. Analysis: Qualitative analysis of verbatims.

Population: Nursing staff doctors + nurses + clinical research officer.

Note: Refine the design according to the results and repeat the evaluation until the device for depositing the

sample in the intended location in the collection cassette is satisfactory. Conduct the usability test when this

stage is reached.

Usability test

Objectives: To assess the number of usability errors and to identify the causes. To assess the number of non-

compliant deposits of the sample into the cassette. To assess the understanding of the training.

Method: Usability test with video recording, interview and questionnaire. 1 session per participant.

Population: Patients with bipolar disorder type I, 3 age groups (18-24, 25-62, over 62), 1 male and 1

female/group. Non-medical carers, 3 age groups (18-24, 25-62, over 62), 1 male and 1 female/group.

Course of the session: Presentation of the device allowing the sample to be placed in the location provided

in the cassette selected following the initial evaluations, with the associated instructions. Explanation of the

use by the trainer, as in a real situation. Immediately afterwards, the user will carry out all the tasks

requested, following only the instructions, without any external help. The session will be filmed to allow

analysis (number of hesitations during sampling, number of times the instructions are consulted).

Immediately after the collection, the volume of saliva deposited in its place will be recorded in the

observation book, as well as the presence or absence of bubbles/foam. Proposal of the SUS questionnaire

with an interview targeted on the difficulties of use encountered, including the understanding of the

instructions given.

Data collection: Video recording + observation booklet + questionnaires + note taking.

Data analysis: Quantitative analysis of the number of errors, hesitation/consultation of the instructions, non-

compliant deposits + analysis of SUS + qualitative analysis of verbatims.

with a few users who have to complete some tasks

representing the important functions of the future MD

(IEC/TR 62366-2:2016 - Avril 2016, n.d.).

For the R-Link device, the risk analysis reveals

four tasks for which the risk of use errors leading to

damage is significant: (i) saliva collection, (ii)

insertion into the cartridge, (iii) reading the result, and

(iv) interpreting the results. The formative

evaluations should ensure that the design chosen for

the parts of the device supporting these tasks

effectively eliminates or limits any risk associated

with misuse. It is performed in an iterative way and

the first steps could be done with experts instead of

end users. For each of the four domains mentioned,

two types of formative evaluations are retained: a

face-to-face interview with hospital staff (experts)

and a usability test with patients. A summary sheet

for each of these tasks was designed (Table 2). These

sheets, as the whole file, are not fixed yet and may

evolve according to the progress and design choices

of the project.

ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices

266

4.3.2 Summative Evaluations

The summative evaluation is always the very last step

of the fitness-for-use engineering process. It must

demonstrate that the MD can be used under the

specified conditions of use, by the intended users and

without unacceptable residual risk: it is therefore the

validation step of the device in terms of safety risks

related to use. The summative evaluation must

implement the scenarios relating to the previously

defined dangerous phenomena, under conditions as

close as possible to reality, but without a clinical

effect. Thus, for the summative evaluation to be valid,

it is important to ensure that the participants represent

all the intended users, that all critical tasks are

performed during the test, that the user interface

represents the final design, and that the test conditions

correspond to the real conditions of use.

As with a traditional clinical investigation, a

rigorous protocol must be established, including the

introduction, the objectives of the test and the method

used, the description of the MD, the necessary

equipment and environment, the description of the

participants and the personnel involved, the list of

tasks to be carried out, the methods of data collection

and analysis, an operating procedure for the test and,

if necessary, a description of the training.

5 CONCLUSION

The objective of this work was to give indications to

the R&D team concerning the regulatory constraints

likely to influence the design and to initiate the

engineering suitability plan. Thus, although many

questions remain today, this very early participation

has already allowed and will subsequently allow the

technical team to orientate itself towards what we

hope will be a high-performance, reliable and safe

product.

Thus, carrying out a usability plan at a very early

stage of design is entirely possible and even desirable

because the analyses carried out make it possible to

feed the design and orientate the choices by

identifying the needs of the users and the constraints

of the usage environments. The specifications for use

will thus be issued accordingly, making it possible to

prevent the risks associated with the use of the DM.

However, the plan cannot be fixed at this stage. It will

have to be adapted as the design, technical choices

and the results of the various analyses and formative

evaluations carried out during the project progress.

This work highlight the importance of the usability

aspect from the very beginning of a project. As we

discussed in a previous paper (Charrière et al., 2021),

the development process of a MD should not be

envisaged in a linear way, with a separation between

partners. A dynamic vision has to be adopted, because

the choice of technical solutions or specialities on

which research and design efforts should be made

depends on several factors such as: acceptability to

end-users, risk analysis, technical feasibility,

production cost, regulatory constraints and the return

on investment that the manufacturer can expect.

ACKNOWLEDGEMENTS

R-LiNK has received funding from The European

Union’s Horizon 2020 Research and Innovation

Program Under Grant Agreement N° 754907.

Many thanks to the teaching team of the

Universitary diploma “Methodologies in clinical

evaluation of medical devices” leaded by the

TECH4HEALTH network.

REFERENCES

510(k) Premarket Notification. (n.d.). Retrieved 3 June

2021, from https://www.accessdata.fda.gov/scripts/

cdrh/cfdocs/cfpmn/pmn.cfm?ID=K031579

Affairs, A. S. for P. (2013, September 6). System Usability

Scale (SUS). Department of Health and Human

Services. system-usability-scale.html

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An

Empirical Evaluation of the System Usability Scale.

International Journal of Human–Computer Interaction,

24(6), 574–594. https://doi.org/10.1080/1044731080

2205776

Charrière, K., Azzopardi, C., Nicolier, M., Lihoreau, T.,

Bellivier, F., Haffen, E., & Wacogne, B. (2021).

Functional Analysis to Drive Research and Identify

Regulation Requirements: An Example with a Lithium

Monitoring Device: Proceedings of the 14th

International Joint Conference on Biomedical

Engineering Systems and Technologies, 300–307.

https://doi.org/10.5220/0010382503000307

Finger-stick lithium test. (n.d.). Retrieved 2 June 2021,

from https://www.mdedge.com/psychiatry/article/

61028/finger-stick-lithium-test

Fisic | Documentation. (n.d.). Retrieved 2 June 2021, from

https://www.medimate.com/medimate/en/product/doc

umentation

Fisic | Lithium self test. (n.d.). Retrieved 26 May 2021, from

https://www.medimate.com/medimate/en/bipolar/selft

est

Floris, A., Staal, S., Lenk, S., Staijen, E., Kohlheyer, D.,

Eijkel, J., & Berg, A. van den. (2010). A prefilled,

ready-to-use electrophoresis based lab-on-a-chip

Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring

Device

267

device for monitoring lithium in blood. Lab on a Chip,

10(14), 1799–1806. https://doi.org/10.1039/C003899G

Goodwin, G. M., Haddad, P. M., Ferrier, I. N., Aronson, J.

K., Barnes, T. R. H., Cipriani, A., Coghill, D. R., Fazel,

S., Geddes, J. R., Grunze, H., Holmes, E. A., Howes,

O., Hudson, S., Hunt, N., Jones, I., Macmillan, I. C.,

McAllister-Williams, H., Miklowitz, D. M., Morriss, R.,

… Young, A. H. (2016). Evidence-based guidelines for

treating bipolar disorder: Revised third edition

Recommendations from the British Association for

Psychopharmacology. Journal of Psychopharmacology

(Oxford, England), 30(6), 495–553. https://doi.org/

10.1177/0269881116636545

Health, C. for D. and R. (2019, September 2). Applying

Human Factors and Usability Engineering to Medical

Devices. U.S. Food and Drug Administration; FDA.

https://www.fda.gov/regulatory-information/search-

fda-guidance-documents/applying-human-factors-and-

usability-engineering-medical-devices

IEC/TR 62366-2:2016—Avril 2016. (n.d.). Retrieved 21

May 2021, from https://www.boutique.afnor.org/

norme/iec-tr-62366-22016/-/article/864646/xs127839

INSTAREAD LITHIUM SYSTEM * Adverse Event MAUDE.

(n.d.). Retrieved 2 June 2021, from

https://fda.report/MAUDE/MDR/672100

James, S. L., Abate, D., Abate, K. H., Abay, S. M., Abbafati,

C., Abbasi, N., Abbastabar, H., Abd-Allah, F., Abdela,

J., Abdelalim, A., Abdollahpour, I., Abdulkader, R. S.,

Abebe, Z., Abera, S. F., Abil, O. Z., Abraha, H. N.,

Abu-Raddad, L. J., Abu-Rmeileh, N. M. E.,

Accrombessi, M. M. K., … Murray, C. J. L. (2018).

Global, regional, and national incidence, prevalence,

and years lived with disability for 354 diseases and

injuries for 195 countries and territories, 1990–2017: A

systematic analysis for the Global Burden of Disease

Study 2017. The Lancet, 392(10159), 1789–1858.

https://doi.org/10.1016/S0140-6736(18)32279-7

Malhi, G. S., Bell, E., Bassett, D., Boyce, P., Bryant, R.,

Hazell, P., Hopwood, M., Lyndon, B., Mulder, R.,

Porter, R., Singh, A. B., & Murray, G. (2021). The 2020

Royal Australian and New Zealand College of

Psychiatrists clinical practice guidelines for mood

disorders. The Australian and New Zealand Journal of

Psychiatry, 55(1), 7–117. https://doi.org/10.1177/0004

867420979353

Malhi, G. S., Bell, E., Outhred, T., & Berk, M. (2020).

Lithium therapy and its interactions. Australian

Prescriber, 43(3), 91–93. https://doi.org/10.18773/

austprescr.2020.024

Malhi, G. S., Gershon, S., & Outhred, T. (2016).

Lithiumeter: Version 2.0. Bipolar Disorders, 18(8),

631–641. https://doi.org/10.1111/bdi.12455

Malhi, G. S., Tanious, M., & Gershon, S. (2011). The

lithiumeter: A measured approach. Bipolar Disorders,

13(3), 219–226. https://doi.org/10.1111/j.1399-

5618.2011.00918.x

Miura, T., Noma, H., Furukawa, T. A., Mitsuyasu, H.,

Tanaka, S., Stockton, S., Salanti, G., Motomura, K.,

Shimano-Katsuki, S., Leucht, S., Cipriani, A., Geddes,

J. R., & Kanba, S. (2014). Comparative efficacy and

tolerability of pharmacological treatments in the

maintenance treatment of bipolar disorder: A

systematic review and network meta-analysis. The

Lancet Psychiatry, 1(5), 351–359. https://doi.org/

10.1016/S2215-0366(14)70314-1

Muñoz, M., Eijkel, J., Floris, A., Staal, S. s, Ríos, Á., & Van

den Berg, A. (2011). The development of a point of care

creatinine measurement using disposable ready to use

microchip capillary electrophoresis. Journal of The

Electrochemical Society - J ELECTROCHEM SOC, 2.

NF EN 12464-1—Juillet 2011. (n.d.). Retrieved 21 May

2021, from https://www.boutique.afnor.org/norme/nf-

en-12464-1/lumiere-et-eclairage-eclairage-des-lieux-

de-travail-partie-1-lieux-de-travail-

interieurs/article/701391/fa158788

NF EN 62366-1/A1—Août 2020. (n.d.). Retrieved 21 May

2021, from https://www.boutique.afnor.org/norme/nf-

en-62366-1-a1/dispositifs-medicaux-partie-1-applicati

on-de-l-ingenierie-de-l-aptitude-a-l-utilisation-aux-dis

positifs-medicaux/article/922059/fa194085

NF EN ISO 14971—Décembre 2019. (n.d.). Retrieved 21

May 2021, from https://www.boutique.afnor.org/

norme/nf-en-iso-14971/dispositifs-medicaux-

application-de-la-gestion-des-risques-aux-dispositifs-

medicaux/article/918584/fa190418

Nieuwe mogelijkheden voor een lithiummeting op de poli en

in de huiskamer. (2019, June 13). Psyfar.

http://www.psyfar.nl/tijdschrift/editie/artikel/t/nieuwe-

mogelijkheden-voor-een-lithiummeting-op-de-poli-en-

in-de-huiskamer

Oldis, M., Murray, G., Macneil, C. A., Hasty, M. K., Daglas,

R., Berk, M., Conus, P., & Cotton, S. M. (2016).

Trajectory and predictors of quality of life in first

episode psychotic mania. Journal of Affective

Disorders, 195, 148–155. https://doi.org/10.1016/

j.jad.2016.02.018

Scott, J., Etain, B., & Bellivier, F. (2018). Can an Integrated

Science Approach to Precision Medicine Research

Improve Lithium Treatment in Bipolar Disorders?

Frontiers in Psychiatry, 9, 360. https://doi.org/10.3389/

fpsyt.2018.00360

self test lithium—Google Search. (n.d.). Retrieved 2 June

2021, from https://www.google.com/search?q=self+

test+lithium&hl=en&biw=1525&bih=699&sxsrf=ALe

Kk00VBPQf4Bo8ckOE4s2PV8S7AarP0Q%3A16226

25871402&ei=T063YM7aF6LuxgPH0oPgAg&oq=sel

f+test+lithium&gs_lcp=Cgdnd3Mtd2l6EAMyBAgjEC

cyBAgjECc6BwgjELECECc6BggAEAcQHjoHCCM

QsAIQJzoICAAQCBAHEB46CAgAEAgQDRAeUP

wvWOlEYNpJaAJwAHgAgAF4iAGxCJIBAzcuNJgB

AKABAaoBB2d3cy13aXrAAQE&sclient=gws-

wiz&ved=0ahUKEwiOvqu10PjwAhUit3EKHUfpAC

wQ4dUDCA0&uact=5

Staal, S., Ungerer, M., Floris, A., Brinke, H.-W. T.,

Helmhout, R., Tellegen, M., Janssen, K., Karstens, E.,

Arragon, C. van, Lenk, S., Staijen, E., Bartholomew, J.,

Krabbe, H., Movig, K., Dubský, P., Berg, A. van den,

& Eijkel, J. (2015). A versatile electrophoresis-based

self-test platform. ELECTROPHORESIS, 36(5), 712–

721. https://doi.org/10.1002/elps.201400428

ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices

268

Troubles bipolaires: Diagnostiquer plus tôt pour réduire le

risque suicidaire. (n.d.). Haute Autorité de Santé.

Retrieved 10 May 2021, from https://www.has-

sante.fr/jcms/c_2560925/fr/troubles-bipolaires-

diagnostiquer-plus-tot-pour-reduire-le-risque-

suicidaire

Vieta, E., Berk, M., Schulze, T. G., Carvalho, A. F., Suppes,

T., Calabrese, J. R., Gao, K., Miskowiak, K. W., &

Grande, I. (2018). Bipolar disorders. Nature Reviews

Disease Primers, 4(1), 1–16. https://doi.org/10.1038/

nrdp.2018.8

WHO | The global burden of disease: 2004 update. (n.d.).

WHO; World Health Organization. Retrieved 10 May

2021, from https://www.who.int/healthinfo/

global_burden_disease/2004_report_update/en/

Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring

Device

269