Is Usability Engineering Anticipation Possible during the
Initial Research Actions? An Example with the R-Link in
Vitro Self-monitoring Device
K. Charrière
, C. L. Azzopardi
, S. Pelayo
3,4 c
, M. Nicolier
1,5 d
, T. Lihoreau
1,4 e
F. Bellivier
6,7,8,9 f
, E. Haffen
1,5,10 g
and B. Wacogne
1,2 h
Centre Hospitalier Universitaire de Besançon, Centre d’Investigation Clinique, INSERM CIC 1431,
25030, Besançon Cedex, France
FEMTO-ST Institute, Université de Bourgogne Franche-Comté, CNRS, 15B Avenue des Montboucons,
25030 Besancon, Cedex, France
Centre Hospitalier Universitaire de Lille, Centre d’Investigation Clinique, INSERM CIC-IT 1403, F-59000 Lille, France
Tech4Health Network - FCRIN, France
Department of Clinical Psychiatry, CHU de Besançon, UR LINC 481 Laboratoire de Recherches Intégratives en
Neurosciences & psychologie Cognitive, University Bourgogne Franche-Comté, 25030, Besançon Cedex, France
AP-HP, GH Saint-Louis, Lariboisière, F. Widal, Department of Psychiatry and Addiction Medicine,
75475 Paris Cedex 10, France
Inserm, U1144, Paris, F-75006, France
Université Paris Descartes, UMR-S 1144, Paris, F-75006, France
Université Paris Diderot, Sorbonne Paris Cité, UMR-S 1144, Paris, F-75013, France
FondaMental Foundation, Creteil, Hôpital Albert Chenevier, Pôle Psychiatrie, 40 rue de Mesly, 94000 Créteil, France
{emmanuel.haffen, bruno.wacogne},
Keywords: Self-monitoring in Vitro Medical Device, Lithium, Usability, European in Vitro Medical Devices Regulation.
Abstract: Bipolar disorders are severe and complex psychiatric disorders and lithium remains one of the most effective
drugs for relapse prevention. Despite its effectiveness, prescription of lithium therapy can be complicated
because of its narrow therapeutic range. Furthermore, adherence to treatment is generally low. One means of
improving adherence would be to make the patient an actor of his/her treatment. The possibility to control the
lithium level with a device that can be used at home would favor this involvement. Although the main part of
the work to produce a device is research and development, regulatory analysis, including usability, should not
be neglected. Indeed, some design choices should be made taking into account usability constraints. This
ensure the fabrication of a device which will be safe, effective and well accepted by the intended users. In this
conference, we present actions taken in this direction during the R-Link project.
The R-Link project, "Response to Lithium Network",
is a collaborative project funded by the European
Commission (Grant agreement 754907). It
proposes a clinical study involving people with
bipolar disorder type I when lithium treatment is
initiated (NCT04209140). The consortium includes
22 European partners among which research
institutes, hospitals, clinical investigation centers and
companies. It is led by Prof. Franck Bellivier
(Department of Psychiatry and Addiction Medicine -
Charrière, K., Azzopardi, C., Pelayo, S., Nicolier, M., Lihoreau, T., Bellivier, F., Haffen, E. and Wacogne, B.
Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring Device.
DOI: 10.5220/0010971600003123
In Proceedings of the 15th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2022) - Volume 1: BIODEVICES, pages 259-269
ISBN: 978-989-758-552-4; ISSN: 2184-4305
2022 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
Expert Centers University of Paris Diderot - INSERM
The goal is to identify early biomarkers that will
allow stratification of patients with bipolar I disorder
according to their Lithium (Li) response. This response
is being assessed prospectively over a two-year period
based on a thorough clinical assessment coupled with
measurements of blood omics, anatomical/structural
magnetic resonance imaging (MRI) and
derived markers. These markers will be tested as
predictors of response status at the end of the study.
Each patient will be involved in the study for two years.
Translation will be assessed in terms of positive and
negative predictive values of the markers, usefulness
of the markers when used alone or in combination,
patient acceptability, and cost-effectiveness. As it is
essential to monitor adherence to treatment, interactive
software for self-assessment of mental status will be
introduced and electronic reminders will be offered
throughout the study. A device that will allow self-
monitoring of salivary lithium levels at home will be
developed to be provided to patients. This last point is
the focus of this paper.
Indeed, the design and development of this device
raise some interesting questions related to the
compatibility between (i) the design choices of the
device and its usability and (ii) the regulatory
framework to be compliant with. The regulatory
analysis guides some design choices. In a context
where the device is still at the conceptual stage and its
design is progressing at the pace of the complex
regulatory analysis, can we already plan and conduct
a usability engineering process?
In this paper, we will present the different aspects
of usability engineering process on a general basis
and we will specify what was performed in the frame
of the R-Link project. Regulatory aspects must be
treated but will not be described in this
communication. After an introductory part on bipolar
disorders and the technical progress of the R-Link
device, we will detail the usability studies plan before
Bipolar disorders are severe and complex psychiatric
disorders that affect approximately 45 million people
worldwide (James et al., 2018). In France, it is
estimated that between 1 and 2.5% of the population
is affected by these disorders, but it seems that these
figures are underestimated. It is one of the most
serious psychiatric pathologies, frequently leading to
suicide attempts: 50% of patients with bipolar
disorder will make at least one suicide attempt, and
15% will die (Troubles bipolaires, n.d.)
[not dated]. In
addition, bipolar disorder often leads to functional
impairment and reduced quality of life (Oldis et al.,
2016) and is associated with a decrease in lifespan of
approximately 10 years. The World Health
Organization has ranked this condition among the 10
most worrying of the 21
century (WHO | The Global
Burden of Disease, n.d.).
According to the DSM-5 (Diagnostic and
Statistical Manual of Mental Disorders-5th edition),
bipolar disorders can be classified into bipolar I
disorder, bipolar II disorder, cyclothymia and residual
categories. This sub-classification depends on the
severity and duration of manic (or hypomanic) and
depressive episodes (Vieta et al., 2018).
Bipolar disorder is recurrent, even when
diagnosed and treated. Various molecules are
available to treat bipolar disorders, among them are
mood stabilizing agents. Clinically, the main actions
that qualify a molecule as a mood stabilizer are its
effects at both ends of the mood spectrum (depression
and mania) and its ability to maintain euthymia by
preventing future mood instability. According to
these factors, lithium is the best and therefore the gold
standard mood stabilizing agent (Malhi et al., 2021).
According to the network meta-analysis by Miura
et al., lithium remains one of the most effective drugs
for relapse prevention and should remain the first-line
treatment (Miura et al., 2014).
Current recommendations call for a serum lithium
concentration between 0.6 mM and 0.8 mM for the
most effective treatment. In the acute manic phase,
concentrations can be increased to 1 mM, depending
on the patient's tolerance (Malhi et al., 2020). Despite
its effectiveness, lithium therapy can be complicated to
administer. Indeed, lithium can cause safety problems
due to its narrow therapeutic range. Below 0.5 mM
lithium, treatment may be ineffective and may lead to
relapse. Above 1.5 mM, there is a risk of toxicity. The
Li intoxication symptoms are variable and depend on
the intoxication severity. Nevertheless, if lithium levels
are correctly controlled, it seems that its long-term
toxicity may be limited (Malhi et al., 2020). According
to the practical guide of Malhi et al., follow-up should
be performed during the initial maintenance phase as
well as whenever there is a significant change in
therapy or when adverse effects occur (Malhi et al.,
2011, 2016).
Despite existing guidelines, many clinicians
remain reliant on an empirical "trial and error"
approach to effective lithium prescribing. Indeed, 18
to 24 months is often required to ensure a clinically
meaningful effect of lithium, with shorter-term
ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices
outcomes not reliably predicting prophylactic
outcomes. In addition to concerns about potential side
effects, this trial-and-error strategy likely leads to
increased non-adherence to treatment potentially
increasing the likelihood of treatment failure. For
example, only 30% of patients treated with lithium
show an excellent long-term response, most show a
partial response, and up to one-third do not respond
(Scott et al., 2018).
Furthermore, adherence to prescribed treatment is
generally low in most chronic illnesses including
bipolar disorder, with nonadherence as high as 50%
of most patients (Goodwin et al., 2016). The
possibility to strongly involve patients through
regular and home self-monitoring would be a
valuable help, probably allowing for increased
adherence to treatment but also for finer monitoring
of lithium levels. This is why a part of the R-Link
H2020 project aims to develop such a device.
The R-Link device aims to improve adherence to
treatment for patients with bipolar disorder type I,
prevent lithium overdose, prevent relapse into a
manic or depressive phase.
To achieve these goals, the idea is to help patients
to become active in their treatment - and more
particularly in its monitoring - by regularly
monitoring their salivary lithium levels.
Although there are still many uncertainties to be
resolved before an usable product is available for the
first pilot studies, the final configuration of the device
is already broadly defined (Figure 1). It will consist
of three distinct parts. Two parts will be single-use: a
system for collecting the patient's saliva (A) and a
"cartridge" containing the reactive zone and the
solutions necessary for the reaction (B). The third part
will be the device itself, i.e. the reusable apparatus (C)
allowing: (i) the driving of the solutions on the
dedicated reaction zone, (ii) the reading of the
reaction, (iii) the display and recording of the results.
Figure 1: Diagram of the 3 parts of the final device. A.
saliva collection system, B. cartridge with reagent area and
C. reader-actuator for performing, reading and interpreting
the reaction.
As mentioned above, the prototype is not yet
available but some technical solutions have already
been defined and technical validation tests are
currently underway. It is therefore possible - and
necessary to meet the time constraints set by the
H2020 project - to move forward in parallel on certain
tasks, including the implementation of a usability
Usability is an integral part of the MDR/IVD, in
particular point 19, chapter II of Annex VIII
concerning “protection against risks arising from
devices intended for self-diagnosis or diagnosis near
the patient […]”. So, usability engineering process
aims to improve the safety of use of the device and
ultimately the safety of the patients as end-users by
reducing the risks associated with errors in use during
normal use of the medical device. Usability studies
have to be mobilized to anticipate the risks of
abnormal use, in order to avoid, as much as possible,
the associated errors. The process should be
documented in the usability studies file for obtaining
CE marking.
Usability is defined by the 62366-1 standard (NF
EN 62366-1/A1 - Août 2020, n.d.) as "the
characteristic of the user interface that facilitates use
and thus establishes the effectiveness, performance
and satisfaction of the user in the intended use
environment". The usability engineering process is a
risk management process focused on potential use
errors. This usability process is closely intertwined
with the standard 14971 for the application of risk
management to MD (Medical Device) (NF EN ISO
14971 - Décembre 2019, n.d.).
The usability engineering process is an iterative
process that applies to all stages of the MD life cycle
and for all users. It concerns, of course, the use of the
device itself with the user interface, but also the
accompanying documentation and the delivered
training. It must take into account the end users
(patients and non-medical caregivers) and the
secondary users such as the medical staff who will be
responsible for training in the use of the device or the
staff who will have to manufacture, package, store,
maintain, recycle or dispose of the device. We have
related here only the end users: patients and non-
medical caregivers.
Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring
Figure 2: Schematic of the usability plan for the R-Link device.
4.1 Use Specifications
Establishment part of the use and functional
specifications was done during the functional analysis
(Charrière et al., 2021). However, the usage
specifications do not only include the required
functions of the final device but must also establish
the characteristics of the environment in which the
device will be used, as well as the characteristics of
the users, considering both the physical and cognitive
characteristics of the primary and secondary users.
The main steps of the proposed usability
engineering plan for the R-Link device are
summarized in Figure 2. They consist in establishing
first, the usability specifications and second, the
functional specifications (Figure 2, points 1 and 2).
The usability-related safety characteristics must be
then established accordingly and will complete the
technical risk analysis made by the manufacturer
(Figure 2, point 2). On this basis, the dangerous
situations and the different scenarios arising from
them can be identified to guide the MD design. Future
assessments can then be planned to test to what extent
the design of the device prevents that use errors occur
(Figure 2, points 3 and 4). The evaluation plan for the
user interface (Figure 2, point 5) should be
established integrating formative evaluations (Figure
2, point 6). It may be necessary to run iterative
evaluations with several models or demonstrators
(Figure 2, points 6 and 7), before reaching a system
satisfactory for conducting summative evaluation(s)
(Figure 2, point 9).
4.1.1 Intended Use Environment
The device is intended to be used at the patient's
home, by the patient himself or by non-professional
caregivers. Environmental characteristics are
therefore likely to vary according to location,
especially countries. For example, the first models of
the R-Link device will have to be connected into the
mains. In France, the voltage is 220 V, whereas it is
110 V in the United States.
The patient could be away from home at the time
of the test. Ideally, the device should be easily
transportable and usable in mobile conditions. It will
therefore be important to provide an appropriate
device size and weight.
The appropriate luminous flux to illuminate a
space varies according to the room. Recommended
levels can be found in NF EN 12464-1 standard
"indoor lighting for workplaces" (NF EN 12464-1 -
Juillet 2011, n.d.). It is desirable that the result can be
read from 20 cm to 50 cm under appropriate light
Since the device is intended for home use, the
temperature can be varied in the range of 14°C to
35°C. However, previous summer heat waves should
be taken into account. If this is not the case, the
manufacturer will ensure that this risk is controlled by
clearly indicating it in the instructions or by adding an
internal control to the device.
Based on the reagent cost, the estimated
production costs after industrialization and, above all,
the recommendations of the project's partner
physicians, patients will be encouraged to perform a
test every 15 days. This frequency could be adapted
throughout the project duration.
1. Prepare use specification
Conduct meetings and brainstorming with medi cal staff
and manufacturer
6. Establish user interface specification
7. Design and implement the user interface - 8. Perform formative evaluations
Conceptualize Implement Evaluate
Develop user
interface design
Models with
instructions for use
Formative evaluation:
face-to-face interview
Ite rati on 1
Refine user
interface design
Hight fidelity
demonstrators +
instructions for use
Formative evaluation:
usability test
Itera ti on 2
9. Perform summative evaluation
Usability test Evaluate residual risks related to usability
3. Identify hazards, hazardous situations and hazard-
related scenarios
4. Select the hazard-related use scenarios for
2. Identify characteristics for safety
Ta s k a n a l y s i s
Review publicly
Conduct meetings, brainstormings with partners
(medical, research, manufacturer)
Conduct meetings and brainstorming with medical staff
and manufacturer
5. Establish user interface evaluation plan
Refine user interface
Refined models and
instructions for use
Formative evaluation:
face-to-face interview
Iteration 2+X (if needed)
Refine user interface
Refined models and
instructions for use
Formative evaluation:
face-to-face interview
Iteration 1+X (if needed)
ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices
The description of the technical environment of
the device cannot yet be finalized at this stage of the
project. However, some characteristics can now be
specified: hardware configuration such as processor
speed, memory size, network, storage, input and
output devices; screen type and size, resolution and
color depth; whether or not the visual interface
elements (such as text or symbols) can vary in size
(and size(s) available); configuration of the electronic
board; assistive technologies available if required.
4.1.2 Target Users
User characteristics (functional, physical, sensory
and cognitive capabilities, experience, knowledge
levels and behaviors) could impact the safe and
effective use of the device.
For example, elderly people may have reduced
visual acuity or polyarthritis problems. A small text
on a screen or a too complicated handling of the
device will most likely lead to user errors. Since the
ultimate goal is to eliminate sources of error related
to perception, cognition or handling as much as
possible, it is important to correctly identify the
primary users (i.e. the person who will use the device
in its actual medical use) and the secondary users (i.e.
all persons who may have the device in their hands
during its life cycle, from manufacture to disposal).
In the case of the R-Link device, the primary users
(see Table 1) of the device are patients with bipolar
disorder type I. Bipolar disorder affects both men and
women, regardless of social class or location. The
illness can occur throughout the lifespan, from the age
of 15 to over 60. If patients are unable to use the
device due to physical or cognitive impairments,
caregivers may do it for them and then become the
primary users. Bipolar disorder causes comorbidity
that can lead to impairments, and patients (or
caregivers) may have age-related physical and
cognitive impairments, such as loss of vision,
hearing, dexterity, etc. Patient and non-professional
caregiver categories for the device should include:
adults (18-49 years old), seniors (50-64 years old),
and the elderly (65 years old and older).
4.2 User-centered Safety Features
Risk analysis is often understood as an analysis of
technical risks like electrical, thermal or biological
risks. They are related to a failure of the device or of
a component, and therefore do not depend on the way
the device is used, i.e. on the interaction between user
and interface.
However, some risks are directly related to this
interface/user interaction and can be the result of user
interface design problems. For example, the result is
not clearly readable or difficult to interpret, resulting
in a more or less serious damage (Health, 2019).
Therefore, the risk analysis - and the entire risk
Table 1: Primary user characteristics.
Age range: 18 years and above. Education and literacy: all levels of education, including
illiterate. All types of socio-economic, ethnic, cultural status. Language: French.
Many women have long nails. Patients may have xerostomia and other co-morbidities
(alcohol and drug use, panic disorders, obsessive-compulsive disorders, eating disorders,
personality disorders, overweight and obesity, diabetes, cardiovascular disease). The
majority of patients have no physical disorders. Some patients - particularly those over 40
years of age - may have age-related vision problems (such as presbyopia). Some patients -
particularly those aged 50 and over - may have progressive hearing loss and/or dexterity
and strength limiting disorders such as tremors, arthritis...Some patients may have native
disorders such as visual impairment, hearing impairment or physical disability. Some
patients may have cognitive impairments in executive function, learning and verbal
memory. With advancing age, patients may have much greater impairments in information
Patients are generally not proficient in the use of medical devices. Some patients -
particularly older ones - may not be comfortable using the device, as they may be
latecomers to computer technology.
Type of
Unknown. To be determined in formative and summative evaluations. There is a strong
preference among partners for learning in a consultation, delivered by the doctor and/or
trained medical staff.
Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring
management plan - must also include the risks
associated with the use of the device throughout its
life cycle. It is therefore necessary to be able to
identify the hazards, estimate and quantify the
associated risks, control them and be able to monitor
the effectiveness of these measures (NF EN ISO
14971 - Décembre 2019, n.d.).
Here, the analysis is focused on the risks related
to the use of the device. The analysis of technical
risks, resulting from a failure of the device, will not
be dealt with. Use errors analysis is difficult to carried
out when the technical solutions are not yet known
and when the development of the device is not
advanced, which is the case for the R-Link device.
Some of main trends are already decided in terms of
design: a saliva sample is inserted in the system
manually or automatically, a chemical reaction takes
place, the result is read by an analyzer and delivered
to the patient who must interpret it and react
There are analytical approaches for identifying
hazard-related tasks or scenarios. Such an approach is
based on the task analysis method, which breaks
down the process of using the device into discrete
sequences of tasks. This analysis has been applied to
the R-Link device.
To perform the salivary lithium level self-test, all
parts of the R-Link device are required: the reader, a
cartridge and a saliva collector (Figure 1). The
cartridge and saliva sampler are independent of the
reader. Five major steps have been identified for
performing salivary lithium self-testing with the R-
Link device: (i) collect saliva using a saliva sampler,
(ii) insert the saliva sample in the designated area, (iii)
insert the cartridge into the R-Link reader, (iv) after a
few minutes, the result appears on the screen and (v)
the patient reads and interprets the result.
For each of these tasks, a questioning based on the
WWWWHW model (Who, What, Where, When,
How, Why) is performed. Based on this questioning,
we identified anticipated subtasks that will be
performed by the patient, with the exception of the
automated tasks.
Based on these identified subtasks, the user risk
analysis can start relying on a Failure Mode and
Effects Analysis (FMEA) method. It is used to
identify all the hazards and harms associated with the
use of the device according to its characteristics and
its intended use. In order to conduct this analysis in
the best way, all project partners (clinicians,
researchers and manufacturers) must be involved. For
each of the previously defined subtasks, it is
determined whether or not a hazard can be associated
with. This hazard may lead - either on its own or as a
result of a sequence of events - to a dangerous
situation that will result in damage for the user. The
risk level is then assessed according to the probability
and severity of the damage.
If the risk level is high, risk control measures must
be put in place to ensure that the residual risk is
acceptable. At the research and development step, a
certain number of control methods could be
suggested. The final choice of the control method will
be made considering the adequacy between use added
value and production costs.
For the R-Link device, several types of damage
have been identified. The most serious is an erroneous
chemical reaction leading to a false result, namely an
over- or under-estimation of the lithium level. In both
cases, the damage is severe.
In case of overestimation of the lithium level by
the device, the patient might actually be beyond the
zone for which no toxicity is to be feared.
Nevertheless, this risk is to be put in comparison with
the patient's feeling. Indeed, lithium overdoses are
often well estimated by the patient who then
immediately contacts his doctor.
In case of lithium level underestimation, the
patient would probably not be aware of it and would
risk a relapse - either into a manic state or into a
depressive state. It is precisely these cases that the R-
Link device targets in priority. Thus, in both cases,
the damage to the patient could be significant and
countermeasures must be taken to reduce it.
Several causes could be at the origin of this bad
estimation: too high temperature, expired
consumable, bad salivary sampling, bad reading and
bad interpretation of the result delivered by the
device. To reduce these risks, several control methods
are suggested: designing the device with a
thermostatic chamber, or at least incorporating a
temperature controller; designing the device with an
integrated expiration date controller; training end
users in saliva sampling and deliver clear instructions
for use; making sure that the result is clearly
Other non-critical errors of use have been
identified. For example, if the patient does not
connect the device properly to the power source, the
test cannot be performed. Nevertheless, this problem
should be rare and will not cause any direct damage
since the test cannot be performed. It should also be
easily controlled by learning how to use the device
and a clear instruction manual.
ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices
4.2.1 Review of Public Databases
A review of available databases was also conducted
to identify known use errors with similar devices:
MAUDE (Manufacturer and User Facility Device
Experience), Web of Science, PUBMED. Only one
search carried out with the key words "self-test
lithium" on google gave interesting results (Self Test
Lithium - Google Search, n.d.). The first comes from
the Dutch company FISIC: the Medimate Multireader
(Fisic | Lithium Self Test, n.d.). The second comes
from ReliaLAB, an American company: the Instaread
lithium system (Finger-Stick Lithium Test, n.d.).
For the Instaread lithium system an adverse
reaction report exist. . This report mentions that the
results obtained with the Instaread lithium system can
differ of up to 0.5 mM compared with the results
obtained during a laboratory test. (INSTAREAD
LITHIUM SYSTEM * Adverse Event MAUDE, n.d.).
Finally, the 510k data sheet for this MD/IVD is
available, but it only enumerates device performance
data (510(k) Premarket Notification, n.d.). No data
regarding usability was found.
More documentation is available from the second
MD/IVD, the Medimate Multireader from the
company FISIC (Fisic | Documentation, n.d.). This
one is not FDA approved but is EC labelled according
to the European Directive for IVDs (98/79/EC). In a
study, authors aim to evaluate the usability of the
Medimate Multireader when used by the patient for
self-testing at home, or when used in a health care
facility for point-of-care testing. Healthcare workers
(for point-of-care testing) and patients (for home
testing) completed a System Usability Scale (SUS)
questionnaire. The SUS is a validated method to
quickly assess the perceived usability of a system and
consists of 10 items covering different aspects such
as complexity, ease of learning, frequency of use
(Affairs, 2013; Bangor et al., 2008). Based on this
scale, authors concluded that the usability of their
device is "good", even if the blood collection was
considered unpleasant and/or difficult in terms of
sampled volumes.
The analysis of the competing devices is a key
point, which allows to anticipate the requirements
expected for similar devices. Thus, the studies for the
design and then the validation of the R-Link device -
similar in its specification of use to the Medimate
Multireader and Instaread lithium system - could be
inspired by this already compliant competition for a
diffusion on the European market or for the American
market. For the Instaread lithium system, the 510k
data sheet of the system could be a source of
inspiration for the performance validations of our
MD/IVD as well as the instructions for use (complete
and abbreviated), the study designs used and the
various articles published in peer-reviewed journals
from the company FISIC (Floris et al., 2010; Muñoz
et al., 2011; Nieuwe Mogelijkheden Voor Een
Lithiummeting Op de Poli En in de Huiskamer, 2019;
Staal et al., 2015).
4.3 Formative and Summative
Although the R-Link device is at a very early
development stage, it is possible to anticipate future
evaluations. In addition to the 62366-1 and 2 standard
(IEC/TR 62366-2:2016 - Avril 2016, n.d.; NF EN
62366-1/A1 - Août 2020, n.d., pp. 62344–2), the FDA
guide for manufacturers and their staff is freely
available and is a good support to design the plan of
the different usability evaluations of a device (Health,
2019). Usability evaluations can be classified into
two categories depending on the objective: formative
and summative evaluations.
4.3.1 Formative Evaluations
Formative evaluations should help in the design of the
MD during its development and focus primarily on
points that could jeopardize the safety of use
identified during the risk analysis and on undefined
design options. They should complement the
preliminary analyses (task analyses, risk analyses)
and reveal previously unidentified errors in use. Thus,
formative evaluations should be performed
throughout the development process, depending on
the amount of information needed for the design, the
complexity of the device and its use, the variability of
the user population or the conditions of use. They can
be done with very simple mockups, even drawings, or
with very advanced prototypes (Health, 2019).
Standard 62366-2 recommends several types of
methodologies for conducting these formative
evaluations, including face-to-face interviews,
cognitive walkthroughs, and/or usability tests. For
face-to-face interviews to be productive, the
objectives must be established beforehand and an
interview guide defined. This guide should not
present closed questions but include short, open-
ended, organized questions around topics of
discussion. In the cognitive walk, a very preliminary
design - which may be in the form of drawings - is
presented to a small group of people. A session
involves a single participant who must imagine
his/her reactions to the MD and verbalize all his/her
thoughts and actions. Usability tests are conducted
Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring
Table 2: Example of summary sheet; task "Insert the sample in the slot provided in the cartridge".
Description of the use scenario related to the hazardous
test result
The system for transferring saliva from the collection tube to the
cassette has not yet been determined. The user has difficulties in
transferring saliva from one container to another. The user does
not insert a sufficient volume into the cassette and/or causes
numerous bubbles in the reaction area. The chemical reaction does
not take place correctly, leading to an over- or underestimation of
the lithium level.
Anxiety, relapse
or risk of toxicity
Use of the saliva
collection device
is difficult for the
Formative evaluation(s) - "Sample tube / cartridge / leaflet" interface
Face-to-face interviews
Objective: To assess the understanding of the instructions in relation to the use of the system and the clarity
of the training.
Method: Face-to-face interviews with an interview grid focused on the understanding of the instructions and
the instructions given by the trainer.
Presentation of a low definition model, then high definition, allowing the sample to be placed in the
cassette, with the associated instructions. Collect opinions on the clarity of instructions. Explanation of the
use of the device. Collect opinions on the clarity of the use of the device after explanation.
Data collection: audio recording and note taking. Analysis: Qualitative analysis of verbatims.
Population: Nursing staff doctors + nurses + clinical research officer.
Note: Refine the design according to the results and repeat the evaluation until the device for depositing the
sample in the intended location in the collection cassette is satisfactory. Conduct the usability test when this
stage is reached.
Usability test
Objectives: To assess the number of usability errors and to identify the causes. To assess the number of non-
compliant deposits of the sample into the cassette. To assess the understanding of the training.
Method: Usability test with video recording, interview and questionnaire. 1 session per participant.
Population: Patients with bipolar disorder type I, 3 age groups (18-24, 25-62, over 62), 1 male and 1
female/group. Non-medical carers, 3 age groups (18-24, 25-62, over 62), 1 male and 1 female/group.
Course of the session: Presentation of the device allowing the sample to be placed in the location provided
in the cassette selected following the initial evaluations, with the associated instructions. Explanation of the
use by the trainer, as in a real situation. Immediately afterwards, the user will carry out all the tasks
requested, following only the instructions, without any external help. The session will be filmed to allow
analysis (number of hesitations during sampling, number of times the instructions are consulted).
Immediately after the collection, the volume of saliva deposited in its place will be recorded in the
observation book, as well as the presence or absence of bubbles/foam. Proposal of the SUS questionnaire
with an interview targeted on the difficulties of use encountered, including the understanding of the
instructions given.
Data collection: Video recording + observation booklet + questionnaires + note taking.
Data analysis: Quantitative analysis of the number of errors, hesitation/consultation of the instructions, non-
compliant deposits + analysis of SUS + qualitative analysis of verbatims.
with a few users who have to complete some tasks
representing the important functions of the future MD
(IEC/TR 62366-2:2016 - Avril 2016, n.d.).
For the R-Link device, the risk analysis reveals
four tasks for which the risk of use errors leading to
damage is significant: (i) saliva collection, (ii)
insertion into the cartridge, (iii) reading the result, and
(iv) interpreting the results. The formative
evaluations should ensure that the design chosen for
the parts of the device supporting these tasks
effectively eliminates or limits any risk associated
with misuse. It is performed in an iterative way and
the first steps could be done with experts instead of
end users. For each of the four domains mentioned,
two types of formative evaluations are retained: a
face-to-face interview with hospital staff (experts)
and a usability test with patients. A summary sheet
for each of these tasks was designed (Table 2). These
sheets, as the whole file, are not fixed yet and may
evolve according to the progress and design choices
of the project.
ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices
4.3.2 Summative Evaluations
The summative evaluation is always the very last step
of the fitness-for-use engineering process. It must
demonstrate that the MD can be used under the
specified conditions of use, by the intended users and
without unacceptable residual risk: it is therefore the
validation step of the device in terms of safety risks
related to use. The summative evaluation must
implement the scenarios relating to the previously
defined dangerous phenomena, under conditions as
close as possible to reality, but without a clinical
effect. Thus, for the summative evaluation to be valid,
it is important to ensure that the participants represent
all the intended users, that all critical tasks are
performed during the test, that the user interface
represents the final design, and that the test conditions
correspond to the real conditions of use.
As with a traditional clinical investigation, a
rigorous protocol must be established, including the
introduction, the objectives of the test and the method
used, the description of the MD, the necessary
equipment and environment, the description of the
participants and the personnel involved, the list of
tasks to be carried out, the methods of data collection
and analysis, an operating procedure for the test and,
if necessary, a description of the training.
The objective of this work was to give indications to
the R&D team concerning the regulatory constraints
likely to influence the design and to initiate the
engineering suitability plan. Thus, although many
questions remain today, this very early participation
has already allowed and will subsequently allow the
technical team to orientate itself towards what we
hope will be a high-performance, reliable and safe
Thus, carrying out a usability plan at a very early
stage of design is entirely possible and even desirable
because the analyses carried out make it possible to
feed the design and orientate the choices by
identifying the needs of the users and the constraints
of the usage environments. The specifications for use
will thus be issued accordingly, making it possible to
prevent the risks associated with the use of the DM.
However, the plan cannot be fixed at this stage. It will
have to be adapted as the design, technical choices
and the results of the various analyses and formative
evaluations carried out during the project progress.
This work highlight the importance of the usability
aspect from the very beginning of a project. As we
discussed in a previous paper (Charrière et al., 2021),
the development process of a MD should not be
envisaged in a linear way, with a separation between
partners. A dynamic vision has to be adopted, because
the choice of technical solutions or specialities on
which research and design efforts should be made
depends on several factors such as: acceptability to
end-users, risk analysis, technical feasibility,
production cost, regulatory constraints and the return
on investment that the manufacturer can expect.
R-LiNK has received funding from The European
Union’s Horizon 2020 Research and Innovation
Program Under Grant Agreement N° 754907.
Many thanks to the teaching team of the
Universitary diploma “Methodologies in clinical
evaluation of medical devices” leaded by the
TECH4HEALTH network.
510(k) Premarket Notification. (n.d.). Retrieved 3 June
2021, from
Affairs, A. S. for P. (2013, September 6). System Usability
Scale (SUS). Department of Health and Human
Services. system-usability-scale.html
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An
Empirical Evaluation of the System Usability Scale.
International Journal of Human–Computer Interaction,
24(6), 574–594.
Charrière, K., Azzopardi, C., Nicolier, M., Lihoreau, T.,
Bellivier, F., Haffen, E., & Wacogne, B. (2021).
Functional Analysis to Drive Research and Identify
Regulation Requirements: An Example with a Lithium
Monitoring Device: Proceedings of the 14th
International Joint Conference on Biomedical
Engineering Systems and Technologies, 300–307.
Finger-stick lithium test. (n.d.). Retrieved 2 June 2021,
Fisic | Documentation. (n.d.). Retrieved 2 June 2021, from
Fisic | Lithium self test. (n.d.). Retrieved 26 May 2021, from
Floris, A., Staal, S., Lenk, S., Staijen, E., Kohlheyer, D.,
Eijkel, J., & Berg, A. van den. (2010). A prefilled,
ready-to-use electrophoresis based lab-on-a-chip
Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring
device for monitoring lithium in blood. Lab on a Chip,
10(14), 1799–1806.
Goodwin, G. M., Haddad, P. M., Ferrier, I. N., Aronson, J.
K., Barnes, T. R. H., Cipriani, A., Coghill, D. R., Fazel,
S., Geddes, J. R., Grunze, H., Holmes, E. A., Howes,
O., Hudson, S., Hunt, N., Jones, I., Macmillan, I. C.,
McAllister-Williams, H., Miklowitz, D. M., Morriss, R.,
… Young, A. H. (2016). Evidence-based guidelines for
treating bipolar disorder: Revised third edition
Recommendations from the British Association for
Psychopharmacology. Journal of Psychopharmacology
(Oxford, England), 30(6), 495–553.
Health, C. for D. and R. (2019, September 2). Applying
Human Factors and Usability Engineering to Medical
Devices. U.S. Food and Drug Administration; FDA.
IEC/TR 62366-2:2016—Avril 2016. (n.d.). Retrieved 21
May 2021, from
(n.d.). Retrieved 2 June 2021, from
James, S. L., Abate, D., Abate, K. H., Abay, S. M., Abbafati,
C., Abbasi, N., Abbastabar, H., Abd-Allah, F., Abdela,
J., Abdelalim, A., Abdollahpour, I., Abdulkader, R. S.,
Abebe, Z., Abera, S. F., Abil, O. Z., Abraha, H. N.,
Abu-Raddad, L. J., Abu-Rmeileh, N. M. E.,
Accrombessi, M. M. K., Murray, C. J. L. (2018).
Global, regional, and national incidence, prevalence,
and years lived with disability for 354 diseases and
injuries for 195 countries and territories, 1990–2017: A
systematic analysis for the Global Burden of Disease
Study 2017. The Lancet, 392(10159), 1789–1858.
Malhi, G. S., Bell, E., Bassett, D., Boyce, P., Bryant, R.,
Hazell, P., Hopwood, M., Lyndon, B., Mulder, R.,
Porter, R., Singh, A. B., & Murray, G. (2021). The 2020
Royal Australian and New Zealand College of
Psychiatrists clinical practice guidelines for mood
disorders. The Australian and New Zealand Journal of
Psychiatry, 55(1), 7–117.
Malhi, G. S., Bell, E., Outhred, T., & Berk, M. (2020).
Lithium therapy and its interactions. Australian
Prescriber, 43(3), 91–93.
Malhi, G. S., Gershon, S., & Outhred, T. (2016).
Lithiumeter: Version 2.0. Bipolar Disorders, 18(8),
Malhi, G. S., Tanious, M., & Gershon, S. (2011). The
lithiumeter: A measured approach. Bipolar Disorders,
13(3), 219–226.
Miura, T., Noma, H., Furukawa, T. A., Mitsuyasu, H.,
Tanaka, S., Stockton, S., Salanti, G., Motomura, K.,
Shimano-Katsuki, S., Leucht, S., Cipriani, A., Geddes,
J. R., & Kanba, S. (2014). Comparative efficacy and
tolerability of pharmacological treatments in the
maintenance treatment of bipolar disorder: A
systematic review and network meta-analysis. The
Lancet Psychiatry, 1(5), 351–359.
Muñoz, M., Eijkel, J., Floris, A., Staal, S. s, Ríos, Á., & Van
den Berg, A. (2011). The development of a point of care
creatinine measurement using disposable ready to use
microchip capillary electrophoresis. Journal of The
Electrochemical Society - J ELECTROCHEM SOC, 2.
NF EN 12464-1—Juillet 2011. (n.d.). Retrieved 21 May
2021, from
NF EN 62366-1/A1—Août 2020. (n.d.). Retrieved 21 May
2021, from
NF EN ISO 14971—Décembre 2019. (n.d.). Retrieved 21
May 2021, from
Nieuwe mogelijkheden voor een lithiummeting op de poli en
in de huiskamer. (2019, June 13). Psyfar.
Oldis, M., Murray, G., Macneil, C. A., Hasty, M. K., Daglas,
R., Berk, M., Conus, P., & Cotton, S. M. (2016).
Trajectory and predictors of quality of life in first
episode psychotic mania. Journal of Affective
Disorders, 195, 148–155.
Scott, J., Etain, B., & Bellivier, F. (2018). Can an Integrated
Science Approach to Precision Medicine Research
Improve Lithium Treatment in Bipolar Disorders?
Frontiers in Psychiatry, 9, 360.
self test lithium—Google Search. (n.d.). Retrieved 2 June
2021, from
Staal, S., Ungerer, M., Floris, A., Brinke, H.-W. T.,
Helmhout, R., Tellegen, M., Janssen, K., Karstens, E.,
Arragon, C. van, Lenk, S., Staijen, E., Bartholomew, J.,
Krabbe, H., Movig, K., Dubský, P., Berg, A. van den,
& Eijkel, J. (2015). A versatile electrophoresis-based
self-test platform. ELECTROPHORESIS, 36(5), 712–
ClinMed 2022 - Special Session on Dealing with the Change in European Regulations for Medical Devices
Troubles bipolaires: Diagnostiquer plus tôt pour réduire le
risque suicidaire. (n.d.). Haute Autorité de Santé.
Retrieved 10 May 2021, from https://www.has-
Vieta, E., Berk, M., Schulze, T. G., Carvalho, A. F., Suppes,
T., Calabrese, J. R., Gao, K., Miskowiak, K. W., &
Grande, I. (2018). Bipolar disorders. Nature Reviews
Disease Primers, 4(1), 1–16.
WHO | The global burden of disease: 2004 update. (n.d.).
WHO; World Health Organization. Retrieved 10 May
2021, from
Is Usability Engineering Anticipation Possible during the Initial Research Actions? An Example with the R-Link in Vitro Self-monitoring