Toward a Compare and Contrast Framework for COVID-19 Contact
Tracing Mobile Applications: A Look at Usability
Cristiano Storni
1a
, Damyanka Tsvyatkova
2b
, Ita Richardson
1,2 c
, Jim Buckley
1,2 d
,
Manzar Abbas
2
, Sarah Beecham
2
, Muslim Chochlov
1,2
, Brian Fitzgerald
2
, Liam Glynn
1
,
Kevin Johnson
1
, John Laffey
3
, Bairbre McNicholas
3
, Bashar Nuseibeh
2
, James O’Connell
2
,
Derek O’Keeffe
3
, Ian R O'Keeffe
2
, Mike O’Callaghan
1
, Abdul Razzaq
2
, Kaavya Rekanar
2
,
Andrew Simpkin
3
, Jane Walsh
3
and Thomas Welsh
2
1
University of Limerick, Republic of Ireland
2
The Science Foundation Ireland Research Centre for Software, Lero, University of Limerick, Republic of Ireland
3
National University of Ireland Galway, Republic of Ireland
Keywords: COVID-19, mHealth, Evaluation Taxonomy, Usability.
Abstract: This paper reports on the progress in the project COVIGILANT, which is aimed at developing an evaluation
taxonomy for Contact Tracing Applications (CTAs) for COVID-19. Specifically, this article describes the
development of Usability, one pillar of the COVIGILANT taxonomy, discussing the classification and
decision-making processes, and the initial model validation. The validation process was undertaken in two
stages. First, we validated how the Usability pillar could be used to evaluate the Irish Health Services
Executive (HSE) COVID-19 CTA. While this supported many of the attributes that we had within the
Usability pillar, it also identified issues. We made amendments based on these, and undertook a second study,
this time evaluating 4 CTAs used in other countries. This has led to the completion of the Usability pillar,
which can now be used to evaluate global CTAs.
1 INTRODUCTION
As the novel coronavirus (Covid-19) spreads across
the globe with limited treatments and no vaccine as
yet discovered, governments and public health
institutions look at a series of non-pharmaceutical
intervention strategies to limit the spread of the virus.
In this context, contact tracing (CT) is seen as a
promising strategy to identify, isolate and contain
outbreaks. Although traditionally a manual process,
digital contract tracing applications (CTA) have been
proposed to leverage the pervasive use of smartphone
devices and increase the accuracy of the CT process.
This paper reports on progress in the
‘COVIGILANT’ project, a project aimed at
developing a compare and contrast evaluation
framework for CTAs which would help to improve
a
https://orcid.org/0000-0003-1601-0900
b
https://orcid.org/0000-0002-1686-7413
c
https://orcid.org/0000-0002-5493-2837
d
https://orcid.org/0000-0001-6928-6746
existing solutions (Buckley et al., 2020). For this
purpose, the COVIGILANT project aims at better
understanding the population’s perception of Contact
Tracing Apps (O’Callaghan et al., 2020) in the Irish
context and to develop a taxonomy to compare and
contrast the different aspects of contact tracing apps
developed globally. The COVIGILANT taxonomy
currently includes 7 pillars which were developed by
the project team as aspects to be examined when
evaluating CTAs using compare and contrast:
Characteristics, Effectiveness, Performance, User
Autonomy/Self Determination, Data Protection,
Transparency and Usability. Specifically, within each
pillar, there are attributes, sub-attributes and a
corresponding set of questions which are posed
during CTA evaluation. In this paper, we focus on the
development of the Usability pillar of the
COVIGILANT taxonomy. We use literature to
Storni, C., Tsvyatkova, D., Richardson, I., Buckley, J., Abbas, M., Beecham, S., Chochlov, M., Fitzgerald, B., Glynn, L., Johnson, K., Laffey, J., McNicholas, B., Nuseibeh, B., O’Connell, J.,
O’Keeffe, D., O’Keeffe, I., O’Callaghan, M., Razzaq, A., Rekanar, K., Simpkin, A., Walsh, J. and Welsh, T.
Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability.
DOI: 10.5220/0010307005570565
In Proceedings of the 14th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2021) - Volume 5: HEALTHINF, pages 557-565
ISBN: 978-989-758-490-9
Copyright
c
2021 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
557
develop the pillar, and illustrate its effectiveness by
using it to evaluate 5 existing CTAs which are being
used in different countries.
The first section provides a definition of COVID-
19 CTAs, their types, the international regulations for
their development and design, and limitations found
in the existing frameworks. The second section
reports on the development of the COVIGILANT
taxonomy, specifically its Usability pillar and the
structure of the attributes within. This section also
explains how we tested, reviewed and refined our first
iteration of the Usability pillar. The paper concludes
with a summary of the findings and outlines plans for
future work.
2 CTA FOR COVID-19:
DEFINITION, TYPES, EU
REGULATIONS AND
AVAILABLE FRAMEWORKS
Paper-based CT strategies have existed through
previous pandemics (Swanson et al., 2018). However,
these are intense and require tedious manual entry
which is prone to human error. The use of technology
for tracing activity has shown promising results in the
monitoring process of Ebola (Danquah et al., 2019)
and has generated understandable enthusiasm as a
means to prevent the further spread of COVID-19
(Goggin, 2020). In this section, we first provide a
definition of CTAs and their various types as they are
discussed in the growing academic literature on this
subject. We outline some of the regulations and
principles suggested for their design, discussing
existing frameworks offered for evaluating CTAs.
2.1 Definitions and Different Types of
CTAs
CTAs are mobile applications that are developed for
and used in smartphone technology. According to the
World Health Organisation (WHO) (2020), these
applications are “also known as proximity tracking
tools”, that “use location-based (GPS) or Bluetooth
technology to find and trace the movements of
individuals to identify people who may have been
exposed to an infected person”. Different types of
CTA are being developed. These use different
1
Transparency about the purpose, transparency about the
design choices, transparency about the benefits of using the
application, minimum amount of personal data collection,
protect users by using pseudonymous identifiers, ensure
approaches to contact tracing and different
approaches to data storage, management and usage.
To help classify these approaches, the WHO (2020)
considered two main forms of CTAs: a) centralised
form where the processing of the contact history data
is central, usually undertaken by health authorities
and b) decentralised where the data is kept in the
users’ devices. In both cases, GPS or Bluetooth
technologies are used to collect data and include alert
notification functions to warn users of a potentially
infectious exposure.
2.2 EU Regulations for the
Development and Design of CTAs
To develop effective app solutions, a series of design
requirements were proposed by the European
Commission to ensure that the use of the digital tool:
a) will address accessibility and inclusiveness and be
on a voluntary basis,
b) be approved by the national health authority to be
monitored, and
c) will include forms of encryption for preserving
personal data.
Additionally, once the pandemic is over, the app
can be deactivated and disassembled (eHealth
Network, 2020).
Similarly, the Information Commissioner's Office
(ICO) (2020) published ten core principles
1
to follow
when designing CTAs. These have been grouped to
cover the application development life cycle. Some of
these principles discuss the importance of building
“core requirements” which include “user experience
(UX) design and other appropriate in-app
transparency mechanisms”, and that they “should
adopt a user centric design approach” (ICO, 2020).
However, despite all these regulations, requirements
and principles, the CTAs will be effective only if they
are integrated into the “existing public health system”
and adopted by the majority of the population (WHO,
2020).
2.3 Existing Frameworks for CTAs and
Their Limitations
Our first steps were to look at the available academic
and grey publications discussing evaluation
frameworks for CTAs. A Cho et al. (2020) and De
Carli et al. (2020) offer two frameworks which
users have control over the app, store collected data for a
minimum amount of time as users have control over it,
encryption of data, voluntary participation and ensure
privacy and security.
HEALTHINF 2021 - 14th International Conference on Health Informatics
558
primarily focus on privacy issues. Gasser et al. (2020)
propose a framework whose scope is the ethical and
legal challenges of CTAs, while Dar et al. (2020) look
mainly at feasibility and effectiveness aspects.
Despite certain differences, these frameworks
commonly highlight the specific importance of
privacy and data protection in CTAs, and this led us
to acknowledge that we need a specific pillar about
this aspect in our taxonomy. Only the assessment
framework for mobile and web-based applications
developed for COVID-19 suggested by Vokinger et
al. (2020) includes usability aspects. Vokinger et al.’s
framework is based on “an existing trustworthiness
checklist for digital health applications” and
comprises eight domains (i.e., Purpose, Usability,
Information Accuracy, Transparency, Organisational
Attributes/Reputation, Privacy and User
Control/Self-Determination) (2020). The authors,
however, also highlighted the following limitations:
1) The framework was specifically designed to the
Swiss privacy law aspects which stress
transparency and user privacy but limits it to the
Swiss context, and
2) Usability criteria and technical characteristics of
CTAs are included but not developed sufficiently.
We have identified this as a research gap, and our
research question for this paper is:
How can Usability attributes be integrated into a
taxonomy through which CTAs can be compared,
contrasted and possibly improved?
3 RESEARCH METHOD:
CREATING AND ASSESSING
USABILITY PILLAR FOR CTA
To address the limited scope of the CTA frameworks
we initially reviewed with respect to Usability, and to
further develop a Usability pillar, three researchers
assessed three further types of literature in detail.
First, we looked at the most recent guidelines for
mobile apps (Alturki & Gay, 2019; Goel et al., 2018;
Shitkova et al., 2015; Weichbroth, 2020), and
specifically for mobile health apps (e.g., mHealth)
(Kasali et al., 2019; Kaur & Haghighi, 2016; Xcertia,
2019). This literature is important because it
overviews a number of Usability attributes and
guidelines. For instance, 75 attributes for the
Usability of mobile applications were distinguished
and analysed through a systematic literature review
by using only the Scopus database (Weichbroth,
2020). The article discussed only the attributes that
appeared more than once (N=24). Xcertia (2019)
provides guidelines on five key areas of Privacy,
Security, Usability, Operability and Content to ensure
that mobile health apps cover the needs of clinicians
and end-users. We also searched more traditional
literature on Usability including: the Usability
standards ISO 9241-11, Nielsen Usability framework
(Nielsen, 1993), Preece et al. (2015) Usability
taxonomy (1998/2019) and notions from Norman’s
psychology (Norman, 2013) of everyday objects
(1998/2012, e.g., Affordances, Constraints and
Mapping). This literature gave us a rich range of
notions to focus on for the development of a Usability
pillar (e.g., User Satisfaction is found in all mentioned
frameworks, Efficiency and Learnability are
mentioned on most of them). Eventually, we also
acknowledged the key importance of Accessibility
which is often included as an aspect of Usability and
which led us to further review the literature and
standards on Accessibility (Ballantyne et al., 2018;
ETSI, 2018; EU, 2016) and Universal Design (UD)
of mHealth apps (Harrington et al., 2017; Kascak et
al., 2014). That literature includes design
requirements for users with various type of
impairments (e.g., visual, motor, cognitive), but also
‘special’ categories of users such as the elderly. This
initial review helped us to identify an initial list of 53
attributes relevant to CTA, and Table 1 shows all the
used sources (first column) and attributes (second
column).
The initial list of attributes was discussed by a
subset of the COVIGILANT team (8 researchers),
and against the 7 initial pillars for the COVIGILANT
taxonomy: Characteristics, Usability, Effectiveness,
Performance, User Autonomy/Self Determination,
Data Protection and Transparency. We acknowledge
that some issues (which in a different context could
be treated under Usability) deserved a more specific
focus in CTAs. Based on this consideration, aspects
such as Effectiveness, Performance, User Autonomy
and Transparency were developed as pillars on their
own, independent from the Usability group, by other
members of the team. In line with existing CTA
frameworks, Privacy and Data Protection were also
treated independently.
Seven of the discussed attributes were specific to
Usability and were used as starting points for the
development of the Usability pillar (version 1.0)
(Figure 1). These were:
1) Subjective Satisfaction: “refers to how pleasant
it is to use the system” (Nielsen, 2009);
2) Availability (New): inequalities in the access
based on internet connectivity and platform
dependency;
Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability
559
Table 1: An overview of the most relevant attributes identified in academic publications.
Frameworks Proposed attributes
Contact tracing apps for COVID-19
(Vokinger et al., 2020)
Purpose, Usability, Information Accuracy, Organisational Attributes
/Reputation, Transparency, Privacy, User Control/Self-Determination
mHealth
(Kasali et al., 2019; Kaur & Haghighi, 2016;
Xcertia, 2019)
Efficiency, Satisfaction, Effectiveness, Learnability, Memorability,
Cognitive Load, Simplicity, Universality, Aesthetics,
Security, Usefulness, Resources, Troubleshooting,
Ongoing App Evaluation, Name of the Application
UD for mHealth apps for older individuals
(Harrington et al., 2017; Kascak et al., 2014)
Navigation, Affordances, Interaction, Equitable Use, Flexibility,
Ease of Use, Errors, Low Physical Effort,
Size and Space for Approach and Use
Accessibility
(Ballantyne et al., 2018)
Perceivable, Operability, Understandability, Robustness,
Design, System, Content
Mobile applications
(Alturki & Gay, 2019; Goel et al., 2018;
Shitkova et al., 2015; Weichbroth, 2020)
Attractiveness, Comprehensibility, Accessibility, Consistency,
Training, Trust, Battery Consumption, Less Storage Consumption,
Adaptability, Performance, Layout, Platform Dependency,
Onboarding, Speed, ISO 9241-11 (Efficiency, Satisfaction and
Effectiveness)
Figure 1: Visualisation of the initial structure of the Usability pillar (version 1.0).
3) Accessibility: design guidelines for individuals
with cognitive, visual, hearing and dysmotility;
4) Flexibility: allow users to perform tasks in
various ways to accelerate their performance;
5) Effectiveness: the ability of a user to complete a
task to achieve their goals;
6) Interaction: design of the User Interface (UI) and
user interactions;
7) Ongoing App Evaluation: presence of user
research and iterative evaluations after the app
was released.
We included Accessibility under Usability.
Furthermore, we found that Platform Dependency
and Internet Connectivity are also elements of
Accessibility to be accounted for as CTAs might not
work in all smartphones and typically require an
active connection to be installed during setup. To
accommodate this, we included an ‘Availability’
category under Accessibility. Satisfaction was
changed to ‘Subjective Satisfaction’ adhering to
Nielsen's definition (2009).
HEALTHINF 2021 - 14th International Conference on Health Informatics
560
After identifying these seven high-level
dimensions, our job was to again review the literature,
so that we could populate these high-level categories
with attributes, sub-attributes and specific questions
to be asked when using our Usability pillar to assess
existing CTAs. For each attribute and sub-attribute, a
definition was given (based on that literature) and at
least one probing question (also based on literature
when this is available, or freely formulated by us
otherwise). We included a commentary on the
feasibility of such questions, detailing how one could
possibly answer them (e.g., the Devil’s Advocate
approach). This ensured that, not only had we
identified probing questions, but they could be
answered. This resulted in the top seven attributes
being expanded into specific sub-attributes with 77
related questions. A subset of our results is shown in
Table 1. To validate the Usability pillar, we tested
initial list of questions by applying them to the Irish
Health Services Executive (HSE) ‘COVID Tracker’
2
contact tracing app.
3.1 First Assessment of Usability Pillar:
HSE App Assessment
Our initial draft on the Usability pillar was tested with
the Irish HSE app, and a series of issues both in the
structure and in the questions were immediately
apparent:
1. The usability structure looked quite unbalanced
and disproportionate as:
a) two attributes had many sub-attributes (i.e.,
Accessibility and Interaction)
b) three attributes had no sub-attributes (i.e.,
Flexibility, Effectiveness and Subjective
Satisfaction)
c) some of the attributes (i.e., Accessibility and
Interaction) had sub-attributes with the same
labels (i.e., User Interaction and User Interface)
2. A few of the attributes were found too broad as:
a) Effectiveness is discussed in two places; as a
pillar of the COVIGILANT taxonomy, and as
one of the attributes in the Usability pillar
b) Accessibility included three sub-attributes (i.e.,
Functional Performance Statements, User
Interface and User Interaction)
c) Subjective Satisfaction was found to be too
subjective; it required better-formulated
questions
3. Many of the attributes and their subsequent
questions helped to identify more than one issue,
indicating:
2
https://covidtracker.gov.ie/
a) wrong positions of the attributes (i.e.,
Accessibility and Flexibility) and/or questions
(i.e., needed to relocate the question in the sub-
attribute Low Physical Effort)
b) not well-formulated questions (i.e., question
used for the sub-attribute User Interaction)
4. Most of the questions were developed for the
Interaction attribute. This provoked further
discussions on criteria for inclusion and exclusion
based on the assessment approach (e.g., heuristic
evaluation, should some attribute in Usability be
task-based rather than general about the whole
CTA?)
Table 2 presents a short summary of the issues
found with the appropriate solutions.
To achieve a more balanced structure on Usability
(version 2.0), we address the issues uncovered by
testing version 1.0 with the Irish HSE app (Table 2),
but we also refined the taxonomy based on the work
of Alonso-Ríos et al. (2010). We found this work to
be very helpful. Their work overviews and compares
existing Usability taxonomies towards an integrative
framework offering an exhaustive description and
definition of Usability attributes. It offered the
opportunity to clarify our terminology (e.g., the
difference between clarity and consistency) and to
expand and rearrange our categories to develop a
hierarchical structure of Usability, removing
redundancy and repetitions. We made several
changes in the Usability structure and questions:
1. ‘Effectiveness’ and ‘Interaction’ attributes were
re-labelled respectively into Design Effectiveness
and User Interaction (Tasks Specific). Each of
these attributes has four sub-attributes. For the
former, these attributes are covering the capacity
of the system, interface, and interaction design
(i.e., Completeness, Configurability, User
Interface and Helpfulness). For the latter, they are
looking at the interaction with the app UI in
performing a particular task (i.e., Efficiency,
Robustness, Clarity of Interaction with Elements
and Consistency of Interaction with Elements).
2. We created two new sub-attributes for Subjective
Satisfaction (i.e., Motivations for High Score and
Motivations for Low Score).
3. Universality is a new attribute; it refers to the
ability of the app to be used by different users
(different because of impairments or culture); it
has two sub-attributes Accessibility and Cultural
Universality.
4. Questions were further refined in a bottom-up
fashion. This approach helped to identify a limited
number of specific questions related to the
Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability
561
attributes which were more effective for doing a
CTA review.
We undertook a second validation phase, using the
Usability pillar again to evaluate 4 further CTAs more
specifically (e.g., individual questions picked up
unique CTA issues), and categories and definitions
were clearer.
3.2 Four CTAs Assessments: Results
and Amendments to the Usability
Pillar
We identified four CTAs
3
with different characteristics
and used the Usability version 2.0 against them. These
CTAs were developed for health authorities and users
in EU and non-EU countries. They were selected on
the basis that they had different characteristics, for
example, content, user interface design and
navigational elements. This variety allowed us to test
the Usability pillar structure and questions to detect
where improvements are possible or needed. We found
these activities very useful in two ways. First, it
particularly helped to add/improve some of the
questions we had under Accessibility (i.e., for the input
fields and activation of accessibility technologies in the
device settings) and it helped in the Universality
attribute for the Language subsection. Second, it
helped to identify new sub-attributes to be added to the
pillar such as Notifications, Content and Age/Parental
Control (Figure 2).
Table 2: Examples of issues identified in the usability structure and questions (version 1.0) along with the appropriate
solutions.
Attribute/
Sub-attributes/Questions
Refinement issues Solutions
Attribute:
Effectiveness
a) Use the same label with a pillar examining
the effectiveness of contact tracing function
b) Does not include sub-attributes
a) Re-label as a Design Effectiveness
b) Create four sub-attributes, i.e., Completeness,
Configurability, User Interface and Helpfulness.
Attribute:
Accessibility
a) Clashing with Universality
b) Includes many sub-attributes (i.e.,
Functional Performance Statements, User
Interface and User Interaction) overlapping
with more traditional Usability issues
a) Create a new attribute Universality with two sub-
attributes, i.e., Accessibility and Cultural Universality
b) Create three sub-attributes for Accessibility, i.e.,
Functional Performance, UI Elements, Accessible
Interactions
Attribute:
Interaction
a) Include most of the questions developed for
the Usability pillar
b) Refers to the actual interaction with an
interface in the execution of specific tasks
a) Re-label as User Interaction (Tasks Specific)
b) Create four sub-attributes, i.e., Efficiency,
Robustness, Clarity of Interaction with Elements and
Consistency of Interaction with Elements
Sub-attribute:
User Interaction
a) Appears in two attributes, i.e., Accessibility
and Interaction
b) Refers to the actual interaction with an
interface in the execution of specific tasks
a) Create only one main attribute, i.e., User
Interaction (Tasks Specific) that includes four sub-
attributes, i.e., Efficiency, Robustness, Clarity of
Interaction with Elements and Consistency of
Interaction with Elements
Sub-attribute:
User Interface
a) Appears in two attributes, i.e., Accessibility
and Interaction
b) Refers to the User Interface elements and
how accessible they are designed
a) Re-label as UI Elements
b) Relocate the sub-attribute in the attribute
Universality, sub-attribute Accessibility
Question:
Q1: Is multimodality
offered?
a) Question assesses more than one issue, i.e.,
Accessibility and User Interaction
b) Current position - in the attribute
Interaction, sub-attribute User Interaction
c) Refers to the multiple modes available for
interaction, i.e., Flexibility/Multimodality
a) Create a new sub-attribute with a label, i.e.,
Flexibility/Multimodality, attribute User Interaction
(Tasks Specific)
b) Relocate the question in the sub-attribute
Flexibility/Multimodality,
c) Formulate a better question, i.e.,
Q1: Can the same tasks be executed in different
ways?
Question:
Q1: Can a user complete
a task without scrolling?
a) Current position - in the sub-attribute
Navigation, attribute Accessibility
b) Refers to the reduction in physical effort,
i.e., sub-attribute Low Physical Effort
a) Relocate the question in the sub-attribute Low
Physical Effort, sub-attributes Accessibility and
Accessible Interactions
3
AMAN https://amanapp.jo/en
Corona-Warn-App https://www.coronawarn.app/en/
NOVID https://www.novid.org/
PathCheck SafePlaces https://pathcheck.org/
HEALTHINF 2021 - 14th International Conference on Health Informatics
562
Figure 2: Visualisation of the final Usability pillar (version 2.0) after application to the four CTAs.
Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability
563
For example, to support individuals who have
various limitations and disabilities, some of the CTAs
use various options for the input fields used for self-
reporting of a positive COVID-19 diagnosis, e.g.,
manually entering the data, voice control and QR
code. Likewise, promoting accessibility, it was found
that all apps support more than one spoken language
where selection was either in the app during the
activation process, in the app settings or in the device
system when picking up the preferred language.
Questions were improved to cover and assess the
availability of such options in the apps. In addition, a
few questions were added to assess whether the app
design accommodates the OS-level changes from the
device settings (e.g., font, colour and contrast) and the
accessibility features after they are activated from the
device settings (e.g., screen readers). We also
acknowledge age restriction as an important aspect of
CTAs. Only the Irish HSE app offers age restrictions
during the activation process, the other CTAs we
analysed describe these restrictions in the section
'Terms of Use'. To frame this aspect, we followed the
guidelines for inclusiveness (eHealth Network,
2020), and we added a new subgroup 'Age/Parental
control'.
The implementation of these amendments
brought us to finalise version 2.0 of our Usability
pillar, which we will continue to assess both
internally, by addressing redundancies and relocation
of attributes within the 7 pillars, and externally, by
continuing to test the Usability structure and
questions with real CTAs to stress its scope and
limitation.
4 USABILITY PILLAR: FUTURE
WORK
We presented the research question: How can
Usability attributes be integrated into a taxonomy
through which CTAs can be compared, contrasted
and possibly improved? Our approach was to draw on
different sources for design requirements (health
apps, mobile apps, accessibility guidelines (noting
particularly the elderly who are an at-risk category for
COVID-19)). This top-down approach, drawing on
existing literature and helping us to identify 53 initial
attributes, was complemented with bottom-up,
iterative-refinement of Usability version 1.0.
Through an initial test using the Irish HSE COVID-
19 CTA, we identified both the questions that worked
and the limitations in the initial structure and
questions. We undertook further exploration of
available literature, expanding beyond healthcare,
and proposed a more balanced structure by adding
new attributes and formulating task-specific
questions up to a refined version 2.0 of our Usability
pillar. The same strategy was applied to the other 6
pillars (see Welsh et al., 2020, for work on Data
Protection pillar) included in the COVIGILANT
taxonomy. The next planned activities are focused on
an amalgamation of all pillars, the assessment of the
full set of questions again in order to propose a refined
evaluation framework that can effectively evaluate
various types of CTAs. We are considering how
different questions might concern and be relevant for
different stakeholders in the CTAs ecology, for
example, developers, Health Departments, data
protection authorities. We are exploring how different
‘vistas’ could be created to align with the different
concerns of different stakeholders, thus offering a
tailored subset of pertinent questions to understand
where CTAs could be improved.
ACKNOWLEDGEMENTS
This work was supported, in part, by Science
Foundation Ireland grant 13/RC/2094 and the
COVIGILANT Science Foundation Ireland grant
20/COV/0133.
REFERENCES
Alonso-Ríos, D., Vázquez-García, A., Mosqueira-Rey, E.,
& Moret-Bonillo, V. (2010). Usability: A critical
analysis and a taxonomy. International Journal of
Human-Computer Interaction, 26(1), 53–74.
Alturki, R., & Gay, V. (2019). Usability Attributes for
Mobile Applications: A Systematic Review.
Ballantyne, M., Jha, A., Jacobsen, A., Scott Hawker, J., &
El-Glaly, Y. N. (2018). Study of accessibility
guidelines of mobile applications. ACM International
Conference Proceeding Series, 305–315.
https://doi.org/10.1145/3282894.3282921
Buckley, J., Abbas, M. M., Chochlov, M., Fitzgerald, B.,
Glynn, L., Johnson, K., Laffey, J., McNicholas, B.,
Nuseibeh, B., Connell, J., O’Keeffe, D., O’Keeffe, I. R.,
O’Callaghan, M., Razzaq, A., Rekanar, K., Richardson,
I., Simpkin, A., Storni, C., Tsvyatkova, D., … Welsh,
T. (2020). A Proposal for the SFI Covid-19 Rapid
Response Funding Call. https://tinyurl.com/y4qbvk8c
Cho, H., Ippolito, D., & Yu, Y. W. (2020). Contact Tracing
Mobile Apps for COVID-19: Privacy Considerations
and Related Trade-offs. ArXiv Preprint
Danquah, L. O., Hasham, N., MacFarlane, M., Conteh, F.
E., Momoh, F., Tedesco, A. A., Jambai, A., Ross, D. A.,
& Weiss, H. A. (2019). Use of a mobile application for
HEALTHINF 2021 - 14th International Conference on Health Informatics
564
Ebola contact tracing and monitoring in northern Sierra
Leone: A proof-of-concept study. BMC Infectious
Diseases, 19(1).
Dar, A. B., Lone, H., Zahoor, S., Amin Khan, A., & Naaz,
R. (2020). Applicability of Mobile Contact Tracing in
Fighting Pandemic (COVID-19): Issues, Challenges
and Solutions.
De Carli, A., Franco, M., Gassmann, A., Killer, C.,
Rodrigues, B., Scheid, E., Schoenbaechler, D., &
Stiller, B. (2020). WeTrace -- A Privacy-preserving
Mobile COVID-19 Tracing Approach and Application.
1–15. http://arxiv.org/abs/2004.08812
eHealth Network. (2020). Mobile applications to support
contact tracing in the EU’s fight against COVID-19:
Common EU Toolbox for Member States.
ETSI. (2018). ETSI EN 301 549 - V2.1.2 - Accessibility
requirements for ICT products and services. 2, 1–
152.https://www.etsi.org/deliver/etsi_en/301500_3015
99/301549/02.01.02_60/en_301549v020102p.pdf
EU. (2016). DIRECTIVE (EU) 2016/2102 OF THE
EUROPEAN PARLIAMENT AND OF THE
COUNCIL of 26 October 2016 on the accessibility of
the websites and mobile applications of public sector
bodies. Official Journal of the European Union, 327, 1–
15. https://eur-lex.europa.eu/eli/dir/2016/2102/oj
Gasser, U., Ienca, M., Scheibner, J., Sleigh, J., & Vayena,
E. (2020). Digital tools against COVID-19: Framing
the ethical challenges and how to address them.
Goel, S., Nagpal, R., & Mehrotra, D. (2018). Mobile
Applications Usability Parameters: Taking an Insight
View. In Mishra D., Nayak M., & Joshi A (Eds.),
Information and Communication Technology for
Sustainable Development (Vol. 9, pp. 35–43). Springer,
Singapore.
Goggin, G. (2020). COVID-19 apps in Singapore and
Australia: reimagining healthy nations with digital
technology. Media International Australia,
1329878X2094977.
Harrington, C. N., Ruzic, L., & Sanford, J. A. (2017).
Universally Accessible mHealth Apps for Older
Adults: Towards Increasing Adoption and Sustained
Engagement. International Conference on Universal
Access in Human-Computer Interaction, 3–12.
ICO. (2020). COVID-19 Contact tracing: data protection
expectations on app development.
https://ico.org.uk/media/for-organisations/documents/
2617676/ico-contact-tracing-recommendations.pdf
Kasali, F. A., Taiwo, O. O., Akinyemi, I. O., Alaba, O. B.,
Awodele, O., & Kuyoro, S. O. (2019). An Enhanced
Usability Model for Mobile Health Application. 17(2),
20–29. https://sites.google.com/site/ijcsis/
Kascak, L. R., Rebola, C. B., & Sanford, J. A. (2014).
Integrating Universal Design (UD) principles and
mobile design guidelines to improve design of mobile
health applications for older adults. Proceedings - 2014
IEEE International Conference on Healthcare
Informatics, ICHI 2014, 343–348.
Kaur, E., & Haghighi, P. D. (2016). A context-aware
usability model for mobile health applications. ACM
International Conference Proceeding Series, 181–189.
https://doi.org/10.1145/3007120.3007135
Nielsen, J. (1993). Usability engineering. Morgan
Kaufmann. https://doi.org/10.1201/b16768
Nielsen, J. (2009). What Is Usability? In C. Wilson (Ed.),
User Experience Re-Mastered: Your Guide to Getting
the Right Design. Morgan Kaufmann.
Norman, D. (2013). The design of everyday things: Revised
and expanded edition (2nd ed.). Basic Books.
O’Callaghan, M. E., Buckley, J., Fitzgerald, B., Johnson,
K., Laffey, J., McNicholas, B., Nuseibeh, B., O’Keeffe,
D., O’Keeffe, I., Razzaq, A., Rekaner, K., Richardson,
I., Simpkin, A., Storni, C., Tsvyatkova, D., Walsh, J.,
Welsh, T., & Glynn, L. (2020). A National Survey of
attitudes to COVID-19 Digital Contact Tracing in the
Republic of Ireland. https://doi.org/10.21203/rs.3.rs-
40778/v1
Preece, J., Sharp, H., & Rogers, Y. (2015). Interaction
design: beyond human-computer interaction. John
Wiley & Sons. Chicago.
Shitkova, M., Holler, J., Heide, T., Clever, N., & Becker, J.
(2015). Towards Usability Guidelines for Mobile
Websites and Applications. Wirtschaftsinformatik,
1603–1617.
Swanson, K. C., Altare, C., Wesseh, C. S., Nyenswah, T.,
Ahmed, T., Eyal, N., Hamblion, E. L., Lessler, J.,
Peters, D. H., & Altmann, M. (2018). Contact tracing
performance during the Ebola epidemic in Liberia,
2014-2015. PLoS Neglected Tropical Diseases, 12(9).
https://doi.org/10.1371/journal.pntd.0006762
Vokinger, K. N., Nittas, V., Witt, C. M., Fabrikant, S. I., &
von Wyl, V. (2020). Digital health and the COVID-19
epidemic: an assessment framework for apps from an
epidemiological and legal perspective. Swiss Medical
Weekly, 150(1920), w20282.
Weichbroth, P. (2020). Usability of mobile applications: A
systematic literature study. IEEE Access, 8, 55563–
55577.
Welsh, T., Rekanar, K., Abbas, M., Chochlov, M.,
Fitzgerald, B., Glynn, L., Johnson, K., Laffey, J.,
McNicholas, B., Nuseibeh, B., O’Connell, J., O’Keeffe,
D., O’Keeffe, I. R., O’Callaghan, M., Razzaq, A.,
Richardson, I., Simpkin, A., Storni, C., Tsvyatkova, D.,
… Buckley, J. (2020). Towards a Taxonomy for
Evaluating Societal Concerns of Contact Tracing Apps.
To Appear In 2020 7th International Conference on
Behavioral, Economic, and Socio-Cultural Computing
(BESC).
WHO. (2020). Digital tools for COVID-19 contact tracing:
annex: contact tracing in the context of COVID-19, 2
June 2020. June. https://www.who.int/publicati
ons/i/item/WHO-2019-nCoV-Contact_Tracing-Tools_
Annex-2020.1
Xcertia. (2019). Xcertia mHealth App Guidelines. Xcertia.
https://www.himss.org/sites/hde/files/media/file/2020/
04/17/xcertia-guidelines-2019-final.pdf
Toward a Compare and Contrast Framework for COVID-19 Contact Tracing Mobile Applications: A Look at Usability
565