“The Algorithm Will See You Now”: Exploring the Implications of

Algorithmic Decision-making in Connected Health

Noel Carroll

1,2

, Ita Richardson

1,3,4

and Raja Manzar Abbas

1,3

Lero - The Irish Software Research Centre, Ireland

Business Information Systems, NUI Galway, Galway, Ireland

Department of Computer Science and Information Systems, University of Limerick, Limerick, Ireland

ARC – Ageing Research Centre, HRI- Health Research Institute, University of Limerick, Limerick, Ireland

Keywords: Algorithm, Algorithmic Decision-making, Connected Health, Artificial Intelligence.

Abstract: Despite abundant literature theorizing on Connected Health innovations to support decision-making, the

extant literature provides sparse coverage on users’ awareness of algorithmic decision-making. As a result,

little is known regarding the role of algorithmically generated insights which directly influence clinical

decisions nor the consequences of distancing clinicians and patients from decision-making capabilities.

Indeed, recent studies highlight the growing emphasis on algorithmic decision-making but there is a need to

raise questions as to how this is impacting on the risk and quality of delivering care. In this article, a summary

of key concerns from the literature is provided, and a discussion on the implications of algorithmic decision-

making in Connected Health is presented. In addition, a research roadmap is presented to draw more research

focus on the role of algorithmically generated insights in Connected Health.

1 INTRODUCTION

Fuelled by ongoing discussions on advancements in

technology and data science to facilitate algorithmic

decision-making, there is a growing body of literature

that expresses the need to explore its implications.

Burrell (2016; p.1) argues that “opacity seems to be

at the very heart of new concerns about ‘algorithms’

among legal scholars and social scientists” but we

need to focus the discourse on getting inside the

algorithms themselves.

An algorithm may be defined as a process or set

of rules to calculate or solve a problem which is

typically carried out by a computer. Therefore,

algorithms can be viewed as a set of step-by-step

instructions to achieve a desired result in a finite

number of moves (Orlikowski and Scott 2015) which

act on data. Using data as input, algorithms produce

an output; for example, a risk classification for a loan,

or whether an email should be considered as spam.

Burrell (2016; p.1) explains that algorithms “are

opaque in the sense that if one is a recipient of the

output of the algorithm (the classification decision),

rarely does one have any concrete sense of how or

why a particular classification has been arrived at

from inputs”. Indeed, technological advances

continues to mask our ‘black box society’ through

powerful yet unnoticed algorithms that control how

data is collected and processed to present information

and ultimately influence decision-making.

While there has been much speculation around the

role of technology giants in recent years in collecting

all kinds of data about citizens and how the so-called

surveillance capitalism is creating another layer of

secrecy across society and trust in software solutions,

awareness for similar implications in algorithmic

decision-making goes undocumented. Yet, the

implications of ubiquitous and pervasive digital

technologies for healthcare and public health are

profound (Lupton 2014).

In this article, we examine some of the

assumptions around the role of algorithmically

generated insights which influence healthcare

decisions through Connected Health innovation.

2 CONNECTED HEALTH

Connected health is a socio-technical model for

healthcare management and delivery by using

technology to provide healthcare services. The

Connected Health phenomena (Carroll, 2016; Carroll

758

Carroll, N., Richardson, I. and Abbas, R.

“The Algorithm Will See You Now”: Exploring the Implications of Algorithmic Decision-making in Connected Health.

DOI: 10.5220/0009178507580765

In Proceedings of the 13th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2020) - Volume 5: HEALTHINF, pages 758-765

ISBN: 978-989-758-398-8; ISSN: 2184-4305

et al. 2016a) have continued to grow for several

reasons including, the growth in mobile phone

devices, increased availability of mobile applications

(‘app’) technologies, improved Internet connectivity,

growing availability of personal and health-related

data, and growing pressures on healthcare providers

to seek alternative means to deliver healthcare

services. Thus, ‘Connected Health’ is a term that is

used to encompass the wide range of technologies

that are used for healthcare such as digital health

(Lupton, 2014), eHealth (Oh, 2005), mHealth

(Gagnon et al. 2015), health informatics (Coiera,

2015) and health education (Glanz, 2008).

Connected Health technologies are now explicitly

designed for medical and health purposes (Carroll and

Richardson, 2016a), contributing to the digital health

phenomenon that has recently emerged and many of

which are controlled by specific algorithmic decision-

making techniques. Connected Health technologies

may bring other benefits, such as standardized care,

and greater control over the delivery of care.

Algorithmic decision-making through Connected

Health technologies give significant benefits, ranging

from improved diagnosis, thereby delivering better

patient care and improving the support of clinical

decision-making. This enhances hospital

productivity, lowers costs, and reduces medication

errors (Aron et al., 2011). Algorithmic decision-

making in healthcare ought to be viewed as critically

important whereby decision-making can have

significant consequences, including potentially fatal

outcomes, on the quality and safety of care.

In essence, Connected Health is far reaching and

focuses on the convergence of digital technologies

with health, healthcare, living, and society to enhance

the efficiency of healthcare delivery and make

medicine more personalized and precise. To enhance

the efficiency of healthcare delivery, this places more

emphasis on the role of algorithms to gather data and

encapsulate a process or set of rules to be followed in

calculations or other problem-solving operations

(Gruber, 2019), for example, through Connected

Health technologies, yet little research has explored

the implications of algorithmic decision-making in

Connected Health and its impact on decision-making.

2.1 Decision Making in Healthcare

The crucial element of high-quality care in healthcare

is the accuracy, efficacy, and expediency of clinical

decision-making. It is important, therefore, that we

understand both its importance and the range of

strategies that are used to make decisions (Croskerry,

2002). Technological advances have encouraged the

development of new technologies that drive

connectivity across the healthcare sector such as

software apps, gadgets and systems that personalise,

track, and manage care using just-in-time information

exchanged through various patient and community

connections (Leroy et al., 2014; Carroll, 2016). This

paradigm shift has contributed to advancing

healthcare practice, highlighting our growing reliance

and need for algorithmic decision-making to support

healthcare decisions due to technological

advancements such as with artificial intelligence (AI).

For example, research explains how the performance

of AI algorithms can be highly dependent on the

population used in the training sets (for example,

algorithm training and testing for cancer screening) to

ensure that the results are broadly applicable (Topol,

2019).

The outcome of algorithmic decision has

significant implications when individual find

themselves at an intersection of medicinal

possibilities, diverging pathways have extraordinary

and significant results with lasting ramifications.

These include, for instance, decision-making in major

surgeries, prescriptions to be taken for the rest of a

patient’s life, and screening and symptomatic tests

that can trigger upsetting interventions. But, while

there have been benefits, there have been several high

profiles and costly technology failures within

healthcare in recent years, leading to the importance

of having a published and defined algorithmic

decision-making structure to decrease the risk of

failures (Lepri et al. 2017).

3 ‘BLACK BOX SOCIETY’:

ALGORITHMIC DECISION-

MAKING IN CONNECTED

HEALTH

The last decade has witnessed the widespread

diffusion of digitized devices that have the ability to

monitor the minutiae of our everyday lives (Hedman

et al., 2013) enabling data to flow across devices

guided by algorithms to shape information and make

decisions and predictions about individuals by

recognizing complex patterns in complex datasets.

3.1 Promoting the “Right to

Explanation”

Governments have made increasing efforts to protect

citizens’ rights within the digital world. For example,

the European Union’s General Data Protection

“The Algorithm Will See You Now”: Exploring the Implications of Algorithmic Decision-making in Connected Health

759

Regulation (GDPR) provides data protection and

privacy for all individual citizens of the European

Union and the European Economic Area – which

extends to the use of algorithms. Regardless, the use

of algorithmic decisions in an increasingly wide range

of applications has led some scepticism around

technology companies and their growing dominance

in our so-called ‘black-box society’ (Pasquale, 2015).

As a result, there has been growing demands for

increased transparency in algorithmic decision-

making. However, the regulatory requirements

around transparency are often unclear and are open to

some interpretation; for example, in GDPR

(Goodman and Flaxman, 2017). Goodman and

Flaxman (2017) explain that regulation efforts such

as GDPR place restrictions on automated individual

decision-making (that is, algorithms that make

decisions based on user-level predictors) that

“significantly affect” users. GDPR also presents a

requirement on the “right to explanation”. As outlined

in Articles 13 and 14, when profiling takes place, a

data subject has the right to “meaningful information

about the logic involved”. Goodman and Flaxman

(2017) explain that this requirement prompts the

question: what does it mean, and what is required, to

explain an algorithm's decision?

There have been efforts to categorise barriers to

transparency. For example, Burrell (2016)

distinguishes between three broad barriers to

transparency we can associate with algorithms: (1)

opacity as intentional corporate or state secrecy, i.e.

where decision-making procedures are kept from

public scrutiny (2) opacity as technical illiteracy, i.e.

simply having access to underlying code is

insufficient, and (3) opacity that arises from the

characteristics of machine learning algorithms and

the scale required to apply them usefully, i.e. a

mismatch between human reasoning and styles of

interpretation and machine capabilities. Such barriers

to transparency have far reaching consequences

across many sectors, especially in healthcare, where

one can view it as an evolving critical decision-

making system whereby decision often leads to “life

or death” outcomes.

3.2 Connected Health as a Safety

Critical Decision-making System

Critical systems are systems where failure or

malfunction will lead to significant negative

consequences (Lyu, 1996). These systems may have

strict requirements for security and safety to protect

the user or others (Leveson, 1986). By safety critical,

we refer to system failure may lead to loss of life or

serious personal injury. This is particularly important

within a healthcare context where we may view

Connected Health as an extension of a safety critical

decision-making system.

Within the healthcare sector, the complexity of

delivering healthcare services is becoming less clear.

This presents new implications for patients,

clinicians, and the wider society, given that decisions

are increasingly automated, and decision-making

algorithms may not always be transparent. Within a

healthcare system, many decisions are made by

human beings as a result of interpreting medical data

and images generated by computer algorithms.

Healthcare innovations such as Connected Health

promise to increase accuracy and reduce human bias

in important decisions. Specifically, within a

Connected Health context, algorithmic decision-

making occurs when data are collected through

digitized devices carried by individuals such as

smartphones and technologies with inbuilt sensors

built – and subsequently processed by algorithms,

which are then used to make (data-driven) decisions.

Decisions are typically based on relationships

identified in patterns of data - yet decision-makers

often ignore or are not fully aware of why such

relationships may be present (Mayer-Schonberger

and Cukier, 2013). In addition, non-clinical

professionals, such as software engineers and data

scientists are typically tasked with developing

Connected Health solutions. Without clinical input,

they may be misinformed on healthcare best practice,

medicine management, or simply reinforce existing

biases and disparities under the guise of algorithmic

neutrality (Carroll and Richardson, 2016a).

3.3 Connected Health Data-driven

Decisions

In Connected Health, the phrase “data-driven

decision-making” is used, which often alludes to

describe how healthcare organisations integrate

objective information to inform and improve all sorts

of decisions. Data-driven decisions made through

digital devices are promoted and trusted on the basis

of providing personalised insights on individual

behaviour and health status. They also result in the

narrowing of their choices while the diffusion of

Connected Health devices become normalised across

society. However, as clinical practice increasingly

adopts Connected Health solutions, we continue to

distance clinicians and patients form the mechanics of

the decision-making process. Thus, there is a growing

power of the algorithm to influence the provision of

care, for example, whereby (often untrained) software

HEALTHINF 2020 - 13th International Conference on Health Informatics

760

developers and data scientists play a key and

influential role in the provision of healthcare. This

raises significant concerns and there is a need to re-

examine assumptions around this. Furthermore,

ethics and potential unwanted consequences must be

considered.

3.4 Risks of Algorithmic

Decision-making

Research indicates that users expect that algorithms

will help human decision-makers to avoid their own

prejudices by adding consistency to make decisions

(Zerilli et al. 2018). However, algorithms introduce

new risks which often go undocumented as more

focus is placed on the Connected Health innovation

and devices. Algorithms can replicate institutional

and historical biases, amplifying disadvantages

lurking in data points like such as health status ratings

or scores (Coiera, 2019; Ransbotham et al., 2016).

Even if algorithms remove some subjectivity from the

care pathways, humans are still very much involved

in final decisions. Arguments that cast “objective”

algorithms as fairer and more accurate than fallible

humans fail to fully recognise that in most cases, both

play a role particularity in safety critical healthcare

decisions.

On a wider societal level, and arguably a less

critical impact, another element of Connected Health

promotion that has been led by consumers is self-

tracking and new innovations which support mobile

devices and associated software that can monitor and

measure many aspects of bodily functions and

activities and geolocation details. Algorithmic

decision-making across Connected Health devices

reports on a myriad of body functions, sensations and

indicators ranging from blood glucose, body weight

body mass index and physical activity, which are

monitored through wearable and internal sensors and

collected to support decision-making processes. Yet,

algorithmic decision-making means that

discriminations are increasingly being made by an

algorithm, with few individuals actually

understanding what is included in the algorithm or

even why. Criado Perez (2019) argues, for example,

that many such algorithms are based on men-only

data, although outputs are used by women. In other

words, it is seen as being sufficient that an algorithm

is successfully predictive, never mind if the reasons

for the associations found in the data from different

sources are unknown. We argue that this is likely to

create problems when no one in a healthcare system,

for example, a hospital context, really understands

why some decisions are made nor how they were

influenced based on Connected Health algorithms.

4 THE NEXT WAVE OF

ALGORITHMIC

DECISION-MAKING IN

CONNECTED HEALTH

The importance of exploring the implications of

algorithmic decision-making in Connected Health

will be further realised by the growth of AI and

machine learning. AI refers to the simulation of

human intelligence in machines that are programmed

to think like humans and mimic their actions such as

learning and problem-solving. Machine learning is an

application of AI that provides systems with the

ability to automatically to learn from experience.

Therefore, machine learning allows systems to adjust

to new inputs and perform human-like tasks through

algorithms and statistical models that perform a

specific task without using explicit instructions and

rely on patterns and inference instead.

Hollis et al. (2019) explains that with more health

data availability, and the recent developments of

efficient and improved machine learning algorithms,

there is a renewed interest for AI in healthcare. In

general, the objective of adopting AI in healthcare is

to help health professionals improve patient care

while also reduce costs. However, the other costs of

AI, including ethical issues when processing personal

health data by algorithms, should be considered

(Hollis et al. 2019).

AI and machine learning are continuing to

currently dominate research efforts in healthcare

(Gruber, 2019), from planning to care pathway

recommendations to predictive analytics. There are

emerging research trends which demonstrate how

healthcare providers can exploit the use of AI for

getting routine results at a faster rate, health insurers

can better understand risk assessments, and respond

to patient contact call centres and automate drug

dispensary in hospital pharmacy. However,

algorithms are often trained on “data sets that are

riddled with data gaps” (Criado Perez, 2019, p xii),

and as they are often ‘black-box’ systems, users

cannot identify nor take these into account when

supporting our decisions. While technological

advancements such as AI present a new

transformative power in algorithmic decision-

making, research calls for a regulatory framework

related to Software-as-a-Medical-Device (Carroll and

Richardson, 2016b). This is important to provide a

“The Algorithm Will See You Now”: Exploring the Implications of Algorithmic Decision-making in Connected Health

761

new approach for Connected Health technologies to

offer a more tailored fit to healthcare needs.

5 A RESEARCH ROADMAP

This section summarises some of the main research

gaps on the implications of algorithmic decision-

making in Connected Health and presents a research

roadmap. Scholars are encouraged to consider the

following seven key research themes, namely: (i)

ethical implications; (ii) open science implications;

(iii) accountability and responsibility implications;

(iv) ageing implications; (v) data literacy

implications; (vi) healthcare professional skills gap;

and (vii) regulatory implications associated with

algorithmic decision-making in Connected Health.

5.1 Ethical Implications

Technological advancements such as IoT, AI,

machine learning and cloud technology, has been one

of the most important trends over the past couple

years. AI promises to transform society on the scale

of the industrial, technical, and digital revolutions

before it and will accelerate solutions to large-scale

problems in myriad of fields, including healthcare. In

addition, IoT technologies are increasingly infusing

our lives as the interplay of people, computing, data,

and things is continuously evolving. Thus, IT

innovation has been developing at astonishing speeds

since its inception, often rapidly changing healthcare

in new and quite unexpected forms. This raises new

concerns around ethical implications. For example,

Coiera (2019) explains that while AI will be applied

to classic pattern recognition tasks such as diagnosis

or treatment recommendation, it is likely to be as

disruptive to clinical work as it is to care delivery. In

addition, digital scribe systems that use AI to

automatically create electronic health records

promise great efficiency for clinicians but may lead

to potentially very different types of clinical records

and workflows.

Other examples include radiology, whereby AI is

likely to see image interpretation become an

automated process with diminishing human

engagement. Thus, there needs to more focus on

machine ethics in Connected Health with a view to

investigate the role of artificial moral agents, robots

or artificially intelligent computers that behave

morally or as though moral and indeed challenge the

idea that AI can itself be held accountable. Thus, the

research question arises: What are the ethical

considerations for algorithmic decision-making

within a Connected Health context?

5.2 Open Science Implications

Open Science is the practice of science in such a way

that others can collaborate and contribute, where

research data and other research processes are freely

available, under terms that enable reuse,

redistribution and reproduction of the research and its

underlying data and methods. Amidst these activities

however, it is worth noting that the Open Science

movement is still not universally welcome (Osborne,

2015).

Among the issues raised against Open Science are

worries that the movement can unleash into the public

domain unprecedented amount of materials beyond

our capacity to process them, thereby degrading the

peer review quality and adding more stress on the

discoverability and spread of new knowledge.

However, many scientists surveyed by Mann et al.

(2009), identified with Open Science in principle, but

have not made any action plan to share their data and

software tools because the existing research funding

and evaluation structures offer no incentives to justify

the extra efforts to circulate their resources. Paton and

Kobayashi (2019) explain the ecosystem of software

development, data sharing, education, and research in

the AI community has, in general, adopted an Open

Science ethos that has driven much of the recent

innovation and adoption of new AI techniques.

However, within the healthcare domain, adoption

may be inhibited by the use of “black-box” systems,

where only the inputs and outputs of those systems

are understood, and clinical effectiveness and

implementation studies are missing.

As Connected Health and clinical decision

support systems begin to be implemented in

healthcare systems around the world, further

openness of clinical effectiveness and mechanisms of

action may be required by safety-conscious

healthcare policy-makers to ensure they are clinically

effective in real world use. This leads to a research

question: How can Open Science present an action

plan to be transparent on algorithmic decision-

making within a Connected Health context?

5.3 Accountability and Responsibility

Implications

Accountability serves to ensure responsible

development and use of algorithmic systems such that

they improve human rights and benefit society

(Nissenbaum, 1994). An important difference

HEALTHINF 2020 - 13th International Conference on Health Informatics

762

between transparency and accountability is that

accountability is primarily a legal and ethical

obligation on an individual or organisation to account

for its activities, accept responsibility for them, and to

disclose the results in a transparent manner.

Transparency, logs of data provenance, code changes

and other record keeping are important technical

tools, but ultimately accountability depends on

establishing clear chains of responsibility.

Accountability ultimately lies with a (legal) person

(Cooper, 2011).

Automated decision-making algorithms are now

used throughout industry and government,

underpinning many processes. Given that such

algorithmically informed decisions have the potential

for significant societal impact, software developers

and product managers design and implement

algorithmic systems in publicly accountable ways.

Accountability in this context includes an obligation

to report, explain, or justify algorithmic decision-

making as well as mitigate any negative social

impacts or potential harms.

Koene et al. (2019) present an EU initiative for a

governance framework for algorithmic accountability

and transparency. They describe how algorithmic

systems are increasingly being used as part of

decision-making processes in both the public and

private sectors, with potentially significant

consequences for individuals, organisations and

societies as a whole. Koene et al. (2019) draw focus

on how the same properties of scale, complexity and

autonomous model inference, however, are linked to

increasing concerns that many of these systems are

opaque to the people affected by their use and lack

clear explanations for the decisions they make. This

lack of transparency increases the risk of undermining

meaningful scrutiny and accountability. This is a

significant concern when these systems are applied as

part of decision-making processes that can have a

considerable impact on people’s human rights (e.g.

allocation of health and social service resources). We

posit the research question: What governance

structures are required to ensure accountability and

responsible use of algorithmic systems within a

Connected Health context?

5.4 Ageing Implications

Societal, demographic and economic changes have

encouraged us to reconsider how we deliver health

and social care to older people and their families in

our communities (Carroll et al., 2016b). The

worldwide increase in the ageing population presents

an urgent need for new technologies to improve the

quality of life for the elderly. In recent years we have

seen the rapid development of healthcare

technologies along with the widespread use of the

Internet, mobile technologies, data analytics and

artificial intelligence in healthcare – moving towards

more personalized care. However, we must also

consider how do algorithms consider the age and

changing healthcare needs of patients. Our research

question becomes: How can we ensure that

algorithmic decision-making aligns with the

complexity of longevity, i.e. an ageing population

within a Connected Health context?

5.5 Data Literacy and Intelligence

Data literacy is the ability to read, work with, analyse,

and argue with data. Much like literacy as a general

concept, data literacy focuses on the competencies

involved in working with data. For example, at best

current science on various machine learning methods

described artificial narrow intelligence (ANI), i.e. the

first level of intelligence created by humans. This

implies that algorithms are useful in recognising

patterns and gleaning topics from blocks of text or

deriving the meaning of whole documents from a few

sentences. With increasing efforts to achieve artificial

general intelligence (AGI) to abstract concepts from

limited experience and transferring knowledge

between domains and then moving towards

superintelligence, this has the potential to allow

machines to demonstrate some level of consciousness

(Goertzel, 2014). There are limited studies which

consider how to exploit the rich body of medical

evidence to develop frameworks for conceptualising

the algorithm itself and support clinical teams,

beyond decision support systems (O’Leary et al.

2014). This raises the question: What are the key

requirements between data literacy and intelligence

on algorithmic decision-making for stakeholders

within a Connected Health context?

5.6 Healthcare Professionals Skills Gap

Healthcare education is continually evolving to meet

the global healthcare needs of society. While there are

inherent links between healthcare professionals’

educational development and patient safety, there is

growing concern regarding the mismatch in

healthcare professionals’ technological skills and

how technological innovators are informed of

healthcare needs (Carroll et al. 2018).

There is an opportunity to experiment with

algorithmic decision-making in simulated clinical

learning environments. For example, university-

“The Algorithm Will See You Now”: Exploring the Implications of Algorithmic Decision-making in Connected Health

763

simulated clinical skills laboratories provide a safe

innovation environment for healthcare solution

developers to experiment with implementing new

algorithms to improve healthcare practice. Thus, we

need to understand: How can we develop healthcare

education and training on algorithmics decision-

making for Connected Health technology solutions?

5.7 Regulatory Implications

Connected Health is a rapidly developing field never

before witnessed across the healthcare sector. It has

the potential to transform healthcare service systems

by increasing its safety, quality and overall efficiency

(Kvedar et al., 2014). However, as medical devices

and algorithms continuously rely more on software

development, one of the core challenges is examining

how Connected Health is regulated – often impacting

Connected Health innovation adoption and usage.

Many of these regulatory developments fall under

“medical devices”, giving rise to Software-as-a-

Medical Device (SaaMD) yet we need to re-examine

how regulation governs the development and usage of

algorithms which guide decision-making processes in

practice, for example, the role and impact of GDPR

as a requirement on the “right to explanation” in

Connected Health innovation. We posit the research

question: What are the key regulatory requirements

for algorithmic decision-making for software-as-a-

medical device within a Connected Health context?

6 DISCUSSION & CONCLUSION

The future of Connected Health aims to apply data

sciences, machine learning, AI and IoT to tackle the

health problems and challenges faced by patients and

the care professionals. For example, tracking

personalized health indicators regularly such as blood

pressure and heart rate can help with the management

of the health and well-being of patients with heart

issues.

New technologies developed in the digital

industry, particularly in the emerging interfacing area

between big data and AI, are changing the way

healthcare delivery is decided upon and can have an

enormous economic impact on healthcare provision.

We are witnessing growing research efforts in

healthcare in the development of new smart sensing,

new algorithms, and new systems or devices for

personalised healthcare. One of the fundamentals of

these developments is to ensure that healthcare data

can be accessed and analysed effectively in order to

support accurate decision-making. This article

focuses on the algorithmic decision-making process

and the need to uncover key enabling and inhibiting

factors to support and deliver healthcare services. It

is becoming increasingly important for healthcare

technologies to invest in technology and to explore

how technology may be part of that solution.

We explain that central to Connected Health

innovation is the process of algorithmic decision-

making. Therefore, it is important that healthcare

stakeholders understand the need for improved

transparency and ethical considerations in Connected

Health algorithms. Therefore, as part of our future

research, we propose a research roadmap on key

topics. Scholars are encouraged to consider the seven

key research themes, namely: (i) ethical implications;

(ii) open science implications; (iii) accountability and

responsibility implications; (iv) ageing implications;

(v) data literacy implications; (vi) healthcare

professional skills gap; and (vii) regulatory

implications associated with algorithmic decision-

making in Connected Health.

ACKNOWLEDGEMENTS

This work was supported with the financial support

of the Science Foundation Ireland grant 13/RC/2094

and co-funded under the European Regional

Development Fund through the Southern & Eastern

Regional Operational Programme to Lero - the Irish

Software Research Centre (www.lero.ie).

REFERENCES

Aron, R., Dutta, S., Janakiraman, R. and Pathak, P.A., 2011.

The impact of automation of systems on medical errors:

evidence from field research. Information Systems

Research, 22(3), pp.429-446.

Burrell, J., 2016. How the machine ‘thinks’: Understanding

opacity in machine learning algorithms. Big Data &

Society, 3(1), pp.1-12.

Carroll, N. 2016. Key success factors for smart and

connected health software solutions. Computer, 49(11),

pp.22-28.

Carroll, N. and Richardson, I. 2016a. Aligning healthcare

innovation and software requirements through design

thinking. In Proceedings of the international workshop

on software engineering in healthcare systems (pp. 1-

7). ACM.

Carroll, N. and Richardson, I., 2016b. Software-as-a-

medical device: demystifying connected health

regulations. Journal of Systems and Information

Technology, 18(2), pp.186-215.

HEALTHINF 2020 - 13th International Conference on Health Informatics

764

Carroll N, Kennedy C, Richardson I. 2016a. Challenges

towards a connected community healthcare ecosystem

(CCHE) for managing long-term conditions.

Gerontechnology.14(2), pp.64-77

Carroll N, Travers M, Richardson I. 2016b. Evaluating

Multiple Perspectives of a Connected Health

Ecosystem, 9th International Conference on Health

Informatics (HEALTHINF), Rome, Italy, Feb. 21-23.

Carroll, N., Richardson, I., Moloney, M. and O’Reilly, P.,

2018. Bridging healthcare education and technology

solution development through experiential innovation.

Health and Technology, 8(4), pp.255-261.

Coiera, E. 2015. Guide to health informatics. CRC Press.

Coiera, E. 2019. The price of artificial intelligence.

Yearbook of medical informatics, 28(01), 047-051

Cooper, B. 2011. Judges in Jeopardy: Could IBM's Watson

Beat Courts at Their Own Game. Yale LJF, 121, pp.87-

100.

Criado Perez, C., 2019. Invisible women: Exposing data

bias in a world designed for men. Chatto & Windus,

London.

Croskerry, P., 2002. Achieving quality in clinical decision

making: cognitive strategies and detection of

bias. Academic Emergency Medicine, 9(11), pp.1184-

1204.

Gagnon, M.P., Ngangue, P., Payne-Gagnon, J. and

Desmartis, M. 2015. m-Health adoption by healthcare

professionals: a systematic review. Journal of the

American Medical Informatics Association, 23(1),

pp.212-220.

Glanz, K., Rimer, B.K. and Viswanath, K. eds., 2008.

Health behavior and health education: theory, research,

and practice. John Wiley & Sons.

Goertzel, B., 2014. Artificial general intelligence: concept,

state of the art, and future prospects. Journal of

Artificial General Intelligence, 5(1), pp.1-48.

Goodman, B. and Flaxman, S., 2017. European Union

regulations on algorithmic decision-making and a

“right to explanation”. AI Magazine, 38(3), pp.50-57.

Gruber, K. 2019. Is the future of medical diagnosis in

computer algorithms?. The Lancet Digital Health, 1(1),

pp.e15-e16.

Hedman, J., Srinivasan, N. and Lindgren, R., 2013. Digital

traces of information systems: Sociomateriality made

researchable. Proceeding of the 34th ICIS, Milan, Italy.

Hollis, K.F., Soualmia, L.F. and Séroussi, B., 2019.

Artificial Intelligence in Health Informatics: Hype or

Reality?. Yearbook of Medical Informatics, 28(01),

pp.003-004.

Koene, A., Clifton, C., Hatada, Y., Webb, H. and

Richardson, R., 2019. A governance framework for

algorithmic accountability and transparency. European

Parliamentary Research Service, Tech. Rep. PE, 624.

Accessed online on 07/01/2020: https://www.europarl.

europa.eu/RegData/etudes/STUD/2019/624262/EPRS

_STU(2019)624262_EN.pdf

Kvedar, J., Coye, M.J. and Everett, W., 2014. Connected

health: a review of technologies and strategies to

improve patient care with telemedicine and telehealth.

Health Affairs, 33(2), pp.194-199.

Lepri, B., Staiano, J., Sangokoya, D., Letouzé, E. and

Oliver, N., 2017. The tyranny of data? the bright and

dark sides of data-driven decision-making for social

good. In Transparent data mining for big and small data

(pp. 3-24). Springer, Cham.

Leroy, G., H. Chen and T. C. Rindflesch 2014. Smart and

Connected Health [Guest editors' introduction]. IEEE

Intelligent Systems 29(3), pp.2-5.

Leveson, N.G., 1986. Software safety: Why, what, and

how. ACM Computing Surveys (CSUR), 18(2),

pp.125-163.

Lupton, D. 2014. Health promotion in the digital era: a

critical commentary. Health promotion international,

30(1), pp.174-183.

Lyu, M.R., 1996. Handbook of software reliability

engineering (Vol. 222). CA: IEEE Computer Society

Press.

Mann, F., Walter, B. V., Hess, T., and Wigand, R. T. 2009.

Open access publishing in science. Communications of

the ACM, 52, pp.135-139.

Mayer-Schönberger, V. and Cukier, K., 2013. Big data: A

revolution that will transform how we live, work, and

think. Houghton Mifflin Harcourt.

Nissenbaum, H., 1994. Computing and accountability.

Communications of the ACM, 37(1), pp.72-81.

O’Leary, P., Carroll, N. and Richardson, I., 2014. The

practitioner's perspective on clinical pathway support

systems. International Conference on Healthcare

Informatics (pp. 194-201). IEEE.

Oh, H., Rizo, C., Enkin, M. and Jadad, A. 2005. What is

eHealth (3): a systematic review of published

definitions. Journal of medical Internet research, 7(1),

p.e1.

Orlikowski, W. J., and Scott S. V. 2015. The Algorithm and

the Crowd: Considering the Materiality of Service

Innovation, MIS Quarterly 39 (1), pp. 201-216.

Osborne, R. 2015. Open Access Publishing, academic

research and scholarly communication. Online Inform.

Rev. 39, pp.637-648.

Pasquale, F. 2015. The black box society. Harvard

University Press.

Paton, C. and Kobayashi, S., 2019. An Open Science

Approach to Artificial Intelligence in Healthcare.

Yearbook of medical informatics. 28(01), pp.047-051.

Ransbotham, S., Fichman, R.G., Gopal, R. and Gupta, A.,

2016. Special section introduction – ubiquitous IT and

digital vulnerabilities. Information Systems Research,

27(4), pp.834-847.

Topol, E.J., 2019. High-performance medicine: the

convergence of human and artificial intelligence.

Nature medicine, 25(1), pp.44-56.

Zerilli, J., Knott, A., Maclaurin, J. and Gavaghan, C., 2018.

Transparency in Algorithmic and Human Decision-

Making: Is There a Double Standard? Philosophy &

Technology, pp.1-23.

“The Algorithm Will See You Now”: Exploring the Implications of Algorithmic Decision-making in Connected Health

765