Designing a Social Machine for the Heart Manual Service

Vanessa Hanschke

, Areti Manataki

, Cristina Adriana Alexandru

, Petros Papapanagiotou

Carolyn Deighan

, Louise Taylor

and Dave Robertson

Microsoft, Milan, Italy

School of Informatics, The University of Edinburgh, Edinburgh, U.K.

The Heart Manual Department, NHS Lothian, Edinburgh, U.K.

Keywords:

Social Machine, Heart Manual, Self-management Rehabilitation, Participatory Design.

Abstract:

Social machines are emerging as a focus of research within the ﬁeld of informatics as they begin to become the

central administrator of our everyday communications. The difﬁculty of applying such systems to specialised

contexts, such as healthcare, calls for guidelines on how to design them, so that they become truly useful.

In collaboration with the Heart Manual Department, this project is an attempt at ﬁnding suitable methods for

designing social machines in a healthcare context. It suggests that adopting a participatory approach where

stakeholders are active, equal participants throughout the design process leads to a more usable, likeable, and

thus more successful social machine. We describe the process of designing a social machine for the Heart

Manual service, in which requirements were elicited through various participatory design methods and a proof

of concept evaluation was carried out with a prototype. The prototype was received largely positively and

scored highly on the System Usability Scale, indicating the success of the proposed methodology.

1 INTRODUCTION

The Heart Manual (The Heart Manual Department,

2016) is the UK’s leading home-based supported self-

management programme for individuals recovering

from acute myocardial infarction and/or revasculari-

sation. With the help of a team of facilitators (typi-

cally nurses, psychologists or general practitioners),

who are based in different locations nationally and in-

ternationally, it guides patients through a series of ses-

sions which empower them to improve their lifestyle.

Apart from the initial training that facilitators receive,

there is no infrastructure for community building, and

knowledge exchange takes place in an ad-hoc fash-

ion, with individual facilitators contacting the Heart

Manual (HM) team to clarify their questions.

Social machines (Hendler and Berners-Lee, 2010)

can provide an appropriate infrastructure for such in-

teractions. Through a closer interaction between hu-

mans and machines, such systems can support com-

munication and the sharing of experience, and con-

tribute to a sense of community. Despite recent re-

search interest in this topic, there are still no best prac-

tices for designing successful social machines.

In this paper, we propose a participatory approach

to the design of healthcare social machines. We posit

that the involvement of users throughout the design

process can address usability and likeability aspects,

which are key success factors for health technologies.

In particular, we describe our experience in employ-

ing participatory design methods to develop a social

machine for the HM facilitators. Background infor-

mation is provided in Section 2 and our methodology

is introduced in Section 3, followed by an account of

requirement elicitation in Section 4. We next present a

ﬁrst prototype created to collect quick feedback (Sec-

tion 5) and how this was further adapted and evaluated

by HM facilitators (Section 6). We conclude with an

overview of lessons learnt and future work.

2 BACKGROUND

The Heart Manual (The Heart Manual Department,

2016) is a home based cardiac rehabilitation pro-

gramme, supported by trained facilitators and ev-

idenced by three randomized control trials (Clark

et al., 2011). It consists of six weekly sessions

that include education, exercise, relaxation and stress

management. The programme was digitised in 2015

Hanschke V., Manataki A., Alexandru C., Papapanagiotou P., Deighan C., Taylor L. and Robertson D.

Designing a Social Machine for the Heart Manual Service.

DOI: 10.5220/0006249004350440

In Proceedings of the 10th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2017), pages 435-440

ISBN: 978-989-758-213-4

435

(Deighan et al., 2015), and there is now the oppor-

tunity to extend it with recent advances in large and

distributed open systems, such as social machines.

Social machines are technology-enabled social

systems, seen as computational entities governed by

both computational and social processes (Hendler and

Berners-Lee, 2010). They are deﬁned as “Web-based

socio-technical systems in which the human and tech-

nological elements play the role of participant ma-

chinery with respect to the mechanistic realization of

system-level processes” (Smart et al., 2014). Social

machines are a recent research theme, with related

work focusing mostly on analysing existing exam-

ples, such as Facebook, Wikipedia and Stack Over-

ﬂow (Shadbolt et al., 2013; Smart et al., 2014). Lit-

erature on designing social machines is relatively

sparse. Donath (Donath, 2014) approaches this

topic in terms of conceptualising communities and

strengthening ties, while Murray-Rust and Robertson

(Murray-Rust and Robertson, 2015) suggest using ex-

isting social machines that bring together designers

and developers to generate new social machines. Both

approaches push for an understanding of the users and

a close tailoring of the social machine to its partici-

pants. Yet we are still lacking guidelines for practical

requirement elicitation for new social machines.

In healthcare, a number of social machines aim

to bring patients together to help them live health-

ier lives. PatientsLikeMe.com, for instance, con-

nects patients with similar conditions, allowing for

peer support and knowledge sharing, while the Fit-

bit and Nike+ FuelBand online communities operate

as behavioural interventions that complement activ-

ity trackers. However, the majority of such machines

lack a clinical evaluation, and there are concerns

around conﬁdentiality and patient self-diagnosis.

Similarly to the proposed HM social machine,

other machines connect clinicians rather than patients

(e.g. doc2doc.bmj.com). A recent review (Rolls et al.,

2016) explores how social media have helped health

professionals worldwide create virtual communities

where they exchange knowledge and network. How-

ever, the paper does not give a comprehensive re-

view of design methodologies. The closest example

in this literature were the trials for the Midwifery Fo-

rum (Brooks et al., 2004). However, the participatory

design mentioned in the paper was limited to proto-

type evaluation, rather than collaborative design.

3 METHODOLOGY

The lack of guidelines for designing social machines

is a considerable gap, especially in the healthcare con-

text, where technology uptake is relatively slow. In-

cluding the capabilities of the users in the interface

design can have a signiﬁcant effect on its usability

and, thus, the success of a healthcare social machine.

We thus adopt a participatory approach to the de-

sign of a social machine for the HM facilitators. Par-

ticipatory design is a user-centred design methodol-

ogy, in which the end-user is made a full participant

of the design process, typically by interacting with

mock-ups, prototypes and other tools that represent

developing systems (Schuler and Namioka, 1993; Si-

monsen and Robertson, 2012). We hypothesise that

participatory design is an effective method for the de-

sign of a social machine of health services with re-

spect to system likeability and usability, as measured

by a small-scale evaluation of a web-based prototype.

An iterative development process was followed,

consisting of three main phases: requirements elicita-

tion, low-tech paper-based prototype design and high-

tech wireframe prototype development. Inspired by

the spiral model of software development, each phase

included a repetition of steps, namely planning ob-

jectives, collecting information from users, analysing

their feedback and creating a next level prototype (or

list of requirements, in the case of Phase 1).

4 ESTABLISHING

REQUIREMENTS

We utilised Shadbolt’s constructs for classifying so-

cial machines (Shadbolt et al., 2013; Smart et al.,

2014) to establish a set of questions which provide

a framework for gathering requirements for a social

machine. The questions devised cover the tasks and

purpose of participation, participants and their roles,

as well as motivation and incentives (see Figure 1).

Three main methods were used for establishing re-

quirements, described hereafter:

Training session: We ﬁrst attended a two-day train-

ing session that prepares healthcare professionals to

become facilitators of the HM programme. This al-

lowed us to gain a clearer understanding of their role

and the challenges they face (e.g. cultural issues that

might impede patients from following dietary restric-

tions). To address such challenges, the session also

made apparent the usefulness of a space for facilita-

tors to discuss their opinions and experiences, share

good practice and provide peer support, especially in

the case of facilitators working in remote areas.

Brainstorming session: This was organised in the

form of a “Future Workshop” towards idea genera-

tion (Simonsen and Robertson, 2012). Future work-

shops are common in participatory design and consist

HEALTHINF 2017 - 10th International Conference on Health Informatics

436

Figure 1: Questions to guide requirement gathering for a social machine, and answers for the HM social machine.

of three stages: i) critiquing the present, ii) envision-

ing the future and iii) implementing - moving from

present to the future. Four clinicians participated in

the workshop: a health psychologist, an assistant psy-

chologist, a specialist nurse who is also an HM facili-

tator, and the lead of the HM team. Following the Fu-

ture Workshop structure, we ﬁrst focused on existing

communication between clinicians, and then we dis-

cussed the goals of the HM social machine. Based on

these goals, the participants were then asked to devise

dream solutions, unrestricted by technical knowledge

or possibilities. Three examples of social machines

were then presented to them: Facebook, doc2doc and

Stack Overﬂow. These were carefully chosen, so as to

prompt discussions around different forms of commu-

nication, privacy and conﬁdentiality regulations, as

well as user reputation, respectively. This presenta-

tion initiated a new discussion about the dream solu-

tions, leading to a jointly ordered list of requirements.

Telephone interviews: Three semi-structured inter-

views with additional facilitators were organised so

as to verify some of the ideas collected previously, as

well as to clarify points where participants had dif-

fering opinions. For example, participants gave their

opinion about the motivation for a social machine and

the associated concerns, the need for private groups

or private messaging, anonymous posting, etc.

The brainstorming session and telephone inter-

views were recorded (with the written consent of the

participants), transcribed and analysed following top-

down thematic analysis, as guided by the questions in

Figure 1. Outputs included explicit answers to these

questions (presented in the same ﬁgure), leading to

a list of functional and non-functional requirements.

Among functional requirements, we distinguish the

following elements: discussion forum, quizzes, sur-

veys, blogs, events and notepad. Posting anony-

mously and tagging content as helpful were deemed

desirable, but setting up groups or private messaging

were not. The list of non-functional requirements in-

cluded high levels of security and privacy, low main-

tainability and the option to report forum abuse.

5 FIRST PROTOTYPE AND ITS

EVALUATION

Following best practices in participatory design (Si-

monsen and Robertson, 2012) and based on the re-

quirements gathered, we created a low-ﬁdelity, paper-

based website prototype, so as to quickly obtain feed-

back and generate ideas at an early stage of the design

process, before committing to design decisions that

would be harder to change later on. A set of pages

were designed to capture the main functionality of the

website: i) homepage, ii) discussions, iii) submitting

a discussion (see Figure 2) and iv) user proﬁle.

A prototyping session was organised to collect

feedback, with the partaking of the four participants

from the brainstorming session. Each participant was

questioned individually around two activities. In the

ﬁrst activity, they were given a set of cut out items

(e.g. buttons and taskbars) which they could arrange

Designing a Social Machine for the Heart Manual Service

437

Figure 2: Paper-based prototype for submitting a discus-

sion.

on a sheet of paper to reﬂect their idea of the home-

page. In the second activity, they were presented sug-

gestions for the design of the screens to elicit opinions

and given screen print-outs to annotate. Topics dis-

cussed included the notes facility and the presentation

of ground rules for participating in the discussions, so

as to avoid misinformation and breach of conﬁden-

tiality. The prototyping session was audio-recorded,

handwritten notes were taken and any visual material

created was documented with photographs. These re-

sources were analysed qualitatively using a combina-

tion of top-down and bottom-up thematic analysis. In

general, it was found that participants were pleased

with the layout that was presented to them. Opinions

were evenly split on the presentation of the ground

rules and on a simple versus busy look of the home-

page. A pop-up window and a busy feel, respectively,

were chosen by the designer for the next version of

the prototype, which would be further evaluated, thus

reducing the associated risk.

6 FINAL PROTOTYPE AND ITS

EVALUATION

A hi-ﬁ web-based prototype was developed by adapt-

ing the ﬁrst prototype given the feedback gathered.

It was presented on a browser and made from linked

screens to generate a feel of the interactions that

would take place (see Figure 3). Its main difference

to the ﬁrst prototype was that users could interact with

the system and see its dynamics in operation. The pro-

Figure 3: The home page in the ﬁnal prototype.

totype was not fully functional, but a proof of concept.

It was designed to work for certain scenarios that were

set up for the evaluation: 1) starting a discussion, 2)

entering a discussion point, 3) marking a contribution

as helpful, 4) reporting a contribution as harmful, 5)

entering an event, 6) adding items to favourites, and

7) accessing favourites.

The prototype was tested with seven potential

users (ﬁve members of the HM team and two facil-

itators) for usability and likeability. Three of the HM

team members had participated in the ﬁrst prototyp-

ing session, while the remaining two had not been in-

volved in any of the prior activities. The two addi-

tional facilitators had been interviewed for establish-

ing requirements. The evaluation was divided in two

main activities (i.e. think aloud and questionnaire),

while a survey was set up for carrying it out remotely.

Think Aloud and Short Structured Interviews: For

this part of the evaluation, participants were presented

with the prototype on a laptop. They were given four

sets of tasks: i) submit a new discussion, ii) ﬁnd a dis-

cussion, mark it as a favourite, mark and report a con-

tribution, iii) ﬁnd an event and mark it as a favourite

and iv) revisit favourites. The participants were asked

to “think aloud” while completing these tasks, i.e.

explain their thought process, so as to make explicit

where they were stuck and why, what they were look-

ing for, what their expectations were, etc. Following

their think aloud, participants gave their opinion about

the presented features and their intuitiveness.

Questionnaire: Participants were next presented

with a questionnaire around usability and likeabil-

ity, which included three sets of questions: i) SUS

scale (Brooke, 2013), ii) how often (in a scale from 1,

HEALTHINF 2017 - 10th International Conference on Health Informatics

438

Figure 4: Accuracy and completeness in Think Aloud tasks.

i.e. never, to 5, i.e. very often) participants thought

they would use certain features and iii) whether they

thought certain aspects were particularly enjoyable,

unenjoyable or concerning and why.

Survey: Participants who performed the evaluation

remotely were emailed an online survey containing

the same questionnaire and Think Aloud tasks, and

a link to the prototype. They were also asked to put

remarks in comments if they were stuck at any point,

especially if they were unable to complete the task.

The ﬁve participants of the HM team carried out

the different evaluation activities in the presence of

the researcher. One of the facilitators completed the

evaluation using the survey, while the other completed

it remotely, with the researcher on the phone and

watching the screen navigation in real time.

The Think Aloud sessions and structured inter-

views were audio recorded (with the participants’ per-

mission), transcribed and, together with the survey

data, analysed qualitatively using a combination of

top-down and bottom-up thematic analysis. Inter-

view or on-line survey replies on intuitiveness were

attributed a ranking from 1 to 5. During the Think

Aloud session, the time for completing each task

was recorded, and notes were taken about whether it

was completed successfully and the number of errors.

These data were analysed quantitatively with regards

to efﬁciency, task accuracy and completeness. Ques-

tionnaires were also analysed quantitatively.

Figure 4 presents the accuracy and completeness

measured during the Think Aloud session. All partic-

ipants were able to complete all tasks independently,

except for one participant who required further expla-

nations on two tasks. The errors observed had to do

with misunderstanding instructions and with the fact

that the search bar lacked functionality at this stage.

Regarding efﬁciency, the average time in seconds

for each of Tasks 1-4 was: 57.4 (SD 57.4), 67.4 (SD

43.5), 21 (SD 8) and 35 (SD 19.4), respectively. Note

that the standard deviations here are inevitably large,

given the small number of participants recruited.

Based on participant remarks and answers during

the Think Aloud session and the short structured in-

terviews, all tasks scored highly on intuitiveness: the

average intuitiveness score (on a scale from 1 to 5) for

Tasks 1-4 was 4.6, 4.4, 5 and 3.7, respectively.

The average SUS score was 86.4 (SD 18.02), indi-

cating “excellent” satisfaction, according to published

grade rankings of SUS scores (Bangor et al., 2009).

With the exception of one participant who gave a

score of 50 (i.e. “ok”), the rest of the scores ranged

from 80 (i.e. “Good”) to 100 (i.e.“Best imaginable”).

The results to the question of how often partici-

pants would use certain features ranged from 3 (i.e.

every now and then) to 4.43 (i.e. between regularly

and very often). The former was assigned to the use

of the notepad for keeping personal notes, while the

latter to reading discussions and looking up events.

A variety of features were found to be enjoyable.

Discussions, in particular, were mentioned by ﬁve out

of the seven participants. For instance, a facilitator

mentioned: “Starting a discussion. I liked that. And

I liked it because if I know I’ve got a thought in my

head and I can put it out there and see what my col-

leagues and counterparts have. So it can validate the

ideas I might have”. Three facilitators mentioned that

the favourites feature also seemed very useful. No

feature was pointed out as unenjoyable or concerning.

7 CONCLUSIONS

Based on the quantitative measures presented in Sec-

tion 6, the hi-ﬁ prototype was a great success, per-

forming well on both satisfaction and effectiveness.

The SUS score was very positive, corresponding to

an “excellent” usability grade. Only one user was un-

able to complete two tasks independently, totalling an

overall average of 92% success per user. The overall

error rate was low, with no errors on many tasks and

less than one error on average for tasks with errors.

Some of the errors, such as accessing the search bar

(which was not yet functional), are easily resolvable.

The efﬁciency of the tasks can only be compared be-

tween each other, as we do not have an absolute mea-

sure of how long these should take. A learning effect

between Tasks 1 and 3 can be noted, as they take a

similar amount of clicks, but the latter is faster.

The system seemed to have a high level of accep-

tance, as most features were marked to be used be-

tween “every now and then” and “very often”. This

shows that the features proposed in the prototype were

in line with the expectations of the participants. This

is also demonstrated by the fact that discussions were

highlighted as a particularly enjoyable feature.

Designing a Social Machine for the Heart Manual Service

439

This interdisciplinary project highlighted several

themes which we expect to be recurring when design-

ing social machines for health professionals, given the

general characteristics of the domain. Firstly, design-

ing for time constraints is crucial, given that clini-

cians are very busy. Hence, an easily learnable inter-

face is recommended, where all features are directly

accessible on display. Security is another important

theme, and access control aspects should be carefully

thought out. Furthermore, social machines that sup-

port knowledge sharing should allow users to identify

misinformation, which is critical in healthcare.

The choice of participatory design allowed us to

identify these themes and concerns, which might not

have been addressed without input from the users.

Our overall experience of using participatory design

in healthcare was greatly positive. Participants were

very motivated and gave thorough and careful feed-

back. On the other hand, recruiting participants was

the biggest issue, as healthcare professionals are gen-

erally very busy and communication channels tend to

be controlled. We worked around this issue by ensur-

ing that different types of stakeholders were involved

in the study, from experienced to new facilitators and

from isolated workplaces to busy hospitals.

The range of participatory design methods used

provided us with a wealth of both quantitative and

qualitative data and allowed for a good level of user

engagement and interaction. The two-hour brain-

storming session, which resulted in 26 pages of tran-

script, allowed participants to exchange and visualise

ideas, and thus served as a good basis for establish-

ing requirements. The phone interviews did not pro-

vide the same level of idea generation, but carrying

them out separately with each facilitator provided a

safe space for them to express their personal views on

a HM machine. Furthermore, the paper-based proto-

typing session was found to be very useful, as it al-

lowed us to concretise ideas at an early stage.

The questions presented in Figure 1 were partic-

ularly useful for designing the HM social machine.

They guided the requirement elicitation process, from

structuring the brainstorming session to analysing the

qualitative data obtained. We would, hence, recom-

mend their use to other social machine designers.

In the future, we wish to continue using the partic-

ipatory design methodology and implement the feed-

back received as part of the current study. In order

to create a truly usable social machine for the Heart

Manual service, we plan to include more participants

and to investigate new topics, such as the moderat-

ing role of the HM team. Applying the methodology

presented in this paper to the design of other social

machines is another exciting avenue for future work.

ACKNOWLEDGEMENTS

This research was supported by the EPSRC SociaM

project under grant EP/J017728/1.

REFERENCES

Bangor, A., Kortum, P., and Miller, J. (2009). Determining

what individual SUS scores mean: Adding an adjec-

tive rating scale. J. Usability Studies, 4(3):114–123.

Brooke, J. (2013). SUS: a retrospective. Journal of usability

studies, 8(2):29–40.

Brooks, F., Pospopa, C., and Schott, P. (2004). Midwifery

on the net: new communication technology. British

Journal of Midwifery, 12(2):107–110.

Clark, M., Kelly, T., and Deighan, C. (2011). A system-

atic review of the heart manual literature. European

Journal of Cardiovascular Nursing, 10(1):3–13.

Deighan, C., Pagliari, C., Michalova, L., Elliott, J., Brook-

mann, F., and Taylor, L. (2015). Digital heart manual:

making technology accessible with a user-friendly re-

source. In Health Informatics Scotland Conference.

British Computer Society.

Donath, J. (2014). The social machine: designs for living

online. MIT Press.

Hendler, J. and Berners-Lee, T. (2010). From the seman-

tic web to social machines: A research challenge for

AI on the world wide web. Artiﬁcial Intelligence,

174(2):156–161.

Murray-Rust, D. and Robertson, D. (2015). Bootstrapping

the next generation of social machines. In Crowd-

sourcing, pages 53–71. Springer.

Rolls, K., Hansen, M., Jackson, D., and Elliott, D. (2016).

How health care professionals use social media to cre-

ate virtual communities: An integrative review. Jour-

nal of Medical Internet Research, 18(6):e166.

Schuler, D. and Namioka, A., editors (1993). Participatory

Design: Principles and Practices. L. Erlbaum Asso-

ciates Inc., Hillsdale, NJ, USA.

Shadbolt, N. R., Smith, D. A., Simperl, E., Van Kleek, M.,

Yang, Y., and Hall, W. (2013). Towards a classiﬁcation

framework for social machines. In Proceedings of the

22nd International Conference on World Wide Web,

pages 905–912. ACM.

Simonsen, J. and Robertson, T. (2012). Routledge Interna-

tional Handbook of Participatory Design. Routledge.

Smart, P., Simperl, E., and Shadbolt, N. (2014). A taxo-

nomic framework for social machines. In Social Col-

lective Intelligence, pages 51–85. Springer.

The Heart Manual Department (2016). The Heart Manual.

http://www.theheartmanual.com.

HEALTHINF 2017 - 10th International Conference on Health Informatics

440