Experiences in Designing HCI Studies for Real-time Interaction

across Distributed Crowds and Co-located Participants

Franco Curmi

and Conrad Attard

Department of Marketing, Faculty of Education Management and Accountancy, University of Malta, Msida, Malta

Computer Information Department, University of Malta, Msida, Malta

Keywords: Real-time HCI, Real-time Interaction, Instant-method, In-the-Wild, Distributed Participants, Distributed

Interaction, Co-located Participants.

Abstract: This paper is a post-hoc reflective case study from the point of view of the research investigators. The authors

share the experience of designing and deploying four studies that involve real-time interaction between

distributed crowds and co-located participants. We first recount the challenges that these uncommon, yet

increasingly necessary, HCI research contexts afford. We then present the learning outcomes from 1) the

‘designing’, 2) the setting up, 3) the real-time dynamics and 4) the interaction between distributed and co-

located participants. From this we deduce the impact for the four stakeholders in these contexts 1) the

distributed crowd, 2) the co-located participants, 3) the system owners and 4) the researchers. This meta-

research approach is motivated by our struggle to find more ‘Researcher-experience’ cases during the early

stages of the studies. This contribution in experience sharing is intended to help HCI researchers who are

planning studies in this field.

1 INTRODUCTION

This work presents the experience as a post-hoc

reflective case study after a five-year timeframe of

conducting HCI studies with real-time distributed

interaction of online crowds. In this paper, we

aggregate the insights from four studies in a sports

context. Through the lens of the researchers, we share

the challenges that these HCI settings afford. These

studies required the development of a series of

custom applications that allow distributed crowds to

communicate their emotional support to athletes in

real-time. Through four in-the-wild deployments we

then broadcast sensor-data from athletes to globally

distributed spectators in custom-designed data

visualisations. These visuals help spectators build an

understanding of the athletes’ remote performance.

Concurrently, supporters were prompted to

externalise their support in the form of remote

cheering Across the studies we were interested in

investigating four central questions to 1) understand

the experience of the data-sharing athletes while

https://orcid.org/0000-0003-4264-6646

https://orcid.org/0000-0001-9111-6807

receiving remote support, 2) identify factors that

influence remote spectators’ behaviour during live

events, 3) identify motivations for using real-time

spectator support systems, and 4) provide guidelines

for researchers and designers that seeks to facilitate

support from remote spectators during sports events.

In this paper, we aggregate the learning from these

individual studies and use the experience to share the

challenges faced by researchers when planning and

conducting complex but increasingly necessary HCI

research that involves real-time interaction across

distributed participants. This is intended to guide

other researchers who are planning studies in this

exciting research area.

2 BACKGROUND

This work was composed of four phases as follows.

1. We first conducted a feasibility study to assess the

viability of investigating real-time support from

remote crowds in a sporting context, identify any

178

Curmi, F. and Attard, C.

Experiences in Designing HCI Studies for Real-time Interaction across Distributed Crowds and Co-located Participants.

DOI: 10.5220/0008980801780185

In Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020) - Volume 2: HUCAPP, pages

178-185

ISBN: 978-989-758-402-2; ISSN: 2184-4321

ethical issues that may arise from the study and

gather preliminary insights on how to design

systems for remote spectator support. This

feasibility study was composed of desktop

research and two in-the-wild deployments during

two sporting events (Curmi, 2013).

2. With the insights gained in Phase 1, we then

designed and build BioShare, a customizable

research tool that facilitates sharing live data over

social networks and allows remote spectators to

send instant feedback (Curmi, 2014).

3. With BioShare, we than developed and deployed

HeartLink, in an ad-hoc in-the-wild 5k event with

5 athletes and 140 remote spectators. HeartLink is

an application that allows athletes and spectators

to interact in real-time during events (Curmi,

2015).

4. Finally, we conducted a fourth in-the-wild

deployment during a 24-hour 170-mile relay race

with 13 athletes and 261 spectators (Curmi, 2017).

The next sections briefly describe each of these

individual studies. We present the methods used

together with an overview of the learning outcomes

from each of these phases.

2.1 Phase 1: Feasibility Study

Consisting in Desktop Research, a

Pilot Study and a User Study

Phase 1 (Curmi, 2013), assessed the viability of

investigating remote crowd support. It gathered

insights on possible ethical issues that should be taken

into consideration when deploying events in-the-wild

within this context and captured requirements for

system design.

Through desktop research we first reviewed

existing commercial mobile phone applications that

were designed for sports activities. We found that

applications at the time of conducting the study, did

not allow spectators to communicate with athletes

during events. We also identified that academic

research on sports applications is very limited

particularly when it comes to the sharing of live

personal data. In this light, before conducting in-the-

wild deployments, through a survey, we assessed the

readiness of participants from a university setting to

share personal data while conducting sports activities.

A pilot study and a user study were then

conducted. These investigated the technical issues

involved when athletes share data in the wild. These

also gathered primary data on the athletes’ and the

spectators’ experience. A pilot study took place

during a triathlon in the Lake District and focused

primarily on validating the technology. A user study

was then conducted during a charity run in Lancaster,

UK. This focused primarily on capturing the

participants’ experience. Analysis of the data that was

captured through observations, server-interaction

logs, interviews and content analysis of online

discussions during the events, indicated that research

in remote-crowd support is worth pursuing.

However, the use of third-party communication

applications that were used to share athletes’ data

within an in-the-wild research context, presented

several challenges that included a lack of control on

data integrity and reliability. These also limited the

ability to measure user experience and behaviour thus

motivated the development of a bespoke data sharing

system for researching remote spectator support:

BioShare.

In summary, the contributions of this phase 1)

confirmed that further research in remote crowd

support is worth pursuing, 2) provided preliminary

insights on how to build crowd support systems

around athletes and spectators, and 3) highlighted the

need to create dedicated tools for researchers working

in this area.

2.2 Phase 2: System Design and

Development

In Phase 2 (Curmi, 2014) we designed and developed

BioShare. The requirements capturing for developing

Bioshare involved three stages.

We first reanalyze the data collected in Phase 1

and identified key system requirements.

We were interested in making Bioshare relevant

for other researchers working in this area.

Consequently, to validate whether the insights gained

from our experience in deploying two in-the-wild

studies matches the requirements of researchers who

developed closely related systems, we then compared

and contrasted our insights with those of closely

related systems that are referenced in literature.

We found that systems that are referenced in

literature lack details on how these systems were

developed and details on issues that emerged during

their development, if any. Thus, we further

investigate past systems’ development by

interviewing HCI researchers who created closely

related data sharing systems for research applications.

The developed prototype consisted in a native

mobile application that broadcasts users’ locative and

physiological data over mobile networks and received

feedback from online crowds. A web interface

together with a dedicated backend allowed distributed

crowds to follow and communicate with the data-

sharing users. BioShare is open-source and is

Experiences in Designing HCI Studies for Real-time Interaction across Distributed Crowds and Co-located Participants

179

designed such that it can be configured for different

study requirements.

In addition to contributing BioShare as a tool for

researchers, this phase also contributed a set of

requirements for spectator support systems in the

presented context. These include ethical

considerations, design for adaptability and a

framework to empower the user over the shared data.

2.3 Phase 3: Deployment in an 5k-road

Race

A customised version of BioShare, HeartLink, was

then deployed in a 5k-road race with 5 athletes and

140 remote spectators. In this deployment, we 1)

captured the experience of athletes when sharing data

and receiving remote support in real-time and 2)

identified key influencers to the supporters’

behaviour during a live sport event. Pilot studies

suggested that spectator engagement is influenced by

both the data that is presented (e.g. the effort that the

athlete is exerting) as well as the social relation

between the athlete and the supporter. To validate

this, we recruited two spectator groups. One spectator

group was recruited through the athletes’ own social

networks. Thus, the spectators in this group knew the

athletes. A second spectator group was recruited from

a crowdsourcing platform and thus these spectators

had no social connection with the athletes.

Additionally, to compare whether different data

types, particularly heart rate data, influences the

spectators’ engagement, all the spectators were

randomly assigned to one of two conditions. One

group was presented with locative data while a

second group was presented with both locative data

and heart rate data of the athletes. The results

indicated that having a social tie with the athletes

increases engagement when supporting the athletes.

These spectators cheered the athletes more and spent

more time supporting. Spectators who were presented

with the additional heart-rate data in their interface

also cheer significantly more. Additionally, through a

focus group, the athletes suggested that the

motivation for athletes to use remote spectator-

support systems is dependent on the effort that the

task entails and the degree of loneliness that the event

presents. To further investigate this, we then

conducted a fourth in-the-wild deployment during a

24-hour 170-mile long relay race across the UK.

In summary, Phase 3 (Curmi, 2015) contributed

the following:

Through quantitative data, the work highlighted

differences in spectator behaviour across spectators

who are presented with different visuals, and

spectators who have different social relationships

with the athletes. For example, we found that

spectators who are presented with additional

information about the heart rate of remote participants

are likely to feel more emotional and consequently

cheer more.

Through qualitative data, the study identified key

motivations for using live remote cheering systems.

For example, we identified that spectators’ behaviour

depended on their understanding of why the athletes

are conducting the task (e.g. the assumed athletes’

egoistic vs. altruistic objectives in participating in an

event). With regards to the athletes’ motivation, we

identify that the impact that the cheering has on the

athletes is relative to their expectations. This and

similar outcomes, are congruent with existing

theories of expectations management (Boehm, 2000)

and self-determination theory (Ryan and Deci, 2000).

2.4 Phase 4: Deployment in a 170-mile

Relay Race

For this event, BioShare was customised and

embedded in a running relay-baton form factor

(Curmi, 2017). This baton works as an interface

between the remote crowd and the athletes. The

baton’s form-factor also provides enough space to

store the needed energy for the 24-hour long event.

Following a co-design process with the athletes, the

prototyped baton collects and broadcast data in real-

time and vibrates whenever a remote supporter clicks

a cheer button on the web interface. Additionally, the

baton also calls out the name of the person who sent

the ‘cheer’. In this way, the athletes get an awareness

of where the cheers came from.

Through these deployments, we further analysed

and deduce user-motivations for using real-time

crowd-support systems. Athletes reported motivation

from: receiving remote support, building followship,

having a proof of accomplishment, satisfying a social

need to connect with others, democratising sport

events and facilitating mindfulness about the event,

among others.

Additionally, the data collected through these

deployments provided insight on key factors that need

to be taken into consideration when engineering real-

time crowd support systems. These are presented in

three categories:

1) Spectators’ expressiveness i.e. the design of how

spectators can externalize their support. This can

range from a highly controlled form (e.g. simple

binary ‘Likes’) to a more open approach such as user-

generated communication (e.g. live audio streaming

of aggregate cheers from spectators’ microphones).

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

180

2) Context applicability i.e. we identify contexts

where remote spectator support seems most pertinent.

Findings indicate that these systems seem to be most

valuable in challenging events and where the athletes

feel lonely (e.g. participating in an unaccompanied

setting at night-time). On the other hand, remote

support appears less useful in competitive events.

3) The design of the data flows within the social

network. Here designers need to consider how system

users (athletes, spectators and organisers)

communicate and design communication flows.

3 METHODOLOGICAL

CONSIDERATIONS

A common element in these studies is the adoption of

an in-the-wild approach. Unlike traditional

experimental methods that take place within the lab

(Johnson, 2012), the method go beyond observing

existing practice and present opportunity to evaluate

novel technology in the place where the technology is

intended to be used (Morris, 2012; Marshall, 2011).

Research in the wild is an old practice. Centuries-old

inter-continent expeditions that inform ship design

may classify within the definition. However, over the

last decade, research in-the-wild became a common

research practice in HCI. As in our study, HCI

researchers often seek to explore new technology, test

prototypes in the location in which they are intended

for and understand how people interpret and

appropriate the technology (Dahlbäck, 1993; Gaver

2013; Kittur, 2008, Rogers 2011).

Prototypes were deployed with participants in

different cities, countryside pathways, cycle lanes,

nature parks and inside a lake. For example, the final

prototype deployment connects athletes running a

170-mile race, from coast to coast, across the UK. In

this setting, research in-the-wild allowed compare

and contrast the effect of mobile data connectivity on

the proposed technology across different

environments within the same deployment.

Evaluating technology in-the-wild poses a

number of added challenges. These challenges go

beyond the lack of comfort that out of the studio

participants are presented with (Rogers, 2011). An in-

the-wild study may suffer from lack of control that a

lab facilitates (Laseki, 2013). Consequently,

extrapolating specific effects becomes difficult and

researchers need to interpret data that is influenced by

several externalities and interdependencies

(Rogstadius, 2011). Using multiple methods of data

collection often compensates for this. Where

possible, we triangulate findings across different data

sources. Eight data collection methods were used

throughout the study, namely, surveys, literature

reviews, focus groups, semi-structured interviews

with athletes, spectators and HCI researchers, content

analysis of social network comments posted during

deployments, quantitative data of online users’

interaction that is collected by the data servers,

observations, and research through designing four

data telemetry prototypes and four online-crowd

interfaces.

The challenges that research in-the-wild presents

are widely documented in literature (Mueller, 2010;

Fox, 2006; Rogers, 2011). However, over and above

these challenges, this work faced additional unusual

dynamics. Each of these augments the complexity of

running the study (Figure 1). Namely, 1) the need for

co-ordinating a group of co-located participants that

are conducting a challenging task in-the-wild, 2) the

need of co-ordinating a group of globally distributed

participants and 3) the need for all activities to operate

in real-time with synchronous interaction at a global

scale. The latter does not afford the traditional lab

recruitment approach where the researcher schedules

participants at a time when it is most convenient for

each participant. In such HCI studies, all the

participants have to synchronise with the live event.

In this context, recruiting participants,

particularly online spectators, requires rigorous

planning. Online spectators may be less difficult to

recruit than co-located athletes since much less effort

is needed when participating in an online task than

when participating in a physically challenging task

such as a long-distance race. Additionally, there is

typically no travelling involved. The participants do

not feel they are being watched and they might do

work in parallel to following and/or supporting the

athletes. For example, Manson points out that

participants might have coffee while engaging in an

online event (Manson, 2012). Another important

consideration is ‘Attrition’. In a lab experiment, it is

very unlikely that a participant walks out of an

experiment due to unstated pressure from being in a

face-to-face situation. This does not apply to an

online environment where participants may easily

leave the experiment at the click of a button. The

participants may also be distracted by various other

factors such as surfing other websites, making errands

or experience technical system failure, to mention a

few. To monitor this, we placed occasional prompts

to monitor attention. The system then logs the time

taken for each viewer to respond and this measure

may then be compared to different participant groups

and collected datasets.

Experiences in Designing HCI Studies for Real-time Interaction across Distributed Crowds and Co-located Participants

181

Additionally, to mitigate complexity, we start

with a small-scale deployment that has few

participants, to then increment the scale of the

deployment iteratively. This approach promises 1)

incremental improvement, 2) contains any emergent

ethical issues and 3) minimises risk of failure.

Figure 1: Methodological influencing contexts.

4 FINDINGS AND REFLECTION

4.1 Conducting RIW Deployments with

Synchronous Interaction between

Distributed Participants

As earlier stated, all the deployments were conducted

in the wild. This made planning, organising and

deploying extremely challenging. Moreover, each

study involved multiple user groups that were not

only in the wild but also distributed across different

locations. Additionally, the interaction under

investigation was synchronous. Thus, the participants

needed to synchronize themselves to the study rather

than the study synchronises to the participants. The

need for participants to synchronise with the study

limits the number of participants that can take part in

the study as their participation is not only conditioned

by their willingness and appropriateness to the sample

group but also by their personal schedule. This

restriction is particularly visible when the user group

is friendsourced, that is, the participants have a social

tie with the athletes. On the other hand, this is less

restricting for outsourced participants, that is, where

participants have no social connection with the

athletes as the sample frame may be larger.

Outsourced participants were recruited from

CrowdFlower, a crowdsourcing platform.

Crowdsourcing platforms provide a large enough

pool of participants (crowd workers) who are seeking

work that matches their expected enjoyment and

financial return. The enjoyment value is a major

factor in the recruitment process. Many studies show

that crowdsourced participants value the pleasantness

of a given task (Rogstadius, 2011). This impact both

the engagement of participants in the task and also the

reputation on the platform (through post-task-

completion feedback) of the researcher who issued

the study.

4.2 Contrasting Real-time in the Wild

vs Lab

Inversely should there was a lab variant of the studies,

the challenges would have been very different. 1) The

researchers would not have been bound with

recruiting a large group of participants to perform at

one specific global time. 2) The researchers would

have had more control over the environment and

confounding variables would have been limited. For

example, weather conditions would have been

minimally influential on the study, if any. Windows

would have been closed and, say, a treadmill could be

kept the same gradient for all participants such that all

the participants would have been presented with an

identical controlled experience. Similarly, cheers

could have been computer generated from predefined

patterns that would simulate remote cheering crowds.

3) Running such an experiment would have been

easier because the researcher would be in the comfort

of the well-known, tried and tested, “lab”. The

researcher could have wired, handled and observed

one participant at a time.

However, no matter how controlled this

environment would have been, it would have never

been anything close to the real thing; the in-the-wild

environment with real crowds sending self-initiated

cheers at that moment in time.

In hindsight, an in the wild deployment that

involves distributed crowds, like the four

deployments in this study, are more unforgiving then

an in the lab approach where single participants

sequentially conduct offline sessions. For example, if

there is an issue with the system, such as what

happened in the pilot study due to downtime on US

Amazon Web Services, then the researcher needs to

coordinate a crowd that is distributed. This is

challenging, not only because the investigation

involves a crowd but also because the data is live and

distributed. In this case, the research event is likely to

fail or at minimum, the research objectives would

change. Moreover, a new event is likely to require

Research

In the

wild

setting

Co-located

particip-

ants

Distribute

d online

crowd

Out-

sourced

crowd

Friend-

sourced

crowd

Research

Team

Synchro-

nous

interact-

ion

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

182

coordinating a new crowd. Should that have been in

the lab, a participant would have been ranked as an

outlier or replaced with an additional lab session.

Finally, in a lab variant, systems can be tested, and

researchers can pose as dummies. In an in-the-wild

version where crowds operate synchronously,

systems are difficult to test comprehensively. For

example, collecting enough participants to simulate a

crowd to test a system in situ is often impractical.

Furthermore, if the researcher does manage to do this,

in most cases, the in-the-wild environment is likely to

change over time. Hence, reliability across all

variables is challenging at best. For example, data

telemetry that is dependent on mobile network

coverage (reception) may be influenced by the

density of users that happen to be in that public area

and of which the researchers have little or no control.

These challenges that research in the wild with

real-time interaction brings to the table further

highlight the differences of real-time in the wild over

an in-lab study. These differences emphasise the

distinct values that both approaches present the

researcher with. Based on the above considerations,

we recommend the following to researchers who

intend conduct research on the lines of this work:

1. The researcher should seek to control any variable

that can be controlled but plan contingencies for

unexpected events.

2. Conducting meticulous planning minimizes the

risks of unexpected outcomes.

3. In these contexts, observations during the event as

part of the research method is highly important

and details should be documented during or

immediately after the event.

4. The researcher should have a communication

channel with remote participants. This is essential

for ad-hoc coordination should unplanned

phenomena occur.

Roles for documenting events and coordinating

events should be assigned to separate individuals.

Due to the complexity that such tasks entail, we

recommend that researchers build a research team

where each member is assigned a pre-designed role.

Different studies and conditions would necessitate

different roles for supporting staff. In the case of the

study in Phase 4, the 170-mile relay race, the

recommended roles for the event such that the

researchers can focus on core areas were as follows:

1) A researcher was assigned to coordinate the online

crowd. This person would, for example, message the

crowd should there be a need to do so during the live

event and answer any queries that online participants

may have. 2) A person needs to coordinate the co-

located athletes. 3) An experienced researcher

journals the event. 4) A videographer and/or

photographer may provide grounded content for post-

event analysis or in support of post-event

publications.

4.3 Managing Interfaces with Multiple

Stakeholders

The deployed interfaces were driven by two main

sources: The athletes and the spectators. In their

work, “Designing the Spectator Experience”, Reeves

et al. classify these interfaces as public interfaces.

These are public interfaces not because the interfaces

are out in the wild but rather because of the “extent to

which [the] performer’s manipulations of an interface

and their resulting effects are hidden, partially

revealed, fully revealed or even amplified for

spectators.” (Reeves, 2005 p.741). In recent years,

there have been numerous discussions within related

communities, such as SIGCHI, on interfaces that are

moving away from providing an individual dialogue

but rather are designed for a crowd (Boulos 2011;

Brady, 2014) and driven by a crowd (Laseki, 2013).

In most of these cases, as it is the case of the above

studies, the crowd is distributed.

In a real-life cheering context, within an open

sporting event, the human interaction is intended to

be public. Spectators cheer from the sides of a

racecourse or from the stands at a stadium. On the

other hand, in the digital world, there is by far more

human-human communication that is designed for a

private setting rather than a public one. Public

telephones for example are enclosed in boxes or

photo kiosks (Rogers, 2011). There are also multiple

levels of engagement. There are spectators who

follow the data through the crowd-powered interface,

hence they can follow the data of the athletes and the

data that is driven by the crowd (i.e. the live cheering,

the spectators live comments as the event unfolds and

the number of spectators that join and leave the event,

during the event). There are also the supporters, that

is, those spectators who do not simply follow the data

but also interact with the interface and the athletes by

cheering and commenting, hence contributing to the

live discussion. Finally, there are the athletes whose

interaction is highly automated, both the sharing of

data, and in receiving feedback from the crowd. In a

way, the interface of the athletes is hidden and

inexistent. It is an extension of the crowd’s interface.

For example, the relay baton that was deployed in

Phase 4, opens a channel to the crowd. The athletes

do not interact with the baton, but the baton automates

the communication from the athlete to the crowd. The

baton captures the data and sends it to the crowd

Experiences in Designing HCI Studies for Real-time Interaction across Distributed Crowds and Co-located Participants

183

without any interaction from the user (the athlete).

There are also the co-located spectators, who,

although they might not interact with the cheering

system or the online crowd, they may also influence

the online environment through the athlete’s

environment. In this regard, Reeves et al. add another

dimension to public vs. private dichotomy;

manipulations and effects, where manipulations are

the actions of the ‘performer’ (in our case these are

the actions of the athlete). On the other hand, the

effect is the impact of the manipulations; a click on a

cheer button triggers a vibration (direct effect) and

may make the athlete aware of the support being sent.

The athlete may perform better and his or her exertion

may influence the gradient of the chart representing

the live heart rate (indirect effect).

The work of Reeves et al. helps us position the

authors’ interface within the spectrum of interfaces.

Interface categories include 1) Magical, this refers to

interfaces that hide the manipulations but the effects

are revealed (e.g. wizard of oz interface (Reeves,

2005)), 2) Secretive, where both the effects and the

manipulations are hidden (e.g. within a competitive

game), 3) Suspenseful, where manipulations are

hidden but the effects are revealed and finally 4)

Expressive, where both the manipulations and the

effects are revealed or amplified. Within this

taxonomy, the proposed cheering system positions

itself in the expressive quadrant. Spectators’ actions

are channelled to the athletes and amplified through

haptic and sound synthesisers. The athletes’

performance is sensed and amplified to all connected

spectators within the spectators’ interfaces.

The real-time cheering system associates another

dimension to this. The interface is not only generated

and interacted-with by the individual and displayed to

the spectators as a crowd, but this crowd also drives

the interface. In other words, the interface (including

the cheers, the number of online spectators and the

live comments that make up the interface) is

generated from the crowd. These become, as Michael

Bernstein coined, crowd-powered interfaces. Crowd-

powered interfaces are “interfaces that rely on human

activity traces or human computation to provide

benefits to the end user.” (Bernstein 2010 p.347).

Undoubtedly, the cheering process is explicitly

conducted for the benefit of an end user, the athlete.

We argue that this process relies on both ‘human

activity traces’ and ‘human computation’. They rely

on human activity traces as the distributed individuals

trigger the cheers, and each has his or her intentions

and motivations to cheer. The human computation

component is highlighted in the user interviews.

Upon interview, the spectators showed interest in

maximising the positive impact that the cheers could

have on the athlete. In this regard, spectators devised

strategies such as leaving more cheers towards the

end of the race, ‘such that the cheered athletes do not

get used to the cheering’. These strategies are

reflected in the cumulative cheering plots.

4.4 Managing Multiple

Communication Modalities

Across the studies, we looked at primarily two

communication flows. Informing the online crowd

(spectators) and informing the athletes. The athletes’

awareness of spectators’ behaviour and their support,

can contribute to build a sense of ‘liveness’(Mueller,

2010). However, it can also generate pressure on the

athlete. The sense of ‘being observed’ that real-time

remote support systems create, may make the athlete

feel obliged to perform or feel embarrassed for

mistakes since spectators are following the

performance. The modality that is adopted to

communicate the crowd’s support to the athlete is an

influential factor in the design of the athletes’

experience. The deployments explored tactile and

sound alerts to communicate the cheering crowd.

Results showed that the effect of the communication

modality is dependent on different externalities over

and above the modality itself. These include the

context (e.g. the background noise within the

environment), the trustworthiness of the cheering

crowd (e.g. whether there are spammers among the

cheering crowd who might misuse the modality, say,

send inappropriate messages) and the individual

personalities of the athletes that the set modality is

communicating with. For example, when the

modalities where calibrated to generate the same

intensity of tactile feedback for all participants, some

athletes did not feel the set vibration. This seemed

related to the athlete’s body mass and athletes with

larger arms were more likely not to feel the vibrations

that were triggered by the telemetry device thus

emphasising the need for personalisation.

5 CONCLUSION

We believe that these findings are just the beginning

of this research area and we trust that other studies

will follow. What is presented here may provide the

preliminary groundwork for real-time remote crowd

support. Our findings suggest that this research

domain promises high impact in many research fields,

including social network theory, crowd psychology

and commercial applications in sports.

HUCAPP 2020 - 4th International Conference on Human Computer Interaction Theory and Applications

184

Finally, a more challenging but equally

interesting area is that of studying how these systems

could be personalized for individual needs and

expectations. Results indicated that different athletes

react differently to cheering. Through a psychological

framework, further work could reveal which traits

determine the relevance or otherwise of remote

cheering for individual personalities.

REFERENCES

Bernstein S Michael. 2010. Crowd-powered

interfaces. Proc. UIST'10, ACM Press, 347–350.

http://doi.org/10.1145/1866218.1866220

Boehm B. 2000. The art of expectations

management. Computer 33, 1: 122–124.

http://doi.org/10.1109/2.963135

Boulos N Maged and Resch Bernd. 2011. Crowdsourcing,

citizen sensing and sensor web technologies for public

and environmental health surveillance and crisis

management: trends, OGC standards and application

examples. International Journal of Health

Geographics 10, 67: 1–29. https://doi.org/10.1186/

1476-072X-10-67

Brady E and Bigham J P. 2014. Crowdsourcing

Accessibility: Human-Powered Access

Technologies. Interaction 8, 4: 273–372.

http://dx.doi.org/10.1561/1100000050

Curmi Franco, Maria Angela Ferrario, and Jon Whittle.

2014. Sharing real-time biometric data across social

networks. Proc. DIS’14 Designing Interactive Systems,

ACM Press: 657–666. http://doi.org/10.1145/

2598510.2598515

Curmi Franco, Maria Angela Ferrario, and Jon Whittle.

2017. Biometric data sharing in the wild_ Investigating

the effects on online sports spectators. Journal of

Human Computer Studies 105: 56–67. http://doi.

org/10.1016/j.ijhcs.2017.03.008

Curmi Franco, Maria Angela Ferrario, Jen Southern, and

Jon Whittle. 2013. HeartLink: open broadcast of live

biometric data to social networks. Proc. CHI'13 Human

Factors in Computing Systems, ACM Press: 1749–

1758. http://doi.org/10.1145/2470654.2466231

Curmi Franco, Maria Angela Ferrario, Jon Whittle, and

Florian Floyd' Mueller. 2015. Crowdsourcing

Synchronous Spectator Support: (go on, go on, you're

the best)n-1. Proc. CHI'15 Human Factors in

Computing Systems, ACM Press: 757-

766.http://doi.org/10.1145/2702123.2702338

Dahlbäck N , Jönsson A, and Ahrenberg L. 1993. Wizard

of Oz studies: why and how. Proc. Intelligent User

Interfaces '93, ACM Press: 193-200.

https://doi.org/10.1016/0950-7051(93)90017-N

Fox A , Davies N , and De Lara E. 2006. Real-world

ubicomp deployments: Lessons learned. Pervasive

Computing 5, 3: 21–23. https://doi.org/10.1109/

MPRV.2006.58

Gaver W W, Beaver J, and Benford S. 2003. Ambiguity as

a resource for design. Proc. CHI'03 Human Factors in

Computing Systems, ACM Press: 233-240.

https://doi.org/10.1145/642611.642653

Johnson Rose, Rogers Yvonne, Van der Linden Janet, and

Nadia Bianchi-Berthouze. 2012. Being in the thick of

in-the-wild studies: the challenges and insights of

researcher participation. Proc. CHI'12 Human Factors

in Computing Systems, ACM Press: 1135–1144.

http://doi.org/10.1145/2207676.2208561

Kittur Aniket, Chi Ed H, and Suh Bongwon. 2008.

Crowdsourcing user studies with Mechanical Turk.

Proc. CHI'08 Human Factors in Computing Systems,

ACM Press: 453-456. http://doi.org/10.1145/1357054.

1357127

Lasecki S Walter, Miller D Christopher, and Bigham P

Jeffrey. 2013. Warping time for more effective real-

time crowdsourcing. Proc. CHI'13 Human Factors in

Computing Systems, ACM Press, 2033.

http://doi.org/10.1145/2470654.2466269

Marshall Paul , Morris Richard, Rogers Yvonne,

Kreitmayer Stefan, and Davies Matt. 2011. Rethinking

“multi-user”: an in-the-wild study of how groups

approach a walk-up-and-use tabletop interface. Proc.

CHI'11 Human Factors in Computing Systems, ACM

Press: 3033–3042. http://doi.org/10.1145/1978942.

1979392

Mason Winter and Suri Siddharth. 2012. Conducting

behavioral research on Amazon’s Mechanical

Turk. Behavior research methods 44, 1: 1–23.

https://doi.org/10.3758/s13428-011-0124-6

Morris R Robert and Picard Rosalind. 2012.

Crowdsourcing Collective Emotional

Intelligence. arXiv.org.

https://arxiv.org/abs/1204.3481v1

Mueller Florian, Vetere Frank, Gibbs R Martin, Edge

Darren, Agamanolis Stefan, and Sheridan G Jennifer.

2010. Jogging over a distance between Europe and

Australia. Proc. UIST'10, ACM Press, 189–198.

http://doi.org/10.1145/1866029.1866062

Reeves Stuart, Benford Steve, O'Malley Claire, and Fraser

Mike. 2005. Designing the spectator experience. Proc.

CHI'05 Human Factors in Computing Systems, ACM

Press: 741-750. https://doi.org/10.1145/1054972.

1055074

Rogers Yvonne. 2011. Interaction design gone wild:

striving for wild theory. interactions 18, 4.

http://doi.org/10.1145/1978822.1978834

Rogstadius J, Kostakos V, Kittur A, B Smus, and Laredo J.

2011. An Assessment of Intrinsic and Extrinsic

Motivation on Task Performance in Crowdsourcing

Markets. Proc. ICWSM: 321-328. https://doi.org/

10.13140/RG.2.2.19170.94401

Ryan M Richard and Deci L Edward. 2000. Self-

determination theory and the facilitation of intrinsic

motivation, social development, and well-

being. American psychologist 55, 1: 68–78.

Experiences in Designing HCI Studies for Real-time Interaction across Distributed Crowds and Co-located Participants

185