Experiences in Designing HCI Studies for Real-time Interaction
across Distributed Crowds and Co-located Participants
Franco Curmi
and Conrad Attard
Department of Marketing, Faculty of Education Management and Accountancy, University of Malta, Msida, Malta
Computer Information Department, University of Malta, Msida, Malta
{franco.curmi, conrad.attard}@um.edu.mt
Keywords: Real-time HCI, Real-time Interaction, Instant-method, In-the-Wild, Distributed Participants, Distributed
Interaction, Co-located Participants.
Abstract: This paper is a post-hoc reflective case study from the point of view of the research investigators. The authors
share the experience of designing and deploying four studies that involve real-time interaction between
distributed crowds and co-located participants. We first recount the challenges that these uncommon, yet
increasingly necessary, HCI research contexts afford. We then present the learning outcomes from 1) the
‘designing’, 2) the setting up, 3) the real-time dynamics and 4) the interaction between distributed and co-
located participants. From this we deduce the impact for the four stakeholders in these contexts 1) the
distributed crowd, 2) the co-located participants, 3) the system owners and 4) the researchers. This meta-
research approach is motivated by our struggle to find more ‘Researcher-experience’ cases during the early
stages of the studies. This contribution in experience sharing is intended to help HCI researchers who are
planning studies in this field.
This work presents the experience as a post-hoc
reflective case study after a five-year timeframe of
conducting HCI studies with real-time distributed
interaction of online crowds. In this paper, we
aggregate the insights from four studies in a sports
context. Through the lens of the researchers, we share
the challenges that these HCI settings afford. These
studies required the development of a series of
custom applications that allow distributed crowds to
communicate their emotional support to athletes in
real-time. Through four in-the-wild deployments we
then broadcast sensor-data from athletes to globally
distributed spectators in custom-designed data
visualisations. These visuals help spectators build an
understanding of the athletes’ remote performance.
Concurrently, supporters were prompted to
externalise their support in the form of remote
cheering Across the studies we were interested in
investigating four central questions to 1) understand
the experience of the data-sharing athletes while
receiving remote support, 2) identify factors that
influence remote spectators’ behaviour during live
events, 3) identify motivations for using real-time
spectator support systems, and 4) provide guidelines
for researchers and designers that seeks to facilitate
support from remote spectators during sports events.
In this paper, we aggregate the learning from these
individual studies and use the experience to share the
challenges faced by researchers when planning and
conducting complex but increasingly necessary HCI
research that involves real-time interaction across
distributed participants. This is intended to guide
other researchers who are planning studies in this
exciting research area.
This work was composed of four phases as follows.
1. We first conducted a feasibility study to assess the
viability of investigating real-time support from
remote crowds in a sporting context, identify any
ethical issues that may arise from the study and
gather preliminary insights on how to design
systems for remote spectator support. This
feasibility study was composed of desktop
research and two in-the-wild deployments during
two sporting events (Curmi, 2013).
2. With the insights gained in Phase 1, we then
designed and build BioShare, a customizable
research tool that facilitates sharing live data over
social networks and allows remote spectators to
send instant feedback (Curmi, 2014).
3. With BioShare, we than developed and deployed
HeartLink, in an ad-hoc in-the-wild 5k event with
5 athletes and 140 remote spectators. HeartLink is
an application that allows athletes and spectators
to interact in real-time during events (Curmi,
4. Finally, we conducted a fourth in-the-wild
deployment during a 24-hour 170-mile relay race
with 13 athletes and 261 spectators (Curmi, 2017).
The next sections briefly describe each of these
individual studies. We present the methods used
together with an overview of the learning outcomes
from each of these phases.
2.1 Phase 1: Feasibility Study
Consisting in Desktop Research, a
Pilot Study and a User Study
Phase 1 (Curmi, 2013), assessed the viability of
investigating remote crowd support. It gathered
insights on possible ethical issues that should be taken
into consideration when deploying events in-the-wild
within this context and captured requirements for
system design.
Through desktop research we first reviewed
existing commercial mobile phone applications that
were designed for sports activities. We found that
applications at the time of conducting the study, did
not allow spectators to communicate with athletes
during events. We also identified that academic
research on sports applications is very limited
particularly when it comes to the sharing of live
personal data. In this light, before conducting in-the-
wild deployments, through a survey, we assessed the
readiness of participants from a university setting to
share personal data while conducting sports activities.
A pilot study and a user study were then
conducted. These investigated the technical issues
involved when athletes share data in the wild. These
also gathered primary data on the athletes’ and the
spectators’ experience. A pilot study took place
during a triathlon in the Lake District and focused
primarily on validating the technology. A user study
was then conducted during a charity run in Lancaster,
UK. This focused primarily on capturing the
participants’ experience. Analysis of the data that was
captured through observations, server-interaction
logs, interviews and content analysis of online
discussions during the events, indicated that research
in remote-crowd support is worth pursuing.
However, the use of third-party communication
applications that were used to share athletes’ data
within an in-the-wild research context, presented
several challenges that included a lack of control on
data integrity and reliability. These also limited the
ability to measure user experience and behaviour thus
motivated the development of a bespoke data sharing
system for researching remote spectator support:
In summary, the contributions of this phase 1)
confirmed that further research in remote crowd
support is worth pursuing, 2) provided preliminary
insights on how to build crowd support systems
around athletes and spectators, and 3) highlighted the
need to create dedicated tools for researchers working
in this area.
2.2 Phase 2: System Design and
In Phase 2 (Curmi, 2014) we designed and developed
BioShare. The requirements capturing for developing
Bioshare involved three stages.
We first reanalyze the data collected in Phase 1
and identified key system requirements.
We were interested in making Bioshare relevant
for other researchers working in this area.
Consequently, to validate whether the insights gained
from our experience in deploying two in-the-wild
studies matches the requirements of researchers who
developed closely related systems, we then compared
and contrasted our insights with those of closely
related systems that are referenced in literature.
We found that systems that are referenced in
literature lack details on how these systems were
developed and details on issues that emerged during
their development, if any. Thus, we further
investigate past systems’ development by
interviewing HCI researchers who created closely
related data sharing systems for research applications.
The developed prototype consisted in a native
mobile application that broadcasts users’ locative and
physiological data over mobile networks and received
feedback from online crowds. A web interface
together with a dedicated backend allowed distributed
crowds to follow and communicate with the data-
sharing users. BioShare is open-source and is
designed such that it can be configured for different
study requirements.
In addition to contributing BioShare as a tool for
researchers, this phase also contributed a set of
requirements for spectator support systems in the
presented context. These include ethical
considerations, design for adaptability and a
framework to empower the user over the shared data.
2.3 Phase 3: Deployment in an 5k-road
A customised version of BioShare, HeartLink, was
then deployed in a 5k-road race with 5 athletes and
140 remote spectators. In this deployment, we 1)
captured the experience of athletes when sharing data
and receiving remote support in real-time and 2)
identified key influencers to the supporters’
behaviour during a live sport event. Pilot studies
suggested that spectator engagement is influenced by
both the data that is presented (e.g. the effort that the
athlete is exerting) as well as the social relation
between the athlete and the supporter. To validate
this, we recruited two spectator groups. One spectator
group was recruited through the athletes’ own social
networks. Thus, the spectators in this group knew the
athletes. A second spectator group was recruited from
a crowdsourcing platform and thus these spectators
had no social connection with the athletes.
Additionally, to compare whether different data
types, particularly heart rate data, influences the
spectators’ engagement, all the spectators were
randomly assigned to one of two conditions. One
group was presented with locative data while a
second group was presented with both locative data
and heart rate data of the athletes. The results
indicated that having a social tie with the athletes
increases engagement when supporting the athletes.
These spectators cheered the athletes more and spent
more time supporting. Spectators who were presented
with the additional heart-rate data in their interface
also cheer significantly more. Additionally, through a
focus group, the athletes suggested that the
motivation for athletes to use remote spectator-
support systems is dependent on the effort that the
task entails and the degree of loneliness that the event
presents. To further investigate this, we then
conducted a fourth in-the-wild deployment during a
24-hour 170-mile long relay race across the UK.
In summary, Phase 3 (Curmi, 2015) contributed
the following:
Through quantitative data, the work highlighted
differences in spectator behaviour across spectators
who are presented with different visuals, and
spectators who have different social relationships
with the athletes. For example, we found that
spectators who are presented with additional
information about the heart rate of remote participants
are likely to feel more emotional and consequently
cheer more.
Through qualitative data, the study identified key
motivations for using live remote cheering systems.
For example, we identified that spectators’ behaviour
depended on their understanding of why the athletes
are conducting the task (e.g. the assumed athletes’
egoistic vs. altruistic objectives in participating in an
event). With regards to the athletes’ motivation, we
identify that the impact that the cheering has on the
athletes is relative to their expectations. This and
similar outcomes, are congruent with existing
theories of expectations management (Boehm, 2000)
and self-determination theory (Ryan and Deci, 2000).
2.4 Phase 4: Deployment in a 170-mile
Relay Race
For this event, BioShare was customised and
embedded in a running relay-baton form factor
(Curmi, 2017). This baton works as an interface
between the remote crowd and the athletes. The
baton’s form-factor also provides enough space to
store the needed energy for the 24-hour long event.
Following a co-design process with the athletes, the
prototyped baton collects and broadcast data in real-
time and vibrates whenever a remote supporter clicks
a cheer button on the web interface. Additionally, the
baton also calls out the name of the person who sent
the ‘cheer’. In this way, the athletes get an awareness
of where the cheers came from.
Through these deployments, we further analysed
and deduce user-motivations for using real-time
crowd-support systems. Athletes reported motivation
from: receiving remote support, building followship,
having a proof of accomplishment, satisfying a social
need to connect with others, democratising sport
events and facilitating mindfulness about the event,
among others.
Additionally, the data collected through these
deployments provided insight on key factors that need
to be taken into consideration when engineering real-
time crowd support systems. These are presented in
three categories:
1) Spectators’ expressiveness i.e. the design of how
spectators can externalize their support. This can
range from a highly controlled form (e.g. simple
binary ‘Likes’) to a more open approach such as user-
generated communication (e.g. live audio streaming
of aggregate cheers from spectators’ microphones).
2) Context applicability i.e. we identify contexts
where remote spectator support seems most pertinent.
Findings indicate that these systems seem to be most
valuable in challenging events and where the athletes
feel lonely (e.g. participating in an unaccompanied
setting at night-time). On the other hand, remote
support appears less useful in competitive events.
3) The design of the data flows within the social
network. Here designers need to consider how system
users (athletes, spectators and organisers)
communicate and design communication flows.
A common element in these studies is the adoption of
an in-the-wild approach. Unlike traditional
experimental methods that take place within the lab
(Johnson, 2012), the method go beyond observing
existing practice and present opportunity to evaluate
novel technology in the place where the technology is
intended to be used (Morris, 2012; Marshall, 2011).
Research in the wild is an old practice. Centuries-old
inter-continent expeditions that inform ship design
may classify within the definition. However, over the
last decade, research in-the-wild became a common
research practice in HCI. As in our study, HCI
researchers often seek to explore new technology, test
prototypes in the location in which they are intended
for and understand how people interpret and
appropriate the technology (Dahlbäck, 1993; Gaver
2013; Kittur, 2008, Rogers 2011).
Prototypes were deployed with participants in
different cities, countryside pathways, cycle lanes,
nature parks and inside a lake. For example, the final
prototype deployment connects athletes running a
170-mile race, from coast to coast, across the UK. In
this setting, research in-the-wild allowed compare
and contrast the effect of mobile data connectivity on
the proposed technology across different
environments within the same deployment.
Evaluating technology in-the-wild poses a
number of added challenges. These challenges go
beyond the lack of comfort that out of the studio
participants are presented with (Rogers, 2011). An in-
the-wild study may suffer from lack of control that a
lab facilitates (Laseki, 2013). Consequently,
extrapolating specific effects becomes difficult and
researchers need to interpret data that is influenced by
several externalities and interdependencies
(Rogstadius, 2011). Using multiple methods of data
collection often compensates for this. Where
possible, we triangulate findings across different data
sources. Eight data collection methods were used
throughout the study, namely, surveys, literature
reviews, focus groups, semi-structured interviews
with athletes, spectators and HCI researchers, content
analysis of social network comments posted during
deployments, quantitative data of online users’
interaction that is collected by the data servers,
observations, and research through designing four
data telemetry prototypes and four online-crowd
The challenges that research in-the-wild presents
are widely documented in literature (Mueller, 2010;
Fox, 2006; Rogers, 2011). However, over and above
these challenges, this work faced additional unusual
dynamics. Each of these augments the complexity of
running the study (Figure 1). Namely, 1) the need for
co-ordinating a group of co-located participants that
are conducting a challenging task in-the-wild, 2) the
need of co-ordinating a group of globally distributed
participants and 3) the need for all activities to operate
in real-time with synchronous interaction at a global
scale. The latter does not afford the traditional lab
recruitment approach where the researcher schedules
participants at a time when it is most convenient for
each participant. In such HCI studies, all the
participants have to synchronise with the live event.
In this context, recruiting participants,
particularly online spectators, requires rigorous
planning. Online spectators may be less difficult to
recruit than co-located athletes since much less effort
is needed when participating in an online task than
when participating in a physically challenging task
such as a long-distance race. Additionally, there is
typically no travelling involved. The participants do
not feel they are being watched and they might do
work in parallel to following and/or supporting the
athletes. For example, Manson points out that
participants might have coffee while engaging in an
online event (Manson, 2012). Another important
consideration is ‘Attrition’. In a lab experiment, it is
very unlikely that a participant walks out of an
experiment due to unstated pressure from being in a
face-to-face situation. This does not apply to an
online environment where participants may easily
leave the experiment at the click of a button. The
participants may also be distracted by various other
factors such as surfing other websites, making errands
or experience technical system failure, to mention a
few. To monitor this, we placed occasional prompts
to monitor attention. The system then logs the time
taken for each viewer to respond and this measure
may then be compared to different participant groups
and collected datasets.
Additionally, to mitigate complexity, we start
with a small-scale deployment that has few
participants, to then increment the scale of the
deployment iteratively. This approach promises 1)
incremental improvement, 2) contains any emergent
ethical issues and 3) minimises risk of failure.
Figure 1: Methodological influencing contexts.
4.1 Conducting RIW Deployments with
Synchronous Interaction between
Distributed Participants
As earlier stated, all the deployments were conducted
in the wild. This made planning, organising and
deploying extremely challenging. Moreover, each
study involved multiple user groups that were not
only in the wild but also distributed across different
locations. Additionally, the interaction under
investigation was synchronous. Thus, the participants
needed to synchronize themselves to the study rather
than the study synchronises to the participants. The
need for participants to synchronise with the study
limits the number of participants that can take part in
the study as their participation is not only conditioned
by their willingness and appropriateness to the sample
group but also by their personal schedule. This
restriction is particularly visible when the user group
is friendsourced, that is, the participants have a social
tie with the athletes. On the other hand, this is less
restricting for outsourced participants, that is, where
participants have no social connection with the
athletes as the sample frame may be larger.
Outsourced participants were recruited from
CrowdFlower, a crowdsourcing platform.
Crowdsourcing platforms provide a large enough
pool of participants (crowd workers) who are seeking
work that matches their expected enjoyment and
financial return. The enjoyment value is a major
factor in the recruitment process. Many studies show
that crowdsourced participants value the pleasantness
of a given task (Rogstadius, 2011). This impact both
the engagement of participants in the task and also the
reputation on the platform (through post-task-
completion feedback) of the researcher who issued
the study.
4.2 Contrasting Real-time in the Wild
vs Lab
Inversely should there was a lab variant of the studies,
the challenges would have been very different. 1) The
researchers would not have been bound with
recruiting a large group of participants to perform at
one specific global time. 2) The researchers would
have had more control over the environment and
confounding variables would have been limited. For
example, weather conditions would have been
minimally influential on the study, if any. Windows
would have been closed and, say, a treadmill could be
kept the same gradient for all participants such that all
the participants would have been presented with an
identical controlled experience. Similarly, cheers
could have been computer generated from predefined
patterns that would simulate remote cheering crowds.
3) Running such an experiment would have been
easier because the researcher would be in the comfort
of the well-known, tried and tested, “lab”. The
researcher could have wired, handled and observed
one participant at a time.
However, no matter how controlled this
environment would have been, it would have never
been anything close to the real thing; the in-the-wild
environment with real crowds sending self-initiated
cheers at that moment in time.
In hindsight, an in the wild deployment that
involves distributed crowds, like the four
deployments in this study, are more unforgiving then
an in the lab approach where single participants
sequentially conduct offline sessions. For example, if
there is an issue with the system, such as what
happened in the pilot study due to downtime on US
Amazon Web Services, then the researcher needs to
coordinate a crowd that is distributed. This is
challenging, not only because the investigation
involves a crowd but also because the data is live and
distributed. In this case, the research event is likely to
fail or at minimum, the research objectives would
change. Moreover, a new event is likely to require
In the
d online
coordinating a new crowd. Should that have been in
the lab, a participant would have been ranked as an
outlier or replaced with an additional lab session.
Finally, in a lab variant, systems can be tested, and
researchers can pose as dummies. In an in-the-wild
version where crowds operate synchronously,
systems are difficult to test comprehensively. For
example, collecting enough participants to simulate a
crowd to test a system in situ is often impractical.
Furthermore, if the researcher does manage to do this,
in most cases, the in-the-wild environment is likely to
change over time. Hence, reliability across all
variables is challenging at best. For example, data
telemetry that is dependent on mobile network
coverage (reception) may be influenced by the
density of users that happen to be in that public area
and of which the researchers have little or no control.
These challenges that research in the wild with
real-time interaction brings to the table further
highlight the differences of real-time in the wild over
an in-lab study. These differences emphasise the
distinct values that both approaches present the
researcher with. Based on the above considerations,
we recommend the following to researchers who
intend conduct research on the lines of this work:
1. The researcher should seek to control any variable
that can be controlled but plan contingencies for
unexpected events.
2. Conducting meticulous planning minimizes the
risks of unexpected outcomes.
3. In these contexts, observations during the event as
part of the research method is highly important
and details should be documented during or
immediately after the event.
4. The researcher should have a communication
channel with remote participants. This is essential
for ad-hoc coordination should unplanned
phenomena occur.
Roles for documenting events and coordinating
events should be assigned to separate individuals.
Due to the complexity that such tasks entail, we
recommend that researchers build a research team
where each member is assigned a pre-designed role.
Different studies and conditions would necessitate
different roles for supporting staff. In the case of the
study in Phase 4, the 170-mile relay race, the
recommended roles for the event such that the
researchers can focus on core areas were as follows:
1) A researcher was assigned to coordinate the online
crowd. This person would, for example, message the
crowd should there be a need to do so during the live
event and answer any queries that online participants
may have. 2) A person needs to coordinate the co-
located athletes. 3) An experienced researcher
journals the event. 4) A videographer and/or
photographer may provide grounded content for post-
event analysis or in support of post-event
4.3 Managing Interfaces with Multiple
The deployed interfaces were driven by two main
sources: The athletes and the spectators. In their
work, “Designing the Spectator Experience”, Reeves
et al. classify these interfaces as public interfaces.
These are public interfaces not because the interfaces
are out in the wild but rather because of the “extent to
which [the] performer’s manipulations of an interface
and their resulting effects are hidden, partially
revealed, fully revealed or even amplified for
spectators.” (Reeves, 2005 p.741). In recent years,
there have been numerous discussions within related
communities, such as SIGCHI, on interfaces that are
moving away from providing an individual dialogue
but rather are designed for a crowd (Boulos 2011;
Brady, 2014) and driven by a crowd (Laseki, 2013).
In most of these cases, as it is the case of the above
studies, the crowd is distributed.
In a real-life cheering context, within an open
sporting event, the human interaction is intended to
be public. Spectators cheer from the sides of a
racecourse or from the stands at a stadium. On the
other hand, in the digital world, there is by far more
human-human communication that is designed for a
private setting rather than a public one. Public
telephones for example are enclosed in boxes or
photo kiosks (Rogers, 2011). There are also multiple
levels of engagement. There are spectators who
follow the data through the crowd-powered interface,
hence they can follow the data of the athletes and the
data that is driven by the crowd (i.e. the live cheering,
the spectators live comments as the event unfolds and
the number of spectators that join and leave the event,
during the event). There are also the supporters, that
is, those spectators who do not simply follow the data
but also interact with the interface and the athletes by
cheering and commenting, hence contributing to the
live discussion. Finally, there are the athletes whose
interaction is highly automated, both the sharing of
data, and in receiving feedback from the crowd. In a
way, the interface of the athletes is hidden and
inexistent. It is an extension of the crowd’s interface.
For example, the relay baton that was deployed in
Phase 4, opens a channel to the crowd. The athletes
do not interact with the baton, but the baton automates
the communication from the athlete to the crowd. The
baton captures the data and sends it to the crowd
without any interaction from the user (the athlete).
There are also the co-located spectators, who,
although they might not interact with the cheering
system or the online crowd, they may also influence
the online environment through the athlete’s
environment. In this regard, Reeves et al. add another
dimension to public vs. private dichotomy;
manipulations and effects, where manipulations are
the actions of the ‘performer’ (in our case these are
the actions of the athlete). On the other hand, the
effect is the impact of the manipulations; a click on a
cheer button triggers a vibration (direct effect) and
may make the athlete aware of the support being sent.
The athlete may perform better and his or her exertion
may influence the gradient of the chart representing
the live heart rate (indirect effect).
The work of Reeves et al. helps us position the
authors’ interface within the spectrum of interfaces.
Interface categories include 1) Magical, this refers to
interfaces that hide the manipulations but the effects
are revealed (e.g. wizard of oz interface (Reeves,
2005)), 2) Secretive, where both the effects and the
manipulations are hidden (e.g. within a competitive
game), 3) Suspenseful, where manipulations are
hidden but the effects are revealed and finally 4)
Expressive, where both the manipulations and the
effects are revealed or amplified. Within this
taxonomy, the proposed cheering system positions
itself in the expressive quadrant. Spectators’ actions
are channelled to the athletes and amplified through
haptic and sound synthesisers. The athletes’
performance is sensed and amplified to all connected
spectators within the spectators’ interfaces.
The real-time cheering system associates another
dimension to this. The interface is not only generated
and interacted-with by the individual and displayed to
the spectators as a crowd, but this crowd also drives
the interface. In other words, the interface (including
the cheers, the number of online spectators and the
live comments that make up the interface) is
generated from the crowd. These become, as Michael
Bernstein coined, crowd-powered interfaces. Crowd-
powered interfaces are “interfaces that rely on human
activity traces or human computation to provide
benefits to the end user.” (Bernstein 2010 p.347).
Undoubtedly, the cheering process is explicitly
conducted for the benefit of an end user, the athlete.
We argue that this process relies on both ‘human
activity traces’ and ‘human computation’. They rely
on human activity traces as the distributed individuals
trigger the cheers, and each has his or her intentions
and motivations to cheer. The human computation
component is highlighted in the user interviews.
Upon interview, the spectators showed interest in
maximising the positive impact that the cheers could
have on the athlete. In this regard, spectators devised
strategies such as leaving more cheers towards the
end of the race, ‘such that the cheered athletes do not
get used to the cheering’. These strategies are
reflected in the cumulative cheering plots.
4.4 Managing Multiple
Communication Modalities
Across the studies, we looked at primarily two
communication flows. Informing the online crowd
(spectators) and informing the athletes. The athletes
awareness of spectators’ behaviour and their support,
can contribute to build a sense of ‘liveness’(Mueller,
2010). However, it can also generate pressure on the
athlete. The sense of ‘being observed’ that real-time
remote support systems create, may make the athlete
feel obliged to perform or feel embarrassed for
mistakes since spectators are following the
performance. The modality that is adopted to
communicate the crowd’s support to the athlete is an
influential factor in the design of the athletes’
experience. The deployments explored tactile and
sound alerts to communicate the cheering crowd.
Results showed that the effect of the communication
modality is dependent on different externalities over
and above the modality itself. These include the
context (e.g. the background noise within the
environment), the trustworthiness of the cheering
crowd (e.g. whether there are spammers among the
cheering crowd who might misuse the modality, say,
send inappropriate messages) and the individual
personalities of the athletes that the set modality is
communicating with. For example, when the
modalities where calibrated to generate the same
intensity of tactile feedback for all participants, some
athletes did not feel the set vibration. This seemed
related to the athlete’s body mass and athletes with
larger arms were more likely not to feel the vibrations
that were triggered by the telemetry device thus
emphasising the need for personalisation.
We believe that these findings are just the beginning
of this research area and we trust that other studies
will follow. What is presented here may provide the
preliminary groundwork for real-time remote crowd
support. Our findings suggest that this research
domain promises high impact in many research fields,
including social network theory, crowd psychology
and commercial applications in sports.
Finally, a more challenging but equally
interesting area is that of studying how these systems
could be personalized for individual needs and
expectations. Results indicated that different athletes
react differently to cheering. Through a psychological
framework, further work could reveal which traits
determine the relevance or otherwise of remote
cheering for individual personalities.
Bernstein S Michael. 2010. Crowd-powered
interfaces. Proc. UIST'10, ACM Press, 347–350.
Boehm B. 2000. The art of expectations
management. Computer 33, 1: 122–124.
Boulos N Maged and Resch Bernd. 2011. Crowdsourcing,
citizen sensing and sensor web technologies for public
and environmental health surveillance and crisis
management: trends, OGC standards and application
examples. International Journal of Health
Geographics 10, 67: 1–29. https://doi.org/10.1186/
Brady E and Bigham J P. 2014. Crowdsourcing
Accessibility: Human-Powered Access
Technologies. Interaction 8, 4: 273–372.
Curmi Franco, Maria Angela Ferrario, and Jon Whittle.
2014. Sharing real-time biometric data across social
networks. Proc. DIS’14 Designing Interactive Systems,
ACM Press: 657–666. http://doi.org/10.1145/
Curmi Franco, Maria Angela Ferrario, and Jon Whittle.
2017. Biometric data sharing in the wild_ Investigating
the effects on online sports spectators. Journal of
Human Computer Studies 105: 56–67. http://doi.
Curmi Franco, Maria Angela Ferrario, Jen Southern, and
Jon Whittle. 2013. HeartLink: open broadcast of live
biometric data to social networks. Proc. CHI'13 Human
Factors in Computing Systems, ACM Press: 1749–
1758. http://doi.org/10.1145/2470654.2466231
Curmi Franco, Maria Angela Ferrario, Jon Whittle, and
Florian Floyd' Mueller. 2015. Crowdsourcing
Synchronous Spectator Support: (go on, go on, you're
the best)n-1. Proc. CHI'15 Human Factors in
Computing Systems, ACM Press: 757-
Dahlbäck N , nsson A, and Ahrenberg L. 1993. Wizard
of Oz studies: why and how. Proc. Intelligent User
Interfaces '93, ACM Press: 193-200.
Fox A , Davies N , and De Lara E. 2006. Real-world
ubicomp deployments: Lessons learned. Pervasive
Computing 5, 3: 21–23. https://doi.org/10.1109/
Gaver W W, Beaver J, and Benford S. 2003. Ambiguity as
a resource for design. Proc. CHI'03 Human Factors in
Computing Systems, ACM Press: 233-240.
Johnson Rose, Rogers Yvonne, Van der Linden Janet, and
Nadia Bianchi-Berthouze. 2012. Being in the thick of
in-the-wild studies: the challenges and insights of
researcher participation. Proc. CHI'12 Human Factors
in Computing Systems, ACM Press: 1135–1144.
Kittur Aniket, Chi Ed H, and Suh Bongwon. 2008.
Crowdsourcing user studies with Mechanical Turk.
Proc. CHI'08 Human Factors in Computing Systems,
ACM Press: 453-456. http://doi.org/10.1145/1357054.
Lasecki S Walter, Miller D Christopher, and Bigham P
Jeffrey. 2013. Warping time for more effective real-
time crowdsourcing. Proc. CHI'13 Human Factors in
Computing Systems, ACM Press, 2033.
Marshall Paul , Morris Richard, Rogers Yvonne,
Kreitmayer Stefan, and Davies Matt. 2011. Rethinking
“multi-user”: an in-the-wild study of how groups
approach a walk-up-and-use tabletop interface. Proc.
CHI'11 Human Factors in Computing Systems, ACM
Press: 3033–3042. http://doi.org/10.1145/1978942.
Mason Winter and Suri Siddharth. 2012. Conducting
behavioral research on Amazon’s Mechanical
Turk. Behavior research methods 44, 1: 1–23.
Morris R Robert and Picard Rosalind. 2012.
Crowdsourcing Collective Emotional
Intelligence. arXiv.org.
Mueller Florian, Vetere Frank, Gibbs R Martin, Edge
Darren, Agamanolis Stefan, and Sheridan G Jennifer.
2010. Jogging over a distance between Europe and
Australia. Proc. UIST'10, ACM Press, 189–198.
Reeves Stuart, Benford Steve, O'Malley Claire, and Fraser
Mike. 2005. Designing the spectator experience. Proc.
CHI'05 Human Factors in Computing Systems, ACM
Press: 741-750. https://doi.org/10.1145/1054972.
Rogers Yvonne. 2011. Interaction design gone wild:
striving for wild theory. interactions 18, 4.
Rogstadius J, Kostakos V, Kittur A, B Smus, and Laredo J.
2011. An Assessment of Intrinsic and Extrinsic
Motivation on Task Performance in Crowdsourcing
Markets. Proc. ICWSM: 321-328. https://doi.org/
Ryan M Richard and Deci L Edward. 2000. Self-
determination theory and the facilitation of intrinsic
motivation, social development, and well-
being. American psychologist 55, 1: 68–78.