Measuring Immersion in Experiences with Biosensors

Preparation for International Joint Conference on Biomedical Engineering

Systems and Technologies

Paul J. Zak

1,2

and Jorge A. Barraza

2,3

Center for Neuroeconomics Studies, Claremont Graduate University, 160 E 10

St, Claremont, CA, U.S.A.

Immersion Neuroscience, 340 S. Lemon Ave #2358, Walnut, CA 91789, U.S.A.

Department of Psychology, University of Southern California, Los Angeles, CA, U.S.A.

Keywords: Immersion, Engagement, Consumer Neuroscience, Mass Experiences.

Abstract: When people are engaged in an immersive task or experience, they can become so absorbed in it that they

lose track of time and place. Narrative transportation, has similar effects, producing meaningful psychological

responses and influencing behavior. Those seeking to create immersive experiences typically rely on

inaccurate self-report to measure immersion. We describe research from our group on the neuroscience of

immersion and our development of a physiologic sensor, algorithm, and software suite that measures

immersion. Our studies show that immersion predicts enjoyment, recall of information, and actions after an

experience with 75%-95%% accuracy depending on the outcome measure. We discuss the trade-offs when

developing sensor technologies designed for non-laboratory environments.

1 WHAT IS IMMERSION?

Immersion, an experience of deep involvement with

an aspect of the environment, is a psychological and

neurological process that is not well understood but

has very strong and meaningful implications in

everyday life. The concept of immersion has its roots

in prehistory as bands of homo sapiens related stories

of hunts and travels in order to instruct and perhaps

entertain members of their clans. Aristotle's two great

books, Poetics and Rhetoric lay the theoretical

foundation for immersion. While Poetics created a

structure for a dramatic play or narrative reading to

have an emotional impact on audiences, Rhetoric

identifies how a speaker can persuade others by

structuring his or her narrative. Combining these two

approaches led to the development of the dramatic arc

used in nearly all storytelling. The five parts of the

dramatic arc (exposition, rising action, climax, falling

action, and dénouement) were meant to transport

audiences into the narrative so they would experience

a simulacrum of the story. In the 1990s,

psychologists coined the term "narrative

transportation" and showed that a key aspect of

transportation occurred when an audience member

empathized with story characters (Gerrig, 1993).

Empathy for characters can persuade audiences to

change their attitudes and opinions. For many

viewers, watching the physical training the boxing

movie Rocky or the self-sacrifice in the film about the

Battle of Thermopylae, 300, combined with an

emotionally-compelling reason for the suffering,

motivated audience members to join a gym or take up

boxing or enroll in a boot camp-type training

program.

Immersion is similar to "flow" (Csikszentmihalyi,

1997), a state of concentration or absorption in an

activity, like work or performance. While flow states

require an active participation in a task, immersive

experiences can be purely passive, but have similar

positive psychological effects like deep concentration

and feelings of awe or transcendence. Immersion

encompasses the concept of flow, defining it beyond

active boundaries to include passive experiences like

transportation in a story.

Herein, we present a review of the work we have

done to quantify individual and group immersion

using commercial wearable sensors.

Zak, P. and Barraza, J.

Measuring Immersion in Experiences with Biosensors - Preparation for International Joint Conference on Biomedical Engineering Systems and Technologies.

DOI: 10.5220/0006758203030307

In Proceedings of the 11th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2018) - Volume 4: BIOSIGNALS, pages 303-307

ISBN: 978-989-758-279-0

303

2 WHY SHOULD WE MEASURE

IMMERSION?

The experience economy depends on creating highly

immersive experiences for employees and consumers

alike.

2.1 Organizations

Companies strive to engage their employees in their

work beyond extrinsic motivation. The burgeoning

field of positive organizational behavior (POB)

provides increasing evidence that traditional extrinsic

motivators such as pay and benefits only weakly

engage employees (Seligman & Csikszentmihalyi,

2000; Peterson & Seligman, 2004). Seligman and

Csikszentmihalyi (2000, p. 5) point out that today’s

employees are increasingly “seen as decision makers,

with choices, preferences, and the possibility of

becoming masterful.” POB reflects an increasing

emphasis on “psychological capacities that can be

measured, developed, and effectively managed for

performance improvement” (Luthans, 2002: 59).

Flow states are more likely to occur at work than

during recreation due in part to having clearly defined

goals (Gardener, Csikszentmihaly, & Damon, 2002).

Flow has been tied to performance by improving

concentration and motivation (Engeser & Rheinberg,

2008).

2.2 Consumer Experiences

Consumers are increasingly demanding, and paying

for, extraordinary experiences. As Joseph Pine and

James Gilmore articulate in The Experience Economy

(2012), premium pricing and differentiated products

and services require that experiences faced by

consumers be extraordinary. This means the entire

customer journey, from messaging, to online

information gathering, to in-store interactions, to the

purchasing ceremony, needs to be integrated and

customized for each client.

The push toward immersive content is seen in

marketing and advertising. For instance, the last few

years have seen a rise in 360-degree video,

augmented reality, and virtual reality technologies.

Virtual reality has been shown to increase emotional

engagement and longer engagement periods over

traditional content. Retail companies like Sephora are

creating augmented reality apps that allow the

consumer to engage with their products outside the

store. Indeed, there may be substantial hype around

the use of immersive technologies in marketing and

advertising, but it is clear that immersion itself is

becoming key component to build quality content.

2.3 The Self

Our previous research on immersive experiences

suggests that there is much we can learn about the

impact of immersive experiences on human

flourishing. If immersion is a positive state people

seek out, then it is likely that people will want to learn

about what experiences are immersive for them

specifically. Coupled with the rise of the quantified

self, particularly with information to improve

psychological well-being and productivity, we expect

that the need to quantify immersive experience by the

individual will grow in the coming years.

3 MEASURING IMMERSION

3.1 Traditional Measures

We can be immersed in stories, entertainment, work,

and consumer experiences, but not always. The same

experience can play out differently for different

people. The current state of the art to measure the

quality of experiences for consumers is to ask people

their opinions in a survey that might not be answered

for days or weeks after the experience. For instance,

there exist several scales for story immersion

including the Narrative Engagement Scale (Busselle

& Bilandzic, 2009) and the Narrative Transportation

Scale (Green & Brock, 2000). Task flow has been

measured via self-report using Flow Short Scale

(Rheinberg et al., 2003), the Flow State Scale-2 (FSS-

2) and Dispositional Flow Scale-2 (DFS-2) for

physical activities (Jackson & Eklund, 2002).

Surveys are known to be rife with biases,

including poor recall of event details, providing

socially acceptable answers, and easier recall of

negative experiences. Focus groups, having

participants turn dials, and mass emails all suffer from

a fatal flaw: people cannot accurately or consistently

report their unconscious emotional reactions to

experiences. Moreover, there is a dynamic nature to

experiences that cannot be measured simply by

asking static post-experience questions.

Although it would seem that simply asking people

about their experiences could reveal how immersive

they were, self-report is often unreliable and difficult

to compare across individuals. Moreover, how people

respond to traditional survey scales is heavily

impacted by culture, current physiologic state outside

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

304

the target experience (e.g. fatigue, hunger), and

socioeconomic status.

3.2 Biosensor Measurement

3.2.1 Defining Immersion Biologically

Our decade-plus of research has shown that

immersion depends on two key elements: attention to

the experience and emotional engagement during it.

Both sympathetic and parasympathetic systems

are indicative of attention and emotional engagement.

Attention is associated with energy expended. People

are more likely to attend to stimuli eliciting

sympathetic arousal (Boucsein, 2012; Kensinger,

2004; MacLeod & Matthews, 2004). Activity in both

sympathetic and parasympathetic systems occurs in

response to emotional stories (Eisenberg, Fabes, et

al., 1988; Eisenberg, Fabes, Schaller, Miller, et al.,

1991; Eisenberg, Schaller, et al., 1988). A key

component of the parasympathetic nervous system,

the vagus nerve, is proposed to be central to the

mammalian “social-engagement system” (Porges,

2007), with vagal activity being linked with affective

experiences, most notably empathic concern (e.g.,

Oveis, Cohen, Gruber, Shiota, Haidt, & Keltner,

2009) and trait and state experiences of positive

emotion (DiPietro, Porges, & Uhly, 1992; Oveis et

al., 2009).

3.2.2 Research

Our lab first uncovered this effect by studying the

immersive properties of stories (Barraza & Zak,

2009; Barraza et al., 2015; Lin et al., 2013; Zak,

2015). This series of studies measured and

manipulated neural activity, showing that narratives

that sustain attention and generate emotional

resonance with the story's characters are judged as

more enjoyable, the information better remembered

weeks later, and are more likely to motivate prosocial

costly actions than those that lack one or both of these

responses. For instance, our DARPA (Defense

Advanced Research Projects Agency)-funded

research identified predicted costly actions after a

narrative with 82% accuracy in 2014 (Barraza, et al.,

2015).

The key, we found, is that both neural signatures

for attention and emotional resonance must be present

for costly actions to occur. We have measured

attention in a variety of ways, but an increase in heart

rate and/or in electrodermal activity are robust

measures of the energy expended to sustain attention.

Our research has shown that emotional resonance

corresponds to an oxytocin response measured in

blood samples that correlate with increases in vagal

tone as measured with an electrocardiogram (ECG).

4 TECHNOLOGY

In the last year, our lab has developed software that

uses wearable sensors to capture neural signals

associated with attention (increases in heart rate and

electrodermal activity) and vagal tone (increases in

heart rate variability). Our published research shows

that immersion predicts both individual and group

behaviors, not just intentions or other self-report

measures (Barraza et al., 2015; Zak, 2017). The

behaviors we have been able to accurately forecast

include: donations to charity, recall of brands and

information two weeks after viewing messages,

YouTube views, social media shares, and sales

bumps.

We are currently using off-the-shelf (OTS)

sensors for data collection and have built algorithms

to measure key neurologic variables by testing these

devices against research-grade peripheral neurologic

sensors. Our scalable sensor solution provides

algorithms that quantify one's immersion that varies

form 0-10, so everyone can understand it: higher

score means more immersion. Using advanced signal

processing techniques, we can now measure and

display immersion in real-time.

4.1 How It Works

Participants are invited to put a small sensor on a

stretchy band on a forearm. We call this the IN Band.

The IN Band can be hidden underneath one's shirt

sleeve or worn showing. The IN Band automatically

syncs to a PC or mobile device and collects

individualized, real-time immersion data from four

different signals associated with the brain's control of

the heart and gut. Our signal processing algorithms

show immersion data for one person or 100 within 10

seconds after activating the sensor. It will collect data

for up to 10 hours on a single charge. The cornerstone

algorithm is called the Immersion Quotient

(IQ)

that shows second-by-second immersion while a

participant watches an ad, or shops, or attends a sales

training, or works.

4.2 Using Commercial Sensors

In the last year, our lab has developed a passive

wearable forearm sensor and software suite that will

allow us to capture the neurocorrelates of immersion

Measuring Immersion in Experiences with Biosensors - Preparation for International Joint Conference on Biomedical Engineering Systems

and Technologies

305

in field experiments. The sensor uses the Valencell®

biometric chip which has been benchmarked for

providing above 99% reliability as compared to a

cardiac chest strap. A primary reason for our

investment in this new technology is to permit us to

run field experiments that provide ecologically valid

data to identify immersive experiences.

The problem with neuroscience techniques is they

are often uncomfortable, for example, having

participants lie in an intimidating and noisy MRI

scanner, or being poked with needles to draw blood,

or being wired up to a scary looking

electroencephalogram (EEG) machine. This is fine

for laboratory research, but limits the ability to

measure what real people do in real situations. Our

initial research on immersion was funded by the U.S.

Department of Defense and the U.S. Intelligence

Community and we were required to create

technologies that accurately predicted people's

behaviors and could be used anywhere--even a theater

of war.

We also needed to identify "robust" brain signals

to predict people's behavior. A serious problem in all

neuroscience studies is "signal extraction." Most of

what the brain does is keeping people upright,

breathing, and conscious. For any experience people

have, a small portion of brain activity is responding

to that stimulus. Over the course of 12 years, we very

methodically traced pathways in the brain using

functional MRI, drug infusions, and EEGs in order to

find brain signals that consistently measured people's

immersion in a message, ad, or experience, and

accurately predicted their subsequent actions,

including purchases, donations to charity, recall of

information, and social media shares.

The other problem one encounters when

measuring brain activity is that so much data is

collected, it is difficult to extract robust predictive

signals. "Robust" is a statistical term meaning that the

signal predicts in all or most settings. Scientists seek

to resolve the robustness problem by controlling the

environment (running experiments in laboratories),

using sophisticated equipment, and by using very

well-trained PhDs to process the "big data" that are

collected. From a business perspective, this means

that, for example, the neuromarketing studies Zak and

Barraza have been doing for businesses are

expensive, must be custom-designed for each client,

and take weeks to get results.

To resolve these issues, and build a large and

scalable solution for businesses, we focused on

building a scalable platform to provide a rapid

measure of neural immersion that could be used

outside the laboratory.

4.3 Creating a Platform

We have automated the signal processing by building

a software suite that measures immersion in real time.

It uses the off-shelf sensor wearable that wirelessly

collects immersion data from the peripheral nervous

system using our Immersion Quotient

algorithm.

Our signal processing and predictive algorithms work

through in an online platform that clients use to

collect their own data. Our web interface means that

multiple events can be viewed by clients as they are

collected. Data are archived and can be downloaded

as data files for later detailed analyses and

comparison.

4.4 Real World Application Problems

Going with an off-shelf sensor and automated

algorithms comes with several potential problems.

The sensors are not as reliable or accurate in

measuring peripheral physiology as research grade

sensors. Moreover, there is greater data drop off and

motion artefact when collecting physiological data

outside the laboratory. Fortunately, wearable sensors

have vastly improved, even in the last five years, with

many sensor manufacturers competing on quality and

price. Moreover, the relatively low cost of the sensors

makes it feasible to collect data from hundreds of

people simultaneously.

Some concerns still exist, however. Wearable

sensors still need to consider light absorption issues

(e.g., skin tone) that may reduce signal fidelity. Also,

with the ease of use, there is the possibility of less

optimal placement by poorly trained users. As with

any wearable autonomic sensor, optical noise and

blood perfusion can introduce statistical noise into

measurement as well.

5 CONCLUSIONS

Immersive experiences are high valued but until

recently, could only be measured through unreliable

retrospective self-reports. Our development of a

wearable OTS sensor, paired with sophisticated

algorithms to process data and a simple-to-use

interface have produced the first passive real-time

immersion sensor. Our research team will be

exploring the many ways such a sensor can be used,

from marketing and messaging, to employee

motivation, to health and welfare in order to create

more engaging and valuable experiences for

consumers.

Real 2018 - Special Session on Assessing Human Cognitive State in Real-World Environments

306

REFERENCES

Barraza, J.A., Alexander, V., Beavin, L.E., Terris, E.T. and

Zak, P.J., 2015. The heart of the story: Peripheral

physiology during narrative exposure predicts

charitable giving. Biological psychology, 105, pp.138-

143.

Barraza, J.A. and Zak, P.J., 2009. Empathy toward

strangers triggers oxytocin release and subsequent

generosity. Annals of the New York Academy of

Sciences, 1167(1), pp.182-189.

Busselle, R. and Bilandzic, H., 2009. Measuring narrative

engagement. Media Psychology, 12(4), pp.321-347.

Csikszentmihalyi, M., 1996. Flow and the psychology of

discovery and invention. New York: Harper Collins.

Gerrig, R.J., 1993. Experiencing narrative worlds: On the

psychological activities of reading. Yale University

Press.

Green, M.C. and Brock, T.C., 2000. The role of

transportation in the persuasiveness of public

narratives. Journal of personality and social

psychology, 79(5), p.701.

Jackson, S.A. and Eklund, R.C., 2002. Assessing flow in

physical activity: The flow state scale–2 and

dispositional flow scale–2. Journal of Sport and

Exercise Psychology, 24(2), pp.133-150.

Lin, P.Y., Grewal, N.S., Morin, C., Johnson, W.D. and Zak,

P.J., 2013. Oxytocin increases the influence of public

service advertisements. PloS one, 8(2), p.e56934.

Measuring Immersion in Experiences with Biosensors - Preparation for International Joint Conference on Biomedical Engineering Systems

and Technologies

307