Smartphone-based Device for Checking Mental Status in Real Time
Mayumi Oyama-Higa
1, 2
, Wenbiao Wang
, Shigeo Kaizu
1, 2
, Terufumi Futaba
and Taira Suzuki
Chaos Technology Research Laboratory, Otsu, Japan
Kwansei Gakuin University, Nishinomiya, Japan
PricewaterhouseCoopers Aarata, Tokyo, Japan
Ryukoku University, Kyoto, Japan
J. F. Obern University, Tokyo, Japan
Keywords: Smartphone-based Device, Largest Lyapunov Exponent, Real-Time Mental Health Check-up for Highway
Drivers, Application to the Caring for Amyotrophic Lateral Sclerosis Sufferers.
Abstract: In this article, we present a smartphone-based device for checking mental status in real time, which for the
first time enables real-time check-up of mental status with a smartphone. With this device, by measuring
pulse waves, two important mental health indicators can be visualized at the same time: the largest
Lyapunov exponent obtained from non-linear analysis of pulse waves, and the autonomic nerve balance.
Before the development of this device, the measurement of these indicators had already been conducted in
thousands of experiments, and their relationship with individual’s mental status had been intensively studied
in recent years. This device enables users to conduct the measurement and capture the mental status
dynamically, without the limitation of place and time. It has the potential application in preventing accidents
due to failure of emotional management. The device is convenient to use and cost-effective.
Problems related to mental disorder, or simply
mental changes, can lead to serious consequences. In
order to measure and analyse individual’s mental
status for early detection of potential mentally
related problems, we have performed a number of
studies in recent years, whose general description
can be found in Oyama’s 2012 book. It has been
discovered that chaotic fluctuations obtained from
the fingertip pulse waves contain information of the
central nervous system. In particular, the largest
Lyapunov exponent (LLE), which quantifies the
variation of the attractor trajectory, can serve as an
indicator of mental immunity. Besides, we have
developed the measuring device in order that a
measurement of fingertip pulse waves can be
conducted anywhere at any time.
For the results of our recent studies, specifically,
the method of measuring and analysing fingertip
pulse waves have been applied to various research
subjects concerning detection of mental changes
(Oyama-Higa et al., 2008; Wang et al., 2012) and
check-up for mental and cognitive disorders, such as
dementia (Oyama-Higa and Miao, 2006; Oyama-
Higa et al., 2008; Pham et al., 2015) and depression
(Oyama-Higa et al., 2008; Hu et al., 2011; Pham et
al., 2013).
This paper presents our improvement in the
measuring device: We have made the device more
convenient by connecting the pulse wave sensor to a
smartphone. Moreover, we calculate and display
values of autonomic nerve balance (ANB) at the
same time, which indicates whether sympathetic
nerve or parasympathetic nerve predominates. Thus,
we have made it possible to examine minute changes
in mental condition by graphic display of LLE and
ANB. Although this device still needs improving in
some aspects, it is so far the only existing device that
makes such mental check-up possible and
convenient with a smartphone.
We introduce the two key indexes calculated and
displayed by our measuring device: LLE and ANB.
In addition, an index of vascular age will also be
Oyama-Higa, M., Wang, W., Kaizu, S., Futaba, T. and Suzuki, T.
Smartphone-based Device for Checking Mental Status in Real Time.
DOI: 10.5220/0005655401370142
In Proceedings of the 9th International Joint Conference on Biomedical Engineering Systems and Technologies (BIOSTEC 2016) - Volume 4: BIOSIGNALS, pages 137-142
ISBN: 978-989-758-170-0
2016 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
2.1 Largest Lyapunov Exponent (LLE)
Time series data with deterministic chaos can be
constituted by fingertip plethysmograms (Tsuda,
Tahara and Iwanaga, 1992). Let
x (i), i = 1, 2, … (1)
denote the time series data. Using the method of
delays, the phase space is reconstructed with vectors
represented as
X(i) = ( x(i), x(i-τ), …, x(i-(d-1)τ) )
= { x
(i) }
k=1, …, d
where τ is a constant delay, d is the embedding
dimension and x
(i) is defined as
(i) = x(i-(k-1)τ), k = 1, …, d. (3)
In our study where the time series are recorded from
fingertip pulse waves, studies (Sano and Sawada,
1985; Sumida et al., 2000) have shown that the
optimal choices for the constant delay and the
embedding dimension are
τ = 50 ms (4)
d = 4. (5)
The largest Lyapunov exponent (LLE) is a measure
of complexity that reflects the divergence and
instability of the attractor trajectory. Let X(t) evolve
with time starting with an initial trajectory X(0).
Then, LLE can be calculated as
where the separation of the trajectories is
represented by
and the initial separation is represented by
in the phase space. In the measuring device in our
previous studies as mentioned above, the method
proposed by Rosenstein et al. in 1993 is applied for
estimating the LLE. For convenience, the value of
LLE is normalized to a range of 0-10 in the display
of our device.
Figure 1 shows the plethysmogram and attractor
obtained from the measurements, and LLE obtained.
Our previous studies have shown that the values of
LLE of a mentally healthy individual fluctuate
within a reasonable scope (from 2-7, centred at 5).
Figure 1: Plethysmogram (top), attractor (right) and LLE
When LLE is abnormally high, the mental
immunity of the individual is so strong that he or she
is likely to go to extremes: such individual can be
easily irritated and take unexpected actions. On the
other hand, when it is abnormally low, the mental
immunity is so weak that the individual is prone to
mental illnesses. In other words, a high LLE
indicates a mental status of adapting to the external
environment (we simply called it “external
adaptation” in some of our previous articles), while a
low LLE indicates a status of “internal focusing”.
2.2 Autonomic Nerve Balance (ANB)
Spectral analysis of heart rate variability can
evaluate the activity of the autonomic nervous
system. We consider the high frequency (HF, 0.15-
0.40 Hz) component which represents
parasympathetic nerve activity, and the low
frequency (LF, 0.04-0.15 Hz) component which is
an index of sympathetic nerve activity. In our study,
autonomic nerve balance (ANB) is defined as a
normalized index ranging from 0 to 10 as follows.
ANB = 10 B / 3.5, (9)
B = ln (LF) / ln (HF). (10)
From the ranges of LF and HF, we can clearly
observe that B ranges from 1, when both LF and HF
take the value 0.15, to approximately 3.5, when LF
is at the minimum while HF is at the maximum.
Therefore, ANB goes from approximately 2.86 to 10.
For convenience, in our current study, we still apply
a 0-10 valued graph to display the result of ANB.
ANB < 5 indicates predominance of parasympathetic
nerve while ANB > 5 indicates sympathetic
The computation method has also been
embedded in the measuring device that has been
used so far in our studies, and it has been registered
as a U.S. patent (Higa, 2011).
BIOSIGNALS 2016 - 9th International Conference on Bio-inspired Systems and Signal Processing
2.3 Vascular Age
In addition to LLE and ANB, a normalized index of
vascular age will also be calculated and displayed.
This index is defined as
2.8571 100
where N is the physical balance of the blood vessel,
computed using second derivative of
photoplethysmogram (SDPTG) (Sano et al., 1985).
N is comparable to the actual age: a high N indicates
unbalanced function of the vessel and vulnerability
to sclerosis, while a low N indicates plasticity of the
vessel. Note that σ is monotonically increasing with
respect to N, ranging approximately from 0 to 10 as
N goes from 0 to 100.
In this article concerning only LLE and ANB
will be discussed in details, so we omit further
explanation of the vascular age.
3.1 General Description
The software that runs on Android phones is named
“Lady Alys”. It has two main parts: the engine part,
in charge of the computation process, and the driver
portion, for connecting the sensor. It can also be
connected to the Internet to save and deliver the
measured data. The main feature, as stated above, is
the real-time display of LLE and ANB during the
pulse wave measurement, which enables the users
(subjects) to check their mental status easily.
The sensor determines the haemoglobin levels in
the finger capillary vessels using infrared
spectroscopy (an 840 nm light-emitting diode is
equipped), and performs digital transformation that
converts 200 Hz to 12 bits.
Values of LLE calculated from the pulse wave,
for clarity, are displayed in 10 colours according to
the values, from red to blue in colour depth. The
closer the colour is to red, the higher the chaotic
fluctuation, and thus the external adaptability, is
reflected. On the contrary, a colour close to blue
represents a low fluctuation and in this case the user
is internally focused
Besides fingertips, pulse waves from earlobes
can also be used as inputs. Since the sensor is
portable, even long-time measurement will not cause
much inconvenience. Furthermore, equipped with
devices such as Bluetooth, it is also possible to take
the data wirelessly.
Figure 2: Smartphone and sensor.
The acquired data can be saved in the database
on the Internet. These data can be further analysed
with advanced analytical software “Dr. Lyspect”
(computer-based). “Dr. Lyspect” is an updated
version of “Lyspect” that we developed (described
in details in Oyama-Higa et al., 2012), which has
been used as a measuring device in our recent
studies cited in the Introduction section.
3.2 Main Screens
In this section, we introduce the main screens of the
smartphone-based device. Figures 3 and 4 show the
starting screen and the pulse wave display
Figure 3: Starting screen.
Fluctuations in LLE due to emotional changes
are displayed in colour: the higher LLE, the closer to
red, while the lower the LLE, the closer to blue.
During the measurement, after the trial time period
of the first 17 seconds, values of LLE are recorded
and displayed in colour every second. It turns red
when the LLE is greater than 5, and darkens as the
value increases. On the other hand, when the LLE is
less than 5, the screen turns blue and grows lighter
Smartphone-based Device for Checking Mental Status in Real Time
as the value decreases.
Figure 4: Pulse wave screen.
Figure 5: Coloured display of LLE.
Figure 6: Time-series display of LLE (in red) and ANB (in
Values of LLE and ANB are also displayed in
time series. However, for ANB, the trial time period
at the beginning of the measurement requires 60
Moreover, LLE and ANB are displayed in
Cartesian coordinates in real time, which visualizes
the correlation of the two indexes.
At the end of the measurement, three semi-
circular graphs are displayed, showing the results of
the average values of LLE, ANB and vascular age
respectively. As to Dr. Lyspect, these three graphs
will also be displayed, among other detailed results,
on the results window.
Figure 7: Display of the Cartesian coordinate plane (ANB:
X-axis, LLE: Y-axis).
Figure 8: Semi-circular graphs of Lady Alys (top; from
top to bottom: LLE, ANB and vascular age) and Dr.
Lyspect (bottom; from left to right: LLE, vascular age and
BIOSIGNALS 2016 - 9th International Conference on Bio-inspired Systems and Signal Processing
There are various applications of the device. We
illustrate two typical ones.
4.1 Real-time Check-up of Mental
Status for Highway Drivers
One is the possible application for the enhancement
of driver safety.
We collected the real-time LLE data of a driver
who was driving on a highway (Figure 9).
Figure 9: Driving experiment on a highway.
We observe that his LLE suddenly increased at
several time points. These time points are actually
when the driver was driving past another car, when
he was passing a car that was stuck in the way due to
accident, and when he was checking a monitoring
camera. All these behaviours require carefulness and
concentration that make the driver tense up. In
Section 2.1, we have explained that a high value of
LLE indicates strong mental immunity, which is
inevitable in adapting oneself to the external
A too high LLE represents the status of excessive
stress that may give burden to the driver. On the
other hand, however, if his LLE keeps low under
such circumstances, the driver will be prone to
operation error.
Therefore, self-adjustment is necessary in both
extreme cases. “Lady Alys” can facilitate such self-
adjustment in that it makes the real-time mental
index visible to the driver who is driving. From this
viewpoint, this device can help enhance traffic
4.2 Caring for Amyotrophic Lateral
Sclerosis Sufferers
The other is the potential application in the caring
for the sick.
We conducted an experiment on the
communication between a male end-stage
Amyotrophic Lateral Sclerosis (ALS) sufferer and
his wife. The patient’s pulse waves were measured
when his wife was talking to him for 16 minutes.
The comprehensive result is obtained with “Dr.
Lyspect” (Figure 10) (Undoubtedly, “Lady Alys”
would be enough for the observation of time series
of LLE).
Figure 10: Conversation to end-stage ALS patients.
With end-age ALS, the patient was bedridden
without any ability to initiate and control any
voluntary movement, so he was not able to make
any physical reaction to wife’s talk.
However, from the graph of time series of LLE
(bottom of Figure 10), we observe significant mental
reaction of the patient. The LLE rose to the first
peak when the wife was talking about their daughter,
and then the second peak appeared when her topic
moved to their parents. Since a high value of LLE
indicates the mental status of external adaptation,
this result reveals that the patient was actually
listening to his wife attentively when she was talking
about his concern.
The device can not only enable us to observe the
emotional reaction of the patient, but also gratify the
carer, namely, the wife in this experiment who
realized that her husband did make response to her
In addition to ALS, we also conducted similar
experiments with encephalopathy sufferers, and
obtained the same conclusion.
In such a way, when physical communication is
not possible, the simple measurement with “Lady
Alys” can be of help for both carers and carees.
With this device, measurement with Android tablet
or mobile phone can be easily conducted,
Smartphone-based Device for Checking Mental Status in Real Time
irrespective of the limitation of place and time. The
“Lady Alys” can help the users observe their real-
time changes in mental status. The display with
intuitive colours makes the values of LLE more
visible. By conducting the measurement during
various daily activities, the users are able to perform
real-time mental check-up and then self-control
according to his or her mental status.
The measurement is also possible to involve two
or more subjects, in order to observe the reactions in
their relationships, especially the carer-caree
In addition, admittedly, there are several
drawbacks with the system. For example, instead of
showing changes in the screen colour, an acoustic
alarm would be more applicable, as it can make the
driver much easier to recognize the alarm without
sporadic attention to the smartphone screen. Such
improvement is under consideration and will serve
as a subject of our future study. Further development
of wireless devices and the improvement of sensors
are also in progress.
We would like to express our sincere thanks to Mrs.
Mitsuko Tanabiki, Director, and the nursing staff
members of Himeji Himawari Nursery School, for
her cooperation in our experiments.
Higa, M. (2011) Autonomic nervous balance computation
apparatus and method therefor, U.S. Patent US
2011/0313303 A1.
Hu, Y., Wang, W., Suzuki, T. and Oyama-Higa, M. (2011)
‘Characteristic extraction of mental disease patients by
nonlinear analysis of plethysmograms’, AIP Conf.
Proc., vol. 1371, pp. 92-101.
Oyama-Higa, M. and Miao, T. (2006) ‘Discovery and
application of new index for cognitive psychology’,
2006 IEEE Conf. on Systems, Man, and Cybernetics
Proc., vol. 4, pp. 2040–2044.
Oyama-Higa, M., Miao, T., Kaizu, S. and Kojima, J.
(2012) ‘Mental health self-check system using
“Lyspect” ’, Proc. of the Sixth International
Symposium on e-Health Services and Technologies,
Sixth International Symposium on e-Health Services
and Technologies (EHST 2012), Geneva, pp.9-18.
Oyama-Higa, M., Miao, T., Tsujino, J. and Imanishi, A.
(2008) ‘Possibility of mental health self-checks using
divergence of pulse waves’, Proc. of the First
International Conference on Biomedical Electronics
and Devices, BIOSIGNALS 2008, Funchal, pp. 361-
Oyama, M. (2012) Psychology of mental flexibility
(English edition, Kindle), Seattle: Amazon Services
International, Inc.
Pham, T.D., Oyama-Higa, M., Truong, C.T., Okamoto, K.,
Futaba, T., Kanemoto, S., Sugiyama, M. and Lampe, L.
(2015) ‘Computerized assessment of communication
for cognitive stimulation for people with cognitive
decline using spectral-distortion measures and
phylogenetic inference’, PLos One, vol. 10(3),
Pham, T.D., Thang, T.C., Oyama-Higa, M., Nguyen, H.X.,
Saji, H. and Sugiyama, M. (2013) ‘Chaos and
nonlinear time-series analysis of finger pulse waves
for depression detection’, Proc. of the International
Conference on Bio-inspired Systems and Signal
Processing, BIOSIGNALS 2013, Barcelona, pp. 298-
Rosenstein, M.T., Collinsa, J.J. and De Luca, C.J. (1993)
‘A practical method for calculating largest Lyapunov
exponents from small data sets’, Physica D: Nonlinear
Phenomena, vol. 65, pp. 117-134.
Sano, M. and Sawada, Y. (1985) ‘Measurement of the
Lyapunov spectrum from a chaotic time series’, Phys.
Rev. Lett., vol. 55, pp. 1082.
Sano, Y., Kataoka, Y., Ikuyama, T., Wada, M., Imano, H.,
Kawamura, K., Watanabe, T., Nishida, A. and Osanai,
H. (1985) ‘Evaluation of peripheral blood circulation
to predict the possible impairment of circulatory
organs by quadratic differential plethysmography and
its application (in Japanese)’, Science of Labour, vol.
61, no. 3, pp. 129-143.
Sumida, T., Arimitu, Y., Tahara, T. and Iwanaga H.
(2000) ‘Mental conditions reflected by the chaos of
pulsation in capillary vessels’, International Journal of
Bifurcation and Chaos, vol. 10, pp. 2245-2255.
Tsuda, I., Tahara T. and Iwanaga. I. (1992) ‘Chaotic
pulsation in capillary vessels and its dependence on
mental and physical conditions’, International Journal
of Bifurcation and Chaos, vol. 2, pp. 313-324.
Wang, W., Hu, Y., Oyama-Higa, M., Suzuki, T., Miao, T.
and Kojima, J. (2012) ‘Analysis of
electroencephalogram and pulse waves during music
listening’, Proc. of the Sixth International Symposium
on e-Health Services and Technologies, Sixth
International Symposium on e-Health Services and
Technologies (EHST 2012), Geneva, pp. 31-35.
BIOSIGNALS 2016 - 9th International Conference on Bio-inspired Systems and Signal Processing