Feeling Hungry: Association of Dietary Patterns
with Food Choices using Scene Perception
Shoma Berkemeyer
a,
and Julius Sch
¨
oning
b,
Osnabr
¨
uck University of Applied Sciences, Osnabr
¨
uck, Germany
Keywords:
Scene Perception, Dietary Patterns, Food Choices, Nutritional Behavior, Gaze Analytics, Multimodal Data
Analysis, Visualization, Nutritional Patterns.
Abstract:
Studies on nutrition have historically concentrated on food-shortages and over-nutrition. The physiological
states of feeling hungry or being satiated and its dynamics in food choices, dietary patterns, and nutritional
behavior, have not been the focus of many studies. Currently, visual analytic using easy-to-use tooling offers
applicability in a wide-range of disciplines. In this interdisciplinary pilot-study we tested a novel visual an-
alytic software to assess dietary patterns and food choices for greater understanding of nutritional behavior
when hungry and when satiated. We developed software toolchain and tested the hypotheses that there is no
difference between visual search patterns of dishes 1) when hungry and when satiated and 2) in being vegetar-
ian and non-vegetarian. Results indicate that food choices can be deviant from dietary patterns but correlate
slightly with dish-gazing. Further, scene perception probably could vary between being hungry and satiated.
Understanding the complicated relationship between scene perception and nutritional behavioral patterns and
scaling up this pilot-study to a full-study using our introduced software approaches is indispensable.
1 INTRODUCTION
Hunger and forms of under-nutrition has been one
of the nutritional research focus, investigating food
and/or nutrient shortage and their impact on diseases
and survival (Rubin, 2018; Berkemeyer, 2012). Sci-
entific inquiry into over-nutrition has been extremely
well-documented, with studies on obesity, metabolic
syndrome, diabetes, cancers, and other diseases of
civilization (Berkemeyer, 2009). Little though is
known about food choice and food perception in the
state of being hungry or satiated, a complex of phys-
iological and behavioral processes. The attitude to-
wards food, including food culture, and individual
emotional relation to, perception of and preference
of food, would co-determine food choice. Short-term
food choices and long-term dietary patterns affect nu-
tritional status, all which have bearing on etiology
and therapy of nutrition-based diseases. (Breer et al.,
2017; Payne et al., 2010; MacCormack and Lindquist,
2019).
The research field of visual analytic provides ap-
plications capturing gaze-data and visualizing the
a
https://orcid.org/0000-0002-2125-7468
b
https://orcid.org/0000-0003-4921-5179
Both author contribute equally
gazes as perception of the scene. In the field of nutri-
tion the use of visual analytic is in its nascent stages.
Thus, we combined these disciplines in this paper,
opening up attractive opportunities to observe the vi-
sual search pattern on food, thereby finding applica-
tion in nutritional research.
Next to the development of an easy-to-use
toolchain for conducting and analyzing studies on
scene perception of dishes for nutritional research, a
pilot study was conducted. Our pilot study objective
was to investigate scene perception with food gaze on
dishes when hungry (before lunch) and the food gaze
at satiety (after lunch), taking into account the prior
knowledge of dietary patterns. The underlying as-
sumption of our pilot-study was that involuntary gaze
movements on food would be an index for sponta-
neous nutritional choices, with stated priors of nutri-
tional patterns to augment our current understanding
of nutritional behavior.
In the following sections we report our pilot study
goals, the tooling developed for this pilot study, and
state the hypotheses. In Methods we describe the set-
ting, participants, development of the visual analytic
tooling for assessing gaze-data to index scene percep-
tion and the data-analysis conducted. This is followed
up by presentation of the Results and Discussion.
188
Berkemeyer, S. and Schöning, J.
Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception.
DOI: 10.5220/0010146101880195
In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2020), pages 188-195
ISBN: 978-989-758-480-0
Copyright
c
2020 by SCITEPRESS Science and Technology Publications, Lda. All rights reserved
2 GOALS FOR PILOT STUDY
In an interdisciplinary team comprising of nutritional,
computer, and behavioral scientists, two goals were
identified. Firstly, creating software for conduct-
ing surveys with eye-tracking at the site of food-
consumption. Second goal was to test the analysis
toolchain for first insights on before and after lunch
scene perception. We thus set up our pilot study to-
wards these two goals and for testing our developed
software under realistic conditions.
2.1 Tooling
This application of scene perception in nutritional
studies required data-collection directly at food-
eating locations, e.g., canteens or food-courts. For
our study, the university canteen, Mensa, setting was
chosen due to nearness and easy feasibility of pilot-
testing. We assumed that people would have lim-
ited time for lunch-break, as stipulated by employ-
ers, classically half-an-hour. Thus, the study’s imple-
mentation goal was identified as a mobile hardware-
software solution that would work without special
equipment, such as chin-rests, and allow running each
experiment at minimum time.
The second challenge, next to data-gathering, was
data-storage for an interdisciplinary team. The con-
cept of using multimedia container formats for stor-
ing survey-data (Sch
¨
oning et al., 2017c,b), would al-
low first exploratory visual data analysis without any
special tool. By improving our existing implementa-
tion, a more meaningful visualization was to be pro-
grammed, cf. Figure 4. The current approach and
the future programming requirement would allow nu-
tritional scientists to explore the data collected with-
out additional requirement of special self-compilation
or costly proprietary software, enabling newer dimen-
sions in nutritional analytics.
2.2 Hypotheses
The prototyped survey tool enabled us to collect
questionnaire-based information and gaze point on
stimuli types such as photos, video footage, and an-
imations.
For this pilot-study, the first null-hypothesis was
defined as 1) that there is no difference between vi-
sual search patterns of dishes when participants were
hungry and when satiated. As the second hypothesis,
we expected 2) no difference between visual search
patterns and a priori stated dietary pattern.
(a) Step I:
answering a
questionnaire (10s)
(b) Step II:
recalibrate the low-cost eye
tracking device (2sec)
(c) Step III:
gazing on stimuli —
different dishes (40s)
(d) Step V: having lunch
Step V: gazing on stimuli
again (40sec)
Figure 1: Steps of the study designed; not illustrated: Step
IV have lunch.
3 METHODS
The study was conducted in January 2020 under the
specifications of the declaration of Helsinki (World
Medical Association, 2013). Informed consent was
obtained prior to study participation. Ethical clear-
ance was invalid for this pilot-study as no unique
person-related, and identifiable data was gathered.
3.1 Setting
The pilot-study was based at a tertiary educational
setting. Study participants included students, staff,
or visitors of the tertiary educational institution. On
the day of survey, four different main dishes and sev-
eral side dishes were on offer at the canteen namely:
Dish 1: organic spaghetti with organic soya bolog-
nese, Dish 2: chicken schnitzel with peach and hol-
landaise sauce, Dish 3: gemstone pumpkin curry,
Dish 4: beef sliced Esterhazy in vegetable sauce, Dish
5: side dishes i.e., potatoes, salad, soup, and noodles.
3.2 Study Design, Variables and Tooling
A cross-sectional study design was used for this ex-
ploratory study. For this early-stage feasibility study,
Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception
189
a total of ten participants were recruited, which gen-
erated a medium volume of gaze-data. The recruit-
ment ensued random selection of people visiting the
canteen on the survey-day, given participant willing-
ness to take part in the study and informed writ-
ten study-consents were available. The inclusion
of ten participant for a pilot feasibility was deter-
mined by the period of lunch-break, of about half an
hour per person, over an entire lunch period of two
hours between 12.00 noon and 14.00 p.m. on the
day of data-collection and the use of a single com-
puter based terminal equipped with visual analytic for
data-gathering. Study participation, even so random
stochastically determined, can not rule out study par-
ticipation bias by those, who were inherently more
interested in nutrition, compared to those, who were
less interested in nutrition. The study was imple-
mented by one study-personnel on the survey day, the
software developed for this pilot study integrated a
self-training sequence for those conducting the study
allowing easy use and applicability.
Dietary patterns included non-vegetarian and veg-
etarian; non-vegetarian was defined as omnivore,
healthy omnivore nutrition (Deutsche Gesellschaft f
¨
ur
Ern
¨
ahrung e. V., 2020), and paleo-diet. Vegetarian
was defined as ovo-lacto-vegetarian, ovo-vegetarian,
lacto-vegetarian and vegan diets; raw-food, whole-
food and flexitarian diets; and pesco-vegetarian diet.
Dietary pattern was assumed to indicate long-term nu-
tritional pattern and behavior. Food choices were de-
fined by canteen menus that were chosen by the study
participants on the survey-day. Food choices were
assumed to indicate short-term nutritional behavior.
There were five menus for choice on the study-day.
Background data were collected on age, gender, and
country of birth.
For the mobile hardware platform, a standard mid-
class laptop with a built-in, detachable low-cost eye-
tracking device was used. Estimating the computing
power required for mobile software solution, it was
determined that a low-cost laptop would not be able
to simultaneously handle multi-threading, necessary
for synchronous visualization of the stimuli, and data-
storage of gaze data. We, hence, used a standard mid-
class laptop for the pilot study, laptop specifications
were Intel i5 CPU equipped with a Tobii EyeX eye
tracker.
As seen in Figure 1, our software
1
covered the
whole acquisition process of data, including human-
machine-interaction, e.g., for the questionnaire and
for calibration of the eye tracking device. We provide
this software as free, open-source software, which is
1
The source code can be downloaded at https://www.
hs-osnabrueck.de/prof-dr-julius-schoening/chira2020
capable of providing a timed graphical user interface
for presenting the stimuli at a defined time and in-
teroperable to different low-cost eye tracking device,
in the programming language of C++. The software
given as open-source for public research is also pro-
vided with an open-source free widget toolkit QT.
A simple calibration routine was implemented for
the use of low-cost eye-tracking device along with the
minimization of time spent for the study by each par-
ticipant. As shown in Figure 1(b), the participant was
therefore asked to fixate on a cross in the middle of the
screen. The computer recognized a fixation over one
second within an offset of 50 pixels, and the software
re-calibrated the gaze tracking.
The study-design required that the stimuli presen-
tation, as illustrated in Figure 1(c) would be flex-
ible showing the day’s dishes available at the can-
teen. The stimuli were not compiled with the soft-
ware. Rather the stimuli, e.g., photos of dishes, were
placed in a specified folder next to the executable
built-in software. In this way, this pilot-study devel-
oped an easy, simple, and flexible solution for inter-
changeable stimuli. By naming convention, the order,
as well as the duration of the stimulus, was set. For
example, the photograph titled “dish1 3000.jpg” was
shown for 3000msec followed by “dish2 1242.jpg”
and so on. To avoid restarting the software repeat-
edly, a guided dialog was provided for the survey in-
structor, allowing back and forth movements with in-
struction repeats. Thus, the software applicability al-
lowed easy, on-site training for people conducting the
study, thereof saving time in study-specific schooling
and training of personnel.
To regain participants after lunch we had a reward
for the study participants built in to the study-design.
Participants received a visualization of their gaze in
the format compatible with standard multimedia play-
ers (Sch
¨
oning et al., 2017a). This is described in detail
in the following section.
3.3 Analysis
For scene perception analysis, the gaze trajectories
on each dish were visualized for our investigation.
Therefore static heat-maps, as shown in Figure 2, for
each shown dish per participant, were generated by a
simple Python script. Next to the five different dishes,
cf. Subsection 3.1, a white screen was presented as
well for zero-error correction calibration. For further
analyses, the differences in the gaze patterns when
hungry and when satiated were visualized.
According to the hypotheses 1), the resulting heat-
maps were composed next to each other for visual in-
spection. In addition to the heat-maps per participant,
CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications
190
(a) dish 4: before having lunch (b) dish 4: after having lunch
(c) dish 5: before having lunch (d) dish 5: after having lunch
Figure 2: Cumulated gaze patters off all participants on stimulus dish 4 and dish 5 , after having lunch the participants explore
the outer areas of the stimulus.
cumulative heat-maps per dish for all participants was
generated, cf. Figure 2. Finally 2), cumulative heat-
maps over vegetarians and non-vegetarians were gen-
erated as well.
We compared the visual search pattern when hun-
gry and when satiated as an assessment of nutritional
behavior and compared it to participant’s food choice
of mid-day meal. Thereby, we visually examined,
thereby testing, the possibility of using visual search
pattern heat-maps for assessing nutritional behavior.
4 RESULTS
The overview of all study participants (cf. Appendix
Table 1) shows that of the ten study participants,
seven were non-vegetarian and three were vegetar-
ians, thereof one vegan, one vegetarian, and one
pescovegetarian. Age-range of the participants was
23-40 years. There were three women and seven men.
Of the seven non-vegetarians, two were women and
five men, and of the three vegetarians one was man
and two were women.
Testing hypothesis 1), Figure 3 suggests that there
was no significant difference between visual search
patterns when hungry and when satiated using in-
dividual gaze-data. Visual inspection indicated that
there could be gaze-deviation in cumulative gaze-
patterns, i.e., when hungry and when satiated. How-
ever, this observation could also be due to, that the
same photos were presented before and after lunch,
which remains a study confounder.
Dish 1 Dish 2 Dish 3 Dish 4 Dish 5 white screen
0
5
10
15
4.14%
7.76%
5.12%
2.48%
9.72%
0.53%
similarity in visual search pattern %
Figure 3: Similarity of visual search pattern when hungry
and when satiated, accumulated over all participants.
Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception
191
Of the seven non-vegetarians, three gazed at
dish 2, and two chose dish 2 (chicken schnitzel with
peach and hollandaise sauce), one chose an undis-
closed dish 5; two gazed at dish 4, and one chose
it and the other chose dish 2. The seventh non-
vegetarian was undecided and chose dish 1 (organic
spaghetti with organic soy bolognase). Of the seven
non-vegetarians, six were decided and one undecided
at study-begin, and of the six only three chose what
they gazed at. Of the three vegetarians, the vegan
participant gazed and chose dish 3 (gemstone curry).
One was undecided and chose an undisclosed dish 5.
The third vegetarian (pescovegetarian) gazed, and
chose dish 2. Of the three vegetarians, two chose dif-
ferent dishes than at onset; one food-choice was dif-
ferent from the dietary pattern, one remained undis-
closed.
In total, only four (one vegetarian and three non-
vegetarian) of the ten participants made food choices
that coincided with what they gazed at. Two were
undecided in food-choice from onset on (one veg-
etarian and one non-vegetarian). Of the remain-
ing four participants (one vegetarian and three non-
vegetarian), three food choices were different from
what was gazed at, yet within the dietary pattern (non-
vegetarian) and in one case food choice was different
from the stated dietary pattern (vegetarian). We pro-
vide above-trend results for our hypothesis 2), which
would require a full-scale study with large sample
size.
5 DISCUSSION
Our pilot study indicates that individual visual search
patterns were similar when hungry and when satiated,
though cumulative visual search patterns suggest that
participants could have gazed on food (central plate)
when hungry and peripheral dish-area when satiated,
which though remains confounded by the presenta-
tion of same before-after pictures. Initial analysis in-
dicates that food choices tended to vary within dietary
patterns, and at times food choices could be different
from dietary patterns. The chance of greater undis-
closed food choices was probably related to higher
dietary pattern deviant choices, indicating nutritional
behavior pattern to be more complex than the stated
long-term dietary pattern or short-term food choice.
Our pilot study further provides that the development
of a low-cost tooling device can be used for nutri-
tional studies with the development of more easy-to-
use software for visual analytic.
Exploring and analyzing the temporal behavior of
the scene perception, the idea of using any multime-
dia player for exploratory analysis (Sch
¨
oning et al.,
2017a,c,b) was adopted and improved in our study. In
this process we developed and improved the use of vi-
sual analytic to its stated purpose Thomas and Cook
(2005), i.e., to derive insight from massive, dynamic,
ambiguous, and often conflicting data, to detect the
expected and discover the unexpected.
We further added transparent heat-map images as
subtitle tracks instead of the use of the pure text-based
universal subtitle format (USF), thereby we increased
the meaning of gaze points per frame without leaking
the ability to be played by the use of standard multi-
media player such as the VLC Player. With our novel
approach, refer Figure 4 (b) with the USF in Figure
4 (a), the information content was increased, because
several gaze sampling points on the pixel were able to
be visualized with colors. Despite these extensions,
our approach still conformed to the multimedia con-
tainer format and provided both instantaneous visual-
ization with multimedia plays and the storage of all
data in raw formats.
Our pilot study indicates that food choices and
dietary pattern in frame of nutritional behavior pat-
tern show variance. Very little is known about the
decision-making process in food choices across the
human life-span. Ogden and Roy-Stanley (2020) re-
ported in children that food decision was automatic
or considered or sanctioned. Greater autonomy in
food decision-making does not automatically imply
healthy food choices (Breer et al., 2017). Further,
sanctioned behavior might also present a viable op-
tion in nutritional advice with diseases (Berkemeyer,
2009). MacCormack and Lindquist (2019) hypothe-
sized that people experience hunger as emotions and
misattribution process (Payne et al., 2010), to demon-
strate that hunger (Rubin, 2018; Berkemeyer, 2012)
shifts affective perceptions in negative but not in neu-
tral or positive contexts. A role for intuitive eating to
promote health has also emerged requiring further in-
vestigation (Rubin, 2018; Keirns and Hawkins, 2019).
A further application of our study results can be in
the practice of how restaurants prepare food. Much of
the processes are pre-defined and much of food prepa-
ration,such as, chopping, slicing, base sauce prepara-
tions are done in advance. Everything has its place
and order in a professional kitchen to facilitate dif-
ferent people accessing same objects with these stan-
dardizations. Use of visual data analytic in scene per-
ception in this practice-fields could help, for exam-
ple, in training staff in simulations or real-life practice
evaluations, which would be additional covariates in
food choices. Food presentation has been shown to
affect food choice (Ogden and Roy-Stanley, 2020).
Our limitations include the number of partici-
CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications
192
(a) universal subtitle format (USF) for visualization (b) transparent image as sub station alpha overlays as
subtitles for visualization
Figure 4: Gaze visualization with standard multimedia player; (a) former approach (Sch
¨
oning et al., 2017a) and (b) our new
approach providing a heatmap in each frame.
pants in our pilot study, which can be addressed in
a full-study. Dichotomous grouping into vegetarians
and non-vegetarians might be inadequate, requiring a
third intermediate-grouping of semi-non-vegetarians
including pescovegetarians and/or flexitarians. Fur-
ther, our before-after study can be improved with
assessment of food cultural and emotional factors,
which co-determine food choices. Acquisition of
healthy food-habits can be improved with reeduca-
tion, incorporating emotional association with food,
which too remains an additional dimension applica-
tion of the current pilot-study requiring future focus.
Finally, even so participants were randomly recruited
on the survey day we can not rule out participation
bias by those, who are interested in nutrition, which
remains a study-limitation. The use of cumulative
heat-maps would require greater optimization, which
can be conducted in future research in study-samples
similar to and diverse from the current pilot samples,
in order to generate long-term valid and reliable re-
sults.
This pilot-study indicates that by understanding
scene perception and gaze-data, a greater understand-
ing of nutritional patterns can be facilitated, includ-
ing the future development of innovative nutritional
therapy and innovative ways of preparing and serving
food. Thus, a small unobtrusive bay leaf at the right
place could probably instinctively help to make better
food decisions, e.g., as a food-nudging intervention.
Our pilot-study shows that such future goals can be
the mainstay of innovative interdisciplinary research
providing new, open, easy-to-use tools and exchange
of different scientific approaches and perspectives.
REFERENCES
Berkemeyer, S. (2009). Acid–base balance and weight
gain: Are there crucial links via protein and organic
acids in understanding obesity? Medical Hypotheses,
73(3):347–356.
Berkemeyer, S. (2012). Starvation versus calorie restric-
tion: Our road to food insecurity or health. Journal of
Nutrition & Food Sciences, (S1).
Breer, N., Gendig, C., and Berkemeyer, S. (2017). The re-
lationship of migration, age, income and dietary pat-
terns with body mass index in a cross-sectional analy-
sis of ebagil-study. In Eighth EUSPR Conference and
Members Meeting.
Deutsche Gesellschaft f
¨
ur Ern
¨
ahrung e. V. (2020).
Vollwertige ern
¨
ahrung. https://www.dge.de/
ernaehrungspraxis/vollwertige-ernaehrung/ (2020-
01-30).
Keirns, N. G. and Hawkins, M. A. W. (2019). Intuitive eat-
ing, objective weight status and physical indicators of
health. Obesity Science & Practice, 5(5):408–415.
MacCormack, J. K. and Lindquist, K. A. (2019). Feeling
hangry? when hunger is conceptualized as emotion.
Emotion, 19(2):301–319.
Ogden, J. and Roy-Stanley, C. (2020). How do children
make food choices? using a think-aloud method to ex-
plore the role of internal and external factors on eating
behaviour. Appetite, 147:104551.
Payne, B. K., Hall, D. L., Cameron, C. D., and Bishara,
A. J. (2010). A process model of affect misattri-
bution. Personality and Social Psychology Bulletin,
36(10):1397–1408.
Rubin, O. (2018). The precarious state of famine research.
The Journal of Development Studies, 55(8):1633–
1653.
Sch
¨
oning, J., Faion, P., Heidemann, G., and Krumnack, U.
(2017a). Providing video annotations in multimedia
containers for visualization and research.
Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception
193
Sch
¨
oning, J., Gert, A., Ac¸ik, A., Kietzmann, T., Heide-
mann, G., and K
¨
onig, P. (2017b). Exploratory mul-
timodal data analysis with standard multimedia player
— multimedia containers: a feasible solution to make
multimodal research data accessible to the broad au-
dience. In Proceedings of the 12th Joint Conference
on Computer Vision, Imagingand Computer Graphics
Theory and Applications (VISAPP), pages 272–279.
SCITEPRESS.
Sch
¨
oning, J., Gundler, C., Heidemann, G., K
¨
onig, P., and
Krumnack, U. (2017c). Visual analytics of gaze data
with standard multimedia player. Journal of Eye
Movement Research, pages 1–14.
Thomas, J. J. and Cook, K. A., editors (2005). Illuminating
the Path: The Research and Development Agenda for
Visual Analytics. IEEE Computer Soc., Los Alamitos,
Calif.
World Medical Association (2013). World medical associ-
ation declaration of helsinki. JAMA, 310(20):2191.
CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications
194
APPENDIX
Table 1: Overview of all study participants of the pilot-study.
ID Age (years) Gender Brith Country Mensa-Type Nutritional Pattern Food Gaze (when
hungry)
Food choice (satiated)
P280413 31 Woman Germany Staff NP-NV Beef Sliced Esterhazy
in Vegetable Sauce
Chicken schnitzel
with peach and
hollandaise sauce
P281612 30 Man Netherlands Staff NP-NNV (Vegan) Gemstone Pumpkin
Curry
Gemstone Pumpkin
Curry
P283111 23 Man Germany Student NP-NV Chicken schnitzel
with peach and
hollandaise sauce
A different dish
P283312 38 Man Germany Staff NP-NV Chicken schnitzel
with peach and
hollandaise sauce
Chicken schnitzel
with peach and
hollandaise sauce
P283811 25 Man Germany Student NP-NV Undecided Organic spaghetti
with organic soya
bolognese
P284111 31 Woman Germany Student NP-NNV (Vegetar-
ian)
Undecided A different dish
P284811 32 Woman Germany Student NP-NV Chicken schnitzel
with peach and
hollandaise sauce
Chicken schnitzel
with peach and
hollandaise sauce
P285111 32 Man Germany Student NP-NNV (Vegetarian
with fish)
Organic spaghetti
with organic soy
bolognese
Organic spaghetti
with organic soy
bolognese
P285611 40 Man Germany Staff NP-NV Chicken schnitzel
with peach and
hollandaise sauce
Chicken schnitzel
with peach and
hollandaise sauce
P285710 34 Man Germany Staff NP-NV Beef Sliced Esterhazy
in Vegetable Sauce
Beef Sliced Esterhazy
in Vegetable Sauce
Feeling Hungry: Association of Dietary Patterns with Food Choices using Scene Perception
195