The Missing Tip: Lack of Micro-Movements Impairs Navigation Realism
in Artificial Social Agents
Jacob Sharp
a
and Ulysses Bernardet
b
College of Engineering and Physical Sciences, Aston University, Aston St, Birmingham, U.K.
Keywords:
Artificial Social Agents, Navigation, Micro-Movements, Virtual Environments, Path Planning, Realism,
Immersion, Standardisation.
Abstract:
Navigation is critical to an intelligent social agent’s ability to interact with the world and any other agent,
virtual or otherwise. In order to create a truly realistic artificial social agent, unconscious human micro-
movements need to be simulated. We see this as an important goal for the research area. Examples of these
micro-movements include orienting while walking and back-stepping, strafing with attention focused else-
where, and micro-orientations during locomotion. We postulate that there is a gap in research around these
micro-movements within the field of navigation that we hope to contribute to filling. Most research in this field
is focused on the understandably important pathfinding aspect of navigation; moving between two spatial lo-
cations. There is little to no research being done on micro-movements and making a truly realistic navigation
system for artificial social agents. Moreover, there exists no canonical way of describing these movements
and ”micro-movements” that are so characteristic for human spatial behaviour. Here we propose a set of stan-
dardised descriptors of movement configurations, that will be able to be used as building blocks for spatial
behaviour experimentation, and as the basis for behaviour generation models. We see this as an important tool
in the creation of navigation systems that are able to more readily include these kinds of behaviours, with hope
that the aforementioned configurations will improve development of realistic movement systems.
1 INTRODUCTION: HIERARCHY
OF NAVIGATIONAL REALISM
The navigational behaviour of an artificial social
agent plays a key role in the perception of realism hu-
mans feel when interacting with it. However, much
current research into this area is focused primarily
on allowing intelligent artificial social agents to navi-
gate to their intended destination realistically, or to in-
crease the visual fidelity of the artificial social agents
themselves. This has created a gap in the litera-
ture with little to no research done into the micro-
movements and orientation that we believe give an
artificial social agent a much higher sense of real-
ism and therefore foster a higher sense of immer-
sion in the user; the subjective feeling of being in an-
other world (Bartle, 2004). This is important in vir-
tual experiences as it has been shown that a higher
sense of immersion is key in creating effective vir-
tual worlds. With virtual experiences being used
a
https://orcid.org/0000-0002-5577-3962
b
https://orcid.org/0000-0003-4659-3035
more often for important experiences such as train-
ing (Lele, 2013), (Merchant et al., 2014), the onus
is there to create immersive worlds in which the ex-
perience that humans gain is real and worthwhile.
(Latoschik et al., 2017) confirmed this with their work
on avatar realism and it’s effect on various measures,
finding that humaneness and attractiveness were both
increased with human-like avatars over wooden man-
nequins, though eeriness was also increased. These
findings were supported by (P
¨
utten et al., 2009) look-
ing into whether social presence can be elicited for
virtual avatars, with results showing that they can,
with the more behavioural realism the artificial social
agent shows increasing this measure. Other than be-
havioural realism, movement is another key area of
the realism of an artificial social agent, though it can
be seen as one of the most important, (Pedica and
Vilhj
´
almsson, 2008) found from their work on vir-
tual avatars for online chat rooms that social percep-
tion and reactive manoeuvring in the form of group-
ing around o-spaces (Burgoon and Kendon, 1992)
seemed to give the avatars a heightened level of re-
alism showing that the orientation and positioning
380
Sharp, J. and Bernardet, U.
The Missing Tip: Lack of Micro-Movements Impairs Navigation Realism in Artificial Social Agents.
DOI: 10.5220/0011798300003393
In Proceedings of the 15th International Conference on Agents and Artificial Intelligence (ICAART 2023) - Volume 1, pages 380-386
ISBN: 978-989-758-623-1; ISSN: 2184-433X
Copyright
c
2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)
of people or agents can be just as important. This
need for realistic movement can be seen again in the
propagation of motion capture technology in many
forms of media. This high cost, high time invest-
ment form of animation, uses recorded humans to cre-
ate realistic movements for virtual characters. The
capture of the micro-movements and unconscious ac-
tions is what make motion capture so powerful, if
there were a way to simulate these micro-movements
without the heavy cost of motion capture, the real-
ism and believability of artificial social agents could
increase without a heavy investment. This in turn
would increase the overall immersion of the expe-
rience (Pedica and Vilhj
´
almsson, 2008). However
the main focus of pathfinding, determining the path
the agent takes from (a) to (b) is understandable, as
for these micro-movements and navigations to occur,
there must be a movement scenario for them to occur
within. Thus, the construction of a believable human
movement model can be viewed as a sort of pyramid
(Figure 1). Inspired in part by A. Maslow’s hierar-
chy of human needs (Maslow, 1943), with pathfind-
ing laying the foundational systems for higher level
behavioural systems to utilise. We see the micro-
movement behaviours at the ”tip” of the pyramid that
are described in this paper as being supremely impor-
tant in the development of realistic human artificial
social agent, and that these behaviours, small move-
ments, navigation and orientation all have a very im-
portant part to play in creating a realistic, immersive
artificial social agent that evidences some facsimile
of life (Pedica and Vilhj
´
almsson, 2008). For humans,
these things happen subconsciously without thought
(Burgoon and Kendon, 1992) but for artificial social
agents, these need to be firstly measured from human
participants, canonised and studied and then designed
in such a way for an AI to perform these small actions.
The current article addresses the scientific commu-
nity, specifically around artificial social agents. It’s
goal is to increase the awareness of the shortcomings
in the area of locomotion for artificial social agents,
and realism in virtual worlds. The research areas
are introduced first, with Pathfinding, Social Robotics
and Micro-Movements explored in more detail. Ex-
amples of current locomotion data will be presented
to illustrate the issue with current locomotion data
collection, before moving onto the proposal of this
paper; the presentation of the formalisation of these
micro-movements we unconsciously make during lo-
comotion.
Figure 1: The requirements of a realistic human artificial
social agent as a hierarchical pyramid; Pathfinding describ-
ing the routing method that generates the path for an arti-
ficial social agent to take. Locomotion is the actual move-
ment the agent employs to displace itself to the intended
direction. Micro-Movements are the small movements and
orientations humans use during navigation such as rotating
while stepping, that when lacking cause a lowering of real-
ism of the agent.
1.1 Pathfinding - Getting from (a) to (b)
Pathfinding as a subset of navigation has a large body
of research due to it’s necessity regarding the control
models of real-world robots, however this robotic fo-
cus has led to most systems having very low mea-
sures of realism when applied to an artificial social
agent. When investigating multi agent systems (Wang
et al., 2013) focused on creating a dynamic system
for controlling multiple agents in an immersive en-
vironment, with special attention on path planning
for avoidance. The purpose of this research was to
increase the effectiveness of training simulations for
evacuation of multiple agents and other real-time ap-
plications of crowd simulation, however misses the
importance of increasing the realism of the individ-
ual movement of each agent. (Zhukov and Iones,
2000)’s work on navigational control for intelligent
agents instead focused on the creation of navigational
maps, with the goal of decreasing computational load
and increasing the complexity of navigation for artifi-
cial social agents, but still only designed these nav-
igational maps to translate an agent from point (a)
to point (b) with no simulation of higher movement
functions. With (Raees and Ullah, 2021) also focus-
ing on pathfinding without the complexity of micro-
movements we discuss in this paper. (Olcay et al.,
2020) Took the extremely interesting angle of de-
signing a simultaneous, collision-free motion plan-
ning system for fully autonomous robots, allowing
groups of fully autonomous robots to motion-plan,
even within an environment with moving obstacles
The Missing Tip: Lack of Micro-Movements Impairs Navigation Realism in Artificial Social Agents
381
or poor sensor range. This kind of SLAM (simulta-
neous localisation and mapping) control system will
be hugely important as a facet of an autonomous true
virtual human. However for the goal of creating com-
puter controlled artificial social agents which move in
a realistic way, autonomy is not a requirement. Re-
inforcing the research into navigation within an envi-
ronment with movable obstacles, (Djerroud and Ali-
Ch
´
erif, 2021) created ”VICA” a vicarious cognitive
architecture for autonomous robots, though this re-
search differs as it follows the ”theory of mind” in
saying that a form of ”vicariance” is important for
a robot’s strategy to interact with the outside word.
This architecture employs a multi agent system to al-
low the robot a representation of how it’s interactions
would cause the outside world to react. (Sutera et al.,
2021) have pushed the field of marrying navigation
with learning even further, by using ultra-wide band
technology for precise tracking combined with a low-
cost point-to-point local planner learnt with deep re-
inforcement learning (the notion of intelligent agents
taking actions to maximise a cumulative reward, see
(Akalin and Loutfi, 2021)), they are able to path-find
robustly in noisy and complex environments. This
is something important for robots in real-world envi-
ronments, but unnecessary for artificial social agents,
who by virtue of their medium already have access to
all data on their environment. These approaches once
again however, all miss a vital component of realism
in these artificial social agents, the micro behaviours
that we propose need formalisation.
1.2 Approaches in the Social Robotics
Domain
Robotics has a wealth of valuable data on naviga-
tion due to the field of Human-robot interaction (HRI)
being one of the largest in artificial social agents,
with increasing amounts of research being done in the
area due to the uptake of complex and use-specific
robots that exist in the world today (Goodrich and
Schultz, 2007). One may think this domain has re-
search into micro-movements due to it’s size, how-
ever even though the field of HRI is so large, the
most related research in this field is focused primar-
ily on interactive social robots. For example (Ghaz-
ali et al., 2019) Looked at the effect of social cues
in robots on user’s psychological reactance, liking
among other psychological measures, however they
do not investigate navigational realism as a social
cue, instead focusing only head mimicry and social
praise timing. (Liu et al., 2018) investigated human-
robot behaviour in a shopkeeper scenario and in-
cluded locomotion in the multi-model behaviour of
the robot. Finding that cross-validation on the train-
ing data showed higher social appropriateness of the
robot’s behaviours. Though once again this research
was conducted on a wheeled form of locomotion, and
realistic human movement was not the intention of
the research. Apart from investigations into social
robots such as these, the bulk of research is directed
around navigation systems that create efficient (a) to
(b) routes for robotic agents (Olcay et al., 2020), (Li
et al., 2019). Despite the field of HRI being so large,
it is still missing research into the micro-movements
that we describe in this paper. This is primarily due
to robotics in general not being advanced enough in
their mimicry of human movement in a reliable way
to focus on these higher-level behaviours, causing the
area of advanced realism of movement to be some-
thing that will need to be researched in the future.
2 HOW DO HUMANS NAVIGATE?
2.1 Micro-Movements - The Small
Movements We Make
Research into the small movements we unconsciously
make is an even smaller subset of the navigation field
and is ongoing, and these behaviours give a true sense
of realism when simulated or replicated well, though
most of this research is focused on face or arm move-
ments, rather than the implicit orientations and move-
ments we make during locomotion. These locomotion
movements are what we hope to formalise, as work
on facial realism and animation realism is quite ma-
ture in comparison due to the important role the face
has for humanoid perception, as well as due to mo-
tion capture as a technology. For example (Davison
et al., 2018) in their work in the field of micro fa-
cial movements have created a formalised dataset of
micro-facial movements, poised to become the new
standard. No such dataset exists for this kind of data
relevant to human navigational movements however,
and this sort of gap in the field explains why even in
highly funded, yet unreleased video games such as
”Star Citizen” (see (Ahrens et al., 2019)) non player
character performance is still substandard, and often
consists of movement to a position before rotating and
continuing a path. (Onishi et al., 2003) investigated
creating a new laboratory application to record hu-
man robot movements and test new humanoid robots,
that describe in their future work section the need to
capture human locomotion data accurately to make
robots that realistically move like humans. This is a
good example of the need for these realistic move-
ments being recognised, with work being done to im-
ICAART 2023 - 15th International Conference on Agents and Artificial Intelligence
382
prove the data capture systems that are used to cre-
ate spatial control systems which once again confirms
the necessity for further research into agents’ naviga-
tion if the aim is to increase the ”sense of realism”.
(Kuffner et al., 2003) have done incredibly impor-
tant work in the field, with their goal of creating a
digital human, including sophisticated digital models
of human physiology and biomechanics. However at
least at this stage, they have been more focused on
arm motion and upper body movements at a set lo-
cation, rather than the movements we describe here.
(Kagami et al., 2003) looks at using a motion cap-
ture system, force plates and distributed force sensors
to record data from a human participant, as well as a
humanoid ”H7” robot, however they found difficulty
comparing the human data with the H7 due to issues
in differences between the mechanical nature of a hu-
man and a robot, (including link parameters, walk-
ing speed, step length, step cycle and mechanisms).
These issues however do not exist in the virtual world
unless explicitly designed, and the large amount of
heavy data Kagami recorded emphasises the need for
a formalisation of these naturalistic, realism focused,
micro-movements. Even (Gratch et al., 2002) in their
excellent paper on virtual human realism fail to men-
tion navigation in any form, they note that the broad
range of requirements virtual humans have poses a
serious problem for research and the technology in
general. But do not touch on the specifics of micro-
movement that we describe here.
2.2 Human Movement Data
Human navigational behaviour is complex and the
data required to fully understand it can have a lot of
nuances, due to the innate complexity of the move-
ment it is representing. When looking at tracked data
from a human in an environment, such as the illustra-
tion in Figure 2 it is clear a simple location tracking of
the participant gives a good representation of where
the human moved, but not in what order, or where
they were facing during the movement. If one was
developing a human artificial social agent control sys-
tem with an emphasis on realism, data like this may
help in creating the pathfinding control, but this data
is not able to be easily utilised in the creation of a con-
trol model that includes micro-movements. When the
shoulders of the participant are tracked as in Figure
3, this gives a much better sense of where the human
was facing during locomotion and can even convey
a sense of turning while moving and strafing. This
begins to highlight the complexity of human locomo-
tion, as humans do not just face forward and move
along a straight axis, and a better way to describe the
different parts of this data would go a long way in in-
creasing the ease of implementing micro behaviours
in artificial social agents.
Figure 2: Centre of mass of a person moving through space
between 4 different spatial locations.
When looking at the illustrative human accelera-
tion data in Figure 4, one can once again see human
locomotion is not as simple as moving to a location
and stopping to turn and continuing along the calcu-
lated path, acceleration is more than just a linear up
and down. The data instead includes a large array
of different movements and angular velocities, further
highlighting how complex this data can become, and
again showing that humans indeed do employ micro-
movements that if simulated can improve the realism
of an agent, are so often missed out during research
into navigation and are the exact movements we pro-
pose need to be formalised. With annotation of the
human navigational data (Figure 3, one can see that
each part of the movement data can be ascribed to a
specific micro-movement, though even the rotate in
place annotated here may be closer to rotate in walk.
These micro-movements are described in more detail
within the next section.
This issue of translating human movement into the
virtual space has been wrestled with for a long time,
and though we do not yet have a solution for this prob-
lem, we do propose a framework for describing and
formalising this behaviour and improving ease of col-
lecting useful data.
3 FORMALISING
MICRO-MOVEMENT
BEHAVIOUR
To have realistic agent behaviour, we need to de-
velop spatial models that understand these micro be-
The Missing Tip: Lack of Micro-Movements Impairs Navigation Realism in Artificial Social Agents
383
Figure 3: Orientation during locomotion between 4 dif-
ferent spatial locations, with specific micro-movement be-
haviours from the proposed formalisation annotated.
Figure 4: These Plots of Orientation, distance and velocity
show that the orientation of a human is constantly changing
and complex in it’s nature, there is a lot of changes in accel-
eration, velocity and orientation rather than a simple (a) to
(b) with constant velocity.
haviours, and to that end this paper proposes the for-
malisation of a testing framework, by associating the
following descriptors to these micro behaviours with
diagrams representing the 6 behaviours in Figure 5.
If we assume that there are a finite set of movement
strategies that humans will use, one can divide human
locomotion into 6 key behaviours. We see these be-
haviours as consisting of; (A) ”Linear walk, where
an agent walks in a direct line congruent with their
orientation direction. (B) ”Backwards walk, where
an agent back-steps without turning their orientation.
(C) ”Strafe walk,” in which an agent steps to the side
while facing forward. (D) ”Curved walk, where an
agent is orienting gradually as they walk. (E) ”Rotate
in Place, where an agent may turn on the spot without
movement, and finally (F) ”Rotate in Walk,” in which
an agent is rotating their orientation as they step in a
specific direction, for example, turning as you back-
step into a forward walk. One can view any naviga-
tion from point (a) to point (b) as a combination of any
number of these actions, resulting in the ability for in-
vestigations into this field to more readily explain and
describe the movement behaviours they record. At
the very least, it is important for the field of agent
navigation to recognise these important navigational
micro-movements, as they are paramount in creating
the next step in virtually real humanoid agents that
foster a sense of immersion in users.
4 WHERE TO NEXT?
From the research discussed above, it is clear that
there exists a gap within the field of navigational
research centred around the simulation of micro-
movements that humans unconsciously employ dur-
ing path-planning and execution of movement. Be-
haviours such as these have been found to foster
a higher sense of immersion within virtual worlds,
something that is key to creating impactful, efficient
virtual simulations. This is especially important in
the field of virtual reality training, as the closeness to
the real-world is what makes it such a powerful tool.
With the formalisation of these micro-movements that
occur during locomotion, researchers will be able
to look at complex human locomotion data and as-
cribe these formalised terms to the different micro-
movement techniques that are employed. This will
result in researchers being more well equipped to
tackle the issue of realistic human movement in a vir-
tual space, creating this much needed and oft missing
sense of realism to the artificial social agents that in-
habit these virtual spaces. Formalisation is just the
first step however, and this future work must look at
collecting data on these micro-movements during lo-
comotion from real-life human participants, and cre-
ate an autonomous system that is able to replicate in
a realistic way and simulate these micro-movement
behaviours for an artificial social agent. Whether by
reinforcement learning systems, finite control mod-
els or some other system. This is specifically for AI
controlled agents in virtual spaces, as user-controlled
agents do not run into this issue as the user is in full
control of the navigation and thus any strange move-
ment is overlooked. The list that has been outlined
here can be seen as complete, though is open to being
improved upon, and is a starting point for the stan-
dardisation of these concepts for ease of understand-
ing and formalisation.
ICAART 2023 - 15th International Conference on Agents and Artificial Intelligence
384
Figure 5: Proposed list of atomic micro-movements, complete in that we posit there is no further way to break these motions
down in a meaningful way. The start location of the agent is the lighter blue circle, the darker blue circle denotes the end
location of movement. The arrows on the circle show body orientation, the yellow arrows denote the displacement path
irrespective of body orientation. (A) Linear walk in a direct line congruent with body orientation. (B) Linear back-step in a
direct line. (C) Strafe (side-step) to the side while facing forward. (D) Curved walk with gradual body orientation along the
curve. (E) Rotating in place with no translation of the center of mass of the body, the original and final orientation differ by
180
. (F) Rotation during displacement along a path, the original and final orientation also differ by 180
.
REFERENCES
Ahrens, J. P., Istipliler, B., Isaak, A., and Steininger, D. M.
(2019). The star citizen phenomenon & the “ulti-
mate dream management” technique in crowdfunding.
In 40th International Conference on Information Sys-
tems, ICIS 2019.
Akalin, N. and Loutfi, A. (2021). Reinforcement learning
approaches in social robotics.
Bartle, R. A. (2004). Designing Virtual Worlds.
Burgoon, J. K. and Kendon, A. (1992). Conducting Inter-
action: Patterns of Behavior in Focused Encounters.
Contemporary Sociology.
Davison, A. K., Lansley, C., Costen, N., Tan, K., and Yap,
M. H. (2018). SAMM: A Spontaneous Micro-Facial
Movement Dataset. IEEE Transactions on Affective
Computing.
Djerroud, H. and Ali-Ch
´
erif, A. (2021). VICA: A vicari-
ous cognitive architecture environment model for nav-
igation among movable obstacles. ICAART 2021 -
Proceedings of the 13th International Conference on
Agents and Artificial Intelligence, 2(Icaart):298–305.
Ghazali, A. S., Ham, J., Barakova, E., and Markopoulos,
P. (2019). Assessing the effect of persuasive robots
interactive social cues on users’ psychological reac-
tance, liking, trusting beliefs and compliance. Ad-
vanced Robotics.
Goodrich, M. A. and Schultz, A. C. (2007). Human-robot
interaction: A survey. Foundations and Trends in
Human-Computer Interaction, 1(3):203–275.
Gratch, J., Rickel, J., Andre, E., Cassell, J., Petajan, E.,
and Badler, N. (2002). Creating Interactive Virtual
Humans: Some Assembly Required. IEEE Intelligent
Systems.
Kagami, S., Mochimaru, M., Ehara, Y., Miyata, N., Nishi-
waki, K., Kanade, T., and Inoue, H. (2003). Measure-
ment and comparison of human and humanoid walk-
ing. In Proceedings of IEEE International Sympo-
sium on Computational Intelligence in Robotics and
Automation, CIRA.
Kuffner, J., Chestnutt, J., Kagami, S., Latombe, J. C., Nishi-
waki, K., Hodgins, J., LaValle, S., Inaba, M., Yamane,
K., and Inoue, H. (2003). Motion planning for digital
humans. Proceedings of IEEE International Sympo-
sium on Computational Intelligence in Robotics and
Automation, CIRA, 2:912–917.
Latoschik, M. E., Roth, D., Gall, D., Achenbach, J., Walte-
mate, T., and Botsch, M. (2017). The effect of avatar
realism in immersive social virtual realities. Proceed-
ings of the ACM Symposium on Virtual Reality Soft-
ware and Technology, VRST, Part F1319.
Lele, A. (2013). Virtual reality and its military utility. Jour-
nal of Ambient Intelligence and Humanized Comput-
ing, 4(1):17–26.
Li, R., Van Almkerk, M., Van Waveren, S., Carter, E.,
and Leite, I. (2019). Comparing Human-Robot
Proxemics between Virtual Reality and the Real
World. ACM/IEEE International Conference on
Human-Robot Interaction, 2019-March:431–439.
Liu, P., Glas, D. F., Kanda, T., and Ishiguro, H. (2018).
The Missing Tip: Lack of Micro-Movements Impairs Navigation Realism in Artificial Social Agents
385
Learning proactive behavior for interactive social
robots. Autonomous Robots.
Maslow, A. H. (1943). A theory of human motivation. Psy-
chological Review.
Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-
Kennicutt, W., and Davis, T. J. (2014). Effectiveness
of virtual reality-based instruction on students’ learn-
ing outcomes in K-12 and higher education: A meta-
analysis. Computers and Education.
Olcay, E., Schuhmann, F., and Lohmann, B. (2020). Col-
lective navigation of a multi-robot system in an un-
known environment. Robotics and Autonomous Sys-
tems, 132:103604.
Onishi, M., Odashima, T., Asano, F., and Luo, Z. (2003).
Development of PC-based 3D dynamic human inter-
active robot simulator. In Proceedings of IEEE Inter-
national Symposium on Computational Intelligence in
Robotics and Automation, CIRA.
Pedica, C. and Vilhj
´
almsson, H. (2008). Social perception
and steering for online avatars. Lecture Notes in Com-
puter Science (including subseries Lecture Notes in
Artificial Intelligence and Lecture Notes in Bioinfor-
matics), 5208 LNAI:104–116.
P
¨
utten, A. M. V. D., Kr
¨
amer, N. C., and Gratch, J. (2009).
Who’s there? Can a Virtual Agent Really Elicit Social
Presence? Design, pages 1–7.
Raees, M. and Ullah, S. (2021). RUN: rational ubiqui-
tous navigation, a model for automated navigation and
searching in virtual environments. Virtual Reality,
25(2):511–521.
Sutera, E., Mazzia, V., Salvetti, F., Fantin, G., and Chi-
aberge, M. (2021). Indoor point-to-point navigation
with deep reinforcement learning and ultra-wideband.
In ICAART 2021 - Proceedings of the 13th Inter-
national Conference on Agents and Artificial Intelli-
gence.
Wang, Y., Dubey, R., Magnenat-Thalmann, N., and Thal-
mann, D. (2013). An immersive multi-agent sys-
tem for interactive applications. Visual Computer,
29(5):323–332.
Zhukov, S. and Iones, A. (2000). Building the navigational
maps for intelligent agents. Computers and Graphics
(Pergamon), 24(1):79–89.
ICAART 2023 - 15th International Conference on Agents and Artificial Intelligence
386