Emerging Complexity: Communication between Agents in a MAS for

Shape-shifting TUIs

Helen Hasenfuss

Abbeyfeale, Co. Limerick, Ireland

Keywords: Communication, Shape-shifting, Self-assembling, Multiagent Systems.

Abstract: Communication is an essential component for creating shape-shifting, interactive interfaces. This paper

discusses the early stages in constructing a communication protocol for a specific agent (the Dod (Hasenfuss,

2019)), to be used in a multiagent system (MAS). It is part of a larger study focused on developing novel

interactive interfaces. Supporting the attempt to create a viable blueprint for an agent design, communication

was explored from the perspective of artificial intelligence (AI). The components necessary for

communication are tied to deeper constructs that cater for qualities such as coping mechanisms,

understanding, interpretation and awareness. In the process of developing aware or conscious agents, the

challenges of ethical usage of such technology comes to the foreground. Raising the necessary questions

around these issues even in the development of a blueprint, can ensure a more informed and wholesome

adaptation of the design once these issues can be resolved (e.g. final application, construction material,

working environment, etc.).

1 INTRODUCTION

As a continuation from last year's paper presented at

CHIRA 2019 (Hasenfuss, 2019), this paper aims to

discuss the aspect of communication necessary

between agents in a multiagent systems (MAS) but

also the need for machine learning or artificial

intelligence (AI), in order for agents to function

autonomously. A MAS consists of a quantity of

individual agents operating together to create larger

macro structures. A biological example is that of an

ant or termite colony.

The capacity for agents to learn specifics about

their environment and given task is essential in order

to be able to achieve shape shifting, tangible user

interfaces (TUIs). The learning process should ideally

accommodate the following actions:

 learn from past experiences

 remember optimum cluster formations in the

creation of specific 3D structures,

 cope with partial malfunctions,

 Cope with unexpected event (falling of an edge,

being pushed by other agents, etc

 recognise actions of the user and respond to

them,

https://orcid.org/0000-0001-5703-2685

 forget irrelevant or obsolete information

This list is not exhaustive but aims to illustrate the

complexity of each task when it is necessary to build

or code these behaviours from the beginning.

Alongside the question of intelligence, is its

connection to the concept of ethics. Whilst ethics is a

difficult topic to define, it is useful to consider the

effect, of applying certain values in the early stages

of technology development. Contrary to the current

trend of designing machines that behave ethically

(e.g. self-driving cars), a focus of this paper will be to

consider design parameters that can define or

accommodate ethical restraints.

The Dod design will be used to demonstrate the

process of creating rulesets necessary to initiate

autonomy in artificial agents. The Dod is a design

based on a dodecahedron, with a retractable arm

extending from each of the 12 facets (Hasenfuss,

2019), see Figure 1.

The overall result of the study that produced the

Dod design, is a blueprint of a MAS agent design that

can be used as is, or adapted to future technological

developments or applications.

Section two will highlight some of the works that

have focused exclusively on agent communication

Hasenfuss, H.

Emerging Complexity: Communication between Agents in a MAS for Shape-shifting TUIs.

DOI: 10.5220/0010140800440055

In Proceedings of the 4th International Conference on Computer-Human Interaction Research and Applications (CHIRA 2020), pages 44-55

ISBN: 978-989-758-480-0

and have created working prototypes. It highlights

how different approaches to communication can

influence design choices and also have an impact on

the design itself.

Figure 1: The Dod.

The Dod represents a primary focus on

developing a physical agent design. As a result, the

communication procedure envisioned for the Dod

adapts to the physical affordances. It also draws

inspiration from organic behaviours of lifeforms that

have similar physical qualities, (e.g. as exhibited by

octopuses, sea anemones, or starfish). This process

will be discussed in section three.

In section four, the rulesets defined for the current

state of the Dod will be explored. This relates to the

local structures it is capable of creating (e.g. line,

curve and cluster) and its effect on the overall macro

structure. Elements such as textural changes, colours

and movement can be useful in the design of user

interactions with shape-shifting technology.

The proof that biological systems have already

catered for many extraordinary designs is evident in

the diverse number of research projects that are

directly or peripherally influenced by elements of

nature (Ridden, 2015; Petersen et al., 2014; JPL,

2012). Therefore, it is not entirely unrealistic to

consider developing the Dod into the biological

domain. Continuing along this avenue not only brings

technological challenges but also ethical ones.

Section five will discuss the relevancy of ethical

parameters that can arise when designing AI agents,

as is required for shape-shifting technology.

2 RELATED WORK

The concept of communication in a MAS has

received a large proportion of attention in academic

research because it is a) accessible, particularly in

digital format, b) relatable and provides insight into a

diverse range of social groups and c) there is still a

great deal that is not completely understood. MAS

communication is not just used in shape-shifting

technology but also helps explore concepts such as

swarm computing, crowd or insect behaviour and

networked systems such as IoT. With respect to the

behaviour traits and communication of Dod agents,

since the aim was not to build a fully functioning

prototype. The projects listed in this section were

explored for their suitability and inspired the

proposed communication behaviours, for the Dod,

detailed in section four. Three of the projects are

based on working prototypes and effectively illustrate

the interconnected relationship between

communication and physical structure. The method of

communication can also influence the style of self-

assembly (static or dynamic). Several different

approaches for programming these methods have

been developed as a result (Butera, 2002; Le Goc et

al., 2016; Özgür et al., 2017; Romanishin et al., 2015;

Roudaut et al., 2016;Rubenstein, 2014). These

approaches will be briefly described in the following

list. Determining which approach is most efficient,

with respect to a computer-based system, is still being

experimented with.

 In his proposal of pushpin computing, Lifton

describes information travelling from one agent

to another via programming fragments: a

distributed sensor network (Lifton, 2002). These

fragments pass on the commands to their next

nearest neighbour until the require task is

completed. In analysing the graphical data of

this work, it is possible to draw an analogy

between the spread of commands amongst the

agents to the propagation of a viral infection, i.e.

an exponential growth. The potential exists for

this to be achieved through physical contact or

defining a sensory range.

 Another suggestion detailed in the project

Proteo, is to have several seed agents dispersed

throughout the system. These would act as core

points around which the other agents can gather

and orientate themselves. They would have

slightly different coding and be able to make

more managerial decisions (Le Goc, 2016;

Bojinov et al., 2002).

 ‘Ghost’ trails (Uhrmacher et al., 2009), which

are very similar to the pheromone trails left by

ants and other insects, have been developed to

indicate location and type of message (food:

energy, danger: obstacle, etc). Depending on

how many agents use the trail, it is strengthened.

An initially chaotic field of trails eventually

becomes ordered into optimum paths (Tero et

al., 2007).

 Organisation according to stigmergy is another

approach. It is based on the mechanism by

which agents can organize through

commonalities in the environment (Tummolini

Emerging Complexity: Communication between Agents in a MAS for Shape-shifting TUIs

et al., 2009) and is very closely linked to the

ghost trails approach. The main principle is

based on the fact that a single ant can lay a trail

that indicates a good food source. This anomaly

of difference or change in the environment

influences the next actions taken by the same ant

or others that find the trail and strengthen it

(Tummolini et al., 2009). It is also possible to

alter the environment to influence the behaviour

or reaction of the agent.

2.1 Existing Platforms

Adaptability, flexibility, stability, endurance and

robustness are desirable qualities of a MAS

communication protocol. Kilobot, Zooids and Cellulo

are three projects that embody these qualities and will

be briefly discussed in order to provide valuable

insights that can help define behavioural

characteristics of any kind of physical agent, and in

this case the Dod.

2.1.1 Kilobot

Rubenstein details the effective swarm

communication of robots numbering in the hundreds

(approx. 1024 Kilobots) (Rubenstein et al., 2014).

The Kilobot is a circular robot with IR sensing for

distance and positioning calculation, and vibration

motors for locomotion (Rubenstein et al., 2014).

These robots are based on defining boundaries rather

than the placement of each agent, and they self-

organise into shapes (quasi 2D planar shapes) as

opposed to self-assembling into 3D structures (i.e.

along the z-axis).

Defining boundaries indicates a great deal of

system flexibility with respect to packing patterns for

self-organisation. Being able to sense or recognise

boundaries is also a crucial skill for an autonomous

agent, especially in the process of adapting to

specific, complex, prescribed boundary conditions

such as a keyed interface. Such flexibility is also

indicative of a natural approach to organic assembly

(Rubenstein et al., 2014). It incorporates the ability to

be adaptable and responsive to changing

circumstances which can be beneficial in the survival

of a particular design or behaviour. Each agent in this

project is only aware of itself in relation to its local

neighbouring agents. Localisation is achieved by

averaging the distance samples between the sensing

agent and its surrounding neighbours. Distance is

defined by the communication between agents, i.e.

the update of messages. Maintaining a list of sampled

distances ensures that behavioural adjustments can be

made based on the comparison of values. For

example, if a robot were pushed over a ledge it would

realize from the sudden change in several distance-

values, that its position had changed suddenly and not

as part of the self-organising process.

The following observations that emerged from the

Kilobot project can be applicable and relevant for any

multiagent system:

 Variation in the ability of agents due to

components (motion, extension, learning,

transmitting, etc).

 Rare or unpredictable events causing errors

either from within or external environmental

influence (e.g. internal influence: short circuit,

external influence: poke)

 Message corruptions – multiple message per

channel, chatter between agents, random noise.

This can have a domino effect on other

dependant factor, e.g. failing to sense

boundaries

 Anisotropic boundary measurements

2.1.2 Zooids

Zooids is a project based on swarm computing to

create user interfaces. The main aim of this research

is to close the gap between actuated tangible table-

tops, whereby solid materials can be manipulated, and

shape displays that use the method of physical

deformation to manipulate digital data. Like a natural

ant hive, this project has a centralised control system

to coordinate the Zooids. Similar in physical structure

to Kilobots, Zooids are battery powered, circular

robots; they move according to motor driven wheels

and they can self-organise. They utilise capacitive

touch for sensing and communicate through a radio

receiver (Le Goc, 2016). Overall, the physical Zooid

agent represents only a quarter of the entire MAS: an

application sets the goal, then a simulation is

responsible for path planning, proceeding to a server

that sends commands to each individual Zooids. Once

each Zooid receives its instructions specific

predefined algorithms are initiated and executed as

required. Of interest is the conclusion that to be

considered viable for real-time applications, a MAS

or programmable matter-based interface must be able

to execute any function or motion or shape change in

‘the order of one second’ (Le Goc, 2016; Nielson,

1995). This relates to the user’s expectation, focus

and need for feedback regarding system states. The

following qualities are described by the authors which

merit careful consideration.

 Continuous versus discrete positioning: the

advantage of swarm computing is the possibility

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

of availing of continuous positioning.

Continuous values contain more information,

provide greater accuracy and resolution, and

possibly even render a more faithful

representation of physical matter. Discrete

values are most often used in the digital domain

and represent values that have been interpolated

in order to convey the essence contained within

digital data, e.g. an image that is converted into

square pixels.

 The concept of a system being able to adapt to a

fixed number of elements or one where the agent

count differs, i.e. active and inactive agents.

This is true of natural systems. A MAS must be

capable of dealing with agents that malfunction

or, if for example a shortage of energy is

detected, function with a reduced capacity of

agents.

 The ability of agents to change their roles is a

quality that is also echoed in certain ant species

(Gordon, 2010). The ability to decide which

function is appropriate to the current task

contributes to the awareness level, enabling the

agent to determine the next course of action with

greater independence.

2.1.3 Cellulo

The last project worth considering with respect to

inter-agent communication is Cellulo. Cellulo is a

tangible table-top based interface whose main

objectives are to be robust, reliable, affordable and

versatile within a classroom or educational setting. As

these are their main aims, the communication

protocols have not reached the level of sophistication

required for the envisioned autonomous agents of a

MAS. However, it is possible to consider elements

that can be used to create a robust foundation ruleset

for MAS agents. Cellulo agents communicate via

Bluetooth and have a downward facing camera to

read the printed microdot patterns over which they

move. The agents are capable of holonomic

movement due to an omnidirectional ball drive

(Özgür, 2017). Each agent communicates, via

Bluetooth, with a ‘master’ tablet. Bluetooth

technology limits the number of agents that can be

active at any given stage and this system is an

example of how each agent is instructed what to do

individually. An important quality in this project, is

the ability of the Cellulo agents to cope with

unexpected events, e.g. kidnapping: when a child

removes an agent from the table top, or physical

manipulation: pushing an agent against the

designated direction, etc. This is due to the relative

simplicity of the system with respect to its movement

and environmental localisation capabilities, i.e. a

camera reading a microdot pattern and following an

instruction. The simplicity of its ruleset means the

Cellulo agents exhibit predictable behaviour. As a

result, it is possible to focus on apparent collective

behaviour: the final, perceived behaviour versus the

actual, inner functions.

These are functioning projects that achieve

effective communication in physical MAS, ranging

from 7 to 1000 agents. Whilst the core issue of

awareness is not yet fully solved, it is possible to see

the potential of using such systems as basic templates

for communication protocols of artificial agents. The

following section explores the basic movement-based

mechanisms that are available to the Dod design, with

the aim of eventually leading to defining which

communication protocol is best suited for future

developments of a Dod MAS.

3 COMMUNICATION TYPES

As mentioned previously the term MAS has had most

usage describing digitally simulated, multiagent

systems. An advantage in testing communication

protocols, such as the ones described in section two,

via digital simulations, is the possibility to manipulate

the variable of Time. Whilst it is possible to observe

patterns and development of communication in real-

time, the timespan for such observations potentially

precludes any useful application of the collected data.

Digital simulations are also a contributing factor as to

why communication models have received

continuous interest and research. The development of

communication and the means to embody it are

usually symbiotic-ally interlinked within a system. In

the domain of man-made agents, however, it is

sometimes necessary to consider one characteristic

over the other, e.g. the physical appearance before

communication technique or vice versa: choosing a

specific communication style that delineates the

resulting physical design (Gorbet et al., 1998; Lifton

et al., 2004; Nakayasu, 2010; Richardson et al.,

2004). The Dod is primarily designed to

accommodate self-assembly and the ability to create

complex, 3D macro structures. In this case the

element of communication has secondary priority, i.e.

it must develop from the physical structure. The field

of Biomimicry had a strong influence on defining

design parameters for the Dod. The octopus, star fish,

sea anemone and euplectella aspergillum were

referenced in relation to how a design such as the Dod

could move, how its internal circuitry could be

Emerging Complexity: Communication between Agents in a MAS for Shape-shifting TUIs

accommodated and what kind of behaviours would be

possible. For example, the brain of an octopus is

divided between an internal component and its skin.

Rather than having just one central processing unit,

its skin / appendages also have a degree of processing

power allowing it to execute complex camouflaging

with greater ease. Examining these structures,

provides insight into existing communication and

awareness models and how they function in relation

to their given physical form. The act of

communication will be separated into two processes

for this paper: intrinsic and extrinsic communication.

3.1 Intrinsic Communication

Communication based on an intrinsic level would

indicate that each agent receives the same knowledge

to begin with, but after initiation, is responsible for

sensing, storing and analysis of any extraneous

information that is gathered. The agent communicates

within itself more so than with its neighbours in order

to make sense of its surrounding. These types of agent

could easily become leading agents, particularly if

placed among agents that have less knowledge. This

setup creates individualistically orientated, valuable

agents because of the resources required to facilitate

each agent with learning capacity, memory storage,

energy to transmit and direct, knowledge processing,

decision making. More energy is consumed by

individual agents but the potential exists that these

agents can make decisions based on the information

collected.

3.2 Extrinsic Communication

Communication based on an extrinsic level would

indicate that the external environment, like

stigmergy, controls each agent. Such a setup would

require very precise and definitive instructions from

the environment to achieve specific goals. Aside from

environmental influences, this type of

communication requires a greater proportion of

agent-to-agent interaction. In this setup it could be

possible to pool information together, collected by

each Dod, such that the common knowledge base of

the MAS is maintained. While the potential exists for

memory storage to be used up faster, it would ensure

that each Dod is aware of the overall system. This is

useful for determining system states or if the Dod

based MAS is applied in a sensor network

application.

Applying these concepts, consider the following

example: a group of 50 adult humans is asked to link

together in order to create the shape of the letter ‘B’.

Each person understands the concept of the letter.

One solution is if people link hands and follow one

person who walks the outline of the shape. By being

linked everyone not only knows the state of the

system (i.e. how many people there, how far to spread

out, when to move, etc.) but they also learn more

about their environment. The people following do not

always know where or when the shape ends because

they are part of a linked chain.

Alternatively, people can also arrange themselves

individually by realizing where they are needed and

what needs to be done to complete the structure. Each

person takes the responsibility to be aware of the

overall goal as well as their individual place. This

ability means that the people are not tied to a specific

place in the shape and can adapt to or compensate for

unforeseen events.

The example highlights the strengths and

weaknesses in both approaches of communication

and the behaviours that can emerge as a result. The

ideal case would be a combination of intrinsic and

extrinsic influences. Each agent should be influenced

by its environment, i.e. be aware of surrounding Dods

and the environment in which they move, enabling

them to make informed decisions, but also

information should be passed from one agent to

another to foster a collective awareness. Despite the

application of these types of behaviours, i.e. direct

coding, falls outside the scope of this study, the

application of learning algorithms is an important

aspect of future work for the Dod agent.

Considering these approaches has an impact on

the development of the Dod design, since a

fundamental difference exists between an internally

driven and an environmentally driven motivation to

self-assemble or self-organise. For example, in ant

hives, it emerges that the lack of a singular

hierarchical chain of command ensures that a system

can expand and maintain optimum flexibility

(Gordon, 2010). Alternatively, the concept of seed

particles could be translated into representing the seed

particles as being physically different - slightly larger,

or have a different configuration, etc. to provide a

structural initiation marker instead of digitally

communicating its significance. Whilst extrinsic and

intrinsic communication describes how an agent acts

and learns within the entire system, another layer of

programming exists that is specific to each individual

agent. This layer of programming refers to a

fundamental ruleset and is evident in biological and

inorganic agents.

The model presented by Kristinn Thórisson was

used as a method of structuring the task of creating a

fundamental rule set for the Dod. In summary, he

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

highlights three distinct layers of awareness in an

autonomous agent (Thórisson, 1996):

 The inner or bottom layer refers to the agent’s

rudimentary functions, i.e. how it fundamentally

works: the system mechanisms, the rules and

coding that characterize it. This layer of

awareness functions even if the incoming

sensory information is not being processed.

 The 2nd or middle layer refers to the behaviours

of the agent. These behaviours emerge and are

defined as a result of the interactions between

the 1st level rules. This encompasses the variety

of combinations in which the rudimentary

mechanisms can function together.

 The 3rd level and what can possibly be

considered the topmost level, is one in which the

agent behaviour interacts with the environment

in which it is in, i.e. its surroundings and the

objects therein as well as other agents. This is

the boundary between internal and external

environments.

Once the process of awareness begins, each layer is

interlinked and appropriately informs the next layer.

This model indicates the bi-directionality of

awareness but also the balance between ‘hardcoded’

behaviours found in the innermost layer and the

flexible, adaptability of the remaining layers in their

participation of information exchange. For example,

consider an agent whose 1st level, basic rule set

consists of moving, talking and collecting. The 2nd

level behaviours that could emerge from this are

moving, talking, collecting, foraging (moving +

collecting), storing information (talking + collecting)

and spreading of information (talking + moving). The

behaviour of an agent is continuously being informed

by its basic rule set and in the 3rd level, the agent acts

and communicates outside of itself, i.e. in the

environment. Factors such as stigmergy may become

relevant as well as contact with other agents and

whether or not they have similar or different

behaviours. This description is an example of a purely

mechanical or logical rule set and can be considered

to have relatively predictable variables. The

introduction of emotions as an unpredictable and

undefinable ruleset, clearly indicates the complexity

of interaction that can begin to emerge.

The integration of these three levels provides the

capacity for varying degrees of awareness and

thereby communication abilities. This includes being

able to prioritize knowledge through active learning,

forgetting irrelevant information, pooling knowledge

between agents and passing on or receiving

knowledge gained from past experiences or different

parts of the system.

In future MAS developments, the depth of

autonomy required for agents will predominantly

define the refinement of the basic ruleset. For this

study the Dod is considered to function on a purely

mechanical level.

4 Dod RULESET

The Dod’s fundamental ruleset (the inner layer),

emerged from the affordances of its physical design.

As mentioned in section three, existing systems that

have a similar shape or mechanisms also helped

inform possible behavioural rules and patterns. For

example, octopus and anemones explore their

surrounding via appendages or tentacles that can

extend or retract, are flexible and are essential in

completing everyday tasks such as environmental

exploration, foraging, protection, defence,

camouflage, etc. With the ability to extend arms, the

Dod could also use it’s extendable arms as ‘feelers’

to scope out its surrounding, or communicate with

neighbouring Dods, etc.

An important differentiation is that these

rulesets do not automatically ensure an innate

awareness. In the Dod’s case, they currently cater for

the rudimentary mechanisms (e.g. creating line or

cluster structures) that are possible through its design

(e.g. extending arms, semi-spherical nature, etc). This

means that if the Dod develops further, with respect

to construction material, sensory ability, user

application - this ruleset should remain unaffected;

the equivalent to default factory settings. For the

following description of developing a fundamental

ruleset, the Dod hypothetically determines its position

through IR sensors that are set into each arm / facet.

4.1 Level 1 Mechanisms

This section details how a Dod physically functions

on its own and it includes the description of how

features such as lines, curves and clusters are

constructed. The following points are deemed as

being necessary components for the basic level of

Dod awareness, from which fundamental rules can be

formed.

Body Mechanisms

 Extend and retract arms

 Connect & disconnect to other arms

 Orientate to any 12 arms

 Sense next closest Dod

 Sense the immediate environment

Emerging Complexity: Communication between Agents in a MAS for Shape-shifting TUIs

Computational Mechanism

 Emit ready to connect tag = attachable

 Emit connected tag = attached and # of arm

 Emit not functioning tag = arm failure

 Emit standby tag = waiting for use

 Send and receive command messages

 Memory: the ability to store past, present and

future tags in command line

 Memory: the ability to remember and forget

(once a command has past or a goal has been

achieved)

Fundamental Rules:

1) An arm can go in (0) or out (1),

2) Any arm will always have 5 surrounding it.

3) The decision on which arm is extended or not

will eventually depend on weighted probability.

This is based on which configurations are most

advantageous given the different states.

4) When exploring the environment, a Dod extends

its arms like the feelers of an insect. When it

encounters a solid boundary, it determines that it

is either a surface meaning it can potentially

move in that direction or it is another Dod. The

concept of awareness must be considered at a

very primeval level particularly when

considering scalability. Dods are designed to feel

rather than see therefore in order to check

whether a Dod is connected they must either emit

a connect tag: like a pheromone or must receive

force feedback such as resistance when pushed.

5) To construct a line or curve only two arms are

required by each Dod and in a cluster a minimum

of three Dods are required with a vast variety of

arm states possible.

6) It is currently envisioned that commands are

passed from one Dod to another via contact.

From these basic guidelines, it is possible to construct

a set of states, e.g. tags that determine specific actions

and behaviours. The term tag is symbolic of the piece

of coding that the Dod would work with and act upon,

similar to the program fragments as described in

Lifton’s thesis on Pushpin Computing (Lifton, 2002).

To date three states have been established for the

Dod: line, curve and cluster. They can be viewed as

basic structural components for 2D and 3D structures.

The root position is one in which the Dod rests

completely on one facet. When the arm connected to

this facet is extended, it will be defined as the

standing arm for ease of visualisation. Another stable

configuration is when all arms are extended and the

standing arm and its mirror arm are retracted: the

edges of the surrounding extended arms support the

Dod, see Figure 2. Due to the inherent symmetry of

the dodecahedron it is possible for any facet to

become the root position. This is what is meant by the

Dod being able to orientate to any of the 12 arms, if

the sensor ID remains static for each Arm, then Arm

ID is relative to the sensor value, e.g. Arm 1 can

become Arm 7. Therein lies great flexibility, because

it is possible for the Dod to reposition and orientate

itself irrespective of the task, location and orientation.

Figure 2: Root position – (a) root position with standing

arms retracted & extended indicated by the red lines, (b)

Facet / Arm allocation.

Figure 2b. illustrates the mapping of the remaining

arms when in the root position configuration, Arm 1

(bottom hemisphere) and Arm 12 (top hemisphere)

are designated as the default standing arms in root

position.

4.1.1 The Line Tag

Line tag (LT): In the line state, the fundamental rule

is: opposite arms always engage or activate together.

The arms can A) both extend, B) retract or C) have

one extend and one retract. It is possible to adjust

which arms are retracted or extended, resulting

in straight-lined structures of varying heights, see

Figure 3.

Figure 3: Line heights - Varying heights due to the

configuration of arms that are retracted or extended.

4.1.2 The Curve Tag

Curve tag (CT): In the curve state, the fundamental

rule is: opposite arms never engage or activate

together. The Dod is capable of two types of angle

positions. There is the AcCurve tag (AcC) for an

acute angle (root position + arm from lower

hemisphere) or the ObCurve tag (ObC) for an obtuse

angle (root position + arm from upper hemisphere).

Rather than activating opposing arms, the first

arm is identified and then another arm either on the

upper or lower hemisphere of the dodecahedron can

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

be activated. This creates acute or obtuse angles

without much strain on a mechanism such as a joint

or lever. Due to the semi-spherical but faceted nature

of the Dod, the facets between upper and lower

hemisphere are offset to each other This influences

the dimensionality of the circle or type of line formed

with this tag, i.e. instead of a flat circle, there is a

slight twist or wobble because of the offset facets.

It takes a minimum of 6 Dods to form a circle

using obtuse angles, Figure 4a. Applying this tag to a

line, creates a spiralling, curved line, Figure 4b.

Figure 4: (a) circle of Dods, (b) curvy line of Dod.

4.1.3 The Cluster Tag

The last rule relates to the Cluster tag. A cluster is

comprised of Dods with specific arm configurations.

A configuration in this instance is defined as the

position of the arm states (extended or retracted) such

that another Dod can attach to these active arms and

form an anchor. The main usage for this tag is to fill

space, thereby creating a variety of texture and

emulating material density. This can be achieved via

three options:

1) All arms retracted = a dense, compact filling of

Dods

2) All arms extended = a porous filling of Dods

3) Via configurations = a starting point of three

Dods connected in one of four specific patterns.

These connections each in turn provide the

opportunity for a wide variety of configurations

to which other Dods can connect and continue to

build structures. In this manner, it is possible to

juxtapose areas of dense and sparse clustering of

Dods.

Whilst it is possible to define or code the line and

curve tag with greater accuracy, the challenge arose

in attempting to define the Dod’s behaviour, to enable

them to cope in a state of greater uncertainty. The rule

had to instruct the Dod in how to behave (what arms

to extend and / or connect) but then also to allow for

further interactions to evolve independently

according to each Dod’s decision. In the cluster state,

there is still scope for many adaptations. Enabling the

Dod with the capacity to learn the most often used

configurations, when emulating specific materials,

might only for example require 3 and 7 armed Dods

as opposed to all arms retracted or extended, etc. This

latter point precedes the possibility that whilst the

current suggestion highlights that all arms are extend-

able it may emerge that only a certain number of arm

configurations are necessary. Superfluous

configurations may be eliminated, thereby creating a

more efficient Dod. A method of determining the

most successful configurations could be through

evolutionary algorithms. Once a Dod modelled in

such a way, the potential exists to define which

configurations would be best suited for specific

applications.

A cluster consists of a minimum of three agents

connecting and attaching with each other and a cluster

anchor is comprised of the six arms (2 from each Dod)

being arranged in specific configurations. When a

cluster anchor is generated, two craters are formed on

either side of the anchor via the adjacent arms, into

which other single Dods can attach. This enables two

sides to be delineated: Side A and Side B, Figure 5.

The craters on either side determine the arm

configuration necessary for another Dod to attach -

this configuration is defined by three arms: 3 arms

extended (3E), 3 arms retracted (3R), 2 extended arms

and 1 retracted arm (2E1R) or 2 retracted arms and 1

extended arm (2R1E). The number and variety of

craters formed is dependent on the state of the arms

adjacent to the anchoring arms. These are described

through the definition of four Cases.

Figure 5: Craters - A crater formation of 2E1R Arms is

evident on Side A and a R1E Arms, crater is visible on Side

4.1.4 Case A and B

The state of all arms retracted (Case A) or all arms

extended (Case B) will generally become unstable if

similar state Dods come together. This instability is

due to the fitting error caused by the dihedral angle.

The Dod design enables packing to continue despite

not being a perfect fit (Teich et al., 2016). These cases

(and thereby the configurations) should ideally have

the lowest probability weighting, alternatively they

can be implemented primarily for generating dense or

porous masses.

Emerging Complexity: Communication between Agents in a MAS for Shape-shifting TUIs

Case A

When all arms are retracted, craters are formed on

side A and B that generate configurations to accept

totally retracted Dods or Dods with three adjacent

arms retracted (3R), see Figure 6.

Figure 6: Case A.

Case B

When all arms are extended, craters are formed on

side A and B that generate configurations to accept

totally extended Dods or Dods with three adjacent

arms extended (3E), see Figure 7.

Figure 7: Case B.

Case C and D

These cases present clusters that have more variety

with respect to the configuration possibilities. They

can be implemented in the construction of materials

that require a mixed density.

When ADod1 has 2 extended arms, ADod2 has 2

retracted arms, ADod3 has 2 retracted arms, it is

possible to generate approx. 34 craters with different

three-arm configurations in this setup (some

configurations are mirrored through symmetry and

therefore not counted), see Figure 8.

Figure 8: Case C.

When ADod1 has 2 extended arms, ADod2 has 2

retracted arms, ADod3 has 1 extended & 1 retracted

arm it is possible to generate approx. 40 craters with

different three-arm configurations in this setup, see

Figure 9.

Figure 9: Case D.

The aim of this section is to illustrate the rulesets that

will eventually define how the Dod behaves and

communicates. It is the inner layer according to

Thórisson’s awareness model and represent the

inherent rulesets that each Dod can avail of. Greater

in-depth detail of the Dods fundamental rulesets can

be read in the original study (Hasenfuss, 2018). The

next section will discuss the implications of learning

algorithms, important factors to consider in the

communication process and the ethics involved in

using AI.

5 DISCUSSION

The previous section has presented the rudimentary

functions envisioned for the Dod. These are primarily

influenced by its physical affordances. The kind of

sensing capabilities the Dod can eventually avail of,

will be in part dependant on the end application, and

the environment it will be used in. In conjunction with

the programming described in the Zooids and Kilobot

projects, it is feasible to postulate that it would be

possible to create a high-fidelity working prototype of

a Dod based MAS. With the addition of each layer of

awareness, as proposed by Thórisson, the degree of

complexity increases. Feynman’s response to

complex systems had a strong influence in the process

of modelling the Dod’s fundamental rule-sets. He

suggests that complex systems are comprised of very

simple rules. There is a hierarchy of complexity that

builds upon the layers of variables that affect the

system and the reason it appears complex is the

distance of understanding between the simple rules

and the final product or system (Dallas, 2015). For

example, in a system that has four independent

variables acting on it, it is possible to understand the

system in its entirety but also to potentially predict

future behaviour dependent on the changes in those

four variables. In contrast consider if the same system

was now exposed to 25 variables: 10 of which are

independent, 2 of which are dependent on 6 others, 8

are interlinked with 11 (3 of which belong to the

group of 6) and 5 that occur only when there is a

change in 4 of the other variables (3 of the interlinked

group and 1 of the dependent). It becomes clear that

as the simple rules increase in complexity within

themselves, it is more difficult to model the system

accurately. The number of variables that affect the

system grows significantly and the ability to predict

its behaviour is greatly decreased (University of

Groningen, 2016). Whilst this is only a fictitious

example it offers perspective into the task of defining

fundamental rulesets for autonomous man-made

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

agents that will eventually be able to interact with, act

on and react to their environment. Another concept,

also illustrated through this example, is with the

increase of complexity comes an increased

probability of error and unpredictability. In the

process of developing a working blueprint for a MAS

agent design, rather than a working prototype, it has

been possible to consider alternative influences, such

as error and randomness. The two influences that will

be discussed in this section is the role and

interpretation of error and the ethical considerations

in dealing with intelligent agents.

A counterpoint to the building of structures and

the creation of order is the release of entropic energy

back into a system. In the proposed MAS, the aim is

to achieve 3D self-assembled, known or newly

imagined, structures; generating order from a mass of

disorganized and chaotic agents. Error is an integral

part of this process and as a designer rather than

attempting to account or provide a solution for every

possible error, it is possibly more important to enable

the system to cope with error itself. Qualities such as

self-repair and being able to function despite minor

system failures can have substantial advantages, not

only for the system itself but on the scope of

resources, energy efficiency and sustainability. The

concept of coping, also inherently holds the ability to

interpret error, which in theory provides a more

wholesome degree of awareness. In general, error is

considered to be unacceptable predominantly because

it delineates how a system or product has not

functioned as intended. It is true that certain errors are

fatal to systems and clearly define the space between

functioning and being irreparably broken. However,

before these detrimental errors occur, there is a degree

of scope or tolerance, whereby errors, mistakes or

failures are essential components in the process of

learning and creativity. It introduces an element of

unpredictability. Whereas unpredictability,

randomness and chance have been primarily

embraced in the arts-based disciplines, this state of

existence or outcome proves more problematic in the

sciences, particularly in engineering (for valid

reasons). Throughout the development of the Dod,

error was explored with respect to its suitability to be

considered a creative or mutative force in the self-

assembling and design process. Whereas the act of

learning from mistakes is a relatively intuitive

procedure for most organic organisms, translating this

into an artificial algorithm becomes an interesting

challenge. Should artificial agents be able to

recognise the difference between error and creativity?

If they can make that distinction, shouldn’t they be

able to use their interpretation for problem-solving

rather than avoiding the behaviour or action, that

caused an ‘error’, altogether? It may be that through

error, alternative, possibly even better, methods for

completing tasks can evolve. For example, exploring

how people with a neurodiversity learn (e.g. autism

or dyslexia) can provide valuable insights into

alternative processing, interpretation and

understanding mechanisms that are involved in

learning. It has the potential to assist in creating more

adaptable learning algorithms. By imbuing agents

with this degree of learning ability, a natural

progression for agents is to continue learning beyond

the confines of their original programming. It raises

questions of sentience and is being continuously

explored through the diverse range of human

expression: from logic to creativity.

AI is a field in its own right and is still receiving

a large amount of academic and industry attention:

self-driving cars, human-robot collaborations. As

with any design project, the Dod fulfilled certain

design characteristics that influenced its physical

shape:

1 A semi-spherical shape with an irregular,

cratered surface.

2 Non-hierarchical chain of command: autonomy

to function as individuals

3 The ability to morph: surface topology and

fundamental form

4 One material make-up and scalability –

structural affordances and inherent material

qualities

5 Bi-directionality – the ability to assemble and

dis-assemble

6 Behavioural simplicity

The projects listed in section two, demonstrate

working communication protocols that could be

transferred to the Dod design. Adaptations would

have to be made in order to accommodate the 3D

assembly aspect, however, elements regarding agent-

to-agent interactions and overall environmental

awareness are developed and tested. The concept of

prioritising design parameters that favour physical

shape over communication is also applicable with

respect to an agent’s autonomy and awareness. A

distinction must be made relating to an agent’s

functional and intelligent communication. Functional

communication is similar to the 1st two levels of

awareness described by Thórisson; it enables the

agent to function physically, to sense and form

operations using that sensory information. Intelligent

communication begins to integrate the environment

into the awareness of the agent. It must now assess its

situation relative to others and contextualise itself

Emerging Complexity: Communication between Agents in a MAS for Shape-shifting TUIs

within the environment. In conjunction with this, the

agents are also required to work together in order to

create complex structures; this requires continuous

learning and adaptation. The learning process itself

encompasses remembering and adapting to past

experiences, forgetting irrelevant information, and

incorporating and interpreting new information. A

variety of these aspects have been incorporated into

AI learning algorithms. The degree of complexity and

awareness that is beginning to emerge, is tending

toward the spectrum whereby a system should ideally

begin to operate on its own accord as opposed to

being instructed at each step by a researcher or user.

In relation to the Dod, a question of ethics arose

towards the end of the original study, when exploring

the future developments of the research highlighted a

natural progression for the construction of such

agents to be achieved through biological 3D printing.

Through the study of biomimicry, it is clear that the

biological constructs are still superior to their

artificial counterparts (Kriegman et al., 2020). Even

though powerful advances in material science are

being made with respect to emulating biological

material, the multifaceted nature of biological

material is difficult to reproduce. For example, the

ability of cells to self-repair, to adapt, to decay

without a detrimental effect on the environment, to

metamorphize, the diverse types, etc. Placing this

postulation into context with current research, a

recent study has successfully utilised frog stem cells

to create squishy robots (Kriegman et al., 2020).

Essential cells (early-stage skin and heart cells) have

been removed from one organism and have been used

to create another. Whilst the scientific and

technological developments from this research are

valuable and progressive, does the fact that biological

material is being used, alter moral or ethical

implications? Is there a difference between

reconstituting agents from another existing organism

and growing a new agent from a blank canvas?

Combining the technological advances in AI

algorithms (both logic and emotion), with the

advances in creating artificial agents from biological

material, requires a conscious engagement with the

idea of sentience. For example, instead of developing

an artificial agent with this degree of intelligence, it

may be necessary to consider sacrificing a degree of

autonomy or awareness, in order to avoid ethical

conflicts regarding subjugating sentient agents to a

user’s will.

It is not the aim of this paper to attempt a

definition of sentience but primarily to highlight that

design parameters of technological endeavours

should also consider the consequences of specific

choices, as well as fulfilling the actual goal. For

example, if a new laptop only lasts seven years (after

which its parts may become obsolete) can the

components be reused, or properly recycled? What

impact does it have on people, and the environment?

The gap between what is envisioned for shape-

shifting technology and reality is still very large and

will take more time to reduce. The reason these

ethical or moral questions become relevant is because

they will inevitably influence future designs and

interactions with AI technology. The art of keeping a

machine as a machine, as opposed to emulating a

human brings subjects such as philosophy side by

side science and design. In the attempts to replicate,

physical or mental, aspects of humanity, there is a

striving to understand the intangible elements such as

emotions, the soul, the mind, and phenomenology. In

pure replication there is a risk of not only reproducing

aspects of humanity that function but also the inherent

flaws. As was a guiding principle in the physical

design of the Dod, when developing artificial or new

agents it is necessary to emulate not directly replicate

existing systems (Hasenfuss, 2019).

6 CONCLUSION

The scope of developing new interactive, interfaces is

beginning to extend beyond the traditional hardware,

digital and material science research, and is beginning

to incorporate elements that distinguish organic from

artificial, inorganic agents. This paper has presented

an exploration of communication possible for the Dod

design, based on the physical affordances of the shape

itself. Undertaking the process of creating

fundamental rulesets for a novel agent, brought

concepts of AI, its effects and resulting outcomes into

the foreground of the design process. As designers it

is not only important to fulfil the design brief but it is

the responsibility of designers to consider their design

from as many perspectives as possible: the

advantages, disadvantages and its consequences.

In relation to the existing frameworks used to

create 3D relief or complete 3D shape-shifting

interfaces (Hasenfuss, 2019), a multiagent system

framework appears to be most suitable. A large

proportion of knowledge pertaining to these systems

can and is being obtained from existing biological

MAS. With time and further technological

advancements, it will be a question of moving from

replicating these biological systems, to emulating

them. In the emulation process, elements with are

currently receiving individual attention in interface

research (e.g. physical form, functionality, energy

CHIRA 2020 - 4th International Conference on Computer-Human Interaction Research and Applications

manipulation, communication, etc.), will be

amalgamated in order to create shape-shifting

technology that can fulfil human user requirements.

ACKNOWLEDGEMENTS

I would like to thank my family for their continued

support and my supervisor, Dr. Mikael Fernstrӧm, for

his guidance throughout my study. Thanks also go to

the Irish Research Council for funding the first 3

years of this study (Project ID: GOIPG/2013/351).

REFERENCES

Hasenfuss, H., Reinventing the Cube: an alternative agent

design for shape-shifting technology, in 3rd

International Conference on Computer-Human

Interaction Research and Applications, 2019,

Scitepress: Vienna, p. 15-27.

Ridden, P. Festo unveils robotic ants, butterflies and

chameleon tongue gripper. Gizmag, 2015, [Accessed

14 June 2016], Available from: http://www.

gizmag.com/festo-bionicants-flexshapegripperemotion

butterflies/36765/.

Petersen, K., Nagpal, R., and Werfel, J., Termes: An

autonomous robotic system for three-dimensional

collective construction. Self-organisng Systems

Research Group, 2014.

JPL, Jet Propulsion Laboratory, Gripping foot mechanisms

for anchoring and mobility in microgravity and extreme

terrain. 2012, [Accessed 13 June 2016], Available

from: http://www-robotics.jpl.nasa.gov/tasks/show

Task.cfm?FuseAction=ShowTask&TaskID=206&tdaI

D=700015.

Butera, W.J., Painting the Computer, in Program in Media

Arts and Sciences, 2002, Massachusetts Institute of

Technology. p. 188.

Le Goc, M., Kim, L., Parsaei, A., Fekete, J.D., Dragicevic,

P., Follmer, S., Zooids: Builidng blocks for swarm user

interfaces, in Proceedings of the Symposion on User

Interface Software and Technology (UIST), 2016, New

York.

Özgür, A., Lemaignan, S., Johal, W., Beltran, M., Briod,

M., Pereyre, L., Mondada, F., Dillenbourg, P., Cellulo:

Versatile Handheld Robots for Education, in

ACM/IEEE International Conference on Human-Robot

Interaction, 2017, Vienna, Austria. pp. 119-127.

Romanishin, J.W., et al. 3D M-Blocks: Self-reconfiguring

robots capable of locomotion via pivoting in three

dimensions. in 2015 IEEE International Conference on

Robotics and Automation (ICRA), 2015.

Roudaut, A., et al. Cubimorph: Designing modular

interactive devices. in 2016 IEEE International

Conference on Robotics and Automation (ICRA), 2016.

Rubenstein, M., Cornejo, A., and Nagpal, R.,

Programmable self-assembly in a thousand-robot

swarm. Science, 2014. 345(6198): p. 795.

Lifton, J.H., Pushpin computing: a platform for distributed

sensor networks, in School of Architecture and

Planning, 2002, Massachusettes Institute of

Technology.

Bojinov, H., Casal, A., and Hogg, T., Multiagent control of

self-reconfigurable robots. Artificial Intelligence,

2002. 142.

Uhrmacher, A. and Weyns, D., Multi-agent systems:

simulation and applications. 2009, Boca Raton.

Teich, E.G., et al., Clusters of polyhedra in spherical

confinement. Proceedings of the National Academy of

Sciences, 2016. 113(6): p. E669-E678

Tero, A., Kobayashi, R., and Nakagaki, T., A mathematical

model for adaptive transport network in path finding by

true slime mold. Journal of Theoretical Biology, 2007.

244(4): p. 553-564.

Tummolini, L., Mirolli, M., and Castelfranchi, C.,

Stigmergic cues and their uses in coordination - an

evolutionary approach, in Multi-Agent Systems:

Simulation and Applications, 2009. p. 33.

Nielsen, J. 10 usability heuristics for user interface design.

1995 3 November 2016]; Available from: https://www.

nngroup.com/articles/ten-usability-heuristics/.

Gordon, D. M., Colonial Studies. Boston Review, 2010: p.

59 - 62.

Gorbet, M.G., Orth, M., and Ishii, H., Triangles: tangible

interface for manipulation and exploration of digital

information topography, in Proceedings of the SIGCHI

Conference on Human Factors in Computing

Systems1998, ACM Press/Addison-Wesley Publishing

Co.: Los Angeles, California, USA. p. 49-56.

Lifton, J.H., Broxton, M., and Paradiso, J.A., Distributed

Sensor networks as sensate skin. BT Technology

Journal, 2004. 22(4): p. 32-44.

Nakayasu, A., Himawari: shape memory alloy motion

display for robotic representation, in CHI, Human

Factors in Computing Systems2010: Atlanta, GA, USA.

pp 4327 - 4332.

Richardson, B., et al., Z-Tiles: Building Blocks for Modular,

Pressure-Sensing Floorspace, in Computer Human

Interaction2004, ACM: Vienna, Austria.

Thórisson, K.R., Communicative Humanoids: A

Computational Model of Psychosocial Dialogue Skills,

in School of Architecture and Planning1996,

Massachusetts Institute of Technology.

Hasenfuss, H., A design exploration of an agent template

for multiagent systems (MAS) for shape shifting

tangible user interfaces., in Computer Science and

Information Systems2018, Univesity of Limerick.

Dallas, D., Richard Feynman The World from another point

of view [HD] 2015: Yorkshire Public Television. p.

36:41.

Univerisity of Groningen, Decision making in a complex

and uncertain world, F. Learn, Editor 2016.

Kriegman, S., et al., A scalable pipeline for designing

reconfigurable organisms. Proceedings of the National

Academy of Sciences, 2020. 117(4): p. 1853.

Emerging Complexity: Communication between Agents in a MAS for Shape-shifting TUIs