A COMPUTATIONAL MODEL FOR VISUAL METAPHORS

Interpreting Creative Visual Advertisements

Angela Schwering, Kai-Uwe Kühnberger, Ulf Krumnack, Helmar Gust, Tonio Wandmacher

Institute of Cognitive Science, University of Osnabrück, Albrechtstr. 28, 49076 Osnabrück, Germany

Bipin Indurkhya and Amitash Ojha

International Institute of Information Technology, Gachibowli, Hyderabad 500 032, Andhra Pradesh, India

Keywords: Visual metaphor, Perceptual metaphor, Creativity, Analogical reasoning.

Abstract: Coming up with new and creative advertisements is a sophisticated task for humans, because creativity

requires breaking conventional associations to create new juxtaposition of familiar objects. Using objects in

an uncommon context attracts the viewer’s attention and is an effective way to communicate a message in

advertisements. Perceptual similarity seems to be a major source for creativity in the domain of visual

metaphors, e.g. replacing objects by perceptually similar, but conceptually different objects is a technique to

create new and unconventional interpretations. In this paper, we analyze the role of perceptual similarity in

advertisements and propose an extension of Heuristic-Driven Theory Projection, a computational theory for

analogy making that can be used to automatically compute interpretations of visual metaphors.

1 INTRODUCTION

Visual metaphors can often be found in

advertisement, caricature, and fine arts (Forceville,

1996; Carroll, 1994; Hausman, 1989). Generating

novel and eye-catching visual metaphors is a highly

sophisticated task requiring creativity, because their

underlying meanings are crucially based on

unconventional conceptualizations and the detection

of new associations. Even interpreting such

metaphors requires creativity (Indurkhya 2007).

Perceptual similarity seems to play a major role

in visual metaphors. Mapping objects of a source to

objects of a target domain based on their common

visual appearance helps to bridge the gap between

incompatible conceptualizations and anchors the

interpretation of their metaphorical relation.

The cognitive mechanism of deliberate

deconceptualization, which is needed in order to

establish a new conceptualization, is a difficult task,

because humans are constrained by conceptual

associations that are learned during lifetime.

Furthermore, it requires a significant amount of

cognitive effort to break away from these

associations. Computers, on the other hand, do not

have such conceptual associations of their own

accord and therefore they can be helpful in finding

and interpreting creative metaphors (Indurkhya

1997). Our aim in this paper is to design

computational systems that can model the process of

interpreting visual metaphors.

The remainder of this paper is structured as

follows: in section 2, we present related work on

modeling creativity in visual analogies and

metaphors. Section 3 exemplifies creativity and

visual metaphors in the domain of advertisements. In

section 4, we introduce “Heuristic-Driven Theory

Projection”, a formal framework developed for

analogy making. We explain how this framework

can be adapted to analyze visual metaphors and

provide a creative interpretation. Section 5 illustrates

the application domain of our approach with several

examples. Section 6 concludes the paper.

2 RELATED WORK

There have been many approaches to modeling

analogies and creativity underlying them.

Hofstadter (1995) persuasively argued that the

processes of generating representations and mapping

are intimately intertwined in creative analogies.

339

Schwering A., Küehnberger K., Krumnack U., Gust H., Wandmacher T., Indurkhya B. and Ojha A. (2009).

A COMPUTATIONAL MODEL FOR VISUAL METAPHORS - Interpreting Creative Visual Advertisements.

In Proceedings of the International Conference on Agents and Artiﬁcial Intelligence, pages 339-344

DOI: 10.5220/0001660303390344

 SciTePress

O’Hara and Indurkhya (1995) modeled the

interaction between representation and mapping in

the domain of geometric analogies. Dastani,

Indurkhya and Scha (2003) proposed an algebraic

model to formalize this interaction in the Copycat

domain of Hofstadter. However, all these

approaches are limited to artificial and rather simple

domains such as letter strings or geometric figures.

These domains have the advantage of being

controllable so that the formal models can be

systematically evaluated, but they do not scale up to

the wide range of examples in ads, art and media.

There have been some studies of cognitive

mechanisms underlying creativity (Gordon, 1961;

Schön, 1963; Rodari, 1996). What they all agree on

is that the key step is to break the conventional

conceptualization of a given object or situation.

Rodari also emphasizes that one needs to get closer

to the perceptual image of the objects and create a

resonance between the images of the source and the

target. Creating strange juxtaposition of familiar

objects, ignoring their conceptual properties and

focusing on perceptual and visual appearance only is

one way to break the existing associations, and

discover new and meaningful interpretations.

Even in language-based metaphors, perceptual

resemblance has often been the basis for

understanding metaphorical expressions. For

instance, the following lines from the poem

Seascape by Stephen Spender:

There are some days the happy ocean lies

Like an unfingered harp, below the land …

Here the metaphorical relation between an ocean and

the unfingered harp can only be established at a

perceptual level, where the sunlight reflected on the

ripples of a calm ocean, making them look like the

strings of a harp. Such synergy of perceptual images

is essential in understanding the meaning of the

poem. This is very difficult, if not impossible, to

obtain by conceptual analysis alone (Indurkhya

1992; Indurkhya et al. 2008).

3 CREATIVITY AND VISUAL

METAPHORS IN ADS

Many visual metaphors rely on perceptual similarity.

Coming up with attractive and effective

advertisements is a difficult and highly creative

process. Figure 1 and additional figures in section 5

show advertisements promoting different products or

ideas. They may serve as examples for how the

perceptual similarity of objects is used to visualize

and communicate a message.

Figure 1: Advertisements for “Clorets”, a chewing gum

that is supposed to help eliminating mouth odors.

Obviously, the visual appearance of objects plays

a major role in the creation of the advertisement

depicted in Figure 1. In the beginning, the associated

objects – tongue and sock – are not similar at all,

although the depicted sock appears in Figure 1

where usually the tongue would be expected. The

perceptual similarity together with the contextual

embedding of the sock in the mouth of a person is

the starting point to establish an association between

two objects, which is moved in a second step to a

conceptual level. The (conceptual) similarity can

only be created by the metaphorical comparison

(Indurkhya 1994). The feature “bad odor” of a

tongue might be (in principle) known before, but it

is new from the cognitive agent’s point of view: it is

newly created in the cognitive agent’s mental

representation of the tongue.

It is important to notice that – based on and

triggered from the visual appearance of the two

initially incomparable objects – a transfer of

properties from the concept “sock” to the concept

“tongue” can be realized that yields a plausible

interpretation of this advertisement. This transfer of

properties is the basis for a creative and non-

conventional interpretation of the advertisement.

4 COMPUTATIONAL MODEL

FOR VISUAL METAPHORS

Metaphors, like analogies, are established via

associating certain elements from the source domain

with elements from the target domain. Via

establishing an alignment between elements, which

at the first sight are not very similar, knowledge

about the elements in the source domain can be

transferred and applied in the target domain and lead

to a new conceptualization of the target domain.

ICAART 2009 - International Conference on Agents and Artificial Intelligence

340

4.1 Heuristic-Driven Theory Projection

Heuristic-Driven Theory Projection (HDTP) is a

formal theory for computing analogical relations

between a source and a target domain. HDTP has a

logical basis: the source and the target domain are

formalized as theories based on a many-sorted first-

order logic. It computes analogies by associating

constants, functions, relations, and (complex)

formulas between target and source domain. Besides

analogies, it was also applied to learning linguistic

metaphors in the domain of technical devices (Gust

et al. 2007). In the following, we explain how HDTP

can be extended to analyze visual metaphors.

HDTP uses anti-unification to identify common

patterns in the source and target domain. Anti-

unification (Plotkin 1970) is a syntactical operation

that compares two terms and identifies the most

specific generalization subsuming both terms. More

precisely, anti-unification of two terms t

and t

can

be interpreted as finding a generalized term t of t

and t

which may contain variables, together with

two substitutions θ

and θ

of variables, such that

tθ

= t

and tθ

= t

. Because there are usually many

possible generalizations, anti-unification tries to find

the most specific one. Based on the classical theory

of anti-unification of terms, HDTP extends this

approach to allow also the anti-unification of

formulas of a first-order logical language

(Krumnack et al. 2007). This results in the

possibility to generalize whole theories of two given

domains in order to generate a structural description

of the underlying commonalities.

Figure 2 shows two examples: f(a) and f(b) are anti-

unified to f(X) where X is a variable replacing the

different arguments of the function. The second

example shows a simple form of second-order anti-

unification: f(a) and g(a) are anti-unified to F(a).

The different function symbols are replaced by a

variable, while the common argument remains.

Given two theories Th

and Th

modeling source

and target domain as input, the HDTP algorithm

computes the analogy between the two domains.

Due to the limited space,

Table 1 roughly sketches

the algorithm. A detailed specification of syntactic,

semantic and algorithmic properties of HDTP can be

found in (Gust et al. 2006; Schwering et al.

accepted).

formulas representing

the source domain

formulas representing

the target domain

generalized formulas common to

source and target

generalized theory Th

Examples

f(a) f(b)

f(X)

f(a) g(a)

F(a)

Figure 2: Establishing the analogical relation between the

source theory Th

and the target theory Th

and

constructing the general theory Th

Table 1: The HDTP Algorithm to compute analogical

relation between a source and a target theory.

Input: A theory Th

of the source domain and a

theory Th

of the target domain represented in a

predicate logic language.

Output: A generalized theory Th

such that the

input theories Th

and Th

can (partially) be

reestablished by substitutions.

Algorithm: Selection and generalization of facts

and rules. Select an axiom from Th

according to a

heuristics h. In HDTP, this heuristics could select

formulas according to their complexity, i.e. prefer

less complex literals to complex rules. Afterwards,

select an axiom from Th

and construct a

generalization (together with corresponding

substitutions).

Optimize the generalization w.r.t. a given

heuristics and update Th

w.r.t. the result of this

process. The heuristics used by HDTP orders the

generalizations according to the complexity of

their substitutions (e.g. length of substitutions).

Transfer (project) facts and laws of Th

to Th

provided they are not generalized yet. Test (using

an oracle) whether the transfer is consistent with

. This can be done via experiments or using

world knowledge in a database.

4.2 Visual Metaphor Formalization

Knowledge about the source and the target domain

must be captured formally to enable a computational

model to analyze metaphors. HDTP is a logical

framework using first-order logic as representational

language. In order to establish a metaphorical

relation between “sock” in the source domain and

“tongue” in the target domain (Figure 1), HDTP

requires a specification of the involved domains.

The main extension to the standard HDTP

formalizations is the distinction in facts referring to

the visual appearance and other conceptual facts that

refer to the non-visual background knowledge.

A COMPUTATIONAL MODEL FOR VISUAL METAPHORS - Interpreting Creative Visual Advertisements

341

We capture the shape at different levels of detail:

at the very basic level we distinguish regions, lines

and points. A line can be further described as being

linear or curved, regularly curved like waves or

irregularly curved. A region can be approximated by

different mathematical attributes like quadratic,

rectangular, circular, and oval. Perceptual similarity

is a multifaceted phenomenon: besides common

shape, it might be caused by common color, texture,

or sometimes by a similar spatial arrangement of

objects. Of course, the simple description of the

appearance in the following tables is incomplete, but

for this introductory example it should suffice.

Table 2: Formalization of the source domain.

Sorts

object:sock,

object:nose

property:bad,

property:region …

Facts referring to visual appearance

shape(sock, region)

in(mouth, sock)

above(nose, mouth)…

Facts referring to conceptual properties

function(sock, keepWarm)

function(sock, provideComfort)

odor(sock, bad) …

Table 2 describes the knowledge about socks: at

the level of visual appearance, a sock has a regional

shape. Furthermore, spatial properties of parts of the

face can be covered, e.g. that the nose is above the

mouth and the sock is in the mouth. Conceptual

background knowledge about socks is crucial,

because certain facts about the source domain need

to be transferred and applied to the target domain

and will provide the creative interpretation of the

metaphor. The background knowledge is usually

very large. The essential information to interpret this

metaphor – the smell of the sock – must be included

in the conceptual facts to come up with the correct

interpretation.

Table 3: Formalization of the target domain.

Sorts

object:tongue,

object:nose

property:region,…

Facts referring to visual appearance

shape(tongue, region)

in(mouth, tongue)

above(nose, mouth) …

Facts referring to conceptual properties

---

A tongue is also described by properties referring

to its visual appearance. The visual appearance can

be rather similar to socks: the tongue also covers a

region that can be approximated by a polygon, and it

has a uniform texture. The tongue is in the mouth.

Furthermore, some visual information about the

context, i.e. the face in which the tongue appears,

may be available. Although humans have much

conceptual knowledge of tongues, the target domain

contains no facts referring to conceptual properties

of the tongue. This is left empty, because existing

conceptual knowledge could only distract from

establishing new creative knowledge. It is necessary

for the deconceptualization which is essential for the

interpretation of the metaphor.

4.3 Computation of Visual Metaphor

The process of analyzing visual metaphors covers

the same steps as the usual analogy-making process

on which HDTP is based: the retrieval of an

appropriate source domain, the mapping of the

analogous elements and the transfer of potentially

meaningful knowledge from the source to the target

domain. The difference between ordinary analogies

and visual metaphors lies in the mapping process: it

is the perceptual similarity between two objects

which causes humans to establish a metaphorical

relation. In visual metaphors, the mapping is based

purely on the visual properties. HDTP restricts the

anti-unification to facts referring to visual

appearance only. Afterwards, in the transfer phase,

HDTP focuses on facts referring to conceptual

background knowledge and transfers non-visual

conceptual properties. Figure 3 illustrates the

process with the “Clorets” advertisements

introduced in section 3. The combination of the face

with a sock as the tongue can be interpreted with an

analogical mapping between sock and tongue.

Figure 3: The visual metaphor can be interpreted via

analogical mapping between a sock and the tongue.

HDTP goes through all facts describing the

visual appearance of the target domain and searches

successively for alignable facts describing the visual

appearance of the source domain. Suitable facts are

2) Transfer

1) Mapping

Source Target

Analogical Mapping

ICAART 2009 - International Conference on Agents and Artificial Intelligence

342

those which can be anti-unified and lead to the most

specific generalization with a minimal set of

substitutions. HDTP re-uses existing substitutions

and tries to minimize the overall number and

complexity of substitutions. In the running example,

the anti-unification process is executed as follows:

The first axiom from the target domain

shape(tongue, region) is aligned with

shape(sock, region) from the source domain

and generalized to

shape(X, region) where the

variable X is substituted by tongue on the target

domain and by sock on the source domain. The next

formula chosen from the target domain for anti-

unification is

in(mouth, tongue). The

counterpart in the source domain is

in(mouth,

sock). In this case, HDTP reuses the already

established substitutions and generalizes both

formulas to the formula

in(mouth, X) where X is

the same variable as before. This process is

continued. The more visual properties can be

mapped, the more perceptually similar are both

objects. The mapping phase is finished if no visual

property is left in the target domain which is not

anti-unified or if no suitable mapping can be created.

The second phase is the transfer of conceptual

knowledge. Conceptual knowledge about socks is

usually quite extensive, but only very few facts

make sense in the context of this metaphor. Function

aspects of socks – e.g. keeping feet warm and

providing comfort – are not applicable to tongues.

However, bad odor of socks is applicable and

therefore a candidate for the analogical transfer.

HDTP transfers every fact referring to the

conceptual properties and checks afterwards for their

applicability. This can be tested by an oracle that

checks the compatibility (consistency, saliency etc.)

of the transfer. Of course, such decisions require a

spelled-out and large database of background

knowledge about the target domain.

5 APPLICATION SCENARIOS

The following pictures show different

advertisements for or against a product or an idea.

Their interpretation originates in some kind of

perceptual similarity. HDTP in the modified version

as described above is a promising approach to

establish a creative interpretation of these visual

metaphors. This approach can be used to

automatically interpret advertisements, but also to

support ad designers to come up with creative ideas.

Source:

Transfer:

Target:

Figure 4: The advertisement associates a mascara wand

applicator with a needle. It calls out on boycott of animal-

tested products.

Figure 4 shows an object which is a combination

of a syringe and a mascara wand applicator. Both

objects share the same overall longish shape, a

cylindrical tube and a spiky top. The object also has

the typical features of the mascara wand applicator

and the needle: the needle of the syringe with the

brush at the top of a mascara wand applicator. Based

on the perceptual similarity, a mapping between the

syringe and the wand applicator is established.

While mascara is associated with beauty, a syringe is

associated with illness or even death. In the

metaphorical interpretation, associated properties of

the syringe are applied to cosmetics.

Figure 5: With this advertisement “Crafted from Nature”

the natural origin of the material is stressed.

Figure 5 shows on the left an advertisement for

cotton shirts: an orange leaf with the shape at the top

resembling a collar of a shirt. The rain pearls on the

leaf representing the freshness and the pure nature

while the association to clothes is only created via

the perceptual similarity. Note that leaves do not

look like shirts in general, but they can be presented

in a particular shape to look similar to shirts. The

leaf characteristics are the arrangement of veins and

the typical autumn color.

Figure 6 shows an advertisement against

smoking. The shape of the “smoke-bag” resembles a

plastic bag. The deathly effect of a plastic bag put

over a head is applied to smoking. The pictures on

the right show smoke of a cigarette and a plastic bag

to illustrate the perceptual similarity. Here again, the

perceptual similarity is on two different levels: while

the material of the bag is made out of smoke (mouth

and the nose of the little boy breathing the smoke),

the shape resembles a plastic bag.

A COMPUTATIONAL MODEL FOR VISUAL METAPHORS - Interpreting Creative Visual Advertisements

343

Figure 6: The advertisement on the left shows a small

child choking on a plastic bag made of smoke. It states

“Smoke isn’t suicide. It’s murder.”

6 CONCLUSIONS AND FUTURE

WORK

Perceptual similarity seems to play a major role in

the generation and interpretation of visual

metaphors: Two conceptually different objects are

associated with each other due to their similar

appearance. Based on this new alignment,

conceptual properties of the source can be

transferred and applied to the target, which enables a

completely new, metaphoric interpretation of the

target. In this paper, we suggest a formal framework

to analyze metaphorical relations: HDTP computes

an interpretation via a mapping based on common

facts describing the visual appearance. Afterwards it

transfers conceptual properties.

Further research shall investigate at a broader level

what influences perceptual similarity and how it can

be formalized. A set of properties describing the

visual appearance will be defined. The domain of

visual ads is suitable for analyzing creativity in

visual metaphors, because it is as challenging as fine

arts, but simpler and better structured. This eases the

evaluation of a computational system. The

interpretation of visual metaphors will be compared

to human interpretations.

REFERENCES

Carroll, N. (1994). Visual Metaphor. In J. Hintikka (ed.)

Aspects of Metaphor, Kluwer Academic Publishers,

Dordrecht, The Netherlands, 1994, pp. 189-218.

M. Dastani, B. Indurkhya, and R. Scha, (2003). An

Algebraic Approach to Modeling Analogical

Projection in Pattern Perception, Journal of

Experimental and Theoretical Artificial Intelligence

15(4): 489-511

Forceville C. (1996). Pictorial Metaphor in Advertising.

London/New York: Routledge.

Gordon, W. J. J. (1961). Synectics: The development of

creative capacity. Harper & Row, New York.

Gust, H., K.-U. Kühnberger and U. Schmid (2006).

Metaphors and heuristic-driven theory projection

(HDTP). Theoretical Computer Science 354(1):98-

117.

Gust, H., K.-U. Kühnberger and U. Schmid (2007).

Ontologies as a cue for the metaphorical meaning of

technical concepts. In A. Schalley and D. Khlentzos

(eds.): Mental States: Evolution, Function, Nature.

Amsterdam/Philadelphia, Benjamins, pp. 191-212.

Hausman, C.R. (1989). Metaphor and Art. Cambridge:

Cambridge University Press.

Hofstadter, D., and The Fluid Analogies Research Group

(1995). Fluid concepts and creative analogies. Basic

Books, New York.

Indurkhya, B. (1992). Metaphor and cognition. Dodrecht,

Kluver.

Indurkhya, B. (1994). A computational perspective on

similarity-creating metaphor. Research in Humanities

Computing. S. Hockey and N. Ide. Oxford, UK,

Clarendon Press. 3:145-162.

Indurkhya, B. (1997). Computers and Creativity.

Unpublished manuscript. Based on the keynote

speech “On Modeling Mechanisms of Creativity,”

delivered at Mind II: Computational Models of

Creative Cognition, 1997. URL:

http://www.iiit.ac.in/~bipin/.

Indurkhya, B. (2007). Creativity in Interpreting Poetic

Metaphors. In T. Kusumi (ed.): New Directions in

Me-taphor Research, Hitsuji Shobo, Tokyo, pp. 483-

501.

Indurkhya, B., K. Kattalay, A. Ojha and P. Tandon (2008).

Experiments with a creativity-support system based on

perceptual similarity. In Proceedings of the 7th

International Conference of Software Methodologies,

Tools and Techniques, Sharjah, United Arab Emirates.

Krumnack, U., A. Schwering, H. Gust and K.-U.

Kühnberger (2007). Restricted higher-order anti-

unification for analogy making. In Proceedings of the

20th Australian Joint Conference on Artificial

Intelligence (AI07), Gold Coast, Australia, Springer.

O'Hara, S. and B. Indurkhya, (1995). “Adaptation and Re-

Description in the Context of Geometric Proportional

Analogies,” AAAI-95 Fall Symposium Series:

Adaptation of Knowledge for Reuse, pp. 80-86.

Plotkin, G. D. (1970). A note on inductive generalization.

Machine Intelligence 5:153-163.

Rodari, G. (1996). The grammar of fantasy. Teachers &

Writers Collaborative, New York.

Schön, D. A. (1963). Displacement of concepts.

Humanities Press, New York.

Schwering, A., U. Krumnack, K.-U. Kühnberger and H.

Gust (accepted). Syntactic Principles of Heuristic-

Driven Theory Projection. Special Issue on Analogies

- Integrating Cognitive Abilities. Journal of Cognitive

Systems Research.

ICAART 2009 - International Conference on Agents and Artificial Intelligence

344