Multimodal Approach in Advacing the Visual Literacy

Adya Arsita, Kusrini, and Terra Bajraghosa

Institute Seni Indonesia Yogyakarta, Jl. Parangtritis Km 6.5 Sewon Bantul, Yogyakarta, Indonesia

Keywords: Multimodal, Visual Literacy, Graphic Novel.

Abstract: This research aims to introduce multimodality as an analytic tool to analyze visual texts which are rich in

semiotic modes, especially those that juxtapose more than one pictorial forms elaborated with verbal text,

such as in a graphic novel. Graphic novel is chosen as the object of the study because it elaborates drawings,

photographs, other forms of pictures, and verbal texts. This research chose several graphic novels to highlight

that such form of literature is not ‘cheap’ or childish, yet it could promote visual literacy which might have

not been considered as ‘academic’ all along. Literary studies were conducted in compiling some samples and

collecting qualitative data. Samples were purposively selected and narrowed down to five samples to help

finding out the significance of learning multimodality. The semiotic modes discussed include the visual,

verbal, written text, gesture, and even color. While the meta-functions posited to help analyzing a whole text

cover three domains, namely ideational, interpersonal, and textual. The findings will bring the audience to

recognize that using multimodal approach is suitable to understand a visual text thoroughly and hopefully it

will also escalate the use of various visuals as a way of effective communication in enhancing the visual

literacy.

1 INTRODUCTION

Multimodality has been an intriguing topic nowadays

because it has been receiving more attention. People

communicate one to another with several different

modes at the same time. In this context,

communication is considered as an action of

exchanging information from the sender to the

receiver and at the same time also constructing

meaning.

Communication is a way to exchange information

(Danesi, 2004). In everyday life, communication can

expand further when making and delivering the

message, that the message must be spread and finally

it will reach the receiver. Danesi stated, human’s

communication is unique in the way they use more

than one mode (modality) and medium, being

multimodal and multimedia. All human’s senses have

the ability to be the modes of communication. Our

five senses are the modality to experience our

surroundings, to encode, to store, and to construct

meaning from each information received.

In communication, the construction of codes and

signs is critical. The choice and the implementation

of signs will influence how a communication is

conducted and they will create effects that might

appear during the process. That will also include

whether the message is being effectively and

accurately delivered. Kress and van (Leeuwen,2006)

use the term multimodality to explain how the value

of truthfulness and the credibility of a statement are

constructed through a message. Commonly, this will

be deducted from the representation of a particular

social group as communication agents. Therefore, it

is obvious that the role of a message in a

communication process to construct meanings of

reality is essential. So does what happens with visual

communication. The presented visual medium is able

to form an understanding of the world represented in

it.

Visual medium used in communication may vary

from written text, hand drawing picture to

photograph, and even sculptures. One interesting

phenomenon to be observed in visual communication

is the use of various media at once, which could be

referred to as visual multimodality. An example of

media juxtaposing some different visuals is graphic

novel. It is easy to find and it has been passed down

from generations to generations, known as comic.

The ubiquity of visuals encounter in comic or graphic

novel could be considered massive. Graphic novels

are rich in semiotic modes and all is visualized.

Arsita, A., Kusrini, . and Bajraghosa, T.

Multimodal Approach in Advacing the Visual Literacy.

DOI: 10.5220/0008544200140019

In Proceedings of the 1st International Conference on Interdisciplinary Arts and Humanities (ICONARTIES 2019), pages 14-19

ISBN: 978-989-758-450-3

Comprehending the whole text (verbal and pictorial)

as an intact message may not be as easy as it seems

because of its complexity. Due to the name ‘comic’ it

bears, its popularity seems to remain as ‘childish’.

Hence, the term graphic novel would be more

attractive and suitable for a larger scope of audience.

No wonder more publishers and bookstores distribute

more graphic novels for they have been gaining fame

even when they contain science and history. To

generally sum up, there is no huge difference between

comic and graphic novel. Advertisements both on

television and printed actually carry the same

complexity as in graphic novels, but it is only for a

very short time viewing.

Therefore, this paper aims to introduce

multimodality as an analytic tool to analyze visual

texts which are rich in semiotic modes, especially

those that juxtapose more than one pictorial forms

elaborated with verbal text, such as in a graphic novel.

The diverse visuals presented a graphic novel make it

interesting to analyze.

The research is conducted by doing literary

research and desk studies by compiling some samples

and collecting qualitative data. Samples are

purposively selected and narrowed down to five

samples, solely to introduce the application in using

multimodal approach to promote visual literacy and

its importance in making meaning in graphic novels.

2 ADOPTING MULTIMODAL AS

AN APPROACH

Multimodality is comprehended as a part from

communication study and social semiotics. The term

multimodality refers to a communication practice

which involves several different modes at the same

time (Kress & van Leeuwen, 1996). When

communicating, people tend to use more than one

mode, for as we speak to someone, not only the

speech is uttered, but it is also followed by certain

gesture, gaze, and even written text. How people

engage themselves in a communication indicates a

practice of multimodality.

Each mode in multimodal has its own power and

capacity in constructing meaning. As an example, a

communication occurs during a seminar may include

speaking and listening (auditory), eye contact and

body language (gesture), printed papers, slide show

with graphics and pictures, and written text in slide

show (visual). To sum up, such communication is

multimodal (Danesi, 2004). Those modes may show

a great disparity, however none could be regarded as

stronger than or more dominant than the others. Each

mode has its own characteristic to create meaning

partially, yet together they will build a distinct

meaning and somehow people will understand those

chains of modes as ‘a good presentation in a seminar’.

Jewit confirmed that all modes have an important

part in contributing the construction of meaning in

different ways (Jewitt, 2008). As stated by Kress, a

mode is a socially shaped and culturally given

semiotic resource in constructing meaning (Kress,

2010). Therefore, it is considered substantial to

appreciate and understand each resource used in that

process (Jewitt & Kress, 2003).

Aforementioned, a multimodal analysis is derived

from a social semiotics and communication theories,

and it considers that every resource is important in

meaning potential within a distinct and specific

context. To apply multimodality as an approach, a

shift of theorising shall be implied.

2.1 Metafunction

Grammar, which initially was only for anything

related to language and linguistics, it has been

enabled to analyze all visual matters in this visual era.

Visual grammar treats all visual images as if they are

languages. Among all of those different semiotic

modes applied at the same time in a text (graphic

novel as a whole), the text has a broader context. It is

not only in the form of written or verbal form, but

signifies far more than that. The text is all practices

that signifies, being meaningful in the form of images

and even sounds (Barker, 2000).

In relation to social semiotics, the social function

of a text and its operation of the semiotic resources to

make meaning are regarded as carrying three

metafunctions namely ideational / logical,

interpersonal, and textual (Kress & van Leeuwen,

2006). The exploration of those three metafunction

described by Kress and van Leeuwen is derived from

Halliday’s theory of seeing grammar not as a fixed

rule, but more as a resource to deliver meaningful

message (Halliday, 2004). The first is ideational /

logical which is exposing pictures or images as a

representation of empirical aspects experienced by

human being. The second is interpersonal which

observes the interrelation of someone with the others

and amongst all of present semiotic modes. The

amalgamation of each sign or mode to build an intact

meaning is the third metafunction, the textual, which

breathes coherence and relevance with the other two.

Altogether, this process will lead to a finding that

each mode is subjectively motivated and shaped by

certain culturally given resources. In the context of

Multimodal Approach in Advacing the Visual Literacy

visual multimodal, the power of visual resources is

very important in a communication process. Kress

and van Leeuwen revealed that there are two

important things to be considered in a communication

practice, interaction and representation (Kress and

van Leeuwen, 1996). By interaction, it means that the

communication will always need participants who

shall understand each other, showing that

communication is effective. While choosing the

proper and adequate expression in communication

implies the representation.

2.2 Graphic Novel as a Visual

Communication Media

Graphic novel, a ‘new term’ for comic as a medium

for storytelling, can be regarded as a communication

medium. The message in the form of story is based on

an idea or concept to be delivered to the readers.

Comic is a channel to deliver story in a (pictorial)

visual form.

Seen from the point of view of visual

communication design, graphic novel is applying

some graphic design elements such as images

(drawing) and typography, into a particular

composition, to send message to the readers.

The cycle of communication process used as the

communication model in graphic novel is expected to

help understanding its visual communication process

that will enhance the appreciation towards comic or

graphic novel as a media that is able to communicate

various ideas.

In this context, the source as the message’s sender

is the comic artist, merely from one component- they

are the drawing artist and the author, and can be from

many other components. The initial idea and the

objective of the story are originated from the source,

with the story maker (author) as the main component,

to define plot, scenes, actions, as well as to provide

narration and dialogue. The other components are the

comic artists (draftsmen), from the pencillers, inkers,

colorists, letterer, or depending on the work flow, that

will interpret a story into images. Started with

pencillers who will define the layout and

composition, then followed by the inkers to

emphasize the style, and followed by written text

attachment as paralanguage inspired by the author

and at last being colored (if needed) for color will

bring out different meaning in the extension of the

message (Duncan & Smith, 2009). The receivers are

the comic readers, who will send their feedback

through open forums such as discussion group,

articles in newspapers, and even in a more private

way which is by personal letter or emails. Those

feedbacks could easily be found indirectly in social

media or in a discussion session related to comic

publication. On web-based comics (webtoon), the

feedback is directly expressed in the comments

provided by platform of the online comic providers.

The message implied in a comic is presented in

the form of storytelling, with the comic itself as the

channel. There are two types of messages in the

communication of comic, the encoded message (sent

by the source) and the decoded message (received by

the readers (audience). Comic has many codes and

they operate as particular system of general meaning

for the groups of certain culture or culture-derivative

in the form of signs’ organization, consists of signs

and their convention in how codes are combined to

form a far more complex message (Fiske, 2011).

The format of the message in a graphic novel is

the harmonization of comic-style drawing and

photograph or other visual text. A precise example for

that is a graphic novel entitiled ‘The Photographer:

Into War-Torn Afghanistan with Doctors without

Borders’, published in 2011, was made by Didier

Lefѐvre and Emmanuele Guibert. It presents the

readers with the comic-style drawing and black and

white photograph. The juxtaposition of different

images forms a message that could complete one

another. The comic drawings are arranged in series of

panels and the photographs are in frames-like-panels.

Verbal texts and word balloons accompany the

comic, while captions and narrations set to the

photographs, sometimes slightly apart just to make

the photographs remain intact as they are supposed to.

As a centre of phenomenon, comic is able to

collide and to blur the boundaries between verbal text

and pictorial texts (Eisner, 2004). Reading a comic is

a perceptual experience that does not only involve the

decoding of pictorial symbols, but also the linguistics

as well as the inter-animation between images and

words (Duncan & Smith, 2009).

3 ADVANCING VISUAL

LITERACY USING

MULTIMODAL XPLORATION

Defining visual literacy is quite strenuous since it has

not got a specific and fixed term. Ausburn and

Ausburn define visual literacy as a group of skills that

make each person understands and intentionally use

visuals to communicate with the others (Ausburn and

Ausburn, 1978). Considered as the pioneer who

coined the term ‘visual literacy’, Debes, defined

visual language as the intentional use of culturally

ICONARTIES 2019 - 1st International Conference on Interdisciplinary Arts and Humanities

given sign to establish patterns in acquiring

communication (Debes, 1969).

Visual literacy in general is something that

already dwells within our daily life. The term visual

literacy will always remind us that we are all visual

creatures. Visual literacy is basically all about

understanding what you see.

Below is the sample of a graphic novel entitled

‘Maus’ that received a Pulitzer Prize in 1992. That

time was the benchmark of embracing comic book,

also known as graphic novel, as a part of literary

world.

Figure 1: “Maus” by Art Spiegelman http://www.

goodreads.com/book/show/15195.The_Complete_Maus

(accessed on 20th May 2019).

‘Maus’ is special in a way it gives a touch of a

different visual, comic hand-drawings and a portrait.

The presence of the portrait seems to emphasize the

importance of the memory of a mother and a best

friend of the graphic novel artist. He seems to want

telling his readers how those two important persons

mean a lot to him. The juxtaposition of black and

white hand-drawing and the black and white portrait

seem to indicate the sorrow happening in the story.

The irony and metaphor are presented in the

characters, the mice. The story narrated through word

balloons and narration show the representation of the

political and social situation happening at that

moment.

Figure 2: Fumetti entitled ‘Doomlord” (1980s), http://

www.cisforcomics.gr/doomlord/ (accessed on 5th May

2019).

The above fumetti or photonovel was actually

considered to be one of the first photonovels back

then. The collaboration of photographs with the

iconic word balloons as in comic would not be

directly understood as a pure comic form even though

all elements of comic present in that photonovel. The

images look so realistic, however to make them less

realistic, the panels are all in black and white, leaving

the cover alone stays in full color. The convention

format of a comic somehow will lead the readers to

comprehend it as a comic book, in a new way.

Figure 3: Interdependence of verbal text and picture,

supporting a scene. Losing one of it, the meaning of the

story might not be understood. Source: “Ayam Majapahit”

(Ismail:1997).

The collaboration of pictorial and linguistics in a

graphic novel is the combination of images and

written text that enable a graphic novel to be an

effective communication media. The combination is

unnecessarily in balanced, however, the interaction

and the proper stress amongst the elements would

build a good communication (Duncan & Smith,

2009). Words are spread all over the big upper panel

in picture 5, with one image of a seemingly traditional

Multimodal Approach in Advacing the Visual Literacy

house. The simplicity shown in that panel actually

does not offer the same simplicity, as the readers need

to discern the sequence of the word balloons. Failing

to follow the sequence will make that panel losing its

meaning.

Figure 4: Situation in Afghanistan in “The Photographer”.

Source: The Photographer (Guibert, et.al., 2009).

Figure 5: One of the main characters in “The Photographer”

(left:comic version and right: photograph version). Source:

The Photographer (Guibert, et.al., 2009).

The above images, picture 6 and picture 7 show

how the graphic novel drawings are elaborated with

photographs. Picture 7 almost seems redundant by

showing the same image of a man. Yet, those two

visuals with their own characteristic and mode, show

their own strength in making meaning. They

complement each other in visuality. The left panel

with typical comic drawing emphasizes its presence

as a comic, while the right panel with typical black

and white photograph (even without a single word

balloon) define its ubiquity as a photograph carrying

a message that what is drawn on the left panel is

inspired by the real life, a real man. Without

appreciating all visuals, lacking information about the

reality of the story might happen to the readers.

Figure 6: Onomatopoeia representing some sound effects in

a graphic novel. Source: Ophir (Studio Bajing Loncat,

1999) and Kecoa (Yudi, 1998).

Onomatopeia is often regarded as the sound effect

of a comic, in this context it could also be referred as

sound lettering. This iconic presentation is almost as

famous as the word balloons, that once people see it,

will identify the media as a comic or graphic novel.

This onomatopoeia has a strength in narrating the

sequential story or to indicate an event not presented

in the panels (Duncan & Smith, 2009).

Understanding the types of word balloons in

comic is regarded as important, for the shapes and the

thickness of the lines are quite distinctive. Various

types of word balloons defines thoughts, direct

utterance, and even calm talking (Saraceni, 2013).

ucapan

biasa

Bisik-bisik

ucapan

r esmi

mon o l o g

bat in / pikir an

Figure 7: Variations of word balloons. (Bajraghosa, 2012).

4 CONCLUSIONS

In visual communication, multimodality can become

an alternative in the format of a message. However, it

is mandatory to have a good comprehension on verbal

text, images, and other things that cannot be replaced

by thousands of words. It is resulted from the

complexity of the images themselves. The graphic

novel seems unique in its own way, the information

is solid and compact yet there is a hint of ‘freshness’

in it. For those who are not familiar with comic

reading, to understand the combination of text and

visual, could be assisted by the presence of the

photographs. The nature of a photo as the recorder of

reality can be implied as the element of continuity,

which will result in the understanding of the whole

comic story with the messages meant to be delivered.

Another advantage of visual multimodality in

communication is how the message enters the mind

and the perception of the readers. The impression of

ICONARTIES 2019 - 1st International Conference on Interdisciplinary Arts and Humanities

the related images from an event can be more striking

when visual as in picture is present. Hence, readers

unnecessarily imagine for themselves how an event

happens. The stimuli of the images or the visual

sensory dashes because written texts and photographs

or other visuals are attached. Even so, further research

on it will still have to be done.

REFERENCES

Ausburn, Lynna and Floyd B. Ausburn., 1978. “Visual

literacy: background, theory and practice” (Accessed

from https://www.researchgate.net/publication/

234650756_Visual_Literacy_Background_Theory_an

d_Practice, on 4th June 2019)

Bajraghosa, Terra., 2012, Visualisasi bahasa dalam komik

indonesia, Program Pascasarjana Pengkajian Seni ISI

(thesis), Yogyakarta.

Barker, Chris., 2000. Cultural studies: theory and practice.

London: SAGE Publication.

Danesi, Marcel. 2004. Messages, signs, and meaning: a

basic textbook in semiotics and communications theory

(3rd Edition). Toronto: Canadian Scholars’ Press Inc.

Debes, John L. 1969. Some Hows and Whys of Visual

Literacy Educational Screen and Audiovisual Guide,

Vol.48, No.1

Duncan, Randy and Matthew J. Smith., 2009, The power of

comics: history, form, and culture, New York:

Bloomsburry Academy.

Eisner, Will., 2004, Comics and sequential arts, Florida:

Poorhouse Press

Fiske, John. (2011), Introduction to cultural and

communication studies or cultural and communication

studies; Sebuah Pengantar Paling Komprehensif,

Jalasutra: Yogyakarta.

Guibert, Emmanuel., Didier Lefevre, Frederic Lemercier,

2009, The Photographer : Into war-torn afghanistan

with doctors without borders, New York: First Second

Halliday, M.A.K. dan Matthiessen, Christian, M.I.M. 2004.

An introduction to functionnal grammar, 3rd edition.

Great Britain: Arnold.

Ismail, Ahmad Faisal & Kirikomik. 1997. Ayam majapahit,

Jakarta: Balai Pustaka.

Jewitt, Carey. 2008. Multimodality and literacy in school

classrooms (Review for Research in Education), Vol.

32. DOI: 10.3102/0091732X07310586

Jewitt, C., & Kress, G. (Eds.). 2003. Multimodal literacy.

New York: Peter Lang.

http://www.cisforcomics.gr/doomlord/ (accessed on 5th

May 2019)

http://www.goodreads.com/book/show/15195.The_Compl

ete_Maus (accessed on 20th May 2019)

Kress, Gunter and Theo van Leeuwen. 2006. Reading

Images: The Grammar of Visual Design, Routledge:

New York.

Kress, Gunther R. and Theo Van Leeuwen. 2001.

Multimodal Discourse: The Modes And Media Of

Contemporary Communication, London: Cardiff

University.

Saraceni, Mario. 2003. The Language of Comics,

Routledge: New York.

Studio Bajing Loncat.1999. Ophir, Mizan: Bandung

Sulistia, Yudi and Arief Yaddie. 1998. Kecoa, Balai

Pustaka: Jakarta.

Multimodal Approach in Advacing the Visual Literacy