An Augmented Reality Machine Translation Agent
Arbi Haza Nasution
1
, Yoze Rizki
2
, Salhazan Nasution
3
, Rafi Muhammad
1
1
Department of Informatics Engineering, Universitas Islam Riau, Pekanbaru, Indonesia
2
Department of Informatics Engineering, Universitas Muhammadiyah Riau, Pekanbaru, Indonesia
3
Department of Informatics, Universitas Riau, Pekanbaru, Indonesia
Keywords:
Machine Translation, Augmented Reality, Chatbot.
Abstract:
English is a language used as a universal communication tool. Therefore, without English skills, a person will
have a difficulty to communicate properly and correctly in the international scope. This research developed
an application of augmented reality-based translating machine that provides the education to students with
different media in order to increase students’ interest in learning English. This application used library
Vuforia sdk which is able to display 3-dimensional characters with markerless techniques in the form of
augmented reality. The final result of this study was an application that can be used on smartphones with
Android operating system. Based on the results of the application testing, it is concluded that this application
can display 3-dimensional characters in dim light with light intensity of 28 lux at a distance of 10cm-60cm
and viewing angle of 10
◦
-90
◦
. After reviewing the application, 95% of the correspondents stated that this
application is good so it can help students to relearn English outside the school.
1 INTRODUCTION
According to Yamin (2017), the current development
of information technology makes all developing
countries improve the quality of their human
resources as an effort to face global competition.
English is one language that is used as a universal
communication tool in the international scope.
Moreover, Galih et al. (2017) states that
English iscurrently a foreign language introduced in
elementary schools because children aged 6-12 years
have a brilliant learning period called the golden age
(Saputra and Indonesia, 2014; Pangestika et al., 2017;
Mariani and Ananta, 2017).
The learning facilities at school are still
conventional in which teachers deliver the lessons
assisted by textbooks as teaching guides in front
of the class. As a result, this makes students less
interested in the learning process.
This research generates a system in the form
of an attractive English learning tool to increase
children’s learning interest at the school age. This
system translates a text into sound in Indonesian to
English and vice versa. A smartphone is needed
as a medium to run the application. Characters
in 3-dimensional form will translate questions from
users, either words or sentences that have been
previously inputted (Dikdok, 2017; Efendi, 2014).
2 LITERATURE REVIEW
There are several prior works being discussed in this
section. The first study was an implementation
of augmented reality systems conducted by
Yoga Aprillion Saputra, (2014), entitled ”The
Implementation of Augmented Reality (AR) in
Archaeological Fossils at the Bandung Geological
Museum”. The second study becoming the reference
for the language translation process was conducted
by Galih Vidia Pangestika, et al. (2017) entitled
”An Android-Based English Language Learning
Application for Elementary School Students”. The
next research was conducted by Mariani, et al. (2017)
entitled ”The Development of SMS Response and
Phone Call Applications Using Android Text To
Speech and Proximity Sensors for Drivers” as a
reference for the implementation of Text To Speech
method (Mariani and Ananta, 2017; Pangestika et al.,
2017; Saputra and Indonesia, 2014).
Based on the literature reviews of the previous
research, it can be concluded that the creation of
an augmented reality-based machine translation that
utilizes markerless techniques and Vuforia SDK as a
Nasution, A., Rizki, Y., Nasution, S. and Muhammad, R.
An Augmented Reality Machine Translation Agent.
DOI: 10.5220/0009146301630168
In Proceedings of the Second International Conference on Science, Engineering and Technology (ICoSET 2019), pages 163-168
ISBN: 978-989-758-463-3
Copyright
c
 2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved
163

supporting library has never been done.
2.1 Machine Translation
There are three different kind of machine translation.
The rule-based method is a technique that
uses standard language rules in the process of
transliteration (Rahman et al., 2014; Dewantara et al.,
2013). Hansel (2009) states that statistical machine
translation utilizes a machine translation paradigm
in which the translation results are generated on the
basis of statistical models using parameters obtained
from the analysis of the collections of parallel
two-language texts. The neural machine translation
is a new feature of google translate that works by
translating all sentences at once, so the translation
looks more natural, accurate and not weird when it is
read.
In the research of Nasution, et.al (2017), Machine
Translation (MT) is very useful in supporting
multicultural communication. Existing Statistical
Machine Translation (SMT) which requires high
quality and quantity of corpora and Rule-Based
Machine Translation (RBMT) which requires
bilingual dictionaries, morphological, syntax, and
semantic analyzer are scarce for low-resource
languages. Due to the lack of language resources,
it is difficult to create MT from high-resource
languages to low-resource languages like Indonesian
ethnic languages. Nevertheless, Indonesian ethnic
languages’ characteristics motivate us to introduce
a Pivot-Based Hybrid Machine Translation (PHMT)
by combining SMT and RBMT with Indonesian as
a pivot which we further utilize in a multilingual
communication support system(Nasution et al., 2017;
Panggabean, 2016).
2.2 Pivot-based Hybrid Machine
Translation
In the research of Nasution, et.al (2018), Google
Translate service and bilingual dictionary service
were combined as a composite service in the language
grid. There are more than a hundred high-resource
languages available in the Google Translate service.
To this date, two Indonesian ethnic languages, i.e.,
Javanese and Sundanese, are available in Google
Translate service alongside the official language,
Indonesian (Nasution et al., 2018; Nugroho, 2005).
It is unlikely that Google Translate can provide
the rest of Indonesian ethnic languages in the near
future, since the available corpora for Indonesian
ethnic languages are still scarce. In order to
bridge the gap between high-resource languages and
low-resource languages, in this case between English
and Minangkabau, a quicker approach is to create
an English-Minangkabau PHMT with Indonesian
as the pivot. Since Minangkabau has 61.59%
lexical similarity with Indonesian based on ASJP,
the morphology and syntax are similar. Therefore,
Indonesian-Minangkabau word-to-word translation is
expected to be acceptable.
2.3 Language Grid
Toru Ishida (2018) mentioned that globalization
increasingly demands multilingual communication
on the Internet, as well as in local communities.
To create customized collaboration tools to support
multilingual communities, the Language Grid was
established ten years ago. It has been improving
web-based services to communities throughout the
world by providing highly adaptable infrastructure
and access to a wide variety of language resources
and services (Ishida et al., 2018; Nasution et al., 2017;
Nasution, 2018).
3 RESEARCH METHOD
3.1 System Overview
Based on the results of the research analysis, it
can be concluded that the Augmented reality-based
Translating Machine has two criteria. This
Augmented reality-based Translating Machine can
interact with users by translating text from Indonesian
into English and vice versa, and by displaying
sound as the result of translation and animated 3D
characters. Augmented reality-based Translating
Machine is markerless, which means that it does not
use printed markers to display 3D animation models.
Figure 1 explains the bird view of process from
input in the form of text to output in the form of
animation object and speech translation results.
Figure 1: Whole System Overview.
ICoSET 2019 - The Second International Conference on Science, Engineering and Technology
164

3.2 Interactive Words
Interactive is a matter related to two-way
communication or something that is mutually
acting, active and interconnected and has reciprocity
between one another (Warsita, 2008). In this
system, the word “interactive” is classified into two
categories, namely special and general. When a
user types a word in the application, the word will
be matched to the database. If it is in the database,
the 3-dimensional character will say an interactive
word consisted in a special interactive word table
randomly. Otherwise, if the word typed by the user
does not exist in the database, the 3-dimensional
character will utter an interactive word consisted
in a general interactive word table randomly. In
this system, the interactive word consists of two
languages, Indonesian and English.
Examples of general and specific interactive
words can be seen in the following Table 1.
Table 1: Chatbot Corpus
Category Keywords
#Random
Statement
Food
fried rice, meatball,
fried chicken, fried
potatoes, egg
3 for each
keyword
Color
red, yellow, green,
blue, white
3 for each
keyword
Animal
chicken, goat, cow,
cat, dog
3 for each
keyword
Transportation
plane, car, motorcycle,
bike, train
3 for each
keyword
Fruit
grape, apple, banana,
mango, pineapple
3 for each
keyword
General None 5
3.3 Flowchart
In this study, the design of the application used a
flowchart in order to show the workflow done by
the system as a whole. In general, the flows of the
application ofAugmented Reality-Based Translating
Machine were as follows:
The flow diagram of the application of augmented
reality-based Translating Machine can be seen in
Figure 2 and Figure 3.
The flows of the system of an interactive machine
based on augmented reality can be explained as
follows:
1. The user inputs the text.
2. The text is checked in the database.
Figure 2: System flowchart (Augmented reality part).
Figure 3: System flowchart (Language translation part).
3. If the text is in a special interactive database, the
system would produce an interactive word output
in a form of a text.
4. If the text does not exist in a special interactive
database in the previous stage, the system
would access the general interactive database and
generated output from general interactive words
in form of text randomly.
5. The output of interactive words is sent to the text
to speech API to be changed into sound.
6. Character says the word or sentence to the user as
output.
The information about the system flow for
An Augmented Reality Machine Translation Agent
165

the interactive word of Augmented reality-based
Translating Machine can be seen in Figure 4.
Figure 4: Interactive Word System Flowchart.
3.4 How the Application Works
This Augmented reality-based Translating Machine
utilizes a markerless technique, which means that
a marker used to display 3D characters has not
been registered since the application making. The
application will search and mark locations in the
camera area as markers, and the location is listed as
a marker to display the model of 3D characters. An
overview of how the application works can be seen in
Figure 5.
4 RESULTS AND DISCUSSION
The following is the interface of the application of
augmented reality-based machine translation.
Figure 5: Application Interface
Figure 5(a) is a picture before the user presses the
image button and Figure 5(b) is a picture after the user
presses the image button.
In this subchapter, we discuss the results of the
application testing that has been made. Some of the
tests that have been carried out include light intensity
testing, viewing angle testing, distance testing,
markerless detection location testing, translation
testing, and interactive word testing.
4.1 Black Box Testing Scenarios
Black box testing on the application of augmented
reality translating machine was conducted to test each
function of the interface input in the application,
in order to know whether the interface input was
in accordance with the expected output. A black
box testing result shows that all the system designed
match to table 3.1 functionally work as expected.
4.2 Light Intensity Testing
Light intensity testing was conducted inside and
outside the room with different light intensities. This
test was conducted to find out whether the application
of augmented reality translating machine translator
could track and display animated models at different
light sources.
The conclusion of the test on light intensity can be
seen in Table 2.
Table 2: Application test results against light intensity
Test
Case
Light
Intensity
Wait
Time
Result
Test
Results
Daytime
Outdoor
230 lux 1 Second
3D
Character
showed
Success
Outdoor
Night
Day
28 lux 1 Second
3D
Character
showed
Success
Indoor 1130 lux 1 Second
3D
Character
showed
Success
Indoor 322 lux 1 Second
3D
Character
showed
Success
Indoor 0 lux 1 Second
3D
Character
not showed
Not
successful
Based on the results of the light intensity testing
in Table 2, it can be concluded that the application
of machine translators cannot mark the location or
tracking markerless if the light intensity is 0 lux. In
other words, the markerless method in Vuforia did not
require light even if there was little tracking on the
target.
ICoSET 2019 - The Second International Conference on Science, Engineering and Technology
166

4.3 Distance and Angle Testing
The distance and angle testing was done to find out
how far and at what angle the markerless method on
Vuforia sdk displayed the 3D characters. This test was
carried out with bright light. The test was repeated at
a minimum distance of 10cm with an angle of 10
◦
to
the farthest distance of 60cm at an angle of 90
◦
.
The results of testing distance and angle of the
location can be seen in Table 3.
Table 3: Distance and Angle Testing
Action Testing
Result
Test
ResultsDistance Angle
10 cm 10
◦
Character
3D showed
Success
60
◦
Character
3D showed
Success
90
◦
Character
3D showed
Success
20 cm 10
◦
Character
3D showed
Success
60
◦
Character
3D showed
Success
90
◦
Character
3D showed
Success
30 cm 10
◦
Character
3D showed
Success
60
◦
Character
3D showed
Success
90
◦
Character
3D showed
Success
40 cm 10
◦
Character
3D showed
Success
60
◦
Character
3D showed
Success
90
◦
Character
3D showed
Success
50 cm 10
◦
Character
3D showed
Success
60
◦
Character
3D showed
Success
90
◦
Character
3D showed
Success
60 cm 10
◦
Character
3D showed
Success
60
◦
Character
3D showed
Success
90
◦
Character
3D showed
Success
Based on the data of the test results in Table
3, it can be concluded that with a distance
of at least 10cm and an angle of 10, the
application of the translating machine is still ableto
display 3-dimensional characters well, and the
translating machine application is still able to display
3-dimensional characters properly with the furthest
distance testing of 60 cm with a taking angle of
60
◦
and 90
◦
.
4.4 Types of Tracking Object Testing
Testing the types of tracking object with the
markerless method was carried out to find out the best
object or place in marking the location by Vuforia sdk
using the markerless technique. This test was carried
out with 3 types of objects.
The conclusion of the overall results of testing the
types of tracking object can be seen in Table 3. Based
on the testing conducted on the tracking object, it can
be concluded that Vuforia sdk using the markerless
method cannot be used on all tracking object fields
as listed in Table 3. It is because if the object lacks of
image features, the 3D characters will not appear even
though the light and color on the object are sufficient.
4.5 Evaluation
The evaluation was performed by giving
questionnaires to 20 people, in order to find out
the responses from users about the application of
augmented reality-based translating machine. The
results of the evaluation after giving questionnaires
to 20 respondents can be seen in Table 4.
Table 4: Correspondent Percentage
Correspondent Percentage
Excellent Very Good Good Not Good
4 15 1 0
Overall, the results of the questionnaire were
calculated by using the tabulation formula to get
the results of the percentage of each answer to
the questionnaire. Each of these percentages is as
follows:
1. Excellent : 4/20*100% = 20%
2. Very Good : 15/20*100% = 75%
3. Good : 1/20*100% = 5%
4. Not good : 0/20*100% = 0%
5 CONCLUSIONS
The research and the design of the application of
augmented reality-based translating machine have
An Augmented Reality Machine Translation Agent
167

been successfully implemented and a series of tests
have been conducted to test the capabilities of the
application and the following results are obtained.
The application can be used as a reference in learning
word pronunciation and translation from English into
Indonesian and Indonesian into English. However,
it cannot track well if there is no light. It also
cannot display the 3-dimensional characters if there
are few details on the marker. The minimum distance
to obtain good results in displaying 3-dimensional
characters is 10cm from the marked location point.
At a distance of 60cm with taking angles above 10
◦
to
90
◦
, the application still can display 3-dimensional
characters properly. The application can be used both
outdoors and indoors.
REFERENCES
Dewantara, I. M. A. Y., Crisnapati, P. N., Kesiman,
M. W. A., and Darmawiguna, I. G. M. (2013).
Augmented reality book pengenalan gerak dasar tari
bali. Kumpulan Artikel Mahasiswa Pendidikan Teknik
Informatika (KARMAPATI), 2(8):1022–1028.
Dikdok (2017). Naikan kemampuan google translate
hadirkan neural machine translation untuk banyak
bahasa, jurnal apps.
Efendi, I. (2014). Pengertian augmented reality. Diambil
kembali dari it-jurnal: http://www. it-jurnal.
com/2014/05/Pengertian-Augmented-Reality-AR.
html.
Ishida, T., Murakami, Y., Lin, D., Nakaguchi, T., and Otani,
M. (2018). Language service infrastructure on the
web: the language grid. Computer, 51(6):72–81.
Mariani, H. T. and Ananta, M. T. (2017). Pengembangan
aplikasi respons sms dan panggilan telepon
menggunakan android text to speech dan proximity
sensor bagi pengemudi mobil. Jurnal Pengembangan
Teknologi Informasi dan Ilmu Komputer e-ISSN,
2548:964X.
Nasution, A. H. (2018). Pivot-based hybrid machine
translation to support multilingual communication for
closely related languages. World Transactions on
Engineering and Technology Education, 16(2):12–17.
Nasution, A. H., Murakami, Y., and Ishida, T. (2018).
A generalized constraint approach to bilingual
dictionary induction for low-resource language
families. ACM Transactions on Asian and
Low-Resource Language Information Processing
(TALLIP), 17(2):9.
Nasution, A. H., Syafitri, N., Setiawan, P. R., and Suryani,
D. (2017). Pivot-based hybrid machine translation
to support multilingual communication. In 2017
International Conference on Culture and Computing
(Culture and Computing), pages 147–148. IEEE.
Nugroho, A. (2005). Analisis dan perancangan sistem
informasi dengan metodologi berorientasi objek.
Bandung: Informatika.
Pangestika, G. V., Wikusna, W., and Suryadi, A. H. (2017).
Aplikasi pembelajaran bahasa inggris untuk murid
sekolah dasar berbasis android. eProceedings of
Applied Science, 3(3).
Panggabean, H. (2016). Urgensi dan posisi bahasa
inggris di indonesia. Universitas Methodist Indonesia,
Medan.
Rahman, A., Ernawati, and Funny Farady, C. (2014).
Rancang bangun aplikasi informasi universitas
bengkulu sebagai panduan pengenalan kampus
menggunakan metode markerless augmented reality
berbasis android. Jurnal Teknik Informatika, 7(2).
Saputra, Y. A. and Indonesia, T. (2014). Implementasi
augmented reality (ar) pada fosil purbakala di
museum geologi bandung. Jurnal Ilmiah Komputer
dan Informatika (KOMPUTA), 1:1–8.
ICoSET 2019 - The Second International Conference on Science, Engineering and Technology
168