A Study of the Effectiveness of English Speaking of Teachable Agent

using AI Chatbot

Kyung A Lee

, Soon-Bum Lim

and Shankara Narayanan Nagarajan

Dept. of IT Engineering & Research Institute of ICT Convergence, Sookmyung Women's University, Republic of Korea

TATA Technologies Europe Limited, U.K.

Keywords: Artificial Intelligence Chatbot, Virtual Agents, Pedagogical Agents, Teachable Agents, Educational Software.

Abstract: In an environment where English is a foreign language (English as a foreign language: EFL), English learners

use AI voice chatbots for English-speaking practice activities. They enhance their speaking motivation and

provide opportunities for communication practice, ultimately leading to English language learning. It can

improve their speaking skills. However, if they are preschoolers or elementary school students with no

experience learning English, a conversation may not be possible using the AI voice chatbot system. In this

study, we propose a teachable agent using an AI voice chatbot that can be easily used even for elementary

school students and can enhance the learning effect. The existing Teachable Agent is a method that makes

inferences with the knowledge acquired from the learner and answers questions using a path search algorithm.

However, applying the Teachable Agent system to language learning is complex, an activity based on tense,

context, and memory. This paper proposed a new TA method by reflecting the learner's English pronunciation

and level to the teachable agent and generating the agent's answer according to the learner's error. Moreover,

a teachable Agent AI chatbot prototype was implemented with an AI voice chatbot.

1 INTRODUCTION

In an environment where English is a foreign

language (EFL), learners do not have or lack

opportunities to use English in daily communication

and feel psychological pressure to speak English.

Recently, various IT technologies and pedagogical

theories have been combined. In an environment

where English is a foreign language (EFL), learners

do not have or lack opportunities to use English in

daily communication and feel psychological pressure

to speak English. And research is actively conducted

to overcome these environmental factors and increase

the learning effect. There are cases of applying AI

voice chatbot as an educational engineering tool in the

English education field (Sung, M.C., 2020).

It has been found that when AI voice chatbot is

used for English-speaking practice activities. It is

possible to provide optimized learning to individual

learners, increase learners' motivation to speak, and

provide opportunities for communication practice,

ultimately improving their English-speaking ability

(Hwang, Yohan & Lee, Hyejin., 2021).

However, various problems exist, such as AI

chatbots' recognition rate, ambiguous learners'

pronunciation, and lack of target expression learning

(Chu, S. Y., 2021).

Among them, an AI voice chatbot is a system-

driven program in which learners conduct

conversations as guided by the voice bot, and it is

difficult for learners who need to be fluent in English.

If the learner is a preschooler or elementary school

student, a conversation may not be possible using the

AI voice chatbot system. Therefore, in this study, we

will propose an AI voice chatbot that can be easily

used by preschoolers or elementary school students

and can increase the learning effect.

The teaching method is one of the more effective

approaches to learning as part of the Learning-by-

teaching method. In a virtual environment, the learner

plays an active role by teaching a computer agent

called the Teachable Agent (Sandra Y. Okita and

Daniel L. Schwartz, 2013).

The teaching activities in the teachable agent

method allow the instructor to understand and master

the primary learning content more thoroughly. As

repeated training entails analyzing and elaborating the

learning content from various perspectives, in-depth

learning occurs through the teaching process.

The most representative Teachable Agent is

308

Lee, K., Lim, S. and Nagarajan, S.

A Study of the Effectiveness of English Speaking of Teachable Agent using AI Chatbot.

DOI: 10.5220/0011730300003393

In Proceedings of the 15th International Conference on Agents and Artiﬁcial Intelligence (ICAART 2023) - Volume 1, pages 308-314

ISBN: 978-989-758-623-1; ISSN: 2184-433X

 2023 by SCITEPRESS – Science and Technology Publications, Lda. Under CC license (CC BY-NC-ND 4.0)

Betty's Brain. Betty's Brain is a software environment

created by the Teachable Agents Group at Vanderbilt

University to help students advance their

understanding of metacognitive technologies and

enrich their ecosystem knowledge as part of their

science curriculum.

They compared the results of studying in two

ways: second-year middle school students learn

biology through software and learn it on their own and

teach it to a program called 'Betty's Brain'. As a result,

it was found that students study longer when trying to

teach software than when studying for themselves

(Kittaya Leelawong & Gautam Biswas, 2008),

(Taylor, R.P(Ed),1980), (Gautam Biswas1 & James R.

Segedy1 & Kritya Bunchongchit, 2015).

It enables students to learn about science concepts

by implementing a learning-by-teaching paradigm. In

teaching Betty, a virtual student, and testing whether

Betty has learned well, learners can check how much

they know the concept. It has been limited to research

teaching somewhat limited topics (such as conceptual

learning areas of the curriculum) to Teachable Agents.

At the heart of these Teachable Agents is the concept

map approach, that learners teach the Teachable

Agents by drawing and editing concept maps to create

information structures, allowing learners to learn

concepts related to data or science, such as causal

impacts (e.g., ecosystems, climate change or

temperature regulation). Because English learning is

based on tense, memory, and context, not concepts or

causal relationships, it is impossible to learn using the

concept map method (Lee Ingu, 2022) (Nalin

Chhibber, 2019) (Gautam Biswas, Krittaya

Leelawong , Daniel Schwartz , Nancy Vye, 2005).

Language learning should be able to provide

communicative practice close to an actual

conversation through the steps of presentation of the

expression to be learned and repeated practice. It

should provide an opportunity for learners to correct

errors on their own, induce them to use various

expressions, and learn discourse skills.

This study applies the Teachable Agent system to

language learning, an activity based on tense, context,

and memory rather than simple one-to-one

correspondence knowledge.

A new Teachable Agent method was proposed by

reflecting the learner's English pronunciation and

level to the teachable agent and generating the agent's

answer according to the learner's error. And the

Teachable Agent AI chatbot prototype was

implemented.

2 TEACHABLE AGENT MODEL

DESIGN

2.1

Overall Structure of the System

For learners to learn English using artificial

intelligence (AI) voice chatbots, Learners must have

sufficient communication skills to communicate in

English. Non-native languages, especially when using

AI chatbot systems for preschoolers or elementary

school students.

There is a limit to inducing conversations directly

with young learners to make it possible for learners to

practice speaking, and it is difficult for students to

learn voluntarily and continuously. The student's

interest in learning quickly decreases, and the

learning duration is short. To compensate for these

shortcomings, we designed Odinga Agent, a

Teachable Agent-type AI voice chatbot prototype that

allows learners to improve their skills while teaching

the agent.

The overall composition of Odinga Agent is shown

in Figure 1. It is designed to increase learning

efficiency by implementing a character chatbot rather

than a voice chatbot. As the learner's skills improve,

the agent's speaking skills also improve so that free

conversations such as asking or answering questions

to learners are possible.

Figure 1: System configuration.

Odinga Agent consists of 4 modules, as shown in

Table 1.

The Teaching Module is a module in which

learners teach English to an agent. The learner speaks

English first, and the agent follows along.

The Check Module is the part where the learner

listens to and evaluates the agent's utterance, and the

agent utters it by reflecting the learner's utterance as

it is.

The reward module is a function for motivation

for learning, and it provides items or coins as a reward

for the behavior taught by the learner.

A Study of the Effectiveness of English Speaking of Teachable Agent using AI Chatbot

309

The Free Talking Module allows learners to talk

freely with agents.

Table 1: 4 Modules of Odinga Agent.

Module Description

Teaching

Module

- Learners teach English

-The learner speaks first, and the character

speaks afte

Check

Module

-The learner evaluates the character

-Reflect the learner's deficiencies in the

character and make them react.

Reward

Module

-The learner pays the character a reward.

-Learning increases or decreases Odinga IQ

and EQ.

-Odinga learning ability changes according

to IQ and EQ.

Free

Talking

Module

-The learner communicates with the

character

-Free conversation with characters of the

same level as the learner's abilit

2.2 Database for Teachable Agent

Level-specific learning contents for Odinga agents

are designed to expand vocabulary and sentences and

increase application power based on basic sentence

patterns and sentences frequently used in English

conversation. Specifically, it includes patterns and

applied expressions that are essential to know at the

lower grade level of elementary school students in the

United States, interests of children of the same age,

and topics commonly used in school. (see Fig 2).

Figure 2: English Curriculums.

The automatic generation of chatbot sentences used

Natural Language Processing technology, and for this

purpose, 300,000 QA sets, including animation

scripts, were collected. The collected sentences were

labeled based on each sentence's object name

recognition and intention classification results. All

question and response sentences were classified into

a limited number of subjects and generated composed

sentences by level.

The level was set as ARI (The Automated

Readability Index) Score (E.A. Smith, RJ Senter,

1967) for the written sentences. The calculation

formula for level setting is the same as Equation (1).

ቀ

௖௛௔௥௔௖௧௘௥௦

௪௢௥ௗ௦

ቁ ൅ 0.5 ቀ

௪௢௥ௗ௦

௦௘௡௧௘௡௖௘௦

ቁ െ 21.43

(1)

2.3 Teaching Module Design

In the Teaching Module, the learner selects a sentence

to be taught to the Odinga Agent and tells the

corresponding sentence. If a particular score is

reached by evaluating the fluency and accuracy of the

sentence, the Odinga Agent repeats the learner's

utterance. If a particular score is not obtained, the

Odinga Agent says, "umm.." will do. If a particular

score is not reached even after repeating it three times,

the doctor appears and tells the exact sentence

utterance. When the first learning goal is completed,

an item or coin is given as a reward.

2.4 Check Module Design

In the OX feedback system, a function for the learner

to evaluate the character; if the score is less than 70

based on the learner's utterance score, the agent utters

an incorrect answer. However, the incorrect answer is

implemented in 3 types of 0, 1, 2, and 2 utterances in

chunk units. As shown in Figure 3, the learner

evaluates OX by looking at the character's answer,

and if it is X, it is configured to re-utter. If the learner's

incorrect answer continues, it is designed to receive

help from an AI teacher called a doctor. The doctor

was set to motivate learners by listening to the correct

pronunciation and evaluating the learner's

pronunciation. As shown in Figure 3, the learner

utters up to 9 sentences while teaching Odinga Agent

with a learning design to maximize the learner's

speaking frequency and time.

Figure 3: OX Feedback & Iterative learning system.

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

310

2.5 Free Talking Module

When a learner's English proficiency increases to a

certain level, he can free-talk with an Odinga agent.

As shown in Figure 4, when a learner asks a question

to the Odinga agent, Odinga answers at the level of

the sentence taught by the learner. If the learner asks

an Odinga agent a higher-level question than the

sentence taught, the Odinga agent will say, "Sorry.

What was that? Teach me, please." answer and

encourage the learner to teach.

Figure 4: Free Talking Method.

3 ODINGA SYSTEM

3.1 System Configuration

As shown in Figure 5, the system configuration

diagram of Odinga agent proposed in this paper has

changed the function of the general AI voice chatbot's

Dialogue management system to Teachable Agent.

The development environment of this service is

Android Studio, and the code is written in Java. Since

voice recognition and voice output should be possible,

Google Speech API was used. The learner uses the

Speech-To-Text (STT) API to input an English

conversation for learning. The character can answer

using the Text-To-Speech (TTS) API for the

conversation corresponding to the input English text.

Figure 5: Odinga Agent System.

3.2 Dialogue Management System

It recognizes the learner's speech and converts it into

text through STT, creates a sentence waveform, as

shown in Figure 6, and creates the speech waveform

of the Odinga Agent in conjunction with the learner's

waveform.

The learner's utterance is calculated as a learner's

confidence and similarity score. This score is applied

as a weight according to the learner's IQ/EQ to

motivate learning.

According to the learner's utterance score, the

Odinga Agent's correct/incorrect answer is

determined, and the weak part of the learner is created

as the Odinga Agent's incorrect answer.

3.3 Agent Emotion Function

The EQ system with game elements was designed to

generate interest continuously and form bonds

through intellectual growth and the emotional

expression of learners and characters. The IQ & EQ

system set Odinga Agent’s IQ to increase and EQ to

decrease as practice continued, and the learner

provided snacks and gifts to the Odinga Agent with

coins acquired through learning so that the EQ could

be increased.

Figure 6: Agent Feedback.

IQ means the learning level of the Odinga Agent,

EQ represents the emotions of the Odinga Agent, and

IQ does not rise or decrease as learning progresses.

EQ randomly decreases as learning progresses and

increases with items, games, and emotional

exchanges. If the EQ is low, Odinga will refuse to

learn or make mistakes frequently during learning. In

addition, as shown in Figure 7, a reaction according

to the learner's utterance was implemented to add

visual interest.

Figure 7: Agent Reactions.

A Study of the Effectiveness of English Speaking of Teachable Agent using AI Chatbot

311

4 LEARNING EFFECTIVENESS

EVALUATION

4.1 Method

To verify this effectiveness, we evaluated Odinga

Agent satisfaction for teachers and elementary school

students. The subjects who participated in the

evaluation were five English teachers of elementary

schools and 30 students from 3rd grade to 6th grade,

ten each for upper, middle, and lower English

proficiency.

The first questionnaire was an English teacher

group, and the functionality, persistence, and

satisfaction of the Odinga agent were evaluated on a

5-point Likert scale. The questionnaire items are

shown in Table 2.

Table 2: Teachers’ Survey Questions.

Category

Question

Efficiency Is it acceptable to understand learners'

pronunciation and expression?

Sustainability Do students continue to use it?

Satisfaction Do the students find it exciting and fun?

The second survey was conducted with a student

group. The students used the Odinga Agent in class

for 40 minutes once a week for 16 weeks, they

conducted a 5-point satisfaction measurement for the

survey items. The survey items are shown in Table 3.

Table 3: Students’ Survey Questions.

Category Question

Efficiency Does Odinga understand what I say in Eng

lish?

Will the doctor correct me if I speak Engl

ish incorrectl

Sustainabilit

lan to continue usin

Odin

Satisfaction Is it fun to study English using Odinga?

Study Time How much time does it take to study with

Odinga?

4.2 Results

4.2.1 Comparison of Responses Between Groups

As for the evaluation results by groups for Odinga

Agent, the teacher group showed higher satisfaction

than the student group in all three items.

In the evaluation items, both teachers and students

showed the highest intention to continue using it and

showed low scores for satisfaction. (see Fig. 8).

Figure 8: Comparison by Question.

As for the overall average, satisfaction was shown

in the order of teachers > 4th grade > 3rd and 5th

grade > 6th grade (see Fig. 9).

Figure 9: Comparison by Group.

4.2.2 Comparison of Responses by Question

In the Efficient evaluation, the teacher and 4th grade

scored 4.6 points and 4.7 points, respectively. 6th

grade showed the lowest evaluation. (see Fig. 10).

Figure 10: Result of efficient.

In the sustainability evaluation of the intention to

continue using Odinga Agent, teachers and 4th

graders showed high scores, and 3rd graders had the

lowest evaluation (see Fig. 11).

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

312

Figure 11: Result of Sustainability.

In the satisfaction evaluation of whether to continue

using the Odinga Agent when learning English,

teachers and students were satisfied primarily with

similar scores (see Fig.12).

Figure 12: Result of Satisfaction.

4.2.3 Comparison of Responses by English

Level

In the case of the student group, when compared by

English proficiency, the functional evaluation was

highest in the upper group at 4.3, the sustainability

evaluation was highest in the middle group at 4.5, and

the satisfaction evaluation was highest in the lower

group at 4.2(see Fig.13).

Figure 13: Result of English level.

4.2.4 Overall Results

In the evaluation of Odinga Agent, teachers'

evaluation was somewhat higher than that of students.

This result is because using the Odinga Agent allowed

English teachers to solve the difficulties they felt in

the English-speaking class.

Students in the 4th grade who spent the longest

time using Odinga Agent showed high overall

evaluation. According to these results, the learning

effect appears only when a certain amount of learning

time is secured according to the characteristics of

language learning.

The reason for the lower satisfaction of the 3rd

graders compared to the 4th graders seems to be that

they had difficulties in teaching English because the

difficulty of English sentences was somewhat higher

than their level (see Fig.14).

Figure 14: Total Result.

The difficulty of English sentences needs to be

adjusted through the verification of the ARI index and

the verification of English experts.

5 CONCLUSIONS

This study designed a teaching-type AI chatbot

Teachable Agent to activate personalized, level-

specific education and maximize learning effects in

English education.

This system is a Teachable Agent AI chatbot that

improves the speaking ability of the agent as the

learner's skill improves. It was designed to increase

learning efficiency by implementing a character

chatbot rather than a voice chatbot to escape the

boredom of repeated learning and to make it more like

a human conversational environment. As the learner's

skill improves, the agent's speaking ability also

increases, so free conversations such as asking or

answering questions to the learner are possible. It was

designed to increase learning efficiency by

implementing a character chatbot rather than a voice

chatbot to escape the boredom of repeated learning

and to make it more like a human conversational

environment.

In addition, usability evaluation was conducted

A Study of the Effectiveness of English Speaking of Teachable Agent using AI Chatbot

313

for actual elementary school English teachers and

elementary school students, and the effectiveness of

the Teachable Agent AI chatbot was evaluated.

This study is to verify whether it is possible to

apply the Teachable Agent function to the AI voice

chatbot. It can be applied to motivate students or

preschoolers who are not interested in learning

through role switching to participate voluntarily in

learning.

It is possible to improve the immersion of the

educational effect by allowing children to practice

conversation or conversation using their favorite

character as a model.

In the future, it is necessary to evaluate the

effectiveness of the proposed system in more detail by

comparing it with the AI voice chatbot used in the

existing English education field.

ACKNOWLEDGMENT

This research was supported by Basic Science

Research Program through the National Research

Foundation of Korea(NRF) funded by the Ministry of

Education. (2021R1I1A4A01059)

REFERENCES

Sung,M.C. 2020. Artificial intelligence chatbot developed

by elementary school English pre-service teachers.

Language Study, Korea, 56(1), 97-115.

Hwang, Yohan & Lee, Hyejin. 2021. Artificial Intelligence

and its potential in English education: Deployment and

creation of an AI chatbot by pre-service EFL teachers.

Multimedia-Assisted Language Learning, 24(1), 104-

133.

Chu, S. Y. 2021. A study on the development of AI chatbot-

based English picture book retelling teaching model.

Cheongju National University of Education. Korea.

Sandra Y. Okita and Daniel L. Schwartz. 2013. Learning by

Teaching Human Pupils and Teachable Agents: The

Importance of Recursive Feedback. The Journal Of The

Learning Sciences, 22: 375–412, 2013

Kittaya Leelawong & Gautam Biswas. 2008. Designing

Learning by Teaching Agents: The Betty's Brain

System. International Journal of Artificial Intelligence

in Education 18(3):181-208.

E.A. Smith, RJ Senter. 1967. Automated Readability Index.

Aerospace medical research laboratories. Cincinnati

Univ OH

Lee Ingu. 2022. English Speaking Practice System for Non-

native Speakers Based on Reading Comprehension

Checks. Korea University, Korea.

Taylor,R.P(Ed). 1980. The computer in the school: Tutor,

tool, tutee. New York Teachers College Press.

Gautam Biswas1 & James R. Segedy1 & Kritya

Bunchongchit. 2015. From Design to Implementation to

Practice a Learning by Teaching System: Betty’s Brain.

International Journal of Artificial Intelligence in

Education volume 26, pages350–364.

Nalin Chhibber. 2019. Towards the Learning, Perception,

and Effectiveness of Teachable Conversational Agents,

Waterloo, Ontario, Canada

Gautam Biswas, Krittaya Leelawong , Daniel Schwartz ,

Nancy Vye. 2005. Learning by Teaching: A New Agent

Paradigm for Educational Software. Applied Artificial

Intelligence an International Journal Volume 19, 2005 -

Issue 3-4

ICAART 2023 - 15th International Conference on Agents and Artiﬁcial Intelligence

314