EvaTalk: A Chatbot System for the Brazilian Government Virtual School
Guilherme Guy de Andrade
1
, Geovana Ramos Sousa Silva
1
, Francisco Carlos Molina Duarte J
´
unior
2
,
Giovanni Almeida Santos
1
, F
´
abio L
´
ucio Lopes de Mendonc¸a
1
and Rafael Tim
´
oteo de Sousa J
´
unior
1
1
Electrical Engineering Department, University of Bras
´
ılia, Bras
´
ılia, Brazil
2
National School of Public Administration, Bras
´
ılia, Brazil
Keywords:
Chatbot, Customer Service, Conversational Interface.
Abstract:
EvaTalk is a complete chatbot system developed to attend users from Escola Virtual de Governo (EV.G), which
is a Brazilian virtual school maintained by the federal government. The proposed architecture was based on
a framework to build chatbots, but it was necessary to replace and adapt services to attend EV.G needs. The
architecture is composed of the following modules: Interface for direct interaction; Artificial Intelligence to
comprehend and process messages; Development to deal with the knowledge base; and Business Intelligence
to analyze messages. The first version responded to questions related to Institutional Membership and chitchat.
Still, it was noted that Eva needed more training data considering that the developers could not predict well
user behavior. Therefore, it was necessary to change the conversational data examples and flows to match user
behavior observed after release, which showed an increase in the chatbot’s response confidence. The system
relies mostly on the data collected through the data analysis tools to evolve.
1 INTRODUCTION
Escola Virtual de Governo (EV.G)
1
is a Brazilian vir-
tual school that hosts free and open courses aimed at
public servants in the areas of interest and responsi-
bility of the federal public administration, providing
unification of all government schools and allowing
studies and analysis of the phenomenon of training
in public administration (Teixeira and Pontes, 2017).
EV.G is maintained by the Brazilian federal govern-
ment trough the National School of Public Adminis-
tration (Enap).
The number of enrollments in courses in this plat-
form in 2018 was 442.719, increasing to 940.545 en-
rollments in 2019 according to its open information
dashboard
2
. This growth raises a challenge to the
customer service area. Currently, the support is made
through a ”Contact Us” page, in which the user can fill
a form and have it sent to the support e-mail, where
the staff can answer. Questions are related to pro-
cedures to engage or follow a course or even to be-
come an EV.G partner to provide course material. The
job of the human attendants is to identify the issue
and send the appropriate solution from a collection of
1
https://www.escolavirtual.gov.br/
2
https://emnumeros.escolavirtual.gov.br/indicadores/
default answers, with step-by-step commands. This
approach demands some human effort which grows
together with the platform, which does not have all
the financial resources to have as many attendants as
needed.
To deal with the growth of EV.G and speed up
the user support process, this work proposes a chat-
bot architecture to answer the frequently asked ques-
tions. Chatbots receive natural language from users
and execute one or more related commands to engage
in a conversation, being able to adapt to new informa-
tion or new requests, if it employs machine learning
(Radziwill and Benton, 2017). The proposed imple-
mentation makes use of open-source tools customized
to attend EV.G’s needs.
The development and implementation of a chat-
bot for the EV.G platform is expected to lower the
demand for human support without necessarily extin-
guishing it and solve the most common user doubts.
Also, it will have a good impact on the quality of ser-
vice, because providing an answer quickly have an
impact on the responsiveness factor, which is one of
the variables to measure customer satisfaction (Purna-
masari. et al., 2017). The chatbot’s knowledge-base
will be supplied with the messages previously sent by
users through the ”Contact Us” page and the standard
556
Guy Andrade, G., Silva, G., Duarte Júnior, F., Santos, G., Lopes de Mendonça, F. and Sousa Júnior, R.
EvaTalk: A Chatbot System for the Brazilian Government Virtual School.
DOI: 10.5220/0009418605560562
In Proceedings of the 22nd International Conference on Enterprise Information Systems (ICEIS 2020) - Volume 1, pages 556-562
ISBN: 978-989-758-423-7
Copyright
c
 2020 by SCITEPRESS – Science and Technology Publications, Lda. All rights reserved

responses registered on a document used by the at-
tendants when responding to emails. As a part of the
personification of the chatbot, it was given the name
Eva, and the system is called EvaTalk.
This paper is organized as follows: Section 2
presents other chatbot architectures developed for
Portuguese speaking users; Section 3 raises re-
quirements and specificities of Evatalk; Section 4
shows the designed architecture; Section 5 describes
EvaTalk behaviour with real users; Section 6 proposes
the next steps for improving the system; and Section
7 concludes the paper.
2 RELATED WORKS
Despite relying on recent areas of study, to answer
Portuguese-written questions, some works developed
chatbot systems that were composed mostly by three
layers: user interface, natural language understand-
ing, and a knowledge base. What presents relative
inconsistency are the technologies used, as there are
many tools and algorithms still under development for
this purpose.
Avila et al. (2019) proposed a chatbot architecture
to answer questions related to medicines prices. Med-
iBot is contacted via Telegram and has a custom made
module to understand natural language. It depends on
external resources to load its knowledge base.
Fialho et al. (2013) created a conversational inter-
face system that helps tourists to find answers about
Monserrate. Its learning data demands some effort to
create and maintain because it is defined by XML files
that require long-winded syntax.
Ketsmur et al. (2019) developed a text-to-speech
architecture for a smart home system and Mostac¸o
et al. (2018) proposed one that retrieves information
about agricultural sensor networks. Both are powered
by IBM Watson, which is not open-source software
and has a very limited free version.
However, none of the authors present ways of col-
lecting massive data from the user to understand the
users’ behavior and help the evolving process of the
chatbot by feeding it back collected data. Therefore,
de Lacerda and Aguiar (2019) worked on an open-
source framework that has the basic layers but also
holds a data analytics module.
Even though the architecture shown in Figure 1
and proposed by de Lacerda and Aguiar (2019) is a
complete chatbot system, it was a generalist imple-
mentation. Therefore, this work served as a starting
point to EvaTalk, but it was necessary to modify and
replace some services to create a system suitable for
EV.G customer service.
Figure 1: Chatbot System Architecture Proposed by de Lac-
erda and Aguiar (2019).
3 SYSTEM REQUIREMENTS
Recent studies revealed that users expect chatbots to
behave as humans, but, at the same time, they also
demand much more from it than from a human at-
tendant, so it is important that it is evident for a user
what are the chatbot features and limitations (Jenk-
ins et al., 2007; Følstad et al., 2018). Therefore, Eva
must not pretend to be a human but act as one. Also,
it should provide to the user what are its capabilities,
when convenient in a conversation.
Besides that, due to the emergence of a differ-
ent language in computer-mediated interactions and,
therefore, the occurrence of new discursive genres
online (Giordan and Dotta, 2008), Eva should be
able to understand terms outside standard Portuguese,
as the chat environment implies the occurrence of
shortcuts to speed communication, misspelling and
acronyms (Varnhagen et al., 2010; Al-Sa’Di and
Hamdan, 2005). These behaviors are already notice-
able in emails received by EV.G’S customer service.
The chatbot must be capable of receiving a sentence
in the users’ natural language and produce an ade-
quate response if the subject of the conversation per-
tains to the chatbot’s domain.
Regarding the content that the chatbot must an-
swer, Table 1 lists the subjects that EV.G email re-
ceives questions about. At its final release, it must be
able to answer to all of these subjects. Initially, as a
release strategy, the chat widget will be exclusively
available on the Institutional Membership page
3
of
EV.G. Therefore, it will only answer questions re-
3
https://www.escolavirtual.gov.br/adesao-institucional
EvaTalk: A Chatbot System for the Brazilian Government Virtual School
557

lated to this page and chitchat, making use of data col-
lected through the traditional customer service. This
approach intends to provide a way to evaluate the
architecture and the knowledge-base in an environ-
ment that presents a lower risk. Finally, EvaTalk must
be compatible with EV.G’s visual identity to manage
customer service and have data analysis tools to iden-
tify problems and improve its training.
Table 1: Subjects Answered by EV.G’s Customer Service.
Subject Topics
Account
Sign up
Sign in
Edit personal data
Delete account
Change password
Recover password
Certificates
Issue certificate
Validate certificate
Details
Courses
Course access
Course availability
Anticipate completion
Studies program
Course
Enrollment
Enroll in a course
Proof of enrollment
Validate proof of enrollment
Re-enroll in a course
Cancel enrollment
Institutional
Membership
Categories
Become a partner
Statistics
Course hosting
Partners
Offered services
4 ARCHITECTURE
Figure 2 shows the architecture designed for the
EvaTalk system, which will be detailed later on in this
paper. In addition to the basic layers used in chatbot
systems, it has a layer to run data analysis. These
layers will be called modules and were divided in the
following way: Interface, Artificial Intelligence, De-
velopment and Business Intelligence.
In this architecture, the user interacts directly with
the Interface Module, which receives messages and
sends them to the Artificial Intelligence Module. Be-
fore being handled by the conversational intelligence
tool, the message will pass through middlewares to be
preprocessed. Then, the conversational intelligence
tool sends the message’s response back to the Inter-
face Module, where it is displayed to the user. Both
the message and the response are sent to the Busi-
ness Intelligence Module where they will be indexed
on data analysis tools and stored on a database. It all
happens at running time, but the Development Mod-
ule is modified before running time and it is where the
developers will provide the knowledge-base data col-
lected by the traditional customer service. When the
system is up, this module will serve data to the train-
ing process made by the Artificial Intelligence Mod-
ule.
4.1 Interface Module
An important part of a chatbot is its graphical inter-
face because it is where users have direct interaction
with the system. Some chatbot tools offer the capa-
bility to work with multiple endpoints for user inter-
action for the same bot, at the same time. Although
its a possibility for the future, Evatalk is focused on
using only one endpoint for chat conversations.
During the development of Evatalk, it was ob-
served that some features where important to the user
interface in this specific use case, such as the possibil-
ity to insert buttons on conversations to guide them,
compatibility with the a markup language which en-
ables formatted linking and image viewing.
The alternative chosen for EvaTalk was the use of
a modified version of an open-source software called
WebChat
4
. It has been customized to have visual
identity compatibility with EV.G. The main advan-
tages of WebChat are the ease of customization and
the development of new features. It is also suitable for
chatbots since other tools have more complex func-
tioning to attend multiple channels of communication,
which is not a goal for this work.
4.2 Artificial Intelligence Module
To achieve the expected user experience for Evatalk it
is necessary to let the users express themselves trough
Portuguese-written text. This implies the necessity of
the chatbot being able to understand the users’ natu-
ral language, deal with variations from standard Por-
tuguese language, and maintain a coherent conversa-
tion flow, as per the requirements defined in Section
3.
For a chatbot, two parts are seem as needed for it
to be able to engage in a conversation with the user:
a Natural Language Understanding (NLU) processor
and a dialogue management system. An NLU pro-
cessor is responsible to convert user messages in their
natural language to a machine-readable form, so that
it can be processed in some further steps (Macherey
4
https://github.com/botfront/rasa-webchat
ICEIS 2020 - 22nd International Conference on Enterprise Information Systems
558

Figure 2: Evatalk’s Architecture.
et al., 2001, p.1). A dialogue management system
should receive the NLU processor’s output and pro-
duce a response that fits the context of what the user
said.
There are some tools capable of providing these
components, for Evatalk project the Rasa
5
stack was
chosen. Rasa is an open-source toolkit for building
conversational systems, composed of Rasa NLU and
Rasa Core (Bocklisch et al., 2017), which are its NLU
processor and dialogue management system, respec-
tively. Some factors that contributed to preferring
Rasa are the ease of use due to human-readable train-
ing data formats; pre-defined pipelines for training;
high flexibility in connection with external services;
and an active community that can be useful for trou-
bleshooting.
The choice of the technological stack for a chat-
bot influences in its training data format, because it
needs to be compatible with the technology. Choos-
ing Rasa introduced some concepts such as intents,
stories, utterances, and a domain file. For Rasa, in-
tents are variations of the same sentence that the user
is expected to send in a conversation; stories describe
the conversation flows; utterances are what the chat-
bot can say back to the user; and the domain file act
as an index for all the actions available to the chatbot.
The integration with the Interface Module is made
through a connector. There are standard connectors
to some of the most popular interfaces and messaging
5
https://rasa.com/
services, with input and output channels. However,
to deal with some situations presented in Section 3, it
was necessary to develop a custom connector, which
has an expandable architecture through a middleware
implementation
6
. This middleware processing was
developed to provide greater freedom when perform-
ing text preprocessing when compared to the standard
method included in Rasa, as the middlewares process
messages before they reach the NLU processor. It is
also possible to access and modify data that goes be-
yond the message text, such as session identifiers.
Using the middleware approach, two middlewares
were added to Evatalk. One of them is responsi-
ble for cleaning user messages by removing accents,
punctuation and replacing common web acronyms for
its standard spelling. The list of substitution rules
changes based on the developer’s analysis of the data
collected on user interactions. This middleware re-
quires that NLU training examples follow the same
rules to increase precision. This is not an unexplored
approach since Ferreira. et al. (2017) had to ”remove
and manually correct words or sentences that were
grammatically incorrect” to process the text in his nat-
ural language processing.
Another middleware was implemented to collect
user messages that were sent in a short period. The
dialogue management system default behavior is to
process each message as a First In First Out (FIFO)
queue. This middleware awaits for a new user mes-
6
https://gitlab.evg.gov.br/codigo-aberto/
EvaTalk: A Chatbot System for the Brazilian Government Virtual School
559

sage before passing the collected messages ahead, be
it to the next middleware or the chatbot. If a user
sends two messages in a row and the first one has
punctuation that indicates the end of the sentence, the
middleware sends them separately to be processed.
If the punctuation is not present, the middleware ap-
pends the second message to the first message and
sends them as one.
4.3 Development Module
To manage the chatbot’s content, developers will be
manipulating files that contain response templates,
conversation examples or training data for the NLU
processor. These files require specific formatting that
follows the guidelines defined by the chatbot’s con-
versation management system and NLU processor.
The domain file works as an index for Rasa and is the
one that contains the responses’ templates and maps
all intents, stories, and actions that are described in
other data files. This mapping requires good manage-
ment and effort to keep it synced with the content in
the data files.
To lower the number of errors related to content
formatting, an open-source tool called Rasa Denera-
tor
7
was added to Evatalk’s system. It allows for de-
veloping content without referencing every data item
in the domain file because it will be automatically cre-
ated. As for the responses templates, they are defined
in a separate file and Denerator will take care of ap-
pending them to the domain file generated. Thus, de-
velopers collected question examples from the cus-
tomer service emailbox for each topic of Institucional
Membership and placed at the data folder. The re-
sponses given through email were adapted to fit a chat
environment.
4.4 Business Intelligence Module
Evatalk’s dialogue management system outputs its
conversational data in a specific format and already
has some implemented compatibility layers with
database management systems where it can be saved.
To provide a way to store users conversations, Mon-
goDB
8
was chosen as a database management system
because of its compatibility with other technologies
used in Evatalk and its document-based approach to
saving data.
Data generated by the chatbot contains the user
messages, the chatbot’s responses and the classifica-
tion made by the NLU processor such as the intention
of the user. Collecting and analyzing users’ data is
7
https://pypi.org/project/rasa-denerator/
8
https://www.mongodb.com/
part of the development process of Eva’s content and
EV.G as a platform. Since Evatalk’s system is part of
EV.G’s user support process, platform issues can be
identified through user interactions with the chatbot.
Besides that, the users interactions stored are also
important to add aspects of the users’ natural lan-
guage to data and guide changes in conversational
flows when it is clear that users are not reacting well
to responses or having difficulties following the de-
signed flow.
A data analysis tool was used to allow developers
to see the bigger picture of the interactions between
users and the chatbot. ElastichSearch
9
and Kibana
10
were chosen based on the previous study of de Lac-
erda and Aguiar (2019) which fit the requirements for
Evatalk’s system. These tools allowed the creation of
dashboards with graphics and the extrapolation of the
data collected from interactions.
One difficulty faced with these technological
choices was consuming the data because of the spe-
cific data format of the interactions. To solve that, a
Message Broker was used in conjunction with an evo-
lution
11
of a consumer from de Lacerda and Aguiar
(2019) work. The evolution includes modifications
to make the consumer an independent service, adapt
the data format, collect new data, and add support for
recovering from failures using a database as a restora-
tion point.
5 RESULTS
As expected, the Business Analysis module provided
insights about the first months of the chatbot in pro-
duction. Figure 3 shows the relation between number
of messages and response confidence, weekly.
Initially, as part of initial testing, the chatbot had
only Institutional Membership data and the first users
were mostly professionals that were involved with
EV.G. Therefore, they knew what to ask because they
have been working in the area and, consequently, the
confidence was high. Later, a substantial number of
real users started to interact with the chat widget so
that questions started to deviate from what developers
initially inserted in the knowledge-base.
With real users, confidence started to get low
and the chatbot team had to analyze stored conversa-
tions to understand the user behavior and update the
knowledge-base. Apart from the daily process of in-
cluding user messages that were not comprehended
9
https://www.elastic.co/
10
https://www.elastic.co/products/kibana
11
https://gitlab.evg.gov.br/codigo-aberto/
ICEIS 2020 - 22nd International Conference on Enterprise Information Systems
560

Figure 3: Weekly Number of Messages in Blue and Weekly Average Confidence in Red.
as new training data, because of the decrease in confi-
dence scores, developers made a major change in con-
versational flows and the subjects that the chatbot was
capable of responding.
Conversations showed that, even though the chat
widget was only on the Institutional Membership
page, users tended to ask about other subjects that
were listed in Table 1. Also, users that asked
about Institutional Membership only had an interest
in the main topic, being that ”Become a partner”.
Hence, developers started populating the knowledge-
base with other subjects and their most asked topics,
and also removed the topics that were not popular so
that they would not influence the confidence score of
other questions. These changes happened in the be-
ginning of December 2019 and, as result, confidence
started increase again.
At the end of the year, the chatbot received fewer
messages, as users do not tend to access EV.G as
much in this time of the year. Changes were really
tested at the beginning of 2020, as user interactions
increased and confidence scores maintained a great
average week by week, indicating that conversational
data modification was successful. Still, user interac-
tions keep shaping the knowledge-base day by day,
as we intend to get the confidence score higher and
higher.
6 FUTURE WORKS
The work experience gathered with Evatalk cre-
ation showed the need for a system that allows non-
technical staff to monitor and change the chatbot’s
content. For instance, pedagogues and linguists can
contribute to the creation of content in ways that serve
users more efficiently and also to take care of the lan-
guage used to communicate with users, since the chat-
bot represents a government platform.
Also, since this work implemented a chatbot that
answers simple questions, we hope that it will exe-
cute more complex tasks in the future, like issuing
certificates and help users to find courses that fit their
interests to attend to.
Lastly, future works include the addition of multi-
language support, through automatic translations of
user messages and chatbot content in a translator mid-
dleware. The objective of this middleware is to offer
minimal chatbot support service to users who do not
speak Portuguese.
7 CONCLUSIONS
EvaTalk chatbot system proved to be a promising tool
to lower the demand for human customer service.
EvaTalk: A Chatbot System for the Brazilian Government Virtual School
561

Some issues will still need human assistance, but Eva
can deal with repetitive and mechanical questions,
which are the main problem for EV.G customer ser-
vice. Its first release was important to understand the
complexity of user behavior and the need for evolving
processes, that will require people from many fields of
expertise.
Regarding the development and maintenance of
the entire architecture, the main difficulty is that the
area is constantly evolving and Eva’s development
team must always be prepared to update the tools and
methods used to bring the most humanlike customer
service, that is, the user leaves satisfied with the ser-
vice provided and does not feel the need to communi-
cate with a human. Although the initial content added
to training data was raised by EV.G’s team mem-
bers, Eva’s evolution depends substantially on data
collected from user messages.
ACKNOWLEDGEMENTS
The authors would like to thank the support of
the Brazilian research, development and innova-
tion agencies CAPES (grants 23038.007604/2014-
69 FORTE and 88887.144009/2017-00 PROBRAL),
CNPq (grants 312180/2019-5 PQ-2, BRICS2017-591
LargEWiN, and 465741/2014-2 INCT in Cybersecu-
rity) and FAP-DF (grants 0193.001366/2016 UIoT
and 0193.001365/2016 SSDDC), as well as the co-
operation projects with the Ministry of the Econ-
omy (grants DIPLA 005/2016 and ENAP 083/2016),
the Institutional Security Office of the Presidency of
the Republic (grant ABIN 002/2017), the Adminis-
trative Council for Economic Defense (grant CADE
08700.000047/2019-14) and the General Attorney of
the Union (grant AGU 697.935/2019).
REFERENCES
Al-Sa’Di, R. A. and Hamdan, J. M. (2005). “synchronous
online chat” english: Computer-mediated communi-
cation. World Englishes, 24(4):409–424.
Avila, C., Calixto, A., Rolim, T., Franco, W., Venceslau, A.,
Vidal, V., Pequeno, V., and Moura, F. (2019). Medi-
bot: An ontology based chatbot for portuguese speak-
ers drug’s users. In International Conference on En-
terprise Information Systems, volume 1, pages 25–36.
SciTePress.
Bocklisch, T., Faulkner, J., Pawlowski, N., and Nichol,
A. (2017). Rasa: Open source language under-
standing and dialogue management. arXiv preprint
arXiv:1712.05181.
de Lacerda, A. R. and Aguiar, C. S. (2019). Floss faq chat-
bot project reuse: how to allow nonexperts to develop
a chatbot. In Proceedings of the 15th International
Symposium on Open Collaboration, page 3. ACM.
Ferreira., R. C. B., Thom., L. H., and Fantinato., M. (2017).
A semi-automatic approach to identify business pro-
cess elements in natural language texts. In Proceed-
ings of the 19th International Conference on Enter-
prise Information Systems - Volume 1: ICEIS,, pages
250–261. INSTICC, SciTePress.
Fialho, P., Coheur, L., Curto, S., Cl
´
audio, P., Costa,
ˆ
A.,
Abad, A., Meinedo, H., and Trancoso, I. (2013). Meet
edgar, a tutoring agent at monserrate. In Proceed-
ings of the 51st Annual Meeting of the Association for
Computational Linguistics: System Demonstrations,
pages 61–66.
Følstad, A., Nordheim, C. B., and Bjørkli, C. A. (2018).
What makes users trust a chatbot for customer ser-
vice? an exploratory interview study. In Interna-
tional Conference on Internet Science, pages 194–
208. Springer.
Giordan, M. and Dotta, S. (2008). Estudo das interac¸
˜
oes
mediadas por um servic¸o de tutoria pela internet. Re-
vista Linguagem & Ensino, 11(1):127–143.
Jenkins, M.-C., Churchill, R., Cox, S., and Smith, D.
(2007). Analysis of user interaction with service ori-
ented chatbot systems. In International Conference on
Human-Computer Interaction, pages 76–83. Springer.
Ketsmur, M., Teixeira, A., Almeida, N., and Silva, S.
(2019). Towards european portuguese conversational
assistants for smart homes. In 8th Symposium on
Languages, Applications and Technologies (SLATE
2019). Schloss Dagstuhl-Leibniz-Zentrum fuer Infor-
matik.
Macherey, K., Och, F. J., and Ney, H. (2001). Natural lan-
guage understanding using statistical machine transla-
tion. In INTERSPEECH.
Mostac¸o, G. M., De Souza,
´
I. R. C., Campos, L. B., and
Cugnasca, C. E. (2018). Agronomobot: a smart an-
swering chatbot applied to agricultural sensor net-
works. In 14th international conference on precision
agriculture, volume 24, pages 1–13.
Purnamasari., P., Manda., G. S., and Santoso., A. B. (2017).
The influence of service quality towards costumer sat-
isfaction. In Proceedings of the 2nd Global Confer-
ence on Business, Management and Entrepreneurship
- Volume 1: GC-BME,, pages 214–220. INSTICC,
SciTePress.
Radziwill, N. M. and Benton, M. C. (2017). Evaluat-
ing quality of chatbots and intelligent conversational
agents. arXiv preprint arXiv:1704.04579.
Teixeira, N. and Pontes, B. (2017). Escola Virtual do Gov-
erno Federal: proposta de soluc¸
˜
ao para a capacitac¸
˜
ao
continuada a dist
ˆ
ancia no servic¸o p
´
ublico federal
[Federal Government Virtual School: proposal for a
solution for continuing distance training in the federal
public service]. In XXII Congreso Internacional del
CLAD sobre la Reforma del Estado y de la Adminis-
traci
´
on P
´
ublica. CLAD.
Varnhagen, C. K., McFall, G. P., Pugh, N., Routledge, L.,
Sumida-MacDonald, H., and Kwong, T. E. (2010).
Lol: New language and spelling in instant messaging.
Reading and writing, 23(6):719–733.
ICEIS 2020 - 22nd International Conference on Enterprise Information Systems
562