Mobile Applications as a Means for the Diagnostics and Speech

Development in Children with Speech Disturbances and as a

Resource for Alternative Communication

Natalya Snytnikova, Liudmila Mozheikina and Mariya Timofeeva

Department of Fundamental and Applied Linguistics, Novosibirsk State University,

1 Pirogova Street, Novosibirsk, Russian Federation

Keywords: Mobile Applications, Speech Diagnostics, Speech Development, Specific Language Impairment,

Alternative Communication.

Abstract: This paper presents three mobile application systems which have been designed and developed at the

Department of Fundamental and Applied Linguistics of Novosibirsk State University, Russian Federation.

These mobile applications are “Logoball”, “Developing Speech”, and “PictInteraction”. The diagnostic

system “Logoball” aims at detecting speech defects in children having different language problems. The

programme “Developing Speech” is intended to be used for the development of impressive speech in

children with the specific language impairment. The mobile application “PictInteraction” deals with the

development of coherent speech in children with speech disturbances. It also serves as a means of

alternative communication for such children. The mobile applications are described in some detail. The

paper briefly discusses the strengths and weaknesses of the applications as well as the opportunities they

provide to the children with language problems, to their parents, and to the speech and language therapists

who work with the children having language problems.

1 INTRODUCTION

The paper studies the experience of creating

comprehensive software systems for the diagnostics,

development and rehabilitation of the linguistic

competence in children with various kinds of speech

disturbances. The electronic means of alternative

communication for such children are also presented.

The kinds of speech disturbances vary from the

specific language impairment to the autism spectrum

disorders. The former include language problems at

different speech levels:

1) when the produced speech is not coherent;

2) when the lexical-grammatical composition of

speech is not consecutive;

3) when there is no phonetic or phonemic

understanding;

4) when the verbalization of ideas in general

does not happen.

The latter comprises the cases in which the

communication with the external world is

compromised and a complete refusal from the social

interaction can take place.

The approach to the development of mobile

applications that can help to both diagnose the

language problems and cultivate the linguistic

competence is highly important and challenging.

There are only few mobile applications in Russian

similar to the ones described in this paper. None of

such mobile applications have been recorded so far

on the Records of the Federal Institute for Industrial

Property, on the Records of the Fund of Algorithms

and Programs of the Siberian Branch of the Russian

Academy of Sciences, on the FREEPATENT

website, or in the Registry of the Intellectual

Property FindPatent.ru.

Nowadays speech and language therapists or

teacher-logopedists do not often use both electronic

technologies and mobile learning technologies in

their practice. They usually employ old-fashioned

paper-based systems that are bulky, take a good deal

of space in the therapist’s office and require a lot of

time-consuming work to be prepared. The

difficulties of using the paper-based stimulus

materials are especially noticeable in the diagnostics

of speech development.

466

Snytnikova, N., Mozheikina, L. and Timofeeva, M.

Mobile Applications as a Means for the Diagnostics and Speech Development in Children with Speech Disturbances and as a Resource for Alternative Communication.

DOI: 10.5220/0006794804660473

In Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pages 466-473

ISBN: 978-989-758-291-2

In the course of diagnostics the diagnostic

materials are selected in manual way using a variety

of textbooks with logopedic tests and illustrations.

The results of the diagnostics and the speech errors

are entered in the form. Then they are carried over to

the electronic spread sheets for the further

construction of charts and for the comparison of

results. All this takes plenty of time that the speech

and language therapists could spend much more

productively working with children.

Meanwhile, a mobile application system will

allow rapid and easy access to the well-organized

and structured information. It will also carry out

automatic processing of the results of the diagnostics

or of the results of a particular therapeutic session.

Several years ago a new idea started up at the

Department of Fundamental and Applied Linguistics

of Novosibirsk State University. The idea was about

creating a multi-subject educational project that

would bring together fundamental and applied

research in the fields of applied linguistics, computer

linguistics, psycholinguistics, speech and language

therapy, and cognitive psychology. The project has

resulted in the creation of real electronic systems for

the diagnostics of speech development and for the

development of speech in children with specific

health capabilities, particularly in children with

various kinds of speech disturbances.

Three mobile application systems (=

programmes) in Russian have been elaborated and

tried out thus far. They are the diagnostic

programme “Logoball”, the mobile application

“Developing Speech”, and the mobile application

“PictInteraction”. These programmes are being put

into the speech and language therapy practice

currently.

2 MOBILE APPLICATIONS FOR

THE SPEECH DIAGNOSTICS,

SPEECH DEVELOPMENT, AND

ALTERNATIVE

COMMUNICATION

The three mobile applications in question have been

created since 2014. The diagnostic programme

“Logoball” appeared in 2017. It is intended for the

speech diagnostics in children having different

language problems. It helps to detect speech defects

in children faster and more effectively in comparison

with the traditional methods.

Unfortunately, the parents do not always have an

opportunity to consult a speech and language

therapist. In such instance they can use textbooks

with logopedic games and tasks as well as the

developmental applications for the mobile devices.

The mobile applications are more effective than

paper-based textbooks due to their interactivity

which encourages the involvement of a child in the

gaming and logopedic experience.

The mobile application “Developing Speech”

was made in 2017. It helps to develop the impressive

speech in children with the specific language

impairment. Children and their parents can use this

programme independently, without any participation

of specialists in the field of speech and language

therapy.

The mobile application “PictInteraction” was

created in 2015. It is supposed to develop the

coherent speech in children with speech

disturbances. And it can also be used as a means of

alternative communication by such children.

2.1 Diagnostic Programme “Logoball”

This mobile application allows us to speed up the

process of detecting the speech defects in children. It

became possible as the diagnostic process was

computerized. The following actions were

automated:

1) posting the diagnostic results into the

electronic spread sheets;

2) setting up the diagrams;

3) comparing the diagnostic results at various

stages of the developmental teaching;

4) retaining the lists of errors made by the

children during the process of diagnostics.

The “Logoball” application is structured after

the diagnostics procedure created by N.V.

Nishcheva (Chirkina, 2003). The given structure

consists of nine series and of 33 groups of tasks. The

diagnostic series are as follows:

1) the assignments for the perception of audio-

syllabic word structure;

2) the assignments for the phonemic perception;

3) the assignments for the phonemic analysis

and synthesis;

4) the assignments for the comprehension of

speech in general;

5) the assignments for the comprehension of

word combinations and simple sentences;

6) the assignments for the assessment of the

lexical-grammatical formedness of the child’s

speech as well as of the formedness of the

child’s skills in the inflexion, word-building,

and coherent speech.

Mobile Applications as a Means for the Diagnostics and Speech Development in Children with Speech Disturbances and as a Resource for

Alternative Communication

467

The application is meant to be used on a

notebook computer operating under Windows 7, 8 or

10. The written programming code is divided into

two separate files in order to make the code more

readable and user-friendly. Thus, not only the author

but also anyone who wishes could read the

programme.

The first file comprises the functions, classes and

cycles of the programme. The second file includes

the descriptions of widgets of the programme such

as the buttons, images and labels.

There was a particular problem to be handled,

the problem of preserving the results. It was solved

with the help of one of the Python language

modules, the CSV module. The given module allows

both reading from a CSV format file and writing

some values into a CSV format file. The values in

such a file can be separated with a comma, a semi-

colon, or tabulation.

The computer programming language Python 3.4

has been utilized to develop the application. The

application programming code includes some data

structures such as List, Tuple, Dictionary, and Set.

Let us describe the interface of the programme as

seen by the speech and language therapist and the

child. The diagnostic stimulus materials used in the

tasks are found in two types: 1) materials without

illustrations; 2) materials with illustrations.

If there is a task of the first type on the screen,

the light-yellow buttons appear with the words, word

combinations, or sentences written on them. They

are the words, word combinations, and sentences the

child is going to use while fulfilling the task. In case

a child makes an error responding to a particular

word, the button topped with this word should be

pressed in order to preserve this error in a special

electronic spreadsheet containing errors. An

electronic “piggy box” (“safe”) with the child’s

errors is very important as it helps the speech and

language therapist to set up an individual course of

further developing work for each child.

If there is a task of the second type on the screen,

the illustrations appear that are to be used by a child

in the course of fulfilling the task. At the bottom of

the screen the light-yellow buttons are located

topped with the picture captions. Should a child

mispress such a button, the error is also preserved in

the list of errors made by a child in the course of

fulfilling the task.

During the performance of all the tasks there is

the name of the section and the name of the

diagnostic task specified at the top of the screen. On

the right-hand side of the screen there are the four

buttons topped with the scores from 0 (zero) to 3

(three). They are used to grade the child’s work.

Having pressed the button with the score the

therapist automatically goes on to the next task.

There is no possibility to undo anything. The task-

performance time is not limited depending on the

child’s productivity alone.

Below there are two pictures (Figures 1 and 2)

describing the tasks as seen on the screen by both

the child and the speech and language therapist.

Figure 1: The screenshot of a task without illustrations.

Figure 1 shows a task without illustrations. This

is Task 2.3. “Differentiating by ear the sounds which

are confused during pronunciation.” It is one of the

tasks from Section 2 “Phonemic awareness

(audiolingual differentiation of sounds).”

At the top of the screenshot the following

captions in Russian are written: 2. “Phonemic

awareness (audiolingual differentiation of sounds).

2.3. Differentiating by ear the sounds which are

confused during pronunciation.” On the screen the

following word pairs in Russian are written: Mouse-

Bowl, Rat-Roof, River-Radish, Fringe-Slot, Wind-

Evening, Goats-Plaits.

Task 2.3 is meant to teach children how to

distinguish between the sounds which are confused

in the process of pronunciation. The task is

performed in the following way: the speech therapist

pronounces the pairs of words with confusing

sounds, and the child repeats them after the

therapist.

The words in pairs are similar in sounding, but

they are different in meaning. The word pairs are

displayed in the yellow boxes on the screen. If the

child does not differentiate the phonemes and

repeats the word pairs improperly, the speech

therapist asks clarification questions. These

questions help to clear up whether the child

distinguishes the meanings and the disturbance is

only about pronunciation. Or the meanings of the

words are confused and the disturbance is about

comprehension. It is crucial for establishing the

logopedic diagnosis.

CSEDU 2018 - 10th International Conference on Computer Supported Education

468

The speech therapist marks by mouse clicks all

the word pairs which the child pronounced

improperly. They are automatically recorded in a

special electronic database (“piggy box” or “safe”).

This database can be used during the process of

training when the speech therapist would work with

the words which were pronounced incorrectly during

the process of diagnostics.

Figure 2: The screenshot of a task with illustrations.

Figure 2 shows a task with illustrations. This is

Task 7.3. “Noun Adjective Agreement in the

singular.” It is meant to check the child’s skill to

make agree the adjectives and nouns in the singular

(e.g., green bucket, red flag, blue pencil, etc).

At the top of the screenshot the following

captions in Russian are written: “7. Inflexion. 7.3.

Noun Adjective Agreement in the singular.” At the

bottom there are the following words in Russian:

Flower, Flag, Bucket, Pencil, Saucepan, Leaf.

The speech therapist screens the picture and asks

questions like “What colour is the bucket?”, “In

what bucket shall we pour water?” etc. The child

should demonstrate the skill of Noun Adjective

Agreement.

After the child has fulfilled the task, the speech

therapist gives him/her a grade by pressing one of

the blue buttons on the right side of the screen.

We recommend the following method of

allocation of grades. Grade “0” means that the child

has not coped with the task. Grade “1” means that

the child has fulfilled about 30% of the task. Grade

“2” means that the child has fulfilled from 31% to

70% of the task. Grade “3” means that the child has

fulfilled the task completely or almost completely.

For further development we are going to do the

following things:

• the improvement of illustrations (so that the

children could perceive the interface better);

• the selection of tasks with various levels of

complexity (so that it would be possible to

differentiate between the children having

diverse levels of development and grade the

tasks for them accordingly);

• the addition of gaming blocks to the script (this

can help to enhance the child’s motivation

when tested);

• the introduction of such operations as “Step

backward” and “Pause” (so that it would be

more convenient for the speech and language

therapist to arrange the test procedure);

• the attachment of a work manual to every

diagnostics task (so that the programme could

be used by anyone who feels like it and not

only by the specialists).

A group of 22 experts – speech and language

therapists – were recruited in order to get a feedback

concerning the assessment of the developed

application. The questionnaire was made up. It was

meant to identify the good and bad points of the

programme.

The results of the questionnaire survey indicated

that the good points are the stability and the

operating speed of the application as well as the

congruency of the diagnostic content to the original,

i.e. the diagnostic procedure created by N.V.

Nishcheva mentioned above. The criterion “Quality

of Visualization” got the lowest mark. The work is

now being undertaken to improve this part of the

Logoball diagnostic application.

2.2 Mobile Application “Developing

Speech”

The mobile application “Developing Speech” has

been designed to be used for the development of

impressive speech in children with the specific

language impairment of the 1-st, 2-nd, and 3-rd

levels of speech development.

The children can use the application in order to

fulfill various logopedic tasks independently. There

are 10 tasks at every level which are to be performed

by a child-user. The task performance time is not

limited. The texts of tasks and the actions of the user

are accompanied by sound. After having fulfilled the

tasks the child-user has the possibility to get to know

the results.

The computer programming language Python

with the Kivy library was utilized to develop the

application. The application was made for the tablet

computers (Android operational system).

When the mobile application “Developing

speech” is started, the user gets oneself to the Menu

of the programme. There are three buttons for the

selection of task level and the “Exit” button on the

screen. Having selected one of the levels the user is

requested to perform 10 tasks in succession.

Mobile Applications as a Means for the Diagnostics and Speech Development in Children with Speech Disturbances and as a Resource for

Alternative Communication

469

Every task is located in a new application

window. In the centre of each screen there is a text

containing the task. Above the text there is the play

button used for reproducing the text of the task. That

is the application provides a voice for the task. The

play button can be pressed an unlimited number of

times. So the child can listen to the task as many

times as he/she needs.

A typical task screen displays from one to four

variants of response which can be selected. Each

response button can be selected and pressed just

once. Pressing the response button is accompanied

by an acoustic signal. The picture proposed for the

task changes simultaneously. At the bottom of the

screen there is a task changeover key.

For example, there is a task that examines

whether a child understands and differentiates

prepositions. The child can see several resembling

pictures on the screen. They are: “The kitten is on

the chair”, “The kitten is behind the chair”,

“The

kitten is under the chair”, etc. The child is offered to

press the button with the task. Then the female voice

delivers the question: “Show the picture where the

kitten is on the chair, please.” The child is supposed

to choose the right picture and press the relevant

button. If the task has been fulfilled correctly, he/she

hears “Correct, good for you!” If a mistake has been

made, then he/she hears “Try again, please.”

Some of the tasks can be skipped. The user can

pass on to one of the next tasks of the same level.

The results of the missing exercises are not taken

into account.

After having fulfilled 10 tasks the user presses

the button situated at the bottom of the screen and

gets into the screen containing the final result. In

order to see the result it is necessary to press the

button “Refresh display” situated at the top of the

screen. At the bottom of the screen the button

appears that can be pushed to get to the Menu.

After the fulfillment of any level it is possible to

select this level again. It allows the user to review

the performed tasks once more and to complete the

fulfillment of the skipped tasks. The “Exit” button

in the Menu ends the work of the programme.

The mobile application “Developing Speech”

has been assessed by 20 teacher-logopedists from

the child care centres located in different districts of

Novosibirsk. The assessment has been made using

the following criteria: 1) quality; 2) functionality; 3)

stability; 4) reliability; and 5) accessibility. Five-

point scale has been used. The teachers have

provided the feedback together with their

impressions and suggestions.

On the whole, the new mobile application

“Developing Speech” has obtained the positive

feedback from the professional teacher-logopedists.

They have suggested that some parts of the

programme are to be improved. First, the sound

tracks are to be refined. Second, the sizes of some

images are to be changed. Third, the tasks are to be

made more varied. Fourth, the system of rewards

should be significantly ameliorated.

At the same time, the teachers have specified that

the interface of the programme and the tasks are

adequately elaborated. The respondents are equally

of the opinion that the usage of such mobile

applications together with the speech and language

therapy sessions will allow the children to develop

their speech more effectively.

The experts have estimated positively such

criteria as the functionality, stability, and the

operation speed of the programme. But the

application has also got some weaknesses:

1) there is no such an option as “turning the

screen angle”;

2) the application flies out of control after the

automatic switch-off of the screen;

3) at the start of the programme the files are

loaded rather long – for about 30 seconds.

These weaknesses are to be taken care of as they

can influence the concentration of the child during

the performance of the task and his/her involvement

into the gaming experience.

The mobile application “Developing Speech”

has great potential as it can be used for various kinds

of speech disturbances. In view of this we intend to

select several levels of complexity of tasks and to

specificate the tasks designed for different speech

disturbances.

2.3 Mobile Application

“PictInteraction”

This mobile application system is intended to

develop the coherent speech in children with speech

disturbances. It also allows them to communicate

with the help of the language of pictograms and their

colour layout. One and the same pictogram can

specify an action (if it is red), an object (if it is blue),

or an attribute (if it is green). For example, to smile,

a smile, cheerful. The black colour is used to specify

the function words.

The colour-coded indication helps to turn correct

phrases. A child is able to make up both simple and

compound sentences by arranging the pictograms in

a certain order. This is one of the most important

stages in teaching the cohesion.

CSEDU 2018 - 10th International Conference on Computer Supported Education

470

The “PictInteraction” application can be used by

children with any autistic disorders and speech

disturbances. It is recommended that the application

is exploited independently for alternative

communication. As a means of developing the

coherent speech it can only be used at a particular

therapeutic session.

The application was made for the tablet

computers (the Android operational system). It is

intended to be used in communication on the basis

of the pictograms. It is an interactive option of

learning activities organized with the help of

logopedic activity books by L.B. Baryayeva

(Baryayeva et al., 2007).

We consider the usage of pictograms to be an

effective method that allows satisfying the needs of

children with speech disturbances (and/or

nonspeaking children) for communication.

The programme development comprises three

stages.

1. Stage 1 includes the construction of ontology

required for the description of the object domain.

This is the supplementary stage for the whole

application.

The Protégé 4.3 ontology editor is used as it

allows graphical ontology matching. The Ontology

Web Language is used. In Protégé 4.3 the subclass

Things is created. Then more particular thematic

subclasses, topics, are made up inside the Things

subclass. The topics are: House, Animals, Toys,

Family, Plants, Nature, School, and “I am a child”.

The topics are related to the activity books

mentioned above.

Every topic is divided into several subtypes:

Preposition, Conjunction, Object, Attribute, Activity.

After that the names of the pictograms are specified

inside the topics themselves.

If a pictogram is employed in several topics, all

the necessary topics are linked to it. The pictogram

is automatically displayed in the added topics. In the

right upper rectangle we can see the pictogram file

name (the picture name).

When the graphical ontology scan is made and

the arrow is pointed at the requested object

(pictogram), the file name is shown as well as the

names of all the other topics in which this pictogram

is used. The ontology in question is employed to put

together the user interaction windows in the

integrated software development environment

IntelliJ IDEA.

2. Stage 2 includes the “Teaching” module

development. The module aims to introduce a child

to the “character-symbol” pair which is the symbolic

notation of a real thing represented in the pictogram.

It also aims to help acquire a better understanding of

the pair.

The module trains the children’s skills to link the

pictures of objects with their functions. It also

teaches them the principles of logical construction of

phrases by selecting the necessary symbols

independently.

3. Stage 3 deals with the “Communicator”

module development. The “Communicator” module

is used for communication per se. It is the leading

module of the programme. It comprises the

following three sections: “Alert report”, “I want

…”, and “Phrases”.

1) Section “Alert report”. This section is meant

to be used by a child in case of emergency.

By simply pressing a button a child is able to

report of a situation that needs to be settled

promptly. For example, when he/she comes

down with a headache or needs the lavatory.

2) Section “I want … .” This section is intended

to be used by a child in case he/she wants to

ask for an object. He/she is able just to press

the pictogram of the object. The section

includes a limited number of the most

necessary pictograms.

3) Section “Phrases”. With the help of this

section the phrases can be composed using

the pictograms. In fact, it is the equivalent of

typing the literal text. The section is to be

used for the pictogram communication

exclusively.

Figure 3: The screenshot of the “Communicator” module.

Mobile Applications as a Means for the Diagnostics and Speech Development in Children with Speech Disturbances and as a Resource for

Alternative Communication

471

Figure 3 shows a sentence composed of the

pictograms (the line at the bottom of the screen).

The sentence reads “I like to draw pictures with a

pencil while seated at the table.”

At the top of the screen a child can see a set of

pictograms with varying semantics – Object,

Attribute, Activity, and Function Words. If the child

is in the learning stage, the therapist asks him/her

probing questions about the following things: 1) at

which part of the screen the needed pictogram is

located; 2) what it means; 3) in what sequence the

child should place the pictograms in order to

communicate his/her idea. The child touches the

needed pictogram with his/her finger, then the

pictogram appears as a part of a sentence at the

bottom of the screen.

This activity allows the child to become aware of

the fact that every word has a real idea behind it. It

also teaches to convey the thoughts successfully and

meaningfully.

The assessment of the developed mobile

application “PictInteraction” was conducted. The

assessment was made of how the users perceive the

application. The development designer account was

registered on Google Play where the application was

put.

After that a test group of 50 people was recruited

those people being the parents of children with the

oral speech disturbances. A feedback form was

composed. It was supposed to evaluate the following

three things:

1) how well the children acquire the language of

pictograms;

2) how effective the mobile application is as a

means of communication;

3) how the application influences the

development of coherent speech.

The results from the questionnaire survey

demonstrated that the mobile application in question

is interesting and demanding. But there are some

shortcomings as well. For example, the quality of

pictures is not quite good.

We conducted both the entry testing and the exit

testing of children who had used the “Teaching”

module of the “PictInteraction” application. The

entry testing was done before using the application.

The exit testing was done when the parents decided

that their children had already learnt how to use the

application and could employ it for alternative

communication.

The results of testing have shown that it is

possible to learn the language of pictograms and that

the “Teaching” module is an extremely efficient

tool. There is an obvious improvement of the results

in all types of questions. However, it appears that

there are some complicated questions which have

been answered improperly by many children. We

believe that they should be restated in order to

become more understandable.

Figure 4: The diagram representing the incorrect answers

in the tests.

Figure 4 shows the diagram representing the

number of incorrect answers to the test questions

before using the “Teaching” module and after that.

There are 11 questions in the test. Forty children

took part in the test. The horizontal axis contains the

numbers of questions. The vertical axis contains the

amount of children under test who answered the

questions incorrectly.

The errors made during the entry testing are

denoted with the blue colour. The errors made

during the exit testing are denoted with the red

colour. It can be seen very easily that the children

have made much less errors when answering the test

questions after they have used the given application.

The “PictInteraction” application raises the

possibilities of the speech development in general

and of the coherent speech development in

particular. At present we are engaged in the

following promising activities:

1) improving the quality of the pictogram

graphics;

2) creating the animated pictograms;

3) expanding the “Communicator” module by

adding the tasks which are intended to

develop the coherence of speech as a crucial

parameter of the speech development;

4) testing the application not only for the

children with the autistic spectrum disorders

but also for the children with different

manifestations of the specific language

impairment.

3 CONCLUSIONS

Our work experience in the field of speech and

language therapy with children and our close co-

operation with the teacher-logopedists lead us to

believe that the usage of mobile applications

CSEDU 2018 - 10th International Conference on Computer Supported Education

472

promotes the children’s interest in the speech and

language therapy sessions. Thus, these sessions

become more successful.

Today, children spend a lot of time with their

smart-phones and tablet computers. The use of

mobile applications for the correction of speech

disturbances (together with the performance of

various logopedic tasks) will encourage the children

to work on their own speech in the course of a

computer game. It will also prevent them from the

“light-headed” pastime with their mobile devices.

REFERENCES

Baryayeva, L. V., Loginova, E. T., and Lopatina, L. V.,

2007. I – speak! Child in the household. The tasks

with pictograms: Activity book for the lessons with

children. Drofa Publishing House. Moscow.

Chirkina, G. V. (ed.), 2003. Survey techniques of the

children’s speech: Textbook on the diagnostics of

speech disturbances. ARKTI Publishing House.

Moscow, 3

edition.

Mobile Applications as a Means for the Diagnostics and Speech Development in Children with Speech Disturbances and as a Resource for

Alternative Communication

473