Mobile applications for diagnostics, development, and alternative communication for children with language disorders

Full Text

Problem and goal. Language communication is an essential part of an individual’s private and social life. Thus, it has a significant impact on life quality. Language disorders of different kinds may cause personal issues and prevent the successful development of an individual. Early diagnostics and precise correctional recommendations enable such issues to be resolved efficiently in childhood; postponing corrective efforts to later stages can have serious adverse effects. Indeed, it is vital for children to have convenient and expressive ways to communicate even if they are currently struggling with language problems. Several common manifestations appear in children with language disorders, and indicate a systematic pathology of language activity [1]. These include a range of disorders, from specific language impairment (incoherent speech; a sequential, lexical, and grammatical construction of speech; and a phonetical and phonematical understanding and verbalization of concepts in general) to disorders that can be placed on the autism spectrum (from limited communication with the external world to the complete rejection of social interaction). The systematic nature of speech disorders manifests in the following ways: children begin to speak later, their speech is devoid of grammar, and their insufficiently phonetically framed speech activity is reduced and with age, if not corrected, declines sharply. Such incomplete speech activity also has an effect on other areas of a child’s mental development. Children with such disorders may have unstable attention and not know how to distribute it. Their verbal memory may be deficient. They may not develop verbal and logical thinking as quickly. In addition, they may lag behind their peers in the reproduction of motive assignments, in terms of space-time parameters and tasks that involve fine motor skills. There are data to support that such children have difficulty learning the Russian language in a comprehensive school environment [2]. Our experience reveals that language disorders can be an obstacle to academic success, even if the university does its best to remedy the situation. We are interested in developing software tools designed for the diagnostic and correction of language development in children with special health disabilities. Creating software tools is quite important, especially when dealing with children. The peak of prophylactic and therapeutic activities occurs in pre- and primary school age. Duenser et al. [3] found that, in Australia, 20% of children at the age of 4 years have a language disorder that hinders their development and understanding of the English language. Of the total number of children who undergo special logopedic examinations, 25% wait approximately six months for the preliminary results, while 15% of children wait for more than a year. After receiving the preliminary results, 18% of children wait an entire year before the speech therapy sessions begin. In remote areas, this issue is exacerbated by the shortage of specialists. Lack of diagnosis or additional training can lead to a sharp decline in the progress and literacy of children, increasing the likelihood of problems in social interactions, which can negatively affect a child’s emotional state. The authors affirm the need to create a mobile application that can diagnose and correct language disorders, and that would allow parents to help their children without requiring them to obtain special knowledge in the field of language disorders. At the moment there is no official statistics on speech diagnoses in Russia. One of the causes is differences between practice and official standards that difficulties comparison of particular issues. For example, the International Statistical Classification of Diseases and Related Health Problems (ICD-11) accepted in Russia does not recognize dyslexia. Another cause is a lack of organizational and methodological ways to collect comprehensive information by federal statistical organisms. In general, according to national sources [4; 5], about 20-25% of preschoolers suffer from speech disorders and from general speech underdevelopment in particular. The draft of “The Strategy of Development of Education for Children with Health Disabilities in the Russian Federation until 2030” (dated April 16, 2019)[2] tells us that at the beginning of 2019, more than 1.15 million of physically challenged children and more than 670,000 children with disabilities live in Russia. According to the definition of the World Health Organization, physically challenged children include children with visual, hearing and speech problems, mental retardation, mental development disorder, autism spectrum disorders (ASD), disorders of the musculoskeletal system, and emotional-volitional defects, as well as complex developmental disorders. According to the Strategy, negative trends in the educational system development for physically challenged children should include an increase in the number of children with developmental disorders and complex disorders, in the extent of functional disorders and chronic diseases, and in the number of disorders with an unclear etiology. This requires early complex diagnosis, which would allow for identification of speech disorders in the initial stages and prevent their development in later adulthood. Several years ago, the Department of Fundamental and Applied Linguistics of the Humanitarian Institute of the Novosibirsk State University initiated an interdisciplinary project that integrates fundamental and applied research in applied linguistics, computer linguistics, psycholinguistics, speech therapy and cognitive psychology. The practical result of this project was the creation of software tools for the diagnosis and development of speech disorders in children with special health abilities. To date, we have developed and tested three programs for the language therapy practice: - LogoBall, diagnostic program; - Logokvest, a game-based diagnostic tool; - PiktObschenie (the Russian for ‘communicate through pictures’), a mobile application for the development of coherent speech and alternative communication. Parts of this paper appeared in the proceedings of SocInfo’18 Conference [6]. This extended version includes new results, not mentioned in our previous publication as well as discussion of practical issues of tool evaluation. Methodology. The complexity of disorders observed in recent years, and the increasing number of children in pre-school and school who need speech therapy create such conditions for the speech therapist so that he or she must work quickly. As a result, a modern speech therapist needs mobile facilities that can both make his/her work with prescribed protocols easier and perform high-quality professional diagnostics. This is the reason why mobile applications based on various platforms are so important in speech therapy practice. Their portability, additional visual and auditory stimuli have become key parameters for fast and high-quality diagnostic and special educational work. When comparing the traditional “paper” approach to the modern mobile-based one, Robles-Bykbaev et al. [7; 8] confirmed that the traditional approach has significant shortcomings: a small number of therapists, a large number of children for whom testing is necessary, in addition to planning, a large amount of preparatory work, processing of the results, monitoring, inter alia. Nevertheless, in language therapy practices, paper carriers are still widely used, both in Russia and abroad. One disadvantage is that they are voluminous and take up much of a therapist’s office space. An experimental evaluation of mobile applications’ effect on children with specific language impairments (32 and 26 children respectively) has found that the amount of time needed to achieve an appropriate level of the language development is substantially reduced, compared to the traditional approach. The difficulties in using paper methods are evident in the time-consuming diagnostics of language development: - the diagnostic material is selected manually by examining several collections of illustrations; - the results are recorded in a special form; - the results are then transferred into electronic tables for processing; - decisions are subsequently made and recommendations given. The personal experience of one of the authors reveals that the traditional “paper” method for testing a group of 20-25 preschool children requires up to three weeks to accomplish. Hence, there are at least three reasons to involve modern information technologies: - the diagnostics methods should be applicable on a mass scale; - they should be attractive and non-disturbing for a child; - they should be easily reconfigurable and tunable. Using mobile applications is also highly beneficial for therapists. First, it cuts the amount of time required for preparatory procedures, as well as the time needed for processing all of the data received during a child’s diagnostics. In Russia, therapists spend one-third of their working time preparing and processing paper carriers. Thus, if they spend less time on these tasks, they can devote more time to face-to-face communication with children. Secondly, the therapy process becomes universal: the electronic form allows a specialist to work with a child at any convenient location. Thirdly, the work of the speech therapist is automated: the program records the intermediate and final results, visualizes the results in the form of tables and graphs, and saves them. Children with developmental challenges, including language disorders, require a special learning environment that should meet educational needs and create comfortable conditions for development. For modern children, mobile applications are a natural part of their daily lives. Thus, the use of mobile applications can make receiving help more interesting, more comfortable and more productive. Preparing for the implementation of mobile applications, the authors used an eye-tracking technique to compare the effectiveness of diagnostics when using paper carriers and when using a portable device. Although the authors cannot insist on the completeness of this study, it nevertheless demonstrated that the electronic approach aided the child in focusing on the test for a longer period of time than did the paper approach. Among the substantial number of children’s applications (entertainment, gaming, regulating activities, developing) there are various training programs that allow children to navigate a studied topic independently, or in tandem with an adult. Using these programs, children can be in charge of their own educational trajectory and the application will monitor its own effectiveness in the training process. In addition, mobile applications help to maintain children’s attention on the therapeutic process and motivate young patients for active participation. Moreover, software tools reduce the cost of therapy, making it more affordable. For example, the price of printed sets of testing and training papers can be as high as several hundred dollars in the US market. In contrast with the printed version, software errors can easily be corrected and therapeutic methods can be updated intermittently. The current trend in the development of universal design [9] involves the creation of products and objects that can be used by all people without special adaptation or design. The fundamental principles of universal design are: 1) equality and flexibility in use; 2) simple and intuitive design; 3) easily perceived information. The principles are equally implemented for both adults and children as well as for persons with and without disabilities. Modern products of such design are created specifically as universal. Another popular trend with which the development of technical equipment for persons with disabilities began was the adaptation of products that people corresponding to the conditional standard use for the needs and goals of disabled people. When identifying the potential of a mobile application for working with both “non-speaking” children and children corresponding to the conditional standard, we relied on the experience of creating and using alternative communication tools and their potential for the universal design of modern speech therapy. Let us briefly analyze modern alternative communication applications. Rodriguez and Cumming [10] studied the influence of the Language Builder application, designed specifically for children with speech disorders, on their language skills (i.e., how receptive (passive) vocabulary, expressive vocabulary and the skill to formulate sentences have changed after using this application). In the experiment took part 31 pupils from the first to third grade with identified speech disorders. Each of the participants had to use the Language Builder application for 30 minutes a day. Children were divided into control and experimental groups for the 8-weeks experiment. The data showed that children in the speech therapy group made much more progress in forming syntactic structures compared to the control group. In addition, the author did not reveal significant differences between the two groups in expressive and receptive vocabulary. The authors concluded that using the Language Builder application can increase the skills of forming sentences in children with language delays. Kultsova and Matyushechkin [11] described the development of the Travel and Communication Assistant auxiliary mobile application supporting the independent movement and communication of people with cognitive disorders. The authors claim that the application allows the supervisor to track the movement of the ward along a given route, as well as to communicate using text, voice and messages from pictograms. The team also noted that a special Russian-language extension of the Text2Picto web service was implemented to translate Russian text messages into pictograms [12]. Lee et al. [13] presented the application PuzzleWalk. Adults with ASD are less physically active than adults without such a diagnosis. After analyzing the situation, the authors concluded that smartphone applications with augmented reality can be an effective solution to this problem. In this application, according to the authors, the user can mark his location and frequently visited places on the map and, most importantly, the user can mark his responsibilities in these places and make notes. Such visualization will help people with ASD to structure their thoughts and increase their physical activity. Barman and Deb [14] explain that the main idea of this application is to provide a platform for people with autism and aphasia, allowing them to understand and formulate sentences, which cause difficulties due to certain damaged areas of the brain. The authors believe that the mobile application, which works as a device for formulating sentences, will help to overcome these difficulties. Mooney et al. [15] found that complementary and alternative communication strategies developed for people with chronic aphasia contribute to generative language skills. The authors argue that it is necessary to find a strategy for complementary and alternative communication for people experiencing primary progressive aphasia (PPA) and neurodegenerative dementia (ND). According to the authors, an experiment was conducted in which alternative procedures were used to compare the vocabulary used for retelling in three conditions: lack of technical support, use of photos only and the CoChat application with photos and shortcuts. This study allows concluding that CoChat proposed by the authors provides the necessary support during natural conversations on personal topics for people with PPA. The authors also say that a graphical interface and visualization of words in applications increases the efficiency of the latter. Cheng et al. [16] argued that facial expression awareness is one of the main tools for building social relationships, and recognition of emotion is one of the most crucial problems for people with autism spectrum disorders (ASD). The authors note that mobile learning creates the earning environment that will help adapt people with ASD to recognize other people’s emotions. In this study, the three-dimensional complex facial recognition system (3CFER) has an important place. It was used when working with children who had ASD. With this program, children were shown the simulated faces of people with different emotions. The results demonstrated that the participants in the experiment showed a significant improvement in their understanding of emotions after conducting it several times. The authors note that surprise and shyness were easily identified by almost all participants. The article ends with the conclusion that the mobile training system is promising. Askari et al. [17] presented studies on the ability of children with autism to recognize facial expressions using the humanoid robot Ryan. Six children with ASD and six typical children took part in the experiment: Ryan showed them six core emotions (i.e. anger, disgust, fear, happiness, sadness and surprise) with different levels of intensity. According to the authors, the participants were asked to identify the emotions depicted by Ryan. The results of the study show that there are no general violations in the ability to recognize expressions in the ASD group compared to the control group; however, as noted by the authors, both groups showed flaws in revealing disgust and fear. Increased intensity of Ryan’s facial expressions significantly improved the accuracy of facial recognition. This study proves that both groups were able to recognize the emotions demonstrated by Ryan with high average accuracy. It means that children with ASD can perceive emotions expressed by other people. Whilst the educational programs are quite diverse, the diagnostic programs are not many. For example, the program Speech Doctor (developed by 1SpecialPlace[3]) seems to be interesting. This application allows parents to test the child’s level of language development. The diagnosis is carried out by answering questions; the answer options are yes/no. After the program has received all the answers, they are sent to speech therapists for analysis. After this, a letter with the results of the diagnostics is sent to the parents’ mail. We did not find among the Russian-language applications those that would be directly intended for diagnosis, with the exception of one “Speech Therapist.”[4] It proposes to conduct rapid testing on its own and to identify problems in the pronunciation of sounds in the child. The results offer a set of articulation exercises with their descriptions, aimed at correcting the identified problem. Testing is an assessment of a child’s sound by the “right/wrong” criterion. Unfortunately, “Speech” has a very narrow scope and users note its unclear interface. Lorusso et al. [18] have stated that, at the moment, software tools for aiding in language disorder therapy are not well characterized with regard to therapeutic effectiveness. In addition, there is scarce information about the clinical testing of such solutions, which makes it difficult to assess their effectiveness. Results and discussion. LogoBall. This diagnostic program was designed to automate the process of detecting a child’s speech pathology: create tables and graphs, compare the results of diagnostics at different stages of training, and save the list of errors made by the child during the diagnosis. It is beginning to be introduced into logopedic practices. The application for diagnosing children’s speech has a diagnostic structure, according to Nishchevaya [19], that consists of 9 series and 33 groups. Each diagnostic criterion is subdivided according to the age gradation (4, 5, and 6 years). The diagnostic series are as follows: 1) sound-syllabic structure of the word; 2) phonemic perception; 3) phonemic analysis and synthesis (an example Figure 1); 4) understanding of speech; 5) understanding of word combinations and simple sentences; 6) lexical and grammatical structure of the child's speech; 7) inflection; 8) word formation; 9) coherent speech. Thus, it is further necessary to examine the application from two perspectives: that of the speech therapist and that of the child. The directly stimulated diagnostic material is divided into two types: with and without illustrative support. NervesNilLegVogueCarrotCreamHelp me find the words where the following sound is contained: N Level 1 DoctorsRooksThe snow is melting. A stream flows. The branches are full of... Level 3 Figure 1. Series for sounds and phonematics If the task on the screen does not provide illustrative support, the child can only see buttons with words, phrases or sentences, with which he/she must perform the task. If the child’s answer is wrong, then the app saves information about the mistake. Recording the child’s errors is integral for further constructing the individual trajectory of the developmental work for each child by the speech therapist. If the task on the screen does provide illustrative support, then the child must perform the tasks specified by the illustrations. The child’s errors are also recorded for these tasks. On the right side of the screen, there are buttons numbered from 0 to 3, which represent an evaluation of the child’s performance. The specialist can move to the next task but it is not possible to go back. However, there is no time limit to perform a task. The principal functionality of LogoBall is to record and synchronize the scores set by the language therapist for each task; record the errors made by a child when pressing the incorrect button that corresponds to the incorrect word; provide a simple interface to make the application easy to use; implement an illustrative series to help children concentrate. A session consists of a series of tasks. For example, one task tests the ability of a child to match adjectives with nouns in the singular (e.g. a green bucket, a red flag, a blue pencil, etc.). The therapist shows a picture on the screen and asks questions so that the child can demonstrate his ability to match the adjectives with the nouns: “What color is the bucket? Which bucket will we take to fetch water? Which bucket can we fill with water? To which bucket can we attach a rope?”, etc. After the child has completed a task, the therapist assigns a corresponding score according to the rating system to which he adheres. We recommend using the following system, although we do not insist upon it: - 0 means that the child has absolutely failed the job; - 1 means that the child has performed less than 30% of the task; - 2 means that the task has been partially performed correctly (no more than 70%); - 3 means the task has been performed completely, or almost completely. We deliberately do not prescribe clear criteria for scoring, because in Russian language therapy there are controversial ways of evaluating the results. Thus, the program is purposefully flexible to allow the specialist to choose the criteria independently. To receive feedback on the quality of the developed application, 22 experts, teachers, and language therapists were questioned. We developed a questionnaire, which includes nine evaluation criteria. All of the questions aim at identifying issues with the program. In order to improve the functionality of the application, each criterion is assessed on a five-point scale, where 1 is very poor, 2 is poor, 3 is satisfactory, 4 is good, and 5 is excellent. The open questions asked encouraged respondents to express their desires for the application and specify its shortcomings. Based on the results of the questionnaire, we calculated the results (average score for each criterion of the questionnaire). The most successful aspect of the application is its stability, with a rating of 4.93. The second best was the speed of the application, at 4.86. Subsequently, how well the diagnostic material corresponds to the original received a rating of 4.71. The average scores obtained for the rest of the criteria are as follows: application design - 4; navigation - 4.23; color solution - 4.05; correspondence between illustrations and tasks - 4.1; and application functionality - 4.47. The criterion with the lowest score is the quality of visualization, which received an average score of 3.84. Taking into account the results of the experts’ evaluation, the following conclusions are drawn with regard to improvements that need to be made to the application LogoBall. The first improvement relates to the illustrative series. Illustrations are perceived by the children better if they are decorated in the same style and presented against a white background, without any additional background (context). Second, it is reasonable to place no more than three or four pictures on one slide. This allows the therapist to improve the child’s concentration on a specific task. Third, to work with preschool-age children, it is necessary to include “step back” and “pause” handles because such young children are unable to concentrate on any task for more than 20 minutes and therefore, speech therapists and teachers must conduct diagnostics in two or even three stages. Fourth, the program should contain extensive instructions and explanations for each diagnostic task so that it can be used by anyone, and not just specialists can use it. We believe that the following suggestions will greatly improve the application. To facilitate the child’s perception of the interface, we should continue improving the illustrative application series. This is a challenging task, which involves an examination of complex cognitive problems relating to human-computer interactions. Moreover, tasks should be arranged with respect to their complexity level to allow the therapist to distinguish among children with different levels of language development and thus, differentiate their tasks. Finally, adding several game elements may increase a child’s motivation to undergo diagnostics. Logokvest. At the moment, there is a wide spread of speech disorders associated with problems in a particular language area. They are most effectively resolved at preschool age; however, to correct them, it is necessary to determine exact diagnoses. Various logopedic diagnostics can be carried out in specialized kindergartens and centers. The vast majority of them are questionnaires with tasks involving various areas of the language, such as phonetics, morphology, and psycholinguistic processes of perception and speech formation. Such diagnostics due to the children’s specificity can take several days. This not only slows down the examination process but can also lead to an incorrect diagnosis since a child’s well-being and mood may bias exam results. This form of diagnostics quickly causes a loss of interest and a decrease in concentration of attention. In addition, it was noted that logopedic diagnostics techniques weakly interacts with modern linguistic investigations which describe the language as a system and individual elements of this system with their features and parameters important in the process of communication. This phenomenon may cast doubt on the rigorous validity of diagnostic methods. Our diagnostic experience has shown that there is an urgent need to solve these problems. One of the simplest and universal solutions is the creation of a mobile application. We attempted to create a theoretically justified diagnostic tool in a form that is fascinating for preschool children, taking into account their mental, age characteristics and specifics of speech development. This application Logokvest implements game-based diagnostics to test a child’s level of lexical and grammatical proficiency. It is a set of mini-games grouped into five thematic complexes depending on the game story. The results can be recorded in the program memory. When the application is launched, the user opens the main menu with two buttons: “Play” and “Children” (Figure 2). To start the game, the first button is pressed, to access the data about the children who have been screened and their results - the second. Before the game starts, a window with back and add buttons is displayed. When you click, the second form of registration of the child with the fields “Family, child’s name,” “Age of the child,” “Name or group number” comes out. Underneath the fields are the Back and Add buttons. After pressing the second one, a line with the child’s data appears in the window. In the absence of data in the Age and Group Fields, the default values are “7 Years” and “Rainbow Unicorns” respectively. The unfilled field of surname and name remains empty. There is an opportunity to enter both the name and surname together, and one thing. MENU PLAY CHILDREN Figure 2. Welcoming window of Logokvest ANIMALSPLANTSИзображение выглядит как текст Автоматически созданное описание Figure 3. An example of a Logkvest task window During the game, the child should distribute images of items on the shelves of cabinets with the names of thematic groups of words in which they are included (Figure 3). Each correct ratio adds one point. In the event of an error, the image remains on the shelves of the wrong cabinet. Regardless of the number of points scored, the child continues to play. The levels differ from the thematic groups of words to the principle of the frequency of use of the objects they designate in the task introduction. everyday life. Thirteen images are provided to the child to relate, so he has the right to make three mistakes. Depending on the number of points scored at all levels of the game, the preliminary level of specific language impairment is determined: less than ten - the first, from ten to twenty - the second, from twenty-one to twenty-ten - the third, thirty - the absence of language impairment. The program also allows you to re-pass the game for re-examination. To do this, just click n… In the first complex, the child is invited to select items belonging to a certain group from presented shelves and collect whole things from pieces, thus allowing them to improve the lexical level of their language. The second complex is a journey through a zoo, during which the child must identify the sounds produced by the animals, give the correct names of the offspring of various animals and match the animals and parts of their bodies, composing the forms of possessive adjectives. The third complex encourages the child to assume the identity of an artist. Here, the lexical and grammatical aspects of speech are tested. The child is asked to assemble a palette of certain colors, draw geometric figures and paint objects; all of which is accomplished by composing a word combination in the correct form. In the fourth complex, the child is tasked with “creating” several items from one, dividing them into groups of two and five, and verifying the presence or absence of certain objects; this tests the child’s knowledge of plural forms, and determines whether she/he can use numerals with them, etc. The fifth complex, which is set in the wilderness, tests the child’s ability to form prepositional-case constructions with prepositions and create verbs of the perfective aspect. For each correctly classified object, the child receives one point; errors do not affect the diagnosis, which helps reduce the level of stress. Each mini-game has four levels. The number of points received for each mini-game and each complex is entered into a table, after which the total number of points is counted. The more points a child gets, the less serious is his or her language development problem. The illustrative material is created in a uniform style to ensure that the external component of the program appears attractive, and that the game process is fascinating. When selecting the “Children” button in the main menu, the speech therapist gets the opportunity to view the list of all past games, as well as add the child at the touch of the “Add Child” button. It is not possible to remove a child from the list. After analyzing the analogues and the object of the diagnostics, getting the illustrations created specifically for the program and creating the application code, the goal was achieved. The result was one of the mini-games of the game mobile diagnostics of the lexicon-gram level Logoquest in children of primary school age with language impairment. It has a bright, aesthetically pleasing interface and the ability to count points. The ease of working with the application allows you to use it not only speech therapists, but also parents. At the moment, the program is being tested. PiktObschenie. This mobile application was designed to improve children’s coherent speech and alternative communication. This application allows an individual to communicate using the language of the icons and their color scheme. The same pictogram can denote an action (if it is red), an object (if it is blue), or a sign (if it is green). Examples are homonyms like to smile and a smile. The color black is used for service words. Color indication helps to construct phrases more correctly. Children can make both simple and complex sentences by placing the pictograms in a certain order, which is the most important stage of learning cohesion (Figure 4). ConjunctionSchoolActionsFeatureObjectPreposition Figure 4. An example of the PiktObschenie session: the composed sentence is “I like to draw with a pencil on the table” The application was developed for pictogram-based communication, which is an interactive version of the learning exercises from speech therapy workbooks. Our experience demonstrates that pictograms enable speech-impaired and/or non-speaking children to satisfy their need to communicate. Actions The program consists of two modules: a trainer and a communicator. Before undertaking software implementation, ontological modelling was accomplished. The training module serves to familiarize the child with a symbol-meaning relationship; the module trains the child to associate the images of objects with their functions, and teaches the principles of the logical construction of a phrase by encouraging the child to choose the right symbols. The communicator module is an alternative means of communication and a tool for the development of coherent speech. The main sections of this module are as follows: Specials: the section is designed so that at one touch of a button the child can quickly report a situation that requires immediate resolution (for example, “I have a headache”, “I want to go to the toilet”). I want: the section enables a child to ask quickly for a certain thing, simply by clicking on the object icon. It consists of a limited number of pictographs most needed by the child. Phrases: in this section, the child can put pictograms together to make phrases. The users tested the developed application to assess its perception. We registered a developer account on GooglePlay, where the mobile application was hosted. Subsequently, a test group of 50 people - specifically, parents of children with speech disorders - was recruited. A questionnaire was compiled to assess the level of the children’s familiarization with the language of pictograms, effectiveness of the mobile application as a means of alternative communication, and its influence on the development of coherent speech. The parents’ responses indicate that, in general, the idea of such an application is interesting and in high demand, but there are also some issues; for example, the quality of the pictures. The application is currently being improved and refined to address these issues. Number of errors in tests Test numbers in the app ■ Before training ■ After training Figure 5. Number of errors in the tests occurring at the beginning and at the end Of the 50 people who enrolled, only 40 passed the test; several refused, either for no apparent reason or under the pretext that, for example, the children would find the application boring. Judging by the feedback and the results, the icons initially seemed somewhat scaring, but after a short interaction with the training module, the users’ attitudes changed. It should be mentioned that in contrast with the initial testing where average number of errors (their range is from 8 to 26) was 18 per test after training average number (their range is from 1 to 9) of errors was 5 per test (Figure 5). Overall, the application demonstrates significant potential for speech development, in particular, for the development of coherent speech. Currently, we are working on improving the following areas: - the quality of the graphics of the pictogram base: creating animated pictograms (instead of using the originals [20]); - the communicator module: in particular, adding assignments directly aimed at developing the coherence of speech as a key parameter of speech development; - application testing: to not only include children who are on the autistic spectrum, but also children who suffer from various manifestations of language underdevelopment. The current trend in the development of universal design involves the creation of products and objects that can be used by all people without special adaptation or design. PiktObschenie was developed in 2016-2017 for a tablet PC (Android OS) for communication based on pictograms, which is an interactive version of speech therapy workbooks by L.B. Baryaeva. Using the application, we assumed that it has a number of possibilities for alternative communication, in particular for children with ASD. Let us clarify what we mean by “alternative communication.” All communication systems that are not related to voice are called alternative systems. They can completely replace speech or become an addition to it. They can be a complete alternative in the absence of spoken language: in this case, communication involves the mastery of a completely different communicative system, where nonverbal communicative means are especially important: photographs, drawings, and so on. In the case of PiktObschenie, these are pictograms. Even without a detailed study of the methodology used in the applications, it can be revealed that all of them have the potential to be used for educational purposes for “speaking” children and adolescents with minor speech disorders or for those who want to deeply engage in speech development. Let us analyze the capabilities of PiktObschenie. It consists of three modules described below. Acquaintance with Pictograms. The module aims at making the child familiar with the pictograms and clarifying that he or she understands them and can somehow identify them. In the “Acquaintance with Pictograms” module, it is necessary to show the pictograms to the child, ask him to identify them (Figure 6). Then the child should match some separate pictograms with the picture, which presents to some extent a realistic image of the object, feature or action of interest (for example, a house or a shower as an object; sweeping or grunting as an action of the object; fat or old as a feature of the object). As has been mentioned above, a certain color is assigned to each category of pictograms, and a child must be warned about this in advance. This module presents six of eight topics. The child will have to get acquainted with the following pictograms: ‘I,’ ‘Family,’ ‘Home,’ ‘Toys,’ ‘School,’ ‘Animals.’ For each topic, there are almost similar exercises in six proposed topics: 1) “select the picture:” object, action or feature; 2) “show what an adult will call:” an object, action or feature; 3) “find the same pictures” among the pictures with an object, action or feature; 4) “find the same picture:” object, action or feature. Show the picture Show the picture Figure 6. An exercise from the “Acquaintance with Pictograms” module Figure 7. An exercise from the “Objects. Actions. Features” module Show the toy Show: I love a teddy-bear I play toy blocks Figure 8. An exercise from the “Objects. Actions. Features” module Figure 9. An exercise from the “Objects. Actions. Features” module Objects, actions, features. The purpose of this module is to familiarize with a number of new pictograms that were not presented in the “Acquaintance with Pictograms” module, as well as strengthen the received knowledge about pictograms and do additional exercises (at the adult’s discretion or if the child still has difficulties with identifying some of the pictograms) devoted to the same six topics that are proposed in the module “Acquaintance with Pictograms:” ‘I,’ ‘Family,’ ‘Home,’ ‘Toys,’ ‘School,’ ‘Animals.’ The following types of exercises are presented here (Figures 7-9): 1) “show what I’m talking about:” the child chooses from a pair of sentences made up of pictograms what the adult said; 2) “show the picture:” similar to the first exercise in the “Acquaintance with Pictograms” module. Communication. This module focuses on communication with the help of pictograms and the development of coherent speech. It includes the ‘Phrases’ section, which implements the compilation of phrases and small texts based on pictograms. In fact, this section is similar to typing of an alphabetic text. It is intended for communication with the help of pictograms, and it can also be used as a tool for the development of coherent speech, using the corresponding exercises. The module presents six topics: ‘I,’ ‘Family,’ ‘Home,’ ‘Toys,’ ‘School,’ ‘Animals,’ ‘Nature,’ ‘Plants’ (Figures 10-11). The stages of teaching a child to work with pictograms in a series of workbooks and a mobile application are almost identical: - the child gets acquainted with the sign-symbol (pictogram) and the supervisor clarifies whether the child understands what this or that pictogram means; - the child establishes the connection between images of objects and their functions; then he or she sequentially logically formulates the sentence by independently choosing desired symbols. The first testing of the application (2015) (carried out by parents of non-speaking children with ASD) revealed significant shortcomings, one of which was the lack of guidelines on the application. In this connection, serious work was required to identify the app’s developing potential for children with speech disorders and create speech therapy recommendations for using the application in specific language classes. ToysActionsFeatureObjectPrepositionConjunction Figure 10. Screenshot of the “Communication” module, topic “Toys” NatureConjunctionObjectPrepositionFeatureActions Figure 11. Screenshot of the “Communication” module, topic “Nature” In 2019, we studied the possibilities of the application and revealed its potential not only for alternative communication but also for speech development of preschool children, both typical and those who are diagnosed with general speech disorders. To study the PiktObschenie’s possibilities, we worked with speech therapists who have experience with children with various speech disorders and use technical communication tools in their practice, in particular, computer speech therapy programs and developing mobile applications. We developed a questionnaire for speech therapists and practitioners, which helped study expert data on the developing potential of the application. To evaluate PiktObschenie in practice, we selected the following criteria: - the app usability for both adults and children, i.e. the interface, navigation and the quality of the icons should be at the acceptable level; - the ability to perform certain exercises aimed at developing coherent speech in children, in particular, its leading components: expanding vocabulary and grammatically correct use of words in the sentence as well as coherent speech, i.e. composing various stories with the help of pictograms; - children’s interest in the application; - the application motivates children to make contact with adults. About the PiktObschenie mobile application 56 experts (speech therapists) were questioned. The following positive features were mentioned in working with children who suffer from general speech underdevelopment, using the PiktObshchenie application: - children were interested in the application (89% of the experts’ responses); - many children were easily contacted because of interest (87%); - children did the proposed exercises with no difficulty (mainly on the development of grammar and coherent speech) (71%). Some problems relating to the app interface were detected and included in development plans to fix. Based on the results of our work with speech therapists, we found that PiktObschenie is most productive not for alternative communication of “non-speaking” children (as previously thought), but for the development of coherent speech of children with speech disorders as well as for typical children’s development. Conclusion. To analyse the results of the development and implementation of mobile applications we collaborated with speech therapists. Applications used in children’s speech therapy are discussed at specialized workshops. Questionnaires for experts were created to discuss and evaluate the effectiveness of using applications. The questionnaires have both open and closed questions, and have tasks related to the ratings and scoring of a particular application’s function. Quality, functionality, stability, reliability and availability are evaluated separately. Expert evaluation allows us to improve the applications. Our working hypotheses about the demand and necessity of mobile applications are confirmed by real speech therapy practice. For further discussion, we raise the issue of creating training and methodological complexes for working with mobile applications for both speech therapists/professionals and parents of children with speech disorders, including those with autism spectrum disorders. For example, we are developing specific guidelines, lessons and exercises with which PiktObshchenie will become a tool for specialists. In addition, we are working on the creation of a special textbook for parents and mentors of children, who do not have special pedagogical or speech therapy education about children with speech disorders. This textbook includes a number of recommendations and specific developing techniques. In addition, the text contains QR codes. By scanning these, users will be able to see a record of the real lesson of a professional speech therapist with a child who is using the technique described in the text. Thus, we will address the issue of prolonged assistance for those involved in various rehabilitation practices, using diagnostic and developing mobile applications with children who suffer from speech disorders. Finally, we believe that the technique used in the application can also be adapted for use by adults who have speech problems caused by, for example, accidents or insults.

About the authors

Liudmila B. Mozheikina

Novosibirsk State Technical University

Email: mozhejkina@corp.nstu.ru
ORCID iD: 0000-0001-7470-8616

PhD in Psychology, Chief of the Center for Inclusive Support

20 Prospekt K. Marksa, bldg 8, Novosibirsk, 630073, Russian Federation

Pavel G. Emelyanov

A.P. Ershov Institute of Informatics Systems, Siberian Branch of the Russian Academy of Sciences

Author for correspondence.
Email: emelyanov@mmf.nsu.ru
ORCID iD: 0000-0001-8747-0600

PhD in Computer Science, senior researcher, Mixed Computation Laboratory

6 Prospekt Lavrentieva, Novosibirsk, 630090, Russian Federation

References