Definition of reflection characteristics of educational process participants with artificial intelligence application

Full Text

Problem statement. Communication tools based on conversational artificial intelligence (AI) become more popular and are already used in education. However, results of its application are researched insufficiently. There are articles about AI tools application in educational programs and its influence on application results [1-3]. But artificial intelligence tools are often used just as a part of adaptive learning systems. AI tools are considered to be taking part in the development of metacognitive skills. But some researchers consider that digitalization of education decreases the level of educational results and doesn’t allow the development of metacognitive skills [4; 5]. It is supposed that specific instructional design work provides AI tools application for competences and skills development. Such work would enhance educational program design and quality of learning outcomes instead of contradicting them. It’s necessary first to research AI tools application for metacognitive skills assessment to prove such a thesis. The research presents comparison between definition of reflection indicators of the participants in the educational process with simulation based on AI tools and with expert assessment. In conclusion discussed an issue of how to interpret the results and how to improve design and development of such simulations. The reflection concept has been intensely discussed in modern psychology since the middle of the 20th century. And nowadays there are a few approaches to define it. Both Russian and foreign scientists agree that reflection - is thinking about thinking [6-9]. In Russian research proceedings of reflection essence and mechanism based on methodological action theory of G. Schedrovitsky, V. Zinchenko, V. Lefevr, Yu. Gromyko were the foundation of the reflection concept [10]. System-Thinking-Activity Approach representatives developed the main theses about what reflection is. One of the base terms was “reflective way out”. Mechanism of “reflective way out” exists as the part of activity and is defined as “action beyond action” and it’s necessary to stop current action and “go beyond” it. That’s the reason why the term “way out to reflection” exists. N. Alexeev described the reflection mechanism as a sequence of thought acts. He considered following G. Schedrovitsky ideas that reflective action begins with the stop of current action. The next step is to trace the considered action and draw it like an external object. So, there are three thought acts which reflection based on: action stop, action tracing and action objectifying [11]. N. Alexeev thought that there is one more element that is needed. That element might be constructive and provide action redesign and improvement [8]. So, one more step appeared and called “action alienation” [11]. Social psychology theories point out that reflection influences social interaction and organises it both in groups and in face-to-face communication. For example, there is a thesis about correlation between social conformity and level of reflection [12]. Modern scientists define reflection in communication in the following way: “The reflection in communication is reflection of common social activity, assessment of activity with going out of it” [13]. So, it’s possible to define reflection in communication as going beyond the communication to comprehend the whole situation: surrounding conditions, actors and stakeholders, points of view and actors’ gains. There is an assumption that reflection indicators can be shown in the negotiation process because there are expressed contradictions in actors’ aims. And the negotiation process itself implies going beyond the situation of direct communication and comprehension gains of both negotiation sides, opponents’ situation, his or her point of view, hidden pains and negotiation thesis [14-16]. In particular, in some studies, reflection is described as “a specific quality of mutual knowledge and mutual understanding, which is an important condition for productive negotiations” [17]. The following reflection indicators in negotiation and group communication processes were allocated [17]: - to change point of view and to observe own situation with point of different person; - to describe own actions, conditions, with prior distinction from each other, been inside negotiation process; - to reconstruct surrounding conditions; - to provide new ideas and knowledges which help to solve situation and overcome problem and conflict obstacles in negotiation; - to design and analyze communication strategies. The following reflection indicators can be defined in negotiation process from the point of N. Alexeev reflection model [6; 15; 16]: - participant stops unproductive communication which don’t directed to a search of common satisfactory decision; - participant traces differentiation between gains of sides been both as a side of the negotiation and as the third side; - participant elaborates opponents’ point and ask specifying questions; - participant gives feedback about vision of situation, structures and organises understanding of the others; - participant offers means for communication; - participant takes into account opponents’ gains and points while searching a decision; - participant designs steps of discussion and monitors the time and rules of negotiations; - participant notices and minimises obstacles and risks and messages about it to the others; - participant finalises, approves and retains results. Thus, the above characteristics are reflection indicators that will be observed in a negotiation procedure in the educational process. For this purpose communicative simulation of negotiation based on artificial intelligence tools was created and expert assessment was organised. Research on correlation between metacognitive skills development and AI instruments application are new due to the very recent spread of this technology. And this topic is discussed in two main directions of AI tools application in education: artificial intelligence for instruction and artificial intelligence for educational assessment. As researches declare the AI tools application for instruction concludes in providing sufficient quality feedback to learners. It allows to add instructors, to personalise feedback, to provoke discussions and group interaction. The use of artificial intelligence for assessing learning outcomes was explored in the process of formative assessment of students, when automatic feedback allowed students to adjust the process of solving a problem and thereby achieve better results [18]. However, it was possible to find articles only about subject matter competences in domains of medicine, foreign language learning and STEM subjects. There were found no research about artificial intelligence tool usage to develop reflection [19]. This work demonstrates how to develop reflection with AI based chat-bot application. Analysis of the above scientific papers reveals the problem associated with the need for additional study of metacognitive skills assessment issues in the educational process, in particular, defining the indicators of reflection as one of the most important such competencies. The article presents a study aimed at testing the effectiveness of an AI-based technology to determine the indicators of the reflection in the process of communication in education. Methodology. The comparison between reflection indicators definition with artificial intelligence and with expert assessment is discussed in this article. Simulation of negotiations and expert assessment procedures were developed in the educational process to solve this task. Assessment took place as a part of a competition to define the best volunteer in the community. Methodology of competence assessment was used for reflection indicators definition. And assessment procedures were developed: in online mode with usage of simulation and in offline mode with expert assessment. Based on allocated reflection indicators in the negotiation table of behavioural indicators were composed [20; 21] which could be observed in participants behaviour. According to such observations, outcomes about reflection indicators demonstrated in communication were obtained [20; 21]. Based on the table of behavioural indicators behaviour evaluation criteria were developed for online simulations and offline competition. For offline competitions educational games and modelling negotiation procedures were developed based on proceedings of scientific group leaded by B. Khasan[2] [16]. For online simulations the plots of situations were developed based on interviews with experts in volunteering. These simulations were models of negotiation too and consisted of dialog with AI-companion at the screen. Simulations were implemented with “Dailo” app. “Dailo’ - is an application for speech interactive simulation with AI-companion (Figure). Each phrase of participant in simulation is estimated with artificial intelligence and attributed to a certain pre-configured pattern of participant behaviour. Depending on how the participant's phrase was estimated, the character on the screen demonstrated different responses. And so, the participant can change the dialog direction with his or her actions. Thus, the effect of natural dialog is achieved between participant and simulation. Depending on participants' behaviour (requests and responses) the simulated situation can develop both to win-win negotiation solutions as well as conflict escalation. The development of such an interactive tool became real only with achievements in the field of artificial intelligence. Because otherwise it would be impossible to customise the character's reaction to each of the participant's phrases and make its behaviour so variable. With artificial intelligence, application interaction between system and humans were designed in such a way that the system can react flexibly and adjust feedback to the behavioural option which participants have chosen. Изображение выглядит как текст, снимок экрана, одежда, компьютер Автоматически созданное описание Communicative situations creation and dialogue organisation using artificial intelligence Source: made by the authors. So, during the research reflection indicators and criteria for its definition were developed. Based on these indicators two procedures were developed: simulation with AI-assessment and competition with expert assessment. As the result data was obtained for simulations and expert assessment from 65 participants, learners from schools and colleges. Each of them firstly participated in online simulation then in offline competition. Since the definition of the reflection indicators was carried out according to the same criteria, a quantitative score of competence was obtained for each indicator from 0 to 3 points in online and offline procedures. Then the dynamics of participants' assessments with simulation and with experts were compared with usage of Pirson’s criteria to establish if there is stable correlation between expert and AI-simulation assessment. Results and discussion. As a result, a weak correlation was obtained between the expert and AI-simulation assessment (Table). As could be seen in Table correlation centres are divided into blocks. The first block is related with behavioural indicators about opponents’ gains and interests comprehension, clarifying the interests and its consideration during negotiations. The second block of indicators is related with organisation of interaction, tracing interaction results. Although the correlations between expert assessment and simulations are related with different behavioural indicators, the following can be observed. Determining the correlation between expert assessment and assessment, acquired with artificial intelligence simulation Parameter Assessment using artificial intelligence simulation Understand the points of view of all parties, and offer the best solution to the conflict for all Try to find out the point of view of the other being a conflict stakeholder Use friendly speech and attempt to resolve the conflict being a conflict stakeholder Give arguments, explain them, maintain a normal emotional condition Designate and approve communicational purpose Design interaction in the form of a clear sequence of actions and approve it Finalize, approve and retain results Expert assessment Understand the points of view of all parties, and offer the best solution to the conflict for all 0,143 0,281 0,013 0,195 0,184 0,117 0,104 Try to find out the point of view of the other being a conflict stakeholder 0,248 -0,064 0,148 0,284 -0,051 0,002 0,013 Use friendly speech and attempt to resolve the conflict being a conflict stakeholder 0,096 -0,121 0,079 0,241 0,065 0,087 0,154 Give arguments, explain them, maintain a normal emotional condition 0,200 0,033 0,252 -0,037 -0,074 0,029 -0,069 Designate and approve communicational purpose 0,087 -0,119 -0,022 0,152 0,118 0,277 0,184 Design interaction in the form of a clear sequence of actions and approve it 0,014 0,014 -0,007 0,037 0,048 0,114 0,063 Finalize, approve and retain results 0,061 -0,100 -0,128 0,110 -0,061 0,246 -0,036 Note. Pearson correlation - significant at the 0.05 level (two-tailed). At the first there are “crossed” correlations. It means that simulations and experts “mixed” some indicators with each other. For example, between indicators “Understand the points of view of all parties and offer the best solution to the conflict for all” and “Try to find out the point of view of the other being a conflict stakeholder” cross correlation can be seen. As criteria have some similarities in meaning, it’s possible to mix it during simulations or during expert assessment. This situation can be seen in at least two cases. At the second there is no correlation between the two different blocks of correlations. It can be called conditionally “gains comprehension” and “interaction organisation”. It means that inside each block similar criteria can be mixed, but between different blocks there is precise distinction. Thus, the conclusion is that there are correlations between various demonstrations of the reflection indicators of the participants in the educational process, but in a weakly expressed form. Conclusion. As a result of the study, a weak correlation was found between assessment with simulations based on artificial intelligence and expert assessment of the reflection indicators on the material of communication in the educational process. The obtained results can be interpreted according to organisation of the assessment procedure as well as to AI-simulation application. It is assumed that stronger correlations can be obtained with two changes. On the one hand, it is proposed to improve the assessment procedures themselves with simulations and with expert assessment. On the other hand, it is proposed to further develop the information system based on artificial intelligence in order to further increase the flexibility and sensitivity of the assessment. The following options are offered for further developments in the described direction. 1. It is necessary to change the procedure of reflection indicators definition. Specifically, it’s necessary to assess the competences of participants in the educational process not just with the same behavioural indicators but standardise assessment procedures (simulations and competitions) to provide the same structure and scenarios. 2. It’s necessary to make assessment instruments with AI-simulations more variable and develop more patterns of participants behaviour so that the assessment can be even more flexible and can take into account more behaviour subtleties of the participant in the educational process. It is offered to take some actions for the further development of the “Dailo” application based on artificial intelligence to increase fidelity of semantic analysis. It is necessary to increase variability of answers which can be processed and accordingly variability of feedback and subtleties of behavioural indicators recognition. It’s suggested that if to teach artificial intelligence to recognise a flexible template for skill assessment or with other words to recognise behavioural patterns then recognition fidelity will increase. The further studies accordingly have to be related with this flexible template (behavioural pattern) development based on a relatively large number of simple templates. At the moment, a competency model and a set of behavioural indicators have already been developed to determine the reflection indicators in the communicative process. In the future, each of the behavioural indicators will be refined and finalised. Specifically, an array of expert evaluation data for each behavioural indicator will be analysed, and these data will become simple templates, as the base for a flexible template. So, a behavioural pattern will be formed. For considering AI-simulation completed and ready to work as the assessment application it is necessary to develop behavioural patterns for each competence indicator of different participants in the educational process and teach artificial intelligence how to recognise it. Then it is necessary to test the instrument once again to prove its performance. In general, it can be considered that this research proved the efficiency of AI-simulations for defining reflection indicators in the educational process and determined the further directions to improvement of means and methods for competence assessment with AI tools application. The next stage of the study is to apply simulations as an instructing tool not only for assessment, but also for the competencies development of participants in the educational process.

About the authors

Vadim V. Grinshkun

RUDN University

Email: vadim@grinshkun.ru
ORCID iD: 0000-0002-8204-9179

Academician of the Russian Academy of Education, Doctor of Pedagogy, Professor, Professor of the Department of Information Technologies in Continuous Education

6 Mikluho-Maklaya St, Moscow, 117198, Russian Federation

Sofya I. Dreytser

Moscow City University; LLC “Cerevrum”

Author for correspondence.
Email: dreitsersi562@mgpu.ru
ORCID iD: 0000-0001-8549-1627

PhD student, Institute of Digital Education, Moscow City University; instructional designer, LLC “Cerevrum”

4 2-y Selskokhozyaistvennyi Proyezd, Moscow, 129226, Russian Federation; 42 Bolshoy Bulvar, bldg 1, Moscow, 121205, Russian Federation

References

Supplementary files