Multimodal verification of pedagogical features using SAR analysis

Full Text

Features (specific manifestations of pedagogical interactions at the behavioral level) and the establishment of causal links between the actions of the teacher and learning outcomes [2]. Specific, measurable behavioral features of teachers (gestures, intonation, phrases) are traditionally identified through qualitative observation, which does not allow correlations to be separated from causal relationships. Pedagogical science is dominated by an expert approach to evaluating the effectiveness of pedagogical actions. Studies demonstrate the priority of qualitative methods based on reflective analysis and expert evaluations [3]. However, the subjectivity of interpretation and the inability to reproduce results limit the applicability of such methods for the formation of evidence-based pedagogy. International practice demonstrates more systematic approaches to verification. The CLASS (Classroom Assessment Scoring System) protocol, developed by Pianta et al. [4], has become the standard for observational evaluation of the quality of classroom interactions. However, even CLASS remains a correlational tool: it records the relationship between pedagogical behavior and outcomes, but does not prove causality. The development of multimodal learning analytics has opened up new opportunities for automated feature analysis [5; 6]. Blikstein [7] showed that joint processing of verbal, nonverbal, and paraverbal (voice features) data allows identifying patterns of effective learning. Prieto et al. [8] developed a methodology for the automatic extraction of orchestration graphs from wearable sensor data. Worsley and D’Angelo [9] identified key areas for the development of MMLA in a systematic review, emphasizing the need to move from descriptive to causal analytics. Schlotterbeck et al. [10] demonstrated a cost-effective approach to detecting pedagogical practices through spectral analysis of classroom audio, but their method is also limited to identifying correlations. Martinez-Maldonado et al. [11] emphasize the critical importance of contextual interpretation of automatically detected patterns, which remains an unresolved issue in most MMLA studies. A critical analysis of current approaches reveals a fundamental limitation: all existing methodologies - from expert observation to advanced multimodal analytics - only allow for the identification of correlations between the features under consideration and learning outcomes. Controlled experiments are required to prove causality [2], but conducting such experiments in real educational settings is fraught with ethical and organizational constraints. An alternative solution could be the use of social assistive robotics (SAR). Belpaeme et al. [12] showed in their review of the use of social robots in education that SAR systems can reproduce actions in controlled conditions. Scassellati et al. [13] substantiated the potential of SAR for creating standardized experimental protocols. However, there are no methodologies in the literature for using SAR specifically to verify features identified through multimodal analysis of real-world practice. The objective of the study is to develop and substantiate a scientific and methodological approach to the experimental verification of pedagogical features, Bosenko T.M., Sadykova A.R. RUDN Journal of Informatization in Education. 2026;23(1):25-38 which is intended to overcome the limitations of correlation methods through the integration of multimodal analysis and SAR technologies. Research tasks: to develop a procedure for forming a context-enriched dataset of pedagogical features based on multimodal analysis of educational interactions; to create a methodology for translating the identified features into behavioral modules of a social assistance robot for experimental verification; to conduct a comparative analysis of the effectiveness of the proposed approach with existing methods of verifying pedagogical practices; to justify the prospects for scaling the methodology through agent systems. Methodology. The methodological basis of the study is a comprehensive approach that combines methods of multimodal analysis of educational data, experimental verification of pedagogical features, and comparative evaluation of the effectiveness of methodologies. Multimodal analysis of pedagogical features was based on expert annotation of video data from training sessions (50 video recordings, ~18.3 hours) using Label Studio tools and the author’s Pedagogical Pattern Infrastructure (PPI)1 platform [14; 15]. The procedure included the identification of pedagogical events (specific pedagogically significant episodes of interaction) and their classification into two main categories of features with the recording of verbal, nonverbal, and paraverbal components. To assess the reliability of the labeling, a statistical method for calculating inter-expert agreement (Cohen’s kappa coefficient) was used with the participation of 12 expert methodologists divided into two teams, which made it possible to quantitatively assess the subjectivity of expert assessments (k = 0.68, 95% CI: 0.61-0.75) [16]. SAR analysis (Socially Assistive Robotics Analysis) was implemented through a three-step procedure of translating the identified features into SAR agent training parameters, followed by an experimental evaluation of their causal impact on learning outcomes. This method ensured the transition from correlation analysis to proof of causal relationships between pedagogical features and learning outcomes. A comparative analysis of the effectiveness of the proposed approach was conducted in relation to the CLASS (Classroom Assessment Scoring System) methodology [4] based on quantitative indicators of labor costs, assessment objectivity, and scalability of the verification process for different volumes of analyzed material (50, 100, 200, 500 videos). Architecture of the scientific and methodological approach. The proposed methodology consists of a sequence of seven logically connected stages (Figure 1), providing a complete research cycle from hypothesis to experimental verification. 1 The PPI platform was developed as part of a scientific collaboration between the Russian-Chinese Center for Artificial Intelligence in Education at Moscow City University (MCU, Russia) and Central China Normal University (CCNU, China). Available from: https://github.com/BosenkoTM/ppi_mgpu (accessed: 10.12.2025). Босенко Т.М., Садыкова А.Р. Вестник РУДН. Серия: Информатизация образования. 2026. Т. 23. № 1. С. 25-38 Figure 1. Architecture of the scientific and methodological approach to feature verification Source: created by Тimur М. Bosenko, Albina R. Sadykova. Pedagogical hypothesis formulation. The following illustrative hypothesis was selected for testing the methodology: Taking into account the pedagogical features that constitute growth-oriented feedback has a more significant positive impact on student engagement and academic self-sufficiency than evaluative feedback such as ‘right/wrong’. This hypothesis was chosen based on Dweck’s [17] theoretical propositions about the influence of feedback type on student motivation and the availability of preliminary correlation data [18]. Collection and analysis of multimodal data. To create a reference dataset, author-developed methodology based on expert annotation in Label Studio was used, followed by processing in the PPI web platform. Two teams of experts (6 people each) participated in the experiment, analyzing two categories of pedagogical features: ‘Pedagogical Actions’ and ‘Communicative Modalities’. Each team worked in groups of three to ensure the reliability of the annotation. A unified data collection template (Table 1) ensures that not only the observed features are recorded, but also the critically important pedagogical context. For the category ‘Pedagogical actions’, experts identified the following subcategories: asking open/closed questions, explanations/clarifications, evaluating answers (reward/constructive criticism), organizing group work, redirecting attention, providing feedback, and using scaffolding. For the category ‘Communicative modalities’, the following were recorded: verbal, nonverbal, and paraverbal components of interaction. Bosenko T.M., Sadykova A.R. RUDN Journal of Informatization in Education. 2026;23(1):25-38 Table 1. Template for data collection in the reference dataset Field Source Description Event_ID System Unique event identifier Subject Expert Action initiator (Teacher, Student, Group of students) Category Expert Major category (Pedagogical actions, Communicative modalities) Subcategory Expert Feature subcategory (multiple choice) Action_description Expert Objective description of the action (what is happening) Context and comments Expert Key field: context BEFORE; reaction AFTER; pedagogical meaning Source: compiled by Тimur М. Bosenko, Albina R. Sadykova. Theoretical and empirical validation. The identified signs undergo two-level validation: compliance with pedagogical theory and the presence of a correlation with measurable learning outcomes (engagement, academic performance, self-sufficiency). Translation of signs into the SAR system. The key methodological contribution of this study is the development of a three-step procedure for “programming” a pedagogical feature for a SAR agent. 1. Feature decomposition. The annotated feature from the action_description field is decomposed into atomic multimodal components: Verbal: specific phrases and their sequence; Nonverbal: gestures, postures, position in space; Paraverbal: voice characteristics (tempo, volume, intonation contours). 2. Mapping to SAR agent capabilities. Each component is mapped to specific parameters for automatic recognition: Verbal → features for NLP models (tokens, embeddings, patterns); Nonverbal → features for computer vision (key point coordinates, motion trajectories); Paraverbal → features for audio analysis (low-frequency cepstral coefficients, pitch. 3. Generation of training examples. Atomic components are combined into labeled examples for training the SAR agent with temporal synchronization of modalities accurate to 100 ms. 4. Experimental evaluation (SAR analysis). The trained SAR agent undergoes a two-level evaluation: Technical evaluation - accuracy of feature recognition on a test sample (metrics: Precision, Recall, F1-score); Pedagogical evaluation - a controlled experiment to verify the causal relationship between the recognized feature and learning outcomes. Statistical analysis of differences between conditions allows to draw a causal conclusion about the influence of the feature. Results and discussion. To evaluate the effectiveness of the proposed approach, a comparative analysis was conducted with the traditional CLASS (Classroom Assessment Scoring System) observational methodology [4], which represents the standard for observational assessment of features (Table 2). Босенко Т.М., Садыкова А.Р. Вестник РУДН. Серия: Информатизация образования. 2026. Т. 23. № 1. С. 25-38 Table 2. Comparative analysis of feature verification methodologies Criterion CLASS (observational) Proposed SAR approach Type of knowledge obtained Correlational (relationship between a feature and a result) Causal (causal influence of a feature) Objectivity of evaluation Mean (depends on the observer’s training, ICC 0.60-0.75) High (based on statistics from a controlled experiment) Ability to isolate the influence of a feature Low (many confounders in a real classroom) High (controlled conditions of the SAR experiment) Scalability Low (requires training of observers) High (automatic reproduction of the experiment) Ethical restrictions High (impossibility of manipulation in a real classroom) Minimal (experiments with a robot in controlled conditions) Source: compiled by Тimur М. Bosenko, Albina R. Sadykova. The results of the comparison demonstrate the fundamental advantage of the SAR approach: the transition from establishing correlations to proving causality. Traditional CLASS methodology allows to assert: “Teachers who use feature X have students with higher engagement”. SAR analysis allows to state: “Feature X causes an increase in student engagement”. Architectural differences between approaches. The methodological difference between correlational and causal verification is illustrated in Figures 2 and 3. Figure 2. Verification without SAR analysis (correlation approach) Source: created by Тimur М. Bosenko, Albina R. Sadykova. Figure 3. Verification using SAR analysis (causal approach) Source: created by Тimur М. Bosenko, Albina R. Sadykova. Bosenko T.M., Sadykova A.R. RUDN Journal of Informatization in Education. 2026;23(1):25-38 The problem of subjectivity at the stage of forming a reference dataset. A critical analysis of the proposed methodology reveals a key problem of subjectivity in expert evaluation at the stage of forming a reference dataset (stages 2-5). Despite the use of a structured template, the fields category, subcategory, and especially context and comments are filled in by an expert methodologist, which introduces an element of interpretation. Inter-expert agreement evaluation procedure. To quantitatively evaluate the reliability of the annotation, a pilot agreement study was conducted with 12 expert methodologists divided into two teams. A pool of 50 video recordings of lessons lasting from 15 to 30 minutes (total duration: ~18.5 hours, subject area: English language, grades 5-7) was analyzed by groups of three independent experts. Each trio labeled pedagogical events (an average of 8-10 events per video) according to two main categories with corresponding subcategories. Total amount: 450 pedagogical events. Agreement in the classification of events by category is presented in Table 3. Table 3. Inter-expert agreement on categories of pedagogical features Category k (mean) 95% CI Interpretation* Pedagogical actions 0.71 0.65-0.77 Significant Communicative modalities 0.65 0.58-0.72 Significant Overall consistency 0.68 0.61-0.75 Significant Note: interpretation according to Landis & Koch [16]: k < 0.00 - poor agreement; 0.00-0.20 = weak; 0.21-0.40 = satisfactory; 0.41-0.60 = moderate; 0.61-0.80 = significant; 0.81-1.00 = almost complete agreement. Source: compiled by Тimur М. Bosenko. The lowest agreement was observed in the subcategory of nonverbal features (k = 0.58), which can be explained by the subjectivity of interpreting gestures and facial expressions. Cohen’s kappa coefficient was calculated using the formula [16]: k = (Po - Pₑ) / (1 - Pₑ), where Po is the observed proportion of agreement between experts, Pₑ is the expected proportion of random agreement. The confidence interval (95% CI) was calculated using the bootstrap method (1000 iterations on a sample of 450 events). Comparison with the performance of automated models. For context, inter-expert agreement was compared with the performance of modern automatic feature recognition models (Table 4). It is clear that automatic models based on verbal features (F1 = 0.52-0.67) are inferior to experts (k = 0.68-0.75), and the gap is even more significant for non-verbal features (F1 = 0.38-0.51). Full automation of labeling is not feasible at the current stage of technological development. Solving the problem of subjectivity through an agent system based on SAR analysis. The SAR analysis methodology offers a solution to the problem of subjectivity through a mechanism of experimental feature filtering. Experimental verification mechanism. The process functions as shown in Figure 4. Босенко Т.М., Садыкова А.Р. Вестник РУДН. Серия: Информатизация образования. 2026. Т. 23. № 1. С. 25-38 Table 4. Comparison of the performance of experts and automated models Approach Verbal features Nonverbal features Multimodal Experts (MCU) k = 0.73* k = 0.58* k = 0.68 Demszky et al. [19] F1 = 0.52-0.67 - - Suresh et al. [20] F1 = 0.61 - - Prieto et al. [8] - F1 = 0.38-0.51 - Bosch et al. [21] F1 = 0.71** - - Notes: * k for verbal = mean across the categories ‘Engagement’, ‘Cognitive Support’, ‘Feedback’; for nonverbal = ‘Recognition of Emotional Signals’. ** Only for classification of feedback types on a dataset >1000 hours. Source: compiled by Тimur М. Bosenko. Subjective labeling. Experts form a dataset with inevitable subjectivity (k ≈ 0.68). Some features receive high agreement (priority 1), while others receive low agreement (priority 2). SAR verification. Each feature is tested in a controlled experiment. Approximately 60-70% of features show a statistically significant causal effect, while the remaining 30-40% are rejected regardless of the expert evaluation. Critical result. Both false positives (features unanimously approved by experts but ineffective in the experiment) and false negatives (features with low expert agreement that showed a strong effect) are detected. Knowledge base formation. Verified features form an objective basis for training neural network models for automatic recognition. Thus, SAR analysis performs the function of an “objectivity filter”: objective knowledge about causally effective features is extracted from a subjectively labeled dataset through an objective experiment. Mathematical justification shows a 25-33% increase in the accuracy of identifying effective features compared to expert evaluation alone. Economic efficiency evaluation. Labor costs were compared using video materials from a pilot study conducted as part of scientific cooperation between the Russian-Chinese Center for Artificial Intelligence: 50 video recordings of training sessions lasting 15-30 minutes each (total duration ~18.3 hours, subject area: English language, grades 5-7). For the SAR approach, the entire chain was analyzed: expert annotation (12 experts working in parallel in groups of three) → dataset formation → development of SAR scenarios for 12 features → conducting experiments. For the CLASS approach: training of observers → video coding by two observers → agreement on evaluations. Results for the initial analysis of 50 videos: CLASS approach - 124 person-hours; SAR approach - 113 person-hours; Savings - 9% (costs are comparable due to the initial investment in SAR script development). The key advantage of the SAR approach becomes apparent when scaling (Figure 5). After creating a library of SAR scripts (one-time costs), each new verification requires only the time to conduct experiments, which does not depend on the video length. The CLASS approach requires a linear increase in labor costs: each new video sample requires re-encoding. Bosenko T.M., Sadykova A.R. RUDN Journal of Informatization in Education. 2026;23(1):25-38 Figure 4. Mechanism for overcoming subjectivity through SAR verification Source: created by Тimur М. Bosenko. Босенко Т.М., Садыкова А.Р. Вестник РУДН. Серия: Информатизация образования. 2026. Т. 23. № 1. С. 25-38 Figure 5. Dependence of labor costs on analyzed material amount Source: created by Тimur М. Bosenko. Limitations and prospects for further development. The current version of the methodology has a number of limitations that determine the directions for further research. The first limitation is related to the need to form sufficiently large reference datasets for training SAR agents: to achieve high accuracy in feature recognition, at least 500 hours of labeled video material is required for each feature category, which requires significant expert resources at the initial stage. The second limitation is due to the technical capabilities of modern computer vision and natural language processing systems: automatic recognition of subtle nonverbal features (micro-expressions, complex spatial configurations of interactions) and context-dependent verbal patterns currently achieves an accuracy of F1 = 0.58-0.71, which requires additional expert validation of the results of SAR agents. The third limitation concerns the need to adapt trained SAR agents to different educational contexts: models trained on data from a single subject area or age group [22] may demonstrate reduced performance when transferred to other contexts without fine-tuning [23]. The prospects for further development of the methodology are related to the automation of the verification process and the scaling of SAR agent applications. The first direction involves the creation of self-learning SAR agents capable of automatically replenishing reference datasets through active learning: the agent identifies cases with low prediction confidence and requests expert annotation only for them, which can reduce the need for expert resources by 60-70%. The second direction involves the development of multitasking SAR agents capable of simultaneously recognizing several categories of features and their interrelationships, which will allow analyzing the causal impact on learning outcomes. The third direction is related to the integration of SAR agents into real-time systems to provide teachers with immediate feedback during lessons, which requires the development of effective neural network architectures capable of processing video streams with a delay of no more than 200 ms, and can be used in both intelligent learning systems and adaptive robotic platforms for education. Bosenko T.M., Sadykova A.R. RUDN Journal of Informatization in Education. 2026;23(1):25-38 Conclusion. The proposed scientific and methodological approach based on SAR analysis represents a paradigm shift in the verification of pedagogical features: from passive observation of correlations to active experimental proof of causality. The methodology solves a fundamental problem of traditional approaches - the inability to isolate the influence of individual features in the complex environment of a real classroom. The methodological contribution of the work is the development of a three-step procedure for translating multimodal pedagogical features into SAR agent training parameters, enabling automated recognition and verification of these features. The empirical contribution consists in demonstrating that comparative analysis showed a reduction in verification labor costs during scaling by more than 79% (from 808 to 167 person-hours per 500 videos) when switching from observational to experimental methods and increasing the objectivity of the assessment from ICC 0.60-0.75 to a statistically controlled experiment. The practical contribution is expressed in the creation of a context-enriched reference dataset template that allows not only observable features to be recorded, but also the critically important pedagogical context for training SAR agents. The conceptual contribution lies in the justification of a mechanism for overcoming the subjectivity of expert annotation through multiple SAR verification, which forms an objective knowledge base for training automated feature recognition systems. It is important to emphasize the systemic nature of the proposed approach. SAR analysis is not an end in itself, but a critical stage in a larger architecture for building evidence-based pedagogy. The knowledge base created through experimental verification is the foundation for training SAR agents capable of automatically recognizing effective pedagogical practices. The ultimate goal of the research program is to integrate trained SAR agents into the PPI platform for automatic analysis of pedagogical activities in real time. Such a system will be able to instantly recognize proven effective techniques in teacher behavior, automating the work of expert methodologists and providing teachers with timely, objective, and scientifically based feedback for professional growth. Moreover, trained SAR agents can be integrated into robotic educational platforms, expanding the possibilities of adaptive learning. The proposed methodology paves the way for the digital transformation of teaching theory and methodology, creating a technological basis for the transition of pedagogy from art to evidence-based science and the formation of a new generation of intelligent learning systems based on experimentally verified principles of effective learning and automated recognition of pedagogical features through SAR agents.

About the authors

Timur M. Bosenko

Moscow City University

Email: bosenkotm@mgpu.ru
ORCID iD: 0000-0002-5375-096X
SPIN-code: 7117-2458

Associate Professor at the Department of IT, Management and Technology, Institute of Digital Education

28 Sheremetyevskaya St, Moscow, 129594, Russian Federation

Albina R. Sadykova

Moscow City University

Author for correspondence.
Email: sadykovaar@mgpu.ru
ORCID iD: 0000-0002-1413-200X
SPIN-code: 7107-7576

Professor at the Department of Informatics, Management and Technology, Institute of Digital Education

28 Sheremetyevskaya St, Moscow, 129594, Russian Federation

References

Supplementary files