Learning analytics in Russia and abroad: level of development, trends and prospects

Full Text

Problem statement. Learning analytics is a relatively new multi-disciplinary field of science located at the intersection of pedagogy, computer science and psychology. It is “the measurement, collection, analysis, and reporting of data about learners and their contexts, for the purposes of understanding and optimizing learning and the environments in which it occurs” [1]. In literature, one can find the terms Educational Data Mining (EDM) and Learning Analytics (LA). Although these branches differ in the methods used, they have the same object of study and the same ultimate goal - to improve the quality of education. Therefore, in the context of this work, we will consider EDM and LA synonymous. Due to the uneven distribution of the level of digitalization of world economies, the level of IT technologies, the peculiarities of national policies in the field of education, the current state of art in the field of learning analytics differs. Our study aims at reporting about the current level of LA development and implementation of its tools to learning process, detecting global trends and challenges. We pay special attention to the level of scientific discourse on LA in Russia and national initiatives that can be drivers of LA enhancement. Methodology. The paper presents the results of context analysis of scientific articles and research reports on the world’s best practices of LA and EDM. As learning analytics is an in multidisciplinary field we analyzed several aspects of its development - methodological, technological and application-oriented. Leading positions, challenges. To assess the level of scientific discourse on learning analytics in Russia we reviewed the publications of Russian-speaking authors on the topic. For this purpose, on the electronic platform eLIBRARY.ru, we selected publications in Russian (as well as in English with Russian-language annotations) using the keywords related to the field of study, and performed a quantitative analysis of the dynamics of research interest to LA end EDM. The qualitative analysis the open sources was made to detect main directions of academic interest and find successful cases of LA applications to educational process. Results and discussion. Global trends and challenges. The formation of learning analytics as an independent scientific discipline is associated with the digitalization of education and the accumulation of large volumes of data on the educational process and students. EDM and LA emerged in the 2000s: in 2006 appeared the first monograph on EDM (C. Romero and S. Ventura), in 2008 and 2011 were held the first conferences on EDM and LA. To date, at least three specialized journals are being published - the International Journal of Learning Analytics and Artificial Intelligence for Education, the Journal of Educational Data Mining (indexed in Scopus) and the Journal of Learning Analytics (in Web of Science and Scopus Q1). There are several large actively working analyst communities on the field - the most known are the Society for learning analytics research (SoLAR) and the International EDM society. Educational institutes of the USA are among the most active users of LA tools: they use educational dashboards (for example, the student monitoring panel of Nottingham Trent University, “My Learning Analytics” dashboard in the University of Michigan), early warning systems preventing students’ dropouts (Course Signals system of Purdue University). Recommendation systems based on educational data analysis are implemented at an institutional level (Degree Compass system at Austin Peay University). Universities actively implement data-driven adaptive learning systems (Arizona State University places students into computerized learning courses running on the software of adaptive-learning start-up Knewton). In terms of publication activity, the leading positions are occupied by the United States, Spain, the United Kingdom, Australia, Germany, Canada, India, the Netherlands, Japan, and China [2]. A significant increase in research and publication activity on the LA topics can be expected in European countries due to the development of national policies, the launch of joint projects on a European level and the creation of competence centers [3]. Initially, the most intensive development was observed in such areas of learning analytics as predictive analysis of learning outcomes and monitoring students’ learning progress. These topics are still trending and in several counties, their development is supported by national policies. For example, in the United Kingdom, higher educational institutes are required to produce detailed metrics for the government-established Office for Students [4]. There are numerous research articles devoted to the application of machine learning methods to forecasting student success, learning outcomes or dropouts [5]. Learning Analytics Dashboards become more and more widespread especially at Higher Educational institutions. At the same time, new important directions for the development of LA are gaining importance, such as data-driven analysis and support of self-regulated learning, provision of personalized feedback and support for learners, development of recommendation and early warning systems. The effectiveness of the implementation of the mentioned LA instruments in a real educational process has recently become the main challenge in LA. As it is mentioned in [5] although data-driven optimization of the learning process is the final stage of the learning analytics cycle, most of the research is still devoted to developing predictive models rather than to designing LA interventions. There are several possible reasons why the data-driven interventions into the learning process are still not widespread, which are related both to students and course instructors. On the one hand, if the interventions are aimed at supporting at-risk students they can face the problem of weak metacognitive skills and self-regulation of such students [6]. On the other hand, implementation of supportive interventions means an additional workload for course instructors and teachers, which is usually not regulated. Another challenge is due to the dual nature of LA. As methods and tools of LA come from IT and data science LA instruments are usually designed by IT specialists not much familiar with pedagogical and psychological aspects of the learning process and peculiarities of the studied disciplines. This results in unjustified simplification when modelling such objects as “proper learning behaviour”, “learning style”, “student online interactions” and even “learning performance”. In [7] authors argue that to be effective LA instruments should be grounded in robust and clearly articulated theories of learning. Unfortunately, educators who are competent in the disciplines they teach and more or less familiar with learning theories usually have difficulty developing data analysis tools on their own. However, the technological aspect of LA itself is also problematic. Big Data techniques well-proven for other fields cannot be automatically applied to an educational context. For instance, in physical sciences data is collected in highly controlled, carefully designed and replicable experiments, while educational data is collected incidentally and in circumstances that differ for every single user-resource or user-user interaction [7]. The replicability of such experiments is not only doubtful but also contradicts the overall purpose of LA which is to optimize the learning environment. The data mining technique from other fields should be carefully adapted to LA taking into account both the subjectiveness of education data and goals of data collection and analysis. Learning analytics in Russian Federation. In Russia, data-driven educational analytics have started to develop actively in the last decade due to the later spread of e-learning methods, the introduction of LMS in the learning process of educational institutions, and the emergence of massive online courses. To assess the level of LA development and implementation of LA instruments in Russia, we highlight the following aspects of this process: The research aspect that concerns the development of new or adaptation of already existing methods for educational data collection and analysis. The research topics include identifying predictors of learning success, building student models, designing learners digital twins, creating and testing methods for managing the educational process, designing supportive data-driven interventions, monitoring learners track and performance in e-learning courses, etc. The environment transformation aspect, which means digitalization of the learning process in the educational institutions, the introduction of information systems for collecting educational data with in-built analytical tools at the level of universities and above, organizing access for all stakeholders of the educational process to monitoring data and the results of their tracks’ analysis, the transformation of institutional policies to ensure the possibility of automatizing some processes, conducting targeted corrective measures based on the analysis of the student's educational footprint, etc. The legal regulation aspect that implies modernization of the state policy in the field of education: changing educational standards towards increasing its flexibility, developing regulatory norms for conducting analytics and making data-driven decisions in the educational field. Although development can proceed simultaneously in all directions, we believe that any administrative and state initiatives can be successful only if academics understand the goals and methods of transformations, and if empirical research is carried out and actively discussed. The level of scientific discourse on learning analytics in Russia. On the electronic platform eLIBRARY.ru, we selected publications in Russian (or with Russian-language annotations) using the keywords (in Russian): “educational analytics”, “earning analytics”, “analysis of educational data”, “educational data mining”, “methods (methodology) for educational data analysis”, “methods for educational data mining”, “methodology and technology for educational analytics”, “Big Data in education”, “learning success prediction”, “ learning success modelling”, “student digital footprint”. Then duplications and irrelevant results were removed, and the remaining 187 publications from the resulting collection were divided into five categories: 1) articles indexed in WoS and/or Scopus; 2) articles indexed in RSCI (Russian Science Citation Index - on the Web of Science Platform) not included in category 1; 3) articles in journals included in the list of the Higher Attestation Commission (VAK) not included categories 1 and 2; 4) articles in journals indexed in the RINC (Russian Index of Scientific Citation), not included in categories 1, 2, and 3; 5) proceedings of conferences, indexed in the RINC. The first publication on the LA in Russian appeared in 2013, then the number of publications increased and in 2020 it resulted already in 40 publications per year (Figure). In the period January - October 2022, 42 articles have already been published. We think that a significant increase in the number of publications in 2020-2022 is due to the acute problem of monitoring the educational process in the period of COVID-19 restrictions. Изображение выглядит как текст, снимок экрана, диаграмма, График Автоматически созданное описание Dynamics of the number of publications on LA on eLIBRARY.ru from 2013 to October 2022 Source: compiled by the authors. After reviewing the content of the open-source publications, we found that most of the papers are just announcing the possibility of using learning analytics in education, providing basic information about its goals, objectives and tools. For an initial acquaintance with the essence of educational analytics, the article [8] seems to be the most useful for a novice researcher. The most complete overview of foreign experience in the use of LA tools can be found in [9; 10]. The most popular among Russian researchers LA problem is the prediction of learning success. In our list we found 14 papers, describing cases on solving this problem by means of various methods of statistical analysis and machine learning. The authors predict academic performance at the level of an academic course or MOOC (Massive Open Online Course), the success of university graduation, and the level of competencies development. At the same time, in 8 works, researchers provide full descriptions of empirical studies and determine the most effective predictive methods. We consider the papers [11; 12] to be the most informative on the topic. Several studies are devoted to identifying patterns in the learning behaviour of students and describing their learning strategies with a further purpose to use the results for designing individual learning trajectories. In this case, various clustering algorithms are used and compared, as, for example, in [13]. Only in 6 publications authors describe learning management tools built on the basis of already designed and implemented in the educational process LA instruments. The described tools are used for data collection in the e-learning environment and for analysis of the collected data. These tools are implemented either at the level of individual academic courses [14] or at the university level [15]. Most of the articles describe, discuss, or explore various aspects of educational data mining as applied to higher education. Only 4 publications from our list are devoted to studies of educational data in school (for example, in [16]). Five publications deal with MOOC learning analytics (for example, in [17]). Russian-speaking researchers also pay attention to the methodological and algorithmic basis of educational analytics. They introduce new quality metrics, approaches to the adaption of data analyses and modelling methods to the analysis of the educational process and learners. For instance, in [18] authors propose a methodology for the automated assessment of programming tasks in an electronic environment that is based on statistical analysis. In [19] various probabilistic methods are used for student success modelling, as well as for learning activities assessment. Transformation of educational environments and national policy. Digitalization of education has been declared to be one of the strategic tasks of the Russian Federation as an aspect of the digitalization of economics. Recent years there appeared several national initiatives aimed at development and establishment of regulatory rules and guidelines for digitalization and usage data for educational research. One of such initiatives (University-2035) announces developing approaches to data-driven educational management as one of its priority tasks. One of the important products of University-2035 in the LA field is a Digital Footprint Standard, which covers the rules for sampling, processing and use of the digital footprint in education.[5] In 2019 there was launched the Consortium “Evidence-based digitalization for student success” which states providing analytical support for digitalization for making informed and effective management decisions based on research data as its main goal. About 20 universities have joined the consortium at the moment. One of the Consortium projects, Learning Analytics Research Community, unites academics, education centers’ managers and independent researchers at a platform for communication, sharing educational data and LA-algorithms. Hopefully, the mentioned initiatives will result in significant modernization of policies or in national level research projects. Conclusion. The importance of learning analytics is constantly growing together with digitalization of education, providing educators promising prospects of the data-driven learning process optimization. However, there are several problematic aspects of its development - pedagogical, technological and ethical. Learning analytics in Russia is gaining popularity. There are some examples of successful implementation of some LA tools in the educational process, but in general, the scientific and pedagogical community in Russia is just getting acquainted with this field. Nevertheless, there are several encouraging signals coming both from individual researchers and at the state level. The number of Russian-language publications on the topic is rapidly increasing. Research and educational communities focused on the development of technology-enhanced education have a good chance of receiving funding and other support due to the concerns about the distance education effectiveness that are currently strong in both academic society and the authorities.

About the authors

Tatiana A. Kustitskaya

Siberian Federal University

Author for correspondence.
Email: tkustitskaya@sfu-kras.ru
ORCID iD: 0000-0001-9854-1259
SPIN-code: 5202-8701

Candidate of Physical and Mathematical Sciences, Associate Professor, Associate Professor of the Department of Applied Mathematics and Computer Security

79 Svobodnyi Prospekt, Krasnoyarsk, 660041, Russian Federation

Mikhail V. Noskov

Siberian Federal University

Email: MNoskov@sfu-kras.ru
ORCID iD: 0000-0002-4514-7925
SPIN-code: 3957-7221

Doctor of Physical and Mathematical Sciences, Professor, Professor of the Department of Applied Mathematics and Computer Security

79 Svobodnyi Prospekt, Krasnoyarsk, 660041, Russian Federation

References