Assessment of ESP students’ learning outcomes in a digital learning environment

Full Text

Problem statement. The pandemic period associated with the transfer of educational processes to distant format aroused great interest of educationalists in the theory and practice of ESP teaching in a digital environment. However, the problem of ‘digital’ assessment of ESP competence-based learning outcomes has not received sufficient scientific coverage. In the context of a new educational paradigm, which implies ‘digital transformation’ of universities, the issue of ‘digital’ assessment needs to be studied in details. Over the last decade there have appeared a wide number of scientific papers on teaching methodology in a digital environment. M.E. Vaindorf-Sysoyeva et al. (2020) highlighted the issue of creating a virtual educational environment.[3] V.I. Blinov et al. (2019) put forward the concept of digital didactics as a scientific discipline on the organization of educational processes in a digital society [1]. In the context of linguistic education, the term ‘digital linguodidactics’ is being increasingly used. This term means a scientifically based system for organizing learning in a digital environment. Professor S.V. Titova (2017, 2022) has made a significant contribution to the development of this branch of didactics, with some aspects of digital assessment being considered in her scientific works [2; 3]. At present, teaching ESP to non-linguistic students is regulated by leading methodological approaches Language for Specific Purposes (LSP) [4] and Content Language Integrated Learning (CLIL) [5]. The choice in favor of CLIL, which is often made by foreign language departments, is focused on the parallel acquisition of a subject-specific foreign language and disciplines of the professional cycle [3]. Despite the difference in goal setting, LSP and CLIL are both interdisciplinary, which implies indirect assessment of both components of subject-language integration. This, in turn, requires development of interdisciplinary evaluation tasks. LSP and CLIL methodology involves a departure from the traditional ‘norm-referencing approach’ to evaluation[4], in which the objects of evaluation are compared with a certain standard (the norm), to formative assessment. An appeal to formative assessment based on the theory of constructivism (L. Vygostky, 1934, J. Piaget, 1969) and the concept of ‘Mastery learning’ (B. Bloom, 1956, M. Scriven, 1967) is due to the desire to create conditions for students’ activity-based learning and their conscious interpretation of cognitive experience [6]. The concepts that are mentioned above are in line with the provisions of competence-based approach, requiring adequate assessment of learners’ competences as the expected learning outcomes. Assessment of competencies is a complex problem of educational methodology. With a huge number of scientific papers devoted to this issue, there is still no consensus on how to evaluate it. Nevertheless, it is obvious that with a competence-based approach, both knowledge and practical skills should be evaluated. Otherwise, it is impossible to talk about achieving the goal - formation of competences. In competence-based education the expected learning outcome of ESP training is formation of foreign language professional communicative competence, which represents the ability to “use a foreign language for solving professional tasks” [7]. Undoubtedly, ESP course evaluation should combine knowledge and skill-based communicative tasks based on clearly defined criteria. However, the analysis of ESP assessment practices shows that even in offline format teachers tend to simplify assessment procedures, perceiving competence-based learning outcomes as an abstract concept and therefore making a final judgement on students’ achievements based on tests. This approach is not consistent with either LSP and CLIL methodology, or the competence-based paradigm. Being a fundamental indicator of the education quality, assessment should be a constant process of monitoring the educational and cognitive activities of students. Formative assessment meets this requirement, as it is based on the dynamic observation of each student’ academic growth, does not prioritize comparison with the standard/norm and does not use repressive function. According to L.V. Vilkova (2017), formative assessment is a “step-by-step movement of each student to the best learning outcomes through active inclusion in the analysis (reflection) designed to identify difficulties and gaps in a course acquisition and effectively fill them” [8]. In the context of LSP and CLIL, any practical communicative tasks combining subject-oriented content and being performed through a foreign language[5] (debates, problem discussion, case-study, presentation, laboratory experiment report, graph interpretation, diagram analysis, technical instruction, patent application, minutes, elevator pitch, essay, business letter, scenario, leaflet, blog, etc.) can serve as objects of formative assessment [5]. Criteria assessment rubrics are required to evaluate the above tasks. L.V. Vilkova (2017) identifies essential components of formative assessment, which are accepted in this study as constitutive: 1) ‘objectives-results’ dyad, which implies transfer of learning objectives to expected learning outcomes, which is traditional for competence-based approach; 2) collaborative work with students to define the evaluation criteria, which promotes students’ better understanding of the requirements and creates sense of belonging to the assessment process; 3) self-assessment promoting the development of critical thinking; 4) peer assessment aimed at developing objectivity and mutual support; 5) feedback, providing students with recommendations and tips that make them think better. Following L.V. Vilkova we consider it appropriate to complement summative and formative assessment methods, as it allows to get a clearer picture of the program acquisition [8]. In a digital learning environment, it is advisable to implement formative assessment of ESP learning outcomes on the basis of ‘integrated evaluation principle’, highlighted by the authors of digital didactics concept (Blinov et al., 2019). According to this principle, “the traditional assessment process is transformed into a continuous personalized diagnosis-forming assessment of academic success” [1, p. 52]. This requires a comprehensive analysis of new didactic digital tools capable of providing instant feedback, informing a teacher and a student about the course and results of task fulfillment, issuing personalized ‘troubleshooting’ recommendation and adjusting the immediate goals and scenarios for further development [1, p. 52]. The purpose of this study is to determine conditions for effective assessment of ESP students’ leaning outcomes in a digital learning environment. The study took place at Linguistics faculty of BMSTU with the participation of 58 master students of Engineering Business and Management (EBM) faculty studying ‘Business English’. The research materials included 1) didactic materials of Business English course delivered at BMSTU, including case-studies, supplementary audio and video materials[6] and tests developed and approved by Linguistics faculty; 2) 20 modern platforms used for evaluation purposes - Wizer.me, iSpring, Genially, Interacty, PruffMe, Exam, Яндекс.Формы, StudySmarter, Symbaloo, Google Формы, Quizizz, Pear Deck, Kahoot, Typeform, Lumio, Unio, Learning Apps, Online Test Pad, BrandQuiz, Buncee. Methodology. The methodological basis of the research included key theoretical provisions of formative assessment [9-11], interdisciplinary approaches LSP and CLIL [4; 12], digital didactics [1], digital linguodidactics [2; 3], distant learning.[7] Based on the analysis of scientific literature and Russian educational standards (FGOS), the goals and values of formative assessment, as well as the requirements for ESP learning outcomes of master students majoring in economics were identified. Then, a comparative analysis of 20 platforms used for assessment purposes was conducted with a view to select the best one. The selection criteria included simplicity and convenience of the interface, availability of videoconference mode, chat and cloud storage, uploading audio and video files, creating tests, presentations, sharing files and monitoring results online, free access. As a result, the domestic service pruff.me was selected for creating a digital assessment space (DAS), a part of BMSTU digital learning environment needed for implementing formative assessment of ESP learning outcomes. In the course of study, the analysis of didactic and evaluation materials used for teaching ESP master students majoring in economics at BMSTU was conducted. All the materials were adapted to the tasks of formative assessment. In compliance with the formative assessment methodology, evaluation criteria for oral communicative tasks (case-studies, problem discussions, presentations) were identified in collaboration with the participating students. Based on the identified criteria, assessment schemes for the above practical tasks, as well as all other evaluation materials, were developed and integrated into pruff.me. All the materials have been tested by EBM master students in the process of Business English distant learning. Based on the results of evaluation materials testing, methodological recommendations for ESP teachers have been compiled. Results and discussion. Business English course for EBM master students at BMSTU is designed for 1 year and includes 6 modules. During the training, the students are engaged in in the development of ESP speaking, listening, reading and writing skills in the field of business communication. Within the framework of a course, the students extend their knowledge of international business practice, business education, sales and marketing strategies, quality management, financial control. The ESP training is conducted on the basis of communicative method, with a significant time being devoted to case analysis, discussions, role plays and profession-oriented written tasks. Monitoring and assessment of ESP learning outcomes is carried out on the basis of oral and written interdisciplinary assignments (case-studies, presentations, problem discussions, tests). Critical analysis of ESP online assessment practice at BMSTU showed that despite the absence of explicit criticism of ESP control materials used, the process of evaluation seemed to be one-sided, with the main focus made on electronic tests and oral communicative tasks evaluated intuitively by the teachers. This approach allows to record the average level of ESP course acquisition, but doesn’t contribute to students’ conscious interpretation of their academic achievements [13]. The virtual classroom observation confirmed the importance of creating a digital assessment space for ESP teachers and students, that allows to monitor students’ individual progress and make an adequate assessment of learning outcomes in a digital learning environment. In the course of this research evaluation materials of Business English course were brought into line with the requirements of formative assessment and integrated into pruff.me. The description of these materials including criteria-based assessment schemes is given below. One of the most popular competence-based techniques used for educational and evaluation purposes at universities is a case study, a description of a situation containing a problem or a contradiction and based on real facts. Case study is widely used in interdisciplinary foreign language teaching as it demonstrates students’ ability to apply theory for solving practical tasks and ensures content acquisition through emotional involvement. On the platform pruff.me ESP students can collaboratively solve the case tasks, listen to the audio and discuss the results in videoconference mode. The sample of case study placement on the platform is shown in Figure 1. In the process of collaborative development of case studies evaluation criteria, it was decided to move away from the traditional format of evaluation scale (rubric) in favor of a criteria-based assessment scheme. The scheme combines tasks fulfilment criteria and indicators in a single description of four levels that describes ESP students’ ability to solve a profession-oriented communicative task and informs the participants about the assigned points. For the convenience, each level is assigned its own range of points correlating to the point-rating system adopted at Linguistics faculty and to the traditional five-point scale familiar to students. For instance, level 1 corresponds to a score range of 0-4 or ‘bad’, level 2 - 5-6 points (‘satisfactorily’), level 3 - 7-8 points (‘good’), level 4 - 9-10 points (‘excellent’). The sample of case assessment scheme is presented in Figure 2. The final communicative task fulfilled by ESP students participating in this research in the end of each term is individual multimedia presentation. The public defence of a presentation allows the students to demonstrate so-called ‘soft skills’ including an ability to work with the information sources, insight into the problem, an ability to communicate information in an accessible form, establish contact with the audience. On pruff.me, presentation topics are given in a list. The logic of designing a multimedia presentation assessment scheme is similar to the same of the case study (Figure 2). The assessment criteria for presentation are: 1) relevance of the topic; 2) language and communication; 3) logic and persuasiveness; 4) design; 5) contact with the audience. Figure 1. Case study ‘IT Solutions in Business’ Available from: https://pruffme.com/landing/u2696543/tmp1654084672#/26c59f270a6b86468e4de69ae03015d0 (accessed: 15.06.2022) Изображение выглядит как стол Автоматически созданное описание Figure 2. Case-study assessment scheme Available from: https://upload.pruffme.com/download/?media=963b78b3c4c8dee5419fc7492d1fa532 (accessed: 15.06.2022) One more oral communicative task used in this study for evaluation purposes is a problem discussion on the video. Problem discussion is a kind of an argument aimed at achieving the truth, with participants’ argumentation being the most important characteristic of the task execution. In the course of this assignment, a subject-related video is offered for ESP students’ viewing on the platform. Then the teacher asks questions on the content initiating a discussion. Problem discussion on the video implies an accurate understanding and correct interpretation of what was seen and heard. The assessment scheme of this task is shown in Figure 3. Изображение выглядит как стол Автоматически созданное описание Figure 3. Assessment scheme for the discussion on the video Available from: https://upload.pruffme.com/download/?media=5d9c676f88f4d1f6743199e0716e0889 (accessed: 15.06.2022) When fulfilling all oral communicative tasks (cases, presentations, discussions), the students carry out peer assessment by rating their groupmates achievements on the collaboratively developed criteria (see assessment schemes in Figures 2, 3). From the grades received, the teacher deduces the average point-rating value in accordance with the results of each student. According to the idea of formative and summative assessment tools’ complementarity proposed by L.V. Vilkova, tests were also included in the set of our evaluation tools and placed on pruff.me. In case of misunderstanding or difficulties encountered when performing the test, students have an opportunity to communicate with the teacher in a chat, where every participant can attach a file, record/send an audio or a video reply. The teacher in turn can see an avatar and a name of every student, which makes it much easier to navigate on the platform. Points are awarded for the correct answers to the test questions and summed up with the points received for oral communication tasks. The resulted value is taken as a final student’s score. An opportunity to chat with a teacher on the platform allows students to provide feedback. When clicking on the ‘Message section’ icon, the teacher can see a window with the list of students and their module assignments completed. The students’ messages in the chat are tied to control activities, which greatly facilitates the teacher’s checking work and allows to monitor every student’s academic achievement in dynamics. Изображение выглядит как стол Автоматически созданное описание Figure 4. ESP student self-assessment check-list Available from: https://upload.pruffme.com/download/?media=a6eb11397d60d244d2343ffbd97941c1 (accessed: 15.06.2022) Another important component of formative assessment is self-assessment. The sample of self-assessment checklist filled by each student in the end of the 1st and 2nd terms is presented in Figure 4. The proposed self-assessment check-list placed on pruff.me is designed on the principle of Likert scale, which allows the respondents to carry out a qualitative evaluation of the educational process, students’ perception of the course and also, if necessary, transfer the obtained qualitative data to numerical form. Qualitative characteristics are given in the form of statements, evaluating participants’ achievements during the period under review on the Likert scale from 1 to 5 (‘Over the past semester I’ve learned how to…’), the quality of the course and the teacher’s work (see statements 1-3 in Figure 4). These statements characterize the learning outcomes from the students’ perspective. The developed formative assessment tools were offered to the students participating in the research and caused a positive response. The reviews written by the participants in a free form contained the following comments: ‘it was convenient to chat with the teacher, get explanations’, ‘evaluating each other is a bit tedious, but useful, you pay attention to the details’, ‘it was important to evaluate yourself at the end of the term, it becomes clear what worked and what didn’t’, ‘it’s nice that the teacher is interested in my opinion about the course’. The experience gained during the described research showed the need to compile methodological recommendations for ESP teachers willing to implement formative assessment technology in a digital learning environment. The recommendations represent a step-by-step description of ESP teachers’ actions, taking into account the formative assessment logic and values [14] and technical characteristics of the platform. Conclusion. Within the framework of the performed study, it was defined that ‘digital’ assessment of ESP students’ leaning outcomes requires creation of a digital assessment space, a part of a university digital learning environment, based on a thoroughly selected platform that allows to place all evaluation materials and implies various modes of teacher-students online interaction. In order to implement ESP ‘digital’ formative assessment effectively, it is necessary to fulfill the following conditions: 1) take into account theoretical provisions of ESP&CLIL underlying methodology, when developing evaluation materials; 2) consider formative assessment as a dynamic observation of every student’s academic growth aimed at identifying difficulties and filling the gaps, not at simple measurement of average results; 3) perform the evaluation procedures in accordance with formative assessment essential components, not excluding any of them. In conclusion, implementing formative assessment technology has had a positive effect on ESP students’ motivation making them the center of evaluation process. The ESP teacher, in turn, has been informed about the problems and got an opportunity to correct the course and teaching methods. The prospects for the conducted research lie in the field of designing digital assessment space as an essential component of universities’ digital learning environment and ESP teacher professional development.

About the authors

Kira M. Inozemtseva

Bauman Moscow State Technical University

Author for correspondence.
Email: ikm@bmstu.ru
ORCID iD: 0000-0003-3143-0671

PhD in Education, Associate Professor, Head of Romano-Germanic Languages Department, Faculty of Linguistics

5 2-ya Baumanskaya St, Moscow, 105005, Russian Federation

Elizaveta V. Morozova

Bauman Moscow State Technical University

Email: lizatih@mail.ru
ORCID iD: 0000-0001-7028-9311

senior lecturer, postgraduate student, English for Industrial Engineering Department, Faculty of Linguistics

5 2-ya Baumanskaya St, Moscow, 105005, Russian Federation

Ilya M. Kolesnikov

Bauman Moscow State Technical University

Email: krik.44442@rambler.ru
ORCID iD: 0000-0002-2240-7990

master of Linguistics, Faculty of Linguistics

5 2-ya Baumanskaya St, Moscow, 105005, Russian Federation

References