Evaluation of digital competence level among educators: assessment tools

Full Text

Problem statement. The 21st century is dominated by technology, permeating almost every aspect of our lives, including communication, socialization, learning, and teaching. According to a report by the European Commission, nearly seven out of ten individuals believe they possess sufficient skills to utilize digital devices that are integrated into their daily routines[9]. On a personal level, 40% of people are willing to share their data to enhance medical research and care, provided it is done securely. In emergency situations, such as natural disasters or terrorist attacks, around 30% of individuals are also willing to share their data. Additionally, more than 60% of people expressed a desire for a single digital identification system for all online services. However, despite the digital prevalence, the Digital Economy and Society Index (DESI) analysis revealed that only approximately 54% of individuals aged 16-74 possessed basic digital skills, despite the fact that 87% regularly used the Internet in 2021[10]. The Netherlands and Finland are leading the way in terms of digital proficiency within the European Union (EU), while Romania and Bulgaria are lagging behind. Consequently, a significant portion of the EU population lacks fundamental digital skills, despite the fact that most jobs now require such abilities. The Path to the Digital Decade proposes a goal for the year 2030, aiming to ensure that at least 80% of EU citizens possess basic digital skills. The impact of information and communication technologies (ICTs) on literacy. Over the past few decades, ICTs have revolutionized literacy practices [1]. In response to the pandemic, most educational institutions shifted their teaching methods to online platforms, leading to an intensified reliance on technology within the education sector and various positive impacts [2]. This transition towards a competency-based curriculum, prioritizing students' specialized skills and capabilities, has also necessitated the development of digital literacy among educators - an essential pedagogical challenge [3; 4]. Online education requires educators to acquire proficiency in new interactive learning software, convert physical course materials into digital formats, and create personalized learning pathways [5]. The significance of digital literacy. Digital literacy is increasingly becoming a crucial competence, encompassing the ability to interact, communicate, and collaborate effectively through digital technologies, while acknowledging cultural and generational diversity [6]. The term “digital literacy” was introduced by Gilster [7] in the late 1990s as the ability to understand and use information from various sources via computers. Several publications refer to this definition as the “know-how”. Joosten et al. [8] defines digital literacy as adapting skills to a new medium and mastering core competency. Similar to media literacy and computer literacy, digital literacy is skill-based and relates to functional use of technology and skills adaptation. Recent publications focus on cognitive skills and competences in defining digital literacy. Bennett [9] and Traxler and Lally [10] emphasize the individual's role in digital literacy. Beetham's definition highlights cognitive perspective as access, skills, and practices to confidently use technologies. Chan et al. [11] also stress critical thinking over ICT skills in understanding and using information in various formats. Recent studies propose a shift in focus from digital literacy to a broader model that encompasses diverse knowledge, capabilities, and dispositions required for prospective teachers [12]. Differentiating digital literacy and digital competence. In the literature, digital literacy is often used interchangeably with digital competence (DC); however, they have distinct origins and meanings. Krumsvik [13] describes digital competence as teachers' skill in using ICT with effective pedagogical discernment. Scuotto and Morellato [14] characterize DC as the capability to navigate and confront new technological challenges in an adaptable manner. According to Cazco et al. [15], digital competence is comprised of values, beliefs, knowledge, skills, and attitudes necessary to utilize technology appropriately. The initial definition emphasizes the proficiency of higher education teachers in using ICT and its impact on student learning, whereas the latter definitions focus on cognitive skills, including the capacity to resolve ethical dilemmas and to cultivate or exploit opportunities to achieve digital competence. According to Martin and Grudziecki [16], digital literacy forms the basis of DC. In 2013, the European Commission published the Digital Competence (DigComp) Framework, which comprised five areas and 21 competences, including the term digital literacy [17]. DC is also recognized as one of the European Union's eight key competences for lifelong learning, encompassing multilingualism, mathematics, science, technology, engineering, literacy, digital competence, personal, social and learning to learn competence, as well as citizenship, entrepreneurship, and cultural awareness[11]. It is defined as the confident, critical, and responsible usage and engagement with digital technologies in the contexts of learning, work, and societal participation. Evolution and implementation of the digital competence framework. Over the past decade, the DigComp Framework adopted an incremental approach, progressing from DigComp 1.0 to DigComp 2.2 through four iterations[12]. Upgrades in the latest version include fact-checking online content and sources, consideration of remote or hybrid work contexts, digital accessibility, green and sustainability aspects of digital technology interaction, well-being and safety considerations, and citizens' interactions with AI systems and data literacy. Alongside these developments, various offshoots and variations have emerged, such as DigCompOrg, specifically designed for educational organizations to facilitate self-reflection on their progress in integrating digital learning technologies effectively. The Joint Research Centre's DigCompEdu Framework provides a general reference to support the development of educator-specific digital competences in Europe, catering to both personal and professional needs as well as learners' demands for digital skills improvement. The Digital Decade policy program's objective is to ensure that 80% of the population possesses basic digital skills. Dimensions and applications of the DigComp Framework. The DigComp Framework is a comprehensive yet adaptable framework comprising five major dimensions: competence areas, competence sub-content areas, proficiency levels, examples of knowledge, skills, and attitudes for each competence, and examples of their use in education and employment scenarios. This framework has been widely implemented and utilized within various platforms and applications. For example, the Women4IT project, an international partnership project under the Youth Employment Programme of the Norwegian, Icelandic, and Liechtenstein Foundation, has created Open Educational Resources aligned with the DigComp Framework[13] These resources can be searched based on different criteria, including language, job profiles, type of material, and areas of competence. The project's objective is to increase the number of EU vulnerable girls and young women participating in the digital agenda. Another example is the Digital Competence Wheel, developed by the Center for Digital Dannelse in Denmark, which specializes in digital formation and DC. This project aims to provide a self-assessment overview of existing digital competences and areas for improvement, offering concrete inspiration for enhancing the most relevant digital skills. By reviewing DigComp documents, the project team created an intellectual map comprising over 300 digital competences, which were subsequently condensed into 52 statements for evaluating 16 core competences in the areas of information, communication, production, and safety. Methodology. The aim of this research is to acquire a more profound comprehension of the assessment of educators' digital competence (DC) and to identify the most rigorously validated assessment instruments for measuring DC. To accomplish this objective, we conducted a theoretical analysis to locate contemporary empirical studies focused on evaluating the level of DC among educators. Subsequently, we employed the most prevalent customizable self-reflection tools for DC testing. These tools aid educators by allowing them to identify their strengths and weaknesses in digital skills through the aggregation and visual display of responses as personalized feedback. Specifically, we utilized three tools that are administered by the European Commission and are available in all official languages of the European Union. These tools include SELFIE for schools, which gathers perspectives from students, teachers, and school leaders[14]; SELFIEforTEACHERS, a toolkit designed for primary and secondary school educators that employs a six-point progression model to assess competency proficiency levels (awareness, exploration, integration, expertise, leadership, innovation)[15]; and the Digital Competence Wheel, an online testing tool based on the European Digital Competence Framework (DigComp)[16]. Results and discussion. In this section, an evaluation has been conducted on three digital competence platforms (SELFIEforSchools, SELFIEforTEACHERS, and the Digital Competence Wheel), with one of them (the Digital Competence Wheel) undergoing testing. These three tools can be accessed through the UNESCO UNIVOC platform[17]. This platform houses a comprehensive database of digital competence frameworks that provide definitions and discussions on the necessary digital knowledge, skills, and attitudes for individuals to become digitally competent educators, learners, or citizens[18]. Presently, the database includes 29 frameworks that vary in terms of titles, descriptions, target groups, origins, and publication years. The earliest framework, Skills Framework for International Age[19], was established in 2000 by the SFIA Foundation, a globally recognized nonprofit organization. Its objective was to outline the fundamental digital skills and competencies required across multiple sectors, including education and learning. It has since been embraced by governments, corporations, and individuals in nearly 200 countries and is available in 13 languages. The most recent frameworks were published in 2022. One such framework is DigComp, which describes digital competences for individuals and professionals. It originated from the European Union and is targeted towards policy makers, teachers, trainers, employers, and unions. The database encompasses frameworks that possess both theoretical and practical applications. They serve as tools for describing and enhancing the digital competence of individuals and professionals, developing methodologies to measure these competences, and establishing a shared conceptual framework for competences. Some frameworks provide guidance on teaching the necessary skills and aid educators in integrating technology and digital literacy-related activities into their classroom practices. Others focus on defining the skills required by individuals, educators, and professionals in the 21st century. For example, the Indonesian National Digital Literacy Framework focuses on addressing the challenges of hoaxes, cyberbullying, and online radicalism, which are identified as the most significant potential dangers in Indonesia. There are also platforms designed specifically for teachers: SELFIE, ProFuturo[20], and students - ISTE[21], SELFIE offers an online, interactive tool that enables teachers to receive feedback on their utilization of digital technology in their work. ProFuturo is a digital platform aimed at enhancing teacher training and student learning. The International Society for Technology in Education Standards for students (ISTE) empowers students to voice their opinions and ensures that the learning process is student-driven. These frameworks have been developed at the global level by UNESCO, national level by the European Union, North America, South Africa, Kenya, Singapore, Norway, Australia, and Indonesia, and regional levels by UNESCO UNIVOC, profit organizations such as Microsoft and McKinsey, and nonprofit organizations like the SFIA Foundation. The objective of this paper is to provide a concise review and assessment of three tools presented by the European Commission: SELFIE for Schools, SELFIE for TEACHERS, and the Digital Competence Wheel. SELFIEforSchools. It is an online assessment tool, designed by the European Commission with input from 5,000 education experts, practitioners, and students from European countries. It has been thoroughly tested by over 67,000 school leaders, teachers, and students across 14 European countries in 2017, and their feedback has contributed to its development. Available in over 30 languages, SELFIE can be utilized by primary, secondary, and vocational schools throughout Europe. It consists of a series of questions targeting school administrators, teachers, and students regarding the integration of digital technologies within their educational institutions. Administrators are asked about strategies and policies related to digital technology usage, while teachers are surveyed on their applied teaching methods. Students, on the other hand, are encouraged to share their experiences with digital technologies in learning. Consequently, the enhancement of digital competence within a school relies on both organizational factors, such as leadership and infrastructure, and pedagogical approaches and instructions. The questions provided in SELFIE are categorized into eight distinct areas: leadership, collaboration and networking, infrastructure and equipment, continuing professional development, pedagogy support and resources, pedagogy implementation in the classroom, assessment practices, and student digital competence. Furthermore, schools have the option to personalize their questionnaires and include up to 10 additional questions. Each user group is assigned a specific number of questions within each area. For instance, the infrastructure and equipment section contain thirteen questions for school leaders and teachers, and six questions for students. The answers provided by all three groups are then compiled into a comprehensive report. It is worth noting that a higher response rate leads to a more accurate and valuable report. The result section of SELFIE presents the findings of each area and user group in a color-coded format. School leaders are indicated by the color blue, teachers by red, and students by yellow. This differentiation aids in the interpretation of the results. The overview section for each area provides the average rating given by each user group. However, users can access more detailed information through pop-up windows and symbols, such as the number of questions answered in that area by each user group, the specific results for that area, the average rating per user group, the number of users who provided responses, the rating scale utilized, and the breakdown of answers presented as percentages. The results for each user group are presented in a wheel format, with each segment representing a question from SELFIE and each area being color-coded. It is important to note that only school administrators have access to the full report, although any part of it can be printed in any language. The results obtained from SELFIE offer a snapshot of a school's current standing in terms of its use of digital technology for learning. This assessment also helps identify areas that require improvement in terms of digital implementation. The findings can be utilized for self-reflection, initiating discussions, and developing action plans aimed at enhancing the use of digital technologies for more effective teaching and learning practices. A student from Spain claims: “It is a great opportunity for us, and it is very important that our viewpoints are valued. We are the key people for building, step-by-step, a better use of technologies for learning”. A teacher from Italy argues: “We have clearer goals now and we are working to address the different viewpoints that we saw through the SELFIE school report”. A school leader from Ireland supports them: “SELFIE made consensus easier because it prompted us to have detailed discussions about digital learning” [18]. In general, SELFIE possesses a notable abundance of advantages when compared to its drawbacks. Shifting our attention towards the former, SELFIE can be characterized as: ● comprehensive - it involves the whole school community (school leaders, teachers and students); ● customizable - school can select and add questions and statements to suit its needs; ● matching experience - it allows all participants to answer questions that match their experience, as students, teachers or school leaders; ● free and anonymous - it is free of charge; answers are anonymized and data is secure; ● tailor-made - each school receives a personalized, interactive report which provides both in-depth data and quick insights into strengths and weaknesses. Nevertheless, a significant drawback of SELFIE development lies in the absence of an option for early childhood and higher education. SELFIE is currently limited to three formats, specifically designed for primary, secondary, and vocational schools. However, the integration of ICT in higher education is no longer a mere choice, but rather an imperative need. In fact, Mamdouh Alenezi et al. [18] argue that digital technologies not only enhance teaching and learning, but also foster student engagement, reduce educational disparities, and enable flexible learning anytime and anywhere. Therefore, educators in higher education would greatly benefit from the introduction of SELFIE for this level, as it would allow for a comprehensive assessment of various aspects related to digital technology applications in higher education. SELFIEforTEACHERS. The COVID-19 pandemic has prompted a shift towards remote and online learning, thereby creating an increased demand for professional development programs and support among educators to enhance their digital competence [19]. Educators have expressed the need for a self-assessment tool to evaluate their digital competence, which has been developed by the Joint Research Centre of the European Commission. Subsequently, this tool, known as SELFIEfoTEACHERS, has undergone enhancements and additional support features, originating from its initial version. Its official launch took place on World Teachers Day in October 2021, following a pilot phase involving over 4,000 teachers across five countries. The tool is now accessible in more than 20 languages. SELFIEforTEACHERS is anchored on the DigCompEdu Framework[22]. It serves as a self-reflection tool that aids in the professional growth of teachers' digital competence. Its primary objective is to assist teachers in understanding how digital technologies are utilized in teaching and learning, while facilitating the planning of collaborative professional learning activities with their peers. Moreover, it is designed to support school leaders in fostering the further development of teachers' digital competence. Additionally, teacher educators can utilize this tool to guide their students in charting their own learning pathways and designing courses for their students. Lastly, teacher trainers and education authorities can employ SELFIEforTEACHERS to facilitate teachers' advancement in digital competence. Completing the SELFIEforTEACHERS assessment usually takes around 25 minutes. It comprises a total of 32 competences, spread across six different areas of digital competence in accordance with the European Framework for the Digital Competence of Educators[23]. Each area entails between three and nine specific competences, encompassing diverse aspects wherein technology can potentially be applied. ● Professional engagement (organizational communication, online learning environments, professional collaboration, digital technologies and school level infrastructure, reflective practice, digital life, professional learning through digital technologies, professional learning about digital technologies, and computational thinking). ● Digital resources: searching and selecting, creating, modifying, managing and protecting, sharing. ● Teaching and learning: teaching, guidance, collaborative learning, self-regulated learning, and emerging technologies. ● Assessment: assessment strategies, analyzing evidence, feedback and planning. ● Empowering learners: accessibility and inclusion, differentiation and personalization, actively engaging learners, and blended learning. ● Facilitating learners’ digital competence: information and data literacy, communication and collaboration, content creation, safety and wellbeing, responsible use, problem solving[24]. The competencies span a range of levels, from A1 to C2: A1 represents awareness; A2 signifies attempt; B1 denotes usage; B2 involves analysis, reflection, and redesign; C1 entails support and involvement; and C2 represents initiation and contribution. Teachers are given the opportunity to select the level that best aligns with their current digital competence. Following the completion of all assessment items, teachers receive an automated report detailing their results (A1 - newcomer, A2 - explorer, B1 - integrator, B2 - expert, C1 - leader, C2 - pioneer), along with recommendations for improvement and reaching the next level of competence. Additionally, they are awarded a certificate and digital badge for their participation[25]. SELFIEforTEACHERS offers teachers the ability to establish or join a group, plan collaborative learning activities, exchange practices, and track their progress over time or compare it to the group and global average outcomes. Numerous benefits are associated with this tool, as evidenced by a testimonial from an Irish teacher who states, “It is arriving at the perfect moment. As we transition post-COVID, we are considering how we can meaningfully engage teachers in developing their digital capabilities and competence. One major challenge is to have a clear understanding of their existing competence level. This tool truly benefits teachers”[26]. However, an important drawback of the tool is its dependency on an EU account for usage. We hold the belief that designers are continually working on enhancing the tool. An example of this is the Knowledge Center Digisprong[27], which is a free self-reflection tool based on SELFIEforTEACHERS and the DigCompEdu Framework. Notably, it goes a step further by linking the suggestions in the SELFIE report to training courses. Consequently, teachers receive personalized feedback along with a defined educational pathway. It is crucial to acknowledge that SELFIE for TEACHERS primarily serves as a self-reflection tool that presents subjective data and does not provide a comprehensive and validated 360-degree assessment of teachers' digital competence. Digital Competence Wheel. The Digital Competence Wheel is a practical and effective tool that provides a comprehensive evaluation of an individual's digital competences. It assesses their strengths and weaknesses in the digital realm and offers suggestions for improvement. The process of creating a personal Digital Competence Wheel takes approximately 15 minutes. Participants are asked to respond to a series of 52 questions and rate themselves on a 7-star scale, selecting the description that best matches their abilities. The scale ranges from 1 (to a very small degree) to 7 (to a very large extent). At the conclusion of the assessment, individuals are prompted to select a specific job type to compare their answers with others in a similar role and identify the most crucial competences within the domain of digital literacy. The results of the assessment are then generated in the form of the DC wheel and sent to the participant's email address. By clicking on any competency within their wheel, individuals can access a more detailed explanation of the competence and its corresponding score. Each competency summary highlights areas that require improvement in order to strengthen overall digital literacy. Furthermore, individuals have the opportunity to compare their results with those of other users, gaining insight into their relative strengths and areas that necessitate additional effort. Additionally, participants receive personalized recommendations in the form of a report that lists three competences with the lowest scores. To enhance their overall skill set, individuals are encouraged to follow the suggested exercises associated with these competences. The platform offers two subscription options: free and premium. The free version provides individuals with a snapshot of their current level of digitalization in their professional lives. The premium subscription, however, offers enhanced features that include the ability to build individual profiles, gain a comprehensive overview of an organization's digital competence levels, integrate the organization's learning materials into the platform, establish a direct link between results and resources, and monitor personal progress in digital development. In the current study, the Wheel was tested by six participants in order to conduct further research on the digital competences of lecturers at the University of Porto, Perm National Research Polytechnic University, and the University of Tyumen. The participants consisted of five professors, comprising 2 males and 4 females, within the age range of 35-50 years old. These participants represented various departments, including Computer Design, Environmental Protection, Public Relations, Foreign Languages, Mass Communication, and Educational Studies. All participants were provided with the opportunity to utilize the Wheel to evaluate their digital competences. Table presents the average results and total scores of the six participants across all six areas of competences. The total score indicates the aggregate evaluation of scores across all competences and questions. This score corresponds to one of eight levels of digital literacy: levels 1-2 represent the Foundation level, levels 3-4 signify Intermediate proficiency, levels 5-6 denote Advanced expertise, and levels 7-8 indicate Highly Specialized competence. In general, the participants exhibited a wide range of total scores, spanning from 47% (participant D) to 84% (participant E). A score of 47% corresponds to Level 2, where users are capable of independently performing simple tasks with some guidance when necessary. A score of 53% at Level 3 indicates that users can complete routine and well-defined tasks, as well as solve straightforward problems on their own. At Level 4, a score of 67% signifies users' ability to perform routine tasks and solve straightforward problems independently. Lastly, scores of 84% and 83% at Level 6 demonstrate users' proficiency in completing tasks and identifying the most appropriate solutions for most problems, while also being able to adapt solutions to complex contexts[28]. Results of the Digital Competence Wheel test Areas A B C D E F Information, % 86 62 47 52 85 84 Communication, % 85 68 78 58 83 79 Production, % 81 78 34 31 81 86 Safety, % 75 58 54 46 76 71 Total score, % 82 67 53 47 84 83 Level Advanced 6 Intermediate 4 Intermediate 3 Foundation 2 Advanced 6 Advanced 6 Source: compiled by Elizaveta A. Osipovskaya, Svetlana Yu. Dmitrieva, Anastasiia A. Savelyeva. Regarding specific competences, participant A achieved the highest scores in Collaboration (97%), and participant C excelled in Social Awareness (93%). Collaboration refers to the ability to coordinate efforts in both traditional and digital contexts, while Social Awareness implies the demonstration of appropriate behavior in social interactions, with an awareness of how technology can influence society and social relationships. On the other hand, the lowest scores were obtained in Configuration (Production) by participants C and D, with scores of 14% and 8% respectively. Configuration pertains to the ability to adjust applications and devices and resolve technical problems[29]. The participants received recommendations on how to improve their individual overall scores. Participant A was advised to enhance their competences in Law, Health, and Automation, which scored 60, 67, and 73% respectively. Participant B's lowest scores were in Media Choice, Search, and Configuration (56, 57, 58%). Participant C exhibited deficiencies in Configuration, Automation, and Search competences (14, 19, 29% respectively). Participant D needed to focus on Configuration, Automation, and Health (8, 17, 29%). Lastly, participants E and F had their lowest scores in Health and Data Protection. Generally, Automation and Configuration displayed the lowest scores among the participants. Automation refers to the ability to modify or create digital solutions that can automate tasks, while Configuration involves adjusting applications and devices to personal preferences and resolving technical problems or tasks[30]. In our perspective, these skills may be less essential for a university professor who teaches subjects such as languages or history, as technical specialists may perform these tasks more effectively. Conversely, Health, which encompasses caring for both physical and mental well-being in a technology and media-driven environment, is a much more crucial competence for educators who must embrace technology in their daily lives. The testing of the Digital Competence Wheel has revealed numerous evident advantages. These include the tool's accessibility, visual and interactive nature, user-friendly interface, and provision of credible, reliable, and comprehensive information regarding an individual's level of digital competence. However, the wheel's self-assessment nature may raise concerns about its objectivity when individuals take the test. Recommending anonymous usage of the tool could address this issue and lead to a more objective assessment of professors' digital competence levels. Conclusion. Today, the use of digital technologies empowers teachers and facilitates a transformation in their roles. Consequently, educators who possess the requisite knowledge and proficiency in technology may supplant those who lack such expertise. As digital competencies evolve incessantly, it is imperative for instructors to adapt by continuously updating their professional development programs. Unfortunately, due to the rapid emergence of new technologies, it is not always feasible to maintain pace with these advancements. Hence, training programs for educators should prioritize the cultivation of creativity in employing both established and cutting-edge tools, enabling them to effectively navigate novel, uncertain, and intricate scenarios, exemplified by technologies like GPT chat or other technological innovations. In essence, these programs should foster the development of multidisciplinary competencies, as well as skills in learning and innovation. One pertinent matter that needs to be addressed in future studies revolves around the incorporation of AI in the DigComp Framework. We contend that AI should be viewed in a transversal manner, as its impact encompasses information interpretation, generation, and security. Nevertheless, an organized approach to AI implementation should regard the framework as a modular system, allowing for the creation of specialized educational training modules that focus on specific skills. SELFIE for School offers a comprehensive depiction of a school environment by involving administrators, teachers, and students. Regrettably, its applicability in higher education settings is limited, as the questions are tailored to the school-level context. On the other hand, the Digital Competence Wheel provides a detailed profile of educators and supplies comprehensive information. However, it is crucial to acknowledge that the resulting profile may not be entirely objective. Therefore, the Wheel should be used in conjunction with other assessment methods. One notable drawback of this study is its theoretical nature, compounded by a small sample size. Nonetheless, the findings underscore the variance in competencies among lecturers in different fields, necessitating a focus on cultivating diverse skill sets. Future research endeavors could involve faculty members to diagnose their digital competencies, serving as a basis for proposing targeted training programs to address the identified areas of improvement.

About the authors

Elizaveta A. Osipovskaya

University of Porto

Author for correspondence.
Email: e.osipovskaya@gmail.com
ORCID iD: 0000-0002-4192-511X

PhD student of Doctoral Program in Digital Media

265 Rodrigues de Freitas Ave, Porto, 4099-002, Portuguese Republic

Svetlana Yu. Dmitrieva

Perm National Research Polytechnic University

Email: s.y.dmitrieva@gmail.com
ORCID iD: 0000-0003-0365-6526

PhD in Philology, Associate Professor, Department of Foreign Languages and Public Relations

29 Komsomolsky Prospekt, Perm, 614990, Russian Federation

Anastasiia A. Savelyeva

University of Tyumen

Email: an.a.saveleva@utmn.ru
ORCID iD: 0000-0002-7727-9850

expert, School of Education

18 Semakova St, Tyumen, 625003, Russian Federation

References