Cognitive complexity measures for educational texts: Empirical validation of linguistic parameters

封面

如何引用文章

全文:

详细

The article presents a study conducted within the framework of discourse complexology - an integral scientific domain that has united linguists, cognitive scientists, psychologists and programmers dealing with the problems of discourse complexity. The issue of cognitive complexity of texts is one of the central issues in discourse complexology. The paper presents the results of the study aimed to identify and empirically validate a list of educational texts’ complexity predictors. The study aims to identify discriminant linguistic parameters sufficient to assess cognitive complexity of educational texts. We view text cognitive complexity as a construct, based on the amount of presented information and the success of reader-text interactions. The idea behind the research is that text cognitive complexity notably increases across middle and high schools. The research dataset comprises eight biology textbooks with the total size of 219,319 tokens. Metrics of text linguistic features were estimated with the help of automatic analyzer RuLingva (rulingva.kpfu.ru). Linguistic and statistical analysis confirmed the hypothesis that text syntactic and lexical parameters are discriminative enough to classify different levels of cognitive complexity of educational texts used in middle and high schools. Text parameters that manifest variance in cognitive complexity include lexical diversity (TTR); local argument overlap; abstractness index; number of polysyllabic words, Flesch-Kincaid Grade Level; number of nouns and number of adjectives per sentence. Empirical evidence indicates that the proposed approach outperforms existing methods of text complexity assessment. The research results can be implemented in the system of scientific and educational content expertise for Russian school textbooks. They can also be of some use in the development of educational resources and further research in the field of text complexity.

作者简介

Roman Kupriyanov

Kazan National Research Technological University; Kazan (Volga Region) Federal University

编辑信件的主要联系方式.
Email: kroman1@mail.ru
ORCID iD: 0000-0001-9794-9607

Doctor of Psychology and Associate Professor of the Department of Social Work, Pedagogy and Psychology at Kazan National Research Technological University; Chief Researcher of the “Text Analytics” Research Lab at the Institute of Philology and Intercultural Communication, Kazan Federal University (Kazan, Russia). His areas of research are psycholinguistics, pedagogy of higher education, social psychology and social work. He is the author of more than 120 research articles.

Kazan, Russia

Olga Bukach

Kazan (Volga Region) Federal University

Email: olga.bukach1987@gmail.com
ORCID iD: 0009-0009-8638-5119

Doctor of Philology and Associate Professor of the Department of Theory and Practice of Foreign Language Teaching at the Institute of Philology and Intercultural Communication, Kazan Federal University (Kazan, Russia). She is in charge of organizing and carrying out language testing procedures at the Institute, including but not limited to test construction, as well as using statistical methods for handling and analyzing the obtained data.

Kazan, Russia

Oksana Aleksandrova

RUDN University

Email: alexandrova-oi@rudn.ru
ORCID iD: 0000-0002-7246-4109

Doctor of Philology and Associate Professor of the Department of General and Russian Linguistics at RUDN University (Moscow, Russia). Her areas of research are semantics, cognitive linguistics and discourse analysis. She is the author of more than 60 research articles

Moscow, Russia

参考

  1. Andrews, Glenda & Graeme S Halford. 2002. A cognitive complexity metric applied to cognitive development. Cognitive Psychology 45 (2). 153-219. https://doi.org/10.1016/S0010-0285(02)00002-6
  2. Blake, J. Barry. 2001. Case (2nd ed., Cambridge Textbooks in Linguistics). Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139164894
  3. Bolbakov, Roman G. 2016. Slozhnost’ informatsionnykh konstruktsiy (Complexity of informative structures). Obrazovatel’nye resursy i tehnologii (In Russ.)
  4. Da’as, Rim'a, Chen Schechter & Mowafaq Qadach. 2020. School Leaders’ Cognitive Complexity: Impact on the Big 5 Model and Teachers’ Organizational Citizenship Behavior. Journal of School Leadership 30 (5). 398-423. https://doi.org/10.1177/1052684619896535
  5. Das, Syaamantak, Kumar S. Das Mandal & Anupam Basu. 2020. Cognitive complexity analysis of learning-related texts: a case study on school textbooks. In Vittorini, P., Di Mascio, T., Tarantino, L., Temperini, M., Gennari, R., De la Prieta, F. (eds.), Methodologies and Intelligent Systems for Technology Enhanced Learning, 10th International Conference. MIS4TEL 2020. Advances in Intelligent Systems and Computing, vol 1241. Springer, Cham. https://doi.org/10.1007/978-3-030-52538-5_9
  6. Dehaene, Stanislas. 2007. Les Neurones de la Lecture. Paris: Éditions Odile Jacob.
  7. Dobrynina, Oxana L. 2019. Akademicheskoe pis’mo dlya publikatsionnykh tselei: stilisticheskie pogreshnosti. (Academic writing for publication purposes: Stylistic faults). Vysshee obrazovanie v Rossii 10. 38-49. (In Russ.)
  8. Field, John. 2004. Psycholinguistics: The Key Concepts. London, Routledge (in preparation). Cognitive Validity. Taylor L. ed. In Examining Speaking: (Studies in Language Testing). Cambridge, Cambridge University Press / Cambridge ESOL.
  9. Fitzgerald, Jill, Jeff Elmore, Heather Koons, Elfrieda H. Hiebert, Kimberly Bowen, Eleanor Sanford-Moore & Jackson A. Stenner. 2015. Important text characteristics for early-grades text complexity. Journal of Educational Psychology 107 (1). 4-29. https://doi.org/10.1037/a0037289
  10. Fulcher, Glenn. 1997. Text difficulty and accessibility: Reading formulae and expert judgement. System 25 (4). 497-513. https://doi.org/10.1016/S0346-251X(97)00048-1
  11. Gatiyatullina, Gallya, Marina Solnyshkina, Valery Solovyev, Andrey Danilov, Ekaterina Martynova & Iskander E. Yarmakeev. 2020. Computing Russian Morphological distribution patterns using RusAC Online Server. Proceedings - International Conference on Developments in eSystems Engineering. 393-398.
  12. Gatiyatullina, Gallya, Marina I. Solnyshkina, Valery Solovyev, Andrey Danilov, Ekaterina Martynova & Iskander Yarmakeev. 2020. Computing Russian Morphological distribution patterns using RusAC Online Server Proceedings - International Conference on Developments in eSystems Engineering, DeSE. 2020-December. 393-398.
  13. Gatiyatullina, Galiya M., Marina I. Solnyshkina, Roman V. Kupriyanov& Chulpan R. Ziganshina. 2023. Lexical density as a complexity predictor: The case of Science and Social Studies textbooks. Research Result. Theoretical and Applied Linguistics 9 (1). 11-26. https://doi.org/10.18413/2313-8912-2023-9-1-0-2
  14. Gladkikh, Anatoliy A., Sergey M. Namestnikov & Nikita A. Pchelin. 2017. Effectivnoe perestanovochnoe dekodirovanie dvoichnykh blokovykh izbytochnykh kodov (Efficient Permutation Decoding of Binary Block Redundant Codes). Avtomatizatciya processov upravleniya 1 (47). 67-74. (In Russ.)
  15. Hansen, Kathleen A. 1990. Information richness and newspaper pulitzer prizes. Journalism Quarterly 67 (4). 930-935. https://doi.org/10.1177/107769909006700447
  16. Johnson-Laird, Philip N. 1983. Mental Models. London et al.: Cambridge University Press.
  17. Just, Marcel A. & Patricia A. Carpenter 1987. The Psychology of Reading and Language Comprehension. MA, US: Allyn & Bacon.
  18. Kalinkina, Evgeniya M., Tatyana A. Poyarova & Aida V. Yablokova. 2021. Uchet dinamiki kognitivnoi slozhnosti u podrostkov pri postroenii obrazovatel’nogo protsessa (Taking into account the dynamics of cognitive complexity in teenagers in educational process design). Problemy sovremennogo pedagogicheskogo obrazovaniya 72 (4). 328-331. (In Russ.)
  19. Kholodnaya, Marina A. 2004. Kognitivnye stili. O prirode individual’nogo uma (Cognitive styles. On the nature of individual brainwork). Saint Petersburg. (In Russ.)
  20. Kintsch, Walter & Teun A. van Dijk. 1978. Toward a model of text comprehension and production. Psychological Review 85. 363-394.
  21. Kluyeva, Nadezhda V. 2003. Osobennosti vospitaniya na raznykh vozrastnykh etapakh (Peculiar features of upbringing at different age). Moscow. (In Russ.)
  22. Koda, Keiko 2005. Insights into Second Language Reading. Cambridge: Cambridge University Press.
  23. Kongorskaya, Svetlana A. 2014. Vozrastnye osobennosti razvitiya prostranstvennogo myshleniya podrostkov i starshikh shkol’nikov: ikh vzaimosvyaz’ s uchebnoy uspevaemost’yu (Age features of developing spatial thinking of adolescents and high school students: their relationship with academic performance). Vestnik Buryatskogo gosudarstvennogo universiteta 5. 59-65. (In Russ.)
  24. Kotova, Ekaterina O. 2021. Lingvoekologicheskaya otsenka udobochitaemosti rossiiskikh longridov (statisticheskii podkhod) (Linguoecological assessment of the readability of Russian longreads (Statistical approach). Izvestiya Yuzhnogo federal’nogo universiteta. Philologicheskie nauki 25 (2). 67-76. (In Russ.)
  25. Krioni, Nikolai K., Alexey D. Nikitin & Anastasia V. Filippova 2008. Avtomatizirovannaya sistema analiza parametrov slozhnosti uchebnogo teksta (Automated system for analyzing the complexity parameters of instructional text). Tekhnologiya i organizatsiya obucheniya 155-161. (In Russ.)
  26. Kudzh, Stanislav A. & Viktor Ya. Tsvetkov 2018. Faktory kognitivnoi slozhnosti (Cognitive complexity factors). Informatsionnye tekhnologii v nauke, obrazovanii i upravlenii 6 (10). 34-41. (In Russ.)
  27. Kupriyanov, Roman V., Marina I. Solnyshkina, Mihai Dascalu & Tatyana Soldatkina. 2022. Lexical and syntactic features of academic Russian texts: A discriminant analysis. Research Result. Theoretical and Applied Linguistics 8 (4). 105-122. https://doi.org/10.18413/2313-8912-2022-8-4-0-8
  28. Lavazza, Luigi, Abedallah Abualkishik, Geng Liu & Sandro Morasca. 2022. An empirical evaluation of the “Cognitive Complexity” measure as a predictor of code understandability. Journal of Systems and Software 197. 111561. https://doi.org/10.1016/j.jss.2022.111561.
  29. Matlin, Margaret W. 1998. Cognition, 4th edn.: Harcourt Brace College Pub. NY.
  30. Matskovskiy, Mikhail S. 1976. Problemy chitabel’nosti pechatnogo materiala (Readability issues of printed materials). Smyslovoe vospriyatie rechevogo obscheniya v usloviyakh massovoi kommunikatsii. 126-142. (In Russ.)
  31. McComb, A. Sara & Jane M. Kirkpatrick. 2016. Impact of pedagogical approaches on cognitive complexity and motivation to learn: Comparing nursing and engineering undergraduate students. Nursing Outlook. 64 (1). 37-48. https://doi.org/10.1016/j.outlook.2015.10.006.
  32. McCarthy, Kathryn Soo, Danielle Siobhan Mcnamara, Marina I. Solnyshkina, Fanuza Kh. Tarasova & Roman V. Kupriyanov. 2019. The Russian language test: Towards assessing text comprehension. Science Journal of Volgograd State University. Linguistics 18 (4). 231-247. https://doi.org/10.15688/jvolsu2.2019.4.18
  33. Morin, Edgar. 1992. Method: Towards a Study of Humankind. Vol. 1. New York; Berlin, Bern; Frankfurt/M.; Paris; Wien; Lang.
  34. Morin, Edgar. 2005. Metod. Priroda prirody (Method. The nature of nature). Progress-Traditsiya. Moscow. (In Russ.)
  35. Morin, Edgar. 2021. O slozhnosti (On complexity). Institut obschegumanitarnykh issledovanii. Moscow. (In Russ.)
  36. Nevdakh, Marina M. 2008. Issledovanie informatsionnykh kharakteristik uchebnogo teksta metodami mnogomernogo statisticheskogo analiza (The study of the instructional text information characteristics by the methods of multivariate statistical analysis). Prikladnaya informatika 4. 117-130. (In Russ.)
  37. Perry, William G., Jr. 1981. “Cognitive and ethical growth: The making of meaning”. In Arthur W. Chickering & Associated (eds.), The Modern American College. 76-116. San Francisco: Jossey-Bass.
  38. Pervin, Lawrence A. 1984. Current Controversies and Issues in Personality. 2nd ed-n. John Wiley & Sons.
  39. Petrenko. Victor F. 2010. Osnovy psikhosemantiki (Basic psychsemantics). Moscow: Eksmo. (In Russ.)
  40. Polya, George. 2015. How to Solve It: A New Aspect of Mathematical Method (Princeton Science Library, 34). Princeton Science Li Edition.
  41. Rodrigues, Virgínia T., Luis A. Gonçalves, Soares P. Paolinelli Maciel & Augusto P. Rodrigues de Paiva 2014. Ethical education of an engineer with responsibility for a sustainable world. 2014 IEEE International Symposium on Ethics in Science, Technology and Engineering. 1-7. https://doi.org/10.1109/ETHICS.2014.6893426
  42. Shardlow, Matthew, Richard Evans & Marcoz Zampieri. 2022. Predicting lexical complexity in English texts: the Complex 2.0 dataset. Lang Resources & Evaluation 56. 1153-1194. https://doi.org/10.1007/s10579-022-09588-2
  43. Sharoff, Serge. 2022. What neural networks know about linguistic complexity. Russian Journal of Linguistics 26 (2). 371-390. https://doi.org/10.22363/2687-0088-30178
  44. Sheehan, Kathleen M., Irene Kostin, Yoko Futagi & Michael Flor. 2010. Generating automated text complexity classifications that are aligned with targeted text complexity standards. ETS Research Report Series. 2010 (2). 1-44. http://dx.doi.org/10.1002/j.2333-8504.2010.tb02235.x
  45. Silva, Susana, Filomena Inácio, Daniel Rocha e Sousa, Nuno Gaspar, Vasiliki Folia & Karl Magnus Petersson. 2023. Formal language hierarchy reflects different levels of cognitive complexity. Journal of Experimental Psychology: Learning, Memory, and Cognition 49 (4). 642-660. https://doi.org/10.1037/xlm0001182
  46. Sirotinina, Olga B. 2009. Razgovornaya rech v sisteme funktsionalnyh stiley sovremennogo russkogo literaturnogo yazyka: grammatika (Spoken language within the system of functional styles of the Russian literary language: grammar) 3rd ed-n, Moscow: Librekom. (In Russ.)
  47. Solnyshkina, Marina I., Elena V. Harkova & Mariia B. Kazachkova. 2020. The structure of cross-linguistic differences: Meaning and context of ‘readability’ and its Russian equivalent ‘chitabelnost’. Journal of Language and Education 6 (1). 103-119.
  48. Solnyshkina, Marina I., Ekaterina V. Martynova & Mariya I. Andreeva. 2020. Propositsional’noe modelirovanie dlya otsenki informativnosti teksta (Propositional modeling for evaluating the informative value of a text). Uchenye zapiski natsional’nogo obschestva prikladnoi lingvistiki 3 (31). 47-57. (In Russ.)
  49. Solnyshkina, Marina I. & Alexander S. Kisel’nikov. 2015. Slozhnost' teksta: Ehtapy izucheniya v otechestvennom prikladnom yazykoznanii (Text complexity: Stages of study in domestic applied linguistics). Vestnik Tomskogo gosudarstvennogo universiteta. Filologiya 6.86-99. (In Russ.)
  50. Solnyshkina, Marina I., Danielle S. McNamara & Radif R. Zamaletdinov. 2022. Natural language processing and discourse complexity studies. Russian Journal of Linguistics 26 (2). 317-341.
  51. Solovyev, Valery, Musa Islamova, Marina Solnyshkina, Roman Kupriyanov & Elzara Gafiyatova. 2021. Sentiment analysis for Russian academic texts: A lexicon-based approach. CEUR Workshop Proceedings. 89-97.
  52. Solovyev V., M. Solnyshkina & D. McNamara. 2022. Computational linguistics and discourse complexology: Paradigms and research methods. Russian Journal of Linguistics 26 (2). 275-316. https://doi.org/10.22363/2687-0088-30161
  53. Tsetlin, Valentina S. 1980. Didakticheskie trebovaniya k kriteriyam slozhnosti uchebnogo materiala (Didactic requirements to the complexity criteria of educational material). Novye issledovaniya v pedagogicheskikh naukakh 1 (35). 30-33. (In Russ.)
  54. Ushakov, Konstantin M. 1980. O kriteriyakh slozhnosti uchebnogo materiala shkol'nykh predmetov (On the criteria of complexity of teaching material of school subjects). Novye issledovaniya v pedagogicheskikh naukakh 2 (36). 33-35. (In Russ.)
  55. Valgina, Nina S. 2003. Teoriya teksta (Theory of text). Moscow: Logos. (In Russ.)
  56. Valueva, Ekaterina A., Nina Danilevskaya, Ekaterina Lapteva & Dmitriy Ushakov 2017. Kognitivnaya slozhnost’ khudozhestvennykh tekstov dlya detei: kvantitativnye metody otsenki (Cognitive complexity of literary texts for children: Quantitative methods of evaluation). Voprosy psikholingvistiki 42-61. (In Russ.)
  57. Vygotsky, Lev S. 1978. Mind in Society: The Development of Higher Psychological Processes. Cambridge, MA: Harvard University Press.
  58. Wang, Yingxu, Robert Berwick & Xiangfeng Luo X. 2012. A formal measurement of the cognitive complexity of texts in cognitive linguistics. 2012 IEEE 11th International Conference on Cognitive Informatics and Cognitive Computing, Kyoto, Japan. 94-102. https://doi.org/10.1109/ICCI-CC.2012.6311132
  59. Weir, Cyril J. 1993. Understanding and Developing Language Tests. London, Prentice Hall.
  60. Weir, Cyril J. & Hanan Khalifa. 2008. A cognitive processing approach towards defining reading comprehension. Research Notes, Cambridge ESOL 31. 2-10.
  61. Wijendra, Dinuka & K. Priyantha Hewagamage. 2021. Analysis of cognitive complexity with cyclomatic complexity metric of software. International Journal of Computer Applications 174. 14-19. https://doi.org/10.5120/ijca2021921066.
  62. Zhu, L., He Li, Wu He & Chuang Hong. 2020. What influences online reviews’ perceived information quality? Perspectives on information richness, emotional polarity and product type. The Electronic Library 38 (2). 273-296. https://doi.org/10.1108/EL-09-2019-0208
  63. Zhuravlev, Anatoly F. 1988. Opyt kvantitativno-tipologicheskogo issledovaniya raznovidnostey ustnoy rechi (An experience of quantitative and typological investigation of spoken registers). Raznovidnosti gorodskoy ustnoy rechi, Moscow, Nauka. 84-150. (In Russ.)
  64. Kondakov, Igor M. 2000. Psikhologiya. Illyustrirovannyi slovar’ (Psychology. Illustrated dictionary). Saint Petersburg. Moscow. (In Russ.)
  65. Petrovskiy, Artur V. (ed.). 1998. Kratkiy psikhologicheskiy slovar’ (Brief dictionary of psychology). Rostov-on-Don: Phenix. (In Russ.)
  66. Voronin, Alexander S. 2006. Slovar terminov po obschei i sotsial’noy pedagogike (Glossary of terms on general and social pedagogy). Yekaterinburg. (In Russ.)

版权所有 © Kupriyanov R., Bukach O., Aleksandrova O., 2023

Creative Commons License
此作品已接受知识共享署名-非商业性使用 4.0国际许可协议的许可。

##common.cookie##