Pupil diameter dynamics as an indicator of the respondent’s cognitive load: Methodological experiment comparing CASI and P&PSI

Cover Page

Cite item

Full Text

Abstract

In recent years, the increase in general interest in methods for measuring cognitive load and subjectively perceived mental effort when solving various tasks and in the interpersonal communication was accompanied by an increase in the specific interest of social researchers in the multimodal assessment of the cognitive load of interviewers and respondents based on objective and subjective indicators, including paradata and webcam data, in order to control this load’s impact on the quality of survey data. The authors argue that the possibilities of relatively new approaches to measuring cognitive load with neurophysiological methods (such as the use of wearable devices for oculography - eye tracking and pupillometry - which do not disrupt the natural course of respondents and interviewers activity) are still underestimated, although they allow an accurate time linkage of measured parameters’ dynamics (primarily the size of the pupil) to the question format, mode and phase of survey completion, external influences localized in time, etc. As a rule, quantitative studies of surveys’ cognitive load and its possible impact on the quality of survey data focus on computer-assisted (CAPI) or paper-based (PAPI) interviewing, while the specificity of the cognitive load in the self-completed computerized (CASI) and paper (P&PSI) surveys was not studied. The article presents the results of the methodological experiment based on a modified version of the multimodal approach to the comparative assessment of the cognitive load of interviewers working with paper and computerized questionnaire. We expanded the range of methods for assessing cognitive load by using a wearable oculographic device (eye tracker) to measure the dynamics of pupil size when answering different survey questions. The results of the experiment confirmed the hypothesis about the approximate equivalence of the two modes of survey completion in terms of their cognitive load for younger respondents with a high level of functional computer literacy, and allowed an initial assessment of the technical and metrological capabilities and limitations of the use of pupil dynamics’ indicators, measured with a wearable oculographic device, to study the respondents’ cognitive load.

About the authors

I. F. Deviatko

HSE University; Institute of Sociology of FCTAS RAS

Author for correspondence.
Email: deviatko@gmail.com

доктор социологических наук, заведующая кафедрой анализа социальных институтов

Myasnitskaya St., 11, Moscow, 101000, Russia; Krzhizhanovskogo St., 24/35-5, Moscow, 117218, Russia

M. B. Bogdanov

HSE University

Email: bogdanovmikle@mail.ru

младший научный сотрудник Лаборатории культурсоциологии и антропологии образования

Myasnitskaya St., 11, Moscow, 101000, Russia

D. V. Lebedev

HSE University

Email: zenon-daniil@yandex.ru

стажер-исследователь Международной лаборатории исследований социальной интеграции

Myasnitskaya St., 11, Moscow, 101000, Russia

References

  1. Bogdanov M.B., Lebedev D.V. “Glavnoe ne boyatsya!” - vozdeystvie treningov na ozhidaniya interviyuerov pri perekhode ot PAPI k CAPI [“Do not worry!” - training’s impact on interviewers’ expectations under the transition from PAPI to CAPI]. Sotsiologiya: 4M. 2017; 45. (In Russ.).
  2. Deviatko I.F. Razrabotka podkhoda k kolichestvennoy multimodalnoy otsenke kognitivnoy nagruzki interviyuerov: rezultaty pilotnogo kvazieksperimenta [Developing an approach to the multimodal quantitative assessment of interviewers’ cognitive load: First results of a field quasi experiment]. RUDN Journal of Sociology. 2018; 18 (4) (In Russ.).
  3. Deviatko I.F., Lebedev D.V. Glazami interviyuera, glazami respondenta: kontury novogo podkhoda k otsenke kognitivnoy nagruzki pri provedenii oprosa [Through the eyes of the interviewer, through the eyes of the respondent: Outlining a new approach towards the assessment of cognitive load during the interview]. Monitoring of Public Opinion: Economic and Social Changes. 2017; 5 (1). (In Russ.).
  4. Lebedev D.V., Bogdanov M.B. Perekhod s PAPI na CAPI: opyt interviyuerov i kharakteristiki, vliyayushchie na ikh ozhidaniya [Transition from PAPI to CAPI: Interviewers’ experience and factors influencing their expectations]. Monitoring of Public Opinion: Economic and Social Changes. 2019; 4. (In Russ.).
  5. Terentiev E.A., Mavletova A.M., Kosolapov M.S. Interviyuirovanie s pomoshchiyu kompiyuternykh tekhnologiy v longityudnykh obsledovaniyakh domokhozyaystv [Computer-assisted personal interviewing for longitudinal household studies]. Monitoring of Public Opinion: Economic and Social Changes. 2018; 3 (In Russ.).
  6. Baier D. Computer-assisted versus paper-and-pencil self-report delinquency surveys: Results of an experimental study. European Journal of Criminology. 2018: 15 (4).
  7. Baker R.P., Bradburn N.M., Johnson R.A. Computer-assisted personal interviewing: An experimental evaluation of data quality and cost. Journal of Official Statistics. 1995; 11 (4).
  8. Beatty J., Lucero-Wagoner B. The pupillary system. Handbook of Psychophysiology. 2000; 2.
  9. Beebe T.J., Harrison P.A., Park E., McRae J.A., Evams J. The effects of data collection mode and disclosure on adolescent reporting of health behavior. Social Science Computer Review. 2006; 24 (4).
  10. Chen F., Zhou J., Wang Y., Yu K., Arshad S.Z., Khawaji A., Conway D. Robust Multimodal Cognitive Load Measurement. Cham; 2016.
  11. Colasante E., Benedetti E., Fortunato L., Scalese M., Potente R., Cutilli A., Molinari S. Paper and-pencil versus computerized administration mode: Comparison of data quality and risk behavior prevalence estimates in the European School Survey Project on Alcohol and Other Drugs (ESPAD). PloS One. 2019; 14 (11).
  12. Costa V.D., Rudebeck P.H. More than meets the eye: The relationship between pupil size and locus coeruleus activity. Neuron. 2016; 89 (1).
  13. Couper M.P., Burt G. Interviewer attitudes toward computer-assisted personal interviewing (CAPI). Social Science Computer Review. 1994; 12 (1).
  14. De Leeuw E.D., Hox J.J., Snijkers G. The effect of computer-assisted interviewing on data quality. A review. Market Research Society. 1995; 37 (4).
  15. Eynon R., Fry J., Schroeder R. The ethics of Internet research. Sage Internet Research Methods. Sage; 2008.
  16. Hartmann M., Fischer M.H. Pupillometry: The eyes shed fresh light on the mind. Current Biology. 2014; 24 (7).
  17. Höhne J.K. Eye-tracking methodology: Exploring the processing of question formats in web surveys. International Journal of Social Research Methodology. 2019; 22 (2).
  18. Höhne J.K., Schlosser S., Krebs D. Investigating cognitive effort and response quality of question formats in web surveys using paradata. Field Methods. 2017; 29 (4).
  19. Hoogerheide V., Renkl A., Fiorella L., Paas F., van Gog T. Enhancing example-based learning: Teaching on video increases arousal and improves problem-solving performance. Journal of Educational Psychology. 2019; 111 (1).
  20. Jbara A., Feitelson D.G. How programmers read regular code: A controlled experiment using eye tracking. Empirical Software Engineering. 2017; 22 (3).
  21. Joinson A.N., Ulf-Dietrich R., Buchanan T., Paine Schofield C.B. Privacy, trust, and selfdisclosure online. Human-Computer Interaction. 2010; 25 (1).
  22. Kaminska O., Foulsham T. Real-world eye-tracking in face-to-face and web modes. Journal of Survey Statistics and Methodology. 2014; 2 (3).
  23. Kennedy D.O., Scholey A.B. Glucose administration, heart rate and cognitive performance: Effects of increasing mental effort. Psychopharmacology. 2000; 149 (1).
  24. Laeng B., Sirois S., Gredebäck G. Pupillometry: A window to the preconscious? Perspectives on Psychological Science. 2012; 7 (1).
  25. Laurie H. From PAPI to CAPI: Consequences for Data Quality on the British Household Panel Study. ISER Working Paper Series, 2003. No. 2003-14.
  26. Lucia S., Herrmann L., Killias M. How important are interview methods and questionnaire designs in research on self-reported juvenile delinquency? An experimental comparison of Internet vs paper-and-pencil questionnaires and different definitions of the reference period. Journal of Experimental Criminology. 2007; 3 (1).
  27. Mathôt S. Pupillometry: Psychology, physiology, and function. Journal of Cognition. 2018; 1 (1).
  28. Paas F., Tuovinen J.E., Tabbers H., van Gerven P.W.M. Cognitive load measurement as a means to advance cognitive load theory. Educational Pychologist. 2003; 38 (1).
  29. Schmeck A., Opfermann M., van Gog T., Paas F., Leutner D. Measuring cognitive load with subjective rating scales during problem solving: Differences between immediate and delayed ratings. Instructional Science. 2015; 43 (1).
  30. Schomakers E.M., Chantal L., Müllmann D., Ziefle M. Internet users’ perceptions of information sensitivity - insights from Germany. International Journal of Information Management. 2019; 46.
  31. Schräpler J.-P., Schupp J., Wagner G.G. Changing from PAPI to CAPI: Introducing CAPI in a longitudinal study. Journal of Official Statistics. 2010; 26 (2).
  32. Spark S., Lewis D., Vaisey A., Smyth E., Wood A., Temple-Smith M., Lorch R., Guy R., Hocking J. Using computer-assisted survey instruments instead of paper and pencil increased completeness of self-administered sexual behavior questionnaires. Journal of Clinical Epidemiology. 2015; 68 (1).
  33. Stodel M. But what will people think? Getting beyond social desirability bias by increasing cognitive load. International Journal of Market Research. 2015; 57 (2).
  34. Turner C.F., Ku L., Rogers S.M., Lindberg L.S., Pleck J.H., Sonenstein F.L Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science. 1998; 280.
  35. Wright D.L., Aquilino W.S., Supple A.J. A comparison of computer-assisted and paper-andpencil self-administered questionnaires in a survey on smoking, alcohol, and drug use. Public Opinion Quarterly. 1998.

Supplementary files

Supplementary Files
Action
1. Рис. 1. Эффект взаимодействия способа заполнения анкеты и номера вопроса

Download (160KB)
Indexing metadata
2. Рис. 2. Динамика среднего размера зрачка по вопросам и паузам (11-й респондент)

Download (61KB)
Indexing metadata
3. Рис. 3. Динамика среднего размера зрачка по вопросам и паузам (31-й респондент)

Download (44KB)
Indexing metadata

Copyright (c) 2021 Deviatko I.F., Bogdanov M.B., Lebedev D.V.

Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.

This website uses cookies

You consent to our cookies if you continue to use our website.

About Cookies