Event Construal in Text and Image. Gaze Behavior and Discourse Interpretations

Full Text

Introduction

The studies of text and image in cognitive semiotics have recently integrated experimental methods since one of the key areas in heterosemiotic research is now prognostic analysis aimed at predicting how heterosemiotic construal techniques might influence interpretation. Prognostic assessment in the studies follows a series of experiments exploiting various heterosemiotic stimuli displaying diverse construal patterns. As a result, some behavior reactions contingent with the construal patterns are detected, which allows to range the construal patterns as more or less significant in terms of interpretation reactions they cause. Among the most influential experimental methods testing this contingency we will name oculographic analysis since it helps reveal the gaze patterns and the quantitative data on reading techniques susceptible to construal patterns in heterosemiotic unities of text and image, for instance, see [1–3]. Experimental studies in multimodal semiotics (exploring various heterosemiotic forms), although not exploiting prognostic algorythms of interpretation, still succeed in detecting both typical and atypical gaze patterns of experiment participants perceiving the stimuli diverse in heterosemiotic construal. It is noteworthy, that these results are used in applied research and technology, for instance in gamification, publishing industry, educational literature design.

However, in most cases these are the semiotic characteristics of text and image unities that enjoy experimental research, among them we might name the occupancy space of text and image, the color techniques, the order of appearance [4; 5], the combination techniques like domineering, coordination, and reduplication [6; 7], the semantic types of information presented, for example its denotative or connotative character, participants type, dynamic or static type [8]). It is worth mentioning that the pre-experiment analysis of heterosemiotic construal techniques is seldom carried out and even in this case it is most restricted to discourse type studies [9] or communicative functions [10]. Therefore, cognitive construal techniques with participants, events or perspective construal techniques are rarely considered, although they might as well hamper or stimulate interpretation of heterosemiotic unities.

In this study, we test the event construal techniques in text and image, and later explore their effect onto the experiment participants’ gaze patterns. We then proceed to studying the discourse patterns which the participants employed when they gave account on the information they had received while reading. It is expected that diverse event construal techniques will affect the gaze patterns and the discourse patterns employed, although it is hard to predict which construal techniques will stimulate the diversity in cognitive load redistribution. Experimental studies mostly use monomodal stimuli, still among the most influential factors determining the increase in cognitive load expressed for instance in longer fixation duration and return saccadic movements they name the factors of salience (mostly expressed in the construal techniques frequency) and focusing (focalization of construal techniques), for instance, see [11–13]. In case of event construal techniques, we can also distinguish between more and less focal techniques, cf. achievement and nonachievement events, instant and durative events, intersubjective and mono-participant events, where the first event in each group is more focal, although the effects of salience will be applicable only in context. The research question is whether these more focal event construal techniques will affect the gaze behavior and interpretation responses. Therefore, the effects of combinatorial prevalence are explored in this study.

Event construal techniques in text and image

Event construal techniques have received attention in both cognitive semiotics [14–17] and in cognitive linguistics [18–21], among the key areas of study are dynamicity, aspectuality and entrenchment as well as embodiment construal types. Event typologies are thoroughly discussed in cognitive linguistics, however they cannot in all cases be easily adapted to conform to other semiotic systems analysis, still they may suffice to develop the event typologies applicable to heterosemiotic systems. It is worth mentioning, that event construal studies originated with the famous work of Z. Vendler who differentiated between events-achievements, accomplishments, states, and activities. Vendler’s work [22] has become highly resonant in linguistics, especially in logical semantics. Following Vendler, N. Arutunova [23] and E. Paducheva [24] give a more detailed and linguistically oriented accounts on event types. Communicative and discursive characteristics of events were mostly explored in European and American studies of the 1980s, for instance, see [25; 26]. In later theories event construal becomes associated with participants’ construal and their relations; this approach is a key one in construction grammar [27].

In this research, we will mostly rely on semiotic approaches to linguistic analysis which pave the way to interpretation theories and allow to synchronize several semiotic modes on the same grounds. Semiotic foundations for modes synchronization on linguistic terms received attention in the works of L. Novikov in his theory of ornamental field accumulating different means of poeticism construal [28] and in disclosing dominant structures in composition [29]. These dominant structures creating an ornamental field comprised objects and participants, their actions, inner world construal, event location in time and space, poeticism construal. However, these components need specification to be analyzed in heterosemiotic forms. The most influential typological research in this area is the study presented by V. Demyankov who develops the notion of interpretation coordinates (or landmarks) [30] which may serve to develop the procedure of parametric analysis of event construal in different semiotic systems, not restricted to linguistic system. Among these landmarks, V. Demyankov names stativity / dynamicity, manageability (controllability) / nonmanageability, integrity / non-integrity, momentarily / durative character, repeatability / uniqueness, achievements / non-achievements [30. Р. 323–328].

The interpretation value of these landmarks lies in their more and less focusing character for the interpreter, it may help assess (and predict) the interpretation effects produced on the viewer (reader). For instance, the landmark “event-achievement” is more focal than “non-achievement” as it lacks result or action outcome, however in some cases the decision on focal character depends on the reader’s expectations as well as on the event construal type. Therefore, the character of manageable or non-manageable event, for instance, may be interpreted as focal or non-focal depending on whether it conforms to the macroevent (discourse event) expected structure or is totally unexpected. That explains our decision to address the parameters which might be both focusing and defocusing in the context of heterosemiotic construal. The annotation parameter system with the coded parameter values is given in Table 1. This system will be applied to annotate both text and image components of heterosemiotic unity.

The parameter code values start with 1 for annotating text and with 2 for annotating image. Applying two types of codes allows to conduct statistical processing of two data sets. For instance, annotating Sample 1 (see Fig. 1) we may select the following event construal parameters for image construal: True, veritable, real event (201), Physical contact, perception event (205), Intersubjective event (207), Manageable, planned event (211), Incomplete, interrupted event (213), Continuous event (215), Purposeless, aimless event (217), Event lacking evaluation (220), Event located in space (221), Event without location in time (223), Sporadic, unitary event (226), Descriptive event (228).

Fig. 1. Sample 1
Рис. 1. Образец 1

Table 1/Таблица 1
Event construal parameters / Параметры конструирования события

Annotating its text component, we observe that in some cases events display a different character. The event is also given as True, veritable, real (101) as the text displays a sequence of events which might be real (the event on the children’s playground), however it is Shifting, changing (104) as we see a chain of subsequent microevents employing a change in scene without creating or destroying any event components. The event is shown as Intersubjective (107) which activates several participants, grandma, children, grandma’s own grandson, at the same time as opposed to the event in the image, it is Manageable, planned (111) which describes the deliberate action of grandma presenting it in detail. We may conclude that the text shows a Completed, resultant event (112) as it is a completed narrative with all narrative components described in [31], Orientation in the first two sentences presenting the scene and its participants and objects, Complicating Action in the third sentence presenting assessment of the grandma’s action, although the action itself will be described later. Therefore, the sentences five and six display an amalgam of Complicating Action in Grandma put a 8-kilo weight into her grandson’s pocket and Resolution in Her grandson turned out to be the heaviest of all children. Coda in the final sentence Then Grandma boasted that they give the best food to their grandson allows to present the Grandma’s explanation for what has happened. The event is Continuous (115) as the actions described are mostly of unbounded aspect type [32] in competed, behaved, overweighted, boasted (cf. with the actions like won, tore, killed, eaten or similar). At the same, we encounter Event-achievement (116) because the reasons for the grandma’s action are explained and the desired result is achieved. The event is classified as Undesirable (119) as this action is justified as dishonest, although explicitly. It might have been classified as desirable if the text perspective were the one that is construed by the grandma herself, however it is not so. The event is located in space (121) as there are several objects which help to orient it (swings, pocket), however, this is Event without location in time (124) because there are no time identifiers apart from the time identifier then in the final sentence, but this one colocates two microevents [33] and does not allow to locate the whole event in time. The event is Sporadic, unitary (126) as the action is not repeated, and Causal (127) because the reasons for such a dishonest action are clearly presented.

Therefore, the system elaborated allows to provide annotation in both text and image and to analyze them on the same grounds, which can further be used for statistical processing and exploring the techniques of event construal specifics developed in text and image in contrast.

Methods and procedure

The study applies the triangulation methodological procedure [34], here it comprises the methods of cognitive semiotic modelling (parametrization), annotation and statistical methods to explore the distribution and contingency techniques in event construal in text and image, and cognitive psychological study (oculographic experiment) testing the contingency between construal and interpretation techniques and assessing role of combinatorial prevalence factor. Below, I will briefly describe these three research steps.

Parametrization procedure was described in detail above, it includes the elaboration of sufficient parameter system applicable to both text and image construal analysis. It involves 12 groups of event construal parameters which have coded values for further annotation and statistical processing.

Annotation was carried in UAM Image Tool (www.corpustool.com) which allows to conduct annotation at various levels and annotate image and text separately. The research data are 100 book samples presenting the fragments of text accompanied with illustration, addressed to adults and children. The basic criterion for samples selection was the criterion of diversity, that is the diversity in author and illustrator range, period of issue, style of writing and artistic style, genre of literature, addressee age group. The samples were selected from the works of A. Chekhov, B. Akunin, I. Ilf and Ye. Petrov, S. Marshak, N. Gogol, A. Barto, L. Kaminsky, F. Nonn, E. Tarle, Z. Turlonnas, among others, with illustrations made by Kukryniksy, L. Vladimirsky, V. Chizhikov, R. Bogdanova, I. Oleynikov, A. Bekker, A. Petrova, K. Boldt, V. Vidali, among others. In Figure 2 we demonstrate the structure of Event level and the annotation procedure of the Sample 1 given above.

Fig. 2. Event level structure and annotation procedure in UAM Image Tool
Рис. 2. Структура уровня события и процедура аннотации в UAM Image Tool

There is specificity of annotating in UAM Image Tool which is the necessity to select only one parameter within the group, it explains why we applied paired parameters (and not only one of them) within the parameter system.

The results were further exported and statistically processed in specially designed software HETEROSTAT [35], which allows to receive the activity values of each parameter and each parameter group as well as the correlation values of every pair of parameters. We then may contrast the parameter distribution in text and image and explore the event construal techniques prevailing in text or in image. This analysis is carried out adopting the methods of vector semantic analysis [36–38] which helps detect the contingency between the multi-vector profiles of text and image event construal. With the contingency results we observe the differences between the text and image text construal profiles.

Apart from contingency analysis, we conduct the resonance analysis applying the procedure of overlapping the annotation results in text and image event construal. The procedure involves the search for the parameters with the same activity in text and image; in case a parameter is active in both matrices, it is marked as active in the heterosemiotic unity, in case it is active in text or image only, it is marked as inactive.

To select the samples for further oculographic analysis testing interpretation techniques of event construal in text and image, we conduct factor analysis (principal component analysis as one of its version) in JAMOVI program (https:// www.jamovi.org) to receive typicality and uniqueness values of each sample. Only the samples with varied uniqueness indexes, low and high, should be selected due to their different event construal techniques. In case we detect the changes in eye movement behavior demonstrated by the experiment participants which are contingent with any of the samples, we can easily deduce which event construal parameters, or the combinations of event construal parameters might have caused them. We finally select 6 samples (presented in Open Access web-resources) with different uniqueness index to serve as experiment stimuli (see Figure 3).

The paired samples 1–6 were further used in the experiment which tested the viewers’ oculographic reactions and linguistic reports in terms of the discourse strategies used, which were descriptive, narrative, and speculative. We hypothesized that different Event construal patterns will correlate with specific eye-movement strategies and discourse strategies, which will allow ranging the Event construal operations in terms of their significance for interpretation. 16 participants (students, average age 22) were supposed to view 6 samples, after each sample there was a blank screen for 15 seconds, during this time the participants gave their reports on what they have seen. Their eye movements as well as their reports were recorded during stimuli presentation. Only the reactions of 11 participants were finally considered as in other 5 participants’ eye movement or discourse interpretation reports not all 6 samples were presented adequately.

Fig. 3. Experiment stimuli 1–6
Рис. 3. Стимулы эсперимента 1–6

The SMI Red-x eye tracker was applied in the experiment (binocular system, frequency = 60 Hz, accuracy = 0.4°, head movement 40х20 cm, operating distance = 60–80 cm). Eye movement data during task performance were sampled (fixation and saccade duration, dwell time on the areas of interest); subjects’ free reports were recorded and analyzed in terms of using Descriptive, Narrative, and Speculative discourse strategies. To analyze the discourse strategies and to decide in favor of the strategy adopted we relied on the discourse markers. For Narrative discourse we considered the tense of verbs (mostly present for Descriptive discourse, mostly past or tense shifts for Narrative discourse), the use of conjunctions and adverbs of time, the use of bodily verbs like see and look as well as of actional verbs like make and take. For Descriptive discourse we considered the tense of verbs, the use of parallel clauses, objects and participants naming in parallel structures. For Speculative discourse we looked for the use of modal verbs and words with the semantics of various modality types, the use of word with the semantics of evaluation, and the use of non-referential words, among other markers.

We hypothesized that this triangulation procedure will reveal contingent event construal techniques, eye movement techniques and discourse construal techniques, which will allow to range the significance of event construal parameters for interpretation.

Results Annotation and statistical analysis

I will proceed with the results of annotation and statistical processing of the received data and then present the results of the oculographic experiment. Following the annotation procedure, we explored parametric activity in event construal in text and image. HETEROSTAT program window is given in Figure 4.

Fig. 4. HETEROSTAT window processing the data on Event construal
Рис. 4. Окно программы HETEROSTAT, обрабатывающей данные о конструировании событий

The samples numbers are given on the left, each sample demonstrates its parameter codes, the codes for event construal in text starting with 1, and the codes for event construal in image starting with 2. Parameter activity results helped reconstrue the text and image profiles of Event construal in the samples, see Figure 5.

Fig. 5. Text and Image profiles of Event construal
Рис. 5. Профили конструирования события в тексте и изображении

Pearson correlation test to test profiles contingency revealed that r(26)=0.66, with critical r-values for p=0.01 equal to 0.48, and for p=0.05 the critical r-value is 0.37. High correlation value evidences that in general the direction of vector activity in text and image is similar, however, this analysis does not consider the values themselves. More detailed t-statistics (Paired samples T-test) revealed very low significance of contingency data in text and image, with t(27)=0.017, p=0.99. It shows that the techniques of Event construal are diverse in text and image. To find out which parameters display higher variance, we conducted One sample T-test. It revealed two parameters whose values are relatively similar, these are 107/207 with t(1)=10.83, p=0.059, and 111/211 with t(1)=11.5, p=0.055. This means that in most cases intersubjectivity and manageability in Event construal are reduplicated in text and image, probably for the sake of their focalization. Among the parameters with higher variance, we should name 114/214 with t(1)=1.24, p=0.431, 122/222 with t(1)=1.94, p=0.302, 119/219 with t(1)=2.14, p=0.278, 109/209 with t(1)=2.2, p=0.272, 123/223 with t(1)=2.26, p=0.265, 127/227 with t(1)=2.26, p=0.265. They are Instantaneous event with higher values in text, Event without location in space with higher values in text, Undesirable event with higher values in text, Event not displaying any relations with higher values in image, Event located in time with higher values in text, Causal event with much higher values in text. This means that the text component is more responsible for Event construal techniques, although these are several techniques displaying higher activity in image construal, with Incomplete, interrupted event, Continuous event, Event lacking evaluation, Event located in space, Event without location in time, Sporadic, unitary event, Descriptive event among them.

What is of importance, is the contingency values of Event construal parameters. With critical r(54) of 0.34 at p=0.01 and 0.26 at p=0.05, there is a large number of those which display contingency, however in almost half of the cases this contingency is the result of the specifics of parameter annotation system which allows only one choice at each parameter group and therefore, the parameters work in counteraction. Considering this fact, I will give single parameters (in case there is counteraction) or parameter groups (in case there are more than two parameters representing each group) which demonstrate coordination prevalence in text and image construal. Among parameter groups I will name the event type, with high correlation values. We observe them in the correlation pairs including the parameter Shifting, changing event in image and text, for instance in the correlations of this parameter in image and Intersubjective event in text (r(54)=0.68), this one and Manageable, planned event in text (r(54)=0.71), this one and Completed, resultant event in text (r(54)=0.68). Among single parameters of image having rigid correlations with text parameters are Continuous event (True, veritable, real event with r(54)=0.79, Manageable, planned event with r(54)=0.83, Completed, resultant event with r(54)=0.89), Purposeless, aimless event (True, veritable, real event with r(54)=0.75, Intersubjective event with r(54)=0.72, Manageable, planned event with r(54)=0.77, Completed, resultant event with r(54)=0.77, Sporadic, unitary event with r(54)=0.69, Descriptive event with r(54)=0.7). Displaying lower, but still significant correlations are the parameters Event located in space, Event without location in time, Descriptive event. Overall, we observe almost strict regulations in Event construal parameter distribution in text and image, with continuous changing events located in space lacking evaluation in image, and resultant causal purposeful manageable events located in time in text.

However, it is worth mentioning how text and image can comply to enhance the construal specificity of one and the same parameter or parameters. The resonance analysis applying the procedure of overlapping the annotation results in text and image event construal showed that in some cases parameter activation occurs simultaneously in text and image, obviously to enhance the information transferred. In Figure 6 I present the parameter activity results in text, image and the resonance values.

Fig. 6. Contrastive data on parameter values in text, image, and their resonance values
Рис. 6. Сопоставление данных о значениях параметров в тексте,  изображении и их резонансных значениях

In Figure 6 the parameter codes are presented in the unified way which allows to consider text, image and resonance parameters on the same grounds. The diagram shows that in some cases the resonance values are almost similar to the values of the parameters in text or image. Among them are the events displaying Truth, Shifting, changing event, Completed, resultant event, Continuous event (almost all cases of continuous event construal in text are reduplicated in image), Purposeless, aimless event, Event located in space (almost all cases of continuous event construal in text are reduplicated in image), Sporadic, unitary event. Therefore, we may conclude that these event types are principal for both text and image event construal.

To select the samples for further oculographic analysis testing interpretation techniques of event construal in text and image, I conducted principal component analysis and received typicality and uniqueness values of each sample. Factor analysis revealed 7 principal components within the data with the Uniqueness indices ranging from 0.111 to 0.675. Among the samples I then selected the ones displaying high uniqueness, thus being atypical and presumably requiring cognitive load increase in their interpretation, and the ones quite typical which might not cause interpretation problems. I hypothesize that their eye-movement and discourse interpretations will be different.

Oculographic experiment  and discourse responses analysis

Next, I proceed with the results of oculographic experiment and discourse responses analysis.

6 samples were selected for the experiment, with the uniqueness indices in Event construal ranging from 0.23 to 0.57. They were presented to the participants one after another, however, they fall into 3 groups, demonstrating similarity in several parameters activity and distinct differences in others’.

First, I conducted contingency analysis with each Event construal parameter and eye movement reactions, here the contingency results of fixation duration within the areas of text and image. Overall, there were 54 probes, 8 probes of participants could not be considered due to inconsistent gaze paths received. In Table 2 Descriptive data on eye movement measures are presented.

Table 2 / Таблица 2
Eye movement measures of text and image AOIs / Показатели движения глаз в текстовых и графических зонах интереса

As seen, Shapiro-Wilk analysis reveals that the data have normal distribution, therefore I apply Pearson correlation procedure for contingency analysis. Pearson correlation analysis disclosed several parameters displaying rigid correlation with fixation duration distribution within the areas of text and image (at p=0.01 critical r(53)=0.35). The role of Event construal in the text was assessed first and the parameters contingent with gaze patterns distribution were detected. They are Event type featuring Creation, building, molding / demolishing, destruction event with r(53)=-0.45, Physical contact, perception event with r(53)=0.52, and Static event with r(53)=0.54, Event not displaying any relations with r(53)=-0.54, Event-achievement with r(53)=0.-45, Desirable event with r(53)=0.45, Event lacking evaluation with r(53)=0.45, Event located in time with r(53)=0.-45, Causal event with r(53)=0.-45, Descriptive event with r(53)=0.45. The results evidence that participants had text and image attention distribution in case the image displayed events showing physical contact, the event was static and desirable — under these circumstances the attention on the text component was longer as if helping to reconstruct the missing information. When the creation or destruction event, event-achievement, event located in time or causal event was not present in the text, the participants were more likely to address the image, not the text.

Next, I assessed the role of Event construal in the image. It is worth mentioning that the number of parameters contingent with statistically significant gaze patterns redistribution is twice smaller, which evidences that in general event construal by means of the image is less expected, its expected role is much higher in the text component. Among the Event construal parameters contingent with attention distribution were Intersubjective event with r(53)=0.45, Event without location in time with r(53)=-0.66, Desirable event with r(53)=0.52, Event lacking evaluation with r(53)=-0.64, Event located in time with r(53)=0.52.

Therefore, if the information load typical of the text component is shared in the image component it is the image that attracts longer fixations, perhaps because in this case the event construal is less typical and demands more concentration, the second explanation although is that the image component is enough to reconstrue the information. There are several parameters of event construal whose role is decisive in attention distribution between text and image components and whose activity affects both text and image construal. They are Relation construal in terms of Intersubjectivity, Event location in time and Desirable event, with the last one working differently. In case of its presence either in image or in text it makes the fixation duration longer. We might conclude that the presence of evaluation in all cases affects the fixation duration increase.

Second, I proceed with the discourse responses analysis.

The same correlation analysis procedure was carried out to see where the choice for discourse types applied (descriptive, narrative, and speculative) was contingent with any of the Event construal parameters. The results have revealed that Descriptive discourse responses were stimulated (being highly contingent) by the presence of the following parameters in the text construal: Shifting, changing event with r(53)=0.92, Interactional event (between man and object) with r(53)=0.46, Manageable, planned event with r(53)=0.92, Purposeless, aimless event with r(53)=0.92, Event lacking evaluation with r(53)=0.92, Event without location in space with r(53)=0.46, Event without location in time with r(53)=0.44, Sporadic, unitary event with r(53)=0.44, Descriptive event with r(53)=0.92. Narrative discourse responses were stimulated by the following parameters in the text construal: Creation, building, molding / demolishing, destruction event, Non-manageable, spontaneous, accidental event, Event-achievement, Desirable event, Event located in time, Causal event, all with r(53)=0.75, Repeatable event with r(53)=0.53. Speculative discourse responses were stimulated by the following parameters in the text construal: Interactional event (between man and object), Event without location in space, both with r(53)=0.46, Repeatable event with r(53)=0.62. Thus, the choice of Speculative discourse responses is of a more individual character, although the results suffice to assume which Event construal parameters are highly provocative.

Now, I will report the results on the discourse responses contingent with Event construal in the image. Descriptive discourse responses were contingent with the presence of the following parameters: Shifting, changing event with r(53)=0.49, Intersubjective event with r(53)=0.92, Manageable, planned event and Completed, resultant event, both with r(53)=0.44, Continuous event with r(53)=0.49, Undesirable event, Sporadic, unitary event and Causal event, all with r(53)=0.44. Narrative discourse responses were stimulated by the presence of following parameters: Event not displaying any relations with r(53)=0.75, Nonmanageable, spontaneous, accidental event, Incomplete, interrupted event, Event lacking evaluation, Repeatable event, and Descriptive event, all with r(53)=0.53. Speculative discourse responses were contingent with the following parameters: Shifting, changing event with r(53)=0.46, Non-manageable, spontaneous, accidental event and Incomplete, interrupted event, both with r(53)=0.62, Continuous event with r(53)=0.46, Event lacking evaluation, Repeatable event, and Descriptive event, all with r(53)=0.62.

Contrastive analysis of contingency results demonstrates that Descriptive discourse responses are stimulated by different Event construal parameters in text and image, and there are a lot of parameters displaying rigid correlation with the choice of this response type, whereas Narrative discourse responses are stimulated by partially the same parameters in text and image, and their number is significantly smaller. It might mean that the choice of Narrative discourse responses is rather predetermined and much easier predicted in case several distinct parameters of Event construal were present. The situation with Speculative discourse responses choice is more complicated. I have detected a vast number of Event construal parameters in the image which stimulate the use of Speculative responses, whereas in the text these parameters are few. Besides, the image parameters contingent with Speculative discourse responses display non-manageable, continuous, incomplete character, thus lacking focusing potential, which in turn might have affected the choice of Speculative discourse.

Concluding remarks

The research aimed at testing Event construal specifics in the heterosemiotic format of book page has allowed to reveal the construal techniques in text and image, verify their interpretational value, and disclose the discourse response types contingent with these techniques.

The results conform with the experimental findings reported in [Kaspar, König 2011; Kirtley 2018] in the part concerning the types of parameter activity distribution in text and image components in heterosemiotic formats. However, in our study we do not focus on the synchronization techniques of domineering, coordination, and reduplication revealed in their works, but explore the contingency of Event construal parameters in text and image. The key issue is the techniques themselves in terms of their interpretation. The research has proved that the factors of focusing and typicality play a less significant role in heterosemiotic formats of information, compared with monosemiotic formats studied in terms of salience effects [Giora 2003], the effects of attractors [Gibbs, Tendal 2006], and entrenchment effects [Schmid 2016]. The study has explored the factor of combinatorial prevalence which adds to the effects of focusing and typicality.

The study has proved that Event construal parameters explored in [Tenny, Pustejovsky 2000] to describe text construal specifics can be applied to analysis of images with the idea of exploring both semiotic forms on the same grounds. Apart from that, several of these construal parameters demonstrate high interpretation value, at least in terms of text and image heterosemiotic construal. This approach integrating construal as both information generation and information processing allows to detect and verify the construal techniques outlined in cognitive linguistics and semiotics. However, the idea of their integrating in application to Event construal analysis is not a new one. The parameters bearing relevance to the interpretation were explored in [Demyankov 1983], but in his study mostly morphological and syntactic techniques were deduced, with the non-verbal parameter groups left unverified, although they were specified.

Among the most important findings of this study are the parameter activity and contingency results in Event construal in text and image, with the system of 28 parameters developed and applied for annotation and statistical processing. Their coordination and reduplication effects as components of combinatorial prevalence were also disclosed, which may pave the way for prognostic fundamental and applied studies assessing the role of these parameters or aiming to devise a successful (with the readers) heterosemiotic format.

Parameter activity and contingency results served to detect the uniqueness and typicality effects among the data samples, which in their turn later became stimuli in the oculographic experiment accompanied by comprehension checks aimed at exploring the interpretation value of diverse construal techniques. The results suffice to state that the presence of several parameters in text or image is highly resonant with the gaze reactions of the participants, moreover, the gaze patterns are also dependent on the semiotic format in which these parameters are activated, which suffices to specify different interpretation roles of text and image. The experiment also included the study of participants discourse reactions, which proved highly resultant, since it revealed how the choice of discourse responses is stimulated.

Overall, hopefully the procedure developed and the results achieved may be used to predict the interpretation effects, select the proper construal techniques to achieve the necessary result, and help verify cognitive linguistic and semiotic data.

About the authors

Maria I. Kiose

Author for correspondence.
Email: maria_kiose@mail.ru
ORCID iD: 0000-0001-7215-0604

Doctor habil. of Philology, Associate Professor, Leading Researcher of the Centre for Socio-Cognitive Studies

References

Supplementary files

Supplementary Files

Action

1. Fig. 1. Sample 1

Download (340KB)

Indexing metadata

2. Fig. 2. Event level structure and annotation procedure in UAM Image Tool

Download (467KB)

Indexing metadata

3. Fig. 3. Experiment stimuli 1–6

Download (786KB)

Indexing metadata

4. Fig. 4. HETEROSTAT window processing the data on Event construal

Download (555KB)

Indexing metadata

5. Fig. 5. Text and Image profiles of Event construal

Download (55KB)

Indexing metadata

6. Fig. 6. Contrastive data on parameter values in text, image, and their resonance values

Download (51KB)

Indexing metadata