Metaphoric gestures in simultaneous interpreting

Cover Page

Cite item

Abstract

The paper deals with the degree to which interpreters incorporate visible behaviors from the people they are interpreting into their own practice. Since metaphoric gestures objectify abstract concepts in visible form, it is worth exploring the degree to which interpreters replicate such gestures of those whose speech they are interpreting; this can indicate how much they are employing the original speakers’ mental imagery connected with those abstract concepts. This imagery for the source domain of the metaphor ranges from highly iconic (high metaphoric) to low in iconicity (low metaphoric). The hypothesis is that interpreters use low metaphoric gestures rather than high metaphoric ones, due to the discourse type (interpreted speech). We performed formal visual and semantic analyses of ten-minute videos of interpreting a scientific lecture for the general public on a psychological topic from English into Russian. First, we analyzed the functions of the gestures in the source videos to identify metaphorically used gestures (e.g., depicting abstract ideas); then we studied the functions of the interpreters’ gestures. The results indicate a predominance of low-level, schematic metaphoricity in the interpreters’ gestures (e.g., simple ontological metaphors, as if presenting ideas on the open hand). Such results might be explained by the time pressure which leads to a decrease in mental imagery of the interpreters. We see a difference between the known role of gestures when speakers are formulating their own ideas (in thinking for speaking) and their role in simultaneous interpreting (when speakers are rendering others’ ideas, rather than forming their own ones).

Full Text

  1. Introduction

Human communication has been regarded as something more than purely speech production for quite a long period of time. As part of the process of speaking, people normally not only produce sounds, which then form meaningful units, but also make movements which also bear meaning and add to the communication process. Such movements are known as co-speech gestures and their role in people’s language production has been a subject of debate. There are different points of view, as there is evidence that such gestures not only help create meaning but also help others to understand what is being said. Some scholars state that gestures are used in order to support the verbal context (Drijvers & Özyürek 2017), give extra information, clarify, and focus the attention of the interlocuters on some points, etc. (Dargue et al. 2019, Goldin-Meadow & Alibali 2013, Hilliard & Cook 2016, Hostetter 2011, Melinger & Levelt 2004). However, other researchers highlight the ability of gestures to help with speech production, which makes them important for the speaker. For example, gestures are used by blind speakers in front of other blind speakers (Iverson & Goldin-Meadow 1998) or when speaking on the phone (Wei 2006). There are also claims that it is likely that gestures were the first form of communication, before spoken, verbal language, language appeared (Corballis 2012). Gestures are used across all cultures and languages. They are one of the first means that babies use to communicate with people around them before they learn how to speak.

As gestures are a part of our natural speech and communication processes, they are not unique for each person. Many gestures are repeated in different contexts by different people. However, some gestures are more complex than others. Gestures which entail some form of metaphor are inherently depicting some entity, process, or relation, and in this regard, they are commonly more complex in form (see, for example, Cienki & Müller 2008) than gestures serving common pragmatic and interactive functions (as described in Bavelas et al. 1992, for example). The research question is: if the same information is being presented in different contexts (i.e., by the speaker in the source video and by the interpreters during their performance), will that influence gesture production? There is evidence, for example, that when one sees a person telling a story, one is likely to use the same words and gestures when retelling the story later as the original speaker did (Cassell et al. 1999, McNeill et al. 1994). We suppose that there is a possibility of gesture mirroring during the process of interpreting, i.e., the participants are affected by the nonverbal behavior of a speaker whom they see on the screen and repeat gestures after the speaker.

  1. Metaphor and gesture

The notion of metaphor and its nature have been discussed in linguistics and other fields for decades. Metaphor can be described as a conceptual mapping from one domain to another (Lakoff 1993). Conceptual Metaphor Theory is an approach where metaphor is viewed as being based on connections and similarities between two domains of knowledge or experience, which can even be reflected in manual movements (involving depiction of imagery). Thus, language is not the only modality where metaphor can be represented: metaphorical expressions can also be realized in a non-verbal modality (Cienki & Müller 2008). According to Jakobson, metaphor and metonymy are cognitive and semiotic strategies that play an important role in structuring both verbal and non-verbal messages (Jakobson 1956, 1987).

A multimodal metaphor involves cases when the source and target of the metaphor are represented in different modes (Forceville 2009). Embodiment of a source domain of such a conceptual metaphor does not represent a random movement, but rather it usually involves recurrent forms and form-meaning mappings (Calbris 2003, 1998, 2005, McNeill 1992, 2005, Mittelberg 2006, Müller 2008, Tong 2023).

When discussing metaphor in gesture, it is also of importance to mention the notion of metonymy. Metaphor and metonymy represent cognitive processes of conceptualization and association which are implemented in different forms: verbal and non-verbal. From this theoretical perspective, cognitive linguists debate about the differences between metaphor and metonymy. Whereas metaphor is grounded on cross-domain mappings, metonymy is based on the mappings from within the same domain (Barcelona 2000). Following Mittelberg and Waugh we assume that metonymy leads the way to metaphor: metonymic mapping between a hand movement and a described imaginary object is a prerequisite for the metaphorical mapping between the object and the abstract idea (Mittelberg & Waugh 2009).

Gestures co-occur with speech simultaneously or can anticipate, or even sometimes follow, the verbal output (a word or an utterance). Many verbal concepts, whether abstract or concrete, can be embodied in gestures, particularly metaphoric ones. For our study, we will follow the definition of metaphoric gestures introduced by Cienki and Müller: “…metaphoric gestures… [are] the ones which have the potential to engage an active cross-domain mapping, that is, the cognitive process of understanding something in terms of something else” (Cienki & Müller 2008: 485–486).

Co-speech gestures are present in all types of contexts and discourse. In the current study we explore gestures and their role in simultaneous interpreting (SI). SI entails decoding information from a source language and rendering it as quickly as possible in a target language. This means that several mental processes are active simultaneously, namely attention, perception, reasoning, and working memory (Dayter 2020, Gósy 2007). The Cognitive Load Model (Seeber 2011) and Effort Model (Gile 2009) portray SI as a process which includes four tasks: comprehension, language production, memory storage, and coordination. In order to complete all these tasks completely one needs to keep them in balance and to reduce cognitive overload. To do that, interpreters can use gestures, as they are regarded to be one of the means to deal with stress and cognitive load (Poyatos 1987/2002). For example, Kita et al. (2017) state that hand gestures have a self-oriented function, as they help people with the process of conceptualization, especially if it involves spatial concepts. In addition, gestures of speakers from source texts are also important for SI. In one of his works, Seeber describes how interpreters use gestures to search for additional information when it can be expressed via body or hand movements of the speaker (Seeber 2012, 2017).

The present work will concentrate on metaphoric co-speech gestures, as we regard them an important part of SI that can give an insight into the cognitive processes involved in the interpreting. The mapping of the domains reflected in the combination of movement, the shape of the hands, and the speech might help with processes of conceptualization which can influence and improve the interpreting.

  1. Data and methods

3.1. The video data and methodology

To conduct our experiment, a special setting was created which allowed us to put the participants in conditions similar to what they experience during interpreting at work. For that, an interpreting booth was recreated in the laboratory where the experiment was filmed. To recreate such a booth, we used a table, a computer screen and a large projector screen put in front of the table which blocked the view of the rest of the room. To film the interpreting process, we used two cameras. A wide-angle camera was put in front of the interpreters under the computer screen and a large professional camera with a microphone was standing on a tripod behind the interpreter, to the right. Such camera positioning was used during previous stages of this study and proved to be the best way of filming the participants’ behavior as it allows one to see nearly all hand gestures and other body movements very clearly and precisely. In addition, each interpreter wore eye-tracking glasses (Tobii Pro Glasses II) while completing the interpreting task. These glasses were used to track the eye movement of the participants when they were looking at the computer screen in front of them and they have a built-in camera which showed another angle, i.e., the speaker’s perspective while engaged in the interpreting.

Ten videos of interpreting from English into Russian (ten minutes each) were analyzed (100 minutes in total) for the use of gesture, in addition to the ten-minute source video.

The experiment included several steps. During the very first stage the participants received a vocabulary list several days before the experiment in order to prepare for the interpreting session. The second stage was filming, during which the participants came to the laboratory, signed an informed consent agreement and then were asked to take a seat in the improvised booth. We asked them not to take anything with them for the purposes of the experiment so that they would not be distracted by any objects. In addition, the lack of a paper and a pen to write down words and numbers increased the cognitive load experienced during the process, and one of the aims of our study was to see how the interpreters would handle an increased cognitive load. Then, the participants performed the interpreting of two ten-minute popular science videos about psychology and psychological disorders, with one video involving a high number of gestures by the speaker (373) and the other in which the speaker gestured much less (90 gestures). For this analysis, we used the interpretations of the source video in which the lecturer produced a large number of gestures.

After filming, the videos from the three cameras used were joined to create one composite video using the program Adobe Premiere Pro. The final version was then put into a special software for annotating speech and gesture in audiovisual recordings: ELAN (https://archive.mpi.nl/tla/elan). The annotation was done with tiers that can be created in the program (see Figure 1). The following tiers were taken into account in the current study: the speech, in which we annotated what the participants were saying; the source text, which included the speech of the speaker from the video; gesture phrases, which was used to annotate hand movements of the participants and categorize them into sub-tiers (adapter, pragmatic, representational, deictic) and comments in which special cases or questions were marked by the annotators.

Figure 1. An example of annotation of the source video in ELAN

Figure 2. An example of annotation of the target video data in ELAN

To explore the question of metaphoric gesture mirroring, we annotated gestures both in the source (Figure 1) and target videos (Figure 2). After that, we extracted tiers with representational and pragmatic gestures from the source video and inserted them into ELAN files with the analysis of the interpreting to identify cases of gesture overlap. Only those cases when the speaker’s gestures started before those of the participants were counted (the overlapping in the beginning of the participants’ gestures), as the interpreters need to see at least the beginning of a gesture in order to be able to repeat it.

 3.2. Metaphoric properties of gestures

Gestures were analyzed per their form (Bressem 2013) and function. In the current study we followed the idea that gestures can be polysemantic in nature, i.e., that one gesture may serve not one, but several functions in speech simultaneously (Calbris 2011, Kok et al. 2015). There are four main categories that we used: pragmatic, representational, deictic gestures and adapters.

Pragmatic gestures are hand movements that have discourse-related properties, in order to engage in word searches, highlight or structure some parts of discourse, etc. (Dressel 2020, Fricke 2013, Ladewig 2014, Streeck 2009a). They can be also used when one expresses an attitude or evaluates the topic of the speech. Representational gestures convey semantic meaning by the form and/or movement of the hand(s). Such gestures share some physical properties of their referent (Müller 2014). Deictic gestures are used to create vectors to show the direction or location of some notion in space (Fricke 2002). Adapters are gestures without any semantic function. They are movements like rubbing one’s own hands, scratching, adjusting one’s clothes, etc.

In this article we regard only gestures with different levels of metaphoricity; for this reason, we explore only two major categories: representational and pragmatic gestures. As it was mentioned above, adapters do not involve any semantics, thus, they cannot form metaphors. Deictic gestures are regarded as instruments to create vectors and show locations, and for this reason they were also excluded from the analysis.

Each category has several subcategories. Representational gestures were divided into five subcategories, using an adaptation of Müller’s (1998, 2014) modes of representation: holding, molding, acting, embodying and tracing. Pragmatic gestures were divided into six subcategories: discourse structuring, emphatic, contact establishing, expressing attitude, negation and word search. The forms are illustrated in the following examples.

Holding and molding gestures are used to describe the shape/boundaries of some object/notion and usually include some tension in the hand that is used for depiction (cf. Streeck’s [2009b, Ch. 6] category of ‘handling’). As can be seen in Figure 3, the participant is using her left hand to show the abstract notion of “no connection”/ “нет связи” by holding her left hand up, palm up, and fingers a bit crooked, as if the notion were in her hand.

Acting gestures are used to show some action/movement/process with one’s hand (cf. Streeck’s [2009b, Ch. 6] category of mimesis, depicting action). For instance, the interpreter in Figure 4 is using her hands to show the process of opening files, by turning her palms outward.

«… (нет) связи» / “… (no) connection
Figure 3. Holding gesture

«… мы открываем файл…» / “…we open the file…”
Figure 4. Acting gesture

Embodying gestures represent objects/notions as if the speaker’s hands were the given entity being mentioned. As it is shown in Figure 5, the interpreter is showing two categories, and these categories are represented via her two open hands, palm down. The distinction from the acting mode, described above, is determined in relation to the speech. In Figure 5, the interpreter is not talking about touching (which could be done with open hands), and the hand shape and movement are not iconically related to the verb mentioned (“divided”); rather, the two hands are held in the air with some effort (with tension in the extended fingers, and with hands elevated above the table) when mentioning “two categories”.

«… делится на две категории…» / “… is divided into two categories…”
Figure 5. Embodying gesture

Tracing is used to outline an object/notion and show its shape or a path of movement. As it can be seen in Figure 6, when the interpreter is showing the degree, he puts his hand lower.

«… насколько далеко они должны быть…» / «… how far they are supposed to be….”
Figure 6. Tracing gesture

Pragmatic discourse structuring gestures have several forms, e.g., palms opposite each other, palms up away the body, etc. The main idea is that a part of discourse is being represented or manipulated in gesture (cf. Streeck’s [2009b,  Ch. 8] gesture function called ‘speech handling’) rather than some notion or object and its form (as in Figure 7).

Emphatic gestures are simple biphasic (back and forth) beat movements that do not add any semantic information to the speech (see Figure 8).

«… простой…» / “… simple…”
Figure 7. Discourse structuring gesture

«Давайте / “Let’s       запишем» / write it down”
Figure 8. Emphatic gesture

Negation gestures usually have the ‘away’ movement in them, as if a speaker brushes off the information (Гришина 2015, Bressem & Müller 2014, Harrison 2014, Kendon 2004). In Figure 9, the participant is moving his left hand away from his body, palm down, when using the negation particle ‘not’.

Word search gestures are used when one is in the ‘tip-of-the-tongue’ state and stimulates the process of lexical retrieval by using gestures (e.g., with “motor gestures”: “simple, repetitive, rhythmic movements that bear no obvious relation to the semantic content of the accompanying speech” [Krauss et al. 2000: 263]). In Figure 10, the participant is moving his right hand from side to side while searching for the correct equivalent in the target language.

«… когда это уже больше не нужно» / “… when it’s not necessary anymore
Figure 9. Negation gesture

«… некоторые эээ элементы должны сохраняться» / “… some ehm elements should be kept”
Figure 10. Word search gesture

 In this article we also divide gestures according to their metaphoric properties, i.e., that they metaphorically represent something in addition to the speech. There are two categories that we highlight: high metaphoric gestures and low metaphoric gestures. High metaphoric gestures are representational ones clearly involving at least one of the modes of representation: holding, molding, acting, embodying, and tracing. Low metaphoric are pragmatic gestures with one of the following functions: discourse structuring, negation, or word search.

We regard all representational gestures affiliated with verbal expression of abstract concepts as high metaphoric gestures as they involve comparison of the abstract ideas being verbalized with some physical forms depicted gesturally. This can be seen, for example, in the form of the handshape, the use of the hands to show borders (holding, molding, embodying, tracing) or via the hands’ movement (acting). In McNeill’s classification system, representational gestures are divided into iconic gestures, which depict physical properties of an object, and metaphoric gestures which shape abstract ideas into some concrete form. However, the author also states that iconicity, metaphoricity, deixis and temporal highlighting should be regarded as different dimensions of one gesture rather than as mutually exclusive categories (McNeill 2005).

Low metaphoric gestures are regarded as such because of their vague resemblance to some process or notion that they possess in their form and/or semantics. For example, discourse structuring gestures are usually formed with two hands opposite each other, as if the speaker is shaping some parts of their discourse, presenting it to the listeners. The idea of presentation through holding discourse in one’s hand gives such gestures metaphorical properties (Lapaire 2016, Müller 2004). In many European languages, gestures expressing negation involve movements of the hand with the palm facing away from the speaker (Bressem & Müller 2014). The idea of moving the concept that the speaker does not agree with away from him/her is regarded as entailing metaphor because the idea is not a physical object which can be moved. The last category, word search gestures, entails circular movements that bear the idea of a repeated movement, a thought in process, which stops when the correct word is found and uttered. Such categories of pragmatic gestures as emphatic, contact establishing and expressing attitude are not regarded as metaphorical at all, though they are still important for the purposes of the study as will be discussed below.

The idea of the greater or lesser resemblance of the hands with the source notion creates a metaphoric property in the types of gestures discussed above. In the current paper we explore if such gestures are numerous in interpreters’ speech and whether their use can be traced back to the original lecturer’s gestures.

In the example illustrated in Figure 11 we can see that the speaker in the source video explains the specifics of our memory and how we can get access to it. While describing the idea, the lecturer uses representational gestures with a holding function: first she performs the gesture using two hands and then switches to her right hand while uttering “…access to the content of those memories”. The representational gestures are used by the speaker to describe different abstract notions, e.g., “access”, “content of those memories”. If we analyze the conduct of the simultaneous interpreter while translating this idea, we can observe that:  1) while describing the state (“We are often even unconscious”) the participant uses a pragmatic gesture which serves the function of “Expressing attitude”; 2) in the utterances “…that this memory…” and “…that we have those memories” the interpreter switches smoothly to a representational gesture with a holding function. As we can see, both the speaker in the video and the interpreter use representational and pragmatic gestures while describing abstract notions. In this case, it is interesting to note that the interpreter uses a representational gesture which is similar in its function to the one performed by the speaker.

The hypothesis of the current study is as following: the interpreters will mostly use low metaphoric gestures rather than high metaphoric, as this type of discourse involves explanation of an abstract topic, which will influence their nonverbal behavior. The results of the analysis conducted to test it are presented below.

Figure 11. Comparison of source and target gestures 

  1. Results

The analysis of the source video (the stimulus for the interpreting) showed that the speaker in that video used many gestures, namely 373 gesture phrases in total. The comparison between the use of pragmatic and representational gestures shows the predominance of the first type: 237 vs 100 (Table 1). Such results might be dictated by the topic and the content of the speech, as the speaker explains terms and some scientific facts to the public, thus there is a tendency to organize the speech or to emphasize its parts. This claim is also supported by the fact that there is a predominance of pragmatic discourse structuring and representational holding gestures.

Table 1. Gestures in the source video

Representational gestures

Pragmatic gestures

holding

57

discourse structuring

121

molding

11

emphatic

65

acting

20

contact establishing

19

embodying

12

expressing attitude

21

tracing

0

negation

8

 

 

word search

3

 

Figure 12. Gestures in the source video

Figure 13. Gestures during interpreting

The analysis of the interpreters’ nonverbal behavior showed that in total they used 644 pragmatic and 105 representational gesture phrases. The results support our hypothesis that interpreters tend to use pragmatic gestures with low metaphoric potential. As we can see, the amount of representational gestures is even lower, proportionally, in comparison to the amount used in the target video (Figures 12, 13). That might be explained by the fact that the interpreters are to translate someone else’s speech rather than produce their own discourse aimed at explaining things to the audience. Thus, they use gestures in order to structure their interpreting rather than using gestures aimed at illustrating their own notions or ideas.

Comparing the analysis of the source video to that of the interpreters’ behavior, the results show that the interpreters tended to use pragmatic gestures even when they saw representational gestures on the screen. As we can see from the Table 2 (see below), many times when the original lecturer used gestures of high metaphoricity, the interpreter used gestures with low metaphoricity (for discourse presenting) or with no metaphoricity at all (emphatic gestures). On the other hand, there are also a number of occasions when the interpreters used representational gestures despite the fact that the original speaker was performing pragmatic gestures (38 cases). In most cases, however, the participants used gestures similar to those of the person on the screen.

 Table 2. Gesture overlap 

 

Speaker

Representational

Pragmatic

hold

mold

act

mbdy

trace

disc

emph

cont

attit

neg

wrd srch

Total

Interpreter

Representational

hold

3

0

3

1

1

2

5

2

0

1

0

18

mold

1

0

0

0

0

1

2

0

0

0

0

4

act

2

0

2

0

0

7

3

1

0

1

0

16

mbdy

2

0

1

1

0

1

4

1

0

0

0

10

trace

0

0

0

0

0

4

2

0

2

0

0

8

Pragmatic

disc

22

2

6

5

0

44

18

10

5

0

0

112

emph

22

3

7

10

0

29

42

6

4

3

3

129

cont

1

0

1

0

0

2

2

0

0

2

0

8

attit

3

0

1

0

0

4

0

0

1

1

0

10

neg

2

0

1

0

0

1

3

0

0

2

0

9

wrd srch

2

1

0

0

0

7

3

0

0

0

0

13

 

TOTAL

60

6

22

17

1

102

84

20

12

10

3

337

The analysis of each participant and their gesture profile reveals that though the number of gestures is different for each participant, there is a strong tendency to use pragmatic gestures over representational ones, as can be seen in the Appendix. Table 2 shows the results of pragmatic and representational gestures used without considering the correlation with gestures from the source video. As it can be observed, the use of gestures with high metaphoricity is scarce. The interpreters prefer gestures with low metaphoricity or without any metaphorical qualities at all.

  1. Discussion

The analysis of 10 videos of participants interpreting a video lecture from English into Russian showed a tendency toward using pragmatic gestures with low or no metaphoricity at all. These results might be influenced by the task, as the cognitive load that the participants’ experience might lead to the simplification of gestures. Representational gestures and their high metaphorical properties might be too difficult to produce immediately after the speaker, as such gestures are more complex in their form and require more cognitive resources than the participant can use at the moment of interpreting. However, the results also showed some cases when the interpreters used representational gestures while the speaker was producing pragmatic ones. We assume that this could happen because such gestures were triggered by particular words or notions which were difficult to interpret and the participants used representational gestures to facilitate lexical retrieval (Morsella & Krauss 2004). Nevertheless, the interpreter does not have the time to absorb and process information in the same way as in experiments where listeners reported information after listening to a story and then produced gestures similar to those of the storyteller (Cassell et al. 1999, McNeill et al. 1994). In situations like those of the current study, however, the lack of time to absorb information could influence the kinds of mental representations or growth points (McNeill 1992, 2005, 2013, McNeill & Duncan 2000) that interpreters employ, from which speech and gesture arise.

Such results might also be influenced by another factor, which is the origin of the speech. The interpreting is neither speech nor thought that solely belongs to the interpreter, but it is someone else’s speech. Thus, interpreters might not concentrate on the content of the speech or the speaker’s behavior as much as on the interpreting, i.e., transferring information correctly from one language into another. This main task might be best accomplished when one is using gestures related to discourse, discourse structuring and emphasis, rather than representational gestures that correspond to separate notions or things. In this case, the use of a gesture can be dictated by the communicative intention rather than mental simulation (see The Sketch Model of speech and gesture production [de Ruiter 2000]). In addition, the analysis of the source video also revealed that the speaker predominantly used pragmatic gestures, especially those aimed at structuring discourse or emphasizing its parts. This predominance might be provoked by the type of discourse, as one of the primary aims of the talk was to share information and knowledge. In such situations people tend to use less complex or informative types of gestures (Gerwing & Bavelas 2004).

The hypothesis was also supported by the analysis of the individual profiles of the interpreters, as the results showed that all participants but one used pragmatic gestures far more frequently. The only participant who used representational gestures more frequently did not have many hand movements in general (only 8 gestures in total of the categories we coded).

  1. Conclusion

The present study showed some peculiarities of gestural representation of metaphor performed by simultaneous interpreters. Usually, a simultaneous interpreter works in a special booth, thus there is normally no counterpart that sees their performance. That can have consequences for the behavior of interpreters, which is primarily aimed at lexical retrieval and speech production. The results of our research show that the simultaneous interpreters tended to use gestures with low or even no metaphorical properties. This can be explained in part by the settings of their work, where the primary goal consists in producing the output which will represent the source text as well as possible. Given that SI is performed under severe time pressure, the interpreters mostly tend to use gestures for themselves, as opposed to producing them for an interlocutor (as there is no interlocutor present). This explains the predominance of pragmatic gestures, in particular discourse structuring and emphasizing gestures. Performing gestures normally involved in the presentation and emphasis of ideas may help the interpreter with their own speech production, regardless of the specific ideas that they are rendering at the moment (e.g., Cienki 2023, Lucero et al. 2014, Vilà-Giménez & Prieto 2020). It is different from producing gestures that relate to specific ideas involving particular imagery. This could explain why representational gestures with high metaphorical properties were not used that frequently; such depictive gestures demand more cognitive effort in order to engage with the specifics of the imagery being mapped from a metaphoric source domain. A problematic situation would appear, as the mental resources of the interpreters are already concentrated on a highly demanding cognitive task. Metaphoric imagery was thus found to be reduced in the gesturing of the interpreters studied in comparison with that used by the lecturer in the source video. This suggests a difference between the known role of gestures when speakers are formulating their own ideas (in thinking for speaking) and their role in simultaneous interpreting (when speakers are rendering others’ ideas, rather than forming their own).

Further research will focus on the eye-tracking data that was collected. This might help us gain a better understanding of whether the metaphoric character of the interpreters’ gestures (high versus low) depends on the verbal context in which they are used or on their being triggered by what the participants see on the screen. The semantics of the lexical units used with metaphoric gestures by the original lecturers might have had an impact on how gestures are used by the participants, especially when they needed to interpret complex abstract notions.

 

Appendix

 Table 1. Individual profiles of the participants

Participant

Pragmatic

N

Representational

N

MH-01M

2301

31

2201

6

2302

27

2202

1

2304

4

2203

1

2305

2

2204

2

2306

5

2205

1

TOTAL

 

69

 

11

MH-02M

2301

37

2201

1

2302

69

2203

1

2303

1

2204

5

2304

1

2205

2

2305

3

  

2306

1

  

TOTAL

 

112

 

9

HM-03M

2301

17

2201

4

2302

26

2202

1

2303

6

2203

6

2304

7

2204

3

2305

6

2205

1

2306

10

  

TOTAL

 

72

 

15

MH-04M

2301

4

2201

1

2302

3

2202

1

2303

1

2203

1

2306

3

  

TOTAL

 

11

 

3

HM-05M

2301

1

2201

2

2302

39

2202

3

2303

1

2203

3

TOTAL

 

41

 

8

HM-06M

2301

46

2203

1

2302

12

2204

1

2303

2

  

2305

1

  

TOTAL

 

61

 

2

MH-07M

2301

18

2201

5

2302

10

2204

1

2303

1

  

2304

9

  

TOTAL

 

38

 

6

HM-08M

2301

33

2201

8

2302

52

2203

11

2304

2

2204

2

2305

8

2205

5

TOTAL

 

95

 

26

HM-09M

2301

1

2201

1

2302

1

2202

1

2305

1

2203

2

  

2205

1

TOTAL

 

3

 

5

HM-10M

2301

41

2203

1

2302

12

2204

1

2303

1

  

2304

1

  

2305

1

  

2306

3

  

TOTAL

 

59

 

2

 

×

About the authors

Anna V. Leonteva

Moscow State Linguistic University; Institute of Linguistics, RAS

Author for correspondence.
Email: lentevanja27@gmail.com
ORCID iD: 0000-0001-7234-2999

holds a PhD in Linguistics. She is Head of the Sign Languages Laboratory and Senior Researcher at the Center for Socio-Cognitive Discourse Studies (SCoDis), Moscow State Linguistic University, and Junior Researcher at the Laboratory of Multi-Channel Communication, Institute of Linguistics of Russian Academy of Sciences. Her research interests are cognitive linguistics, co-speech gestures, cognitive load, multimodal communication, argumentative discourse.

Moscow, Russia

Alan Cienki

Vrije Universiteit Amsterdam

Email: a.cienki@vu.nl
ORCID iD: 0000-0003-2951-9722

PhD in Slavic linguistics and is Professor of Language Use & Cognition and English Linguistics in the Department of Language, Literature and Communication at the Vrije Universiteit Amsterdam, Netherlands. His research interests include cognitive linguistics, semantics, gesture studies, metaphor studies, and political discourse analysis.

Amsterdam, Netherlands

Olga V. Agafonova

Moscow State Linguistic University; Institute of Linguistics, RAS

Email: olga.agafonova92@gmail.com
ORCID iD: 0000-0001-8460-8555

Junior Researcher at the Center for Socio-Cognitive Discourse Studies (SCoDis), Moscow State Linguistic University, and Junior Researcher at the Laboratory of Multi-Channel Communication, Institute of Linguistics of Russian Academy of Sciences. Her research interests are cognitive linguistics, multimodal communication, gesture studies, cognitive load and political discourse.

Moscow, Russia

References

  1. Barcelona, Antonio. 2012. The cognitive theory of metaphor and metonymy. In Antonio Barcelona (ed.), Metaphor and metonymy at the crossroads: A cognitive perspective, 1-28. De Gruyter Mouton. https://doi.org/10.1515/9783110894677
  2. Bavelas, Janet B., Nicole Chovil, Douglas A. Lawrie & Allan Wade. 1992. Interactive gestures. Discourse Processes 15. 469-489. https://doi.org/10.1080/01638539209544823
  3. Bressem, Jana & Cornelia Müller. 2014. The family of Away gestures: Negation, refusal, and negative assessment. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body-language-communication: An international handbook on multimodality in human interaction, Vol. 2, 1592-1604. Berlin: De Gruyter Mouton. https://doi.org/10.1515/9783110302028.1592
  4. Bressem, Jana, Silva H. Ladewig & Cornelia Müller. 2013. Linguistic Annotation System for Gestures (LASG). In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Sedinha Teßendorf (eds.), Body-language-communication: An international handbook on multimodality in human interaction, 1098-1124. Berlin/Boston: De Gruyter Mouton. https://doi.org/10.1515/9783110261318.1098
  5. Calbris, Geneviève. 2003. From cutting an object to a clear cut analysis: Gesture as the representation of a preconceptual schema linking concrete actions to abstract notions. Gesture 3 (1). 19-46. https://doi.org/10.1075/gest.3.1.03cal
  6. Calbris, Geneviève. 2011. Elements of Meaning in Gesture. Amsterdam: John Benjamins Publishing Company. https://doi.org/10.1075/gs.5
  7. Cassell, Justine, David McNeill & Karl-Erik McCullough. 1999. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics and Cognition 7 (1). 1-34. https://doi.org/10.1075/pc.7.1.03cas
  8. Cienki, Alan. 1998. Metaphoric gestures and some of their relations to verbal metaphoric expressions. In Jean-Pierre Koenig (ed.), Discourse and cognition: Bridging the gap, 189-204. Stanford: CSLI Publications.
  9. Cienki, Alan. 2005. Image schemas and gesture. In Beate Hampe (ed.), in cooperation with Joseph Grady, From perception to meaning: Image schemas in cognitive linguistics, 421-442. Berlin/New York: Mouton de Gruyter. https://doi.org/10.1515/9783110197532.5.421
  10. Cienki, Alan. 2023. Self-focused versus dialogic features of gesturing during simultaneous interpreting. Russian Journal of Linguistics 27 (4).
  11. Cienki, Alan & Cornelia Müller. (2008). Metaphor, gesture, and thought. In Raymond W. Gibbs Jr. (ed.), The Cambridge handbook of metaphor and thought, 483-501. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511816802.029
  12. Corballis, Michael C. 2012. How language evolved from manual gestures. Gesture 12 (2). 200-226. https://doi.org/10.1075/gest.12.2.04cor
  13. Dargue, Nicole, Naomi Sweller & Michael P. Jones. 2019. When our hands help us understand: A meta-analysis into the effects of gesture on comprehension. Psychological Bulletin 145. 765-784. https://doi.org/10.1037/bul0000202
  14. Dayter, Daria. 2020. Variation in non-fluencies in a corpus of simultaneous interpreting vs. non- interpreted English. Perspectives 29. 489-506. https://doi.org/10.1080/0907676X.2020.1718170
  15. De Ruiter, Jan Peter. 2000. The production of gesture and speech. In David McNeill (ed.), Language and gesture, 248-311. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511620850.018
  16. Dressel, Dennis. 2020. Multimodal word searches in collaborative storytelling: On the local mobilization and negotiation of coparticipation, Journal of Pragmatics 170. 37-54. https://doi.org/10.1016/j.pragma.2020.08.010
  17. Drijvers, Linda & Asli Özyürek. 2017. Visual context enhanced: The joint contribution of iconic gestures and visible speech to degraded speech comprehension. Journal of Speech Language and Hearing Research 60. 212-222. https://doi.org/10.1044/2016_JSLHR-H-16-0101
  18. Forceville, Charles. 2009. Non-verbal and multimodal metaphor in a cognitivist framework: Agendas for research. In Charles J. Forceville & Eduardo Urios-Aparisi (eds.), Multimodal metaphor, 19-42. Berlin: Mouton de Gruyter. https://doi.org/10.1515/9783110215366.1.19
  19. Fricke, Ellen. 2002. Origo, pointing, and speech: The impact of co-speech gestures on linguistic deixis theory. Gesture 2 (2). 207-226. https://doi.org/10.1075/gest.2.2.05fri
  20. Fricke, Ellen. 2013. Towards a unified grammar of gesture and speech: A multimodal approach. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva Ladewig, David McNeill & Sedinha Tessendorf (eds.), Body - language - communication, Volume 1, 733-754. Berlin: De Gruyter Mouton. https://doi.org/10.1515/9783110261318.733
  21. Gerwing, Jennifer & Janet Bavelas. 2004. Linguistic influences on gesture’s form. Gesture 4 (2). 157-195. https://doi.org/10.1075/gest.4.2.04ger
  22. Gile Daniel. 2009. Basic Concepts and Models for Interpreter and Translator Training. Amsterdam, Philadelphia: John Benjamins. https://doi.org/10.1075/btl.8
  23. Goldin-Meadow, Susan & Martha W. Alibali. 2013. Gesture’s role in speaking, learning, and creating language. Annual Review of Psychology 64. 257-283. https://doi.org/10.1146/annurev-psych-113011-143802
  24. Gósy Mária. 2007. Disfluencies and self-monitoring. Govor = Speech 24. 91-110.
  25. Grishina, Elena A. 2015. O russkom zhestikulyatsionnom otritsanii (On gestural negation in the Russian language). Trudy Instituta Russkogo Yazyka im. V. V. Vinogradova 3. 556-604.
  26. Harrison, Simon. 2014. The organisation of kinesic ensembles associated with negation. Gesture 14 (2). 117-41. https://doi.org/10.1075/gest.14.2.01har
  27. Hilliard, Caitlin & Susan W. Cook. 2016. Bridging gaps in common ground: Speakers design their gestures for their listeners. Journal of Experimental Psychology: Learning, Memory, and Cognition 42. 91-103. https://doi.org/10.1037/xlm0000154
  28. Hostetter, Autumn B. 2011. When do gestures communicate? A meta-analysis. Psychological Bulletin 137. 297-315. https://doi.org/10.1037/a0022128
  29. Iverson, Jana M. & Susan Goldin-Meadow. 1998. Why people gesture when they speak. Nature 396 (6708). 228. https://doi.org/10.1038/24300
  30. Jakobson, Roman. 1956. Two aspects of language and two types of aphasic disturbances. In Linda Waugh & Monique Monville-Burston (eds.), On language, 115-133. https://doi.org/10.1515/9783110889598-005
  31. Jakobson, Roman. 1987. On the relation between auditory and visual signs. In Krystyna Pomorska & Stephen Rudy (eds.), Language in literature, 467-473. Cambridge, MA: Belknap Press.
  32. Kendon, Adam. 2004. Gesture: Visible Action as Utterance. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511807572
  33. Kok, Kasper, Kirsten Bergmann, Alan Cienki & Stefan Kopp. 2015. Mapping out the multifunctionality of speakers’ gestures. Gesture 15. 37-59. https://doi.org/10.1075/ gest.15.1.02kok
  34. Krauss, Robert M., Yihsiu Chen & Rebecca F. Gottesman. 2000. Lexical gestures and lexical access: A process model. In David McNeill (ed.), Language and gesture, 261-283. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511620850.017
  35. Lakoff, George. 1993. The contemporary theory of metaphor. In Andrew Ortony (ed.), Metaphor and thought (2nd ed.), 202-251. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139173865.013
  36. Lapaire, Jean-Remi P. 2016. From ontological metaphor to semiotic make-believe: Giving shape and substance to fictive objects of conception with the ‘globe gesture’. Santa Cruz do Sul 41 (70). 29-44. https://doi.org/10.17058/signo.v41i70.6413
  37. Lucero, Ché, Holly Zaharchuk & Daniel Casasanto. 2014. Beat gestures facilitate speech production. Proceedings of the Annual Meeting of the Cognitive Science Society 36. 898-903.
  38. McNeill, David & Susan D. Duncan. 2000. Growth points in thinking for speaking. In David McNeill (ed.), Language and gesture, 141-161. Cambridge: Cambridge University Press. https://doi.org/10.1017/cbo9780511620850.010
  39. McNeill, David, Justine Cassell & Karl-Erik McCullough. 1994. Communicative Effects of Speech-Mismatched Gestures. Research on Language and Social Interaction 27. 223-237. https://doi.org/10.1207/s15327973rlsi2703_4
  40. McNeill, David. 1992. Hand and Mind: What Gestures Reveal about Thought. Chicago, IL: Chicago University Press. https://doi.org/10.2307/1576015
  41. McNeill, David. 2005. Gesture and Thought. Chicago, IL: University of Chicago Press. https://doi.org/10.7208/chicago/9780226514642.001.0001
  42. Melinger, Alissa & Willem J. M. Levelt. 2004. Gesture and the communicative intention of the speaker. Gesture 4. 119-141. https://doi.org/10.1075/gest.4.2.02mel
  43. Mittelberg, Irene & Linda R. Waugh. 2009. Metonymy first, metaphor second: A cognitivesemiotic approach to multimodal figures of thought in co-speech gesture. In Charles J. Forceville & Eduardo Urios-Aparisi (eds.), Multimodal metaphor, 329-358. Berlin: De Gruyter Mouton. https://doi.org/10.1515/9783110215366.5.329
  44. Mittelberg, Irene. 2006. Metaphor and metonymy in language and gesture: Discourse evidence for multimodal models of grammar. Ph.D. Diss. Cornell University.
  45. Morsella, Ezequiel & Robert M. Krauss. 2004. The role of gestures in spatial working memory and speech. The American Journal of Psychology 117 (3). 411-424. https://doi.org/10.2307/4149008
  46. Müller, Cornelia. 2004. Forms and uses of the Palm Up Open Hand: A case of a gesture family? In Cornelia Müller & Roland Posner (eds.), The semantics and pragmatics of everyday gestures, 233-256. Berlin: Weidler.
  47. Müller, Cornelia. 2008. Metaphors Dead and Alive, Sleeping and Waking. A dynamic view. Chicago: Chicago University Press. https://doi.org/10.7208/chicago/ 9780226548265.001.0001
  48. Müller, Cornelia. 2014. Gestural modes of representation as techniques of depiction. In Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill & Jana Bressem (eds.), Body - language - communication, 1687-1702. Berlin: De Gruyter Mouton.
  49. Poyatos, Fernando. 1987/2002. Nonverbal communication in simultaneous and consecutive interpretation: A theoretical model and new perspectives. In Franz Pöchhacker & Miriam Shlesinger (eds.), The interpreting studies reader, 235-246. London: Routledge.
  50. Seeber Kilian G. 2011. Cognitive load in simultaneous interpreting: Existing theories - New models. Interpreting 13 (2). 176-204. https://doi.org/10.1075/intp.13.2.02see
  51. Seeber, Kilian G. 2012. Multimodal input in simultaneous interpreting: An eye-tracking experiment. In Lew N. Zybatow, Alena Petrova & Michael Ustaszewski (eds.), Proceeding of the 1st international conference TRANSLATA, translation and interpreting research: Yesterday-today-tomorrow, 341-347. Frankfurt am Main: Peter Lang.
  52. Seeber, Kilian G. 2017. Multimodal processing in simultaneous interpreting. In John W. Schwieter & Aline Ferreira (eds.), The handbook of translation and cognition, 461-475. Hoboken, NJ: John Wiley & Sons. https://doi.org/10.1002/9781119241485
  53. Streeck, Jürgen. 2009a. Forward-gesturing. Discourse Processes 46 (2-3). 161-179. https://doi.org/10.1080/01638530902728793
  54. Streeck, Jürgen. 2009b. Gesturecraft: The Manu-facture of Meaning. Amsterdam: John Benjamins. https://doi.org/10.1075/gs.2
  55. Tong, Yao. 2023. Embodiment of concrete and abstract concepts: The role of gesture [Unpublished doctoral dissertation]. Vrije Universiteit Amsterdam.
  56. Vilà-Giménez, Ingrid & Pilar Prieto. 2020. Encouraging kids to beat: Children’s beat gesture production boosts their narrative performance. Developmental Science 23. e12967. https://doi.org/10.1111/desc.12967
  57. Wei, Carolyn Y. 2006. Not crazy, just talking on the phone: Gestures and mobile phone conversations. In Proceedings of the 2006 IEEE International Professional Communication Conference, Saratoga. 299-307. https://doi.org/10.1109/IPCC.2006.320363

Copyright (c) 2023 Leonteva A.V., Cienki A., Agafonova O.V.

Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

This website uses cookies

You consent to our cookies if you continue to use our website.

About Cookies