Combination of focus group and experiment methods: opportunities and limitations

Full Text

One of the ways to obtain representative data is the combination of several sociological techniques. Today, the combination of such techniques as focus group and experiment allows researchers to expand their capabilities and understand complex social phenomena in more depth. On the one hand, the use of focus group provides an opportunity to obtain richer qualitative data and to consider different opinions, views and experiences of participants. Focus group creates an atmosphere of group discussion, and interaction of participants can lead to new ideas and a deeper understanding of the issues discussed [2]. However, E. Colucci argues that it is not always possible to properly maintain group dynamics, and often focus group turns into an individual interview conducted in a group setting [3]; thereby, it is necessary to look for non-traditional methods to supplement the research based on focus groups to obtain more qualitative data. For this reason, in addition to the classic focus group, we will consider the experimental approach which allows to identify causal relationships and ensure controlled conditions to examine the impact of certain factors on the variables under study [14]. Experiments can provide more objective and verifiable data that can be used to formulate statements about causal relationships [1].

The focus group method is based on the idea that “collective reactions to a certain stimulus, provoked and revealed in the course of group discussion, are not random, situational reactions of a given meeting, the configurations of which cannot be reproduced in the future, but are rather manifestations of mass consciousness” [10]. This method allows to better understand and interpret collective opinions and the dynamics of group interactions on a relatively small group, which is an important advantage when conducting experiments with a rather small number of participants. Small groups of 10–15 people are used for sociological experiments, which roughly corresponds to the format of the focusgroup research [6]. I.F. Devyatko defines experiment as “an experimental study of the impact of a single factor (or several factors) on the variable under study” [4]. For instance, laboratory (or true) experiment provides the maximum level of control over the independent variable as isolated from external influences. Such control makes it possible to reject competing hypotheses of the observational effect under study [7]. Thus, we will emphasize three advantages of this type of experiment [4]: complete control over the independent variable; isolation from other variables; possibilities of multiple repetition with the same external and internal factors. Moreover, in the laboratory experiment, as in focus groups, the group is formed artificially, which is why participants in most cases are aware of their participation in the study. In this situation, the advantage is also the fact that the experiment can be considered a standard focus group, which increases the sincerity of answers and behavior of its participants [3]. In turn, the presence of a moderator and an assistant provides full control over the experiment. However, “most experiments in social sciences take place in conditions in which the above-mentioned principles cannot be fully provided” [4]. Therefore, advantages of this method can be also achieved in the framework of the focus group method, which also proves the efficiency of their combination. Focus groups also open opportunities for a full-fledged digital recording of experiment from its beginning to the end: first, we can refer to the research data at any time; second, we get an empirical database [10].

Considering the combination of focus group and experiment, we should refer to the works of R. Merton and his co-authors, who directly used this combination [11] “to obtain as complete a report as possible of what a particular situation was like”, focusing on several conceptual tasks that are experimentally feasible: first, labeling an effective stimulus that influences the observed responses, namely “which X or which parts of X in the stimulus situation produced the observed effect”; second, interpreting differences between expected and actual effects, which may contradict theories based on previous research or logical reasoning; third, explaining factors that determine differences in responses between subgroups that are more prevalent in the broader population; fourth, interpreting the occurrence of actual and experimentally generated impacts.

Although Merton outlined a wide range of combination advantages (experiment and focus group), there are practically no applied studies using this methodology. N.V. Vakurova suggests to use a combination of experiment and focus group but in turn [16]: at first, an experiment is conducted, then a focus group is used to identify the social-psychological preconditions for the results of the experiment, i.e., focus group is a complementary technique, not a central one. A similar approach is described in the article by Dutch researchers who studied the influence of the Embodied Conversational Agents (ECA) (chatbots, holograms, robots, etc.) on the evaluation of personal characteristics and on the intentions to use such services [14]: a questionnaire survey was followed by a focus group.

When it comes to combining focus group and experiment, we should admit that these techniques are rarely used simultaneously, so we will consider a few examples in more detail. Thus, one study aimed at “tracing the patterns and specific features of respondents’ behavior under the influence of alcohol: how the type of reasoning changes, how they interact” [15]. The experiment consisted in comparing the answers of participants to the same social-political questions rephrased each time as they were discussed, in moments of different degrees of intoxication. The non-standardized focus group was conducted in a restaurant that created a relaxed, home-like atmosphere, and alcohol as a stimulus for communication increased the sincerity of the participants’ responses, which would have been much more difficult to achieve in a standardized focus group. In a similarly structured study [12], the authors conducted a methodological experiment using focus groups to influence the participants’ attitudes towards adultery. The control variables were psychotype and emotional intelligence, and the dependent variable was suggestibility. The groups were homogeneous in composition and randomized according to emotional intelligence and psychotypes: group 1 consisted of girls with predominantly low emotional intelligence and group 2 — with predominantly high emotional intelligence, one with a pronounced accentuation of each type in each group. The participants’ attitudes were influenced with stimulus material (projective and associative questions, video clips) and with a ‘decoy duck’– a participant who demonstrated a positive attitude towards adultery. After focus groups, the social attitude was measured again in pairs for each psychotype.

Further, we will examine the procedure of the study based on the combination of an experimental approach and focus groups, and its key results [9]. We conducted two focus groups (8 participants in each); groups were randomized by two characteristics — emotional intelligence and psychotype (In both groups, participants with each of the existing radicals were present). The first group was dominated by students with low emotional intelligence, while the second group was dominated by students with high one; the ability to identify fake news in the media was also an important selection criterion. Thus, the control variables were psychotype and emotional intelligence, and the dependent variable was suggestibility.

The first stage consisted of selecting participants by assessing the level of emotional intelligence with the Schutte test, identifying the psychotype based on the questionnaire for accentuated radicals by B.V. Ovchinnikov and I.V. Tyuryapina, and estimating with an online questionnaire the subjective perception of the ability to identify fake news. Participants who were convinced of being able to identify fakes were invited to participate in a focus group. All the above-mentioned steps were taken with one questionnaire.

The main stage of the study was an experiment conducted as a focus group, in which attempts were made to influence the respondents’ attitude to stimulus material and two ‘donkey ducks’. This stage consisted of:

1. Warm-up — respondents were asked a few general questions (e.g., “Do you follow the news, and what news are interesting to you?”) to set the participants up for a conversation on the research issues.
2. Testing with stimulus material — the focus group participants were shown four fake news topics (“A cannibal bear attacked two men in Komi”, “Resolution of the Federal Agency for Tourism on the mandatory assignment of QR codes to tourists traveling in Russia”, “Happy Birthday, Vladimir Vladimirovich!: How Putin’s birthday was celebrated all over the world”, “VGTRK Media Holding and Channel 1 will retouch the faces of ‘foreign agents’”). The moderator and two ‘donkey ducks’ were to convince everyone that all the news were true. At first, the news was presented as text, then, after discussion, the participants were shown the same news as a screenshot from a news website and/or a video fabricated in advance. The organizers included a few mistakes and inaccuracies in these visual forms to make the respondents question the credibility of the news and to further identify the key points the participants pay attention to in the media information. One of the key roles at this stage was played by the ‘donkey ducks’ whose task was to convince the participants of the credibility of the news discussed.
3. Identifying the fake news criteria — the participants were asked to describe their ways for distinguishing true news from fabricated ones.

At the next stage, the participants’ social attitudes towards their ability to identify fake news were measures: at the very end of the focus group, participants were given questionnaires to mark the news which, in their opinion, were fake or true. The last question was again about whether they considered themselves competent in recognizing fake news (the same question was asked at the selection stage).

The analysis of the conducted experiment with the focus group method allowed to make the following conclusions which seem to be the result of the combination of two methods:

• The news “Decree of the Federal Agency for Tourism on the mandatory assignment of QR codes to tourists traveling in Russia” was considered credible by both groups; moreover, one participant noted that he received a relevant newsletter after the ‘duck’ confirmed that this newsletter was really in her e-mail: . “I think I got one of those too” (schizoid radical, above average emotional intelligence [13]). Another participant noted that he had seen the news VGTRK Media Holding and Channel 1 will retouch the faces of ‘foreign agents’” in a satirical TV program — “I think I saw this news in Panorama” (hyperthymic radical, above average emotional intelligence), but this fact did not change the respondents’ opinion in any way; on the contrary, further questions from the moderator and the ‘ducks’ convinced a significant part of the group that this news was true. Thus, the ‘donkey duck’ technique significantly influences the course of the experiment and allows to obtain interesting data.
• For the first group, the issue of the true news category presented during the focus group was quite important. After admitting that all news were fake, the participants continued the discussion. According to the participants, the fact that the researchers presented the news determined their perception of the news as true. The group did not have access to the news on their usual platforms and devices, as gadgets were prohibited. “We see exactly what you are showing us, and we cannot get to the details. From the perspective that you have presented it to us, we may be more likely to believe it. I can’t check at this moment what you have shown us. If I came across it myself on the Internet, I would certainly look at least at the date when this news was published, at the number of other sources that also published it” (psychasthenic radical, emotional intellect below average).
• Participants were shown the barely noticeable mistakes in the news such as dates, links, speech inaccuracies and fake videos. Participants admitted that they tended to take the moderator’s word for it, since the stimulus material was not provided personally but was displayed on the TV screen. “Maybe if everyone had been given the same materials, we would have been more attentive. If we are shown a general picture, and even if you read it out, we are inclined to hear what you tell us, rather than to go and look for wrong and right” (psychasthenic radical, below average emotional intellect). However, it should be stressed again that the ‘ducks’ were actively involved in the conversation and influenced the respondents’ opinion. “In general, it seems that this is the Russian style — first to add to the list of foreign agents, and then this whole story seems to be a matter of time — that they may be cut it out of the broadcast or retouched. It was expected” (duck).
• Compared to the first group, participants with the above average emotional intelligence were less active in discussing the event after the focus group and asked almost no questions. However, they were surprised that the news was not true. They noted that the moderator’s nonverbal reactions also affected the participants’ opinions. “You read out the news so seriously… I believed that they were all real because there was no emotion on your face” (schizoid radical, emotional intellect above average); “We were deceived” (hysteric radical, emotional intellect above average). Thus, representatives of this group are rather empathic and focus mostly on emotions.

To sum up, we can conclude that the groups turned out to be similar on average: respondents agreed in their opinions and tended to make the same mistakes. For example, there was little attention paid to the technical points of the demonstrated news. However, at the end of the focus group, some participants mentioned that they were still suspicious, for instance, the respondent who said he knew how video was edited considered the news on ‘retouching foreign agents’ with skepticism, which he expressed in detail after the results were summarized and the focus group was over.

During the research we identified the following limitations of the techniques used: (1) focus groups are time-limited, so an experiment must be completed within a certain period; however, the time limit can facilitate a more focused discussion by encouraging participants to be brief and straightforward; (2) the group dynamics has a direct impact on the experiment; with the focus group method, not all studies can be successfully conducted, as conformity or cohesion may interfere and cause distortions in the results and participants’ reactions; (3) the presence of a moderator and videotaping in the focus group can influence the respondents’ behavior, so the benefits and risks of having a third-party during the experiment should be considered (respondents may become more cautious, censor their responses or simply adjust to the expectations of the moderator; some participants may become more conscious and pay more attention to their speech and appearance, which may affect the naturalness and spontaneity of their responses); (4) a wide range of psychotypes in the focus group complicates the moderator’s communicative task — he should be aware of each psychotypes as affecting behavioral patterns, accurately select the type and frequency of interaction with each, and to maintain the ‘right’ group dynamics; (5) there is a need in the complete control of the moderator’s facial and physical reactions, which is an extremely difficult task, since, according to P. Ekman, mimic reactions appear involuntarily and are virtually uncontrollable [5]. For this reason, the main task of the moderator is not only to read out the news with a neutral facial expression, but also to control his face and body during the discussion.

Thus, we can conclude that the combination of focus group and experiment was successful: the constructive collaboration of these two techniques allows to obtain a more complete and deeper understanding of the phenomenon under study. First, the focus group provides context and in-depth understanding of the opinions, attitudes and subjective experiences of the participants, helping researchers to obtain more valid data and to identify unexpected factors that may influence the results of the experiment. In its turn, the experiment allows to control and adapt the research conditions and influence respondents with non-standard focus group techniques, which gives an opportunity to test specific hypotheses and find causal relationships. The combination of experiment and focus group can improve the validity of research data: internal validity is ensured by the control of the experiment and the ability to assess causal relationships, while external validity can be achieved with additional examination of the context of social interactions and participants’ opinions. Moreover, the combination of experiment and focus group produced interesting data, for instance, representatives of the hysteroid and asthenic radicals seem to be less susceptible to suggestibility than other psychotypes. In other words, the combination of these methods allows to supplement the design of any study with social-psychological concepts, as the experimental method provides control over social-psychological variables, while focus groups bring the discussion situation closer to how everything works in everyday life.

To ensure such advantages of the two methods’ combination, we need to follow some recommendations: in recruiting, it is necessary to develop more detailed selection criteria for each respondent to exclude competent participants (for example, in the focus group on fake news, journalists were not allowed); seating of focus group participants should be based on their psychotypes and other personal characteristics (for example, a representative of the schizoid radical should be seated next to a person of the hyperthymic radical; when discussing sensitive issues, participants with more expressive traits and attitudes may become confrontational, so a calmer person should be placed between them); the ‘donkey duck’ technique is very influential in shaping the opinions of the focus group participants, therefore, their seating is more important (we believe that it is necessary to use two ‘ducks’ in the discussion, seating them on opposite sides but not opposite to each other, so that to ensure their eye contact with all participants); during focus groups, it is necessary to provide stimulus material in printed form to each participant, because demonstration on the screen limits people with visual impairment — they rely on the moderator’s speech, which is an obstacle for understanding the material).

About the authors

Zh. V. Puzanova

Author for correspondence.
Email: puzanova-zhv@rudn.ru
Miklukho-Maklaya St., 6, Moscow, 117198, Russia

T. I. Larina

Email: larina-ti@rudn.ru
Miklukho-Maklaya St., 6, Moscow, 117198, Russia

T. A. Ignatova

Email: eignatova0304@mail.ru
Miklukho-Maklaya St., 6, Moscow, 117198, Russia

References