Efficiency of Computer Vision in Modulating Happiness: An Interventional Study

1. INTRODUCTION

Happiness is a matter of serious concern. The irony of this statement becomes evident when we consider the daily efforts we put into making ourselves and others happy. Mental health professionals such as psychologists or psychiatrists would assure us that happiness can determine the very chances of survival in some, especially those with suicidal ideation [1, 2]. United Nations General Assembly updates the World Happiness Report every year, and the latest report says that suicide accounts for more than 1 in every 100 deaths [3]. According to the World Health Organization (WHO), among the 970 million people living with mental disorders, 28.9% still suffer from depression [4].

Facial Feedback Hypothesis (FFH): While international agencies aim to implement community-level measures to promote mental health, individuals can also engage in physical activities, such as cardiorespiratory fitness, to improve mental health. This study, however, focuses on a physical activity that involves voluntary contraction of various facial muscles, thereby forcefully causing a smile on the subject’s face. It is well known that facial expressions convey the emotions we feel. There are numerous studies, though, that have shown that facial expressions are not just a result of emotions, but can modulate emotions through feedback mechanisms [5]. The facial feedback hypothesis (FFH) states that, under suitable conditions, altering one’s facial expressions can change the emotions one feels. Then, some studies couldn’t demonstrate the above-mentioned phenomenon. Thus, a debate has been ongoing for almost half a century regarding the existence of such a phenomenon and, if it does exist, precisely under what circumstances.

One important reason for these inconsistencies could be the lack of specificity in the required facial muscle manipulation. In most studies involving FFH, the facial manipulations are performed according to the investigator's instructions. The decision regarding which muscle to contract depends on the belief that a certain muscle acts the most while expressing a certain emotion. Even though there are numerous muscles in our face that can impart an expression, a set of two or three muscles is usually focused upon. Sometimes it is even left to the experimenter to improvise during the experiment, deciding which muscle to contract and how much to contract it. Traditionally, participants are instructed to hold a pen sideways between their teeth, without it touching their lips [6] (Fig. 1).

This results in a smile and is purported to generate happiness. Without a standardised set of instructions, participants might generate unsatisfactory expressions of emotions. Additionally, unintended involvement of other muscles can result in a mixed emotional expression [7]. There is a need for an easily employable, standardised method for facial muscle contraction. This study thus aimed to harness the potential of computer vision, in which the participants’ facial expressions were analysed using a camera. According to this study’s hypothesis, the change in happiness induced by computer-vision-validated facial expressions will be greater than that induced by the traditional method.

Computer Vision (CV) in emotion recognition: Machine Learning, a type of artificial intelligence (AI), is at the core of modern emotion recognition services. In Machine Learning, a large amount of data, in this case, images of people displaying an emotion, are fed into the system, from which the system learns. This type of AI keeps evolving as more data are supplied to it, and it has been at the core of Computer Vision, enabling computers to understand digital images and videos. Software-related companies, including Google and Microsoft, have utilized this technology to recognise emotions expressed by humans [8]. CV relies on tracking specific facial points to determine the type of emotion is being expressed.

Detection of changes in emotion among participants: Most commonly, emotions elicited by facial muscle manipulation have been detected by asking participants to report them on a scale [9]. There are other variables, such as reaction time to a stimulus, that have been used as replacements for emotional status assessment [10]. Often, reaction time is chosen because it is not completely under participants' control, yet it can still quantify changes in emotional state and is easily measurable. Here, participants were asked to categorize the emotion being expressed in a displayed image, and the resulting reaction time was recorded. This decision is based on the congruency effect, which states that a person experiencing a particular emotion will be more efficient at recognising faces which display the same emotion [11].

The underlying mechanism can be explained by the theory of spreading activation [12]. According to this theory, when an individual experiences an emotion, a representation of that emotion in memory is activated, and the activation spreads to related information. This leads to more efficient processing of emotion-congruent material, due to better perception and retrieval. It was also demonstrated that, compared with controls, participants in a positive emotional state tended to rate low prototypic exemplars of the category as members of the category. Additionally, experimenters found a positive correlation between participants’ emotional intensity and the efficiency with which they processed congruent emotion-laden material [13]. Given the possibility that pre-existing emotions could influence reaction times, rather than comparing only the mean reaction times across the two types of facial muscle manipulations, it was decided to focus on changes in reaction times before and after each manipulation. This comparison will depict the capacity of each method to induce happiness, even in those with a pre-existing emotion.

1.2. Role of Mimicry in Emotion Recognition

The capacity of humans to understand others' emotional states by observing their facial expressions appears crucial from an evolutionary perspective [10]. This capacity enabled individuals to avoid the consequences of interacting with someone in a bad mood and to facilitate interactions with those in a favourable state of mind. This detection of emotional status based on facial expressions is augmented by mimicry [18]. Studies have demonstrated that upon viewing a facial expression, the observer automatically mimics it, a process that occurs spontaneously and mostly without the observer’s awareness [19]. Even in instances where the mimicry is not obvious to the human eye, the electromyographic recordings have shown that the facial muscle contraction is associated with a better understanding of others’ displayed emotions. Even a minor stretch of the facial skin stimulates the low-threshold proprioceptive receptors embedded in it. This sends feedback signals to the spinal cord, which then passes them to the higher centres of the brain. This facilitates the simulation of the observed emotion [7]. These emotions nonetheless are considered to be foreign and need not be of the same intensity. The observer can subsequently empathize or remain indifferent, depending on the situation [20].

Additionally, studies have shown that when facial mimicry was prevented, emotion recognition, especially of subtle emotions, was less efficient than when mimicry was allowed [18]. It should be noted here that, even though mimicry is considered to be important in the detection of an emotion, it is not vital for the process. Patients who had paralysis of facial muscles were found to have difficulty in emotion recognition, but were not completely devoid of the capability. This provides strong evidence that mimicry augments emotion detection [7]. It is important to note here that while spontaneous mimicry of emotions resembles bottom-up processing, the facial feedback mechanism resembles top-down processing (Fig. 2).

Spontaneous mimicry, as useful as it is, can also confound. It can result in efficient processing of the incongruent emotional material. To illustrate, if a subject is made to display a happy facial expression, it is expected that images of incongruent emotions, such as sadness and anger, will be processed slowly. But facial mimicry can lead to comparatively better processing of these incongruent emotional materials. In this study, a control phase was planned for all participants, during which subjects were instructed not to express any emotion. As this prevents mimicry, bottom-up processing will not be activated. Specifically, as the subjects did not smile, top-down processing was curtailed in this phase as well.

This study was conducted to compare changes in the proportion of correct responses when recognising happiness, and in reaction times for the same emotion recognition between CV-aided and non-CV-aided (traditional) facial muscle manipulation. This comparison is intended to demonstrate that a precise scoring system based on Computer Vision (CV) can induce higher levels of happiness.

2. MATERIALS AND METHODS

Sample Size: Calculated using a power analysis based on an effect size of 0.66, derived from a similar study [21]. With a power of 90% and a 20% non-response rate, each group's sample size was 72. Thus, a total of 144 participants were considered appropriate for this study.

2.4. Procedure

Participants appearing for the study were assigned to either one of the two groups: CV-assisted or non-CV-assisted, alternately. They were informed that they would have to produce certain shapes with their lips, such as a smile or a dash (−), and that they would have to compulsorily produce the whistling sound (O). For both groups’ control phase, participants were instructed to produce the shape of ‘O’ with their lips, and this required them to purse their lips as in whistling (Fig. 3). This would prevent spontaneous mimicry [18]. Additionally, the 'O' shape of the lips is not consistently associated with the expression of any emotion. Facial Action Coding System is a widely followed framework to classify emotions based on facial movements, and according to it, the ‘O’ shape of the lips does not signify any major emotion [23]. Together, these would ensure that neither top-down nor bottom-up processing is facilitated during the control phase.

For the intervention phase of the CV-assisted group, they had to smile. For the intervention phase of the traditional group, they had to bite on a disposable tongue depressor or a pen sideways, without touching their lips to it, and this would enable them to achieve a dash shape with their lips (Fig. 1). If the shape produced was satisfactory, participants had to continue using it while labeling emotions displayed on a computer screen. Participants were assured of reasonable privacy while producing the shapes mentioned above.

For the emotion recognition task, images belonging to different races (Black and White), of two biological sexes (Male and Female), expressing different intensities (5 grades) of the main emotions under focus (Happiness and Sadness), and images of neutral expression of two sexes and races were shown once. Thus, a total of 40 + 4 = 44 images were required to be labelled for each phase (control and intervention). Grades of expressions range from one to five, with five being the one with the maximum intensity. Sufficient time was provided to the participants to practice labelling the facial expressions and to clear their queries.

Participants were seated approximately 0.80 metres away from a 19.5-inch LCD computer monitor (refresh rate: 60 Hz, display resolution: 1440 x 900). All the images displayed on the screen were resized such that a visual angle of about 13 degrees vertically and 10 degrees horizontally was subtended. The task of displaying these images and measuring the time participants took to detect the emotions in them was carried out using Windows desktop software named Open Sesame (Version 3.3) [24].

Before each image, a fixation dot was displayed, and participants could start only when they felt comfortable. This would display an image for emotion labelling on the screen, and participants had to choose a response as early as possible. If no response was given for 2 seconds, then it was considered an error. There were three keys (A, B, and L) required for identify three different expressions portrayed. The pairing of keys and emotions was not kept constant, though. A total of six permutations were created, and these were counterbalanced across participants. They were instructed to signal the experimenter at the earliest about any distraction or unwillingness to continue, so that the appropriate measure could be taken.

During the intervention phase, participants' facial expressions were continuously monitored using Microsoft Cognitive Services (Version 3.0.0 - 16.11.106.23128) running in Visual Studio on another Windows desktop computer. The application program interface (API) was called once every 3 seconds, so participants' facial expressions were corrected almost as soon as they deviated from the given guidelines. Participants in the CV-assisted group had to smile constantly while performing the tasks during the intervention phase. The threshold for a definitive expression of happiness was set at 80%, and if the Cognitive Service scores fell below this level, a reminder to smile was issued. Even though only one of the two groups was assisted by computer vision, facial expression monitoring was performed by it in both groups (Fig. 4).

3. RESULTS

A total of 12144 reaction time readings were available for statistical analysis (138 participants x 88 timed performances). The participants’ ages ranged from 18 to 35 years (M = 21.65, S.D. = 1.63). The proportion of female participants was 59%, and the proportion of male participants was 41%. Regarding educational background, medical students accounted for 57% of the participants, and engineering students accounted for 43%. Only four participants were predominantly left-handed. None of the participants reported inconvenience while following instructions or requested to stop the trials midway. The CV-assisted and non-CV-assisted (traditional) groups were age-matched (p = 0.933) and sex-matched (p = 0.58).

Overall, during the intervention phase, participants in both the CV-assisted and traditional groups demonstrated an increase in the proportion of correct responses when labelling happy faces. McNemar tests performed on the paired data showed that the difference was statistically significant (Tables 1 and 2). Both CV-assisted and traditional groups continued to show statistically significant improvements (p < 0.05) when male participants were considered. However, among females, only the traditional group showed a significant improvement (p < 0.001). Similarly, CV-assisted and traditional groups showed statistically significant improvements in correct responses when only medical students were considered (p < 0.05), but with engineering students, only the traditional group showed improvement (p = 0.006). Within these groups, when further analyses were performed, the traditional group (see Supplementary Table S1) consistently showed more significant performance improvements than the CV-assisted group (see Supplementary Table S2). These analyses were performed across subgroups defined by the different model parameters shown in the photos, such as sex, race, and levels of happiness expression. Of these, the sex and race parameters are displayed together in the results as Female-Black (FB), Female-White (FW), Male-Black (MB), and Male-White (MW).

Before the intervention, the mean reaction time to identify a photo in the CV-assisted group was 859.39 × 10^-3 seconds (ms), with a standard deviation of 193.75 ms, and in the traditional group it was 857.48 m,s with a standard deviation of 159.62 ms. Table 3 presents the change scores for correct responses post-intervention for both groups. The change in reaction time compared to the control group was analyzed using paired t-tests. A significant change in performance post-intervention was seen in both the CV-assisted [t(69) = 5.36, p < 0.001] and the traditional group [t(67) = 4.77, p < 0.001] (Table 4 and Table 5). These patterns were consistent even when males and females, and medical and engineering students, were analyzed separately for the CV-assisted (see Supplementary Table S3) and traditional groups (see Supplementary Table S4).

Overall, when the proportion of correct responses in the control phase was compared directly between the CV-assisted and traditional groups, no significant difference was found. The lack of significance was also observed for reaction time of correct responses, even when the analysis was done separately for females, males, medical students, and engineering students. Thus, the baseline performances of the two groups were comparable.

Regarding post-intervention performance, the traditional group showed a higher proportion of correct responses than the CV-assisted group. Chi Square/Fisher Exact tests performed on the data showed that the difference was significant (p = 0.014) when the overall set was considered (Table 6). The same was true when females’ responses (p = 0.005) and medical students’ responses (0.008) were considered separately (see Supplementary Table S5).

Further, independent t-tests were applied to reaction times for correct responses. Overall, no significant difference was observed between the CV-assisted and the traditional groups (Table 7). During subgroup analysis, in FW models with grade 5 expression when recognized by medical students, the traditional group performed significantly better, and in FW models with grade 2 expression when recognized by engineering students, the CV-assisted group demonstrated significantly better performance. The rest of them showed non-significant differences (see Supplementary Table S6). Of the two instances that showed significance, there was no consistency in the superiority of either the CV-assisted or the traditional groups. If all the significance tests calculated are considered, it was evident that the greatest advantage of interventions (CV or traditional) occurred when participants viewed photos of white females with a smile of grade 2 intensity.

Next, the effect of the models' gender on emotion labelling was analyzed. It was found that male participants were more likely to label female-neutral expression photos as happy than male-neutral expression photos (p = 0.04). Additionally, all participants, except the engineering students, showed a stronger tendency to label females’ minimal sad photos as happy, but the difference in males’ photos was not statistically significant. Conversely, when the male models exhibited minimal happiness (grade 1), all participants tended to label them as neutral. Compared to the females’ photos, this tendency was significantly higher when the photos were analyzed overall (p < 0.001) and in all the other categories of participants too (Females: p = 0.014, Males: p < 0.001, Medical students: p < 0.001, and Engineering students: p < 0.009). Further, when the level of happiness expression among male models exceeded 1, this behavior was not observed in practice. Thus, mislabelling usually occurred when the models’ (of either sex) expressions were minimal or neutral. However, there was no significant difference in the proportion of correct responses in happiness recognition between participants by sex, even when viewing minimal expressions. Similarly, with respect to the time required for recognize happiness, no consistent pattern was observed based on either the model's sex in the photos or the participants' sex.

Regarding performance on photos with sad expressions, the time to recognize them was considerably longer than for happy expressions. Statistically, the difference was significant overall (p < 0.001) and within the subcategories. It was significant in both the control and intervention phases of both the CV-assisted and the traditional groups. Between the traditional and CV-assisted groups, participants in the latter group took longer to recognize sadness, though the difference was not statistically significant. Compared to the control phase, the time required to recognize sadness post-intervention had uniformly reduced in both the CV-assisted and traditional groups. The difference was even statistically significant overall (CV-assisted group: p = 0.047, traditional group: p = 0.005) and in the sub-groups of males (CV-assisted group: p = 0.03, traditional group: p = 0.015) and engineering students (CV-assisted group: p = 0.003, traditional group: p = 0.023).

Participants’ expressions were analyzed using CV, regardless of group membership. It provided a score directly related to how genuinely the CV expressed happiness. It was found out that the mean scores were significantly higher for the CV-assisted group than the traditional group (p < 0.001). When the mean scores were compared between the control and intervention phases, the latter showed significantly higher scores (p < 0.001) in both the CV-assisted and traditional groups. However, no positive correlation was found between CV scores and participants’ speed of emotion recognition. Pearson correlation coefficient for the CV-assisted group was –0.1, and for the traditional group, it was -0.2. On the other hand, there was no negative correlation found between reaction times and error rates either; r = 0.4 for both groups. Finally, after completing their trials, participants were asked whether they could guess the study's actual purpose; none answered correctly.

4. DISCUSSION

This study aimed to determine whether computer vision is a more effective tool than traditional methods for increasing happiness among research participants. Among the participants, there was a healthy balance of both sexes and students from two different academic backgrounds: medical and engineering. Even though they were all chosen from two educational institutions, they were not associated with any known confounding factors that would twist the results when compared to the general population. It has been shown that decreased sensitivity to bodily feedback with age hampers the capacity to recognise others' and one's own emotions [25]. With emotion recognition being the task of primary importance in this study, the participants chosen can be considered as young adults, with their age ranging from 18 to 35 years.

The CV scores of both the CV-assisted and traditional groups showed almost no correlation with performance in emotion recognition speed. Nonetheless, participants in the traditional group, who were instructed to bite a pen, labelled emotions consistently faster than during the control phase. It has been shown that holding a pen in the mouth increases an individual’s happiness and helps them better recognize it in others [6]. Post-intervention, the traditional group had more correct responses when labelling happy faces than the CV-assisted group. Recognition of happy faces is found to be directly proportional to the happiness generated in some studies [13]. Here, it means that the traditional method of biting on a pen might have induced a higher level of happiness than the CV assistance, and this inference is of primary importance to this study.

Even though some recent studies [5] have not replicated the effect of the facial feedback hypothesis, subsequent studies have [26]. Coles et al. [27] conducted a multi-lab study with approximately 4000 participants spanning 19 countries, which showed the effect of the facial feedback mechanism when using mimicry and a voluntary facial action task. They demonstrated the effect even with the pen-in-mouth technique, but the evidence was less conclusive. A recent study examined the effect of pen-in-mouth manipulation on perception of emotional faces and memory [28]. The manipulation did not affect memory, but the perception of happy faces improved consistently.

As described above, control of one emotion is possible through the expression of another. Conversely, Rychlowska et al. [18] proved that even without expressing a new emotion, the restriction of facial muscle movements can reduce the speed at which emotions are labelled. In the present study, the control phase, during which the restriction was imposed, showed significantly slower emotion recognition and fewer correct responses than the post-intervention phase in both the CV-assisted and traditional groups. Thus, the control phase significantly impaired participants' emotion recognition.

In the present study, happiness was recognized faster than sadness. This is in line with the existing literature, which establishes that individuals tend to recognize emotions that are congruent with their current emotional state of mind. However, in the control phase, when no emotions were expressed, recognition of happiness remained significantly faster. This finding is consistent with the happy categorization advantage [29], which states that happy emotions are recognized faster than other emotions. One reason for this phenomenon is that a happy facial expression shows fewer overlapping changes with other expressions [30]. Instead of a complete analysis of the face, just a salient feature is enough for its identification [31]. Also, people are positively biased about reality and are even in a positive mood most of the time. Thus, expectations and mood play a role in this advantage [29]. Additionally, in the presence of emotions of negative valence, there is a biased reallocation of resources away from higher-order cognitive processing [32]. These can explain the significant differences in reaction times for sadness and happiness.

Regarding the sex of faces, participants have been shown to find female faces happier than male faces [10]. The reason can be traced back to the fact that boys and men are found to be involved in more certain types of aggressive behaviour than girls and women. This itself might be shaped partly by society wherein the aggression by a male is tolerated more [33]. In the present study, neutral expressions of female faces were mislabelled as happy, especially by male participants. Most participants labelled even the minimal sad faces of females as happy and, conversely, labelled the minimal happy faces of males as neutral. Regarding the time it takes participants to recognize emotions, it has been repeatedly shown that males are slower than females [34]. Along with the biological differences between the two sexes, the social upbringing of females can also be ascribed to this, wherein they are encouraged to perceive and express emotions more than males. However, in this study no significant difference in either the speed of recognition or the proportion of correct responses was found based on the sex of the participants.

It is widely believed and quite often proven that as speed increases while performing tasks, more mistakes happen; this is referred to as the ‘speed accuracy trade-off’. In this study, there was no negative correlation between emotion labelling accuracy and reaction time. It is suggested that the pressure under which an individual performs a task makes them more prone to mistakes, not just the speed of performance. Participants would have been under pressure if they had reference values for the expected speed of performance, based on previous experience [35]. The participants in this study might have acted comfortably, without any pressure, as there were no such reference values, nor had they engaged in a similar task previously. This might be the reason why the speed-accuracy trade-off did not exert its influence in this study.

5. STUDY LIMITATIONS

Compared with the control phase, emotion recognition in the intervention phase was significantly improved for both happiness and sadness. As an attempt was made to generate only happiness during the intervention, the congruent emotion (happiness)- related photos were supposed to gain an advantage over other emotions. This may be explained by the fact that the participants had already completed the control phase and were more comfortable with the given task during the intervention phase. Even though this problem was foreseen, it could not be prevented as the intervention phase could not have come before the control phase. The fixed within-subject order and the lack of counterbalancing can be viewed as threats to internal validity.

It is well known that the timed behavioural processes, such as emotion recognition, can be affected by attention and motivation [37]. Even though the participants appeared motivated by the possibility of winning a cash prize, the same cannot be assumed of their attention. This is more valid in the case of the CV-assisted group, as participants were given repeated prompts based on computer vision scores to encourage and maintain a smile. This process, though it undoubtedly improved the CVs provided for participants’ genuineness in happiness expression, might also have inadvertently caused distraction, thereby hampering participants' performance in the CV-assisted group. This might be one of the reasons why the traditional group performed with a higher proportion of correct responses.

CONCLUSION

If one is not feeling happy, but still, if that person can generate happiness from within by giving appropriate expressions, it would be beneficial, especially if that person has an illness such as clinical depression, just as in laughter therapy, where the ‘fake it until you make it’ principle [36] is followed by intentionally laughing out loud; in this study, too, forceful smiling was the action under focus. It was found that the computer vision-guided approach to smiling was inferior at generating happiness compared to the traditional method of biting a pen. Suppose the simple traditional method is the better of the two; still, we should not forget what this study was meant for: the need for uniform, practical guidelines for generating a smile as genuine as possible. If this requirement is met, then the simpler method of displaying a smile can be useful in generating happiness among some of those in need.

REFERENCES