Valence, arousal, and FE recognition scores
We first evaluated behavioral and subjective responses to the dynamic FE stimuli. There were 12 videos in TOTAL, 4 from each category; neutral, joyful, or fearful. FE recognition scores were based on the correct classification of each of the videos into their category, and for any given category could, therefore, range from 0 to 4 (correctly classified videos out of the 4). Valence (mood upon seeing the dynamic FE stimuli) and arousal (excitation by the stimuli) scores ranged from 1 to 9, and were based on a subjective self-evaluation scale after seeing each video again after the main measurement had been completed (see Methods).
Valence scores, arousal scores, and FE recognition scores, all differed across the three categories of videos (neutral, joyful, fearful, p < 0.001, see Fig. 1). Pairwise comparisons showed that joyful FE had the highest valence, arousal, and recognition scores (p < 0.05, Bonferroni-adjusted, two-sided). These results confirm that dynamic FE stimuli from different categories affected participants differently and were recognized as conveying different emotions. It then becomes interesting to assess the potentially different neural responses to these categorical FEs, focusing in this study on the event-related oscillatory responses in delta, theta, and alpha frequency bands.
Event-related oscillatory activity during dynamic FE processing
As discussed in the Introduction, most prior studies used static FE stimuli to study neural mechanisms underlying the processing of facial expressions. Since the neural correlates of dynamic FE processing are comparatively unexplored, we here concretely focused on three frequency bands, previously linked to (different aspects of) FE processing; delta, theta, and alpha. For each, we present time–frequency plots for (1) power analysis which shows the time-locked and/or phase-locked responses to the presentation of dynamic FE videos and (2) phase-locking values which quantify the phase angle synchronization of responses to the dynamic FE videos across trials. Visual inspection showed two separate event-related responses, one after video onset (0 ms), and a second around 1000 ms into stimulus presentation. Statistical analyses focused on comparing data extracted from these two time windows (TW, see Methods for details) for both power and phase-locking analysis across conditions and stimulus categories.
Event-related power analysis
A repeated-measures ANOVA on mean normalized power included factors Time (first TW, second TW), FE (neutral, joyful, fearful), Location (seven electrode clusters, see Methods), and Hemisphere (left, right). Figure 2a shows the event-related power changes in the time–frequency domain (1–13 Hz) as a response to our dynamic FEs per stimulus category for left and right parietal-2 electrodes (P7, P8) in which the main differences were observed. By eye, the two time windows with distinct responses can be observed around 0 and 1000 ms into the stimulus videos.
Delta frequency
There was a main effect of Time (F(df = 1, 17) = 41.965, p = 0.001, and η_p2 = 0.712): the first TW had higher delta power than the second TW (Fig. 2). In addition, FE had an effect (F(df = 2, 34) = 3.406, p = 0.047, and η_p2 = 0.167): delta power in response to the emotional FEs (joyful, fearful) was higher than that of the neutral FE. This difference was due to the responses generated in the second TW, largely, and indeed the Time*FE interaction was statistically significant (F(df = 2, 34) = 7.415, p = 0.003, and η_p2 = 0.304). Post-hoc results showed that the delta power for emotional FEs in the second TW was higher than for neutral FE (p < 0.05) (Fig. 2). There was also a statistically significant Time*Location interaction (F(df = 6, 102) = 7.304, p = 0.003, and η_p2 = 0.301). Post-hoc results showed that occipital areas had the lowest delta power compared to other areas in the second TW (p < 0.05), while there was no such location differentiation in the first TW (p > 0.05). Time*Hemisphere interaction was statistically significant (F(df = 1, 17) = 4.747, p = 0.04, and η_p2 = 0.225). While there was no statistical difference between the hemispheres in the first TW (p > 0.05), delta power was higher in the left hemisphere than in the right hemisphere for the second TW (p < 0.05).
Theta frequency
There was a main effect of Time (F(df = 1, 17) = 79.534, p = 0.001, and η_p2 = 0.824): the first TW had higher theta power than the second TW (p < 0.05) (Fig. 2). The location had an effect (F(df = 6, 102) = 5.986, p = 0.001, and η_p2 = 0.260): theta power was highest in the occipital location (p < 0.05). Time*FE interaction was statistically significant (F(df = 2, 34) = 5.019, p = 0.012, and η_p2 = 0.228). While the responses for different FEs in the first TW did not differentiate (p > 0.05), the theta power for emotional FEs in the second TW was higher compared to the neutral FE (p < 0.05) (Fig. 2). There was a statistically significant Time*Location interaction (F(df = 6, 102) = 4.107, p = 0.025, and η_p2 = 0.195). Post-hoc results showed that the occipital area had higher theta power compared to the anterior areas in the second TW (p < 0.05), while there was no such location differentiation in the first TW (p > 0.05). In addition, Time*Location*Hemisphere interaction was statistically significant (F(df = 6, 102) = 3.989, p = 0.014, and η_p2 = 0.190). In the first TW, the right parietal-2 location had higher theta power (p < 0.05), while there was no hemispheric difference between the locations in the second TW (p > 0.05).
Alpha frequency
There was a main effect of Time (F(df = 1, 17) = 77.993, p = 0.001, and η_p2 = 0.821): the first TW had higher alpha power than the second TW (p < 0.05). There was a statistically significant Time*Location interaction (F(df = 6, 102) = 5.999, p = 0.002, and η_p2 = 0.261). Post-hoc results showed that parieto-occipital areas (parietal-2, occipital) had higher alpha power compared to the other areas in the first TW (p < 0.05), while there was no such location differentiation in the second TW (p > 0.05). Time*Hemisphere interaction was statistically significant (F(df = 1, 17) = 5.294, p = 0.034, and η_p2 = 0.237). Accordingly, there was no hemispheric differentiation in the second TW (p > 0.05), while the right hemisphere had higher alpha power than the left in the first TW (p < 0.05). In addition, the Time*Location*Hemisphere interaction was statistically significant (F(df = 6, 102) = 4.953, p = 0.007, and η_p2 = 0.226). The right temporo-parietal areas (temporal, parietal-2) had higher alpha power than the left temporo-parietal areas in the first TW (p < 0.05), while there was no such differentiation in the second TW (p > 0.05) (Fig. 2).
Event-related phase-locking analysis
A repeated-measures ANOVA on phase-locking value included factors Time (first TW, second TW), FE (neutral, joyful, fearful), Location (seven electrode clusters, see Methods), and Hemisphere (left, right). Figure 2b shows the event-related phase-locking values in the time–frequency domain (1–13 Hz) as a response to our dynamic FEs per stimulus category for left and right parietal-2 electrodes (P7, P8) in which the main differences were observed. By eye, the two time windows with distinct responses can be observed around 0 and 1000 ms into the stimulus videos as in the power analysis.
Delta frequency
There was a main effect of Time (F(df = 1, 17) = 67.762, p = 0.001, and η_p2 = 0.799): the first TW had a higher delta phase-locking value than the second TW (Fig. 2). In addition, FE had an effect (F(df = 2, 34) = 14.701, p = 0.001, and η_p2 = 0.464): delta phase in response to the fearful FEs was higher than that of the joyful and neutral FE (p < 0.05). This difference was due to the responses generated in the second TW, largely, and accordingly, the Time*FE interaction was statistically significant (F(df = 2, 34) = 24.748, p = 0.001, and η_p2 = 0.593). Post-hoc results showed that the delta phase-locking value for fearful FEs in the second TW was higher than for joyful and neutral FE (p < 0.05), while there was no such FE differentiation in the first TW (p > 0.05) (Fig. 2). The location had an effect (F(df = 6, 102) = 11.841, p = 0.001, and η_p2 = 0.411): delta phase-locking value was highest in the parietal-2 and occipital locations (p < 0.05). This location difference is due to the delta responses in the first time window owing to the fact that there was a statistically significant Time*Location interaction (F(df = 6, 102) = 6.547, p = 0.004, and η_p2 = 0.278). Post-hoc results showed that parietal-2 and occipital location areas higher compared to other areas in the first TW (p < 0.05), while there was no such location differentiation in the second TW (p > 0.05). Hemisphere had an effect (F(df = 1, 17) = 18.273, p = 0.001, and η_p2 = 0.518): right delta phase-locking value was higher than in the left (p < 0.05). Time*Hemisphere interaction was statistically significant (F(df = 1, 17) = 4.602, p = 0.047, and η_p2 = 0.213). While there was no statistical difference between the hemispheres in the second TW (p > 0.05), the delta phase-locking value was higher in the right hemisphere than in the left hemisphere for the first TW (p < 0.05). Location*Hemisphere interaction was statistically significant (F(df = 6, 102) = 6.843, p = 0.001, and η_p2 = 0.287). Consequently, the right temporo-parietal areas (temporoparietal, parietal-2) had higher delta phase-locking values than in the left (p < 0.05). This difference was due to the responses generated in the first TW and Time*Location*Hemisphere interaction was statistically significant (F(df = 6, 102) = 3.888, p = 0.024, and η_p2 = 0.186). Post-hoc results showed that the delta phase-locking value in the right temporo-parietal areas was higher than in the left for in the first TW (p < 0.05), while there was no such differentiation in the second TW (p > 0.05) (Fig. 2).
Theta frequency
There was a main effect of Time (F(df = 1, 17) = 127.587, p = 0.001, and η_p2 = 0.882): the first TW had higher theta phase-locking value than the second TW (Fig. 2). FE had an effect (F(df = 2, 34) = 9.857 701, p = 0.002, and η_p2 = 0.367): theta phase in response to the fearful FEs was higher than that of the joyful and neutral FE (p < 0.05). This difference was due to the responses generated in the second TW, largely, and accordingly, the Time*FE interaction was statistically significant (F(df = 2, 34) = 9.166, p = 0.001, and η_p2 = 0.350). Post-hoc results showed that the theta phase-locking value for fearful FEs in the second TW was higher than for joyful and neutral FE (p < 0.05), while there was no such FE differentiation in the first TW (p > 0.05) (Fig. 2). The location had an effect (F(df = 6, 102) = 7.844, p = 0.001, and η_p2 = 0.316): theta phase-locking value was highest in the occipital locations (p < 0.05). This location difference is due to the theta responses in the first time window and consequently, there was a statistically significant Time*Location interaction (F(df = 6, 102) = 3.942, p = 0.012, and η_p2 = 0.188). Post-hoc results showed that occipital areas higher compared to other areas in the first TW (p < 0.05), while there was no such location differentiation in the second TW (p > 0.05). Hemisphere had an effect (F(df = 1, 17) = 4.466, p = 0.05, and η_p2 = 0.208): right theta phase-locking value was higher than in the left (p < 0.05). Time*Location*Hemisphere interaction was statistically significant (F(df = 6, 102) = 3.080, p = 0.035, and η_p2 = 0.153). Post-hoc results showed that the theta phase-locking value in the right temporo-parietal areas (temporoparietal, parietal-2) was higher than in the left for in the first TW (p < 0.05), while there was no such differentiation in the second TW (p > 0.05) (Fig. 2).
Alpha frequency
There was a main effect of Time (F(df = 1, 17) = 71.155, p = 0.001, and η_p2 = 0.807): the first TW had higher alpha phase-locking value than the second TW (Fig. 2). The location had an effect (F(df = 6, 102) = 11.823, p = 0.001, and η_p2 = 0.410): alpha phase-locking value was highest in the parietal-2 and occipital locations (p < 0.05). Hemisphere had an effect (F(df = 1, 17) = 6.967, p = 0.017, and η_p2 = 0.291): right alpha phase-locking value was higher than in the left (p < 0.05). In addition, Location*Hemisphere interaction was statistically significant (F(df = 6, 102) = 6.792, p = 0.002, and η_p2 = 0.285). Post-hoc results showed that the right temporo-parietal areas (temporoparietal, parietal-2) had higher alpha phase-locking values than in the left (p < 0.05). According to post-hoc comparisons, this location, hemisphere difference, and Location*Hemisphere interaction is due to the alpha responses, mainly, in the first time window (0 < 0.05), and consequently, there were statistically significant Time*Location interaction (F(df = 6, 102) = 8.314, p = 0.001, and η_p2 = 0.328), Time*Hemisphere interaction (F(df = 1, 17) = 10.574, p = 0.005, and η_p2 = 0.383), and Time*Location*Hemisphere interaction (F(df = 6, 102) = 4.774, p = 0.011, and η_p2 = 0.219). In addition, Time*FE*Hemisphere interaction was statistically significant (F(df = 2, 34) = 4.872, p = 0.018, and η_p2 = 0.223). Post-hoc results showed that the alpha phase-locking value in the right hemisphere for the neutral FE was higher than in the left in the first TW (p < 0.05), while there was no such differentiation in the second TW (p > 0.05).
A summary representation of the main results of the study is given in Fig. 3.