Dynamic brain networks in spontaneous gestural communication

0
Dynamic brain networks in spontaneous gestural communication

Experimental design

This study used a within-subject design. Dyads were asked to solve RPPs under all three Conditions: both using gestures (‘BOTH’), only one using gestures (‘ONE’) and neither using gestures (‘NO’). Different conditions were distinguished through instructions given to the participants. The sequence of each condition was counterbalanced. Please see the specific instructions and experimental manipulation check in the Supplementary Methods S1.1 and S1.2.

Participants

Fifty-four participants with a mean age of 20.6 ± 2.2 years (27 females, 27 males) were randomly assigned as 27 gender-matched dyads (9 female-male, 9 male-male, 9 female-female) to solve RPPs under three conditions: ‘BOTH’, ‘ONE’ and ‘NO.’ Participants were randomly assigned to either position 1 or position 2, with the participant in position 1 being the one to gesture in the ‘ONE’ condition. In the male-female pairs, we also followed this random assignment. Eventually, 56% of the male-female pairs had the female participant as the one gesturing. Participants were recruited by school-wide online advertising. All participants were right-handed and had normal or corrected-to-normal visual acuity. Participants signed informed consent and were each paid ¥45 for their participation. The study procedure was approved by the University Committee on Human Research Protection of East China Normal University (approval number: HR 160-2019). The sample size in the present study is comparable to previous publications19,28. Moreover, to examine the achieved power in this study, a post-hoc analysis using G*power 3.1 was conducted33. The effect size was set to the smallest reported ηP2 = 0.12 in the behavioural results. The calculated power (1-β) was 0.988 which was acceptable. Written informed consent has been obtained for the publication of all images and related materials.

Procedures

Before starting, participants were asked to complete several pre-tests. The experimental procedure consisted of a 2-min resting-state session, three 5-min task sessions and two 1-min rest sessions. During the 2-min resting session, participants were required to relax with their eyes closed. This session served as the baseline. Prior to the task session, the instructions were introduced. During each task session, participants were asked to generate as many solutions as possible to solve the open-ended realistic problem (e.g. ‘As there are fewer and fewer customers, the school cafeteria will close down soon. What can the owner do to make his business better?’)22,23. Participants were required to talk while taking turns, and report only one idea per turn. During their turns, participants were free to make comments and there was no time limit for each individual turn, allowing participants to explain their ideas fully. If they could not think of an idea during their turn, they were allowed to say ‘pass’ and present an idea during the next turn. The number of ‘passes’ was detailed in the Supplementary Methods S1.1. The effects of task sequence were clarified in the Supplementary Results S2.3 and Supplementary Table 10.

Pre- and post-experimental tests

After each task, participants completed the self-assessment manikin scale34 to measure the valence and arousal of their emotional states. The feelings of depletion, difficulty and enjoyment when performing each task were also rated on scales ranging from 1 (‘not at all’) to 5 (‘very much’).

Behavioural assessments

Participants’ performance on the RPP was measured from four dimensions: fluency, flexibility, originality and feasibility23,24. The fluency score was the total number of ideas. The flexibility score was assessed based on the number of categories of ideas. To offset the impact of fluency on flexibility, the final flexibility score was the number of categories divided by the fluency score. Two trained raters independently coded the flexibility scores for each participant. The inter-rater agreement was satisfactory (internal consistency coefficient [ICC] = 0.83, calculated as Cronbach’s α). The final flexibility score of each participant was the average ratings of the two raters. The originality/feasibility score was assessed using a subjective method. Three trained raters independently assessed the originality/feasibility of each idea on a 5-point Likert scale (1 = not original/feasible at all, 5 = highly original/feasible). The inter-rater agreement of this method was satisfactory (for originality, ICC = 0.81; for feasibility, ICC = 0.70). The score of each idea was obtained by averaging the ratings of the three raters. The final originality/feasibility score of each participant was obtained by averaging the scores of all responses from that participant, and the score of each dyad was the average score of the two members.

The index of cooperation (IOC) was calculated based on the convergence of the ideas, which reflects the perspective-taking behaviours35. The ideas reported by each dyad were listed in chronological order. If one participant reported an idea that belongs to a similar category as the previous one from the other participant, it scored ‘1’. The total number of ideas scored ‘1’ was defined as the ‘converge’, and the IOC for each dyad was calculated using the following equation: IOC = converge/(group fluency—converge). In this regard, this index reflects the level of idea combination and cooperation.

Moreover, the participants’ gestures were videotaped and coded during the entire process. According to previous studies2,5, gestures were divided into four categories based on their functions: (1) Interactive gestures aim to engage the interaction partner in conversation and involve pointing gestures in the partner’s direction (e.g. pointing at someone to indicate that it is their turn to speak). (2) Fluid gestures are repetitive hand movements with no apparent meaning. Specifically, fluid hand movements that are repeated more than three times in a row and have no apparent meaning are considered fluid gestures (e.g. continually tapping the table with the index finger). (3) Object gestures relate to specific objects or their manipulation (e.g. flipping the hands like turning the pages of a book when referring to reading). (4) Emotional gestures are gestures used to express personal feelings (e.g. clenching nervously). Two trained raters counted participants’ gestures of each category based on the recording videos. The inter-rater agreement of this method was satisfactory (interactive gestures: ICC = 0.82; object gestures: ICC = 0.75; fluid gestures: ICC = 0.78; emotional gestures: ICC = 0.71). The final scores were computed by averaging the ratings of the two raters.

fNIRS data acquisition and pre-processing

An ETG-7100 NIRS system (Hitachi Medical Corporation) was used to record the oxyhaemoglobin (HbO) and deoxyhaemoglobin (HbR) concentrations of each dyad. Based on the abovementioned literature suggesting the important role of the mirroring and mentalizing systems in social communication, the optode probe sets were placed on each participant’s PFC (3*5 optode probe set; 22 measurement channels) and right temporal-parietal-occipital (r-TPO) areas (4*4 optode probe set; 24 measurement channels). The registration of the probe sets was based on the 10–20 system. Channel locations were further determined using a 3D-digitiser and subsequently transformed into Montreal Neurological Institute (MNI) coordinates36. Then the MNI coordinates of each fNIRS channel were mapped onto the corresponding regions using the Automatic Anatomical Labelling (AAL) atlas. Specifically, each channel was automatically allocated to the closest AAL region based on its MNI coordinates, and channels within the same AAL region were grouped to create Regions of Interest (ROIs). The MNI coordinates of the optode channels in a normal participant are presented in Supplementary Table 11. Eventually 13 ROIs are created, including frontopolar cortex (FPC), left dorsolateral prefrontal cortex (lDLPFC), right dlPFC (rDLPFC), left inferior frontal gyrus (lIFG), right IFG (rIFG), right middle temporal gyrus (rMTG), right superior temporal gyrus (rSTG), right primary somatosensory cortex (rPSC), right angular gyrus (rAG), right supramarginal gyrus (rSMG), right motor cortex (rMotor), right somatosensory association cortex (rSAC), right visual cortex (V3). Among these, the rIFG, lIFG, rSTG, rPSC, rMotor, rSAC and V3 belong to the mirroring system; while the FPC, lDLPFC, rDLPFC, rMTG, rAG and rSMG belong to the mentalising system. See details in Fig. 2.

Prior to establishing the ROIs, several pre-processing steps were conducted to reduce noise. First, we plotted the time courses of all channels. Two raters worked together to visually check the plots and excluded channels that exhibited obvious noise (for example, the variances of noisy channels ranged from 10 to 30, while the variances of normal channels ranged from 0.5 to 0.8). The noisy channels were replaced with data from neighbouring channels within the same ROI37. On average, each participant had 2.3 noisy channels. Because the noisy channels varied across participants, we did not universally exclude any specific channels. Subsequently, a principal component spatial filter algorithm was used to remove the effects of systemic components, such as blood pressure, respiratory variation and blood flow variation38. Moreover, a correlation-based signal improvement method was utilised to remove motion artefacts39. To collect data within the steady period, the data in the initial and final 30-s periods of each condition were removed, leaving 240 s of data for each task session. Previous studies found that the HbO signal is more sensitive to changes in cerebral blood flow than the HbR signal, therefore we mainly focused on the HbO signal for further analysis40,41.

fNIRS data analysis

The fNIRS data analysis mainly includes two components: (1) intra- and inter-subject functional connectivity (i.e. FC and ISFC) computation, (2) dynamic brain network analysis.

Wavelet transform coherence (WTC) was used to calculate the FC and ISFC. WTC is a method that measures the cross-correlation between two HbO time series across both frequency and time domains. This technique allows us to capture how the relationship between brain regions changes over different frequencies and time periods, providing a detailed understanding of neural connectivity42,43. Fisher’s r-to-z transformation was applied to the WTC values before further analysis to increase normality of the distribution. We calculated the FC and ISFC of all ROI combinations (i.e. 13 × 13 intra-subject FC matrices for each participant, 13 × 13 inter-subject FC matrices for each dyad). For the inter-brain network, the ISFC values between the same ROI pairings were averaged. For the intra-brain network, the FC matrices of the participant 1 and 2 were averaged under the ‘BOTH’ and ‘NO’ conditions, while under the ‘ONE’ condition, the person who used gestures was called the user and the person who did not use gestures was called the observer, and the intra-brain matrices of the two were calculated separately.

To determine the frequency band of interest (FOI), the following procedures were performed: first, data below 0.01 Hz were excluded to avoid low-frequency fluctuations, and data above 0.7 Hz were also excluded to ensure that high-frequency noise such as cardiac activity (0.8–2.5 Hz) was eliminated44. Therefore, the full frequency range is 0.01–0.7 Hz. Second, to avoid bias, the FC/ISFC were averaged across different conditions (i.e. for ISFC: ‘BOTH’, ‘NO’ and ‘ONE’ conditions; for FC: ‘BOTH’, ‘NO’, ‘ONE-user’ and ‘ONE-observer’ conditions) before conducting the following t-tests45. Subsequently, paired sample t-tests were performed to compare ISFC between the task session and baseline session for each ROI combination along the full frequency range to identify the FOI. This process was repeated for FC, comparing the task session against the baseline session. After the false discovery rate (FDR) correction, significant P-values (Pcorr < 0.05) were observed at frequencies between 0.034 Hz and 0.045 Hz (corresponding to the period between 22.2 and 29.7 s) for both FC and ISFC. These results indicated that in this frequency band, both FC and ISFC of the task session were significantly higher than that of the baseline session. Additionally, to investigate if a specific condition extremely influences the determination of the FOI, paired sample t-tests were performed to compare FC/ISFC between the task session and rest session under each condition (see details in the Supplementary Methods S1.3). After the FDR correction, the FOIs for each condition are detailed in Supplementary Table 12 and visually represented in Supplementary Fig. 4. According to these results, the selected FOI almost incorporated significant frequency bands in all conditions. Therefore, this band was identified as the FOI, and the FC and ISFC within the FOI were averaged for further analyses.

The dynamic approach based on sliding windows and k-means clustering was applied to characterise the FC and ISFC states under each condition. First, sliding windows were used to segment the FC and ISFC data. The window size was set to 26 s (according to the FOI) and moved in an increment of 1 s throughout the task, and the FC/ISFC values were then averaged within each time window, resulting in a series of FC/ISFC matrices. Then we averaged these FC/ISFC matrices across groups and applied a k-means clustering algorithm in MATLAB to capture the representative FC/ISFC states (i.e. clusters). K-means clustering is a method that groups data into a specified number of clusters based on their features. It is particularly useful for identifying patterns in large datasets by organising similar matrices together. This technique helps to reveal the underlying structure of the data, allowing us to understand different states of brain networks. Specifically, the Manhattan distance was used to calculate the similarity between the windowed FC/ISFC matrices46, and the number of clusters was identified based on the elbow criterion of the cluster validity index. The elbow criterion is a widely-used technique in cluster analysis for determining the optimal number of clusters (k) in a dataset28,47,48. Specifically, the cluster cost was computed as the ratio of within-cluster distance to between-cluster distance for a range of k values (i.e. 1–15). These computed costs were then graphically represented as a function of cluster number (Supplementary Fig. 5). This strategy aims to minimise the within-cluster distance and maximise the between-cluster distance, while controlling the number of clusters. The appropriate k value was selected at the elbow of the curve, thereby achieving a balanced trade-off between cluster cost and the number of clusters. In this study, we chose k = 3 as the appropriate k value, and after iterating 1000 times to avoid accidental results, the cluster centroids (i.e. representative FC/ISFC states: State 1, State 2 and State 3) were obtained. These cluster centroids derived from the group-averaged FC/ISFC matrices were then used as the initial centroids for the cluster analysis of the single dyad to obtain the dynamic FC and ISFC (dFC & dISFC) states of each dyad.

The dFC and dISFC states were further evaluated by graph-based network analysis49. The ROIs were treated as the nodes of the brain network and the functional connections between any two of them were the edges. To exclude the effects of spurious correlations, a sparsity threshold was utilised to ensure that only the highest correlations remained. The sparsity threshold was the ratio of the number of existing edges divided by the maximum possible number of edges in a network. Specifically, we thresholded each FC/ISFC matrix multiple times over a wide range (0.2 < sparsity < 0.5; interval = 0.01) to obtain the weighted networks. Finally, brain network parameters, including cluster coefficient (Cp), shortest path length (Lp), and global efficiency (globE), were calculated using GRETNA in MATLAB50. Cp represents the extent of local interconnectivity or cliquishness in a network. A higher Cp in brain networks often suggests more efficient information transfer among neighbouring brain regions51. Lp is a harmonic mean length between all pairs of nodes and indicates the average shortest number of edges that must be traversed from one node to another in the network. A shorter Lp in brain networks indicates more direct pathways for information transfer across distant brain regions, which are associated with higher cognitive performance in tasks involving working memory and attention52. globE assesses the efficiency and vitality of information transmission over the entire network. Higher globE suggests that the brain network can process information more parallelly and rapidly, leading to more efficient cognitive processing. It can be associated with better performance on tasks that require integration of information from different brain regions50. A large number of studies have shown that higher Cp, globE and shorter Lp in the intra-brain network are associated with faster idea generation and information integration, promoting the ability to combine disparate ideas into novel constructions and for the inter-brain network, they are related to more efficient interpersonal communication and better teamwork20,31,32,53. Therefore, we selected these three classical indicators to measure the participants’ brain network states during the task.

Moreover, we also calculated the ROI strength of each dFC/dISFC state. The strength of a ROI is defined as the sum of all FC/ISFC connected to this ROI, which indicates its involvement or importance in the network49. It quantifies how much this particular ROI is involved in intra- or inter-brain connectivity. If an ROI has a high ‘strength’, it suggests that this region is highly synchronising with other regions.

Statistical analysis

For behavioural data, one-way repeated measures ANOVA using Condition (‘BOTH’; ‘NO’ vs. ‘ONE’) as the within-subject factor was performed on IOC, RPP fluency, RPP originality and RPP flexibility. Moreover, in order to examine the variation of gestures under different conditions, we also conducted an exploratory analysis. Because there were two participants using gestures in the ‘BOTH’ condition and only one in the ‘ONE’ condition, we divided the number of gestures in the ‘BOTH’ condition by two (the ‘BOTH/2’ condition) and compared it with the ‘ONE’ condition. One-way repeated measures ANOVA using condition (‘BOTH’; ‘BOTH/2’ vs. ‘ONE’) as the within-subject factor was performed on the number of all types of gestures. Post-hoc Bonferroni correction was used to account for multiple comparisons.

For fNIRS data, the dFC/dISFC states were discriminated by the network parameters (Cp, Lp and globE), ROI strength and the FC/ISFC increments. Repeated measures ANOVAs using State (State 1; State 2 vs. State 3) as the within-subject factor were performed under each condition on the network parameters (Cp, Lp and globE) and ROI strength. Moreover, paired-sample t-tests were performed to compare FC/ISFC between the different States and the baseline session, further identifying the FC/ISFC increments under each State. After delineating the characteristics of different states, the occurrence rates of distinct states were used to measure the difference between conditions. Specifically, One-way repeated measures ANOVAs with State (State 1; State 2 vs. State 3) as the within-subjects factor were used to analyse the occurrence rate of each State under various conditions. The FDR correction, post-hoc Bonferroni correction and validation analyses (see details below) were used to avoid false discoveries.

In addition, we also conducted the behaviour locking analysis (Fig. 2b). The time points of the specific gestures were marked and the correspondence dFC/dISFC states were gathered to obtain the brain network when using different kinds of gestures and not using gestures. Repeated measures ANOVA using State (State 1, State 2 vs. State 3) as the within-subject factor was employed to assess whether there were significant differences in the occurrence rates of states among different kinds of gestures. Moreover, we also computed the ROI strength for different kinds of gestures. Repeated measures ANOVA with Gestures (interactive, fluid, object, emotional vs. none) as the within-subject factor was used to determine whether there were significant differences in ROI strength across various gesture types. The FDR correction and post-hoc Bonferroni correction were used. Besides, the brain network parameters of different kinds of gestures were also calculated as an additional exploratory analysis (please see details in the Supplementary Results S2.4 and Supplementary Table 13).

Pearson’s correlations between the occurrence rates of different dFC/dISFC states and the behavioural indices (the IOC/fluency/originality/flexibility scores and number of different gestures) were calculated to reveal the brain-behaviour relationship. Validation analyses (see details below) were used to avoid false discoveries.

Validation analysis

For ISFC, we constructed reshuffled dyads by rearranging individuals from real dyads under the same condition. All analyses were applied to the reshuffled data in the same manner as to the real data. Specifically, we calculated the dISFC matrices of the 27 reshuffled dyads (the same sample size as the real group) and obtained 3 representative dISFC states (the same number of cluster centroids as the real group) using dynamic brain network analysis. Then the occurrence rate of each dISFC state was calculated. This permutation process was repeated 400 times. Since it was difficult to calculate the IOC scores of reshuffled dyads, we only calculated the RPP originality, RPP fluency and the occurrence rate of each dISFC state. The originality/fluency score of the reshuffled dyads was the average score of the two individuals. Different from the real condition, over 95% of the reshuffled groups showed no significant correlations between RPP originality/fluency and the occurrence rate of any dISFC states, and the observed ISFC increments of the real data were in the top 5% of the permutation distribution. Besides, for the sISFC results, the static analysis approach was also applied to the reshuffled data in the same manner as for the empirical data. This permutation process was repeated 1000 times, and the observed effects of the real data were also in the top 5% of the permutation distribution.

For FC, the permuted data were constructed using circular shifts (preserving the temporally autocorrelated structure of the data)54. All analyses were applied to the time-shifted data in the same manner as to the real data. This permutation process was repeated 400 times under each Condition. Different from the real condition, over 95% of the time-shifted groups showed no significant correlations between the behavioural indices and the occurrence rate of any dFC states, and the observed FC increments of the real data were in the top 5% of the permutation distribution.

link

Leave a Reply

Your email address will not be published. Required fields are marked *