Machine-learning applications for differentiation across states/stages of creative thinking based on time-series and time-frequency features of EEG/ERP signals

Full Text

INTRODUCTION

Machine learning (ML) is widely used in the recognition of various “functional” states (emotions, phases of sleeping, tiredness, etc.) from physiological data (particularly, electroencephalography [EEG]); however, no many studies of cognitive state discrimination have been conducted thus far. Moreover, ML is proposed as the basis for brain–computer interface development in different areas of rehabilitation — motor [1], cognitive [2], or some enhancement procedures [3]. Studies on state recognition have shown great variability in ML methods, choice of classification approaches, and input features. The two main approaches to state recognition based on EEG features are as follows: 1) recognition of characteristics of “continuous” EEG data when the state either remains constant or slowly changes over a long time, i.e., from a few seconds to several minutes (spontaneous EEG); or 2) recognition of characteristics of “short” intervals, i.e., lasting up to a couple of seconds when brain activity is associated with fast decisions or perceptive and cognitive responses to some external stimulus, such as event-related potentials (ERP). In the first case, strict synchronization with external events is unnecessary, and features for classification relate to the so-called “spontaneous” EEG, which might be represented as frequency band power features [4–6]. For “transient processes” classifications based on ERP features, time series is mostly used [4]. In addition, no common approaches could be used without testing it for the certain type of the task. Thus, this study aimed to explore and compare different ML approaches that could be used for the classification of creativity stages in continuous EEGs and short ERP trials. In both cases, divergent thinking tasks were used without and with time restriction. As the model situation, the test-control approach was used, and control tasks were considered as possible (according to hypothesized mental processes) and previous creative decisions as stages/states.

Literature data on the implementation of ML in creativity research are limited. In a previous study [7], the two-class system (more and less creative states in alternative uses task and ordinary or uncommon (more creative) uses of everyday objects) achieved an average accuracy of 63.0%, and the ML approach used included spectrally weighted common spatial patterns (CSPs) for feature extraction and quadratic discriminant analysis. Discrimination of more and less creative individuals based on EEG signals had 82.3% accuracy. Data [8] on the differentiation of three classes — creating an original ending, suggesting a synonym, or remembering a well-known ending of a proverb or saying — were explored, with an average accuracy of 48±5% for the best linear classifier using a classifier learner toolbox in Matlab. The physiological effects described for these data among others revealed higher power (8–9 Hz) in the right frontal and left parietal regions at 400–700 ms after stimulus onset while creating original and synonymic endings compared with the control task of remembering the ending of a well-known proverb. In the nonverbal creativity model [9], i.e., divergent thinking during painting, the accuracy for the separation of the background state and creative and noncreative drawing was 66.9% when using a classifier based on a support vector machine (SVM; Gaussian radial basis function and classifier learner). The physiological effects demonstrated a higher percentage of theta and alpha frequencies in the frontal (5–6 and 12–13 Hz), central (4–7 and 8–10 Hz), and parietal (4–5, 6–7, 8–9, and 12–13 Hz) zones during creative sketching in comparison with noncreative lines and object drawing. Creativity is a heterogeneous process in which certain stages can be defined, for example, idea generation, idea elaboration, idea evaluation or generation, and exploratory stages [10]. In a previous study, the specific creative demands were set by instructions [11] and considered models of creative process stages; thus, the performance of these tasks could correspond to the distinct states of creative thinking. In our study, models of “STORY” creation and “PROVERBS” ending creation can be considered different creative states (stages defined by instruction). More clearly, different stages of creative processes can be distinguished in free creative activities using artistic painting. In this case, all stages from idea generation (sketching) via idea elaboration (canvas painting) and idea evolution (color painting) to evaluation (viewing own painting) can be defined and verified by observation.

To the best of our knowledge, not many researchers have attempted to build classifiers to differentiate creative states and stages. However, such classifiers could be used in cognitive rehabilitation [12].

The use of ML algorithms to differentiate stages of the creative process is within fundamental science and allows us to answer the question of whether information about the type of performed mental operations is reflected in a complex EEG pattern. On the contrary, these approaches can be used to create a “passive” brain–computer interface (see, for example, [13]); when using an external stimulus (music and light), the user can be informed about his/her achievement of the “optimal” state for creative activity.

Aim — the study aimed to find the most effective EEG-based classifiers and feature generation’s approaches to differentiate stages and states of creative thinking.

MATERIALS AND METHODS

Stimuli and tasks

Three models of divergent thinking, namely, verbal (“STORIES”) and nonverbal (“PAINTING viART”) without time restriction and verbal (“PROVERBS”) in the ERP paradigm (with time restriction), were applied. In “STORIES” and “viART”, continuous EEGs were analyzed, whereas in the creative model, “PROVERBS” was examined in relatively short trials.

Divergent thinking tasks with continuous electroencephalograms registration

The comparative study using time-series and time-frequency maps (wavelet transformation) classifications at different stages of creating plots of stories (“STORIES”), or visual images (“viART”) was conducted.

Story creation task (“STORY”)

Electroencephalography data from the story creation task [14] were used for the four-class system. The participants created stories using black-and-white pictures with only two personages from the Guilford and O’Sullivan social IQ test [15].

The three tasks were as follows: the free creation task (FCrT), where the participants were told to create the plot before and after the situation based on what they had seen in the picture. The participants mentally pronounced sentences and voluntarily pressed the button when they were ready to tell the story. In the effortful creation task (EffCrT), the participants were asked to continue the story plot, changing the previous line they had just created and using the same picture presentation. FCrT and EffCrT could be considered two stages of the same process — creating the original story with some impasse and overcoming self-induced short memory stereotype — i.e., in a situation, the participants had to expand the story. In the control task, the participants were asked to mentally describe the picture.

Ecological (free) painting with three or five states/classes: “viART” model

The EEG data used for classification were taken from a longitudinal case study with the participation of an artist (J.P.). Classification was conducted for three (final stage of canvas painting, viewing own painting, and background state with opened eyes) or five (sketching in the album, oil transfer to canvas, final stage of the canvas work, viewing own painting, and a background state) classes, respectively. All stages of free creative painting were empirically observed and marked through time by the investigator.

Divergent thinking task with time restriction (event-related potential paradigm)

Electroencephalography data were collected while the participants were creating original endings for known uncompleted proverbs and sayings (Cr) [16]. The control tasks were to recall the commonly known ending of an uncompleted proverb or saying (C) or to give a synonym for proverbs’ ending (Syn). All tasks were associated with the retrieval of information from long-term memory. The difference between creative and noncreative tasks was in overcoming the previously formed stereotype and searching (creation) for a new original ending of a proverb or saying that would significantly change its meaning.

Participants

Fifteen participants (aged 18–20 years, 9 female) took part in the divergent verbal creative study without time restriction (“STORIES” model), and one participant (professional artist, J.P., aged 57 years) was involved in the nonverbal creative painting task (“viART”) without time restriction.

In the model of divergent verbal tasks with time restriction (ERP paradigm), data of 22 participants (aged 18–22 years, 18 female) were analyzed and classified.

All procedures were conducted in accordance with the Declaration of Helsinki (1974) and its subsequent updates. The study was approved by the Ethics Committee of Sechenov Institute of Evolutionary Physiology and Biochemistry of the Russian Academy of Sciences, Saint Petersburg, Russia (Protocol number 2-02, February 2, 2022). Participation in the study was voluntary, and participants could drop out at any moment. All participants were free of any medical or neurological disorders and had normal or corrected vision. Written consent was obtained from all participants before the study in according to the protocol.

Psychological testing

To ensure that the groups had homogenous cognitive abilities, standard progressive matrices [17] were applied. This test is a nonverbal, culturally independent IQ test that measures deductive reasoning through five sets of multichoice tasks. Obtained data revealed average IQ values ~110±8 for the partisipants sample.

EEG/ERP procedure and data registration

EEG/ERPs were recorded with “Mitsar 31-channel” (Fp1, Fpz, Fp2, F7, F3, Fz, F4, F8, FT7, FC3, FCz, FC4, FT8, T3, C3, Cz, C4, T4, TP7, CP3, CPz, CP4, TP8, T5, P3, Pz, P4, T6, O1, Oz, and O2) EEG system (“Mitsar Ltd.”, Saint Petersburg, http://www.mitsar-medical.com) or with the “SmartBCI 24-channels”(“Mitsar Ltd.”, Saint Petersburg, http://www.mitsar-medical.com) (Fp1, Fp2, F7, F3, Fz, F4, F8, T3, C3, Cz, C4, T4, T5, P3, Pz, P4, T6, O1, O2) EEG system through the WinEEG software package (Ponomarev V.A., Kropotov Ju.D., registered for a computer program RF N 2001610516, 08.05.2001). Silver chloride electrodes were positioned according to the modification by 10–10% or 10–20%. Input signals were referenced to the linked ears, filtered between 0.53 and 30.0 Hz, and were digitized at a sampling rate of 500 or 250 Hz correspondingly, with a notch filter of 45–55 Hz. The ground electrode was located between the Fpz and Fz sites on the forehead. Resistance of the electrodes did not exceed 5 kOhm.

Electroencephalography signal artifact correction

The eye-blink artifacts were corrected by zeroing the activation curves of individual independent components corresponding to eye blinks. These components were obtained by the application of independent component analysis (ICA) to the raw EEG fragments. The method has been previously described [18–20]. High- and low-frequency activities were automatically marked as artifacts and were excluded from further analysis. The thresholds were set as follows: (1) 50 µV for the slow waves in the 0–2 Hz band and 2) 35 µV for the fast waves in the 20–35 Hz band.

Electroencephalography signal feature extraction

The search for informative features and a short-term approach for EEG data preprocessing is important for state classification, which could be used in practical applications. The difficulty of classification of EEG signals is associated with the low spatial resolution of this method. To increase the spatial resolution of EEG, the current source density (CSD) transformation was used [21, 22–24], which can be employed for both continuous EEG and ERP. The CSD reduces the volume conduction effect on the signal recorded from the head surface [22, 23] and makes local differences distinguishable that can otherwise be masked by the activities of the neighboring cortical areas [24].

Continuous electroencephalography signal feature extraction (“STORIES” and “viART” models)

The time-series feature vector generation and wavelet time-frequency analysis were used. In both cases, the artifact-free CSD-transformed EEGs were divided into 4-s fragments with a shift of 100 ms for further analysis.

Time-series feature vector generation. The time series of EEG amplitudes from two electrodes (Fz and Pz) were combined into one feature vector: 2000 time points for each 4-s EEG fragment. The number of 4-s EEG fragments (trials) for classification was equalized between classes in each participant individually.

Wavelet time-frequency analysis. Continuous wavelet transform (CWT) was implemented in Matlab [25]. The analytic Morlet wavelet was used to create the CWT (40 voices per octave) in each 4-s fragment. L1 normalization was used by the CWT function. The minimum and maximum scales for the wavelet energy visualization on time-frequency maps were set equally for all states and participants (max = 12 for continuous EEG). The frequency was presented on a logarithmic scale. The amplitude was normalized within the specified range for each sample. Combined together, time-frequency maps (CWT plotted graphs) from frontal (Fz) and parietal (Pz) electrodes formed one image with 224×224 px resolution. Samples of such images were used as trained and test sets for the ResNet50 convolution network for creative state classification. The modified architecture — ResNet18 (with 70 layers) — was tested for CWT combined images. The number of images for classification was comparable between classes in each participant.

Feature extraction in the event-related potentials paradigm

As previously shown [8], the classification of raw time-series signal in the PROVERBS model had an average accuracy of 48±5% for three classes. Thus, we had to explore and compare other approaches to choose more robust classification methods for transient processes such as ERPs.

At this time, for ERP feature extraction, CSP decomposition was applied for the time-series analysis and wavelet time-frequency analysis.

Time-series CSP feature vector generation. The CSP was used for feature generation in the time domain for short-time intervals (1500 ms) after stimuli presentation. CSPs maximize the variance for one class (least-squares sense) but minimize the variance for the other [26]. As the CSP parameter, the number of components was set. The classifier could not accurately distinguish between the two classes with too few components. However, if there were too many components, the classifier weights might be significantly overfit [27]. CSP was calculated on the space of electrodes located in the central regions (F3, Fz, F4, FC3, FCz, FC4, C3, Cz, C4, CP3, CPz, CP4, P3, Pz, and P4) using MNE-Python (https://mne.tools/stable/generated/mne.decoding.CSP.html). The number of components was selected empirically and was equal to 15.

For the multiclass paradigm (three classes in our case), CSPs were calculated by joint approximate diagonalization that might be equivalent to an ICA, and a method of choosing independent components (ICs) that approximately maximize mutual information of ICs and class labels was presented [28].

Wavelet time-frequency analysis. All preprocessing was the same as described above for continuous EEG data with a time window difference — here, the 1500-ms time following stimuli presentation was used for the CWT time-frequency calculation. The minimum and maximum scales for the wavelet energy visualization on the time-frequency maps were set equally for all states and participants (max = 16 for ERP).

Classification methods

Electroencephalography signal time series were classified using algorithms from the classifier learning toolbox in Matlab. The results of the method with the best accuracy classification for each participant were considered. An empirical assessment of the generalization ability of algorithms was performed automatically by K-folds cross-validation: the total number of trials was successively divided into five samples (nonoverlapping “bootstrap”), with four of them (80% trials) included in the training sample and one (20% trials) in the test sample. The training and test samples of the trials did not overlap. The principal component analysis tool in Matlab was used to reduce the dimension of the input feature vector.

The images of time-frequency maps obtained for “STORY”, “viART”, and “PROVERBS” models were classified using a convolutional neural network (CNN, with ResNet50 architecture) in Matlab Deep Network Designer Toolbox. The numbers of classes (three, four, or five) were set at the last fully connected level. The total samples (images) set for each participant was randomly divided into three nonoverlapping samples: test (15–25% depending on the size of the total sample), validation (15–30% of the remaining set), and training. Training options for CNN were set as default in Matlab Deep Network Designer Tool with MaxEpoch of 30 and MiniBatchSize of 32.

The CSP-filtered data (“PROVERBS” models) were classified using SVM with the radial basis (kernel) function (RBF) and with one-against-each approach of multiclass classification (skLearn and Python). In all approaches to the classification, the sample sets for different classes were equal in every participant so the empirical chance level was close to the theoretical chance level (20.0; 25.0; and 33.3% for the five-, four-, and three-class systems, respectively). The classification accuracy far exceeding this threshold was considered significant.

In both continuous tasks (“STORY” and “viART”), the “neurointerface usage conditions” were modeled. The training and testing sample sets in these cases were formed not by bootstrapping from the whole EEG but consequently: training sample (from the first part of each EEG record) and then the testing sample (from the last part of each EEG record).

Statistical analysis

Statistical comparison of different approaches to classification was performed using the Wilcoxon test for dependent samples, and significant differences with p <0.05 were considered. The Wilcoxon test was selected because we did not expect a normal classification accuracy distribution in the participants’ sample. Moreover, in each participant, since the same data were classified, expected dependent classification accuracies were obtained by different methods.

RESULTS

Classification of spontaneous electroencephalography signal characteristics during divergent creative thinking

Story creation model. A four-class classification was made, i.e., creation of a story plot based on a picture (stage 1, FCreT), creation of the story’s plot further changes (stage 2, EffCrT), description of a picture, and background EEG with eyes open (Table 1).

 

Table 1. Accuracy for the four-class classification based on spontaneous EEG features: time series and images of time-frequency maps from the wavelet analysis

Таблица 1. Точность 4-классовой классификации признаков спонтанной ЭЭГ: временн´ых рядов и изображений частотно- временн´ых карт вейвлет-анализа

Subject number	Time-series classification (4 classes)		Wavelet images classification (4 classes) ResNet50; accuracy, %
	Accuracy; %±SD	Classifier
S1	53.5±1.7	Ensemble bagged tree	100
S2	48.8±3.3	Ensemble boosted tree	75
S3	74.3±1.8	Ensemble bagged tree	99.7
S4	43.4±0.2	Tree medium tree	97.2
S5	89.7±1.0	Ensemble bagged tree	100
S6	63.7±2.7	Ensemble bagged tree	100
S7	54.6 ±1.5	Gaussian naive bayes	100
S8	49.8±0.7	Kernel naive bayes	98
S9	87.9±1.5	Ensemble subspace knn	100
S10	45.1±0.5	Ensemble bagged tree	100
S11	57.3±3.0	Ensemble bagged tree	100
S12	59.5±0.5	Ensemble bagged tree	99.3
S13	64.9±1.9	Ensemble bagged tree	100
S14	69.5±1.1	Ensemble bagged tree	100
S15	33.4±0.5	Ensemble subspace Discriminant	77.0
Mean	59.7±15.8		96.4±8.3

 

The level of classification performance with the time-series features was higher than that at the theoretical chance level (25% for the four-class system) in all participants (See Table 1). The types of classifiers showed that better results varied among participants; however, most often (for 9 among 15 subjects), a higher accuracy in recognizing the stages of story creation was demonstrated by the ensemble bagged tree classifier. The classification performance of the CNN (ResNet50) classifier was significantly higher in comparison with the time-series classification: Z=3.4, p <0.0007. Even in participants in whom the classification accuracy of states using time series was very low (participant 15, 33.4%; or participant 4, 43.4%), the use CNN for time-frequency maps’ images significantly increased the classification accuracy (up to 77% and 97.2% correspondingly).

Free artistic painting/creation of visual images (“viART”) model. Five or three stages of creating two oil paintings by a professional artist were classified. These included sketching in the album, sketching on the canvas, color oil painting on the canvas, viewing the ready painting, and background EEG with opened eyes for the five-class system and color oil painting on the canvas, viewing the ready painting, and background EEG with opened eyes for the three-class system.

The time-series classification achieves an accuracy of 75.8% for the three-class system (ensemble bagged tree) and 56.2% for the five-class system (kernel SVM). In both conditions, the levels of classification performance were higher than the theoretical chance level (33.3 and 20%, respectively). The implementation of CNN (ResNet50) for the classification of time-frequency map images increased the accuracy level up to 99% for both conditions. Thus, different stages of the creative process by a professional artist appear to be distinguished using ML.

In both continuous tasks (“STORY” and “viART”), the lighter CNN architecture (ResNet18) was tested to classify time-frequency images, and the neurointerface usage conditions were modeled. In this case, the training and testing samples were formed not by bootstrapping from the whole EEG but consequently: training sample (from the first part of each EEG record) and then testing sample (from the last part of each EEG record). The classification accuracy in this mode was expectably lower, with an in average of 70% [11 SD].

Classification of creative thinking stages by ERP (single trial) features. The results of the implementation of different approaches to the single-trial classification in creative task performance are presented in Table 2.

 

Table 2. Classification accuracy for the three-class discrimination (create original proverb ending, recall ending, and find a synonym to the ending) based on event-related potential features: common spatial pattern for the time series and images of the time-frequency maps from the wavelet analysis

Таблица 2. Точность классификации трёх классов (придумать оригинальное окончание пословицы, вспомнить окончание и назвать синоним к окончанию) на сновании признаков вызванных потенциалов: пространственной фильтрации временн´ых рядов (CSP) и изображений частотно-временн´ых карт вейвлет-анализа

Subject number	Time-series (3 classes) by CSP (SVM); Accuracy %±SD	Wavelet images (3 classes) ResNet50; Accuracy %
S1	78.9±7.2	51
S2	81.9±5.6	32
S3	77.4±5.9	54
S4	83.3±4.2	47
S5	80.0±7.4	36
S6	90.4±2.6	53
S7	80.2±4.8	44
S8	56.2±8.3	40
S9	95.3±3.5	37
S10	79.3±5.2	36
S11	85.3±2.9	36
S12	86.8±4.6	38
S13	76.5±6.7	43
S14	88.1±6.5	29
S15	73.0±7.5	32
S16	81.5±2.7	40
S17	98.1±2.5	43
S18	77.3±4.6	55
S19	72±6.2	57
S20	80.9±6.5	59
S21	70.9±6.3	50
S22	78.3±3.5	38
Mean	80.5±8.8	43.2±8.8

Note: CSP — common spatial pattern; SVM — support vector machine.

Примечание: CSP — общий пространственный фильтр; SVM — машина/метод опорных векторов.

 

The performance of the SVM classifier based on the spatial filtration of time-series data (CSP) was significantly higher than that of the CNN (ResNet50) classifier for the time-frequency map images: Z=4.1; p <0.00004 (See Table 2). The minimal decoding accuracy with the CSP feature generation was 56.2% (participant 8), whereas the CNN classifier had an accuracy of 33.3%, which was close to the theoretical chance level threshold in 9 of the 22 participants.

The applied multiclass spatial components filtering CSP with the following SVM classifier demonstrated high discriminative accuracy with more mixing of Cr and Syn tasks, as these states are closer to each other than to the control task (Table 3).

 

Table 3. Average confusion matrix for the three-class system in the “PROVERBS” task using common spatial pattern for the time series

Таблица 3. Усреднённая матрица смешивания при классификации трёх классов в задании «ПОСЛОВИЦЫ» с использованием пространственной фильтрации (CSP) временн´ых рядов

True labels, %	Predicted labels, %
	Cr	C	Syn
Cr	78.0	8.7	13.3
C	8.4	86.2	5.5
Syn	15.0	5.3	79.7

Note: Cr (creative) — to create an original proverb’s ending; C (control) — to recall commonly known endings; Syn (synonym) — to find out a synonym to the proverbs’ endings.

Примечание: Cr (творческое) — найти синоним к собственному варианту окончания пословицы; C (контроль) — вспомнить общеизвестное окончание пословицы; Syn (синоним) — найти синоним к известному окончанию пословицы.

 

The confusion matrices in the classification procedure can give additional information for the physiological individual and group data analysis that could be used complementarily for the evaluation of more and less close states based on the discriminated feature vectors.

DISCUSSION

In this comparative classification study, we aimed to develop an approach to classify creative states and stages using EEG times-series and time-frequency analyses. The study results supported the requirement of different classification methods for EEGs in long-lasting/continuous creative states and fast creative tasks.

The wavelet analysis and classification of time-frequency images showed higher effectiveness for distinguishing long-lasting creative states with the decoding accuracy for four classes of up to 96.4±8.3 [SD] compared with the time-series analysis (best results, 59.6±15.8 [SD]). Compared with the time-series analysis, the wavelet (time-frequency) analysis brings EEG power ratio for different frequency bands. It appears that the frequency characteristics effectively describe some stable states formed during creative activity at different stages that could be separated by classification. Physiological data could clarify features that may be sensitive to EEG signal classification. Thus, in [14], the frequency structure and spectral power differences between free story creation (FCrT) and effortful story creation (EffCrT) were revealed. Compared with FCrT, the EffCrT (creative task with overcoming of self-induced stereotype) demonstrated a higher percentage in 9–10, 10–11, and 11–12 Hz and increased power in the temporal and occipital areas. Moreover, an increase of alpha activity was discussed in accordance with attentional-defocused states and blockage from external information, which could be important for effective creative activity.

At present, frequency-specific EEG features have been effectively used to distinguish emotional states based on EEG data [5]. Approaches for classifying creative and emotional states might be similar because these states can have some “stable”/reproducible patterns; however, they can undergo smooth rearrangements. Another question under investigation is the assessment of not only the frequency but also the spatial characteristics of the EEG for classification: in this case, the EEG is a three-dimensional array with the estimation of time, power, and spatial location of the electrodes on the head surface [29–31]. For emotion recognition by EEG features, Wang et al. [31] used electrode–frequency distribution maps calculated based on short-time Fourier transformation as features and CNN with residual blocks for classification, which achieved 90.59% accuracy for the three-class system (positive, neutral, and negative emotions). Kim et al. [32] used a 3D spatiotemporal representation of EEG signals as features and CNN with a channel bottleneck module (CNN-BN) as a classifier and reached accuracy up to 99% for the two-class classification system of emotional states (valence and arousal). In the present study, we used time-frequency maps calculated for two electrodes located in the frontal (Fz) and parietal (Pz) regions, combined into one image. This allowed us to capture both the temporal and spatial distributions of EEG power features in various creative states/expected stages. Similar to the recognition of emotional states from the EEG data, the use of CNN for image classification (time-frequency maps) was effective and provided a mean classification accuracy of up to 96% [8.3 SD] in our case.

Wavelet map classification of three and five classes (states) from the painting phases of a professional artist (in “ecological” condition of art studio) reached an accuracy of up to 99.0% in both cases and was higher than the chance level in the time-series classifications, with 75.8% and 56.2%, respectively. Classifications of long-lasting creative states in professional participants were previously attempted [9, 33]. In the study by Sasaki et al. [33], states of creative music performance (guitar improvization by proficient musicians) compared with noncreative task (scales on guitar) were classified with a mean group accuracy of 75.0% (min, 47.6; max, 92.9%).

Based on high classification accuracy, it was suggested that there could be some common “specific EEG patterns” for classification despite individual variations for the two tested models — “STORY” and “viART” — that failed. When the data of one participant was excluded from the common sample sets (“STORY”), or EEG features from one canvas were suggested to be classified by taking for test EEG data from the other painting (“viArt”), the accuracy was at the chance level. However, when a commonly trained set was formed from EEGs of all participants and the test set also included data from all participants, the classification results were also approximately 90%. Thus, it could be a problem of highly organized CNN memory abilities that we have to check in the future.

In contrast to the differentiation of long-lasting creativity states, the implementation of time-frequency maps for single-trial ERP classifications between creative and noncreative cognitive activities was insufficient.

The mean classification accuracy (with ResNet50 as classifier and time-frequency images as features) for the three-class system (“PROVERB” model) was 43.2% [8.8 SD], and 9 of the 22 participants had an accuracy level <40.0% (with 33.3% at the theoretical chance level). The low classification accuracy might be caused by the small number of samples for CNN training, since 104 trials for each class were proposed for the participants to fulfill. For state discrimination by the ERP features, time-series features were mostly applied [4]. In studies with the classification based on ERP features, the brain responses to different stimuli were mostly classified, for example — target or non-target objects in BCI spellers, erroneous stimuli, or face perception [34–36]. In this study, we attempted to distinguish short single-trial time intervals (1500 ms) connected to different mental operations in response to the same stimuli (same set of proverbs). Only a few attempts were made to distinguish creative and noncreative states based on EEG features [7–9, 33]. ML using spectrally weighted CSPs (SpecCSP) algorithm for EEG feature extraction attained a mean of >63.9% classification performance for verbal creative compared with noncreative task performance (alternative use task) [7]. In our previous study, the classification of time-series single-trial data for the PROVERBS model gave a mean group accuracy level of 48.7±5.0% [8], which was higher than the theoretical chance level (33.3% for the three-class system) but required improved accuracy for practical applications. Нere, in addition to converting EEG from referential montage to CSD, CSP was used for the classification of data from 15 electrodes located in the central frontal and parietal regions. Group-averaged decoding accuracy for three states (creating an original ending, finding/naming a synonym, and recalling the ending of a known proverb or saying) was 80.5±8.8 (min, 56.2±8.3; max, 98.1±2.5). Since we decoded creative and cognitive states according to the instructions given to the participants (without taking into account their response), the type of activity (creative/noncreative) at short intervals for finding an answer (1500 ms) already led to a reorganization of brain bioelectrical activity detectible using the ML approach.

CONCLUSION

ML approaches appear to be effectively used for the discrimination of creative and noncreative states and stages of creative activity in both ordinary people and professionals. We implemented wavelet time-frequency image classifications by convolutional neural network (ResNet50 architecture), which achieved a mean classification accuracy of 96.4% for the four-class system (“STORY” creation model) and up to 99% for the three- and five-class system during oil painting by a professional artist (“viART” model). In both cases, a high discriminative strength of convolutional neural network was demonstrated for long-lasting states (several minutes). Nevertheless, these are pilot data that should be further explored to exclude the situation of just convolutional neural network memory for physiological samples.

This high discriminative strength of convolutional neural network for time-frequency maps could be used for continuous cognitive neurointerface in the case of overcoming the barrier of training length, which might be decided through effective pretraining of the convolutional neural network for specific electroencephalography features. The modeled neurointerface usage situation (consequence order of training and testing samples) with lighter convolutional neural network architecture expectably diminished the convolutional neural network classification strength, with an accuracy of approximately 70% [11% SD].

For short-time single-trial creative responses, the CSP-based support vector machine classifier demonstrated greater accuracy, with a mean accuracy of 83%. Thus, creative states and stages of creative activity could be recognized using machine learning methods for the development of cognitive interfaces.

ADDITIONAL INFORMATION

Funding source. This work was supported by the Russian Science Foundation (grant N 22-28-02073).

Competing interests. The authors declare that they have no competing interests.

Authors’ contribution. N.V. Shemyakina — elaboration of the study designs and data analysis, data collection, preprocessing and search of EEG data classification approaches, literature review, article’s text writing; G.S. Velikoborets — data preprocessing and search of ERP data classification approaches, participation in article writing, text editing; Zh.V. Nagornova — data collection, preprocessing and analysis; elaboration and testing classification approaches, writing the text of the article; collection and analysis of literary sources.

All authors confirm that their authorship meets the international ICMJE criteria (all authors have made a significant contribution to the development of the concept, research and preparation of the article, read and approved the final version before publication).

ДОПОЛНИТЕЛЬНО

Источник финансирования. Научное исследование проведено при поддержке Российского научного фонда (грант № 22-28-02073).

Конфликт интересов. Авторы декларируют отсутствие явных и потенциальных конфликтов интересов, связанных с публикацией настоящей статьи.

Вклад авторов. Н.В. Шемякина — разработка концепции, методологии и подходов исследования; проведение исследований, предобработка, анализ и классификация данных, обзор литературы, написание текста и редактирование статьи; Г.С. Великоборец — реализация кода для создания частотно-временн΄ых карт по ЭЭГ/ВП, поиск подходов и классификация times-series для коротких проб (CSP-преобразование); Ж.В. Нагорнова — проведение исследований, анализ и классификация ЭЭГ/ВП-данных, поиск и тестирование подходов к классификации ЭЭГ-данных, написание текста и редактирование статьи.

Все авторы подтверждают соответствие своего авторства международным критериям ICMJE (все авторы внесли существенный вклад в разработку концепции, проведение исследования и подготовку статьи, прочли и одобрили финальную версию перед публикацией).

About the authors

Natalia V. Shemyakina

Sechenov Institute of Evolutionary Physiology and Biochemistry of the Russian Academy of Sciences

Author for correspondence.
Email: shemyakina_n@mail.ru
ORCID iD: 0000-0002-8936-0082
SPIN-code: 2938-4469

Cand. Sci. (Biol.)

Russian Federation, Saint Petersburg

Gleb S. Velicoborets

Sechenov Institute of Evolutionary Physiology and Biochemistry of the Russian Academy of Sciences

Email: velikoborecz90@mail.ru
ORCID iD: 0000-0003-0737-0787
SPIN-code: 9725-8392
Russian Federation, Saint Petersburg

Zhanna V. Nagornova

Sechenov Institute of Evolutionary Physiology and Biochemistry of the Russian Academy of Sciences

Email: nagornova_zh@mail.ru
ORCID iD: 0000-0002-6476-3141
SPIN-code: 4720-3096

Cand. Sci. (Biol.)

Russian Federation, Saint Petersburg

References

Supplementary files