arXiv subjects

Haruhiko Kishima

Publications and source records attributed to Haruhiko Kishima.

8 recordsLinked to original sources

arXiv2026

Visual Semantic Decoding of Electrocorticography from Video Stimuli using End-to-End Deep Learning

ECoG-based visual semantic decoding enables inference of semantic interpretation of visual perception from complex, noisy brain activity. This study examines the feasibility of visual semantic decoding using an end-to-end deep learning framework using electrocorticography (ECoG). Specifically, the decoding task is to predict visual categories from video stimuli using time-series neural inputs. A previously collected ECoG dataset from participants ($n=17$) with drug-resistant epilepsy is used for analysis. With fewer than 50 training samples per visual category, this study evaluates multiple deep learning approaches, artificial neural network architectures, and frequency-band filtered inputs. The best-performing approach is analyzed to shed light on the discriminative information it relies on across spectral, temporal, and cortical dimensions. The selected decoding system uses mixup augmentation, a Transformer-based encoder, and high-gamma (80-150 Hz) inputs with a 900 ms post-stimulus window. Further analysis shows that early visual cortex (V2-V4), ventral stream visual cortex, MT+ complex with neighbouring visual areas, and lateral temporal cortex contributed substantially to decoding performance. This study demonstrates that an end-to-end deep learning framework can yield promising decoding performance from dynamic visual stimuli without handcrafted features, while the model behavior remains interpretable through spectral, temporal, and cortical dimensions, which are broadly consistent with established neuroscience knowledge.

cs.LG↗

arXiv2025

Brain-aligning of semantic vectors improves neural decoding of visual stimuli

The development of algorithms to accurately decode neural information has long been a research focus in the field of neuroscience. Brain decoding typically involves training machine learning models to map neural data onto a preestablished vector representation of stimulus features. These vectors are usually derived from image- and/or text-based feature spaces. Nonetheless, the intrinsic characteristics of these vectors might fundamentally differ from those that are encoded by the brain, limiting the ability of decoders to accurately learn this mapping. To address this issue, we propose a framework, called brain-aligning of semantic vectors, that fine-tunes pretrained feature vectors to better align with the structure of neural representations of visual stimuli in the brain. We trained this model with functional magnetic resonance imaging (fMRI) and then performed zero-shot brain decoding on fMRI, magnetoencephalography (MEG), and electrocorticography (ECoG) data. fMRI-based brain-aligned vectors improved decoding performance across all three neuroimaging datasets when accuracy was determined by calculating the correlation coefficients between true and predicted vectors. Additionally, when decoding accuracy was determined via stimulus identification, this accuracy increased in specific category types; improvements varied depending on the original vector space that was used for brain-alignment, and consistent improvements were observed across all neuroimaging modalities.

q-bio.NC↗

arXiv2025

EvoBrain: Dynamic Multi-Channel EEG Graph Modeling for Time-Evolving Brain Networks

Dynamic GNNs, which integrate temporal and spatial features in Electroencephalography (EEG) data, have shown great potential in automating seizure detection. However, fully capturing the underlying dynamics necessary to represent brain states, such as seizure and non-seizure, remains a non-trivial task and presents two fundamental challenges. First, most existing dynamic GNN methods are built on temporally fixed static graphs, which fail to reflect the evolving nature of brain connectivity during seizure progression. Second, current efforts to jointly model temporal signals and graph structures and, more importantly, their interactions remain nascent, often resulting in inconsistent performance. To address these challenges, we present the first theoretical analysis of these two problems, demonstrating the effectiveness and necessity of explicit dynamic modeling and time-then-graph dynamic GNN method. Building on these insights, we propose EvoBrain, a novel seizure detection model that integrates a two-stream Mamba architecture with a GCN enhanced by Laplacian Positional Encoding, following neurological insights. Moreover, EvoBrain incorporates explicitly dynamic graph structures, allowing both nodes and edges to evolve over time. Our contributions include (a) a theoretical analysis proving the expressivity advantage of explicit dynamic modeling and time-then-graph over other approaches, (b) a novel and efficient model that significantly improves AUROC by 23% and F1 score by 30%, compared with the dynamic GNN baseline, and (c) broad evaluations of our method on the challenging early seizure prediction tasks.

cs.LG↗

arXiv2025

Intrinsic frequency distribution characterises neural dynamics

Decomposing multivariate time series with certain basic dynamics is crucial for understanding, predicting and controlling nonlinear spatiotemporally dynamic systems such as the brain. Dynamic mode decomposition (DMD) is a method for decomposing nonlinear spatiotemporal dynamics into several basic dynamics (dynamic modes; DMs) with intrinsic frequencies and decay rates. In particular, unlike Fourier transform-based methods, which are used to decompose a single-channel signal into the amplitudes of sinusoidal waves with discrete frequencies at a regular interval, DMD can derive the intrinsic frequencies of a multichannel signal on the basis of the available data; furthermore, it can capture nonstationary components such as alternations between states with different intrinsic frequencies. Here, we propose the use of the distribution of intrinsic frequencies derived from DMDs (DM frequencies) to characterise neural activities. The distributions of DM frequencies in the electroencephalograms of healthy subjects and patients with dementia or Parkinson's disease in a resting state were evaluated. By using the distributions, these patients were distinguished from healthy subjects with significantly greater accuracy than when using amplitude spectra derived by discrete Fourier transform. This finding suggests that the distribution of DM frequencies exhibits distinct behaviour from amplitude spectra, and therefore, the distribution may serve as a new biomarker by characterising the nonlinear spatiotemporal dynamics of electrophysiological signals.

eess.SP↗

arXiv2025

Wirelessly transmitted subthalamic nucleus signals predict endogenous pain levels in Parkinson's disease patients

Parkinson disease (PD) patients experience pain fluctuations that significantly reduce their quality of life. Despite the vast knowledge of the subthalamic nucleus (STN) role in PD, the STN biomarkers for pain fluctuations and the relationship between bilateral subthalamic nucleus (STN) activities and pain occurrence are still less understood. This observational study used data-driven methods by collecting annotated pain followed by a series of corresponding binary pain ratings and wirelessly transmitted STN signals, then leveraging the explainable machine learning algorithm to predict binary pain levels and sort the feature influence. The binary pain levels could be predicted among annotated pain reports corresponding to PD-related pain characteristics. The STN activity from both sides could impact pain prediction, with gamma and beta bands in the contralateral STN and delta and theta bands in the ipsilateral STN showing a prominent role. This study emphasizes the role of bilateral STN biomarkers on endogenous pain fluctuations.

q-bio.NC↗

arXiv2024

SplitSEE: A Splittable Self-supervised Framework for Single-Channel EEG Representation Learning

While end-to-end multi-channel electroencephalography (EEG) learning approaches have shown significant promise, their applicability is often constrained in neurological diagnostics, such as intracranial EEG resources. When provided with a single-channel EEG, how can we learn representations that are robust to multi-channels and scalable across varied tasks, such as seizure prediction? In this paper, we present SplitSEE, a structurally splittable framework designed for effective temporal-frequency representation learning in single-channel EEG. The key concept of SplitSEE is a self-supervised framework incorporating a deep clustering task. Given an EEG, we argue that the time and frequency domains are two distinct perspectives, and hence, learned representations should share the same cluster assignment. To this end, we first propose two domain-specific modules that independently learn domain-specific representation and address the temporal-frequency tradeoff issue in conventional spectrogram-based methods. Then, we introduce a novel clustering loss to measure the information similarity. This encourages representations from both domains to coherently describe the same input by assigning them a consistent cluster. SplitSEE leverages a pre-training-to-fine-tuning framework within a splittable architecture and has following properties: (a) Effectiveness: it learns representations solely from single-channel EEG but has even outperformed multi-channel baselines. (b) Robustness: it shows the capacity to adapt across different channels with low performance variance. Superior performance is also achieved with our collected clinical dataset. (c) Scalability: With just one fine-tuning epoch, SplitSEE achieves high and stable performance using partial model layers.

cs.LG↗

arXiv2023

Fast, accurate, and interpretable decoding of electrocorticographic signals using dynamic mode decomposition

Dynamic mode (DM) decomposition decomposes spatiotemporal signals into basic oscillatory components (DMs). DMs can improve the accuracy of neural decoding when used with the nonlinear Grassmann kernel, compared to conventional power features. However, such kernel-based machine learning algorithms have three limitations: large computational time preventing real-time application, incompatibility with non-kernel algorithms, and low interpretability. Here, we propose a mapping function corresponding to the Grassmann kernel that explicitly transforms DMs into spatial DM (sDM) features, which can be used in any machine learning algorithm. Using electrocorticographic signals recorded during various movement and visual perception tasks, the sDM features were shown to improve the decoding accuracy and computational time compared to conventional methods. Furthermore, the components of the sDM features informative for decoding showed similar characteristics to the high-$γ$ power of the signals, but with higher trial-to-trial reproducibility. The proposed sDM features enable fast, accurate, and interpretable neural decoding.

eess.SP↗

arXiv2021

Voluntary control of semantic neural representations by imagery with conflicting visual stimulation

Neural representations of visual perception are affected by mental imagery and attention. Although attention is known to modulate neural representations, it is unknown how imagery changes neural representations when imagined and perceived images semantically conflict. We hypothesized that imagining an image would activate a neural representation during its perception even while watching a conflicting image. To test this hypothesis, we developed a closed-loop system to show images inferred from electrocorticograms using a visual semantic space. The successful control of the feedback images demonstrated that the semantic vector inferred from electrocorticograms became closer to the vector of the imagined category, even while watching images from different categories. Moreover, modulation of the inferred vectors by mental imagery depended on both the image category and time from the initiation of imagery. The closed-loop control of the semantic vectors revealed an asymmetrical interaction between visual perception and imagery.

q-bio.NC↗