arXiv subjects

Seong-Whan Lee

Publications and source records attributed to Seong-Whan Lee.

At least 163 records · Page 9Linked to original sources

arXiv2021

Voice of Your Brain: Cognitive Representations of Imagined Speech,Overt Speech, and Speech Perception Based on EEG

Every people has their own voice, likewise, brain signals dis-play distinct neural representations for each individual. Al-though recent studies have revealed the robustness of speech-related paradigms for efficient brain-computer interface, the dis-tinction on their cognitive representations with practical usabil-ity still remains to be discovered. Herein, we investigate the dis-tinct brain patterns from electroencephalography (EEG) duringimagined speech, overt speech, and speech perception in termsof subject variations with its practical use of speaker identifica-tion from single channel EEG. We performed classification ofnine subjects using deep neural network that captures temporal-spectral-spatial features from EEG of imagined speech, overtspeech, and speech perception. Furthermore, we demonstratedthe underlying neural features of individual subjects while per-forming imagined speech by comparing the functional connec-tivity and the EEG envelope features. Our results demonstratethe possibility of subject identification from single channel EEGof imagined speech and overt speech. Also, the comparison ofthe three speech-related paradigms will provide valuable infor-mation for the practical use of speech-related brain signals inthe further studies.

cs.HC↗

arXiv2021

FBCNet: A Multi-view Convolutional Neural Network for Brain-Computer Interface

Lack of adequate training samples and noisy high-dimensional features are key challenges faced by Motor Imagery (MI) decoding algorithms for electroencephalogram (EEG) based Brain-Computer Interface (BCI). To address these challenges, inspired from neuro-physiological signatures of MI, this paper proposes a novel Filter-Bank Convolutional Network (FBCNet) for MI classification. FBCNet employs a multi-view data representation followed by spatial filtering to extract spectro-spatially discriminative features. This multistage approach enables efficient training of the network even when limited training data is available. More significantly, in FBCNet, we propose a novel Variance layer that effectively aggregates the EEG time-domain information. With this design, we compare FBCNet with state-of-the-art (SOTA) BCI algorithm on four MI datasets: The BCI competition IV dataset 2a (BCIC-IV-2a), the OpenBMI dataset, and two large datasets from chronic stroke patients. The results show that, by achieving 76.20% 4-class classification accuracy, FBCNet sets a new SOTA for BCIC-IV-2a dataset. On the other three datasets, FBCNet yields up to 8% higher binary classification accuracies. Additionally, using explainable AI techniques we present one of the first reports about the differences in discriminative EEG features between healthy subjects and stroke patients. Also, the FBCNet source code is available at https://github.com/ravikiran-mane/FBCNet.

cs.OH↗

arXiv2021

Decoding Event-related Potential from Ear-EEG Signals based on Ensemble Convolutional Neural Networks in Ambulatory Environment

Recently, practical brain-computer interface is actively carried out, especially, in an ambulatory environment. However, the electroencephalography (EEG) signals are distorted by movement artifacts and electromyography signals when users are moving, which make hard to recognize human intention. In addition, as hardware issues are also challenging, ear-EEG has been developed for practical brain-computer interface and has been widely used. In this paper, we proposed ensemble-based convolutional neural networks in ambulatory environment and analyzed the visual event-related potential responses in scalp- and ear-EEG in terms of statistical analysis and brain-computer interface performance. The brain-computer interface performance deteriorated as 3-14% when walking fast at 1.6 m/s. The proposed methods showed 0.728 in average of the area under the curve. The proposed method shows robust to the ambulatory environment and imbalanced data as well.

cs.HC↗

arXiv2021

Weakly Supervised Thoracic Disease Localization via Disease Masks

To enable a deep learning-based system to be used in the medical domain as a computer-aided diagnosis system, it is essential to not only classify diseases but also present the locations of the diseases. However, collecting instance-level annotations for various thoracic diseases is expensive. Therefore, weakly supervised localization methods have been proposed that use only image-level annotation. While the previous methods presented the disease location as the most discriminative part for classification, this causes a deep network to localize wrong areas for indistinguishable X-ray images. To solve this issue, we propose a spatial attention method using disease masks that describe the areas where diseases mainly occur. We then apply the spatial attention to find the precise disease area by highlighting the highest probability of disease occurrence. Meanwhile, the various sizes, rotations and noise in chest X-ray images make generating the disease masks challenging. To reduce the variation among images, we employ an alignment module to transform an input X-ray image into a generalized image. Through extensive experiments on the NIH-Chest X-ray dataset with eight kinds of diseases, we show that the proposed method results in superior localization performances compared to state-of-the-art methods.

cs.CV↗

arXiv2021

Human Interaction Recognition Framework based on Interacting Body Part Attention

Human activity recognition in videos has been widely studied and has recently gained significant advances with deep learning approaches; however, it remains a challenging task. In this paper, we propose a novel framework that simultaneously considers both implicit and explicit representations of human interactions by fusing information of local image where the interaction actively occurred, primitive motion with the posture of individual subject's body parts, and the co-occurrence of overall appearance change. Human interactions change, depending on how the body parts of each human interact with the other. The proposed method captures the subtle difference between different interactions using interacting body part attention. Semantically important body parts that interact with other objects are given more weight during feature representation. The combined feature of interacting body part attention-based individual representation and the co-occurrence descriptor of the full-body appearance change is fed into long short-term memory to model the temporal dynamics over time in a single framework. We validate the effectiveness of the proposed method using four widely used public datasets by outperforming the competing state-of-the-art method.

cs.CV↗

arXiv2021

Visual Question Answering based on Local-Scene-Aware Referring Expression Generation

Visual question answering requires a deep understanding of both images and natural language. However, most methods mainly focus on visual concept; such as the relationships between various objects. The limited use of object categories combined with their relationships or simple question embedding is insufficient for representing complex scenes and explaining decisions. To address this limitation, we propose the use of text expressions generated for images, because such expressions have few structural constraints and can provide richer descriptions of images. The generated expressions can be incorporated with visual features and question embedding to obtain the question-relevant answer. A joint-embedding multi-head attention network is also proposed to model three different information modalities with co-attention. We quantitatively and qualitatively evaluated the proposed method on the VQA v2 dataset and compared it with state-of-the-art methods in terms of answer prediction. The quality of the generated expressions was also evaluated on the RefCOCO, RefCOCO+, and RefCOCOg datasets. Experimental results demonstrate the effectiveness of the proposed method and reveal that it outperformed all of the competing methods in terms of both quantitative and qualitative results.

cs.CV↗

arXiv2020

Towards Neurohaptics: Brain-Computer Interfaces for Decoding Intuitive Sense of Touch

Noninvasive brain-computer interface (BCI) is widely used to recognize users' intentions. Especially, BCI related to tactile and sensation decoding could provide various effects on many industrial fields such as manufacturing advanced touch displays, controlling robotic devices, and more immersive virtual reality or augmented reality. In this paper, we introduce haptic and sensory perception-based BCI systems called neurohaptics. It is a preliminary study for a variety of scenarios using actual touch and touch imagery paradigms. We designed a novel experimental environment and a device that could acquire brain signals under touching designated materials to generate natural touch and texture sensations. Through the experiment, we collected the electroencephalogram (EEG) signals with respect to four different texture objects. Seven subjects were recruited for the experiment and evaluated classification performances using machine learning and deep learning approaches. Hence, we could confirm the feasibility of decoding actual touch and touch imagery on EEG signals to develop practical neurohaptics.

cs.HC↗

arXiv2020

Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesis

While generative adversarial networks (GANs) based neural text-to-speech (TTS) systems have shown significant improvement in neural speech synthesis, there is no TTS system to learn to synthesize speech from text sequences with only adversarial feedback. Because adversarial feedback alone is not sufficient to train the generator, current models still require the reconstruction loss compared with the ground-truth and the generated mel-spectrogram directly. In this paper, we present Multi-SpectroGAN (MSG), which can train the multi-speaker model with only the adversarial feedback by conditioning a self-supervised hidden representation of the generator to a conditional discriminator. This leads to better guidance for generator training. Moreover, we also propose adversarial style combination (ASC) for better generalization in the unseen speaking style and transcript, which can learn latent representations of the combined style embedding from multiple mel-spectrograms. Trained with ASC and feature matching, the MSG synthesizes a high-diversity mel-spectrogram by controlling and mixing the individual speaking styles (e.g., duration, pitch, and energy). The result shows that the MSG synthesizes a high-fidelity mel-spectrogram, which has almost the same naturalness MOS score as the ground-truth mel-spectrogram.

eess.AS↗

arXiv2020

Design of an EEG-based Drone Swarm Control System using Endogenous BCI Paradigms

Non-invasive brain-computer interface (BCI) has been developed for understanding users' intentions by using electroencephalogram (EEG) signals. With the recent development of artificial intelligence, there have been many developments in the drone control system. BCI characteristic that can reflect the users' intentions led to the BCI-based drone control system. When using drone swarm, we can have more advantages, such as mission diversity, than using a single drone. In particular, BCI-based drone swarm control could provide many advantages to various industries such as military service or industry disaster. BCI Paradigms consist of the exogenous and endogenous paradigms. The endogenous paradigms can operate with the users' intentions independently of any stimulus. In this study, we designed endogenous paradigms (i.e., motor imagery (MI), visual imagery (VI), and speech imagery (SI)) specialized in drone swarm control, and EEG-based various task classifications related to drone swarm control were conducted. Five subjects participated in the experiment and the performance was evaluated using the basic machine learning algorithm. The grand-averaged accuracies were 51.1%, 53.2%, and 41.9% in MI, VI, and SI, respectively. Hence, we confirmed the feasibility of increasing the degree of freedom for drone swarm control using various endogenous paradigms.

cs.HC↗

arXiv2020

Interpreting Deep Neural Networks with Relative Sectional Propagation by Analyzing Comparative Gradients and Hostile Activations

The clear transparency of Deep Neural Networks (DNNs) is hampered by complex internal structures and nonlinear transformations along deep hierarchies. In this paper, we propose a new attribution method, Relative Sectional Propagation (RSP), for fully decomposing the output predictions with the characteristics of class-discriminative attributions and clear objectness. We carefully revisit some shortcomings of backpropagation-based attribution methods, which are trade-off relations in decomposing DNNs. We define hostile factor as an element that interferes with finding the attributions of the target and propagate it in a distinguishable way to overcome the non-suppressed nature of activated neurons. As a result, it is possible to assign the bi-polar relevance scores of the target (positive) and hostile (negative) attributions while maintaining each attribution aligned with the importance. We also present the purging techniques to prevent the decrement of the gap between the relevance scores of the target and hostile attributions during backward propagation by eliminating the conflicting units to channel attribution map. Therefore, our method makes it possible to decompose the predictions of DNNs with clearer class-discriminativeness and detailed elucidations of activation neurons compared to the conventional attribution methods. In a verified experimental environment, we report the results of the assessments: (i) Pointing Game, (ii) mIoU, and (iii) Model Sensitivity with PASCAL VOC 2007, MS COCO 2014, and ImageNet datasets. The results demonstrate that our method outperforms existing backward decomposition methods, including distinctive and intuitive visualizations.

cs.CV↗

arXiv2020

Functional Connectivity of Imagined Speech and Visual Imagery based on Spectral Dynamics

Recent advances in brain-computer interface technology have shown the potential of imagined speech and visual imagery as a robust paradigm for intuitive brain-computer interface communication. However, the internal dynamics of the two paradigms along with their intrinsic features haven't been revealed. In this paper, we investigated the functional connectivity of the two paradigms, considering various frequency ranges. The dataset of sixteen subjects performing thirteen-class imagined speech and visual imagery were used for the analysis. The phase-locking value of imagined speech and visual imagery was analyzed in seven cortical regions with four frequency ranges. We compared the functional connectivity of imagined speech and visual imagery with the resting state to investigate the brain alterations during the imagery. The phase-locking value in the whole brain region exhibited a significant decrease during both imagined speech and visual imagery. Broca and Wernicke's area along with the auditory cortex mainly exhibited a significant decrease in the imagined speech, and the prefrontal cortex and the auditory cortex have shown a significant decrease in the visual imagery paradigm. Further investigation on the brain connectivity along with the decoding performance of the two paradigms may play a crucial role as a performance predictor.

cs.HC↗

arXiv2020

Rotation Invariant Aerial Image Retrieval with Group Convolutional Metric Learning

Remote sensing image retrieval (RSIR) is the process of ranking database images depending on the degree of similarity compared to the query image. As the complexity of RSIR increases due to the diversity in shooting range, angle, and location of remote sensors, there is an increasing demand for methods to address these issues and improve retrieval performance. In this work, we introduce a novel method for retrieving aerial images by merging group convolution with attention mechanism and metric learning, resulting in robustness to rotational variations. For refinement and emphasis on important features, we applied channel attention in each group convolution stage. By utilizing the characteristics of group convolution and channel-wise attention, it is possible to acknowledge the equality among rotated but identically located images. The training procedure has two main steps: (i) training the network with Aerial Image Dataset (AID) for classification, (ii) fine-tuning the network with triplet-loss for retrieval with Google Earth South Korea and NWPU-RESISC45 datasets. Results show that the proposed method performance exceeds other state-of-the-art retrieval methods in both rotated and original environments. Furthermore, we utilize class activation maps (CAM) to visualize the distinct difference of main features between our method and baseline, resulting in better adaptability in rotated environments.

cs.CV↗

arXiv2020

Classification of Imagined Speech Using Siamese Neural Network

Imagined speech is spotlighted as a new trend in the brain-machine interface due to its application as an intuitive communication tool. However, previous studies have shown low classification performance, therefore its use in real-life is not feasible. In addition, no suitable method to analyze it has been found. Recently, deep learning algorithms have been applied to this paradigm. However, due to the small amount of data, the increase in classification performance is limited. To tackle these issues, in this study, we proposed an end-to-end framework using Siamese neural network encoder, which learns the discriminant features by considering the distance between classes. The imagined words (e.g., arriba (up), abajo (down), derecha (right), izquierda (left), adelante (forward), and atrás (backward)) were classified using the raw electroencephalography (EEG) signals. We obtained a 6-class classification accuracy of 31.40% for imagined speech, which significantly outperformed other methods. This was possible because the Siamese neural network, which increases the distance between dissimilar samples while decreasing the distance between similar samples, was used. In this regard, our method can learn discriminant features from a small dataset. The proposed framework would help to increase the classification performance of imagined speech for a small amount of data and implement an intuitive communication system.

eess.SP↗

arXiv2020

Decoding Visual Recognition of Objects from EEG Signals based on Attention-Driven Convolutional Neural Network

The ability to perceive and recognize objects is fundamental for the interaction with the external environment. Studies that investigate them and their relationship with brain activity changes have been increasing due to the possible application in an intuitive brain-machine interface (BMI). In addition, the distinctive patterns when presenting different visual stimuli that make data differentiable enough to be classified have been studied. However, reported classification accuracy still low or employed techniques for obtaining brain signals are impractical to use in real environments. In this study, we aim to decode electroencephalography (EEG) signals depending on the provided visual stimulus. Subjects were presented with 72 photographs belonging to 6 different semantic categories. We classified 6 categories and 72 exemplars according to visual stimuli using EEG signals. In order to achieve a high classification accuracy, we proposed an attention driven convolutional neural network and compared our results with conventional methods used for classifying EEG signals. We reported an accuracy of 50.37% and 26.75% for 6-class and 72-class, respectively. These results statistically outperformed other conventional methods. This was possible because of the application of the attention network using human visual pathways. Our findings showed that EEG signals are possible to differentiate when subjects are presented with visual stimulus of different semantic categories and at an exemplar-level with a high classification accuracy; this demonstrates its viability to be applied it in a real-world BMI.

eess.SP↗

arXiv2020

Few-Shot Object Detection via Knowledge Transfer

Conventional methods for object detection usually require substantial amounts of training data and annotated bounding boxes. If there are only a few training data and annotations, the object detectors easily overfit and fail to generalize. It exposes the practical weakness of the object detectors. On the other hand, human can easily master new reasoning rules with only a few demonstrations using previously learned knowledge. In this paper, we introduce a few-shot object detection via knowledge transfer, which aims to detect objects from a few training examples. Central to our method is prototypical knowledge transfer with an attached meta-learner. The meta-learner takes support set images that include the few examples of the novel categories and base categories, and predicts prototypes that represent each category as a vector. Then, the prototypes reweight each RoI (Region-of-Interest) feature vector from a query image to remodels R-CNN predictor heads. To facilitate the remodeling process, we predict the prototypes under a graph structure, which propagates information of the correlated base categories to the novel categories with explicit guidance of prior knowledge that represents correlations among categories. Extensive experiments on the PASCAL VOC dataset verifies the effectiveness of the proposed method.

cs.CV↗

arXiv2020

Audio Dequantization for High Fidelity Audio Generation in Flow-based Neural Vocoder

In recent works, a flow-based neural vocoder has shown significant improvement in real-time speech generation task. The sequence of invertible flow operations allows the model to convert samples from simple distribution to audio samples. However, training a continuous density model on discrete audio data can degrade model performance due to the topological difference between latent and actual distribution. To resolve this problem, we propose audio dequantization methods in flow-based neural vocoder for high fidelity audio generation. Data dequantization is a well-known method in image generation but has not yet been studied in the audio domain. For this reason, we implement various audio dequantization methods in flow-based neural vocoder and investigate the effect on the generated audio. We conduct various objective performance assessments and subjective evaluation to show that audio dequantization can improve audio generation quality. From our experiments, using audio dequantization produces waveform audio with better harmonic structure and fewer digital artifacts.

eess.AS↗

arXiv2020

Self-Augmentation: Generalizing Deep Networks to Unseen Classes for Few-Shot Learning

Few-shot learning aims to classify unseen classes with a few training examples. While recent works have shown that standard mini-batch training with a carefully designed training strategy can improve generalization ability for unseen classes, well-known problems in deep networks such as memorizing training statistics have been less explored for few-shot learning. To tackle this issue, we propose self-augmentation that consolidates self-mix and self-distillation. Specifically, we exploit a regional dropout technique called self-mix, in which a patch of an image is substituted into other values in the same image. Then, we employ a backbone network that has auxiliary branches with its own classifier to enforce knowledge sharing. Lastly, we present a local representation learner to further exploit a few training examples for unseen classes. Experimental results show that the proposed method outperforms the state-of-the-art methods for prevalent few-shot benchmarks and improves the generalization ability.

cs.LG↗

arXiv2020

Mel-spectrogram augmentation for sequence to sequence voice conversion

For training the sequence-to-sequence voice conversion model, we need to handle an issue of insufficient data about the number of speech pairs which consist of the same utterance. This study experimentally investigated the effects of Mel-spectrogram augmentation on training the sequence-to-sequence voice conversion (VC) model from scratch. For Mel-spectrogram augmentation, we adopted the policies proposed in SpecAugment. In addition, we proposed new policies (i.e., frequency warping, loudness and time length control) for more data variations. Moreover, to find the appropriate hyperparameters of augmentation policies without training the VC model, we proposed hyperparameter search strategy and the new metric for reducing experimental cost, namely deformation per deteriorating ratio. We compared the effect of these Mel-spectrogram augmentation methods based on various sizes of training set and augmentation policies. In the experimental results, the time axis warping based policies (i.e., time length control and time warping.) showed better performance than other policies. These results indicate that the use of the Mel-spectrogram augmentation is more beneficial for training the VC model.

cs.LG↗