arXiv subjects

Sungwon Kim

Publications and source records attributed to Sungwon Kim.

At least 37 records · Page 2Linked to original sources

arXiv2024

DSLR: Diversity Enhancement and Structure Learning for Rehearsal-based Graph Continual Learning

We investigate the replay buffer in rehearsal-based approaches for graph continual learning (GCL) methods. Existing rehearsal-based GCL methods select the most representative nodes for each class and store them in a replay buffer for later use in training subsequent tasks. However, we discovered that considering only the class representativeness of each replayed node makes the replayed nodes to be concentrated around the center of each class, incurring a potential risk of overfitting to nodes residing in those regions, which aggravates catastrophic forgetting. Moreover, as the rehearsal-based approach heavily relies on a few replayed nodes to retain knowledge obtained from previous tasks, involving the replayed nodes that have irrelevant neighbors in the model training may have a significant detrimental impact on model performance. In this paper, we propose a GCL model named DSLR, specifically, we devise a coverage-based diversity (CD) approach to consider both the class representativeness and the diversity within each class of the replayed nodes. Moreover, we adopt graph structure learning (GSL) to ensure that the replayed nodes are connected to truly informative neighbors. Extensive experimental results demonstrate the effectiveness and efficiency of DSLR. Our source code is available at https://github.com/seungyoon-Choi/DSLR_official.

cs.LG↗

arXiv2024

Interpretable Prototype-based Graph Information Bottleneck

The success of Graph Neural Networks (GNNs) has led to a need for understanding their decision-making process and providing explanations for their predictions, which has given rise to explainable AI (XAI) that offers transparent explanations for black-box models. Recently, the use of prototypes has successfully improved the explainability of models by learning prototypes to imply training graphs that affect the prediction. However, these approaches tend to provide prototypes with excessive information from the entire graph, leading to the exclusion of key substructures or the inclusion of irrelevant substructures, which can limit both the interpretability and the performance of the model in downstream tasks. In this work, we propose a novel framework of explainable GNNs, called interpretable Prototype-based Graph Information Bottleneck (PGIB) that incorporates prototype learning within the information bottleneck framework to provide prototypes with the key subgraph from the input graph that is important for the model prediction. This is the first work that incorporates prototype learning into the process of identifying the key subgraphs that have a critical impact on the prediction performance. Extensive experiments, including qualitative analysis, demonstrate that PGIB outperforms state-of-the-art methods in terms of both prediction performance and explainability.

cs.LG↗

arXiv2024

Scaling NVIDIA's Multi-speaker Multi-lingual TTS Systems with Zero-Shot TTS to Indic Languages

In this paper, we describe the TTS models developed by NVIDIA for the MMITS-VC (Multi-speaker, Multi-lingual Indic TTS with Voice Cloning) 2024 Challenge. In Tracks 1 and 2, we utilize RAD-MMM to perform few-shot TTS by training additionally on 5 minutes of target speaker data. In Track 3, we utilize P-Flow to perform zero-shot TTS by training on the challenge dataset as well as external datasets. We use HiFi-GAN vocoders for all submissions. RAD-MMM performs competitively on Tracks 1 and 2, while P-Flow ranks first on Track 3, with mean opinion score (MOS) 4.4 and speaker similarity score (SMOS) of 3.62.

cs.SD↗

arXiv2023

Density of States Prediction of Crystalline Materials via Prompt-guided Multi-Modal Transformer

The density of states (DOS) is a spectral property of crystalline materials, which provides fundamental insights into various characteristics of the materials. While previous works mainly focus on obtaining high-quality representations of crystalline materials for DOS prediction, we focus on predicting the DOS from the obtained representations by reflecting the nature of DOS: DOS determines the general distribution of states as a function of energy. That is, DOS is not solely determined by the crystalline material but also by the energy levels, which has been neglected in previous works. In this paper, we propose to integrate heterogeneous information obtained from the crystalline materials and the energies via a multi-modal transformer, thereby modeling the complex relationships between the atoms in the crystalline materials and various energy levels for DOS prediction. Moreover, we propose to utilize prompts to guide the model to learn the crystal structural system-specific interactions between crystalline materials and energies. Extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, with various real-world scenarios demonstrate the superiority of DOSTransformer. The source code for DOSTransformer is available at https://github.com/HeewoongNoh/DOSTransformer.

cond-mat.mtrl-sci↗

arXiv2023

Simultaneous Bright- and Dark-Field X-ray Microscopy at X-ray Free Electron Lasers

The structures, strain fields, and defect distributions in solid materials underlie the mechanical and physical properties across numerous applications. Many modern microstructural microscopy tools characterize crystal grains, domains and defects required to map lattice distortions or deformation, but are limited to studies of the (near) surface. Generally speaking, such tools cannot probe the structural dynamics in a way that is representative of bulk behavior. Synchrotron X-ray diffraction based imaging has long mapped the deeply embedded structural elements, and with enhanced resolution, Dark Field X-ray Microscopy (DFXM) can now map those features with the requisite nm-resolution. However, these techniques still suffer from the required integration times due to limitations from the source and optics. This work extends DFXM to X-ray free electron lasers, showing how the $10^{12}$ photons per pulse available at these sources offer structural characterization down to 100 fs resolution (orders of magnitude faster than current synchrotron images). We introduce the XFEL DFXM setup with simultaneous bright field microscopy to probe density changes within the same volume. This work presents a comprehensive guide to the multi-modal ultrafast high-resolution X-ray microscope that we constructed and tested at two XFELs, and shows initial data demonstrating two timing strategies to study associated reversible or irreversible lattice dynamics.

cond-mat.mtrl-sci↗

arXiv2023

Conditional Graph Information Bottleneck for Molecular Relational Learning

Molecular relational learning, whose goal is to learn the interaction behavior between molecular pairs, got a surge of interest in molecular sciences due to its wide range of applications. Recently, graph neural networks have recently shown great success in molecular relational learning by modeling a molecule as a graph structure, and considering atom-level interactions between two molecules. Despite their success, existing molecular relational learning methods tend to overlook the nature of chemistry, i.e., a chemical compound is composed of multiple substructures such as functional groups that cause distinctive chemical reactions. In this work, we propose a novel relational learning framework, called CGIB, that predicts the interaction behavior between a pair of graphs by detecting core subgraphs therein. The main idea is, given a pair of graphs, to find a subgraph from a graph that contains the minimal sufficient information regarding the task at hand conditioned on the paired graph based on the principle of conditional graph information bottleneck. We argue that our proposed method mimics the nature of chemical reactions, i.e., the core substructure of a molecule varies depending on which other molecule it interacts with. Extensive experiments on various tasks with real-world datasets demonstrate the superiority of CGIB over state-of-the-art baselines. Our code is available at https://github.com/Namkyeong/CGIB.

q-bio.MN↗

arXiv2023

UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Data

We propose UnitSpeech, a speaker-adaptive speech synthesis method that fine-tunes a diffusion-based text-to-speech (TTS) model using minimal untranscribed data. To achieve this, we use the self-supervised unit representation as a pseudo transcript and integrate the unit encoder into the pre-trained TTS model. We train the unit encoder to provide speech content to the diffusion-based decoder and then fine-tune the decoder for speaker adaptation to the reference speaker using a single $<$unit, speech$>$ pair. UnitSpeech performs speech synthesis tasks such as TTS and voice conversion (VC) in a personalized manner without requiring model re-training for each task. UnitSpeech achieves comparable and superior results on personalized TTS and any-to-any VC tasks compared to previous baselines. Our model also shows widespread adaptive performance on real-world data and other tasks that use a unit sequence as input.

cs.SD↗

arXiv2023

Task-Equivariant Graph Few-shot Learning

Although Graph Neural Networks (GNNs) have been successful in node classification tasks, their performance heavily relies on the availability of a sufficient number of labeled nodes per class. In real-world situations, not all classes have many labeled nodes and there may be instances where the model needs to classify new classes, making manual labeling difficult. To solve this problem, it is important for GNNs to be able to classify nodes with a limited number of labeled nodes, known as few-shot node classification. Previous episodic meta-learning based methods have demonstrated success in few-shot node classification, but our findings suggest that optimal performance can only be achieved with a substantial amount of diverse training meta-tasks. To address this challenge of meta-learning based few-shot learning (FSL), we propose a new approach, the Task-Equivariant Graph few-shot learning (TEG) framework. Our TEG framework enables the model to learn transferable task-adaptation strategies using a limited number of training meta-tasks, allowing it to acquire meta-knowledge for a wide range of meta-tasks. By incorporating equivariant neural networks, TEG can utilize their strong generalization abilities to learn highly adaptable task-specific strategies. As a result, TEG achieves state-of-the-art performance with limited training meta-tasks. Our experiments on various benchmark datasets demonstrate TEG's superiority in terms of accuracy and generalization ability, even when using minimal meta-training data, highlighting the effectiveness of our proposed approach in addressing the challenges of meta-learning based few-shot node classification. Our code is available at the following link: https://github.com/sung-won-kim/TEG

cs.LG↗

arXiv2023

Large-Scale Deep Learning for Multi-Jet Event Classification

We report the largest scale deep learning with High Performance Computing (HPC) to physics analysis with the CMS simulation data in proton-proton collisions at 13 TeV. We build a Convolutional Neural Network (CNN) model that takes low-level information as images considering the geometry of the CMS detector and use this model to discriminate \textit{R}-parity violating super symmetry (RPV SUSY) events from the background events with inelastic quantum process from the Standard Model (QCD multi-jet). We compare the classification performance of the CNN method with that of the widely used cut-based method. The signal efficiency (and expected significance) of the CNN method is 1.85 (1.2) times higher than that of the cut-based method. To speed-up the training, the model training is conducted using the Nurion HPC system at the Korea Institute of Science and Technology Information, which is equipped with thousands of parallel \texttt{Xeon Phi} CPUs. Notably, our CNN model shows scalability up to 1024 nodes.

hep-ex↗

arXiv2023

Predicting Density of States via Multi-modal Transformer

The density of states (DOS) is a spectral property of materials, which provides fundamental insights on various characteristics of materials. In this paper, we propose a model to predict the DOS by reflecting the nature of DOS: DOS determines the general distribution of states as a function of energy. Specifically, we integrate the heterogeneous information obtained from the crystal structure and the energies via multi-modal transformer, thereby modeling the complex relationships between the atoms in the crystal structure, and various energy levels. Extensive experiments on two types of DOS, i.e., Phonon DOS and Electron DOS, with various real-world scenarios demonstrate the superiority of DOSTransformer. The source code for DOSTransformer is available at https://github.com/HeewoongNoh/DOSTransformer.

cs.LG↗

arXiv2023

Ultrafast X-ray imaging of the light-induced phase transition in VO2

Using light to control transient phases in quantum materials is an emerging route to engineer new properties and functionality, with both thermal and non-thermal phases observed out of equilibrium. Transient phases are expected to be heterogeneous, either through photo-generated domain growth or by generating topological defects, and this impacts the dynamics of the system. However, this nanoscale heterogeneity has not been directly observed. Here we use time- and spectrally resolved coherent X-ray imaging to track the prototypical light induced insulator-to-metal phase transition in vanadium dioxide on the nanoscale with femtosecond time resolution. We show that the early-time dynamics are independent of the initial spatial heterogeneity and observe a 200 fs switch to the metallic phase. A heterogeneous response emerges only after hundreds of picoseconds. Through spectroscopic imaging, we reveal that the transient metallic phase is a highly orthorhombically strained rutile metallic phase, an interpretation that is in contrast to those based on spatially averaged probes. Our results demonstrate the critical importance of spatially and spectrally resolved measurements for understanding and interpreting the transient phases of quantum materials.

cond-mat.str-el↗

arXiv2022

Strain and Crystallographic Identification of the Helically Concaved Surfaces of Nanoparticles

Identifying the three-dimensional (3D) crystal-plane and strain-field distributions of nanocrystals is essential for optical, catalytic, and electronic applications. Here, we developed a methodology for visualizing the 3D information of chiral gold nanoparticles with concave gap structures by Bragg coherent X-ray diffraction imaging. The distribution of the high-Miller-index planes constituting the concave chiral gap was precisely determined. The highly strained region adjacent to the chiral gaps was resolved, which was correlated to the 432-symmetric morphology of the nanoparticles and its corresponding plasmonic properties were numerically predicted from the atomically defined structures. This approach can serve as a general characterization platform for visualizing the 3D crystallographic and strain distributions of nanoparticles, especially for applications where structural complexity and local heterogeneity are major determinants, as exemplified in plasmonics.

cond-mat.mtrl-sci↗

arXiv2022

Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance

We propose Guided-TTS, a high-quality text-to-speech (TTS) model that does not require any transcript of target speaker using classifier guidance. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier for classifier guidance. Our unconditional diffusion model learns to generate speech without any context from untranscribed speech data. For TTS synthesis, we guide the generative process of the diffusion model with a phoneme classifier trained on a large-scale speech recognition dataset. We present a norm-based scaling method that reduces the pronunciation errors of classifier guidance in Guided-TTS. We show that Guided-TTS achieves a performance comparable to that of the state-of-the-art TTS model, Grad-TTS, without any transcript for LJSpeech. We further demonstrate that Guided-TTS performs well on diverse datasets including a long-form untranscribed dataset.

cs.SD↗

arXiv2022

Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data

We propose Guided-TTS 2, a diffusion-based generative model for high-quality adaptive TTS using untranscribed data. Guided-TTS 2 combines a speaker-conditional diffusion model with a speaker-dependent phoneme classifier for adaptive text-to-speech. We train the speaker-conditional diffusion model on large-scale untranscribed datasets for a classifier-free guidance method and further fine-tune the diffusion model on the reference speech of the target speaker for adaptation, which only takes 40 seconds. We demonstrate that Guided-TTS 2 shows comparable performance to high-quality single-speaker TTS baselines in terms of speech quality and speaker similarity with only a ten-second untranscribed data. We further show that Guided-TTS 2 outperforms adaptive TTS baselines on multi-speaker datasets even with a zero-shot adaptation setting. Guided-TTS 2 can adapt to a wide range of voices only using untranscribed speech, which enables adaptive TTS with the voice of non-human characters such as Gollum in \textit{"The Lord of the Rings"}.

cs.SD↗

arXiv2022

Perception Prioritized Training of Diffusion Models

Diffusion models learn to restore noisy data, which is corrupted with different levels of noise, by optimizing the weighted sum of the corresponding loss terms, i.e., denoising score matching loss. In this paper, we show that restoring data corrupted with certain noise levels offers a proper pretext task for the model to learn rich visual concepts. We propose to prioritize such noise levels over other levels during training, by redesigning the weighting scheme of the objective function. We show that our simple redesign of the weighting scheme significantly improves the performance of diffusion models regardless of the datasets, architectures, and sampling strategies.

cs.CV↗

arXiv2021

FICGAN: Facial Identity Controllable GAN for De-identification

In this work, we present Facial Identity Controllable GAN (FICGAN) for not only generating high-quality de-identified face images with ensured privacy protection, but also detailed controllability on attribute preservation for enhanced data utility. We tackle the less-explored yet desired functionality in face de-identification based on the two factors. First, we focus on the challenging issue to obtain a high level of privacy protection in the de-identification task while uncompromising the image quality. Second, we analyze the facial attributes related to identity and non-identity and explore the trade-off between the degree of face de-identification and preservation of the source attributes for enhanced data utility. Based on the analysis, we develop Facial Identity Controllable GAN (FICGAN), an autoencoder-based conditional generative model that learns to disentangle the identity attributes from non-identity attributes on a face image. By applying the manifold k-same algorithm to satisfy k-anonymity for strengthened security, our method achieves enhanced privacy protection in de-identified face images. Numerous experiments demonstrate that our model outperforms others in various scenarios of face de-identification.

cs.CV↗

arXiv2021

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models

Denoising diffusion probabilistic models (DDPM) have shown remarkable performance in unconditional image generation. However, due to the stochasticity of the generative process in DDPM, it is challenging to generate images with the desired semantics. In this work, we propose Iterative Latent Variable Refinement (ILVR), a method to guide the generative process in DDPM to generate high-quality images based on a given reference image. Here, the refinement of the generative process in DDPM enables a single DDPM to sample images from various sets directed by the reference image. The proposed ILVR method generates high-quality images while controlling the generation. The controllability of our method allows adaptation of a single DDPM without any additional learning in various image generation tasks, such as generation from various downsampling factors, multi-domain image translation, paint-to-image, and editing with scribbles.

cs.CV↗

arXiv2021

AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate

Non-autoregressive neural machine translation (NART) models suffer from the multi-modality problem which causes translation inconsistency such as token repetition. Most recent approaches have attempted to solve this problem by implicitly modeling dependencies between outputs. In this paper, we introduce AligNART, which leverages full alignment information to explicitly reduce the modality of the target distribution. AligNART divides the machine translation task into $(i)$ alignment estimation and $(ii)$ translation with aligned decoder inputs, guiding the decoder to focus on simplified one-to-one translation. To alleviate the alignment estimation problem, we further propose a novel alignment decomposition method. Our experiments show that AligNART outperforms previous non-iterative NART models that focus on explicit modality reduction on WMT14 En$\leftrightarrow$De and WMT16 Ro$\rightarrow$En. Furthermore, AligNART achieves BLEU scores comparable to those of the state-of-the-art connectionist temporal classification based models on WMT14 En$\leftrightarrow$De. We also observe that AligNART effectively addresses the token repetition problem even without sequence-level knowledge distillation.

cs.CL↗