arXiv subjects

Sungjoon Park

Publications and source records attributed to Sungjoon Park.

At least 19 recordsLinked to original sources

arXiv2026

CtrlVTON: Controllable Virtual Try-On via Visual-Instance-Prompt Segmentation

Virtual try-on (VTO) has made significant progress in realistically transferring garments onto a target person. Yet most systems give the user little control over how a garment should be worn -- its size (loose or fitted), style (e.g., tucked in or untucked, open or closed), and spatial placement on the body. We address this gap with two complementary contributions. First, we define and solve Visual-Instance-Prompt Segmentation via VIP-SAM: given a flatlay image of a garment, segment that specific instance in a photograph of a person wearing it. This is an instance-level task, distinct from the typically studied category-level segmentation. Second, we introduce CtrlVTON, a controllable VTO framework that recasts try-on as an image editing problem and adds segmentation masks as pixel-level control over garment layout, including style, size, and spatial placement on the body. VIP-SAM and CtrlVTON each achieve state-of-the-art results on their respective tasks. In particular, CtrlVTON generates images that follow user-provided layouts far more faithfully than the strongest proprietary editing systems while matching them on garment fidelity.

cs.CV↗

arXiv2025

Neologism Learning as a Parameter-Efficient Alternative to Fine-Tuning for Model Steering

In language modeling, neologisms are new tokens trained to represent a concept not already included in a given model's vocabulary. Neologisms can be used to encourage specific behavior in models, for example by appending prompts with "Give me a neologism answer." Behavioral steering can also be achieved through fine-tuning, albeit with more compute and less flexibility: learning a neologism only trains d parameters and allows the user to still access the model's default behavior. We compare the performance of neologism learning against low-rank adaptation (LoRA) fine-tuning, finding that neologisms outperform fine-tuned models under a matched training setup (same data and hyperparameters). We also investigate self-verbalizations of neologisms, and observe that the model will occasionally make up its own new words when asked about a neologism.

cs.CL↗

arXiv2025

Learning Perceptually Relevant Temporal Envelope Morphing

Temporal envelope morphing, the process of interpolating between the amplitude dynamics of two audio signals, is an emerging problem in generative audio systems that lacks sufficient perceptual grounding. Morphing of temporal envelopes in a perceptually intuitive manner should enable new methods for sound blending in creative media and for probing perceptual organization in psychoacoustics. However, existing audio morphing techniques often fail to produce intermediate temporal envelopes when input sounds have distinct temporal structures; many morphers effectively overlay both temporal structures, leading to perceptually unnatural results. In this paper, we introduce a novel workflow for learning envelope morphing with perceptual guidance: we first derive perceptually grounded morphing principles through human listening studies, then synthesize large-scale datasets encoding these principles, and finally train machine learning models to create perceptually intermediate morphs. Specifically, we present: (1) perceptual principles that guide envelope morphing, derived from our listening studies, (2) a supervised framework to learn these principles, (3) an autoencoder that learns to compress temporal envelope structures into latent representations, and (4) benchmarks for evaluating audio envelope morphs, using both synthetic and naturalistic data, and show that our approach outperforms existing methods in producing temporally intermediate morphs. All code, models, and checkpoints are available at https://github.com/TemporalMorphing/EnvelopeMorphing.

cs.SD↗

arXiv2025

Robust Multi-Omics Integration from Incomplete Modalities Significantly Improves Prediction of Alzheimer's Disease

Multi-omics data capture complex biomolecular interactions and provide insights into metabolism and disease. However, missing modalities hinder integrative analysis across heterogeneous omics. To address this, we present MOIRA (Multi-Omics Integration with Robustness to Absent modalities), an early integration method enabling robust learning from incomplete omics data via representation alignment and adaptive aggregation. MOIRA leverages all samples, including those with missing modalities, by projecting each omics dataset onto a shared embedding space where a learnable weighting mechanism fuses them. Evaluated on the Religious Order Study and Memory and Aging Project (ROSMAP) dataset for Alzheimer's Disease (AD), MOIRA outperformed existing approaches, and further ablation studies confirmed modality-wise contributions. Feature importance analysis revealed AD-related biomarkers consistent with prior literature, highlighting the biological relevance of our approach.

cs.LG↗

arXiv2025

JOLT3D: Joint Learning of Talking Heads and 3DMM Parameters with Application to Lip-Sync

In this work, we revisit the effectiveness of 3DMM for talking head synthesis by jointly learning a 3D face reconstruction model and a talking head synthesis model. This enables us to obtain a FACS-based blendshape representation of facial expressions that is optimized for talking head synthesis. This contrasts with previous methods that either fit 3DMM parameters to 2D landmarks or rely on pretrained face reconstruction models. Not only does our approach increase the quality of the generated face, but it also allows us to take advantage of the blendshape representation to modify just the mouth region for the purpose of audio-based lip-sync. To this end, we propose a novel lip-sync pipeline that, unlike previous methods, decouples the original chin contour from the lip-synced chin contour, and reduces flickering near the mouth.

cs.CV↗

arXiv2024

Interpretable Convolutional SyncNet

Because videos in the wild can be out of sync for various reasons, a sync-net is used to bring the video back into sync for tasks that require synchronized videos. Previous state-of-the-art (SOTA) sync-nets use InfoNCE loss, rely on the transformer architecture, or both. Unfortunately, the former makes the model's output difficult to interpret, and the latter is unfriendly with large images, thus limiting the usefulness of sync-nets. In this work, we train a convolutional sync-net using the balanced BCE loss (BBCE), a loss inspired by the binary cross entropy (BCE) and the InfoNCE losses. In contrast to the InfoNCE loss, the BBCE loss does not require complicated sampling schemes. Our model can better handle larger images, and its output can be given a probabilistic interpretation. The probabilistic interpretation allows us to define metrics such as probability at offset and offscreen ratio to evaluate the sync quality of audio-visual (AV) speech datasets. Furthermore, our model achieves SOTA accuracy of $96.5\%$ on the LRS2 dataset and $93.8\%$ on the LRS3 dataset.

cs.CV↗

arXiv2024

Harmful Suicide Content Detection

Harmful suicide content on the Internet is a significant risk factor inducing suicidal thoughts and behaviors among vulnerable populations. Despite global efforts, existing resources are insufficient, specifically in high-risk regions like the Republic of Korea. Current research mainly focuses on understanding negative effects of such content or suicide risk in individuals, rather than on automatically detecting the harmfulness of content. To fill this gap, we introduce a harmful suicide content detection task for classifying online suicide content into five harmfulness levels. We develop a multi-modal benchmark and a task description document in collaboration with medical professionals, and leverage large language models (LLMs) to explore efficient methods for moderating such content. Our contributions include proposing a novel detection task, a multi-modal Korean benchmark with expert annotations, and suggesting strategies using LLMs to detect illegal and harmful content. Owing to the potential harm involved, we publicize our implementations and benchmark, incorporating an ethical verification process.

cs.CY↗

arXiv2023

FedTherapist: Mental Health Monitoring with User-Generated Linguistic Expressions on Smartphones via Federated Learning

Psychiatrists diagnose mental disorders via the linguistic use of patients. Still, due to data privacy, existing passive mental health monitoring systems use alternative features such as activity, app usage, and location via mobile devices. We propose FedTherapist, a mobile mental health monitoring system that utilizes continuous speech and keyboard input in a privacy-preserving way via federated learning. We explore multiple model designs by comparing their performance and overhead for FedTherapist to overcome the complex nature of on-device language model training on smartphones. We further propose a Context-Aware Language Learning (CALL) methodology to effectively utilize smartphones' large and noisy text for mental health signal sensing. Our IRB-approved evaluation of the prediction of self-reported depression, stress, anxiety, and mood from 46 participants shows higher accuracy of FedTherapist compared with the performance with non-language features, achieving 0.15 AUROC improvement and 8.21% MAE reduction.

cs.CL↗

arXiv2023

Analyzing Norm Violations in Live-Stream Chat

Toxic language, such as hate speech, can deter users from participating in online communities and enjoying popular platforms. Previous approaches to detecting toxic language and norm violations have been primarily concerned with conversations from online forums and social media, such as Reddit and Twitter. These approaches are less effective when applied to conversations on live-streaming platforms, such as Twitch and YouTube Live, as each comment is only visible for a limited time and lacks a thread structure that establishes its relationship with other comments. In this work, we share the first NLP study dedicated to detecting norm violations in conversations on live-streaming platforms. We define norm violation categories in live-stream chats and annotate 4,583 moderated comments from Twitch. We articulate several facets of live-stream data that differ from other forums, and demonstrate that existing models perform poorly in this setting. By conducting a user study, we identify the informational context humans use in live-stream moderation, and train models leveraging context to identify norm violations. Our results show that appropriate contextual information can boost moderation performance by 35\%.

cs.CL↗

arXiv2023

Towards standardizing Korean Grammatical Error Correction: Datasets and Annotation

Research on Korean grammatical error correction (GEC) is limited, compared to other major languages such as English. We attribute this problematic circumstance to the lack of a carefully designed evaluation benchmark for Korean GEC. In this work, we collect three datasets from different sources (Kor-Lang8, Kor-Native, and Kor-Learner) that covers a wide range of Korean grammatical errors. Considering the nature of Korean grammar, We then define 14 error types for Korean and provide KAGAS (Korean Automatic Grammatical error Annotation System), which can automatically annotate error types from parallel corpora. We use KAGAS on our datasets to make an evaluation benchmark for Korean, and present baseline models trained from our datasets. We show that the model trained with our datasets significantly outperforms the currently used statistical Korean GEC system (Hanspell) on a wider range of error types, demonstrating the diversity and usefulness of the datasets. The implementations and datasets are open-sourced.

cs.CL↗

arXiv2022

KOLD: Korean Offensive Language Dataset

Recent directions for offensive language detection are hierarchical modeling, identifying the type and the target of offensive language, and interpretability with offensive span annotation and prediction. These improvements are focused on English and do not transfer well to other languages because of cultural and linguistic differences. In this paper, we present the Korean Offensive Language Dataset (KOLD) comprising 40,429 comments, which are annotated hierarchically with the type and the target of offensive language, accompanied by annotations of the corresponding text spans. We collect the comments from NAVER news and YouTube platform and provide the titles of the articles and videos as the context information for the annotation process. We use these annotated comments as training data for Korean BERT and RoBERTa models and find that they are effective at offensiveness detection, target classification, and target span detection while having room for improvement for target group classification and offensive span detection. We discover that the target group distribution differs drastically from the existing English datasets, and observe that providing the context information improves the model performance in offensiveness detection (+0.3), target classification (+1.5), and target group classification (+13.1). We publicly release the dataset and baseline models.

cs.CL↗

arXiv2022

Theory of transverse magnetization in spin-orbit coupled antiferromagnets

Some antiferromagnets under a magnetic field develop magnetization perpendicular to the field as well as more conventional ones parallel to the field. So far, the transverse magnetization (TM) has been attributed to either spin canting effect or the presence of cluster magnetic multipolar ordering. However, a general theory of TM based on microscopic understanding is still missing. Here, we construct a general microscopic theory of TM in antiferromagnets with cluster magnetic multipolar ordering by considering classical spin Hamiltonians with spin anisotropy that arises from the spin-orbit coupling. First, from general symmetry analysis, we show that TM can appear only when all crystalline symmetries are broken other than the antiunitary mirror, antiunitary two-fold rotation, and inversion symmetries. Moreover, by analyzing spin Hamiltonians, we show that TM always appears when the degenerate ground state manifold of the spin Hamiltonian is discrete. On the other hand, when the degenerate ground state manifold is continuous, TM generally does not appear except when the magnetic field direction and the spin configuration satisfy specific geometric conditions under single-ion anisotropy. Finally, we show that TM can induce anomalous planar Hall Effect, a unique transport phenomenon that can be used to probe multipolar antiferromagnetic structures. We believe that our theory provides a useful guideline for understanding the anomalous magnetic responses of the antiferromagnets with complex magnetic structures.

cond-mat.mes-hall↗

arXiv2022

Unsupervised learning of topological phase diagram using topological data analysis

Topology and machine learning are two actively researched topics not only in condensed matter physics, but also in data science. Here, we propose the use of topological data analysis in unsupervised learning of the topological phase diagrams. This is possible because the quantum distance can capture the shape of the space formed by the Bloch wavefunctions as we sweep over the Brillouin zone. Therefore, if we minimize the volume of the space formed by the wavefunction through a continuous deformation, the wavefunctions will end up forming distinct spaces which depend on the topology of the wavefunctions. Combining this observation with the topological data analysis, which provides tools such as the persistence diagram to capture the topology of the space formed by the wavefunctions, we can cluster together Hamiltonians that give rise to similar persistence diagrams after the deformation. By examining these clusters as well as representative persistence diagrams in the clusters, we can draw the phase diagram as well as distinguish between topologically trivial and nontrivial phases. Our proposal to minimize the volume can be interpreted as finding geodesics in 1D Brillouin zone, and minimal surfaces in 2D and higher-dimensional Brillouin zones. Using this interpretation, we can guarantee the convergence of the minimization under certain conditions, which is an outstanding feature of our algorithm. We demonstrate the working principles of our machine learning algorithm using various models.

cond-mat.mes-hall↗

arXiv2021

Improving Tagging Consistency and Entity Coverage for Chemical Identification in Full-text Articles

This paper is a technical report on our system submitted to the chemical identification task of the BioCreative VII Track 2 challenge. The main feature of this challenge is that the data consists of full-text articles, while current datasets usually consist of only titles and abstracts. To effectively address the problem, we aim to improve tagging consistency and entity coverage using various methods such as majority voting within the same articles for named entity recognition (NER) and a hybrid approach that combines a dictionary and a neural model for normalization. In the experiments on the NLM-Chem dataset, we show that our methods improve models' performance, particularly in terms of recall. Finally, in the official evaluation of the challenge, our system was ranked 1st in NER by significantly outperforming the baseline model and more than 80 submissions from 16 teams.

cs.CL↗

arXiv2021

Temperature selective thermometry with sub-microsecond time resolution using dressed-spin states in diamond

Versatile nanoscale sensors that are susceptible to changes in a variety of physical quantities often exhibit limited selectivity. This paper reports a novel scheme based on microwave-dressed spin states for optically probed nanoscale temperature detection using diamond quantum sensors, which provides selective sensitivity to temperature changes. By combining this scheme with a continuous pump-probe scheme using ensemble nitrogen-vacancy centers in nanodiamonds, a sub-microsecond temporal resolution with thermal sensitivity of 3.7 K$\cdot$Hz$^{-1/2}$ that is insensitive to variations in external magnetic fields on the order of 2 G is demonstrated. The presented results are favorable for the practical application of time-resolved nanoscale quantum sensing, where temperature imaging is required under fluctuating magnetic fields.

quant-ph↗

arXiv2021

KLUE: Korean Language Understanding Evaluation

We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, SemanticTextual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting observations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com.

cs.CL↗

arXiv2021

Dimensional Emotion Detection from Categorical Emotion

We present a model to predict fine-grained emotions along the continuous dimensions of valence, arousal, and dominance (VAD) with a corpus with categorical emotion annotations. Our model is trained by minimizing the EMD (Earth Mover's Distance) loss between the predicted VAD score distribution and the categorical emotion distributions sorted along VAD, and it can simultaneously classify the emotion categories and predict the VAD scores for a given sentence. We use pre-trained RoBERTa-Large and fine-tune on three different corpora with categorical labels and evaluate on EmoBank corpus with VAD scores. We show that our approach reaches comparable performance to that of the state-of-the-art classifiers in categorical emotion classification and shows significant positive correlations with the ground truth VAD scores. Also, further training with supervision of VAD labels leads to improved performance especially when dataset is small. We also present examples of predictions of appropriate emotion words that are not part of the original annotations.

cs.CL↗

arXiv2021

Magnetoelastic coupling enabled tunability of magnon spin current generation in 2D antiferromagnets

We theoretically investigate the magnetoelastic coupling (MEC) and its effect on magnon transport in two-dimensional antiferromagnets with a honeycomb lattice. MEC coeffcient along with magnetic exchange parameters and spring constants are computed for monolayers of transition metal trichalcogenides with Néel order ($\text{MnPS}_3$ and $\text{VPS}_3$) and zigzag order ($\text{CrSiTe}_3$, $\text{NiPS}_3$ and $\text{NiPSe}_3$) by $ab$ $initio$ calculations. Using these parameters, we predict that the spin-Nernst coefficient is significantly enhanced due to magnetoelastic coupling. Our study shows that although Dzyaloshinskii-Moriya interaction can produce spin Nernst effect in these materials, other mechanisms such as magnon-phonon coupling should be taken into account. We also demonstrate that the magnetic anisotropy is an important factor for control of magnon-phonon hybridization and enhancement of the Berry curvature and thus the spin-Nernst coefficient. Our results pave the way towards gate tunable spin current generation in 2D magnets by SNE via electric field modulation of MEC and anisotropy.

cond-mat.mtrl-sci↗