arXiv subjects

Sangmin Lee

Publications and source records attributed to Sangmin Lee.

At least 37 records · Page 2Linked to original sources

arXiv2026

MuCo: Multi-turn Contrastive Learning for Multimodal Embedding Model

Universal Multimodal embedding models built on Multimodal Large Language Models (MLLMs) have traditionally employed contrastive learning, which aligns representations of query-target pairs across different modalities. Yet, despite its empirical success, they are primarily built on a "single-turn" formulation where each query-target pair is treated as an independent data point. This paradigm leads to computational inefficiency when scaling, as it requires a separate forward pass for each pair and overlooks potential contextual relationships between multiple queries that can relate to the same context. In this work, we introduce Multi-Turn Contrastive Learning (MuCo), a dialogue-inspired framework that revisits this process. MuCo leverages the conversational nature of MLLMs to process multiple, related query-target pairs associated with a single image within a single forward pass. This allows us to extract a set of multiple query and target embeddings simultaneously, conditioned on a shared context representation, amplifying the effective batch size and overall training efficiency. Experiments exhibit MuCo with a newly curated 5M multimodal multi-turn dataset (M3T), which yields state-of-the-art retrieval performance on MMEB and M-BEIR benchmarks, while markedly enhancing both training efficiency and representation coherence across modalities. Code and M3T are available at https://github.com/naver-ai/muco

cs.IR↗

arXiv2026

First-principles evidence for conventional superconductivity in a quasicrystal approximant

Quasicrystals (QCs) host long-range order without translational symmetry, a regime in which the very foundations of BCS theory are not straightforwardly applicable, yet experiments on QCs and their approximant crystals (ACs) point to conventional, $s$-wave, electron-phonon coupled superconductivity. Here we test the predictive power of the electron-phonon framework in a representative decagonal AC from first principles. Using state-of-the-art \textit{ab initio} methods, we compute the superconducting properties of the recently discovered AC Al$_{13}$Os$_4$ and quantitatively reproduce its bulk $T_\text{c}$. This constitutes, to our knowledge, the first \textit{ab initio} determination of $T_\text{c}$ for an AC and establishes that the electron-phonon framework is predictive in these systems as well. Using the generalized quasichemical approximation for alloy modeling in the decagonal Al--Os family, we predict tunable superconductivity in Al$_{13}$Os$_{4-x}$Re$_x$ and Al$_{13}$Os$_{4-x}$Ir$_x$; in particular, Al$_{13}$Re$_4$ is dynamically stable and estimated to have a $T_\text{c}$ about 30% above Al$_{13}$Os$_4$. Finally, we discuss the role of ACs as high-fidelity proxies for their parent QCs. Although long-range quasiperiodicity may introduce subtle electronic features, our findings indicate that the key ingredients for superconductivity are already encoded in the local structural motifs preserved by the AC. This places the Al--Os and Al--Re families among the most promising candidates for the highest-$T_\text{c}$ quasicrystalline superconductivity.

cond-mat.supr-con↗

arXiv2026

Interfacial Potential Transduction for Diagnostics

A major barrier to decentralized, near-patient diagnostics is the lack of a signal transduction modality that is both analytically precise and accessible at the point of care. Optical readouts remain instrument-dependent and difficult to miniaturize, while compact electrochemical readouts are prone to matrix-derived signal distortion, limiting their biomarker coverage in real clinical settings. Here, we define interfacial potential transduction as a standardized electrical modality for portable, clinical-grade diagnostics across diverse assay formats. A mechanistic framework identifying key sample matrix parameters within the interfacial potentials transduction system enables control of biofluid-derived interference, and is demonstrated in a widely accessible lateral flow immunoassay format through quantitative detection of estradiol, progesterone, and luteinizing hormone in human plasma with high correlation (r2 > 0.97) to clinical analyzers. Broader applicability across representative diagnostic sectors is further demonstrated through exceptional performance including glucose quantification for biochemical analysis with limit of detection (LOD) of 0.92 ug/dL, HIV p24 capsid protein under an immunomagnetic separation workflow (LOD = 44.8 fg/mL), and hepatitis B virus detection within 5 min via loop-mediated isothermal amplification for molecular diagnostics. Together, these results establish interfacial potentials transduction as a unified diagnostic paradigm for near-patient deployment beyond optical and electrochemical approaches.

q-bio.BM↗

arXiv2026

Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioning

Text-image-to-video (TI2V) generation is a critical problem for controllable video generation using both semantic and visual conditions. Most existing methods typically add visual conditions to text-to-video (T2V) foundation models by finetuning, which is costly in resources and only limited to a few pre-defined conditioning settings. To tackle these constraints, we introduce a unified formulation for TI2V generation with flexible visual conditioning. Furthermore, we propose an innovative training-free approach, dubbed FlexTI2V, that can condition T2V foundation models on an arbitrary amount of images at arbitrary positions. Specifically, we firstly invert the condition images to noisy representation in a latent space. Then, in the denoising process of T2V models, our method uses a novel random patch swapping strategy to incorporate visual features into video representations through local image patches. To balance creativity and fidelity, we use a dynamic control mechanism to adjust the strength of visual conditioning to each video frame. Extensive experiments validate that our method surpasses previous training-free image conditioning methods by a notable margin. Our method can also generalize to both UNet-based and transformer-based architectures.

cs.CV↗

arXiv2026

KD-EKF: Knowledge-Distilled Adaptive Covariance EKF for Robust UWB/PDR Indoor Localization

Ultra-wideband (UWB) indoor localization provides centimeter-level accuracy and low latency, but its measurement reliability degrades severely under Non-Line-of-Sight (NLOS) conditions, leading to meter-scale ranging errors and inconsistent uncertainty characteristics. Inertial Measurement Unit (IMU)-based Pedestrian Dead Reckoning (PDR) complements UWB by providing infrastructure-free motion estimation; however, its error accumulates nonlinearly over time due to bias and noise propagation. Fusion methods based on Extended Kalman Filters (EKF) and Particle Filters (PF) can improve average localization accuracy through probabilistic state estimation. However, these approaches typically rely on manually tuned measurement covariances. Such fixed or heuristically tuned parameters are hard to sustain across varying indoor layouts, NLOS ratios, and motion patterns, leading to limited robustness and poor generalization of measurement uncertainty modeling in heterogeneous environments. To address this limitation, this work proposes an adaptive measurement covariance scaling framework in which reliability cues are learned from historical UWB/PDR trajectories. A large teacher model is employed offline to generate temporally consistent next-position predictions from structured UWB/PDR sequences, and this behavior is distilled into a lightweight student model suitable for real-time deployment. The student model continuously regulates EKF measurement covariances based on prediction residuals, enabling environment-aware fusion without manual re-tuning. Experimental results demonstrate that the proposed KD-EKF framework significantly reduces localization error, suppresses error spikes during Line-of-Sight (LOS)/NLOS transitions, and mitigates long-term drift compared to fixed-parameter EKF, thereby improving measurement robustness across diverse indoor environments.

eess.SP↗

arXiv2026

UniverSR: Unified and Versatile Audio Super-Resolution via Vocoder-Free Flow Matching

In this paper, we present a vocoder-free framework for audio super-resolution that employs a flow matching generative model to capture the conditional distribution of complex-valued spectral coefficients. Unlike conventional two-stage diffusion-based approaches that predict a mel-spectrogram and then rely on a pre-trained neural vocoder to synthesize waveforms, our method directly reconstructs waveforms via the inverse Short-Time Fourier Transform (iSTFT), thereby eliminating the dependence on a separate vocoder. This design not only simplifies end-to-end optimization but also overcomes a critical bottleneck of two-stage pipelines, where the final audio quality is fundamentally constrained by vocoder performance. Experiments show that our model consistently produces high-fidelity 48 kHz audio across diverse upsampling factors, achieving state-of-the-art performance on both speech and general audio datasets.

eess.AS↗

arXiv2025

K-DRIFT: Unveiling New Imagery of the Hidden Universe

Low-surface-brightness (LSB) structures play a crucial role in understanding galaxy evolution by providing significant insights into galaxy interactions, the histories of mass assembly, and the distribution of dark matter. Nevertheless, their inherently faint nature, coupled with observational difficulties such as stray light interference and variations in the sky background, has significantly impeded comprehensive studies of LSB features. The KASI Deep Rolling Imaging Fast Telescope (K-DRIFT) project aims to address these observational challenges by developing off-axis freeform three-mirror telescopes and observational strategies specifically designed for LSB imaging surveys. The first generation of the K-DRIFT (K-DRIFT G1) has been successfully completed, and the forthcoming survey, scheduled to commence shortly, is expected to yield novel insights into the LSB universe. This paper outlines the scientific motivations of the project, discusses the technical challenges encountered, highlights the innovative solutions devised, and describes the future trajectory of the K-DRIFT.

astro-ph.GA↗

arXiv2025

LAMA-UT: Language Agnostic Multilingual ASR through Orthography Unification and Language-Specific Transliteration

Building a universal multilingual automatic speech recognition (ASR) model that performs equitably across languages has long been a challenge due to its inherent difficulties. To address this task we introduce a Language-Agnostic Multilingual ASR pipeline through orthography Unification and language-specific Transliteration (LAMA-UT). LAMA-UT operates without any language-specific modules while matching the performance of state-of-the-art models trained on a minimal amount of data. Our pipeline consists of two key steps. First, we utilize a universal transcription generator to unify orthographic features into Romanized form and capture common phonetic characteristics across diverse languages. Second, we utilize a universal converter to transform these universal transcriptions into language-specific ones. In experiments, we demonstrate the effectiveness of our proposed method leveraging universal transcriptions for massively multilingual ASR. Our pipeline achieves a relative error reduction rate of 45% when compared to Whisper and performs comparably to MMS, despite being trained on only 0.1% of Whisper's training data. Furthermore, our pipeline does not rely on any language-specific modules. However, it performs on par with zero-shot ASR approaches which utilize additional language-specific lexicons and language models. We expect this framework to serve as a cornerstone for flexible multilingual ASR systems that are generalizable even to unseen languages.

cs.CL↗

arXiv2025

SAGE-LD: Towards Scalable and Generalizable End-to-End Language Diarization via Simulated Data Augmentation

In this paper, we present a neural spoken language diarization model that supports an unconstrained span of languages within a single framework. Our approach integrates a learnable query-based architecture grounded in multilingual awareness, with large-scale pretraining on simulated code-switching data. By jointly leveraging these two components, our method overcomes the limitations of conventional approaches in data scarcity and architecture optimization, and generalizes effectively to real-world multilingual settings across diverse environments. Experimental results demonstrate that our approach achieves state-of-the-art performance on several language diarization benchmarks, with a relative performance improvement of 23% to 52% over previous methods. We believe that this work not only advances research in language diarization but also establishes a foundational framework for code-switching speech technologies.

cs.CL↗

arXiv2025

Classical eikonal in relativistic scattering

The classical eikonal is defined to be the generator of all scattering observables in a scattering problem in classical mechanics. It was originally introduced as the log of the quantum S-matrix in the classical limit. But its classical nature calls for a definition and computational methods independent of quantum mechanics. In this paper, we formulate a classical interaction picture which serves as the foundation of the classical eikonal. Our emphasis is on generality. In perturbation theories, both Hamiltonian deformation and symplectic deformation are considered. Particles and fields are treated on a similar footing. The causality prescription of the propagator is essentially the same for non-relativistic and relativistic kinematics. For a probe particle in electromagnetic or gravitational background, we present all order formulas for the perturbative eikonal. In the electromagnetic setting, we also illustrate how the eikonal encodes the information on radiation of external fields.

hep-th↗

arXiv2025

MemoryTalker: Personalized Speech-Driven 3D Facial Animation via Audio-Guided Stylization

Speech-driven 3D facial animation aims to synthesize realistic facial motion sequences from given audio, matching the speaker's speaking style. However, previous works often require priors such as class labels of a speaker or additional 3D facial meshes at inference, which makes them fail to reflect the speaking style and limits their practical use. To address these issues, we propose MemoryTalker which enables realistic and accurate 3D facial motion synthesis by reflecting speaking style only with audio input to maximize usability in applications. Our framework consists of two training stages: 1-stage is storing and retrieving general motion (i.e., Memorizing), and 2-stage is to perform the personalized facial motion synthesis (i.e., Animating) with the motion memory stylized by the audio-driven speaking style feature. In this second stage, our model learns about which facial motion types should be emphasized for a particular piece of audio. As a result, our MemoryTalker can generate a reliable personalized facial animation without additional prior information. With quantitative and qualitative evaluations, as well as user study, we show the effectiveness of our model and its performance enhancement for personalized facial animation over state-of-the-art methods.

cs.CV↗

arXiv2025

UniCoM: A Universal Code-Switching Speech Generator

Code-switching (CS), the alternation between two or more languages within a single speaker's utterances, is common in real-world conversations and poses significant challenges for multilingual speech technology. However, systems capable of handling this phenomenon remain underexplored, primarily due to the scarcity of suitable datasets. To resolve this issue, we propose Universal Code-Mixer (UniCoM), a novel pipeline for generating high-quality, natural CS samples without altering sentence semantics. Our approach utilizes an algorithm we call Substituting WORDs with Synonyms (SWORDS), which generates CS speech by replacing selected words with their translations while considering their parts of speech. Using UniCoM, we construct Code-Switching FLEURS (CS-FLEURS), a multilingual CS corpus designed for automatic speech recognition (ASR) and speech-to-text translation (S2TT). Experimental results show that CS-FLEURS achieves high intelligibility and naturalness, performing comparably to existing datasets on both objective and subjective metrics. We expect our approach to advance CS speech technology and enable more inclusive multilingual systems.

cs.CL↗

arXiv2025

IM-LUT: Interpolation Mixing Look-Up Tables for Image Super-Resolution

Super-resolution (SR) has been a pivotal task in image processing, aimed at enhancing image resolution across various applications. Recently, look-up table (LUT)-based approaches have attracted interest due to their efficiency and performance. However, these methods are typically designed for fixed scale factors, making them unsuitable for arbitrary-scale image SR (ASISR). Existing ASISR techniques often employ implicit neural representations, which come with considerable computational cost and memory demands. To address these limitations, we propose Interpolation Mixing LUT (IM-LUT), a novel framework that operates ASISR by learning to blend multiple interpolation functions to maximize their representational capacity. Specifically, we introduce IM-Net, a network trained to predict mixing weights for interpolation functions based on local image patterns and the target scale factor. To enhance efficiency of interpolation-based methods, IM-Net is transformed into IM-LUT, where LUTs are employed to replace computationally expensive operations, enabling lightweight and fast inference on CPUs while preserving reconstruction quality. Experimental results on several benchmark datasets demonstrate that IM-LUT consistently achieves a superior balance between image quality and efficiency compared to existing methods, highlighting its potential as a promising solution for resource-constrained applications.

eess.IV↗

arXiv2025

Derivative-Free Diffusion Manifold-Constrained Gradient for Unified XAI

Gradient-based methods are a prototypical family of explainability techniques, especially for image-based models. Nonetheless, they have several shortcomings in that they (1) require white-box access to models, (2) are vulnerable to adversarial attacks, and (3) produce attributions that lie off the image manifold, leading to explanations that are not actually faithful to the model and do not align well with human perception. To overcome these challenges, we introduce Derivative-Free Diffusion Manifold-Constrainted Gradients (FreeMCG), a novel method that serves as an improved basis for explainability of a given neural network than the traditional gradient. Specifically, by leveraging ensemble Kalman filters and diffusion models, we derive a derivative-free approximation of the model's gradient projected onto the data manifold, requiring access only to the model's outputs. We demonstrate the effectiveness of FreeMCG by applying it to both counterfactual generation and feature attribution, which have traditionally been treated as distinct tasks. Through comprehensive evaluation on both tasks, counterfactual explanation and feature attribution, we show that our method yields state-of-the-art results while preserving the essential properties expected of XAI tools.

cs.CV↗

arXiv2025

Object-aware Sound Source Localization via Audio-Visual Scene Understanding

Audio-visual sound source localization task aims to spatially localize sound-making objects within visual scenes by integrating visual and audio cues. However, existing methods struggle with accurately localizing sound-making objects in complex scenes, particularly when visually similar silent objects coexist. This limitation arises primarily from their reliance on simple audio-visual correspondence, which does not capture fine-grained semantic differences between sound-making and silent objects. To address these challenges, we propose a novel sound source localization framework leveraging Multimodal Large Language Models (MLLMs) to generate detailed contextual information that explicitly distinguishes between sound-making foreground objects and silent background objects. To effectively integrate this detailed information, we introduce two novel loss functions: Object-aware Contrastive Alignment (OCA) loss and Object Region Isolation (ORI) loss. Extensive experimental results on MUSIC and VGGSound datasets demonstrate the effectiveness of our approach, significantly outperforming existing methods in both single-source and multi-source localization scenarios. Code and generated detailed contextual information are available at: https://github.com/VisualAIKHU/OA-SSL.

cs.CV↗

arXiv2025

Bi2Te3-Sb2Te3-Bi2Te3 Lateral Heterostructures Grown by Molecular Beam Epitaxy

Lateral in-plane heterostructures enable precise control of electronic properties and quantum effects in 2D materials. However, their periodic synthesis is challenging because it requires precise control to maintain sharp, coherent interfaces and compatible growth conditions across different domains. Herein, we report the successful heteroepitaxial growth of Bi2Te3-Sb2Te3-Bi2Te3 and periodic lateral heterostructures on hexagonal boron nitride (hBN) through in-situ multiple growth steps at different stages using a molecular beam epitaxy (MBE) system. These trilateral heterostructures are fabricated by growing triangular or hexagonal Bi2Te3 islands at the very beginning, with typical sizes of several hundred nanometers, on the single-crystalline hBN, followed by the lateral growth of Sb2Te3 to form bilateral heterostructures, and finally growing Bi2Te3 on the side facets of the bilateral heterostructures. The electron microscopy results confirm the core area as Bi2Te3, the intermediate layer as Sb2Te3, and the outermost region as Bi2Te3. The resulting heterostructures are approximately 4-8 nm thick and several hundred nanometers in lateral dimensions. These heterostructures are found to grow epitaxially on hBN (< +-4 deg misalignment), and the individual layers are strongly epitaxially aligned with each other. The in-plane heterojunctions are analyzed using the aberration-corrected (Cs-corrected) high-angle annular dark-field scanning transmission electron microscopy technique. We have explored and established the plasmonic properties of these fabricated Bi2Te3-Sb2Te3-Bi2Te3 lateral heterostructures. In addition, the electronic states and the topological properties of the few quintuple layers (QLs) (2- to 4-QLs) Bi2Te3-Sb2Te3 lateral periodic heterostructures are investigated by first-principles calculations.

cond-mat.mtrl-sci↗

arXiv2025

Question-Aware Gaussian Experts for Audio-Visual Question Answering

Audio-Visual Question Answering (AVQA) requires not only question-based multimodal reasoning but also precise temporal grounding to capture subtle dynamics for accurate prediction. However, existing methods mainly use question information implicitly, limiting focus on question-specific details. Furthermore, most studies rely on uniform frame sampling, which can miss key question-relevant frames. Although recent Top-K frame selection methods aim to address this, their discrete nature still overlooks fine-grained temporal details. This paper proposes QA-TIGER, a novel framework that explicitly incorporates question information and models continuous temporal dynamics. Our key idea is to use Gaussian-based modeling to adaptively focus on both consecutive and non-consecutive frames based on the question, while explicitly injecting question information and applying progressive refinement. We leverage a Mixture of Experts (MoE) to flexibly implement multiple Gaussian models, activating temporal experts specifically tailored to the question. Extensive experiments on multiple AVQA benchmarks show that QA-TIGER consistently achieves state-of-the-art performance. Code is available at https://aim-skku.github.io/QA-TIGER/

cs.CV↗

arXiv2025

Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders

We address the problem of gaze target estimation, which aims to predict where a person is looking in a scene. Predicting a person's gaze target requires reasoning both about the person's appearance and the contents of the scene. Prior works have developed increasingly complex, hand-crafted pipelines for gaze target estimation that carefully fuse features from separate scene encoders, head encoders, and auxiliary models for signals like depth and pose. Motivated by the success of general-purpose feature extractors on a variety of visual tasks, we propose Gaze-LLE, a novel transformer framework that streamlines gaze target estimation by leveraging features from a frozen DINOv2 encoder. We extract a single feature representation for the scene, and apply a person-specific positional prompt to decode gaze with a lightweight module. We demonstrate state-of-the-art performance across several gaze benchmarks and provide extensive analysis to validate our design choices. Our code is available at: http://github.com/fkryan/gazelle .

cs.CV↗