arXiv subjects

Jaemin Jung

Publications and source records attributed to Jaemin Jung.

10 recordsLinked to original sources

arXiv2026

Inference-Time Scaling for Joint Audio-Video Generation

Joint audio-video generation aims to synthesize realistic audio-video pairs that are both semantically aligned with text prompts and precisely synchronized. While existing joint audio-video generation models often require substantial training resources to improve fidelity, Inference-Time Scaling (ITS) has recently emerged as a promising training-free alternative in single-modality domains. However, extending ITS from a single modality to multimodal domains is non-trivial, as it requires balancing multiple heterogeneous objectives. In this paper, we present the first comprehensive study of ITS for joint audio-video generation. We first demonstrate that a multi-verifier framework is essential to address the limitations of single-objective guidance, including asymmetric performance trade-offs and verifier hacking. Through systematic analysis, we then identify an optimal multi-verifier combination that yields balanced improvements across all quality dimensions. Finally, to effectively aggregate diverse reward signals, we propose Adaptive Reward Weighting (ARW), a novel test-time optimization algorithm. ARW treats reward aggregation as an online optimization problem, utilizing learnable parameters to calibrate reward variances without requiring prior knowledge of reward distributions, thereby ensuring robust multi-objective selection. Experimental results on VGGSound and JavisBench-mini benchmarks demonstrate that our framework significantly enhances semantic alignment, perceptual quality, and audio-visual synchronization of generated outputs. Synthesized samples and code are available on the project page: https://jung-jaemin.github.io/ITS-AVGen-Proj.

cs.MM↗

arXiv2025

Exploring the Near and Far-Field Coexistence for RIS-Assisted ISAC Systems: An Adaptive Bandwidth Splitting Approach

Integrated sensing and communication (ISAC) enables the joint use of spectrum and hardware resources for radar sensing and data transmission, serving as a key enabler of next-generation wireless networks. However, most existing ISAC studies have been limited to operation within a single frequency band and have not been designed to adapt to diverse wireless propagation environments or user configurations. To address these limitations, this paper investigates a reconfigurable intelligent surface (RIS)-assisted ISAC system employing an adaptive bandwidth-splitting strategy under near-field (NF) and far-field (FF) coexistence. The system comprises a full-duplex access point (AP), an RIS and multiple users, where an ISAC user (IU) is both a sensing target and a communication user in the NF region, while communication-only users (CUs) rely on the RIS and experience either NF or FF propagation depending on their placement. The proposed system jointly exploits traditional sensing-only (SO) and ISAC bands and adopts uplink non-orthogonal multiple access (NOMA) for simultaneous transmission. We formulate a joint optimization problem for the receive beamforming vector, bandwidth-splitting ratio, and RIS phase shifts to minimize the Cramer-Rao bound (CRB) under rate and resource constraints. An efficient algorithm is developed based on an alternating optimization (AO) framework combined with semi-definite relaxation (SDR). Numerical results demonstrate that the proposed approach significantly outperforms conventional schemes that operate solely in either the ISAC or SO band, achieving superior performance across various RIS and user configurations under hybrid NF and FF coexistence scenarios.

eess.SP↗

arXiv2025

SCORE: Scaling audio generation using Standardized COmposite REwards

The goal of this paper is to enhance Text-to-Audio generation at inference, focusing on generating realistic audio that precisely aligns with text prompts. Despite the rapid advancements, existing models often fail to achieve a reliable balance between perceptual quality and textual alignment. To address this, we adopt Inference-Time Scaling, a training-free method that improves performance by increasing inference computation. We establish its unexplored application to audio generation and propose a novel multi-reward guidance that equally signifies each component essential in perception. By normalizing each reward value into a common scale and combining them with a weighted summation, the method not only enforces stable guidance but also enables explicit control to reach desired aspects. Moreover, we introduce a new audio-text alignment metric using an audio language model for more robust evaluation. Empirically, our method improves both semantic alignment and perceptual quality, significantly outperforming naive generation and existing reward guidance techniques. Synthesized samples are available on our demo page: https://mm.kaist.ac.kr/projects/score

eess.AS↗

arXiv2025

Test-Time Augmentation for Pose-invariant Face Recognition

The goal of this paper is to enhance face recognition performance by augmenting head poses during the testing phase. Existing methods often rely on training on frontalised images or learning pose-invariant representations, yet both approaches typically require re-training and testing for each dataset, involving a substantial amount of effort. In contrast, this study proposes Pose-TTA, a novel approach that aligns faces at inference time without additional training. To achieve this, we employ a portrait animator that transfers the source image identity into the pose of a driving image. Instead of frontalising a side-profile face -- which can introduce distortion -- Pose-TTA generates matching side-profile images for comparison, thereby reducing identity information loss. Furthermore, we propose a weighted feature aggregation strategy to address any distortions or biases arising from the synthetic data, thus enhancing the reliability of the augmented images. Extensive experiments on diverse datasets and with various pre-trained face recognition models demonstrate that Pose-TTA consistently improves inference performance. Moreover, our method is straightforward to integrate into existing face recognition pipelines, as it requires no retraining or fine-tuning of the underlying recognition models.

cs.CV↗

arXiv2024

VoiceDiT: Dual-Condition Diffusion Transformer for Environment-Aware Speech Synthesis

We present VoiceDiT, a multi-modal generative model for producing environment-aware speech and audio from text and visual prompts. While aligning speech with text is crucial for intelligible speech, achieving this alignment in noisy conditions remains a significant and underexplored challenge in the field. To address this, we present a novel audio generation pipeline named VoiceDiT. This pipeline includes three key components: (1) the creation of a large-scale synthetic speech dataset for pre-training and a refined real-world speech dataset for fine-tuning, (2) the Dual-DiT, a model designed to efficiently preserve aligned speech information while accurately reflecting environmental conditions, and (3) a diffusion-based Image-to-Audio Translator that allows the model to bridge the gap between audio and image, facilitating the generation of environmental sound that aligns with the multi-modal prompts. Extensive experimental results demonstrate that VoiceDiT outperforms previous models on real-world datasets, showcasing significant improvements in both audio quality and modality integration.

eess.AS↗

arXiv2024

Bridging the Gap between Audio and Text using Parallel-attention for User-defined Keyword Spotting

This paper proposes a novel user-defined keyword spotting framework that accurately detects audio keywords based on text enrollment. Since audio data possesses additional acoustic information compared to text, there are discrepancies between these two modalities. To address this challenge, we present ParallelKWS, which utilises self- and cross-attention in a parallel architecture to effectively capture information both within and across the two modalities. We further propose a phoneme duration-based alignment loss that enforces the sequential correspondence between audio and text features. Extensive experimental results demonstrate that our proposed method achieves state-of-the-art performance on several benchmark datasets in both seen and unseen domains, without incorporating extra data beyond the dataset used in previous studies.

eess.AS↗

arXiv2023

Disentangling Structure and Style: Political Bias Detection in News by Inducing Document Hierarchy

We address an important gap in detecting political bias in news articles. Previous works that perform document classification can be influenced by the writing style of each news outlet, leading to overfitting and limited generalizability. Our approach overcomes this limitation by considering both the sentence-level semantics and the document-level rhetorical structure, resulting in a more robust and style-agnostic approach to detecting political bias in news articles. We introduce a novel multi-head hierarchical attention model that effectively encodes the structure of long documents through a diverse ensemble of attention heads. While journalism follows a formalized rhetorical structure, the writing style may vary by news outlet. We demonstrate that our method overcomes this domain dependency and outperforms previous approaches for robustness and accuracy. Further analysis and human evaluation demonstrate the ability of our model to capture common discourse structures in journalism. Our code is available at: https://github.com/xfactlab/emnlp2023-Document-Hierarchy

cs.CL↗

arXiv2023

WUDI: A Human Involved Self-Adaptive Framework to Prevent Childhood Obesity in Internet of Things Environment

The Internet of Things (IoT) connects people, devices, and information resources, in various domains to improve efficiency. The healthcare domain has been transformed by the integration of the IoT, leading to the development of digital healthcare solutions such as health monitoring, emergency detection, and remote operation. This integration has led to an increase in the health data collected from a variety of IoT sources. Consequently, advanced technologies are required to analyze health data, and artificial intelligence has been employed to extract meaningful insights from the data. Childhood overweight and obesity have emerged as some of the most serious global public health challenges, as they can lead to a variety of health-related problems and the early development of chronic diseases. To address this, a self-adaptive framework is proposed to prevent childhood obesity by using lifelog data from IoT environments, with human involvement being an important consideration in the framework. The framework uses an ensemble-based learning model to predict obesity using the lifelog data. Empirical experiments using lifelog data from smartphone applications were conducted to validate the effectiveness of human involvement and obesity prediction. The results demonstrated the efficiency of the proposed framework with human involvement in obesity prediction. The proposed framework can be applied in real-world healthcare services for childhood obesity.

cs.SE↗

arXiv2022

Metric Learning for User-defined Keyword Spotting

The goal of this work is to detect new spoken terms defined by users. While most previous works address Keyword Spotting (KWS) as a closed-set classification problem, this limits their transferability to unseen terms. The ability to define custom keywords has advantages in terms of user experience. In this paper, we propose a metric learning-based training strategy for user-defined keyword spotting. In particular, we make the following contributions: (1) we construct a large-scale keyword dataset with an existing speech corpus and propose a filtering method to remove data that degrade model training; (2) we propose a metric learning-based two-stage training strategy, and demonstrate that the proposed method improves the performance on the user-defined keyword spotting task by enriching their representations; (3) to facilitate the fair comparison in the user-defined KWS field, we propose unified evaluation protocol and metrics. Our proposed system does not require an incremental training on the user-defined keywords, and outperforms previous works by a significant margin on the Google Speech Commands dataset using the proposed as well as the existing metrics.

eess.AS↗

arXiv2017

Barrier Enabled IO Stack for Flash Storage

This work is dedicated to eliminating the overhead of guaranteeing the storage order in modern IO stack. The existing block device adopts prohibitively expensive resort in ensuring the storage order among write requests: interleaving successive write requests with transfer and flush. Exploiting the cache barrier command for the Flash storage, we overhaul the IO scheduler, the dispatch module and the filesystem so that these layers are orchestrated to preserve the ordering condition imposed by the application can be delivered to the storage. Key ingredients of Barrier Enabled IO stack are Epoch based IO scheduling, Order Preserving Dispatch, and Dual Mode Journaling. Barrier enabled IO stack successfully eliminates the root cause of excessive overhead in enforcing the storage order. Dual Mode Journaling in BarrierFS dedicates the separate threads to effectively decouple the control plane and data plane of the journal commit. We implement Barrier Enabled IO Stack in server as well as in mobile platform. SQLite performance increases by 270% and 75%, in server and in smartphone, respectively. Relaxing the durability of a transaction, SQLite performance and MySQL performance increases as much as by 73X and by 43X, respectively, in server storage.

cs.OS↗