arXiv subjects

Youngjoon Jang

Publications and source records attributed to Youngjoon Jang.

43 records · Page 3Linked to original sources

arXiv2023

TalkNCE: Improving Active Speaker Detection with Talk-Aware Contrastive Learning

The goal of this work is Active Speaker Detection (ASD), a task to determine whether a person is speaking or not in a series of video frames. Previous works have dealt with the task by exploring network architectures while learning effective representations has been less explored. In this work, we propose TalkNCE, a novel talk-aware contrastive loss. The loss is only applied to part of the full segments where a person on the screen is actually speaking. This encourages the model to learn effective representations through the natural correspondence of speech and facial movements. Our loss can be jointly optimized with the existing objectives for training ASD models without the need for additional supervision or training data. The experiments demonstrate that our loss can be easily integrated into the existing ASD frameworks, improving their performance. Our method achieves state-of-the-art performances on AVA-ActiveSpeaker and ASW datasets.

cs.CV↗

arXiv2023

That's What I Said: Fully-Controllable Talking Face Generation

The goal of this paper is to synthesise talking faces with controllable facial motions. To achieve this goal, we propose two key ideas. The first is to establish a canonical space where every face has the same motion patterns but different identities. The second is to navigate a multimodal motion space that only represents motion-related features while eliminating identity information. To disentangle identity and motion, we introduce an orthogonality constraint between the two different latent spaces. From this, our method can generate natural-looking talking faces with fully controllable facial attributes and accurate lip synchronisation. Extensive experiments demonstrate that our method achieves state-of-the-art results in terms of both visual quality and lip-sync score. To the best of our knowledge, we are the first to develop a talking face generation framework that can accurately manifest full target facial motions including lip, head pose, and eye movements in the generated video without any additional supervision beyond RGB video with audio.

cs.CV↗

arXiv2023

Self-Sufficient Framework for Continuous Sign Language Recognition

The goal of this work is to develop self-sufficient framework for Continuous Sign Language Recognition (CSLR) that addresses key issues of sign language recognition. These include the need for complex multi-scale features such as hands, face, and mouth for understanding, and absence of frame-level annotations. To this end, we propose (1) Divide and Focus Convolution (DFConv) which extracts both manual and non-manual features without the need for additional networks or annotations, and (2) Dense Pseudo-Label Refinement (DPLR) which propagates non-spiky frame-level pseudo-labels by combining the ground truth gloss sequence labels with the predicted sequence. We demonstrate that our model achieves state-of-the-art performance among RGB-based methods on large-scale CSLR benchmarks, PHOENIX-2014 and PHOENIX-2014-T, while showing comparable results with better efficiency when compared to other approaches that use multi-modality or extra annotations.

cs.CV↗

arXiv2022

Metric Learning for User-defined Keyword Spotting

The goal of this work is to detect new spoken terms defined by users. While most previous works address Keyword Spotting (KWS) as a closed-set classification problem, this limits their transferability to unseen terms. The ability to define custom keywords has advantages in terms of user experience. In this paper, we propose a metric learning-based training strategy for user-defined keyword spotting. In particular, we make the following contributions: (1) we construct a large-scale keyword dataset with an existing speech corpus and propose a filtering method to remove data that degrade model training; (2) we propose a metric learning-based two-stage training strategy, and demonstrate that the proposed method improves the performance on the user-defined keyword spotting task by enriching their representations; (3) to facilitate the fair comparison in the user-defined KWS field, we propose unified evaluation protocol and metrics. Our proposed system does not require an incremental training on the user-defined keywords, and outperforms previous works by a significant margin on the Google Speech Commands dataset using the proposed as well as the existing metrics.

eess.AS↗

arXiv2022

Signing Outside the Studio: Benchmarking Background Robustness for Continuous Sign Language Recognition

The goal of this work is background-robust continuous sign language recognition. Most existing Continuous Sign Language Recognition (CSLR) benchmarks have fixed backgrounds and are filmed in studios with a static monochromatic background. However, signing is not limited only to studios in the real world. In order to analyze the robustness of CSLR models under background shifts, we first evaluate existing state-of-the-art CSLR models on diverse backgrounds. To synthesize the sign videos with a variety of backgrounds, we propose a pipeline to automatically generate a benchmark dataset utilizing existing CSLR benchmarks. Our newly constructed benchmark dataset consists of diverse scenes to simulate a real-world environment. We observe even the most recent CSLR method cannot recognize glosses well on our new dataset with changed backgrounds. In this regard, we also propose a simple yet effective training scheme including (1) background randomization and (2) feature disentanglement for CSLR models. The experimental results on our dataset demonstrate that our method generalizes well to other unseen background data with minimal additional training images.

cs.CV↗

arXiv2010

Pulse propagation through a dispersive intracavity medium

In this paper, we study theoretically the behavior of a pulse as it propagates through an intracavity fast-light medium. The method of using a transfer function to determine a pulse after it passes through a cavity is well known. However, this approach cannot be used to determine the behavior of the pulse inside the cavity. To circumvent this constraint, we use an approach that starts by finding a self-consistent solution for a monochromatic field of infinite spatial and temporal extents, and determine its amplitudes before, inside, and after the cavity. We then construct a Gaussian input pulse by adding a set of these waves, properly phased and weighted, to represent a moving pulse before the cavity. Adding these waves at various time intervals then yields the complete spatial profile everywhere, including before, inside and after the cavity. We first confirm the prediction of this model by analyzing the behavior of a pulse passing through an empty cavity, and comparing the prediction of the output with the one produced by the transfer function method. We then apply the technique to a cavity containing a fast-light medium. The resulting model allows us to visualize the behavior of the pulse as it propagates superluminally inside the cavity, and interferes with itself through multiple bounces. For a vanishing group index, an interference pattern is formed immediately after the pulse enters the cavity, with an output pulse emerging with no time delay or distortion. The results obtained here illustrates the physical mechanism behind pulse propagation through a white light cavity, a process we have proposed earlier for realizing a high bandwidth, long delay data buffering system

physics.optics↗

arXiv2010

Pulse delay via tunable white light cavities using fiber optic resonators

Previously, we proposed a data buffering system that makes use of a pair of white light cavities. For application to telecommunication systems, it would be convenient to realize such a device using fiber optic resonators. In this paper, we present the design of such a system, where the white light cavity effect is produced by using stimulated Brillouin scattering. The system consists of a pair of fiber optic white light cavities placed in series. As in the original proposal, the delay time can be controlled independently of the bandwidth of the data pulses. Furthermore, we show how the bandwidth of the system can be made as large as several times the Brillouin frequency shift. We also show that the net delay achievable in such a buffer can be significantly larger than what can be achieved using a conventional recirculating loop buffer.

physics.optics↗