Searcharxiv⌕ Search

arXiv subjects

Hyunjun Kim

Publications and source records attributed to Hyunjun Kim.

39 records · Page 3Linked to original sources

Speaker-adaptive Lip Reading with User-dependent Padding

Lip reading aims to predict speech based on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements. This makes the lip reading models show degraded performance when they are applied to unseen speakers due to the mismatch between training and testing conditions. Speaker adaptation technique aims to reduce this mismatch between train and test speakers, thus guiding a trained model to focus on modeling the speech content without being intervened by the speaker variations. In contrast to the efforts made in audio-based speech recognition for decades, the speaker adaptation methods have not well been studied in lip reading. In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding. The user-dependent padding is a speaker-specific input that can participate in the visual feature extraction stage of a pre-trained lip reading model. Therefore, the lip appearances and movements information of different speakers can be considered during the visual feature encoding, adaptively for individual speakers. Moreover, the proposed method does not need 1) any additional layers, 2) to modify the learned weights of the pre-trained model, and 3) the speaker label of train data used during pre-train. It can directly adapt to unseen speakers by learning the user-dependent padding only, in a supervised or unsupervised manner. Finally, to alleviate the speaker information insufficiency in public lip reading databases, we label the speaker of a well-known audio-visual database, LRW, and design an unseen-speaker lip reading scenario named LRW-ID.

cs.CV↗

Fast Monte-Carlo Approximation of the Attention Mechanism

We introduce Monte-Carlo Attention (MCA), a randomized approximation method for reducing the computational cost of self-attention mechanisms in Transformer architectures. MCA exploits the fact that the importance of each token in an input sequence varies with respect to their attention scores; thus, some degree of error can be tolerable when encoding tokens with low attention. Using approximate matrix multiplication, MCA applies different error bounds to encode input tokens such that those with low attention scores are computed with relaxed precision, whereas errors of salient elements are minimized. MCA can operate in parallel with other attention optimization schemes and does not require model modification. We study the theoretical error bounds and demonstrate that MCA reduces attention complexity (in FLOPS) for various Transformer models by up to 11$\times$ in GLUE benchmarks without compromising model accuracy.

cs.LG↗

Unified Sum-BER Performance Analysis of AF MIMO Beamforming in Two-Way Relay Networks

Unified performance analysis is carried out for amplify-and-forward (AF) multiple-input-multiple-output (MIMO) beamforming (BF) two-way relay networks in Rayleigh fading with five different relaying protocols including two novel protocols for better performance. As a result, a novel closed-form sum-bit error rate (BER) expression is presented in a unified expression for all protocols. A new closed-form high signal-to-noise-ratio (SNR) performance is also obtained in a single expression, and an analytical high-SNR gap expression between the five protocols is provided. We compare the performance of the five relaying protocols with respect to sum-BER with appropriately normalized rate and power, and show that the proposed protocol with four time slots outperforms other protocols when transmit powers from two sources are sufficiently different, and the one with three time slots dominates other protocols when multiple relay antennas are used, at high-SNR.

cs.IT↗