arXiv subjects

Minsu Kim

Publications and source records attributed to Minsu Kim.

At least 145 records · Page 8Linked to original sources

arXiv2022

Transformer Network-based Reinforcement Learning Method for Power Distribution Network (PDN) Optimization of High Bandwidth Memory (HBM)

In this article, for the first time, we propose a transformer network-based reinforcement learning (RL) method for power distribution network (PDN) optimization of high bandwidth memory (HBM). The proposed method can provide an optimal decoupling capacitor (decap) design to maximize the reduction of PDN self- and transfer impedance seen at multiple ports. An attention-based transformer network is implemented to directly parameterize decap optimization policy. The optimality performance is significantly improved since the attention mechanism has powerful expression to explore massive combinatorial space for decap assignments. Moreover, it can capture sequential relationships between the decap assignments. The computing time for optimization is dramatically reduced due to the reusable network on positions of probing ports and decap assignment candidates. This is because the transformer network has a context embedding process to capture meta-features including probing ports positions. In addition, the network is trained with randomly generated data sets. Therefore, without additional training, the trained network can solve new decap optimization problems. The computing time for training and data cost are critically decreased due to the scalability of the network. Thanks to its shared weight property, the network can adapt to a larger scale of problems without additional training. For verification, we compare the results with conventional genetic algorithm (GA), random search (RS), and all the previous RL-based methods. As a result, the proposed method outperforms in all the following aspects: optimality performance, computing time, and data efficiency.

cs.LG↗

arXiv2022

Speaker-adaptive Lip Reading with User-dependent Padding

Lip reading aims to predict speech based on lip movements alone. As it focuses on visual information to model the speech, its performance is inherently sensitive to personal lip appearances and movements. This makes the lip reading models show degraded performance when they are applied to unseen speakers due to the mismatch between training and testing conditions. Speaker adaptation technique aims to reduce this mismatch between train and test speakers, thus guiding a trained model to focus on modeling the speech content without being intervened by the speaker variations. In contrast to the efforts made in audio-based speech recognition for decades, the speaker adaptation methods have not well been studied in lip reading. In this paper, to remedy the performance degradation of lip reading model on unseen speakers, we propose a speaker-adaptive lip reading method, namely user-dependent padding. The user-dependent padding is a speaker-specific input that can participate in the visual feature extraction stage of a pre-trained lip reading model. Therefore, the lip appearances and movements information of different speakers can be considered during the visual feature encoding, adaptively for individual speakers. Moreover, the proposed method does not need 1) any additional layers, 2) to modify the learned weights of the pre-trained model, and 3) the speaker label of train data used during pre-train. It can directly adapt to unseen speakers by learning the user-dependent padding only, in a supervised or unsupervised manner. Finally, to alleviate the speaker information insufficiency in public lip reading databases, we label the speaker of a well-known audio-visual database, LRW, and design an unseen-speaker lip reading scenario named LRW-ID.

cs.CV↗

arXiv2022

VisageSynTalk: Unseen Speaker Video-to-Speech Synthesis via Speech-Visage Feature Selection

The goal of this work is to reconstruct speech from a silent talking face video. Recent studies have shown impressive performance on synthesizing speech from silent talking face videos. However, they have not explicitly considered on varying identity characteristics of different speakers, which place a challenge in the video-to-speech synthesis, and this becomes more critical in unseen-speaker settings. Our approach is to separate the speech content and the visage-style from a given silent talking face video. By guiding the model to independently focus on modeling the two representations, we can obtain the speech of high intelligibility from the model even when the input video of an unseen subject is given. To this end, we introduce speech-visage selection that separates the speech content and the speaker identity from the visual features of the input video. The disentangled representations are jointly incorporated to synthesize speech through visage-style based synthesizer which generates speech by coating the visage-styles while maintaining the speech content. Thus, the proposed framework brings the advantage of synthesizing the speech containing the right content even with the silent talking face video of an unseen subject. We validate the effectiveness of the proposed framework on the GRID, TCD-TIMIT volunteer, and LRW datasets.

cs.CV↗

arXiv2022

Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition

This paper focuses on designing a noise-robust end-to-end Audio-Visual Speech Recognition (AVSR) system. To this end, we propose Visual Context-driven Audio Feature Enhancement module (V-CAFE) to enhance the input noisy audio speech with a help of audio-visual correspondence. The proposed V-CAFE is designed to capture the transition of lip movements, namely visual context and to generate a noise reduction mask by considering the obtained visual context. Through context-dependent modeling, the ambiguity in viseme-to-phoneme mapping can be refined for mask generation. The noisy representations are masked out with the noise reduction mask resulting in enhanced audio features. The enhanced audio features are fused with the visual features and taken to an encoder-decoder model composed of Conformer and Transformer for speech recognition. We show the proposed end-to-end AVSR with the V-CAFE can further improve the noise-robustness of AVSR. The effectiveness of the proposed method is evaluated in noisy speech recognition and overlapped speech recognition experiments using the two largest audio-visual datasets, LRS2 and LRS3.

cs.SD↗

arXiv2022

CoVA: Exploiting Compressed-Domain Analysis to Accelerate Video Analytics

Modern retrospective analytics systems leverage cascade architecture to mitigate bottleneck for computing deep neural networks (DNNs). However, the existing cascades suffer two limitations: (1) decoding bottleneck is either neglected or circumvented, paying significant compute and storage cost for pre-processing; and (2) the systems are specialized for temporal queries and lack spatial query support. This paper presents CoVA, a novel cascade architecture that splits the cascade computation between compressed domain and pixel domain to address the decoding bottleneck, supporting both temporal and spatial queries. CoVA cascades analysis into three major stages where the first two stages are performed in compressed domain while the last one in pixel domain. First, CoVA detects occurrences of moving objects (called blobs) over a set of compressed frames (called tracks). Then, using the track results, CoVA prudently selects a minimal set of frames to obtain the label information and only decode them to compute the full DNNs, alleviating the decoding bottleneck. Lastly, CoVA associates tracks with labels to produce the final analysis results on which users can process both temporal and spatial queries. Our experiments demonstrate that CoVA offers 4.8x throughput improvement over modern cascade systems, while imposing modest accuracy loss.

cs.CV↗

arXiv2022

CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation

Code generation is a longstanding challenge, aiming to generate a code snippet based on a natural language description. Usually, expensive text-code paired data is essential for training a code generation model. Recently, thanks to the success of pre-training techniques, large language models are trained on large-scale unlabelled code corpora and perform well in code generation. In this paper, we investigate how to leverage an unlabelled code corpus to train a model for library-oriented code generation. Since it is a common practice for programmers to reuse third-party libraries, in which case the text-code paired data are harder to obtain due to the huge number of libraries. We observe that library-oriented code snippets are more likely to share similar code sketches. Hence, we present CERT with two steps: a sketcher generates the sketch, then a generator fills the details in the sketch. Both the sketcher and the generator are continually pre-trained upon a base model using unlabelled data. Furthermore, we craft two benchmarks named PandasEval and NumpyEval to evaluate library-oriented code generation. Experimental results demonstrate the impressive performance of CERT. For example, it surpasses the base model by an absolute 15.67% improvement in terms of pass@1 on PandasEval. Our work is available at https://github.com/microsoft/PyCodeGPT.

cs.SE↗

arXiv2022

Multi-modality Associative Bridging through Memory: Speech Sound Recollected from Face Video

In this paper, we introduce a novel audio-visual multi-modal bridging framework that can utilize both audio and visual information, even with uni-modal inputs. We exploit a memory network that stores source (i.e., visual) and target (i.e., audio) modal representations, where source modal representation is what we are given, and target modal representations are what we want to obtain from the memory network. We then construct an associative bridge between source and target memories that considers the interrelationship between the two memories. By learning the interrelationship through the associative bridge, the proposed bridging framework is able to obtain the target modal representations inside the memory network, even with the source modal input only, and it provides rich information for its downstream tasks. We apply the proposed framework to two tasks: lip reading and speech reconstruction from silent video. Through the proposed associative bridge and modality-specific memories, each task knowledge is enriched with the recalled audio context, achieving state-of-the-art performance. We also verify that the associative bridge properly relates the source and target memories.

cs.CV↗

arXiv2022

Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading

Recognizing speech from silent lip movement, which is called lip reading, is a challenging task due to 1) the inherent information insufficiency of lip movement to fully represent the speech, and 2) the existence of homophenes that have similar lip movement with different pronunciations. In this paper, we try to alleviate the aforementioned two challenges in lip reading by proposing a Multi-head Visual-audio Memory (MVM). Firstly, MVM is trained with audio-visual datasets and remembers audio representations by modelling the inter-relationships of paired audio-visual representations. At the inference stage, visual input alone can extract the saved audio representation from the memory by examining the learned inter-relationships. Therefore, the lip reading model can complement the insufficient visual information with the extracted audio representations. Secondly, MVM is composed of multi-head key memories for saving visual features and one value memory for saving audio knowledge, which is designed to distinguish the homophenes. With the multi-head key memories, MVM extracts possible candidate audio features from the memory, which allows the lip reading model to consider the possibility of which pronunciations can be represented from the input lip movement. This also can be viewed as an explicit implementation of the one-to-many mapping of viseme-to-phoneme. Moreover, MVM is employed in multi-temporal levels to consider the context when retrieving the memory and distinguish the homophenes. Extensive experimental results verify the effectiveness of the proposed method in lip reading and in distinguishing the homophenes.

cs.CV↗

arXiv2022

Lip to Speech Synthesis with Visual Context Attentional GAN

In this paper, we propose a novel lip-to-speech generative adversarial network, Visual Context Attentional GAN (VCA-GAN), which can jointly model local and global lip movements during speech synthesis. Specifically, the proposed VCA-GAN synthesizes the speech from local lip visual features by finding a mapping function of viseme-to-phoneme, while global visual context is embedded into the intermediate layers of the generator to clarify the ambiguity in the mapping induced by homophene. To achieve this, a visual context attention module is proposed where it encodes global representations from the local visual features, and provides the desired global visual context corresponding to the given coarse speech representation to the generator through audio-visual attention. In addition to the explicit modelling of local and global visual representations, synchronization learning is introduced as a form of contrastive learning that guides the generator to synthesize a speech in sync with the given input lip movements. Extensive experiments demonstrate that the proposed VCA-GAN outperforms existing state-of-the-art and is able to effectively synthesize the speech from multi-speaker that has been barely handled in the previous works.

cs.CV↗

arXiv2022

Facing to Latency of Hyperledger Fabric for Blockchain-enabled IoT: Modeling and Analysis

Hyperledger Fabric (HLF), one of the most popular private blockchains, has recently received attention for blockchain-enabled Internet of Things (IoT). However, for IoT applications to handle time-sensitive data, the processing latency in HLF has emerged as a new challenge. In this article, therefore, we establish a practical HLF latency model for HLF-enabled IoT. We first discuss the structure and transaction flow of HLF-enabled IoT. After implementing real HLF, we capture the latencies that each transaction experiences and show that the total latency of HLF can be modeled as a Gamma distribution, which is validated by conducting a goodness-of-fit test (i.e., the Kolmogorov-Smirnov (KS) test). We also provide the parameter values of the modeled latency distribution for various HLF environments. Furthermore, we explore the impacts of three important HLF parameters including transaction generation rate, block size, and block-generation timeout on the HLF latency. As a result, this article provides design insights on minimizing the latency for HLF-enabled IoT.

cs.DC↗

arXiv2021

Learning Collaborative Policies to Solve NP-hard Routing Problems

Recently, deep reinforcement learning (DRL) frameworks have shown potential for solving NP-hard routing problems such as the traveling salesman problem (TSP) without problem-specific expert knowledge. Although DRL can be used to solve complex problems, DRL frameworks still struggle to compete with state-of-the-art heuristics showing a substantial performance gap. This paper proposes a novel hierarchical problem-solving strategy, termed learning collaborative policies (LCP), which can effectively find the near-optimum solution using two iterative DRL policies: the seeder and reviser. The seeder generates as diversified candidate solutions as possible (seeds) while being dedicated to exploring over the full combinatorial action space (i.e., sequence of assignment action). To this end, we train the seeder's policy using a simple yet effective entropy regularization reward to encourage the seeder to find diverse solutions. On the other hand, the reviser modifies each candidate solution generated by the seeder; it partitions the full trajectory into sub-tours and simultaneously revises each sub-tour to minimize its traveling distance. Thus, the reviser is trained to improve the candidate solution's quality, focusing on the reduced solution space (which is beneficial for exploitation). Extensive experiments demonstrate that the proposed two-policies collaboration scheme improves over single-policy DRL framework on various NP-hard routing problems, including TSP, prize collecting TSP (PCTSP), and capacitated vehicle routing problem (CVRP).

cs.LG↗

arXiv2021

Non-Terrestrial Networks for UAVs: Base Station Service Provisioning Schemes with Antenna Tilt

By focusing on unmanned aerial vehicle (UAV) communications in non-terrestrial networks (NTNs), this paper provides a guideline on the appropriate base station (BS) service provisioning scheme with considering the antenna tilt angle of BS. Specifically, two service provisioning schemes are considered including the inclusive-service BS (IS-BS) scheme, which makes BSs serve both ground users (GUs) and aerial users (AUs) (i.e., UAVs) simultaneously, and the exclusive-service BS (ES-BS) scheme, which has BSs for GUs and BSs for AUs. By considering the antenna tilt angle-based channel gain, we derive the network outage probability for both IS-BS and ES-BS schemes, and show the existence of the optimal tilt angle that minimizes the network outage probability after analyzing the conflict impact of the antenna tilt angle. We also analyze the impact of various network parameters, including the ratio of GUs to total users and densities of total and interfering BSs, on the network outage probability. Finally, we analytically and numerically show in which environments each service provisioning scheme can be superior to the other one.

eess.SP↗

arXiv2021

Learning Canonical 3D Object Representation for Fine-Grained Recognition

We propose a novel framework for fine-grained object recognition that learns to recover object variation in 3D space from a single image, trained on an image collection without using any ground-truth 3D annotation. We accomplish this by representing an object as a composition of 3D shape and its appearance, while eliminating the effect of camera viewpoint, in a canonical configuration. Unlike conventional methods modeling spatial variation in 2D images only, our method is capable of reconfiguring the appearance feature in a canonical 3D space, thus enabling the subsequent object classifier to be invariant under 3D geometric variation. Our representation also allows us to go beyond existing methods, by incorporating 3D shape variation as an additional cue for object recognition. To learn the model without ground-truth 3D annotation, we deploy a differentiable renderer in an analysis-by-synthesis framework. By incorporating 3D shape and appearance jointly in a deep representation, our method learns the discriminative representation of the object and achieves competitive performance on fine-grained image recognition and vehicle re-identification. We also demonstrate that the performance of 3D shape reconstruction is improved by learning fine-grained shape deformation in a boosting manner.

cs.CV↗

arXiv2021

Precoding Design for Multi-user MIMO Systems with Delay-Constrained and -Tolerant Users

In both academia and industry, multi-user multiple-input multiple-output (MU-MIMO) techniques have shown enormous gains in spectral efficiency by exploiting spatial degrees of freedom. So far, an underlying assumption in most of the existing MU-MIMO design has been that all the users use infinite blocklength, so that they can achieve the Shannon capacity. This setup, however, is not suitable considering delay-constrained users whose blocklength tends to be finite. In this paper, we consider a heterogeneous setting in MU-MIMO systems where delay-constrained users and delay-tolerant users coexist, called a DCTU-MIMO network. To maximize the sum spectral efficiency in this system, we present the spectral efficiency for delay-tolerant users and provide a lower bound of the spectral efficiency for delay-constrained users. We consider an optimization problem that maximizes the sum spectral efficiency of delay-tolerant users while satisfying the latency constraint of delay-constrained users, and propose a generalized power iteration (GPI) precoding algorithm that finds a principal precoding vector. Furthermore, we extend a DCTU-MIMO network to the multiple time slots scenario and propose a recursive generalized power iteration precoding algorithm. In simulation results, we prove proposed methods outperform baseline schemes and present the effect of network parameters on the ergodic sum spectral efficiency.

eess.SP↗

arXiv2021

Ensuring Data Freshness for Blockchain-enabled Monitoring Networks

The age of information (AoI) is a recently proposed metric for quantifying data freshness in real-time status monitoring systems where timeliness is of importance. In this paper, the problem of characterizing and controlling the AoI is studied in the context of blockchain-enabled monitoring networks (BeMN). In BeMN, status updates from sources are transmitted and recorded in a blockchain. To investigate the statistical characteristics of the AoI in BeMN, the transmission latency and the consensus latency are first rigorously modeled. Then, the average AoI, the AoI violation probability, and the peak AoI violation probability are derived in a closed form so as to quantify the performance of BeMN. Furthermore, a simplified form is derived for the AoI violation probability, and it is shown that this quantity can capture the upper or lower bound of the actual AoI violation probability. Simulation results show that each BeMN parameters (i.e., target successful transmission probability, block size, and timeout) can have conflicting effects on the AoI-related performance. Subsequently, design insights are provided to maintain the freshness of the status data in BeMN. Then, experimental results with a real Hyperledger Fabric platform further validate the accuracy of our modeling and analysis.

eess.SP↗

arXiv2021

Securing Fresh Data in Wireless Monitoring Networks: Age-of-Information Sensitive Coverage Perspective

With the development of IoT, the sensor usage has been elevated to a new level, and it becomes more crucial to maintain reliable sensor networks. In this paper, we provide how to efficiently and reliably manage the sensor monitoring system for securing fresh data at the data center (DC). A sensor transmits its sensing information regularly to the DC, and the freshness of the information at the DC is characterized by the age of information (AoI) that quantifies the timeliness of information. By considering the effect of the AoI and the spatial distance from the sensor on the information error at the DC, we newly define an error-tolerable sensing (ETS) coverage as the area that the estimated information is with smaller error than the target value. We then derive the average AoI and the AoI violation probability of the sensor monitoring system, and finally present the η-coverage probability, which is the probability that the ETS coverage is greater than η ratio of the maximum sensor coverage. We also provide the optimal transmission power of the sensor, which minimizes the average energy consumption while guaranteeing certain level of the η-coverage probability. Numerical results validate the theoretical analysis and show the tendency of the optimal transmission power according to the maximum number of retransmissions. This paper can pave the way to efficient design of the AoI-sensitive sensor networks for IoT.

eess.SY↗

arXiv2020

Cross-Domain Grouping and Alignment for Domain Adaptive Semantic Segmentation

Existing techniques to adapt semantic segmentation networks across the source and target domains within deep convolutional neural networks (CNNs) deal with all the samples from the two domains in a global or category-aware manner. They do not consider an inter-class variation within the target domain itself or estimated category, providing the limitation to encode the domains having a multi-modal data distribution. To overcome this limitation, we introduce a learnable clustering module, and a novel domain adaptation framework called cross-domain grouping and alignment. To cluster the samples across domains with an aim to maximize the domain alignment without forgetting precise segmentation ability on the source domain, we present two loss functions, in particular, for encouraging semantic consistency and orthogonality among the clusters. We also present a loss so as to solve a class imbalance problem, which is the other limitation of the previous methods. Our experiments show that our method consistently boosts the adaptation performance in semantic segmentation, outperforming the state-of-the-arts on various domain adaptation settings.

cs.CV↗

arXiv2020

Securing Communications with Friendly Unmanned Aerial Vehicle Jammers

In this paper, we analyze the impact of a friendly unmanned aerial vehicle (UAV) jammer on UAV communications in the presence of multiple eavesdroppers. We first present channel components determined by the line-of-sight (LoS) probability between the friendly UAV jammer and the ground device, and introduce different channel fadings for LoS and non-line-of-sight (NLoS) links. We then derive the secrecy transmission probability satisfying both constraints of legitimate and wiretap channels. We also analyze the secrecy transmission probability in the presence of randomly distributed multiple friendly UAV jammers. Finally, we show the existence of the optimal UAV jammer location, and the impact of the density of eavesdroppers, the transmission power of the UAV jammer, and the density of UAV jammers on the optimal location.

eess.SP↗