arXiv subjects

Kun Song

Publications and source records attributed to Kun Song.

33 records · Page 2Linked to original sources

arXiv2023

The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task

This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.

cs.SD↗

arXiv2023

DSPGAN: a GAN-based universal vocoder for high-fidelity TTS by time-frequency domain supervision from DSP

Recent development of neural vocoders based on the generative adversarial neural network (GAN) has shown obvious advantages of generating raw waveform conditioned on mel-spectrogram with fast inference speed and lightweight networks. Whereas, it is still challenging to train a universal neural vocoder that can synthesize high-fidelity speech from various scenarios with unseen speakers, languages, and speaking styles. In this paper, we propose DSPGAN, a GAN-based universal vocoder for high-fidelity speech synthesis by applying the time-frequency domain supervision from digital signal processing (DSP). To eliminate the mismatch problem caused by the ground-truth spectrograms in the training phase and the predicted spectrograms in the inference phase, we leverage the mel-spectrogram extracted from the waveform generated by a DSP module, rather than the predicted mel-spectrogram from the Text-to-Speech (TTS) acoustic model, as the time-frequency domain supervision to the GAN-based vocoder. We also utilize sine excitation as the time-domain supervision to improve the harmonic modeling and eliminate various artifacts of the GAN-based vocoder. Experiments show that DSPGAN significantly outperforms the compared approaches and it can generate high-fidelity speech for various TTS models trained using diverse data.

cs.SD↗

arXiv2023

Multi-Speaker Expressive Speech Synthesis via Multiple Factors Decoupling

This paper aims to synthesize the target speaker's speech with desired speaking style and emotion by transferring the style and emotion from reference speech recorded by other speakers. We address this challenging problem with a two-stage framework composed of a text-to-style-and-emotion (Text2SE) module and a style-and-emotion-to-wave (SE2Wave) module, bridging by neural bottleneck (BN) features. To further solve the multi-factor (speaker timbre, speaking style and emotion) decoupling problem, we adopt the multi-label binary vector (MBV) and mutual information (MI) minimization to respectively discretize the extracted embeddings and disentangle these highly entangled factors in both Text2SE and SE2Wave modules. Moreover, we introduce a semi-supervised training strategy to leverage data from multiple speakers, including emotion-labeled data, style-labeled data, and unlabeled data. To better transfer the fine-grained expression from references to the target speaker in non-parallel transfer, we introduce a reference-candidate pool and propose an attention-based reference selection approach. Extensive experiments demonstrate the good design of our model.

eess.AS↗

arXiv2023

Gestalt-Guided Image Understanding for Few-Shot Learning

Due to the scarcity of available data, deep learning does not perform well on few-shot learning tasks. However, human can quickly learn the feature of a new category from very few samples. Nevertheless, previous work has rarely considered how to mimic human cognitive behavior and apply it to few-shot learning. This paper introduces Gestalt psychology to few-shot learning and proposes Gestalt-Guided Image Understanding, a plug-and-play method called GGIU. Referring to the principle of totality and the law of closure in Gestalt psychology, we design Totality-Guided Image Understanding and Closure-Guided Image Understanding to extract image features. After that, a feature estimation module is used to estimate the accurate features of images. Extensive experiments demonstrate that our method can improve the performance of existing models effectively and flexibly without retraining or fine-tuning. Our code is released on https://github.com/skingorz/GGIU.

cs.CV↗

arXiv2023

Distribution Aligned Feature Clustering for Zero-Shot Sketch-Based Image Retrieval

Zero-Shot Sketch-Based Image Retrieval (ZS-SBIR) is a challenging cross-modal retrieval task. In prior arts, the retrieval is conducted by sorting the distance between the query sketch and each image in the gallery. However, the domain gap and the zero-shot setting make neural networks hard to generalize. This paper tackles the challenges from a new perspective: utilizing gallery image features. We propose a Cluster-then-Retrieve (ClusterRetri) method that performs clustering on the gallery images and uses the cluster centroids as proxies for retrieval. Furthermore, a distribution alignment loss is proposed to align the image and sketch features with a common Gaussian distribution, reducing the domain gap. Despite its simplicity, our proposed method outperforms the state-of-the-art methods by a large margin on popular datasets, e.g., up to 31% and 39% relative improvement of mAP@all on the Sketchy and TU-Berlin datasets.

cs.CV↗

arXiv2022

AdaVITS: Tiny VITS for Low Computing Resource Speaker Adaptation

Speaker adaptation in text-to-speech synthesis (TTS) is to finetune a pre-trained TTS model to adapt to new target speakers with limited data. While much effort has been conducted towards this task, seldom work has been performed for low computational resource scenarios due to the challenges raised by the requirement of the lightweight model and less computational complexity. In this paper, a tiny VITS-based TTS model, named AdaVITS, for low computing resource speaker adaptation is proposed. To effectively reduce parameters and computational complexity of VITS, an iSTFT-based wave construction decoder is proposed to replace the upsampling-based decoder which is resource-consuming in the original VITS. Besides, NanoFlow is introduced to share the density estimate across flow blocks to reduce the parameters of the prior encoder. Furthermore, to reduce the computational complexity of the textual encoder, scaled-dot attention is replaced with linear attention. To deal with the instability caused by the simplified model, instead of using the original text encoder, phonetic posteriorgram (PPG) is utilized as linguistic feature via a text-to-PPG module, which is then used as input for the encoder. Experiment shows that AdaVITS can generate stable and natural speech in speaker adaptation with 8.97M model parameters and 0.72GFlops computational complexity.

cs.SD↗

arXiv2022

Robust MelGAN: A robust universal neural vocoder for high-fidelity TTS

In current two-stage neural text-to-speech (TTS) paradigm, it is ideal to have a universal neural vocoder, once trained, which is robust to imperfect mel-spectrogram predicted from the acoustic model. To this end, we propose Robust MelGAN vocoder by solving the original multi-band MelGAN's metallic sound problem and increasing its generalization ability. Specifically, we introduce a fine-grained network dropout strategy to the generator. With a specifically designed over-smooth handler which separates speech signal intro periodic and aperiodic components, we only perform network dropout to the aperodic components, which alleviates metallic sounding and maintains good speaker similarity. To further improve generalization ability, we introduce several data augmentation methods to augment fake data in the discriminator, including harmonic shift, harmonic noise and phase noise. Experiments show that Robust MelGAN can be used as a universal vocoder, significantly improving sound quality in TTS systems built on various types of data.

cs.SD↗

arXiv2022

Adaptive neighborhood Metric learning

In this paper, we reveal that metric learning would suffer from serious inseparable problem if without informative sample mining. Since the inseparable samples are often mixed with hard samples, current informative sample mining strategies used to deal with inseparable problem may bring up some side-effects, such as instability of objective function, etc. To alleviate this problem, we propose a novel distance metric learning algorithm, named adaptive neighborhood metric learning (ANML). In ANML, we design two thresholds to adaptively identify the inseparable similar and dissimilar samples in the training procedure, thus inseparable sample removing and metric parameter learning are implemented in the same procedure. Due to the non-continuity of the proposed ANML, we develop an ingenious function, named \emph{log-exp mean function} to construct a continuous formulation to surrogate it, which can be efficiently solved by the gradient descent method. Similar to Triplet loss, ANML can be used to learn both the linear and deep embeddings. By analyzing the proposed method, we find it has some interesting properties. For example, when ANML is used to learn the linear embedding, current famous metric learning algorithms such as the large margin nearest neighbor (LMNN) and neighbourhood components analysis (NCA) are the special cases of the proposed ANML by setting the parameters different values. When it is used to learn deep features, the state-of-the-art deep metric learning algorithms such as Triplet loss, Lifted structure loss, and Multi-similarity loss become the special cases of ANML. Furthermore, the \emph{log-exp mean function} proposed in our method gives a new perspective to review the deep metric learning methods such as Prox-NCA and N-pairs loss. At last, promising experimental results demonstrate the effectiveness of the proposed method.

cs.LG↗

arXiv2021

The Multi-speaker Multi-style Voice Cloning Challenge 2021

The Multi-speaker Multi-style Voice Cloning Challenge (M2VoC) aims to provide a common sizable dataset as well as a fair testbed for the benchmarking of the popular voice cloning task. Specifically, we formulate the challenge to adapt an average TTS model to the stylistic target voice with limited data from target speaker, evaluated by speaker identity and style similarity. The challenge consists of two tracks, namely few-shot track and one-shot track, where the participants are required to clone multiple target voices with 100 and 5 samples respectively. There are also two sub-tracks in each track. For sub-track a, to fairly compare different strategies, the participants are allowed to use only the training data provided by the organizer strictly. For sub-track b, the participants are allowed to use any data publicly available. In this paper, we present a detailed explanation on the tasks and data used in the challenge, followed by a summary of submitted systems and evaluation results.

eess.AS↗

arXiv2021

Size and shape fluctuations of ultrasoft colloids

Ultrasoft colloidal particle fluctuates due to its flexibility. Such fluctuation is essential for colloidal structure and dynamics, but is challenging to quantify experimentally. We use dendrimers as a model system to study the fluctuation of ultrasoft colloids. By considering the dynamic polydispersity in the small-angle neutron scattering (SANS) model, and introducing the fluctuation of invasive water into the contrast in SANS, we reveal the fluctuating amplitudes of the size and shape of the dendrimer of generation 6 at finite concentrations. The size fluctuation is suppressed while the shape fluctuation increases as the weight fraction of dendrimers passes 11%. With neutron spin echo data, we suggest that such crossover originates from the competition between the inter- and intra-particle dynamics. Further investigation on lower-generation samples shows a contrary result, which suggests a structural basis for these dynamic phenomena.

cond-mat.soft↗

arXiv2019

Adaptive Nearest Neighbor: A General Framework for Distance Metric Learning

$K$-NN classifier is one of the most famous classification algorithms, whose performance is crucially dependent on the distance metric. When we consider the distance metric as a parameter of $K$-NN, learning an appropriate distance metric for $K$-NN can be seen as minimizing the empirical risk of $K$-NN. In this paper, we design a new type of continuous decision function of the $K$-NN classification rule which can be used to construct the continuous empirical risk function of $K$-NN. By minimizing this continuous empirical risk function, we obtain a novel distance metric learning algorithm named as adaptive nearest neighbor (ANN). We have proved that the current algorithms such as the large margin nearest neighbor (LMNN), neighbourhood components analysis (NCA) and the pairwise constraint methods are special cases of the proposed ANN by setting the parameter different values. Compared with the LMNN, NCA, and pairwise constraint methods, our method has a broader searching space which may contain better solutions. At last, extensive experiments on various data sets are conducted to demonstrate the effectiveness and efficiency of the proposed method.

cs.LG↗

arXiv2017

Low-loss Broadband Negative Refractive Index due to Nonresonant 2D Helical Chiral Metamaterial

We demonstrated a 2D helical chiral metamaterial that exhibits broadband strong optical activity resulting nonresonant Drude-like response. The strong chirality leads to broadband negative refractive index with high figure of merit (>90) and extremely low loss (<2% per layer). The optical activity are insensitive to the angles of incident electromagnetic waves, thereby enabling more flexibility in polarization manipulation applicaitons.

physics.optics↗

arXiv2017

Metamaterial Perfect Absorber Analyzed by a Meta-cavity Model Consisting of Multilayer Metasurfaces

We demonstrate that the metamaterial perfect absorber behaves as a meta-cavity bounded between a resonant metasurface and a metallic thin-film reflector. The perfect absorption is achieved by the Fabry-Perot cavity resonance via multiple reflections between the "quasi-open" boundary of resonator and the "closed" boundary of reflector. The characteristic features including angle independence, ultra-thin thickness and strong field localization can be well explained by this model. With this model, metamaterial perfect absorber can be redefined as a meta-cavity exhibiting high Q-factor, strong field enhancement and extremely high photonic density of states, thereby promising novel applications for high performance sensor, infrared photodetector and cavity quantum electrodynamics devices.

physics.optics↗

arXiv2016

Broadband angle- and permittivity-insensitive nondispersive optical activity based on chiral metamaterials

Because of the strong inherent resonances, the giant optical activity obtained via chiral metamaterials generally suffers from high dispersion, which has been a big stumbling block to broadband applications. In this paper, we propose a type of chiral metamaterial consisting of interconnected metal helix structures with four-fold symmetry, which exhibits nonresonant Drude-like response and can therefore avoid the highly dispersive optical activity resulting from resonances. It shows that the well-designed chiral metamaterial can achieve nondispersive and pure optical activity with high transmittance in a broadband frequency range. And the optical activity of multi-layer chiral metamaterials is proportional to the layer numbers of single-layer chiral metamaterial. Most remarkably, the broadband behaviors of nondispersive optical activity and high transmission are insensitive to the incident angles of electromagnetic waves and permittivity of dielectric substrate, thereby enabling more flexibility in polarization manipulation.

physics.optics↗

arXiv2015

Broadband dispersionless polarization rotation with composite chiral metamaterials

We propose a planar composite chiral metamaterial (CCMM) by symmetrically inserting a metallic mesh between two layers of conjugated gammadion resonators. As the elaborate CCMM operates in the frequency region of off-resonance, it therefore presents loss-less and dispersion-free features. It can achieve flat polarization rotation simultaneously accompanied with high transmission and extremely low ellipticity in a broad bandwidth. In the meanwhile, this intriguing CCMM shows more superiorities in polarization rotation power and operating bandwidth than the pure chiral metamaterial just composed of conjugated gammadions. Due to the fascinating properties, the proposed CCMM is greatly appealing for controlling the polarization state of the electromagnetic waves.

physics.optics↗