arXiv subjects

Wenbin Wang

Publications and source records attributed to Wenbin Wang.

At least 55 records · Page 3Linked to original sources

arXiv2024

USAT: A Universal Speaker-Adaptive Text-to-Speech Approach

Conventional text-to-speech (TTS) research has predominantly focused on enhancing the quality of synthesized speech for speakers in the training dataset. The challenge of synthesizing lifelike speech for unseen, out-of-dataset speakers, especially those with limited reference data, remains a significant and unresolved problem. While zero-shot or few-shot speaker-adaptive TTS approaches have been explored, they have many limitations. Zero-shot approaches tend to suffer from insufficient generalization performance to reproduce the voice of speakers with heavy accents. While few-shot methods can reproduce highly varying accents, they bring a significant storage burden and the risk of overfitting and catastrophic forgetting. In addition, prior approaches only provide either zero-shot or few-shot adaptation, constraining their utility across varied real-world scenarios with different demands. Besides, most current evaluations of speaker-adaptive TTS are conducted only on datasets of native speakers, inadvertently neglecting a vast portion of non-native speakers with diverse accents. Our proposed framework unifies both zero-shot and few-shot speaker adaptation strategies, which we term as "instant" and "fine-grained" adaptations based on their merits. To alleviate the insufficient generalization performance observed in zero-shot speaker adaptation, we designed two innovative discriminators and introduced a memory mechanism for the speech decoder. To prevent catastrophic forgetting and reduce storage implications for few-shot speaker adaptation, we designed two adapters and a unique adaptation procedure.

cs.SD↗

arXiv2024

Decentralized Feedback Optimization via Sensitivity Decoupling: Stability and Sub-optimality

Online feedback optimization is a controller design paradigm for optimizing the steady-state behavior of a dynamical system. It employs an optimization algorithm as a dynamic feedback controller and utilizes real-time measurements to bypass knowing exact plant dynamics and disturbances. Different from existing centralized settings, we present a fully decentralized feedback optimization controller for networked systems to lift the communication burden and improve scalability. We approximate the overall input-output sensitivity matrix through its diagonal elements, which capture local model information. For the closed-loop behavior, we characterize the stability and bound the sub-optimality due to decentralization. We prove that the proposed decentralized controller yields solutions that correspond to the Nash equilibria of a non-cooperative game.

math.OC↗

arXiv2024

WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge

Sentiment analysis is rapidly advancing by utilizing various data modalities (e.g., text, image). However, most previous works relied on superficial information, neglecting the incorporation of contextual world knowledge (e.g., background information derived from but beyond the given image and text pairs) and thereby restricting their ability to achieve better multimodal sentiment analysis (MSA). In this paper, we proposed a plug-in framework named WisdoM, to leverage the contextual world knowledge induced from the large vision-language models (LVLMs) for enhanced MSA. WisdoM utilizes LVLMs to comprehensively analyze both images and corresponding texts, simultaneously generating pertinent context. To reduce the noise in the context, we also introduce a training-free contextual fusion mechanism. Experiments across diverse granularities of MSA tasks consistently demonstrate that our approach has substantial improvements (brings an average +1.96% F1 score among five advanced methods) over several state-of-the-art methods.

cs.CL↗

arXiv2023

Dynamic Association Learning of Self-Attention and Convolution in Image Restoration

CNNs and Self attention have achieved great success in multimedia applications for dynamic association learning of self-attention and convolution in image restoration. However, CNNs have at least two shortcomings: 1) limited receptive field; 2) static weight of sliding window at inference, unable to cope with the content diversity.In view of the advantages and disadvantages of CNNs and Self attention, this paper proposes an association learning method to utilize the advantages and suppress their shortcomings, so as to achieve high-quality and efficient inpainting. We regard rain distribution reflects the degradation location and degree, in addition to the rain distribution prediction. Thus, we propose to refine background textures with the predicted degradation prior in an association learning manner. As a result, we accomplish image deraining by associating rain streak removal and background recovery, where an image deraining network and a background recovery network are designed for two subtasks. The key part of association learning is a novel multi-input attention module. It generates the degradation prior and produces the degradation mask according to the predicted rainy distribution. Benefited from the global correlation calculation of SA, MAM can extract the informative complementary components from the rainy input with the degradation mask, and then help accurate texture restoration. Meanwhile, SA tends to aggregate feature maps with self-attention importance, but convolution diversifies them to focus on the local textures. A hybrid fusion network involves one residual Transformer branch and one encoder-decoder branch. The former takes a few learnable tokens as input and stacks multi-head attention and feed-forward networks to encode global features of the image. The latter, conversely, leverages the multi-scale encoder-decoder to represent contexture knowledge.

cs.CV↗

arXiv2023

Generalizable Zero-Shot Speaker Adaptive Speech Synthesis with Disentangled Representations

While most research into speech synthesis has focused on synthesizing high-quality speech for in-dataset speakers, an equally essential yet unsolved problem is synthesizing speech for unseen speakers who are out-of-dataset with limited reference data, i.e., speaker adaptive speech synthesis. Many studies have proposed zero-shot speaker adaptive text-to-speech and voice conversion approaches aimed at this task. However, most current approaches suffer from the degradation of naturalness and speaker similarity when synthesizing speech for unseen speakers (i.e., speakers not in the training dataset) due to the poor generalizability of the model in out-of-distribution data. To address this problem, we propose GZS-TV, a generalizable zero-shot speaker adaptive text-to-speech and voice conversion model. GZS-TV introduces disentangled representation learning for both speaker embedding extraction and timbre transformation to improve model generalization and leverages the representation learning capability of the variational autoencoder to enhance the speaker encoder. Our experiments demonstrate that GZS-TV reduces performance degradation on unseen speakers and outperforms all baseline models in multiple datasets.

cs.SD↗

arXiv2023

MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild

Dynamic facial expression recognition (FER) databases provide important data support for affective computing and applications. However, most FER databases are annotated with several basic mutually exclusive emotional categories and contain only one modality, e.g., videos. The monotonous labels and modality cannot accurately imitate human emotions and fulfill applications in the real world. In this paper, we propose MAFW, a large-scale multi-modal compound affective database with 10,045 video-audio clips in the wild. Each clip is annotated with a compound emotional category and a couple of sentences that describe the subjects' affective behaviors in the clip. For the compound emotion annotation, each clip is categorized into one or more of the 11 widely-used emotions, i.e., anger, disgust, fear, happiness, neutral, sadness, surprise, contempt, anxiety, helplessness, and disappointment. To ensure high quality of the labels, we filter out the unreliable annotations by an Expectation Maximization (EM) algorithm, and then obtain 11 single-label emotion categories and 32 multi-label emotion categories. To the best of our knowledge, MAFW is the first in-the-wild multi-modal database annotated with compound emotion annotations and emotion-related captions. Additionally, we also propose a novel Transformer-based expression snippet feature learning method to recognize the compound emotions leveraging the expression-change relations among different emotions and modalities. Extensive experiments on MAFW database show the advantages of the proposed method over other state-of-the-art methods for both uni- and multi-modal FER. Our MAFW database is publicly available from https://mafw-database.github.io/MAFW.

cs.CV↗

arXiv2023

Spin Fluctuations in Sr$_{1.8}$La$_{0.2}$RuO$_4$

We use inelastic neutron scattering to study spin fluctuations in Sr$_{1.8}$La$_{0.2}$RuO$_4$, where Lanthanum doping triggers a Lifshitz transition by pushing the van Hove singularity in the $γ$ band to the Fermi energy. Strong spin fluctuations emerge at an incommensurate wave vector $\mathbf{Q}_{ic} = (0.3,0.3)$, corresponding to the nesting vector between $α$ and $β$ Fermi sheets. The incommensurate antiferromagnetic fluctuations shift toward $(0.25,0.25)$ with increasing energy up to ${\sim}110$ meV. By contrast, scatterings near the ferromagnetic wave vectors $\mathbf{Q} = (1,0)$ and $(1,1)$ remain featureless at all energies. This contradicts the weak-coupling perspective that suggests a sharp enhancement of ferromagnetic susceptibility due to the divergence of density of states in the associated $γ$ band. Our findings imply that ferromagnetic fluctuations in Sr$_2$RuO$_4$ and related materials do not fit into the weak-coupling paradigm, but instead are quasi-local fluctuations induced by Hund's coupling. This imposes significant constraints for the pairing mechanism involving spin fluctuations.

cond-mat.str-el↗

arXiv2023

Pose-disentangled Contrastive Learning for Self-supervised Facial Representation

Self-supervised facial representation has recently attracted increasing attention due to its ability to perform face understanding without relying on large-scale annotated datasets heavily. However, analytically, current contrastive-based self-supervised learning (SSL) still performs unsatisfactorily for learning facial representation. More specifically, existing contrastive learning (CL) tends to learn pose-invariant features that cannot depict the pose details of faces, compromising the learning performance. To conquer the above limitation of CL, we propose a novel Pose-disentangled Contrastive Learning (PCL) method for general self-supervised facial representation. Our PCL first devises a pose-disentangled decoder (PDD) with a delicately designed orthogonalizing regulation, which disentangles the pose-related features from the face-aware features; therefore, pose-related and other pose-unrelated facial information could be performed in individual subnetworks and do not affect each other's training. Furthermore, we introduce a pose-related contrastive learning scheme that learns pose-related information based on data augmentation of the same image, which would deliver more effective face-aware representation for various downstream tasks. We conducted linear evaluation on four challenging downstream facial understanding tasks, ie, facial expression recognition, face recognition, AU detection and head pose estimation. Experimental results demonstrate that our method significantly outperforms state-of-the-art SSL methods. Code is available at https://github.com/DreamMr/PCL}{https://github.com/DreamMr/PCL

cs.CV↗

arXiv2023

Super-resolution of Ray-tracing Channel Simulation via Attention Mechanism based Deep Learning Model

As an emerging approach, deep learning plays an increasingly influential role in channel modeling. Traditional ray tracing (RT) methods of channel modeling tend to be inefficient and expensive. In this paper, we present a super-resolution (SR) model for channel characteristics. Residual connection and attention mechanism are applied to this convolutional neural network (CNN) model. Experiments prove that the proposed model can reduce the noise interference generated in the SR process and solve the problem of low efficiency of RT. The mean absolute error of our channel SR model on the PL achieves the effect of 2.82 dB with scale factor 2, the same accuracy as RT took only 52\% of the time in theory. Compared with vision transformer (ViT), the proposed model also demonstrates less running time and computing cost in SR of channel characteristics.

eess.SP↗

arXiv2022

Electronically phase separated nano-network in antiferromagnetic insulating LaMnO3/PrMnO3/CaMnO3 tricolor superlattice

Strongly correlated materials often exhibit an electronic phase separation (EPS) phenomena whose domain pattern is random in nature. The ability to control the spatial arrangement of the electronic phases at microscopic scales is highly desirable for tailoring their macroscopic properties and/or designing novel electronic devices. Here we report the formation of EPS nanoscale network in a mono-atomically stacked LaMnO3/CaMnO3/PrMnO3 superlattice grown on SrTiO3 (STO) (001) substrate, which is known to have an antiferromagnetic (AFM) insulating ground state. The EPS nano-network is a consequence of an internal strain relaxation triggered by the structural domain formation of the underlying STO substrate at low temperatures. The same nanoscale network pattern can be reproduced upon temperature cycling allowing us to employ different local imaging techniques to directly compare the magnetic and transport state of a single EPS domain. Our results confirm the one-to-one correspondence between ferromagnetic (AFM) to metallic (insulating) state in manganite. It also represents a significant step in a paradigm shift from passively characterizing EPS in strongly correlated systems to actively engaging in its manipulation.

cond-mat.str-el↗

arXiv2022

AutoLV: Automatic Lecture Video Generator

We propose an end-to-end lecture video generation system that can generate realistic and complete lecture videos directly from annotated slides, instructor's reference voice and instructor's reference portrait video. Our system is primarily composed of a speech synthesis module with few-shot speaker adaptation and an adversarial learning-based talking-head generation module. It is capable of not only reducing instructors' workload but also changing the language and accent which can help the students follow the lecture more easily and enable a wider dissemination of lecture contents. Our experimental results show that the proposed model outperforms other current approaches in terms of authenticity, naturalness and accuracy. Here is a video demonstration of how our system works, and the outcomes of the evaluation and comparison: https://youtu.be/cY6TYkI0cog.

cs.MM↗

arXiv2022

Frustrated magnetic interactions in FeSe

The structurally simplest high-temperature superconductor FeSe exhibits an intriguing superconducting nematic paramagnetic phase with unusual spin excitation spectra that are different from typical spin waves; thus, determining its effective magnetic exchange interactions is challenging. Here we report neutron scattering measurements of spin fluctuations of FeSe in the tetragonal paramagnetic phase. We show that the equal-time magnetic structure factor, $\mathcal{S}(\textbf{Q})$, can be effectively modeled using the self-consistent Gaussian approximation calculation with highly frustrated nearest-neighbor ($J_{1}$) and next-nearest-neighbor ($J_{2}$) exchange couplings, and very weak further neighbor exchange interaction. Our results elucidate the frustrated magnetism in FeSe, which provides a natural explanation for the highly tunable superconductivity and nematicity in FeSe and related materials.

cond-mat.supr-con↗

arXiv2021

Pressure-induced charge orders and their coupling to magnetism in hexagonal multiferroic LuFe2O4

Hexagonal LuFe2O4 is a promising charge-order (CO) driven multiferroic material with high charge and spin ordering temperatures. The coexisting charge and spin orders on Fe3+/Fe2+ sites result in novel magnetoelectric behaviors, but the coupling mechanism between the charge and spin orders remains elusive. Here, by tuning external pressure, we reveal three correlated spin-charge ordered phases in LuFe2O4: i) a centrosymmetric incommensurate three-dimensional CO with ferrimagnetism, ii) a non-centrosymmetric incommensurate quasi-two-dimensional CO with ferrimagnetism, and iii) a centrosymmetric commensurate CO with antiferromagnetism. Experimental in-situ single-crystal X-ray diffraction and X-ray magnetic circular dichroism measurements combined with density functional theory calculations suggest that the charge density redistribution caused by pressure-induced compression in the frustrated double-layer [Fe2O4] cluster is responsible for the correlated spin-charge phase transitions. The pressure-enhanced effective Coulomb interactions among Fe-Fe bonds drive the frustrated (1/3, 1/3) CO to a less frustrated (1/4, 1/4) CO, which induces the ferrimagnetic to antiferromagnetic transition. Our results not only elucidate the coupling mechanism among charge, spin and lattice degrees of freedom in LuFe2O4 but also provide a new way to tune the spin-charge orders in a highly controlled manner.

cond-mat.mtrl-sci↗

arXiv2021

Expression Snippet Transformer for Robust Video-based Facial Expression Recognition

The recent success of Transformer has provided a new direction to various visual understanding tasks, including video-based facial expression recognition (FER). By modeling visual relations effectively, Transformer has shown its power for describing complicated patterns. However, Transformer still performs unsatisfactorily to notice subtle facial expression movements, because the expression movements of many videos can be too small to extract meaningful spatial-temporal relations and achieve robust performance. To this end, we propose to decompose each video into a series of expression snippets, each of which contains a small number of facial movements, and attempt to augment the Transformer's ability for modeling intra-snippet and inter-snippet visual relations, respectively, obtaining the Expression snippet Transformer (EST). In particular, for intra-snippet modeling, we devise an attention-augmented snippet feature extractor (AA-SFE) to enhance the encoding of subtle facial movements of each snippet by gradually attending to more salient information. In addition, for inter-snippet modeling, we introduce a shuffled snippet order prediction (SSOP) head and a corresponding loss to improve the modeling of subtle motion changes across subsequent snippets by training the Transformer to identify shuffled snippet orders. Extensive experiments on four challenging datasets (i.e., BU-3DFE, MMI, AFEW, and DFEW) demonstrate that our EST is superior to other CNN-based methods, obtaining state-of-the-art performance.

cs.CV↗

arXiv2020

Magnetoelectric coupling and decoupling in multiferroic hexagonal YbFeO3 thin films

The coupling between ferroelectric and magnetic orders in multiferroic materials and the nature of magnetoelectric (ME) effects are enduring experimental challenges. In this work, we have studied the response of magnetization to ferroelectric switching in thin-film hexagonal YbFeO3, a prototypical improper multiferroic. The bulk ME decoupling and potential domain-wall ME coupling were revealed using x-ray magnetic circular dichroism (XMCD) measurements with in-situ ferroelectric polarization switching. Our Landau theory analysis suggests that the bulk ME-coupled ferroelectric switching path has a higher energy barrier than that of the ME-decoupled path; this extra barrier energy is also too high to be reduced by the magneto-static energy in the process of breaking single magnetic domains into multi-domains. In addition, the reduction of magnetization around the ferroelectric domain walls predicted by the Landau theory may induce the domain-wall ME coupling in which the magnetization is correlated with the density of ferroelectric domain walls. These results provide important experimental evidence and theoretical insights into the rich possibilities of ME couplings in hexagonal ferrites, such as manipulating the magnetic states by an electric field.

cond-mat.mtrl-sci↗

arXiv2020

Sketching Image Gist: Human-Mimetic Hierarchical Scene Graph Generation

Scene graph aims to faithfully reveal humans' perception of image content. When humans analyze a scene, they usually prefer to describe image gist first, namely major objects and key relations in a scene graph. This humans' inherent perceptive habit implies that there exists a hierarchical structure about humans' preference during the scene parsing procedure. Therefore, we argue that a desirable scene graph should be also hierarchically constructed, and introduce a new scheme for modeling scene graph. Concretely, a scene is represented by a human-mimetic Hierarchical Entity Tree (HET) consisting of a series of image regions. To generate a scene graph based on HET, we parse HET with a Hybrid Long Short-Term Memory (Hybrid-LSTM) which specifically encodes hierarchy and siblings context to capture the structured information embedded in HET. To further prioritize key relations in the scene graph, we devise a Relation Ranking Module (RRM) to dynamically adjust their rankings by learning to capture humans' subjective perceptive habits from objective entity saliency and size. Experiments indicate that our method not only achieves state-of-the-art performances for scene graph generation, but also is expert in mining image-specific relations which play a great role in serving downstream tasks.

cs.CV↗

arXiv2019

Neutron Spin Resonance in the Heavily Hole-doped KFe$_{2}$As$_{2}$ Superconductor

We report high-resolution neutron scattering measurements of the low energy spin fluctuations of KFe$_{2}$As$_{2}$, the end member of the hole-doped Ba$_{1-x}$K$_x$Fe$_2$As$_2$ family with only hole pockets, above and below its superconducting transition temperature $T_c$ ($\sim$ 3.5 K). Our data reveals clear spin fluctuations at the incommensurate wave vector ($0.5\pmδ$, 0, $L$), ($δ$ = 0.2)(1-Fe unit cell), which exhibit $L$-modulation peaking at $L=0.5$. Upon cooling to the superconducting state, the incommensurate spin fluctuations gradually open a spin-gap and form a sharp spin resonance mode. The incommensurability ($2δ$ = 0.4) of the resonance mode ($\sim1.2$ meV) is considerably larger than the previously reported value ($2δ$ $\approx0.32$) at higher energies ($\ge\sim6$ meV). The determination of the momentum structure of spin fluctuation in the low energy limit allows a direct comparison with the realistic Fermi surface and superconducting gap structure. Our results point to an $s$-wave pairing with a reversed sign between the hole pockets near the zone center in KFe$_{2}$As$_{2}$.

cond-mat.supr-con↗

arXiv2019

Tuning the Magnetic Ordering Temperature of Hexagonal Ferrites by Structural Distortion Control

To tune the magnetic properties of hexagonal ferrites, a family of magnetoelectric multiferroic materials, by atomic-scale structural engineering, we studied the effect of structural distortion on the magnetic ordering temperature (TN). Using the symmetry analysis, we show that unlike most antiferromagnetic rare-earth transition-metal perovskites, a larger structural distortion leads to a higher TN in hexagonal ferrites and manganites, because the K3 structural distortion induces the three-dimensional magnetic ordering, which is forbidden in the undistorted structure by symmetry. We also revealed a near-linear relation between TN and the tolerance factor and a power-law relation between TN and the K3 distortion amplitude. Following the analysis, a record-high TN (185 K) among hexagonal ferrites was predicted in hexagonal ScFeO3 and experimentally verified in epitaxially stabilized films. These results add to the paradigm of spin-lattice coupling in antiferromagnetic oxides and suggests further tunability of hexagonal ferrites if more lattice distortion can be achieved.

cond-mat.mtrl-sci↗