Searcharxiv⌕ Search

arXiv subjects

Yang Wu

Publications and source records attributed to Yang Wu.

At least 109 records · Page 6Linked to original sources
arXiv2024

GUESS:GradUally Enriching SyntheSis for Text-Driven Human Motion Generation

In this paper, we propose a novel cascaded diffusion-based generative framework for text-driven human motion synthesis, which exploits a strategy named GradUally Enriching SyntheSis (GUESS as its abbreviation). The strategy sets up generation objectives by grouping body joints of detailed skeletons in close semantic proximity together and then replacing each of such joint group with a single body-part node. Such an operation recursively abstracts a human pose to coarser and coarser skeletons at multiple granularity levels. With gradually increasing the abstraction level, human motion becomes more and more concise and stable, significantly benefiting the cross-modal motion synthesis task. The whole text-driven human motion synthesis problem is then divided into multiple abstraction levels and solved with a multi-stage generation framework with a cascaded latent diffusion model: an initial generator first generates the coarsest human motion guess from a given text description; then, a series of successive generators gradually enrich the motion details based on the textual description and the previous synthesized results. Notably, we further integrate GUESS with the proposed dynamic multi-condition fusion mechanism to dynamically balance the cooperative effects of the given textual condition and synthesized coarse motion prompt in different generation stages. Extensive experiments on large-scale datasets verify that GUESS outperforms existing state-of-the-art methods by large margins in terms of accuracy, realisticness, and diversity. Code is available at https://github.com/Xuehao-Gao/GUESS.

cs.CV↗
arXiv2023

Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model

Text-guided motion synthesis aims to generate 3D human motion that not only precisely reflects the textual description but reveals the motion details as much as possible. Pioneering methods explore the diffusion model for text-to-motion synthesis and obtain significant superiority. However, these methods conduct diffusion processes either on the raw data distribution or the low-dimensional latent space, which typically suffer from the problem of modality inconsistency or detail-scarce. To tackle this problem, we propose a novel Basic-to-Advanced Hierarchical Diffusion Model, named B2A-HDM, to collaboratively exploit low-dimensional and high-dimensional diffusion models for high quality detailed motion synthesis. Specifically, the basic diffusion model in low-dimensional latent space provides the intermediate denoising result that to be consistent with the textual description, while the advanced diffusion model in high-dimensional latent space focuses on the following detail-enhancing denoising process. Besides, we introduce a multi-denoiser framework for the advanced diffusion model to ease the learning of high-dimensional model and fully explore the generative potential of the diffusion model. Quantitative and qualitative experiment results on two text-to-motion benchmarks (HumanML3D and KIT-ML) demonstrate that B2A-HDM can outperform existing state-of-the-art methods in terms of fidelity, modality consistency, and diversity.

cs.CV↗
arXiv2023

A Novel 3D Non-stationary Localization-assisted ISAC Channel Model

Integrated sensing and communication (ISAC) has attracted wide attention as an emerging application scenario for the sixth generation (6G) wireless communication system. In this paper, a novel three-dimensional (3D) non-stationary localization-assisted ISAC geometry-based stochastic model (GBSM) is proposed. The locations of the first-bounce scatterer and last-bounce scatterer in the communication channel can be estimated by the particle filter with the assistance of backscattering sensing. The important channel statistical properties of the proposed channel model are simulated and compared with the ray tracing (RT) results, including the delay spread, azimuth angle of departure/arrival (AAoD/AAoA) spread, and elevation angle of departure/arrival (EAoD/EAoA) spread. The simulation results of the proposed channel model show a good agreement with the RT results, which proves the correctness of the proposed channel model. Utilizing the localization parameters of scatterers, the proposed ISAC channel model can better map the real environment.

eess.SP↗
arXiv2023

Observation of the Knot Topology of Non-Hermitian Systems in a Single Spin

The non-Hermiticity of the system gives rise to distinct knot topology that has no Hermitian counterpart. Here, we report a comprehensive study of the knot topology in gapped non-Hermitian systems based on the universal dilation method with a long coherence time nitrogen-vacancy center in a $^{\text{12}}$C isotope purified diamond. Both the braiding patterns of energy bands and the eigenstate topology are revealed. Furthermore, the global biorthogonal Berry phase related to the eigenstate topology has been successfully observed, which identifies the topological invariance for the non-Hermitian system. Our method paves the way for further exploration of the interplay among band braiding, eigenstate topology and symmetries in non-Hermitian quantum systems.

quant-ph↗
arXiv2023

Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs

Most text-driven human motion generation methods employ sequential modeling approaches, e.g., transformer, to extract sentence-level text representations automatically and implicitly for human motion synthesis. However, these compact text representations may overemphasize the action names at the expense of other important properties and lack fine-grained details to guide the synthesis of subtly distinct motion. In this paper, we propose hierarchical semantic graphs for fine-grained control over motion generation. Specifically, we disentangle motion descriptions into hierarchical semantic graphs including three levels of motions, actions, and specifics. Such global-to-local structures facilitate a comprehensive understanding of motion description and fine-grained control of motion generation. Correspondingly, to leverage the coarse-to-fine topology of hierarchical semantic graphs, we decompose the text-to-motion diffusion process into three semantic levels, which correspond to capturing the overall motion, local actions, and action specifics. Extensive experiments on two benchmark human motion datasets, including HumanML3D and KIT, with superior performances, justify the efficacy of our method. More encouragingly, by modifying the edge weights of hierarchical semantic graphs, our method can continuously refine the generated motion, which may have a far-reaching impact on the community. Code and pre-training weights are available at https://github.com/jpthu17/GraphMotion.

cs.CV↗
arXiv2023

An Early Evaluation of GPT-4V(ision)

In this paper, we evaluate different abilities of GPT-4V including visual understanding, language understanding, visual puzzle solving, and understanding of other modalities such as depth, thermal, video, and audio. To estimate GPT-4V's performance, we manually construct 656 test instances and carefully evaluate the results of GPT-4V. The highlights of our findings are as follows: (1) GPT-4V exhibits impressive performance on English visual-centric benchmarks but fails to recognize simple Chinese texts in the images; (2) GPT-4V shows inconsistent refusal behavior when answering questions related to sensitive traits such as gender, race, and age; (3) GPT-4V obtains worse results than GPT-4 (API) on language understanding tasks including general language understanding benchmarks and visual commonsense knowledge evaluation benchmarks; (4) Few-shot prompting can improve GPT-4V's performance on both visual understanding and language understanding; (5) GPT-4V struggles to find the nuances between two similar images and solve the easy math picture puzzles; (6) GPT-4V shows non-trivial performance on the tasks of similar modalities to image, such as video and thermal. Our experimental results reveal the ability and limitations of GPT-4V and we hope our paper can provide some insights into the application and research of GPT-4V.

cs.CL↗
arXiv2023

Experimental study on the principle of minimal work fluctuations

The central quantity in the celebrated quantum Jarzynski equality is $e^{-βW}$, where $W$ is work and $β$ is the inverse temperature. The impact of quantum randomness on the fluctuations of $e^{-βW}$ and hence on the predictive power of the Jarzynski estimator is an important problem. Working on a single nitrogen-vacancy center in diamond and riding on an implementation of two-point measurement of non-equilibrium work with single-shot readout, we have conducted a direct experimental investigation of the relationship between the fluctuations of $e^{-βW}$ and adiabaticity of non-equilibrium work protocols. It is observed that adiabatic processes minimize the variance of $e^{-βW}$, thus verifying an early theoretical concept, the so-called principle of minimal work fluctuations. Furthermore, it is experimentally demonstrated that shortcuts-to-adiabaticity control can be exploited to minimize the variance of $e^{-βW}$ in fast work protocols. Our work should stimulate further experimental studies of quantum effects on the bias and error in the estimates of free energy differences based on the Jarzynski equality.

quant-ph↗
arXiv2023

Towards Open-World Co-Salient Object Detection with Generative Uncertainty-aware Group Selective Exchange-Masking

The traditional definition of co-salient object detection (CoSOD) task is to segment the common salient objects in a group of relevant images. This definition is based on an assumption of group consensus consistency that is not always reasonable in the open-world setting, which results in robustness issue in the model when dealing with irrelevant images in the inputting image group under the open-word scenarios. To tackle this problem, we introduce a group selective exchange-masking (GSEM) approach for enhancing the robustness of the CoSOD model. GSEM takes two groups of images as input, each containing different types of salient objects. Based on the mixed metric we designed, GSEM selects a subset of images from each group using a novel learning-based strategy, then the selected images are exchanged. To simultaneously consider the uncertainty introduced by irrelevant images and the consensus features of the remaining relevant images in the group, we designed a latent variable generator branch and CoSOD transformer branch. The former is composed of a vector quantised-variational autoencoder to generate stochastic global variables that model uncertainty. The latter is designed to capture correlation-based local features that include group consensus. Finally, the outputs of the two branches are merged and passed to a transformer-based decoder to generate robust predictions. Taking into account that there are currently no benchmark datasets specifically designed for open-world scenarios, we constructed three open-world benchmark datasets, namely OWCoSal, OWCoSOD, and OWCoCA, based on existing datasets. By breaking the group-consistency assumption, these datasets provide effective simulations of real-world scenarios and can better evaluate the robustness and practicality of models.

cs.CV↗
arXiv2023

Real-time observation of magnetization and magnon dynamics in a two-dimensional topological antiferromagnet MnBi2Te4

Atomically thin van der Waals magnetic materials have not only provided a fertile playground to explore basic physics in the two-dimensional (2D) limit but also created vast opportunities for novel ultrafast functional devices. Here we systematically investigate ultrafast magnetization dynamics and spin wave dynamics in few-layer topological antiferromagnetic MnBi2Te4 crystals as a function of layer number, temperature, and magnetic field. We find laser-induced (de)magnetization processes can be used to accurately track the distinct magnetic states in different magnetic field regimes, including showing clear odd-even layer number effects. In addition, strongly field-dependent antiferromagnetic magnon modes with tens of gigahertz frequencies are optically generated and directly observed in the time domain. Remarkably, we find that magnetization and magnon dynamics can be observed in not only the time-resolved magneto-optical Kerr effect but also the time resolved reflectivity, indicating strong correlation between the magnetic state and electronic structure. These measurements present the first comprehensive overview of ultrafast spin dynamics in this novel 2D antiferromagnet, paving the way for potential applications in 2D antiferromagnetic spintronics and magnonics as well as further studies of ultrafast control of both magnetization and topological quantum states.

cond-mat.mes-hall↗
arXiv2023

Speech2Lip: High-fidelity Speech to Lip Generation by Learning from a Short Video

Synthesizing realistic videos according to a given speech is still an open challenge. Previous works have been plagued by issues such as inaccurate lip shape generation and poor image quality. The key reason is that only motions and appearances on limited facial areas (e.g., lip area) are mainly driven by the input speech. Therefore, directly learning a mapping function from speech to the entire head image is prone to ambiguity, particularly when using a short video for training. We thus propose a decomposition-synthesis-composition framework named Speech to Lip (Speech2Lip) that disentangles speech-sensitive and speech-insensitive motion/appearance to facilitate effective learning from limited training data, resulting in the generation of natural-looking videos. First, given a fixed head pose (i.e., canonical space), we present a speech-driven implicit model for lip image generation which concentrates on learning speech-sensitive motion and appearance. Next, to model the major speech-insensitive motion (i.e., head movement), we introduce a geometry-aware mutual explicit mapping (GAMEM) module that establishes geometric mappings between different head poses. This allows us to paste generated lip images at the canonical space onto head images with arbitrary poses and synthesize talking videos with natural head movements. In addition, a Blend-Net and a contrastive sync loss are introduced to enhance the overall synthesis performance. Quantitative and qualitative results on three benchmarks demonstrate that our model can be trained by a video of just a few minutes in length and achieve state-of-the-art performance in both visual quality and speech-visual synchronization. Code: https://github.com/CVMI-Lab/Speech2Lip.

cs.CV↗
arXiv2023

Community-Based Hierarchical Positive-Unlabeled (PU) Model Fusion for Chronic Disease Prediction

Positive-Unlabeled (PU) Learning is a challenge presented by binary classification problems where there is an abundance of unlabeled data along with a small number of positive data instances, which can be used to address chronic disease screening problem. State-of-the-art PU learning methods have resulted in the development of various risk estimators, yet they neglect the differences among distinct populations. To address this issue, we present a novel Positive-Unlabeled Learning Tree (PUtree) algorithm. PUtree is designed to take into account communities such as different age or income brackets, in tasks of chronic disease prediction. We propose a novel approach for binary decision-making, which hierarchically builds community-based PU models and then aggregates their deliverables. Our method can explicate each PU model on the tree for the optimized non-leaf PU node splitting. Furthermore, a mask-recovery data augmentation strategy enables sufficient training of the model in individual communities. Additionally, the proposed approach includes an adversarial PU risk estimator to capture hierarchical PU-relationships, and a model fusion network that integrates data from each tree path, resulting in robust binary classification results. We demonstrate the superior performance of PUtree as well as its variants on two benchmarks and a new diabetes-prediction dataset.

cs.LG↗
arXiv2023

Integrating Offline Reinforcement Learning with Transformers for Sequential Recommendation

We consider the problem of sequential recommendation, where the current recommendation is made based on past interactions. This recommendation task requires efficient processing of the sequential data and aims to provide recommendations that maximize the long-term reward. To this end, we train a farsighted recommender by using an offline RL algorithm with the policy network in our model architecture that has been initialized from a pre-trained transformer model. The pre-trained model leverages the superb ability of the transformer to process sequential information. Compared to prior works that rely on online interaction via simulation, we focus on implementing a fully offline RL framework that is able to converge in a fast and stable way. Through extensive experiments on public datasets, we show that our method is robust across various recommendation regimes, including e-commerce and movie suggestions. Compared to state-of-the-art supervised learning algorithms, our algorithm yields recommendations of higher quality, demonstrating the clear advantage of combining RL and transformers.

cs.IR↗
arXiv2023

Direct observation of chiral edge current at zero magnetic field in odd-layer MnBi$_2$Te$_4$

The chiral edge current is the boundary manifestation of the Chern number of a quantum anomalous Hall (QAH) insulator. Its direct observation is assumed to require well-quantized Hall conductance, and is so far lacking. The recently discovered van der Waals antiferromagnet MnBi$_2$Te$_4$ is theorized as a QAH in odd-layers but has shown Hall resistivity below the quantization value at zero magnetic field. Here, we perform scanning superconducting quantum interference device (sSQUID) microscopy on these seemingly failed QAH insulators to image their current distribution. When gated to the charge neutral point, our device exhibits edge current, which flows unidirectionally on the odd-layer boundary both with vacuum and with the even-layer. The chirality of such edge current reverses with the magnetization of the bulk. Surprisingly, we find the edge channels coexist with finite bulk conduction even though the bulk chemical potential is in the band gap, suggesting their robustness under significant edge-bulk scattering. Our result establishes the existence of chiral edge currents in a topological antiferromagnet and offers an alternative for identifying QAH states.

cond-mat.mes-hall↗
arXiv2023

NOFA: NeRF-based One-shot Facial Avatar Reconstruction

3D facial avatar reconstruction has been a significant research topic in computer graphics and computer vision, where photo-realistic rendering and flexible controls over poses and expressions are necessary for many related applications. Recently, its performance has been greatly improved with the development of neural radiance fields (NeRF). However, most existing NeRF-based facial avatars focus on subject-specific reconstruction and reenactment, requiring multi-shot images containing different views of the specific subject for training, and the learned model cannot generalize to new identities, limiting its further applications. In this work, we propose a one-shot 3D facial avatar reconstruction framework that only requires a single source image to reconstruct a high-fidelity 3D facial avatar. For the challenges of lacking generalization ability and missing multi-view information, we leverage the generative prior of 3D GAN and develop an efficient encoder-decoder network to reconstruct the canonical neural volume of the source image, and further propose a compensation network to complement facial details. To enable fine-grained control over facial dynamics, we propose a deformation field to warp the canonical volume into driven expressions. Through extensive experimental comparisons, we achieve superior synthesis results compared to several state-of-the-art methods.

cs.CV↗
arXiv2023

Giant nonlocal edge conduction in the axion insulator state of MnBi2Te4

The recently discovered antiferromagnetic (AFM) topological insulator (TI) MnBi2Te4 represents a versatile material platform for exploring exotic topological quantum phenomena in nanoscale devices. It has been proposed that even-septuple-layer (even-SL) MnBi2Te4 can host helical hinge currents with unique nonlocal behavior, but experimental confirmation is still lacking. In this work, we report transport studies of exfoliated MnBi2Te4 flakes with varied thicknesses down to the few-nanometer regime. We observe giant nonlocal transport signals in even-SL devices when the system is in the axion insulator state but vanishingly small nonlocal signal in the odd-SL devices at the same magnetic field range. In conjunction with theoretical calculations, we demonstrate that the nonlocal transport is via the helical edge currents mainly distributed at the hinges between the side and top/bottom surfaces. The helical edge currents in the axion insulator state may find unique applications in topological quantum devices.

cond-mat.mtrl-sci↗
arXiv2023

Multi-object Video Generation from Single Frame Layouts

In this paper, we study video synthesis with emphasis on simplifying the generation conditions. Most existing video synthesis models or datasets are designed to address complex motions of a single object, lacking the ability of comprehensively understanding the spatio-temporal relationships among multiple objects. Besides, current methods are usually conditioned on intricate annotations (e.g. video segmentations) to generate new videos, being fundamentally less practical. These motivate us to generate multi-object videos conditioning exclusively on object layouts from a single frame. To solve above challenges and inspired by recent research on image generation from layouts, we have proposed a novel video generative framework capable of synthesizing global scenes with local objects, via implicit neural representations and layout motion self-inference. Our framework is a non-trivial adaptation from image generation methods, and is new to this field. In addition, our model has been evaluated on two widely-used video recognition benchmarks, demonstrating effectiveness compared to the baseline model.

cs.CV↗
arXiv2023

Improving Cross-Task Generalization with Step-by-Step Instructions

Instruction tuning has been shown to be able to improve cross-task generalization of language models. However, it is still challenging for language models to complete the target tasks following the instructions, as the instructions are general and lack intermediate steps. To address this problem, we propose to incorporate the step-by-step instructions to help language models to decompose the tasks, which can provide the detailed and specific procedures for completing the target tasks. The step-by-step instructions are obtained automatically by prompting ChatGPT, which are further combined with the original instructions to tune language models. The extensive experiments on SUP-NATINST show that the high-quality step-by-step instructions can improve cross-task generalization across different model sizes. Moreover, the further analysis indicates the importance of the order of steps of the step-by-step instruction for the improvement. To facilitate future research, we release the step-by-step instructions and their human quality evaluation results.

cs.CL↗
arXiv2023

A Near-Optimal Gradient Flow for Learning Neural Energy-Based Models

In this paper, we propose a novel numerical scheme to optimize the gradient flows for learning energy-based models (EBMs). From a perspective of physical simulation, we redefine the problem of approximating the gradient flow utilizing optimal transport (i.e. Wasserstein) metric. In EBMs, the learning process of stepwise sampling and estimating data distribution performs the functional gradient of minimizing the global relative entropy between the current and target real distribution, which can be treated as dynamic particles moving from disorder to target manifold. Previous learning schemes mainly minimize the entropy concerning the consecutive time KL divergence in each learning step. However, they are prone to being stuck in the local KL divergence by projecting non-smooth information within smooth manifold, which is against the optimal transport principle. To solve this problem, we derive a second-order Wasserstein gradient flow of the global relative entropy from Fokker-Planck equation. Compared with existing schemes, Wasserstein gradient flow is a smoother and near-optimal numerical scheme to approximate real data densities. We also derive this near-proximal scheme and provide its numerical computation equations. Our extensive experiments demonstrate the practical superiority and potentials of our proposed scheme on fitting complex distributions and generating high-quality, high-dimensional data with neural EBMs.

cs.CV↗