arXiv subjects

Shiwei Zhang

Publications and source records attributed to Shiwei Zhang.

At least 73 records · Page 4Linked to original sources

arXiv2024

The two-dimensional homogeneous electron gas with symmetric dual-gate screening: exchange-correlation functional and other ground-state properties

The two-dimensional (2D) homogeneous electron gas (HEG) is a fundamental model in quantum many-body physics. It is important to theoretical and computational studies, where exchange-correlation energies computed in it serve as the foundation for density-functional calculations. It is also of direct relevance to a variety of experimental settings, especially with the rapid recent growth in 2D materials and moiré systems. In these experiments, metallic gates are often present, which screen the Coulomb interaction between electrons. The effect of the screening can qualitatively change the behavior of the 2D HEG, and requires accurate many-body computations to capture. In this work, we perform state-of-the-art diffusion Monte Carlo (DMC) calculations in the 2D HEG subjected to symmetric dual-gate screening. We systematically compute the correlation energy across a range of densities and gate separations for both spin unpolarized and fully polarized systems. A global fit is obtained for the correlation energy, using these data and imposing various limiting behaviors obtained from perturbation analysis. The new functional will allow density-functional calculations to be performed for a variety of realistic experimental setups which can accurately account for the presence of gates. We also investigate how the gate screening affects the bulk modulus, pair correlation function, and the structure factor of the 2D HEG, which can potentially be probed in experiments.

cond-mat.str-el↗

arXiv2024

Variational Benchmarks for Quantum Many-Body Problems

The continued development of computational approaches to many-body ground-state problems in physics and chemistry calls for a consistent way to assess its overall progress. In this work, we introduce a metric of variational accuracy, the V-score, obtained from the variational energy and its variance. We provide an extensive curated dataset of variational calculations of many-body quantum systems, identifying cases where state-of-the-art numerical approaches show limited accuracy, and future algorithms or computational platforms, such as quantum computing, could provide improved accuracy. The V-score can be used as a metric to assess the progress of quantum variational methods toward a quantum advantage for ground-state problems, especially in regimes where classical verifiability is impossible.

quant-ph↗

arXiv2024

DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control

Recent advances in customized video generation have enabled users to create videos tailored to both specific subjects and motion trajectories. However, existing methods often require complicated test-time fine-tuning and struggle with balancing subject learning and motion control, limiting their real-world applications. In this paper, we present DreamVideo-2, a zero-shot video customization framework capable of generating videos with a specific subject and motion trajectory, guided by a single image and a bounding box sequence, respectively, and without the need for test-time fine-tuning. Specifically, we introduce reference attention, which leverages the model's inherent capabilities for subject learning, and devise a mask-guided motion module to achieve precise motion control by fully utilizing the robust motion signal of box masks derived from bounding boxes. While these two components achieve their intended functions, we empirically observe that motion control tends to dominate over subject learning. To address this, we propose two key designs: 1) the masked reference attention, which integrates a blended latent mask modeling scheme into reference attention to enhance subject representations at the desired positions, and 2) a reweighted diffusion loss, which differentiates the contributions of regions inside and outside the bounding boxes to ensure a balance between subject and motion control. Extensive experimental results on a newly curated dataset demonstrate that DreamVideo-2 outperforms state-of-the-art methods in both subject customization and motion control. The dataset, code, and models will be made publicly available.

cs.CV↗

arXiv2024

EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models

Recent advancements in generation models have showcased remarkable capabilities in generating fantastic content. However, most of them are trained on proprietary high-quality data, and some models withhold their parameters and only provide accessible application programming interfaces (APIs), limiting their benefits for downstream tasks. To explore the feasibility of training a text-to-image generation model comparable to advanced models using publicly available resources, we introduce EvolveDirector. This framework interacts with advanced models through their public APIs to obtain text-image data pairs to train a base model. Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability. However, it requires large-scale samples of 10 million or more. This incurs significant expenses in time, computational resources, and especially the costs associated with calling fee-based APIs. To address this problem, we leverage pre-trained large vision-language models (VLMs) to guide the evolution of the base model. VLM continuously evaluates the base model during training and dynamically updates and refines the training dataset by the discrimination, expansion, deletion, and mutation operations. Experimental results show that this paradigm significantly reduces the required data volume. Furthermore, when approaching multiple advanced models, EvolveDirector can select the best samples generated by them to learn powerful and balanced abilities. The final trained model Edgen is demonstrated to outperform these advanced models. The code and model weights are available at https://github.com/showlab/EvolveDirector.

cs.CV↗

arXiv2024

FreeMask: Rethinking the Importance of Attention Masks for Zero-Shot Video Editing

Text-to-video diffusion models have made remarkable advancements. Driven by their ability to generate temporally coherent videos, research on zero-shot video editing using these fundamental models has expanded rapidly. To enhance editing quality, structural controls are frequently employed in video editing. Among these techniques, cross-attention mask control stands out for its effectiveness and efficiency. However, when cross-attention masks are naively applied to video editing, they can introduce artifacts such as blurring and flickering. Our experiments uncover a critical factor overlooked in previous video editing research: cross-attention masks are not consistently clear but vary with model structure and denoising timestep. To address this issue, we propose the metric Mask Matching Cost (MMC) that quantifies this variability and propose FreeMask, a method for selecting optimal masks tailored to specific video editing tasks. Using MMC-selected masks, we further improve the masked fusion mechanism within comprehensive attention features, e.g., temp, cross, and self-attention modules. Our approach can be seamlessly integrated into existing zero-shot video editing frameworks with better performance, requiring no control assistance or parameter fine-tuning but enabling adaptive decoupling of unedited semantic layouts with mask precision control. Extensive experiments demonstrate that FreeMask achieves superior semantic fidelity, temporal consistency, and editing quality compared to state-of-the-art methods.

cs.CV↗

arXiv2024

Monte Carlo Methods in the Manifold of Hartree-Fock-Bogoliubov Wave Functions

We explore the possibility to implement random walks in the manifold of Hartree-Fock-Bogoliubov wave functions. The goal is to extend state-of-the-art quantum Monte Carlo approaches, in particular the constrained-path auxiliary-field quantum Monte Carlo technique, to systems where finite pairing order parameters or complex pairing mechanisms, e.g., Fulde-Ferrell-Larkin-Ovchinnikov (FFLO) pairing or triplet pairing, may be expected. Leveraging the flexibility to define a vacuum state tailored to the physical problem, we discuss a method to use imaginary-time evolution of Hartree-Fock-Bogoliubov states to compute ground state correlations, extending beyond situations spanned by current formalisms. Illustrative examples are provided.

cond-mat.str-el↗

arXiv2024

DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models

Emotional talking head generation has attracted growing attention. Previous methods, which are mainly GAN-based, still struggle to consistently produce satisfactory results across diverse emotions and cannot conveniently specify personalized emotions. In this work, we leverage powerful diffusion models to address the issue and propose DreamTalk, a framework that employs meticulous design to unlock the potential of diffusion models in generating emotional talking heads. Specifically, DreamTalk consists of three crucial components: a denoising network, a style-aware lip expert, and a style predictor. The diffusion-based denoising network can consistently synthesize high-quality audio-driven face motions across diverse emotions. To enhance lip-motion accuracy and emotional fullness, we introduce a style-aware lip expert that can guide lip-sync while preserving emotion intensity. To more conveniently specify personalized emotions, a diffusion-based style predictor is utilized to predict the personalized emotion directly from the audio, eliminating the need for extra emotion reference. By this means, DreamTalk can consistently generate vivid talking faces across diverse emotions and conveniently specify personalized emotions. Extensive experiments validate DreamTalk's effectiveness and superiority. The code is available at https://github.com/ali-vilab/dreamtalk.

cs.CV↗

arXiv2024

Downfolding from Ab Initio to Interacting Model Hamiltonians: Comprehensive Analysis and Benchmarking of the DFT+cRPA Approach

Model Hamiltonians are regularly derived from first-principles data to describe correlated matter. However, the standard methods for this contain a number of largely unexplored approximations. For a strongly correlated impurity model system, here we carefully compare a standard downfolding technique with the best possible ground-truth estimates for charge-neutral excited state energies and wavefunctions using state-of-the-art first-principles many-body wave function approaches. To this end, we use the vanadocene molecule and analyze all downfolding aspects, including the Hamiltonian form, target basis, double counting correction, and Coulomb interaction screening models. We find that the choice of target-space basis functions emerges as a key factor for the quality of the downfolded results, while orbital-dependent double counting correction diminishes the quality. Background screening to the Coulomb interaction matrix elements primarily affects crystal-field excitations. Our benchmark uncovers the relative importance of each downfolding step and offers insights into the potential accuracy of minimal downfolded model Hamiltonians

cond-mat.str-el↗

arXiv2024

Correlation effects in magic-angle twisted bilayer graphene: An auxiliary-field quantum Monte Carlo study

Magic angle twisted bilayer graphene (MATBG) presents a fascinating platform for investigating the effects of electron interactions in topological flat bands. The Bistritzer-MacDonald (BM) model provides a simplified quantitative description of the flat bands. Introducing long-range Coulomb interactions leads to an interacting BM (IBM) Hamiltonian, a momentum-space continuum description which offers a very natural starting point for many-body studies of MATBG. Accurate and reliable many-body computations in the IBM model are challenging, however, and have been limited mostly to special fillings, or smaller lattice sizes. We employ state-of-the-art auxiliary-field quantum Monte Carlo (AFQMC) method to study the IBM model, which constrains the sign problem to enable accurate treatment of large system sizes. We determine ground-state properties and quantify errors compared to mean-field theory calculations. Our calculations identify correlated metal states and their competition with the insulating Kramers inter-valley coherent state at both half-filling and charge neutrality. Additionally, we investigate one- and three-quarter fillings, and examine the effect of many-body corrections beyond single Slater determinant solutions. We discuss the effect that details of the IBM Hamiltonian have on the results, including different forms of double-counting corrections, and the need to establish and precisely specify many-body Hamiltonians to allow more direct and quantitative comparisons with experiments in MATBG.

cond-mat.str-el↗

arXiv2024

UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation

Recent diffusion-based human image animation techniques have demonstrated impressive success in synthesizing videos that faithfully follow a given reference identity and a sequence of desired movement poses. Despite this, there are still two limitations: i) an extra reference model is required to align the identity image with the main video branch, which significantly increases the optimization burden and model parameters; ii) the generated video is usually short in time (e.g., 24 frames), hampering practical applications. To address these shortcomings, we present a UniAnimate framework to enable efficient and long-term human video generation. First, to reduce the optimization difficulty and ensure temporal coherence, we map the reference image along with the posture guidance and noise video into a common feature space by incorporating a unified video diffusion model. Second, we propose a unified noise input that supports random noised input as well as first frame conditioned input, which enhances the ability to generate long-term video. Finally, to further efficiently handle long sequences, we explore an alternative temporal modeling architecture based on state space model to replace the original computation-consuming temporal Transformer. Extensive experimental results indicate that UniAnimate achieves superior synthesis results over existing state-of-the-art counterparts in both quantitative and qualitative evaluations. Notably, UniAnimate can even generate highly consistent one-minute videos by iteratively employing the first frame conditioning strategy. Code and models will be publicly available. Project page: https://unianimate.github.io/.

cs.CV↗

arXiv2024

Ferromagnetic semimetal and charge-density wave phases of interacting electrons in a honeycomb moiré potential

The exploration of quantum phases in moiré systems has drawn intense experimental and theoretical efforts. The realization of honeycomb symmetry has been a recent focus. The combination of strong interaction and honeycomb symmetry can lead to exotic electronic states such as fractional Chern insulator, unconventional superconductor, and quantum spin liquid. Accurate computations in such systems, with reliable treatment of strong long-ranged Coulomb interaction and approaching the large system sizes to extract thermodynamic phases, are mostly missing. We study the two-dimensional electron gas on a honeycomb moiré lattice at quarter filling, using fixed-phase diffusion Monte Carlo. The ground state phases of this important model are determined in the parameter regime relevant to current experiments. With increasing moiré potential, the systems transitions from a paramagnetic metal to an itinerant ferromagnetic semimetal and then a charge-density-wave insulator.

cond-mat.str-el↗

arXiv2024

Ground state phases of the two-dimension electron gas with a unified variational approach

The two-dimensional electron gas (2DEG) is a fundamental model, which is drawing increasing interest because of recent advances in experimental and theoretical studies of 2D materials. Current understanding of the ground state of the 2DEG relies on quantum Monte Carlo calculations, based on variational comparisons of different ansatze for different phases. We use a single variational ansatz, a general backflow-type wave function using a message-passing neural quantum state architecture, for a unified description across the entire density range. The variational optimization consistently leads to lower ground-state energies than previous best results. Transition into a Wigner crystal (WC) phase occurs automatically at rs = 37 +/- 1, a density lower than currently believed. Between the liquid and WC phases, the same ansatz and variational search strongly suggest the existence of intermediate states in a broad range of densities, with enhanced short-range nematic spin correlations.

cond-mat.str-el↗

arXiv2024

Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation

Diffusion models have recently achieved remarkable progress in generating realistic images. However, challenges remain in accurately understanding and synthesizing the layout requirements in the textual prompts. To align the generated image with layout instructions, we present a training-free layout calibration system SimM that intervenes in the generative process on the fly during inference time. Specifically, following a "check-locate-rectify" pipeline, the system first analyses the prompt to generate the target layout and compares it with the intermediate outputs to automatically detect errors. Then, by moving the located activations and making intra- and inter-map adjustments, the rectification process can be performed with negligible computational overhead. To evaluate SimM over a range of layout requirements, we present a benchmark SimMBench that compensates for the lack of superlative spatial relations in existing datasets. And both quantitative and qualitative results demonstrate the effectiveness of the proposed SimM in calibrating the layout inconsistencies. Our project page is at https://simm-t2i.github.io/SimM.

cs.CV↗

arXiv2024

Automatic Order Detection and Restoration Through Systematically Improvable Variational Wave Functions

Variational wave function ansatze are an invaluable tool to study the properties of strongly correlated systems. We propose such a wave function, based on the theory of auxiliary fields and combining aspects of auxiliary-field quantum Monte Carlo and modern variational optimization techniques including automatic differentiation. The resulting ansatz, consisting of several slices of optimized projectors, is highly expressive and systematically improvable. We benchmark this form on the two-dimensional Hubbard model, using both cylindrical and large, fully periodic supercells. The computed ground-state energies are competitive with the best variational results. Moreover, the optimized wave functions predict the correct ground-state order with near full symmetry restoration (i.e. translation invariance) despite initial states with incorrect orders. The ansatz can become a tool for local order prediction, leading to a new paradigm for variational studies of bulk systems. It can also be viewed as an approach to produce accurate and systematically improvable wave functions in a convenient form of non-orthogonal Slater determinants (e.g., for quantum chemistry) at polynomial computational cost.

cond-mat.str-el↗

arXiv2024

Metal-insulator transition in transition metal dichalcogenide heterobilayer: accurate treatment of interaction

Transition metal dichalcogenide superlattices provide an exciting new platform for exploring and understanding a variety of phases of matter. The moiré continuum Hamiltonian, of two-dimensional jellium in a modulating potential, provides a fundamental model for such systems. Accurate computations with this model are essential for interpreting experimental observations and making predictions for future explorations. In this work, we combine two complementary quantum Monte Carlo (QMC) methods, phaseless auxiliary field quantum Monte Carlo and fixed-phase diffusion Monte Carlo, to study the ground state of this Hamiltonian. We observe a metal-insulator transition between a paramagnetic and a $120^\circ$ Néel ordered state as the moiré potential depth and the interaction strength are varied. We find significant differences from existing results by Hartree-Fock and exact diagonalization studies. In addition, we benchmark density-functional theory, and suggest an optimal hybrid functional which best approximates our QMC results.

cond-mat.str-el↗

arXiv2024

HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

Single-Program-Multiple-Data (SPMD) parallelism has recently been adopted to train large deep neural networks (DNNs). Few studies have explored its applicability on heterogeneous clusters, to fully exploit available resources for large model learning. This paper presents \OurSystem, an automated system designed to expedite SPMD DNN training on heterogeneous clusters. \OurSystem jointly optimizes the tensor sharding strategy, sharding ratios across heterogeneous devices and the communication methods for tensor exchanges for optimized distributed training with SPMD parallelism. We novelly formulate model partitioning as a program synthesis problem, in which we generate a distributed program from scratch on a distributed instruction set that semantically resembles the program designed for a single device, and systematically explore the solution space with an A*-based search algorithm. We derive the optimal tensor sharding ratios by formulating it as a linear programming problem. Additionally, \OurSystem explores tensor communication optimization in a heterogeneous cluster and integrates it as part of the program synthesis process, for automatically choosing optimal collective communication primitives and applying sufficient factor broadcasting technique. Extensive experiments on representative workloads demonstrate that \OurSystem achieves up to 2.41x speed-up on heterogeneous clusters.

cs.DC↗

arXiv2023

A Recipe for Scaling up Text-to-Video Generation with Text-free Videos

Diffusion-based text-to-video generation has witnessed impressive progress in the past year yet still falls behind text-to-image generation. One of the key reasons is the limited scale of publicly available data (e.g., 10M video-text pairs in WebVid10M vs. 5B image-text pairs in LAION), considering the high cost of video captioning. Instead, it could be far easier to collect unlabeled clips from video platforms like YouTube. Motivated by this, we come up with a novel text-to-video generation framework, termed TF-T2V, which can directly learn with text-free videos. The rationale behind is to separate the process of text decoding from that of temporal modeling. To this end, we employ a content branch and a motion branch, which are jointly optimized with weights shared. Following such a pipeline, we study the effect of doubling the scale of training set (i.e., video-only WebVid10M) with some randomly collected text-free videos and are encouraged to observe the performance improvement (FID from 9.67 to 8.19 and FVD from 484 to 441), demonstrating the scalability of our approach. We also find that our model could enjoy sustainable performance gain (FID from 8.19 to 7.64 and FVD from 441 to 366) after reintroducing some text labels for training. Finally, we validate the effectiveness and generalizability of our ideology on both native text-to-video generation and compositional video synthesis paradigms. Code and models will be publicly available at https://tf-t2v.github.io/.

cs.CV↗

arXiv2023

Observation of an electronic microemulsion phase emerging from a quantum crystal-to-liquid transition

Strongly interacting electronic systems possess rich phase diagrams resulting from the competition between different quantum ground states. A general mechanism that relieves this frustration is the emergence of microemulsion phases, where regions of different phase self-organize across multiple length scales. The experimental characterization of these phases often poses significant challenges, as the long-range Coulomb interaction microscopically mingles the competing states. Here, we use cryogenic reflectance and magneto-optical spectroscopy to observe the signatures of the mixed state between an electronic Wigner crystal and an electron liquid in a MoSe2 monolayer. We find that the transit into this 'microemulsion' state is marked by anomalies in exciton reflectance, spin susceptibility, and Umklapp scattering, establishing it as a distinct phase of electronic matter. Our study of the two-dimensional electronic microemulsion phase elucidates the physics of novel correlated electron states with strong Coulomb interactions.

cond-mat.str-el↗