arXiv subjects

Yong Zhang

Publications and source records attributed to Yong Zhang.

At least 325 records · Page 18Linked to original sources

arXiv2023

VideoCrafter1: Open Diffusion Models for High-Quality Video Generation

Video generation has increasingly gained interest in both academia and industry. Although commercial tools can generate plausible videos, there is a limited number of open-source models available for researchers and engineers. In this work, we introduce two diffusion models for high-quality video generation, namely text-to-video (T2V) and image-to-video (I2V) models. T2V models synthesize a video based on a given text input, while I2V models incorporate an additional image input. Our proposed T2V model can generate realistic and cinematic-quality videos with a resolution of $1024 \times 576$, outperforming other open-source T2V models in terms of quality. The I2V model is designed to produce videos that strictly adhere to the content of the provided reference image, preserving its content, structure, and style. This model is the first open-source I2V foundation model capable of transforming a given image into a video clip while maintaining content preservation constraints. We believe that these open-source video generation models will contribute significantly to the technological advancements within the community.

cs.CV↗

arXiv2023

UCF: Uncovering Common Features for Generalizable Deepfake Detection

Deepfake detection remains a challenging task due to the difficulty of generalizing to new types of forgeries. This problem primarily stems from the overfitting of existing detection methods to forgery-irrelevant features and method-specific patterns. The latter has been rarely studied and not well addressed by previous works. This paper presents a novel approach to address the two types of overfitting issues by uncovering common forgery features. Specifically, we first propose a disentanglement framework that decomposes image information into three distinct components: forgery-irrelevant, method-specific forgery, and common forgery features. To ensure the decoupling of method-specific and common forgery features, a multi-task learning strategy is employed, including a multi-class classification that predicts the category of the forgery method and a binary classification that distinguishes the real from the fake. Additionally, a conditional decoder is designed to utilize forgery features as a condition along with forgery-irrelevant features to generate reconstructed images. Furthermore, a contrastive regularization technique is proposed to encourage the disentanglement of the common and specific forgery features. Ultimately, we only utilize the common forgery features for the purpose of generalizable deepfake detection. Extensive evaluations demonstrate that our framework can perform superior generalization than current state-of-the-art methods.

cs.CV↗

arXiv2023

DeepfakeBench: A Comprehensive Benchmark of Deepfake Detection

A critical yet frequently overlooked challenge in the field of deepfake detection is the lack of a standardized, unified, comprehensive benchmark. This issue leads to unfair performance comparisons and potentially misleading results. Specifically, there is a lack of uniformity in data processing pipelines, resulting in inconsistent data inputs for detection models. Additionally, there are noticeable differences in experimental settings, and evaluation strategies and metrics lack standardization. To fill this gap, we present the first comprehensive benchmark for deepfake detection, called DeepfakeBench, which offers three key contributions: 1) a unified data management system to ensure consistent input across all detectors, 2) an integrated framework for state-of-the-art methods implementation, and 3) standardized evaluation metrics and protocols to promote transparency and reproducibility. Featuring an extensible, modular-based codebase, DeepfakeBench contains 15 state-of-the-art detection methods, 9 deepfake datasets, a series of deepfake detection evaluation protocols and analysis tools, as well as comprehensive evaluations. Moreover, we provide new insights based on extensive analysis of these evaluations from various perspectives (e.g., data augmentations, backbones). We hope that our efforts could facilitate future research and foster innovation in this increasingly critical domain. All codes, evaluations, and analyses of our benchmark are publicly available at https://github.com/SCLBD/DeepfakeBench.

cs.CV↗

arXiv2023

ArchBERT: Bi-Modal Understanding of Neural Architectures and Natural Languages

Building multi-modal language models has been a trend in the recent years, where additional modalities such as image, video, speech, etc. are jointly learned along with natural languages (i.e., textual information). Despite the success of these multi-modal language models with different modalities, there is no existing solution for neural network architectures and natural languages. Providing neural architectural information as a new modality allows us to provide fast architecture-2-text and text-2-architecture retrieval/generation services on the cloud with a single inference. Such solution is valuable in terms of helping beginner and intermediate ML users to come up with better neural architectures or AutoML approaches with a simple text query. In this paper, we propose ArchBERT, a bi-modal model for joint learning and understanding of neural architectures and natural languages, which opens up new avenues for research in this area. We also introduce a pre-training strategy named Masked Architecture Modeling (MAM) for a more generalized joint learning. Moreover, we introduce and publicly release two new bi-modal datasets for training and validating our methods. The ArchBERT's performance is verified through a set of numerical experiments on different downstream tasks such as architecture-oriented reasoning, question answering, and captioning (summarization). Datasets, codes, and demos are available supplementary materials.

cs.CL↗

arXiv2023

On-chip topological transport of optical frequency combs in silicon-based valley photonic crystals

The generation and control of optical frequency combs in integrated photonic systems enables complex, high-controllable, and large-scale devices. In parallel, harnessing topological physics in multipartite systems has allowed them with compelling features such as robustness against fabrication imperfections. Here we experimentally demonstrate on-chip topological transport for optical frequency combs at telecommunication wavelengths, both in classical and nonclassical domains. We access both the quantum frequency combs and dissipative Kerr soliton combs with a micro-resonator. The quantum frequency comb, that is, a coherent superposition of multiple frequency modes, is proven to be a frequency-entangled qudit state. We also show that dissipative Kerr soliton combs are highly coherent and mode-locked due to the collective coherence or self-organization of solitons. Moreover, the valley kink states allow both quantum frequency combs and dissipative Kerr soliton combs with robustness against sharp bends. Our topologically protected optical frequency combs could enable the inherent robustness in integrated complex photonic systems.

physics.optics↗

arXiv2023

PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adapter

The Retrieval Question Answering (ReQA) task employs the retrieval-augmented framework, composed of a retriever and generator. The generator formulates the answer based on the documents retrieved by the retriever. Incorporating Large Language Models (LLMs) as generators is beneficial due to their advanced QA capabilities, but they are typically too large to be fine-tuned with budget constraints while some of them are only accessible via APIs. To tackle this issue and further improve ReQA performance, we propose a trainable Pluggable Reward-Driven Contextual Adapter (PRCA), keeping the generator as a black box. Positioned between the retriever and generator in a Pluggable manner, PRCA refines the retrieved information by operating in a token-autoregressive strategy via maximizing rewards of the reinforcement learning phase. Our experiments validate PRCA's effectiveness in enhancing ReQA performance on three datasets by up to 20% improvement to fit black-box LLMs into existing frameworks, demonstrating its considerable potential in the LLMs era.

cs.CL↗

arXiv2023

Nonintegrability-driven Transition from Kinetics to Hydrodynamics

Nonintegrability plays a crucial role in thermalization and transport processes in many-body Hamiltonian systems, yet its quantitative effects remain unclear. To reveal the connection between the macroscopic relaxation properties and the underlying dynamics, the one-dimensional diatomic hard-point model as an illustrating example was studied analytically and numerically. We demonstrate how the system transitions from kinetic behavior to hydrodynamic behavior as the nonintegrability strength increases. Specifically, for the thermalization dynamics, we find a power-law relationship between the thermalization time and the perturbation strength near integrable regime, whereas in the far from integrable regime, the hydrodynamics dominates and the thermalization time becomes independent of the perturbation strength and exhibits a strong size-dependent behavior. Regarding transport behavior, our results further establish a threshold for the nonintegrable strength of this transition. Consequently, we can predict which behavior dominates the transport properties of the system. Especially, an explicit expression of the thermal conductivity contributed by the kinetics is given. Finally, possible applications were briefly discussed.

cond-mat.stat-mech↗

arXiv2023

Sign-changing solutions to Schiffer's overdetermined problem on wavy cylinder

In this paper, we prove the existence of $k$ families of smooth unbounded domains $Ω_s\subset\mathbb{R}^{N+1}$ with $N\geq1$, where \begin{equation} Ω_s=\left\{(x,t)\in \mathbb{R}^N\times \mathbb{R}:\vert x\vert<1+s\cos \left(\frac{2π}{T(s)}t\right)+s w_s\left(\frac{2π}{T(s)}t\right)\right\},\nonumber \end{equation} such that \begin{equation} -Δu=λu\,\, \text{in}\,\,Ω, \,\, \partial_νu=0,\,\,u=\text{const}\,\,\text{on}\,\,\partialΩ\nonumber \end{equation} admits a bounded sign-changing solution with exactly $k+1$ nodal domains. These results can be regarded as counterexamples to the Schiffer conjecture on unbounded domain. These results also indicate that there exist non-spherical unbounded regions without Pompeiu property. Our construction shows that the condition "$\partialΩ$ is homeomorphic to the unit sphere" is necessary for Williams conjecture to hold. In addition, these conclusions may have potential applications in remote sensing or CT.

math.AP↗

arXiv2023

SAI: Solving AI Tasks with Systematic Artificial Intelligence in Communication Network

In the rapid development of artificial intelligence, solving complex AI tasks is a crucial technology in intelligent mobile networks. Despite the good performance of specialized AI models in intelligent mobile networks, they are unable to handle complicated AI tasks. To address this challenge, we propose Systematic Artificial Intelligence (SAI), which is a framework designed to solve AI tasks by leveraging Large Language Models (LLMs) and JSON-format intent-based input to connect self-designed model library and database. Specifically, we first design a multi-input component, which simultaneously integrates Large Language Models (LLMs) and JSON-format intent-based inputs to fulfill the diverse intent requirements of different users. In addition, we introduce a model library module based on model cards which employ model cards to pairwise match between different modules for model composition. Model cards contain the corresponding model's name and the required performance metrics. Then when receiving user network requirements, we execute each subtask for multiple selected model combinations and provide output based on the execution results and LLM feedback. By leveraging the language capabilities of LLMs and the abundant AI models in the model library, SAI can complete numerous complex AI tasks in the communication network, achieving impressive results in network optimization, resource allocation, and other challenging tasks.

cs.AI↗

arXiv2023

Generalized Color Orderings: CEGM Integrands and Decoupling Identities

In a recent paper, we defined generalized color orderings (GCO) and Feynman diagrams (GFD) to compute color-dressed generalized biadjoint amplitudes. In this work, we study the Cachazo-Early-Guevara-Mizera (CEGM) representation of generalized partial amplitudes and ``decoupling" identities. This representation is a generalization of the Cachazo-He-Yuan (CHY) formulation as an integral over the configuration space $X(k,n)$ of $n$ points on $\mathbb{CP}^{k-1}$ in generic position. Unlike the $k=2$ case, Parke-Taylor-like integrands are not enough to compute all partial amplitudes for $k>2$. Here we give a set of constraints that integrands associated with GCOs must satisfy and use them to construct all $(3,n<9)$ integrands, all $(3,9)$ integrands up to four undetermined constants, and $95 \%$ of $(4,8)$ integrands up to 24 undetermined constants. $k=2$ partial amplitudes are known to satisfy identities. Among them, the so-called $U(1)$ decoupling identities are the simplest ones. These are characterized by a label $i$ and a color ordering in $X(2,|[n]\setminus \{i\}|)$. Here we introduce decoupling identities for $k>2$ determined combinatorially using GCOs. Moreover, we identify the natural analog of $U(1)$ identities as those characterized by a pair of labels $i\neq j$, and a pair of GCOs, one in $X(k,|[n]\setminus \{i\}|)$ and the other in $X(k-1,|[n]\setminus \{j\}|)$. We call them {\it double extension} identities. We also provide explicit connections among different ways of representing GCOs, such as configurations of lines, configurations of points, and reorientation classes of uniform oriented matroids (chirotopes).

hep-th↗

arXiv2023

FateZero: Fusing Attentions for Zero-shot Text-based Video Editing

The diffusion-based generative models have achieved remarkable success in text-based image generation. However, since it contains enormous randomness in generation progress, it is still challenging to apply such models for real-world visual content editing, especially in videos. In this paper, we propose FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask. To edit videos consistently, we propose several techniques based on the pre-trained models. Firstly, in contrast to the straightforward DDIM inversion technique, our approach captures intermediate attention maps during inversion, which effectively retain both structural and motion information. These maps are directly fused in the editing process rather than generated during denoising. To further minimize semantic leakage of the source video, we then fuse self-attentions with a blending mask obtained by cross-attention features from the source prompt. Furthermore, we have implemented a reform of the self-attention mechanism in denoising UNet by introducing spatial-temporal attention to ensure frame consistency. Yet succinct, our method is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model. We also have a better zero-shot shape-aware editing ability based on the text-to-video model. Extensive experiments demonstrate our superior temporal consistency and editing capability than previous works.

cs.CV↗

arXiv2023

ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models

In this work, we investigate the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes. In addition, the generated images should have arbitrary image aspect ratios. When generating images directly at a higher resolution, 1024 x 1024, with the pre-trained Stable Diffusion using training images of resolution 512 x 512, we observe persistent problems of object repetition and unreasonable object structures. Existing works for higher-resolution generation, such as attention-based and joint-diffusion approaches, cannot well address these issues. As a new perspective, we examine the structural components of the U-Net in diffusion models and identify the crucial cause as the limited perception field of convolutional kernels. Based on this key observation, we propose a simple yet effective re-dilation that can dynamically adjust the convolutional perception field during inference. We further propose the dispersed convolution and noise-damped classifier-free guidance, which can enable ultra-high-resolution image generation (e.g., 4096 x 4096). Notably, our approach does not require any training or optimization. Extensive experiments demonstrate that our approach can address the repetition issue well and achieve state-of-the-art performance on higher-resolution image synthesis, especially in texture details. Our work also suggests that a pre-trained diffusion model trained on low-resolution images can be directly used for high-resolution visual generation without further tuning, which may provide insights for future research on ultra-high-resolution image and video synthesis.

cs.CV↗

arXiv2023

Systematic Search for Water Fountain Candidates using the Databases of Circumstellar Maser Sources

Water fountains (WFs) are thought to be objects in the morphological evolution of the circumstellar envelopes of low- and intermediate-mass evolved stars, transitioning from spherically symmetric to asymmetric shapes. We used databases of circumstellar 1612 MHz OH and 22.235 GHz H$_2$O maser sources to search for new WF candidates using the criterion of a larger velocity range of the H$_2$O maser emission compared to that of the OH maser emission. Thus, it is in principle possible to identify WFs with an H$_2$O velocity ranges smaller than those for the previously known WFs. For the OH maser line, we analyzed database entries of 8,474 observations from 2,195 sources, and 6,085 observations from 3,642 sources for H$_2$O maser line. After a close examination of the velocity ranges and line profiles, we identified 11 sources that meet the criterion mentioned above. We examined the IRAS colors of the selected sources and found that two of them (IRAS 19069+0916 and IRAS 19319+2214) are in the color region for post-AGB stars. We find that the maser velocity criterion can discover other astrophysically interesting objects than just WFs. Such objects may include peculiar planetary nebulae with maser emissions and stellar merger remnants.

astro-ph.SR↗

arXiv2023

T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations

In this work, we investigate a simple and must-known conditional generative framework based on Vector Quantised-Variational AutoEncoder (VQ-VAE) and Generative Pre-trained Transformer (GPT) for human motion generation from textural descriptions. We show that a simple CNN-based VQ-VAE with commonly used training recipes (EMA and Code Reset) allows us to obtain high-quality discrete representations. For GPT, we incorporate a simple corruption strategy during the training to alleviate training-testing discrepancy. Despite its simplicity, our T2M-GPT shows better performance than competitive approaches, including recent diffusion-based approaches. For example, on HumanML3D, which is currently the largest dataset, we achieve comparable performance on the consistency between text and generated motion (R-Precision), but with FID 0.116 largely outperforming MotionDiffuse of 0.630. Additionally, we conduct analyses on HumanML3D and observe that the dataset size is a limitation of our approach. Our work suggests that VQ-VAE still remains a competitive approach for human motion generation.

cs.CV↗

arXiv2023

Coverage Dependent H$_2$ Desorption Energy: a Quantitative Explanation Based on Encounter Desorption Mechanism

Recent experiments show that the desorption energy of H$_2$ on a diamond-like carbon (DLC) surface depends on the H$_2$ coverage of the surface. We aim to quantitatively explain the coverage dependent H$_2$ desorption energy measured by the experiments. We derive a math formula to calculate an effective H$_2$ desorption energy based on the encounter desorption mechanism. The effective H$_2$ desorption energy depends on two key parameters, the desorption energy of H$_2$ on H$_2$ substrate and the ratio of H$_2$ diffusion barrier to its desorption energy. The calculated effective H$_2$ desorption energy qualitatively agrees with the coverage dependent H$_2$ desorption energy measured by the experiments if the values of these two parameters in literature are used in the calculations. We argue that the difference between the effective H$_2$ desorption energy and the experimental results is due to the lacking of knowledge about these two parameters. So, we recalculate these two parameters based on experimental data. Good agreement between theoretical and experimental results can be achieved if these two updated parameters are used in the calculations.

astro-ph.GA↗

arXiv2023

AI in the Gray: Exploring Moderation Policies in Dialogic Large Language Models vs. Human Answers in Controversial Topics

The introduction of ChatGPT and the subsequent improvement of Large Language Models (LLMs) have prompted more and more individuals to turn to the use of ChatBots, both for information and assistance with decision-making. However, the information the user is after is often not formulated by these ChatBots objectively enough to be provided with a definite, globally accepted answer. Controversial topics, such as "religion", "gender identity", "freedom of speech", and "equality", among others, can be a source of conflict as partisan or biased answers can reinforce preconceived notions or promote disinformation. By exposing ChatGPT to such debatable questions, we aim to understand its level of awareness and if existing models are subject to socio-political and/or economic biases. We also aim to explore how AI-generated answers compare to human ones. For exploring this, we use a dataset of a social media platform created for the purpose of debating human-generated claims on polemic subjects among users, dubbed Kialo. Our results show that while previous versions of ChatGPT have had important issues with controversial topics, more recent versions of ChatGPT (gpt-3.5-turbo) are no longer manifesting significant explicit biases in several knowledge areas. In particular, it is well-moderated regarding economic aspects. However, it still maintains degrees of implicit libertarian leaning toward right-winged ideals which suggest the need for increased moderation from the socio-political point of view. In terms of domain knowledge on controversial topics, with the exception of the "Philosophical" category, ChatGPT is performing well in keeping up with the collective human level of knowledge. Finally, we see that sources of Bing AI have slightly more tendency to the center when compared to human answers. All the analyses we make are generalizable to other types of biases and domains.

cs.LG↗

arXiv2023

ToonTalker: Cross-Domain Face Reenactment

We target cross-domain face reenactment in this paper, i.e., driving a cartoon image with the video of a real person and vice versa. Recently, many works have focused on one-shot talking face generation to drive a portrait with a real video, i.e., within-domain reenactment. Straightforwardly applying those methods to cross-domain animation will cause inaccurate expression transfer, blur effects, and even apparent artifacts due to the domain shift between cartoon and real faces. Only a few works attempt to settle cross-domain face reenactment. The most related work AnimeCeleb requires constructing a dataset with pose vector and cartoon image pairs by animating 3D characters, which makes it inapplicable anymore if no paired data is available. In this paper, we propose a novel method for cross-domain reenactment without paired data. Specifically, we propose a transformer-based framework to align the motions from different domains into a common latent space where motion transfer is conducted via latent code addition. Two domain-specific motion encoders and two learnable motion base memories are used to capture domain properties. A source query transformer and a driving one are exploited to project domain-specific motion to the canonical space. The edited motion is projected back to the domain of the source with a transformer. Moreover, since no paired data is provided, we propose a novel cross-domain training scheme using data from two domains with the designed analogy constraint. Besides, we contribute a cartoon dataset in Disney style. Extensive evaluations demonstrate the superiority of our method over competing methods.

cs.CV↗

arXiv2023

Domain Generalization via Rationale Invariance

This paper offers a new perspective to ease the challenge of domain generalization, which involves maintaining robust results even in unseen environments. Our design focuses on the decision-making process in the final classifier layer. Specifically, we propose treating the element-wise contributions to the final results as the rationale for making a decision and representing the rationale for each sample as a matrix. For a well-generalized model, we suggest the rationale matrices for samples belonging to the same category should be similar, indicating the model relies on domain-invariant clues to make decisions, thereby ensuring robust results. To implement this idea, we introduce a rationale invariance loss as a simple regularization technique, requiring only a few lines of code. Our experiments demonstrate that the proposed approach achieves competitive results across various datasets, despite its simplicity. Code is available at \url{https://github.com/liangchen527/RIDG}.

cs.CV↗