arXiv subjects

Zhiyuan Ma

Publications and source records attributed to Zhiyuan Ma.

At least 91 records · Page 5Linked to original sources

arXiv2024

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

With the flourishing of social media platforms, vision-language pre-training (VLP) recently has received great attention and many remarkable progresses have been achieved. The success of VLP largely benefits from the information complementation and enhancement between different modalities. However, most of recent studies focus on cross-modal contrastive learning (CMCL) to promote image-text alignment by pulling embeddings of positive sample pairs together while pushing those of negative pairs apart, which ignores the natural asymmetry property between different modalities and requires large-scale image-text corpus to achieve arduous progress. To mitigate this predicament, we propose CMAL, a Cross-Modal Associative Learning framework with anchor points detection and cross-modal associative learning for VLP. Specifically, we first respectively embed visual objects and textual tokens into separate hypersphere spaces to learn intra-modal hidden features, and then design a cross-modal associative prompt layer to perform anchor point masking and swap feature filling for constructing a hybrid cross-modal associative prompt. Afterwards, we exploit a unified semantic encoder to learn their cross-modal interactive features for context adaptation. Finally, we design an associative mapping classification layer to learn potential associative mappings between modalities at anchor points, within which we develop a fresh self-supervised associative mapping classification task to boost CMAL's performance. Experimental results verify the effectiveness of CMAL, showing that it achieves competitive performance against previous CMCL-based methods on four common downstream vision-and-language tasks, with significantly fewer corpus. Especially, CMAL obtains new state-of-the-art results on SNLI-VE and REC (testA).

cs.CV↗

arXiv2024

Exploring Adversarial Robustness of Deep State Space Models

Deep State Space Models (SSMs) have proven effective in numerous task scenarios but face significant security challenges due to Adversarial Perturbations (APs) in real-world deployments. Adversarial Training (AT) is a mainstream approach to enhancing Adversarial Robustness (AR) and has been validated on various traditional DNN architectures. However, its effectiveness in improving the AR of SSMs remains unclear. While many enhancements in SSM components, such as integrating Attention mechanisms and expanding to data-dependent SSM parameterizations, have brought significant gains in Standard Training (ST) settings, their potential benefits in AT remain unexplored. To investigate this, we evaluate existing structural variants of SSMs with AT to assess their AR performance. We observe that pure SSM structures struggle to benefit from AT, whereas incorporating Attention yields a markedly better trade-off between robustness and generalization for SSMs in AT compared to other components. Nonetheless, the integration of Attention also leads to Robust Overfitting (RO) issues. To understand these phenomena, we empirically and theoretically analyze the output error of SSMs under AP. We find that fixed-parameterized SSMs have output error bounds strictly related to their parameters, limiting their AT benefits, while input-dependent SSMs may face the problem of error explosion. Furthermore, we show that the Attention component effectively scales the output error of SSMs during training, enabling them to benefit more from AT, but at the cost of introducing RO due to its high model complexity. Inspired by this, we propose a simple and effective Adaptive Scaling (AdS) mechanism that brings AT performance close to Attention-integrated SSMs without introducing the issue of RO. Our code is available at https://github.com/Biqing-Qi/Exploring-Adversarial-Robustness-of-Deep-State-Space-Models.git.

cs.LG↗

arXiv2024

JWST view of four infant galaxies at z=8.31-8.49 in the MACS0416 field and implications for reionization

New JWST/NIRCam wide-field slitless spectroscopy provides redshifts for four z>8 galaxies located behind the lensing cluster MACS J0416.1-2403. Two of them, "Y1" and "JD", have previously reported spectroscopic redshifts based on ALMA measurements of [OIII] 88 $μ$m and/or [CII] 157.7 $μ$m lines. Y1 is a merging system of three components, and the existing redshift z=8.31 is confirmed. However, JD is at z=8.34 instead of the previously claimed z=9.28. JD's close companion, "JD-N", which was a previously discovered z>8 candidate, is now identified at the same redshift as JD. JD and JD-N form an interacting pair. A new candidate at z>8, "f090d_018", is also confirmed and is at z=8.49. These four objects are likely part of an overdensity that signposts a large structure extending ~165 kpc in projected distance and ~48.7 Mpc in radial distance. They are magnified by less than one magnitude and have intrinsic $M_{UV}$ ranging from -19.57 to -20.83 mag. Their spectral energy distributions show that the galaxies are all very young with ages ~ 4-18 Myr and stellar masses about $10^{7-8}$ ${\rm M_\odot}$. These infant galaxies have very different star formation rates ranging from a few to over a hundred $\rm{M_\odot}$ yr$^{-1}$, but only two of them (JD and f090d_018) have blue rest-frame UV slopes $β<-2.0$ indicative of a high Lyman-continuum photon escape fraction that could contribute significantly to the cosmic hydrogen-reionizing background. Interestingly, these two galaxies are the least massive and least active ones among the four. The other two systems have much flatter UV slopes largely because of their high dust extinction ($A_{\rm V}$=0.9-1.0 mag). Their much lower indicated escape fractions show that even very young, actively star-forming galaxies can have negligible contribution to reionization when they quickly form dust throughout their bodies.

astro-ph.GA↗

arXiv2024

Neural Residual Diffusion Models for Deep Scalable Vision Generation

The most advanced diffusion models have recently adopted increasingly deep stacked networks (e.g., U-Net or Transformer) to promote the generative emergence capabilities of vision generation models similar to large language models (LLMs). However, progressively deeper stacked networks will intuitively cause numerical propagation errors and reduce noisy prediction capabilities on generative data, which hinders massively deep scalable training of vision generation models. In this paper, we first uncover the nature that neural networks being able to effectively perform generative denoising lies in the fact that the intrinsic residual unit has consistent dynamic property with the input signal's reverse diffusion process, thus supporting excellent generative abilities. Afterwards, we stand on the shoulders of two common types of deep stacked networks to propose a unified and massively scalable Neural Residual Diffusion Models framework (Neural-RDM for short), which is a simple yet meaningful change to the common architecture of deep generative networks by introducing a series of learnable gated residual parameters that conform to the generative dynamics. Experimental results on various generative tasks show that the proposed neural residual models obtain state-of-the-art scores on image's and video's generative benchmarks. Rigorous theoretical proofs and extensive experiments also demonstrate the advantages of this simple gated residual mechanism consistent with dynamic modeling in improving the fidelity and consistency of generated content and supporting large-scale scalable training. Code is available at https://github.com/Anonymous/Neural-RDM.

cs.CV↗

arXiv2024

Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking

Recently, stable diffusion (SD) models have typically flourished in the field of image synthesis and personalized editing, with a range of photorealistic and unprecedented images being successfully generated. As a result, widespread interest has been ignited to develop and use various SD-based tools for visual content creation. However, the exposure of AI-created content on public platforms could raise both legal and ethical risks. In this regard, the traditional methods of adding watermarks to the already generated images (i.e. post-processing) may face a dilemma (e.g., being erased or modified) in terms of copyright protection and content monitoring, since the powerful image inversion and text-to-image editing techniques have been widely explored in SD-based methods. In this work, we propose a Safe and high-traceable Stable Diffusion framework (namely Safe-SD) to adaptively implant the graphical watermarks (e.g., QR code) into the imperceptible structure-related pixels during the generative diffusion process for supporting text-driven invisible watermarking and detection. Different from the previous high-cost injection-then-detection training framework, we design a simple and unified architecture, which makes it possible to simultaneously train watermark injection and detection in a single network, greatly improving the efficiency and convenience of use. Moreover, to further support text-driven generative watermarking and deeply explore its robustness and high-traceability, we elaborately design lambda sampling and encryption algorithm to fine-tune a latent diffuser wrapped by a VAE for balancing high-fidelity image synthesis and high-traceable watermark detection. We present our quantitative and qualitative results on two representative datasets LSUN, COCO and FFHQ, demonstrating state-of-the-art performance of Safe-SD and showing it significantly outperforms the previous approaches.

cs.CV↗

arXiv2024

Dense Multimodal Alignment for Open-Vocabulary 3D Scene Understanding

Recent vision-language pre-training models have exhibited remarkable generalization ability in zero-shot recognition tasks. Previous open-vocabulary 3D scene understanding methods mostly focus on training 3D models using either image or text supervision while neglecting the collective strength of all modalities. In this work, we propose a Dense Multimodal Alignment (DMA) framework to densely co-embed different modalities into a common space for maximizing their synergistic benefits. Instead of extracting coarse view- or region-level text prompts, we leverage large vision-language models to extract complete category information and scalable scene descriptions to build the text modality, and take image modality as the bridge to build dense point-pixel-text associations. Besides, in order to enhance the generalization ability of the 2D model for downstream 3D tasks without compromising the open-vocabulary capability, we employ a dual-path integration approach to combine frozen CLIP visual features and learnable mask features. Extensive experiments show that our DMA method produces highly competitive open-vocabulary segmentation performance on various indoor and outdoor tasks.

cs.CV↗

arXiv2024

A Strongly Lensed Dusty Starburst of an Intrinsic Disk Morphology at Photometric Redshift of $z_{\rm ph}>7$

We present COSBO-7, a strong millimeter (mm) source known for more than sixteen years but was just revealed its near-to-mid-IR counterpart by the James Webb Space Telescope (JWST). The precise pin-pointing by the Atacama Large Millimeter Array (ALMA) on the exquisite NIRCam and MIRI images show that it is a background source gravitationally lensed by a single foreground galaxy, and the analysis of its spectral energy distribution by different tools is in favor of photometric redshift at $z_{\rm ph}>7$. Strikingly, our lens modeling based on the JWST data shows that it has a regular, disk morphology in the source plane. The dusty region giving rise to the far-IR-to-mm emission seems to be confined to a limited region to one side of the disk and has a high dust temperature of $>90$~K. The galaxy is experiencing starburst both within and outside of this dusty region. After taking the lensing magnification of $μ\approx 2.5-3.6$ into account, the intrinsic star formation rate is several hundred $M_\odot$~yr$^{-1}$ both within the dusty region and across the more extended stellar disk, and the latter already has $>10^{10}M_\odot$ of stars in place. If it is indeed at $z>7$, COSBO-7 presents an extraordinary case that is against the common wisdom about galaxy formation in the early universe; simply put, its existence poses a critical question to be answered: how could a massive disk galaxy come into being so early in the universe and sustain its regular morphology in the middle of an enormous starburst?

astro-ph.GA↗

arXiv2024

ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation

By leveraging the text-to-image diffusion priors, score distillation can synthesize 3D contents without paired text-3D training data. Instead of spending hours of online optimization per text prompt, recent studies have been focused on learning a text-to-3D generative network for amortizing multiple text-3D relations, which can synthesize 3D contents in seconds. However, existing score distillation methods are hard to scale up to a large amount of text prompts due to the difficulties in aligning pretrained diffusion prior with the distribution of rendered images from various text prompts. Current state-of-the-arts such as Variational Score Distillation finetune the pretrained diffusion model to minimize the noise prediction error so as to align the distributions, which are however unstable to train and will impair the model's comprehension capability to numerous text prompts. Based on the observation that the diffusion models tend to have lower noise prediction errors at earlier timesteps, we propose Asynchronous Score Distillation (ASD), which minimizes the noise prediction error by shifting the diffusion timestep to earlier ones. ASD is stable to train and can scale up to 100k prompts. It reduces the noise prediction error without changing the weights of pre-trained diffusion model, thus keeping its strong comprehension capability to prompts. We conduct extensive experiments across different 2D diffusion models, including Stable Diffusion and MVDream, and text-to-3D generators, including Hyper-iNGP, 3DConv-Net and Triplane-Transformer. The results demonstrate ASD's effectiveness in stable 3D generator training, high-quality 3D content synthesis, and its superior prompt-consistency, especially under large prompt corpus.

cs.CV↗

arXiv2024

Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models

With the emergence of pre-trained vision-language models like CLIP, how to adapt them to various downstream classification tasks has garnered significant attention in recent research. The adaptation strategies can be typically categorized into three paradigms: zero-shot adaptation, few-shot adaptation, and the recently-proposed training-free few-shot adaptation. Most existing approaches are tailored for a specific setting and can only cater to one or two of these paradigms. In this paper, we introduce a versatile adaptation approach that can effectively work under all three settings. Specifically, we propose the dual memory networks that comprise dynamic and static memory components. The static memory caches training data knowledge, enabling training-free few-shot adaptation, while the dynamic memory preserves historical test features online during the testing process, allowing for the exploration of additional data insights beyond the training set. This novel capability enhances model performance in the few-shot setting and enables model usability in the absence of training data. The two memory networks employ the same flexible memory interactive strategy, which can operate in a training-free mode and can be further enhanced by incorporating learnable projection layers. Our approach is tested across 11 datasets under the three task settings. Remarkably, in the zero-shot scenario, it outperforms existing methods by over 3\% and even shows superior results against methods utilizing external training data. Additionally, our method exhibits robust performance against natural distribution shifts. Codes are available at \url{https://github.com/YBZh/DMN}.

cs.CV↗

arXiv2024

DiffSpeaker: Speech-Driven 3D Facial Animation with Diffusion Transformer

Speech-driven 3D facial animation is important for many multimedia applications. Recent work has shown promise in using either Diffusion models or Transformer architectures for this task. However, their mere aggregation does not lead to improved performance. We suspect this is due to a shortage of paired audio-4D data, which is crucial for the Transformer to effectively perform as a denoiser within the Diffusion framework. To tackle this issue, we present DiffSpeaker, a Transformer-based network equipped with novel biased conditional attention modules. These modules serve as substitutes for the traditional self/cross-attention in standard Transformers, incorporating thoughtfully designed biases that steer the attention mechanisms to concentrate on both the relevant task-specific and diffusion-related conditions. We also explore the trade-off between accurate lip synchronization and non-verbal facial expressions within the Diffusion paradigm. Experiments show our model not only achieves state-of-the-art performance on existing benchmarks, but also fast inference speed owing to its ability to generate facial motions in parallel.

cs.CV↗

arXiv2024

EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation

Implementing fine-grained emotion control is crucial for emotion generation tasks because it enhances the expressive capability of the generative model, allowing it to accurately and comprehensively capture and express various nuanced emotional states, thereby improving the emotional quality and personalization of generated content. Generating fine-grained facial animations that accurately portray emotional expressions using only a portrait and an audio recording presents a challenge. In order to address this challenge, we propose a visual attribute-guided audio decoupler. This enables the obtention of content vectors solely related to the audio content, enhancing the stability of subsequent lip movement coefficient predictions. To achieve more precise emotional expression, we introduce a fine-grained emotion coefficient prediction module. Additionally, we propose an emotion intensity control method using a fine-grained emotion matrix. Through these, effective control over emotional expression in the generated videos and finer classification of emotion intensity are accomplished. Subsequently, a series of 3DMM coefficient generation networks are designed to predict 3D coefficients, followed by the utilization of a rendering network to generate the final video. Our experimental results demonstrate that our proposed method, EmoSpeaker, outperforms existing emotional talking face generation methods in terms of expression variation and lip synchronization. Project page: https://peterfanfan.github.io/EmoSpeaker/

cs.CV↗

arXiv2024

JWST's PEARLS: Improved Flux Calibration for NIRCam

The Prime Extragalactic Areas for Reionization and Lensing Science (PEARLS), a JWST GTO program, obtained a set of unique NIRCam observations that have enabled us to significantly improve the default photometric calibration across both NIRCam modules. The observations consisted of three epochs of 4-band (F150W, F200W, F356W, and F444W) NIRCam imaging in the Spitzer IRAC Dark Field (IDF). The three epochs were six months apart and spanned the full duration of Cycle 1. As the IDF is in the JWST continuous viewing zone, we were able to design the observations such that the two modules of NIRCam, modules A and B, were flipped by 180 degrees and completely overlapped each other's footprints in alternate epochs. We were therefore able to directly compare the photometry of the same objects observed with different modules and detectors, and we found significant photometric residuals up to ~ 0.05 mag in some detectors and filters, for the default version of the calibration files that we used (jwst_1039.pmap). Moreover, there are multiplicative gradients present in the data obtained in the two long-wavelength bands. The problem is less severe in the data reduced using the latest pmap (jwst_1130.pmap as of September 2023), but it is still present, and is non-negligible. We provide a recipe to correct for this systematic effect to bring the two modules onto a more consistent calibration, to a photometric precision better than ~ 0.02 mag.

astro-ph.IM↗

arXiv2024

Generative Multi-Modal Knowledge Retrieval with Large Language Models

Knowledge retrieval with multi-modal queries plays a crucial role in supporting knowledge-intensive multi-modal applications. However, existing methods face challenges in terms of their effectiveness and training efficiency, especially when it comes to training and integrating multiple retrievers to handle multi-modal queries. In this paper, we propose an innovative end-to-end generative framework for multi-modal knowledge retrieval. Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases, even when trained with limited data. We retrieve knowledge via a two-step process: 1) generating knowledge clues related to the queries, and 2) obtaining the relevant document by searching databases using the knowledge clue. In particular, we first introduce an object-aware prefix-tuning technique to guide multi-grained visual learning. Then, we align multi-grained visual features into the textual feature space of the LLM, employing the LLM to capture cross-modal interactions. Subsequently, we construct instruction data with a unified format for model training. Finally, we propose the knowledge-guided generation strategy to impose prior constraints in the decoding steps, thereby promoting the generation of distinctive knowledge clues. Through experiments conducted on three benchmarks, we demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.

cs.IR↗

arXiv2023

AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing

With the great success of text-conditioned diffusion models in creative text-to-image generation, various text-driven image editing approaches have attracted the attentions of many researchers. However, previous works mainly focus on discreteness-sensitive instructions such as adding, removing or replacing specific objects, background elements or global styles (i.e., hard editing), while generally ignoring subject-binding but semantically fine-changing continuity-sensitive instructions such as actions, poses or adjectives, and so on (i.e., soft editing), which hampers generative AI from generating user-customized visual contents. To mitigate this predicament, we propose a spatio-temporal guided adaptive editing algorithm AdapEdit, which realizes adaptive image editing by introducing a soft-attention strategy to dynamically vary the guiding degree from the editing conditions to visual pixels from both temporal and spatial perspectives. Note our approach has a significant advantage in preserving model priors and does not require model training, fine-tuning, extra data, or optimization. We present our results over a wide variety of raw images and editing instructions, demonstrating competitive performance and showing it significantly outperforms the previous approaches.

cs.CV↗

arXiv2023

LMD: Faster Image Reconstruction with Latent Masking Diffusion

As a class of fruitful approaches, diffusion probabilistic models (DPMs) have shown excellent advantages in high-resolution image reconstruction. On the other hand, masked autoencoders (MAEs), as popular self-supervised vision learners, have demonstrated simpler and more effective image reconstruction and transfer capabilities on downstream tasks. However, they all require extremely high training costs, either due to inherent high temporal-dependence (i.e., excessively long diffusion steps) or due to artificially low spatial-dependence (i.e., human-formulated high mask ratio, such as 0.75). To the end, this paper presents LMD, a faster image reconstruction framework with latent masking diffusion. First, we propose to project and reconstruct images in latent space through a pre-trained variational autoencoder, which is theoretically more efficient than in the pixel-based space. Then, we combine the advantages of MAEs and DPMs to design a progressive masking diffusion model, which gradually increases the masking proportion by three different schedulers and reconstructs the latent features from simple to difficult, without sequentially performing denoising diffusion as in DPMs or using fixed high masking ratio as in MAEs, so as to alleviate the high training time-consumption predicament. Our approach allows for learning high-capacity models and accelerate their training (by 3x or more) and barely reduces the original accuracy. Inference speed in downstream tasks also significantly outperforms the previous approaches.

cs.CV↗

arXiv2023

Candidate Galaxies at z ~ 11.3--21.8 and beyond: results from JWST's public data taken in its first year

We present a systematic search of candidate galaxies at z > 11.3 using the public Near Infrared Camera data taken by the James Webb Space Telescope (JWST) in its Cycle 1, which include six blank fields totalling 386 sq.arcmin and two lensing cluster fields totalling 48 sq.arcmin. The candidates are selected as F150W, F200W and F277W dropouts, which correspond to z ~ 12.7 (11.3 < z < 15.4), 17.3 (15.4 < z < 21.8) and 24.7 (21.8 < z < 28.3), respectively. Our sample consists of 123 F150W dropouts, 52 F200W dropouts and 32 F277W dropouts, which is the largest candidate galaxy sample probing the highest redshift range to date. The F150W and F200W dropouts have sufficient photometric information that allows contaminant rejection, which we do by fitting to their spectrum energy distributions. Based on the purified samples of F150W and F200W dropouts, we derive galaxy luminosity functions at z ~ 12.7 and 17.3, respectively. We find that both are better described by power law than Schechter function and that there is only a marginal evolution (a factor of < 2) between the two epochs. The emergence of galaxy population at z ~ 17.3 or earlier is consistent with the suggestion of an early cosmic hydrogen reionization and is not necessarily a crisis of the LCDM paradigm. To establish a new picture of galaxy formation in the early universe, we will need both JWST spectroscopic confirmation of bright candidates such as those in our sample and deeper surveys to further constrain the faint-end of the luminosity function at M > -18 mag.

astro-ph.GA↗

arXiv2023

JWST's PEARLS: Transients in the MACS J0416.1-2403 Field

With its unprecedented sensitivity and spatial resolution, the James Webb Space Telescope (JWST) has opened a new window for time-domain discoveries in the infrared. Here we report observations in the only field that has received four epochs (spanning 126 days) of JWST NIRCam observations in Cycle 1. This field is towards MACS J0416.1-2403, which is a rich galaxy cluster at redshift z=0.4 and is one of the Hubble Frontier Fields. We have discovered 14 transients from these data. Twelve of these transients happened in three galaxies (with z=0.94, 1.01, and 2.091) crossing a lensing caustic of the cluster,and these transients are highly magnified by gravitational lensing. These 12 transients are likely of similar nature to those previously reported based on the Hubble Space Telescope (HST) data in this field, i.e., individual stars in the highly magnified arcs. However, these twelve could not have been found by HST because they are too red and too faint. The other two transients are associated with background galaxies (z=2.205 and 0.7093) that are only moderately magnified, and they are likely supernovae. They indicate a de-magnified supernova surface density, when monitored at a time cadence of a few months to a ~3--4 micron survey limit of AB ~ 28.5 mag, of ~0.5 per sq. arcmin integrated to z ~ 2. This survey depth is beyond the capability of HST but can be easily reached by JWST.

astro-ph.GA↗

arXiv2023

Point-like sources among z>11 galaxy candidates: contaminants due to supernovae at high redshifts?

The recent searches for z>11 galaxies using the James Webb Space Telescope have resulted in an unexpectedly high number of candidate objects, which imply at least an order of magnitude higher number density of z>11 galaxies than the previously favored predictions. A question has risen whether there are some new types of contaminants among these candidates. The candidate sample of Yan et al. (2023a), totalling 87 dropouts, is the largest one, and we notice that a number of these candidates are point-like. We hypothesize that the point-source dropouts could be supernovae at high redshifts. Further investigation shows that most of their spectral energy distributions indeed can be explained by supernovae at various redshifts from z ~ 1--15, which lends support to this hypothesis. Attributing such point-source dropouts to supernova contamination cannot eliminate the tension, however, because they only account for ~10% of the Yan et al.'s sample. On the other hand, the discovery of "contaminant" supernovae at z>3 will have a series of important implications. Ironically, the existence of supernovae at $z>10$ would still imply that the previously favored picture of early galaxy formation severely underestimates the global star formation rate density such redshifts. Multiple-epoch JWST imaging will be the simplest and yet the most efficient way to further test this hypothesis.

astro-ph.CO↗