arXiv subjects

Zhifei Zhang

Publications and source records attributed to Zhifei Zhang.

At least 127 records · Page 7Linked to original sources

arXiv2023

SImProv: Scalable Image Provenance Framework for Robust Content Attribution

We present SImProv - a scalable image provenance framework to match a query image back to a trusted database of originals and identify possible manipulations on the query. SImProv consists of three stages: a scalable search stage for retrieving top-k most similar images; a re-ranking and near-duplicated detection stage for identifying the original among the candidates; and finally a manipulation detection and visualization stage for localizing regions within the query that may have been manipulated to differ from the original. SImProv is robust to benign image transformations that commonly occur during online redistribution, such as artifacts due to noise and recompression degradation, as well as out-of-place transformations due to image padding, warping, and changes in size and shape. Robustness towards out-of-place transformations is achieved via the end-to-end training of a differentiable warping module within the comparator architecture. We demonstrate effective retrieval and manipulation detection over a dataset of 100 million images.

cs.CV↗

arXiv2023

Improving Diffusion Models for Scene Text Editing with Dual Encoders

Scene text editing is a challenging task that involves modifying or inserting specified texts in an image while maintaining its natural and realistic appearance. Most previous approaches to this task rely on style-transfer models that crop out text regions and feed them into image transfer models, such as GANs. However, these methods are limited in their ability to change text style and are unable to insert texts into images. Recent advances in diffusion models have shown promise in overcoming these limitations with text-conditional image editing. However, our empirical analysis reveals that state-of-the-art diffusion models struggle with rendering correct text and controlling text style. To address these problems, we propose DIFFSTE to improve pre-trained diffusion models with a dual encoder design, which includes a character encoder for better text legibility and an instruction encoder for better style control. An instruction tuning framework is introduced to train our model to learn the mapping from the text instruction to the corresponding image with either the specified style or the style of the surrounding texts in the background. Such a training method further brings our method the zero-shot generalization ability to the following three scenarios: generating text with unseen font variation, e.g., italic and bold, mixing different fonts to construct a new font, and using more relaxed forms of natural language as the instructions to guide the generation task. We evaluate our approach on five datasets and demonstrate its superior performance in terms of text correctness, image naturalness, and style controllability. Our code is publicly available. https://github.com/UCSB-NLP-Chang/DiffSTE

cs.CV↗

arXiv2023

On the mean-field and semiclassical limit from quantum $N$-body dynamics

We study the mean-field and semiclassical limit of the quantum many-body dynamics with a repulsive $δ$-type potential $N^{3β}V(N^βx)$ and a Coulomb potential, which leads to a macroscopic fluid equation, the Euler-Poisson equation with pressure. We prove quantitative strong convergence of the quantum mass and momentum densities up to the first blow up time of the limiting equation. The main ingredient is a functional inequality on the $δ$-type potential for the almost optimal case $β\in(0,1)$, for which we give an analysis of the singular correlation structure between particles.

math.AP↗

arXiv2023

TopNet: Transformer-based Object Placement Network for Image Compositing

We investigate the problem of automatically placing an object into a background image for image compositing. Given a background image and a segmented object, the goal is to train a model to predict plausible placements (location and scale) of the object for compositing. The quality of the composite image highly depends on the predicted location/scale. Existing works either generate candidate bounding boxes or apply sliding-window search using global representations from background and object images, which fail to model local information in background images. However, local clues in background images are important to determine the compatibility of placing the objects with certain locations/scales. In this paper, we propose to learn the correlation between object features and all local background features with a transformer module so that detailed information can be provided on all possible location/scale configurations. A sparse contrastive loss is further proposed to train our model with sparse supervision. Our new formulation generates a 3D heatmap indicating the plausibility of all location/scale combinations in one network forward pass, which is over 10 times faster than the previous sliding-window method. It also supports interactive search when users provide a pre-defined location or scale. The proposed method can be trained with explicit annotation or in a self-supervised manner using an off-the-shelf inpainting model, and it outperforms state-of-the-art methods significantly. The user study shows that the trained model generalizes well to real-world images with diverse challenging scenes and object categories.

cs.CV↗

arXiv2023

Nonlinear inviscid damping for 2-D inhomogeneous incompressible Euler equations

We prove the asymptotic stability of shear flows close to the Couette flow for the 2-D inhomogeneous incompressible Euler equations on $\mathbb{T}\times \mathbb{R}$. More precisely, if the initial velocity is close to the Couette flow and the initial density is close to a positive constant in the Gevrey class 2, then 2-D inhomogeneous incompressible Euler equations are globally well-posed and the velocity converges strongly to a shear flow close to the Couette flow, and the vorticity will be driven to small scales by a linear evolution and weakly converges as $t\to \infty$. To our knowledge, this is the first global well-posedness result for the 2-D inhomogeneous incompressible Euler equations.

math.AP↗

arXiv2022

SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Model

Generic image inpainting aims to complete a corrupted image by borrowing surrounding information, which barely generates novel content. By contrast, multi-modal inpainting provides more flexible and useful controls on the inpainted content, \eg, a text prompt can be used to describe an object with richer attributes, and a mask can be used to constrain the shape of the inpainted object rather than being only considered as a missing area. We propose a new diffusion-based model named SmartBrush for completing a missing region with an object using both text and shape-guidance. While previous work such as DALLE-2 and Stable Diffusion can do text-guided inapinting they do not support shape guidance and tend to modify background texture surrounding the generated object. Our model incorporates both text and shape guidance with precision control. To preserve the background better, we propose a novel training and sampling strategy by augmenting the diffusion U-net with object-mask prediction. Lastly, we introduce a multi-task training strategy by jointly training inpainting with text-to-image generation to leverage more training data. We conduct extensive experiments showing that our model outperforms all baselines in terms of visual quality, mask controllability, and background preservation.

cs.CV↗

arXiv2022

ObjectStitch: Generative Object Compositing

Object compositing based on 2D images is a challenging problem since it typically involves multiple processing stages such as color harmonization, geometry correction and shadow generation to generate realistic results. Furthermore, annotating training data pairs for compositing requires substantial manual effort from professionals, and is hardly scalable. Thus, with the recent advances in generative models, in this work, we propose a self-supervised framework for object compositing by leveraging the power of conditional diffusion models. Our framework can hollistically address the object compositing task in a unified model, transforming the viewpoint, geometry, color and shadow of the generated object while requiring no manual labeling. To preserve the input object's characteristics, we introduce a content adaptor that helps to maintain categorical semantics and object appearance. A data augmentation method is further adopted to improve the fidelity of the generator. Our method outperforms relevant baselines in both realism and faithfulness of the synthesized result images in a user study on various real-world images.

cs.CV↗

arXiv2022

Quantitative Green's Function Estimates for Lattice Quasi-periodic Schrödinger Operators

In this paper, we establish quantitative Green's function estimates for some higher dimensional lattice quasi-periodic (QP) Schrödinger operators. The resonances in the estimates can be described via a pair of symmetric zeros of certain functions and the estimates apply to the sub-exponential type non-resonant conditions. As the application of quantitative Green's function estimates, we prove both the arithmetic version of Anderson localization and the $(\frac 12-)$-Hölder continuity of the integrated density of states (IDS) for such QP Schrödinger operators. This gives an affirmative answer to Bourgain's problem in\cite{Bou00}.

math-ph↗

arXiv2022

Global stability of the compressible viscous surface waves in an infinite layer

We investigate in this paper the global stability of the compressible viscous surface waves in the absence of surface tension effect with a steady-state violating Rayleigh-Taylor instability and the reference domain being the horizontal infinite layer. The fluid dynamics are governed by the 3-D gravity-driven isentropic compressible Navier-Stokes equations. We develop a mathematical approach to establish global well-posedness of free boundary problems of the multi-dimensional compressible Navier-Stokes system based on the Lagrangian framework, which requires no nonlinear compatibility conditions on the initial data.

math.AP↗

arXiv2022

The number of traveling wave families in a running water with Coriolis force

In this paper, we study the number of traveling wave families near a shear flow under the influence of Coriolis force, where the traveling speeds lie outside the range of the flow $u$. Under the $β$-plane approximation, if the flow $u$ has a critical point at which $u$ attains its minimal (resp. maximal) value, then a unique transitional $β$ value exists in the positive (resp. negative) half-line such that the number of traveling wave families near the shear flow changes suddenly from finite to infinite when $β$ passes through it. On the other hand, if $u$ has no such critical points, then the number is always finite for positive (resp. negative) $β$ values. This is true for general shear flows under mildly technical assumptions, and for a large class of shear flows including a cosine jet $u(y) = {1+\cos(πy)\over 2}$ (i.e. the sinus profile) and analytic monotone flows unconditionally. The sudden change of the number of traveling wave families indicates that long time dynamics around the shear flow is much richer than the non-rotating case, where no such traveling wave families exist.

math.AP↗

arXiv2022

Dynamic stability for steady Prandtl solutions

By establishing an invariant set (1.11) for the Prandtl equation in Crocco transformation, we prove orbital and asymptotic stability of Blasius-like steady states against Oleinik's monotone solutions.

math.AP↗

arXiv2022

Toward Understanding WordArt: Corner-Guided Transformer for Scene Text Recognition

Artistic text recognition is an extremely challenging task with a wide range of applications. However, current scene text recognition methods mainly focus on irregular text while have not explored artistic text specifically. The challenges of artistic text recognition include the various appearance with special-designed fonts and effects, the complex connections and overlaps between characters, and the severe interference from background patterns. To alleviate these problems, we propose to recognize the artistic text at three levels. Firstly, corner points are applied to guide the extraction of local features inside characters, considering the robustness of corner structures to appearance and shape. In this way, the discreteness of the corner points cuts off the connection between characters, and the sparsity of them improves the robustness for background interference. Secondly, we design a character contrastive loss to model the character-level feature, improving the feature representation for character classification. Thirdly, we utilize Transformer to learn the global feature on image-level and model the global relationship of the corner points, with the assistance of a corner-query cross-attention mechanism. Besides, we provide an artistic text dataset to benchmark the performance. Experimental results verify the significant superiority of our proposed method on artistic text recognition and also achieve state-of-the-art performance on several blurred and perspective datasets.

cs.CV↗

arXiv2022

The unconditional uniqueness for the energy-supercritical NLS

We consider the cubic and quintic nonlinear Schrödinger equations (NLS) under the $\mathbb{R}^{d}$ and $\mathbb{T}^{d}$ energy-supercritical setting. Via a newly developed unified scheme, we prove the unconditional uniqueness for solutions to NLS at critical regularity for all dimensions. Thus, together with [18,19], the unconditional uniqueness problems for $H^{1}$-critical and $H^{1}$-supercritical cubic and quintic NLS are completely and uniformly resolved at critical regularity for these domains. One application of our theorem is to prove that defocusing blowup solutions of the type in [54] is the only possible $C([0,T);\dot{H}^{s_{c}})$ solution if exist in these domains.

math.AP↗

arXiv2022

Vanilla Feature Distillation for Improving the Accuracy-Robustness Trade-Off in Adversarial Training

Adversarial training has been widely explored for mitigating attacks against deep models. However, most existing works are still trapped in the dilemma between higher accuracy and stronger robustness since they tend to fit a model towards robust features (not easily tampered with by adversaries) while ignoring those non-robust but highly predictive features. To achieve a better robustness-accuracy trade-off, we propose the Vanilla Feature Distillation Adversarial Training (VFD-Adv), which conducts knowledge distillation from a pre-trained model (optimized towards high accuracy) to guide adversarial training towards higher accuracy, i.e., preserving those non-robust but predictive features. More specifically, both adversarial examples and their clean counterparts are forced to be aligned in the feature space by distilling predictive representations from the pre-trained/clean model, while previous works barely utilize predictive features from clean models. Therefore, the adversarial training model is updated towards maximally preserving the accuracy as gaining robustness. A key advantage of our method is that it can be universally adapted to and boost existing works. Exhaustive experiments on various datasets, classification models, and adversarial training algorithms demonstrate the effectiveness of our proposed method.

cs.CV↗

arXiv2022

GALA: Toward Geometry-and-Lighting-Aware Object Search for Compositing

Compositing-aware object search aims to find the most compatible objects for compositing given a background image and a query bounding box. Previous works focus on learning compatibility between the foreground object and background, but fail to learn other important factors from large-scale data, i.e. geometry and lighting. To move a step further, this paper proposes GALA (Geometry-and-Lighting-Aware), a generic foreground object search method with discriminative modeling on geometry and lighting compatibility for open-world image compositing. Remarkably, it achieves state-of-the-art results on the CAIS dataset and generalizes well on large-scale open-world datasets, i.e. Pixabay and Open Images. In addition, our method can effectively handle non-box scenarios, where users only provide background images without any input bounding box. A web demo (see supplementary materials) is built to showcase applications of the proposed method for compositing-aware search and automatic location/scale prediction for the foreground object.

cs.CV↗

arXiv2022

Fairness-aware Adversarial Perturbation Towards Bias Mitigation for Deployed Deep Models

Prioritizing fairness is of central importance in artificial intelligence (AI) systems, especially for those societal applications, e.g., hiring systems should recommend applicants equally from different demographic groups, and risk assessment systems must eliminate racism in criminal justice. Existing efforts towards the ethical development of AI systems have leveraged data science to mitigate biases in the training set or introduced fairness principles into the training process. For a deployed AI system, however, it may not allow for retraining or tuning in practice. By contrast, we propose a more flexible approach, i.e., fairness-aware adversarial perturbation (FAAP), which learns to perturb input data to blind deployed models on fairness-related features, e.g., gender and ethnicity. The key advantage is that FAAP does not modify deployed models in terms of parameters and structures. To achieve this, we design a discriminator to distinguish fairness-related attributes based on latent representations from deployed models. Meanwhile, a perturbation generator is trained against the discriminator, such that no fairness-related features could be extracted from perturbed inputs. Exhaustive experimental evaluation demonstrates the effectiveness and superior performance of the proposed FAAP. In addition, FAAP is validated on real-world commercial deployments (inaccessible to model parameters), which shows the transferability of FAAP, foreseeing the potential of black-box adaptation.

cs.LG↗

arXiv2022

Feature Importance-aware Transferable Adversarial Attacks

Transferability of adversarial examples is of central importance for attacking an unknown model, which facilitates adversarial attacks in more practical scenarios, e.g., black-box attacks. Existing transferable attacks tend to craft adversarial examples by indiscriminately distorting features to degrade prediction accuracy in a source model without aware of intrinsic features of objects in the images. We argue that such brute-force degradation would introduce model-specific local optimum into adversarial examples, thus limiting the transferability. By contrast, we propose the Feature Importance-aware Attack (FIA), which disrupts important object-aware features that dominate model decisions consistently. More specifically, we obtain feature importance by introducing the aggregate gradient, which averages the gradients with respect to feature maps of the source model, computed on a batch of random transforms of the original clean image. The gradients will be highly correlated to objects of interest, and such correlation presents invariance across different models. Besides, the random transforms will preserve intrinsic features of objects and suppress model-specific information. Finally, the feature importance guides to search for adversarial examples towards disrupting critical features, achieving stronger transferability. Extensive experimental evaluation demonstrates the effectiveness and superior performance of the proposed FIA, i.e., improving the success rate by 9.5% against normally trained models and 12.8% against defense models as compared to the state-of-the-art transferable attacks. Code is available at: https://github.com/hcguoO0/FIA

cs.CV↗

arXiv2022

Dynamics near Couette flow for the $β$-plane equation

In this paper, we study stationary structures near the planar Couette flow in Sobolev spaces on a channel $\mathbb{T}\times[-1,1]$, and asymptotic behavior of Couette flow in Gevrey spaces on $\mathbb{T}\times\mathbb{R}$ for the $β$-plane equation. Let $T>0$ be the horizontal period of the channel and $α={2π\over T}$ be the wave number. We obtain a sharp region $O$ in the whole $(α,β)$ half-plane such that non-parallel steadily traveling waves do not exist for $(α,β)\in O$ and such traveling waves exist for $(α,β)$ in the remaining regions, near Couette flow for $H^{\geq5}$ velocity perturbation. The borderlines between the region $O$ and its remaining are determined by two curves of the principal eigenvalues of singular Rayleigh-Kuo operators. Our results reveal that there exists $β_*>0$ such that if $|β|\leq β_*$, then non-parallel traveling waves do not exist for any $T>0$, while if $|β|>β_*$, then there exists a critical period $T_β>0$ so that such traveling waves exist for $T\in \left[T_β,\infty\right)$ and do not exist for $T\in \left(0,T_β\right)$, near Couette flow for $H^{\geq5}$ velocity perturbation. This contrasting dynamics plays an important role in studying the long time dynamics near Couette flow with Coriolis effects. Moreover, for any $β\neq0$ and $T>0$, there exist no non-parallel traveling waves with speeds converging in $(-1,1)$ near Couette flow for $H^{\geq5}$ velocity perturbation, in contrast to this, we construct non-shear stationary solutions near Couette flow for $H^{<{5\over2}}$ velocity perturbation, which is a generalization of Theorem 1 in [22] but the construction is more difficult due to the $β$'s term. Finally, we prove nonlinear inviscid damping for Couette flow in some Gevrey spaces by extending the method of [4] to the $β$-plane equation on $\mathbb{T}\times\mathbb{R}$.

math.AP↗