arXiv subjects

Ning Yu

Publications and source records attributed to Ning Yu.

At least 91 records · Page 5Linked to original sources

arXiv2022

RepMix: Representation Mixing for Robust Attribution of Synthesized Images

Rapid advances in Generative Adversarial Networks (GANs) raise new challenges for image attribution; detecting whether an image is synthetic and, if so, determining which GAN architecture created it. Uniquely, we present a solution to this task capable of 1) matching images invariant to their semantic content; 2) robust to benign transformations (changes in quality, resolution, shape, etc.) commonly encountered as images are re-shared online. In order to formalize our research, a challenging benchmark, Attribution88, is collected for robust and practical image attribution. We then propose RepMix, our GAN fingerprinting technique based on representation mixing and a novel loss. We validate its capability of tracing the provenance of GAN-generated images invariant to the semantic content of the image and also robust to perturbations. We show our approach improves significantly from existing GAN fingerprinting works on both semantic generalization and robustness. Data and code are available at https://github.com/TuBui/image_attribution.

cs.CV↗

arXiv2022

RelaxLoss: Defending Membership Inference Attacks without Losing Utility

As a long-term threat to the privacy of training data, membership inference attacks (MIAs) emerge ubiquitously in machine learning models. Existing works evidence strong connection between the distinguishability of the training and testing loss distributions and the model's vulnerability to MIAs. Motivated by existing results, we propose a novel training framework based on a relaxed loss with a more achievable learning target, which leads to narrowed generalization gap and reduced privacy leakage. RelaxLoss is applicable to any classification model with added benefits of easy implementation and negligible overhead. Through extensive evaluations on five datasets with diverse modalities (images, medical data, transaction records), our approach consistently outperforms state-of-the-art defense mechanisms in terms of resilience against MIAs as well as model utility. Our defense is the first that can withstand a wide range of attacks while preserving (or even improving) the target model's utility. Source code is available at https://github.com/DingfanChen/RelaxLoss

cs.LG↗

arXiv2022

Artificial Fingerprinting for Generative Models: Rooting Deepfake Attribution in Training Data

Photorealistic image generation has reached a new level of quality due to the breakthroughs of generative adversarial networks (GANs). Yet, the dark side of such deepfakes, the malicious use of generated media, raises concerns about visual misinformation. While existing research work on deepfake detection demonstrates high accuracy, it is subject to advances in generation techniques and adversarial iterations on detection countermeasure techniques. Thus, we seek a proactive and sustainable solution on deepfake detection, that is agnostic to the evolution of generative models, by introducing artificial fingerprints into the models. Our approach is simple and effective. We first embed artificial fingerprints into training data, then validate a surprising discovery on the transferability of such fingerprints from training data to generative models, which in turn appears in the generated deepfakes. Experiments show that our fingerprinting solution (1) holds for a variety of cutting-edge generative models, (2) leads to a negligible side effect on generation quality, (3) stays robust against image-level and model-level perturbations, (4) stays hard to be detected by adversaries, and (5) converts deepfake detection and attribution into trivial tasks and outperforms the recent state-of-the-art baselines. Our solution closes the responsibility loop between publishing pre-trained generative model inventions and their possible misuses, which makes it independent of the current arms race. Code and models are available at https://github.com/ningyu1991/ArtificialGANFingerprints .

cs.CR↗

arXiv2022

Responsible Disclosure of Generative Models Using Scalable Fingerprinting

Over the past years, deep generative models have achieved a new level of performance. Generated data has become difficult, if not impossible, to be distinguished from real data. While there are plenty of use cases that benefit from this technology, there are also strong concerns on how this new technology can be misused to generate deep fakes and enable misinformation at scale. Unfortunately, current deep fake detection methods are not sustainable, as the gap between real and fake continues to close. In contrast, our work enables a responsible disclosure of such state-of-the-art generative models, that allows model inventors to fingerprint their models, so that the generated samples containing a fingerprint can be accurately detected and attributed to a source. Our technique achieves this by an efficient and scalable ad-hoc generation of a large population of models with distinct fingerprints. Our recommended operation point uses a 128-bit fingerprint which in principle results in more than $10^{38}$ identifiable models. Experiments show that our method fulfills key properties of a fingerprinting mechanism and achieves effectiveness in deep fake detection and attribution. Code and models are available at https://github.com/ningyu1991/ScalableGANFingerprints .

cs.CR↗

arXiv2022

Dual Contrastive Loss and Attention for GANs

Generative Adversarial Networks (GANs) produce impressive results on unconditional image generation when powered with large-scale image datasets. Yet generated images are still easy to spot especially on datasets with high variance (e.g. bedroom, church). In this paper, we propose various improvements to further push the boundaries in image generation. Specifically, we propose a novel dual contrastive loss and show that, with this loss, discriminator learns more generalized and distinguishable representations to incentivize generation. In addition, we revisit attention and extensively experiment with different attention blocks in the generator. We find attention to be still an important module for successful image generation even though it was not used in the recent state-of-the-art models. Lastly, we study different attention architectures in the discriminator, and propose a reference attention mechanism. By combining the strengths of these remedies, we improve the compelling state-of-the-art Fréchet Inception Distance (FID) by at least 17.5% on several benchmark datasets. We obtain even more significant improvements on compositional synthetic scenes (up to 47.5% in FID). Code and models are available at https://github.com/ningyu1991/AttentionDualContrastGAN .

cs.CV↗

arXiv2022

Switchable selective interactions in a Dicke Model with Driven Biased term

In this work, we propose a method to investigate controllable qubit-resonator interactions in a Dicke model with driven biased term. The nonlinearity of spectrum, which can be induced by qubit-resonator interactions, plays an important role in such controllable interactions. To gain insight into mechanism of the nonlinearity, we perform a unitary transformation to the Hamiltonian. The results show that the nonlinearity of the transformed Hamiltonian depends on the qubit-resonator coupling strength. The general forms of the effective Hamiltonians are discussed in detail based on the frequency modulation approach. The dynamical evolution can be switched on and off by adjusting the modulation parameters. By utilizing such controllable interactions, we discuss the creation of Dicke states and arbitrary superposition of Dicke states. We also consider the nonlinearity of energy level for the limit of large qubit numbers. In the thermodynamics limit, the kerr type nonlinearity is induced from "magnon"-resonator coupling, and the selective preparation of "magnon" Fock states can be studied under "magnon" scenario.

quant-ph↗

arXiv2021

Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis

The rapid advances in deep generative models over the past years have led to highly {realistic media, known as deepfakes,} that are commonly indistinguishable from real to human eyes. These advances make assessing the authenticity of visual data increasingly difficult and pose a misinformation threat to the trustworthiness of visual content in general. Although recent work has shown strong detection accuracy of such deepfakes, the success largely relies on identifying frequency artifacts in the generated images, which will not yield a sustainable detection approach as generative models continue evolving and closing the gap to real images. In order to overcome this issue, we propose a novel fake detection that is designed to re-synthesize testing images and extract visual cues for detection. The re-synthesis procedure is flexible, allowing us to incorporate a series of visual tasks - we adopt super-resolution, denoising and colorization as the re-synthesis. We demonstrate the improved effectiveness, cross-GAN generalization, and robustness against perturbations of our approach in a variety of detection scenarios involving multiple generators over CelebA-HQ, FFHQ, and LSUN datasets. Source code is available at https://github.com/SSAW14/BeyondtheSpectrum.

cs.CV↗

arXiv2021

KNN, An Underestimated Model for Regional Rainfall Forecasting

Regional rainfall forecasting is an important issue in hydrology and meteorology. This paper aims to design an integrated tool by applying various machine learning algorithms, especially the state-of-the-art deep learning algorithms including Deep Neural Network, Wide Neural Network, Deep and Wide Neural Network, Reservoir Computing, Long Short Term Memory, Support Vector Machine, K-Nearest Neighbor for forecasting regional precipitations over different catchments in Upstate New York. Through the experimental results and the comparison among machine learning models including classification and regression, we find that KNN is an outstanding model over other models to handle the uncertainty in the precipitation data. The data normalization methods such as ZScore and MinMax are also evaluated and discussed.

cs.LG↗

arXiv2021

6 nm super-resolution optical transmission and scattering spectroscopic imaging of carbon nanotubes using a nanometer-scale white light source

Optical hyperspectral imaging based on absorption and scattering of photons at the visible and adjacent frequencies denotes one of the most informative and inclusive characterization methods in material research. Unfortunately, restricted by the diffraction limit of light, it is unable to resolve the nanoscale inhomogeneity in light-matter interactions, which is diagnostic of the local modulation in material structure and properties. Moreover, many nanomaterials have highly anisotropic optical properties that are outstandingly appealing yet hard to characterize through conventional optical methods. Therefore, there has been a pressing demand in the diverse fields including electronics, photonics, physics, and materials science to extend the optical hyperspectral imaging into the nanometer length scale. In this work, we report a super-resolution hyperspectral imaging technique that simultaneously measures optical absorption and scattering spectra with the illumination from a tungsten-halogen lamp. We demonstrated sub-5 nm spatial resolution in both visible and near-infrared wavelengths (415 to 980 nm) for the hyperspectral imaging of strained single-walled carbon nanotubes (SWNT) and reconstructed true-color images to reveal the longitudinal and transverse optical transition-induced light absorption and scattering in the SWNTs. This is the first time transverse optical absorption in SWNTs were clearly observed experimentally. The new technique provides rich near-field spectroscopic information that had made it possible to analyze the spatial modulation of band-structure along a single SWNT induced through strain engineering.

physics.optics↗

arXiv2021

Deep Video Inpainting Detection

This paper studies video inpainting detection, which localizes an inpainted region in a video both spatially and temporally. In particular, we introduce VIDNet, Video Inpainting Detection Network, which contains a two-stream encoder-decoder architecture with attention module. To reveal artifacts encoded in compression, VIDNet additionally takes in Error Level Analysis frames to augment RGB frames, producing multimodal features at different levels with an encoder. Exploring spatial and temporal relationships, these features are further decoded by a Convolutional LSTM to predict masks of inpainted regions. In addition, when detecting whether a pixel is inpainted or not, we present a quad-directional local attention module that borrows information from its surrounding pixels from four directions. Extensive experiments are conducted to validate our approach. We demonstrate, among other things, that VIDNet not only outperforms by clear margins alternative inpainting detection methods but also generalizes well on novel videos that are unseen during training.

cs.CV↗

arXiv2020

GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models

Deep learning has achieved overwhelming success, spanning from discriminative models to generative models. In particular, deep generative models have facilitated a new level of performance in a myriad of areas, ranging from media manipulation to sanitized dataset generation. Despite the great success, the potential risks of privacy breach caused by generative models have not been analyzed systematically. In this paper, we focus on membership inference attack against deep generative models that reveals information about the training data used for victim models. Specifically, we present the first taxonomy of membership inference attacks, encompassing not only existing attacks but also our novel ones. In addition, we propose the first generic attack model that can be instantiated in a large range of settings and is applicable to various kinds of deep generative models. Moreover, we provide a theoretically grounded attack calibration technique, which consistently boosts the attack performance in all cases, across different attack settings, data modalities, and training configurations. We complement the systematic analysis of attack performance by a comprehensive experimental study, that investigates the effectiveness of various attacks w.r.t. model type and training configurations, over three diverse application scenarios (i.e., images, medical data, and location data).

cs.LG↗

arXiv2020

Long-Tailed Recognition Using Class-Balanced Experts

Deep learning enables impressive performance in image recognition using large-scale artificially-balanced datasets. However, real-world datasets exhibit highly class-imbalanced distributions, yielding two main challenges: relative imbalance amongst the classes and data scarcity for mediumshot or fewshot classes. In this work, we address the problem of long-tailed recognition wherein the training set is highly imbalanced and the test set is kept balanced. Differently from existing paradigms relying on data-resampling, cost-sensitive learning, online hard example mining, loss objective reshaping, and/or memory-based modeling, we propose an ensemble of class-balanced experts that combines the strength of diverse classifiers. Our ensemble of class-balanced experts reaches results close to state-of-the-art and an extended ensemble establishes a new state-of-the-art on two benchmarks for long-tailed recognition. We conduct extensive experiments to analyse the performance of the ensembles, and discover that in modern large-scale datasets, relative imbalance is a harder problem than data scarcity. The training and evaluation code is available at https://github.com/ssfootball04/class-balanced-experts.

cs.CV↗

arXiv2020

Inclusive GAN: Improving Data and Minority Coverage in Generative Models

Generative Adversarial Networks (GANs) have brought about rapid progress towards generating photorealistic images. Yet the equitable allocation of their modeling capacity among subgroups has received less attention, which could lead to potential biases against underrepresented minorities if left uncontrolled. In this work, we first formalize the problem of minority inclusion as one of data coverage, and then propose to improve data coverage by harmonizing adversarial training with reconstructive generation. The experiments show that our method outperforms the existing state-of-the-art methods in terms of data coverage on both seen and unseen data. We develop an extension that allows explicit control over the minority subgroups that the model should ensure to include, and validate its effectiveness at little compromise from the overall performance on the entire dataset. Code, models, and supplemental videos are available at GitHub.

cs.CV↗

arXiv2020

Light Nuclei Production in Au+Au Collisions at $\sqrt{s_{\mathrm{NN}}}$ = 5-200 GeV from JAM model

Light nuclei production is sensitive to the baryon density fluctuations and can be used to probe the QCD phase transition in relativistic heavy-ion collisions. In this work, we studied the production of proton, deuteron, triton in central Au+Au collisions at $\sqrt{s_{\mathrm{NN}}}$ = 5, 7.7, 11.5, 14.5, 19.6, 27, 39, 54.4, 62.4 and 200 GeV from a transport model (JAM). Based on the coalescence production of light nuclei, we calculated the energy dependence of rapidity density $dN/dy$ and particle ratios ($d/p$, $t/p$, and $t/d$). More importantly, the yield ratio $N_{t} \times N_{p} / N_{d}^{2}$, which is sensitive to the neutron density fluctuations, shows a flat energy dependence and cannot describe the non-monotonic trend observed by the STAR experiment. Based on the nucleon coalescence, this work can provide constraint and reference to search for the QCD critical point and/or first order phase transition with light nuclei production in future heavy-ion collision experiments.

nucl-th↗

arXiv2020

AI-Powered GUI Attack and Its Defensive Methods

Since the first Graphical User Interface (GUI) prototype was invented in the 1970s, GUI systems have been deployed into various personal computer systems and server platforms. Recently, with the development of artificial intelligence (AI) technology, malicious malware powered by AI is emerging as a potential threat to GUI systems. This type of AI-based cybersecurity attack, targeting at GUI systems, is explored in this paper. It is twofold: (1) A malware is designed to attack the existing GUI system by using AI-based object recognition techniques. (2) Its defensive methods are discovered by generating adversarial examples and other methods to alleviate the threats from the intelligent GUI attack. The results have shown that a generic GUI attack can be implemented and performed in a simple way based on current AI techniques and its countermeasures are temporary but effective to mitigate the threats of GUI attack so far.

cs.CR↗

arXiv2019

Search for the QCD Critical Point by Transverse Velocity Dependence of Anti-deuteron to Deuteron Ratio

We propose the transverse velocity ($β_T$) dependence of the anti-deuteron to deuteron ratio as a new observable to search for the QCD critical point in heavy-ion collisions. The QCD critical point can attract the system evolution trajectory in the QCD phase diagram, which is known as focusing effect. To quantify this effect, we employ thermal model and hadronic transport model to simulate the dynamical particle emission along a hypothetical focusing trajectory near critical point. We found the focusing effect can lead to anomalous $β_T$ dependence of $\bar{p}/p$, $\bar{d}/d$ and $^3\overline{\text{He}}/^3\text{He}$ ratios. We examined the $β_T$ dependence of $\bar{p}/p$ and $\bar{d}/d$ ratios of central Au+Au collisions at $\sqrt{s_{NN}}$ = 7.7 to 200 GeV measured by the STAR experiment at RHIC. Surprisingly, we only observe a negative slope in $β_T$ dependence of $\bar{d}/d$ ratio at $\sqrt{s_{NN}}$ = 19.6 GeV, which indicates the trajectory evolution has passed through the critical region. In the future, we could constrain the location of the critical point and/or width of the critical region by making precise measurements on the $β_T$ dependence of $\bar{d}/d$ ratio at different energies and rapidity.

nucl-th↗

arXiv2019

Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints

Recent advances in Generative Adversarial Networks (GANs) have shown increasing success in generating photorealistic images. But they also raise challenges to visual forensics and model attribution. We present the first study of learning GAN fingerprints towards image attribution and using them to classify an image as real or GAN-generated. For GAN-generated images, we further identify their sources. Our experiments show that (1) GANs carry distinct model fingerprints and leave stable fingerprints in their generated images, which support image attribution; (2) even minor differences in GAN training can result in different fingerprints, which enables fine-grained model authentication; (3) fingerprints persist across different image frequencies and patches and are not biased by GAN artifacts; (4) fingerprint finetuning is effective in immunizing against five types of adversarial image perturbations; and (5) comparisons also show our learned fingerprints consistently outperform several baselines in a variety of setups.

cs.CV↗

arXiv2019

Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture

This paper addresses the problem of interpolating visual textures. We formulate this problem by requiring (1) by-example controllability and (2) realistic and smooth interpolation among an arbitrary number of texture samples. To solve it we propose a neural network trained simultaneously on a reconstruction task and a generation task, which can project texture examples onto a latent space where they can be linearly interpolated and projected back onto the image domain, thus ensuring both intuitive control and realistic results. We show our method outperforms a number of baselines according to a comprehensive suite of metrics as well as a user study. We further show several applications based on our technique, which include texture brush, texture dissolve, and animal hybridization.

cs.CV↗