Searcharxiv⌕ Search

arXiv subjects

Jie Ren

Publications and source records attributed to Jie Ren.

At least 181 records · Page 10Linked to original sources
arXiv2022

Proving Common Mechanisms Shared by Twelve Methods of Boosting Adversarial Transferability

Although many methods have been proposed to enhance the transferability of adversarial perturbations, these methods are designed in a heuristic manner, and the essential mechanism for improving adversarial transferability is still unclear. This paper summarizes the common mechanism shared by twelve previous transferability-boosting methods in a unified view, i.e., these methods all reduce game-theoretic interactions between regional adversarial perturbations. To this end, we focus on the attacking utility of all interactions between regional adversarial perturbations, and we first discover and prove the negative correlation between the adversarial transferability and the attacking utility of interactions. Based on this discovery, we theoretically prove and empirically verify that twelve previous transferability-boosting methods all reduce interactions between regional adversarial perturbations. More crucially, we consider the reduction of interactions as the essential reason for the enhancement of adversarial transferability. Furthermore, we design the interaction loss to directly penalize interactions between regional adversarial perturbations during attacking. Experimental results show that the interaction loss significantly improves the transferability of adversarial perturbations.

cs.LG↗
arXiv2022

Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs

This paper aims to theoretically analyze the complexity of feature transformations encoded in piecewise linear DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN's complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN.

cs.LG↗
arXiv2022

Plex: Towards Reliability using Pretrained Large Model Extensions

A recent trend in artificial intelligence is the use of pretrained models for language and vision tasks, which have achieved extraordinary performance but also puzzling failures. Probing these models' abilities in diverse ways is therefore critical to the field. In this paper, we explore the reliability of models, where we define a reliable model as one that not only achieves strong predictive performance but also performs well consistently over many decision-making tasks involving uncertainty (e.g., selective prediction, open set recognition), robust generalization (e.g., accuracy and proper scoring rules such as log-likelihood on in- and out-of-distribution datasets), and adaptation (e.g., active learning, few-shot uncertainty). We devise 10 types of tasks over 40 datasets in order to evaluate different aspects of reliability on both vision and language domains. To improve reliability, we developed ViT-Plex and T5-Plex, pretrained large model extensions for vision and language modalities, respectively. Plex greatly improves the state-of-the-art across reliability tasks, and simplifies the traditional protocol as it improves the out-of-the-box performance and does not require designing scores or tuning the model for each task. We demonstrate scaling effects over model sizes up to 1B parameters and pretraining dataset sizes up to 4B examples. We also demonstrate Plex's capabilities on challenging tasks including zero-shot open set recognition, active learning, and uncertainty in conversational language understanding.

cs.LG↗
arXiv2022

Trap of Feature Diversity in the Learning of MLPs

In this paper, we focus on a typical two-phase phenomenon in the learning of multi-layer perceptrons (MLPs), and we aim to explain the reason for the decrease of feature diversity in the first phase. Specifically, people find that, in the training of MLPs, the training loss does not decrease significantly until the second phase. To this end, we further explore the reason why the diversity of features over different samples keeps decreasing in the first phase, which hurts the optimization of MLPs. We explain such a phenomenon in terms of the learning dynamics of MLPs. Furthermore, we theoretically explain why four typical operations can alleviate the decrease of the feature diversity.

cs.LG↗
arXiv2022

Defense Against Gradient Leakage Attacks via Learning to Obscure Data

Federated learning is considered as an effective privacy-preserving learning mechanism that separates the client's data and model training process. However, federated learning is still under the risk of privacy leakage because of the existence of attackers who deliberately conduct gradient leakage attacks to reconstruct the client data. Recently, popular strategies such as gradient perturbation methods and input encryption methods have been proposed to defend against gradient leakage attacks. Nevertheless, these defenses can either greatly sacrifice the model performance, or be evaded by more advanced attacks. In this paper, we propose a new defense method to protect the privacy of clients' data by learning to obscure data. Our defense method can generate synthetic samples that are totally distinct from the original samples, but they can also maximally preserve their predictive features and guarantee the model performance. Furthermore, our defense strategy makes the gradient leakage attack and its variants extremely difficult to reconstruct the client data. Through extensive experiments, we show that our proposed defense method obtains better privacy protection while preserving high accuracy compared with state-of-the-art methods.

cs.LG↗
arXiv2022

Why Adversarial Training of ReLU Networks Is Difficult?

This paper mathematically derives an analytic solution of the adversarial perturbation on a ReLU network, and theoretically explains the difficulty of adversarial training. Specifically, we formulate the dynamics of the adversarial perturbation generated by the multi-step attack, which shows that the adversarial perturbation tends to strengthen eigenvectors corresponding to a few top-ranked eigenvalues of the Hessian matrix of the loss w.r.t. the input. We also prove that adversarial training tends to strengthen the influence of unconfident input samples with large gradient norms in an exponential manner. Besides, we find that adversarial training strengthens the influence of the Hessian matrix of the loss w.r.t. network parameters, which makes the adversarial training more likely to oscillate along directions of a few samples, and boosts the difficulty of adversarial training. Crucially, our proofs provide a unified explanation for previous findings in understanding adversarial training.

cs.LG↗
arXiv2022

A Hybrid Josephson Transmission Line and Passive Transmission Line Routing Framework for Single Flux Quantum Logic

The Single Flux Quantum (SFQ) logic family is a novel digital logic as it provides ultra-fast and energy-efficient circuits. For large-scale SFQ circuit design, specialized electronic design automation (EDA) tools are required due to the differences in logic type, timing constraints and circuit architecture, in contrast to the CMOS logic. In order to improve the overall performance of an SFQ circuit, an efficient routing algorithm should be applied during the layout design to perform accurate timing adjustment for fixing hold violations and optimizing critical paths. Thus, a hybrid Josephson transmission line and passive transmission line routing framework is proposed. It consists of four main modules and an exploration of the potential timing performance based on the given layout placement. The proposed routing tool is demonstrated on seven testbench circuits. The obtained results demonstrate that the operating frequency is greatly improved, and all the hold violations are eliminated for each circuit.

cs.ET↗
arXiv2022

Pluralistic Image Completion with Probabilistic Mixture-of-Experts

Pluralistic image completion focuses on generating both visually realistic and diverse results for image completion. Prior methods enjoy the empirical successes of this task. However, their used constraints for pluralistic image completion are argued to be not well interpretable and unsatisfactory from two aspects. First, the constraints for visual reality can be weakly correlated to the objective of image completion or even redundant. Second, the constraints for diversity are designed to be task-agnostic, which causes the constraints to not work well. In this paper, to address the issues, we propose an end-to-end probabilistic method. Specifically, we introduce a unified probabilistic graph model that represents the complex interactions in image completion. The entire procedure of image completion is then mathematically divided into several sub-procedures, which helps efficient enforcement of constraints. The sub-procedure directly related to pluralistic results is identified, where the interaction is established by a Gaussian mixture model (GMM). The inherent parameters of GMM are task-related, which are optimized adaptively during training, while the number of its primitives can control the diversity of results conveniently. We formally establish the effectiveness of our method and demonstrate it with comprehensive experiments.

cs.CV↗
arXiv2022

Twisting the near-field radiative heat switch in hyperbolic antiferromagnets

We study the twisted control of the near-field radiative heat transfer between two hyperbolic antiferromagnetic insulators under external magnetic fields. We show that the near-field heat flux can be affected by both the twist angle $θ$ and the magnitude of the applied magnetic field with different broken symmetries. Irrespective of twist angle, the external magnetic field causes the radiative heat flux to change nonmonotonically, and the minimum heat flux can be found with the magnetic fields of about 1.5 T. Such nonmonotonic behavior is due to the fact that the magnetic field can radically change the nature of the magnon polaritons with time reversal symmetry breaking. The field not only affects the topological structure of surface magnon polaritons, but also induces the volume magnon polaritons that progressively dominate the heat transfer as the field increases. We further propose a twist-induced thermal switch device with inversion symmetry breaking, which can severely regulate radiative heat flux through different magnetic fields. Our findings account for a characteristic modulation of radiative heat transfer with implications for applications in dynamic thermal management.

cond-mat.mes-hall↗
arXiv2022

Theory of Topological Superconductivity in Doped IV-VI Semiconductors

We theoretically study potential unconventional superconductivity in doped AB-type IV-VI semi-conductors, based on a minimal effective model with interaction up to the next-nearest neighbors. According to the experimental implications, we focus on the spin-triplet channels and obtain the superconducting phase diagram with respect to the anisotropy of the Fermi surfaces and the interaction strength. Abundant nodal and nodeless states with different symmetry breaking appear in the phase diagram, and all the states are time reversal invariant and topologically nontrivial. Specifically, the various nodal superconducting ground states, dubbed as the topological Dirac superconductors, are featured by Dirac nodes in the bulk and Majorana arcs on the surface; among the full-gap states, there exist a mirror-symmetry-protected second-order topological superconductor state favoring helical Majorana hinge cones, and different first-order topological superconductor states supporting 4 surface Majorana cones. The experimental verification of the different kinds of superconducting ground states is also discussed.

cond-mat.supr-con↗
arXiv2022

Analytic critical points of charged Renyi entropies from hyperbolic black holes

We analytically study phase transitions of holographic charged Renyi entropies in two gravitational systems dual to the $\mathcal{N}=4$ super-Yang-Mills theory at finite density and zero temperature. The first system is the Reissner-Nordstrom-AdS$_5$ black hole, which has finite entropy at zero temperature. The second system is a charged dilatonic black hole in AdS$_5$, which has zero entropy at zero temperature. Hyperbolic black holes are employed to calculate the Renyi entropies with the entangling surface being a sphere. We perturb each system by a charged scalar field, and look for a zero mode signaling the instability of the extremal hyperbolic black hole. Zero modes as well as the leading order of the full retarded Green's function are analytically solved for both systems, in contrast to previous studies in which only the IR (near horizon) instability was analytically treated.

hep-th↗
arXiv2022

Analytic AC conductivities from holography

We find exact, analytic solutions of the holographic AC conductivity at arbitrary frequency $ω$ and temperature $T$, in contrast to previous works where the AC conductivity was analytically obtained usually at small $ω$ and $T$. These solutions enable us to study the analyticity properties of the current-current correlator $G(ω)$ in detail. The first system we study is the AdS$_5$ planar black hole with momentum dissipation, whose extremal limit has an AdS$_2$ factor. Then we study AdS$_4$ and AdS$_5$ Einstein-dilaton systems whose special cases are maximal gauged supergravities. The solutions show how the poles move and how branch cuts emerge as the temperature varies. As a byproduct, we obtain an R-current correlator in $\mathcal{N}=4$ super-Yang-Mills theory on a sphere at finite temperature in the large $N$ and strong coupling limit.

hep-th↗
arXiv2022

Proximal Quasi-Newton Methods for Multiobjective Optimization Problems

We introduce some new proximal quasi-Newton methods for unconstrained multiobjective optimization problems (in short, UMOP), where each objective function is the sum of a twice continuously differentiable strongly convex function and a proper lower semicontinuous convex but not necessarily differentiable function. We propose proximal BFGS method, proximal self-scaling BFGS method, and proximal Huang BFGS method for (UMOP) with both line searches and without line searches cases. Under mild assumputions, we show that each accumulation point of the sequence generated by these algorithms, if exists, is a Pareto stationary point of the (UMOP). Moreover, we present their applications in both constrained multiobjective optimization problems and robust multiobjective optimization problems. In particular, for robust multiobjective optimization problems, we show that the subproblems of proximal quasi-Newton algorithms can be regarded as quadratic minimization problems with quadratic inequality constraints. Numerical experiments are also carried out to verify the effectiveness of the proposed proximal quasi-Newton methods.

math.OC↗
arXiv2022

Long-Range Order and Quantum Criticality in Antiferromagnetic Chains with Long-Range Staggered Interactions

We study quantum phase transitions in Heisenberg antiferromagnetic chains with a staggered power-law decaying long-range interactions. Employing the density-matrix renormalization group (DMRG) algorithm and the fidelity susceptibility as the criticality measure, we establish more accurate values of quantum critical points than the results obtained from the spin-wave approximation, quantum Monte Carlo and DMRG in literatures. The deviation is especially evident for strong long-range interactions. We extend isotropic long-range interactions to the anisotropic cases and find that kaleidoscope of quantum phases emerge from the interplay of anisotropy of the long-range exchange interaction and symmetry breaking. We demonstrate nonfrustrating long-range interactions induce the true long-range order in Heisenberg antiferromagnetic chains with a continuous symmetry breaking, lifting the restrictions imposed by the Mermin-Wagner theorem.

cond-mat.str-el↗
arXiv2022

Exploring the Impact of Negative Samples of Contrastive Learning: A Case Study of Sentence Embedding

Contrastive learning is emerging as a powerful technique for extracting knowledge from unlabeled data. This technique requires a balanced mixture of two ingredients: positive (similar) and negative (dissimilar) samples. This is typically achieved by maintaining a queue of negative samples during training. Prior works in the area typically uses a fixed-length negative sample queue, but how the negative sample size affects the model performance remains unclear. The opaque impact of the number of negative samples on performance when employing contrastive learning aroused our in-depth exploration. This paper presents a momentum contrastive learning model with negative sample queue for sentence embedding, namely MoCoSE. We add the prediction layer to the online branch to make the model asymmetric and together with EMA update mechanism of the target branch to prevent the model from collapsing. We define a maximum traceable distance metric, through which we learn to what extent the text contrastive learning benefits from the historical information of negative samples. Our experiments find that the best results are obtained when the maximum traceable distance is at a certain range, demonstrating that there is an optimal range of historical information for a negative sample queue. We evaluate the proposed unsupervised MoCoSE on the semantic text similarity (STS) task and obtain an average Spearman's correlation of $77.27\%$. Source code is available at https://github.com/xbdxwyh/mocose.

cs.CL↗
arXiv2022

DrugOOD: Out-of-Distribution (OOD) Dataset Curator and Benchmark for AI-aided Drug Discovery -- A Focus on Affinity Prediction Problems with Noise Annotations

AI-aided drug discovery (AIDD) is gaining increasing popularity due to its promise of making the search for new pharmaceuticals quicker, cheaper and more efficient. In spite of its extensive use in many fields, such as ADMET prediction, virtual screening, protein folding and generative chemistry, little has been explored in terms of the out-of-distribution (OOD) learning problem with \emph{noise}, which is inevitable in real world AIDD applications. In this work, we present DrugOOD, a systematic OOD dataset curator and benchmark for AI-aided drug discovery, which comes with an open-source Python package that fully automates the data curation and OOD benchmarking processes. We focus on one of the most crucial problems in AIDD: drug target binding affinity prediction, which involves both macromolecule (protein target) and small-molecule (drug compound). In contrast to only providing fixed datasets, DrugOOD offers automated dataset curator with user-friendly customization scripts, rich domain annotations aligned with biochemistry knowledge, realistic noise annotations and rigorous benchmarking of state-of-the-art OOD algorithms. Since the molecular data is often modeled as irregular graphs using graph neural network (GNN) backbones, DrugOOD also serves as a valuable testbed for \emph{graph OOD learning} problems. Extensive empirical studies have shown a significant performance gap between in-distribution and out-of-distribution experiments, which highlights the need to develop better schemes that can allow for OOD generalization under noise for AIDD.

cs.LG↗
arXiv2022

Quantum Many-Body Scars in Spin-1 Kitaev Chains

To provide a physical example of quantum scars, we study the many-body scars in the spin-1 Kitaev chain where the so-called PXP Hamiltonian is exactly embedded in the spectra. Regarding the conserved quantities, the Hilbert space is fragmented into disconnected subspaces and we explore the associated constrained dynamics. The continuous revivals of the fidelity and the entanglement entropy when the initial state is prepared in $\vert\mathbb{Z}_k\rangle$ ($k=2,3$) state illustrate the essential physics of the PXP model. We study the quantum phase transitions in the one-dimensional spin-1 Kitaev-Heisenberg model using the density-matrix renormalization group and Lanczos exact diagonalization methods, and determine the phase diagram. We parametrize the two terms in the Hamiltonian by the angle $ϕ$, where the Kitaev term is $K\equiv\sin(ϕ)$ and competes with the Heisenberg $J\equiv\cos(ϕ)$ term. One finds a rich ground state phase diagram as a function of the angle $ϕ$. Depending on the ratio $K/J\equiv\tan(ϕ)$, the system either breaks the symmetry to one of distinct symmetry broken phases, or preserves the symmetry in a quantum spin liquid phase with frustrated interactions. We find that the scarred state is stable for perturbations which obey $\mathbb{Z}_2$-symmetry, while it becomes unstable against Heisenberg-type perturbations.\\ \textit{Accepted for publication in Physical Review Research}

cond-mat.str-el↗